Archives

All posts by atagar

Hi wonderful world. Post-release followup and recent interview with Ben Collier has kept me pretty busy, but have a couple other fun things to report…


Packaging Community

Commonly major releases are followed by followup packaging work and Nyx’s recent release was no exception. But rather than simply work with our delightful packaging community as I usually do I decided instead to bring order to the chaos.

I’m delighted to say we now have a tor-packagers@ list where Tor developers can reach our packaging community, and packagers can subscribe to be notified of new releases. To go along with this I also made a wiki that gives an overview of our packages

packages wiki


Tor Relay: Caer Sidi

In folklore Caer Sidi was an otherworldly fortress, unsuccessfully assailed by the Prydwin in Arthur’s quest for the holy grail. It’s a name I’ve always wanted to use for a relay.

Flavor text aside, while ago I got permission from Dreamhost to run a non-exit relay on their cloud infrastructure so I’d have a busy relay on which to test Nyx. I haven’t tried pushing the envelope, but I can say it’s been a nice low-cost ($6/month) hosting experience thus far. It’s now listed as a good experience on our ISP wiki.

Ben Collier, PhD student with Edinburgh University recently conducted a survey with a number of Tor community members. I found a number of his questions interesting so with his permission sharing the interview here. Enjoy!


Tor office mural by Henry Ward
  1. How did you first get involved with the Tor Project, and with internet freedom activism more generally?

    I got involved with Tor through Google Summer of Code in 2009. Actually, think the application I wrote back then answers most of the questions for ‘why’.

    This application was not accepted. Tor took on several students, but unfortunately their top two picks (me and Runa) applied for the same task so they couldn’t accept us both. In the end though I’m glad for it since that’s how I first got involved authoring Nyx.

    Brenno Winter conducted an interview with me about this back in 2009.

  2. What are the main things you’ve worked on with the Tor Project? Which of these are you the most proud of? What are you working on at the moment, and what does an average day look like for you?

    Actually, Tommy recently wrote a blog post that answers this.

    In short my main involvement with Tor is Nyx (CLI monitor for tor relays) and Stem (Python library for Tor). I ran Tor’s GSoC program for six years but recently I turned in that hat, and now act as our membership secretary (onboarding new folks, maintaining parts of the site, etc).

    As for my average day, generally I do Tor stuff for a couple hours as I sip my morning coffee before work, and take on larger projects over the weekend. My day job is as a software engineer at Amazon.

  3. What would you say are your main motivations for the work you do?

    Tor is interesting in that there’s a wide range of interests in our community. For some its privacy, others counter-censorship or freedom of speech. For me my foremost interest is in Tor as an open source community.

  4. Do you see yourself as part of a Tor community? Do you think there is a strong community around Tor, and what are the main ways that the community interact with one another?

    Yup to both. Most common forms of communication are email, irc, and developer meetings.

  5. Do you think people in the Tor community are quite similar, or are there a lot of different views and perspectives?

    Good question. I’d say that our community is uniform in its libertarian bend. Tor is a privacy and counter-censorship tool after all, so everyone in our community tends to believe in privacy and free speech. That said, this still leaves quite a bit of room for differences. Everything from university academics to Iranian dissidents. And sometimes this leads to some healthy contentions.

    For instance, as mentioned above my foremost interest in Tor is as an open source community. I push heavily toward openness and doing all our work in public. This is somewhat antithetical though to how security and anonymity communities commonly operate. This tug-of-war is good though, with us meeting somewhere in the middle.

  6. What’s your process for doing development work on the tools you’ve worked on? Is it quite collaborative or do you tend to work on your own then feed back communally?

    Unfortunately the later. Tor has subcommunities (such as the core codebase, Guardian, Tails, etc) where multiple developers collaborate on a single codebase, but outside of that our person-to-project ratio is rather poor. I’m the sole developer on Stem and Nyx, which makes best practices such as code reviewing unfeasible.

  7. How do you organise development work in the Tor Project? Do you have a lot of autonomy to make decisions?

    Nick once called Tor a ‘do-ocracy’, which is to say that the person doing the work makes the decisions. For sections of tor where a single person does all-the-things (such as where I work) the person leading the charge has full autonomy. However, other sections where we have a larger developer population operate on different models.

    As for the internal community as a whole we’ve recently started using more formal voting procedures.

  8. What is the balance between feature development and maintenance? Where do new ideas tend to come from?

    This varies between projects and I can only speak to Stem and Nyx. Both of these projects were in the development two full years before their initial release, receiving a high degree of automated test coverage before seeing the light of day. As such maintenance has not been an especially large concern. This comes at the obvious cost though of a long development cycle.

  9. What are the main factors that you consider when making design decisions for your Tor Project work? Could you give me an example of a particularly important/interesting case where you had to make a decision, and how you made this?

    Hmmm, there’s quite a few different forms of design decisions (architecture, scalability, etc). For Stem the most relevant is API design so I’ll speak to that. To derive Stem’s API I dogfood the library (use it myself for projects), as well as keep an eye on how others are using it.

    I have quite a bit of experience when it comes to API design, but really the best way of discovering where rough edges are is to see a library used in practice, and asking yourself how differences in the library could make their code even better.

  10. When you’re making decisions about the projects you’ve worked on for Tor, at the design stages but also in your day-to-day work, do you find that your own personal values, and the values of the organisation, play a big role in these decisions? Do you think "values" are a useful way of talking about technology, and if so, what values do you think the technologies developed by the Tor Project represent?

    Hmmm. Can’t say it’s a foremost thought for me. Mostly I hack on code because I find it fun. This is a volunteer hobby for me, after all.

    I’ve noticed throughout my software engineering career that there’s a wide range of primary motivations. For some it’s impact – they want to change the world. For others it’s challenge. Personally I don’t lean toward those. My interests is in our community and doing quality work. The magnitude of impact isn’t a prime motivator for me – I don’t care overly much if my work greatly changes the world or not. Rather, I just care that the things I do are done well.

    I suppose that’s why I lean toward support and infrastructure roles.

  11. Do you find your other work (e.g. at Amazon) complements your work at Tor, or are they quite separate?

    The two synergize well in that tricks I pick up with Tor tend to benefit Amazon and vice versa. For instance, I first discovered the mocking framework Stem now uses during my dayjob, whereas an Amazon CLIs I develop benefited from my work on Nyx.

    That said, I do also keep a degree of separation. Tor Cloud was a project to provide Amazon cloud images to simplify relay setup. I made a point of not touching it with a ten foot pole. That said, honestly it’s never really been an issue. Work knows about Tor, Tor knows about work, and neither seems to care particularly much about the other.

  12. What are the challenges of onboarding new staff, especially when the organisation is going through a phase of expansion?

    Just the time to discuss 1:1 about what they’d like on the website, get them set up on irc, etc.

  13. The project is Open Source – why do you think this is particularly important, and what benefits and challenges do you find this poses? How does the Tor Project balance the competing views on this – between openness and more traditional approaches to security development?

    Open source is necessary for Tor as a matter of trust. Users depend on Tor to keep them safe, both in their private lives and even more critically in oppressive regimes. If Tor were an uninspectable black box would you trust it? I wouldn’t.

    Tor’s whole design is architected around distributed trust. No single relay knows your identity, no directory authority can mess with you, and by keeping the code open source we can’t impair your anonymity
    either.

    Generally speaking we error toward openness. Exceptions only arise when there’s a need for secrecy. For instance, tor-security@ where sensitive security issues are reported. Another is malicious relay detection so bad actors don’t learn how they’re being caught. However, even those become public eventually (security reports once a fix is available, and the bad relay blacklist is largely public).

  14. What would you say are the main ways that you’ve seen the Tor Project change as an organisation since you’ve been involved?

    Thanks to Shari (our executive director) Tor organizationally has greatly matured. Far less angst about job security and funding for folks employed the 501c3 side. As for the community side we’ve grown. One growing pain has been decision making as it turns out consensus doesn’t scale. Ever tried getting unanimous agreement from ninety people on a contentious topic? That… doesn’t work. As such we now have a formalized voting procedure for decision making.

  15. Do you see law enforcement as posing barriers to the work of the Tor Project? Do you think they understand Tor and its goals?

    Nope. I don’t see law enforcement as an enemy and hope they don’t see us as one either. Roger and others engage with the law enforcement community and we provide tools like Exonerator to make their lives (and lives of relay operators that don’t want kicked down doors) better.

    Just speaking for myself, I was glad to see the Silk Road takedown demonstrate that traditional policing (money trails, informants, etc) still work when it comes to bad actors on Tor. Criminal enterprises have always had IP level privacy through botnets. Our goal is to counteract bulk surveillance and provide individual privacy which I hopefully many (though understandably not all) in the law enforcement community can get behind too.

  16. Are you worried about the potential of governments cracking down on Tor and encryption technologies?

    Not my top concern. True, the Crypto wars of the nineties demonstrated that governments can take a laughably ill-conceived stance when it comes to encryption, but thankfully the Internet is global. Even if the US takes a backward stance in this regard EU jurisdictions don’t seem to be showing any sign of following suit.

  17. Does working on Tor mean you need to be more careful in your own day-to-day online security practices?

  18. Not in particular. I don’t involve myself with anything highly sensitive so don’t think I’m a particularly juicy target. Just about all my Tor involvement is public anyway.

  19. The media has in the past tended to focus on negative stories about Tor, and Hidden Services in particular. Does this bother you, and do you think how Tor is perceived is a problem? Do you think there are any ways to deal with “misuse” of the network that are appropriate, or possible?

    Good questions. While I can certainly see good uses for Hidden Services (such as dissident blogs) I can’t say it’s a feature I’ve found very compelling. As such my work doesn’t focus on it. Lot of others though see promise in it. You’re right that more than anything else within Tor it draws negative press. Kinda irked when that overshadows the rest of what Tor does but oh well, them’s the breaks.

  20. Tor has many uses – from protecting privacy-conscious citizens, to whistleblowing, to fighting censorship in repressive regimes, to fighting corporate surveillance. Which do you personally think are the most important?

    For me personally: privacy and free speech. Authoritarianism requires censorship to survive. It’s unsettling but unsurprising to see denouncement of our free press now that we’re getting our own little dose of extremism here in the US. Freedom of information both press and Tor support are necessary for an informed democracy.

  21. The relay operators I’ve spoken to had very diverse political views, but all had very positive views of the project and the main team. How do you keep the community engaged and happy? What challenges do you face in doing this?

    Glad to hear it, we love our wonderful relay community too!

    Relay operators are just as much a part of our community as developers, activists, and everyone else. By working in the open hopefully they feel included. Actually, just last month our relay community helped me beta test Nyx. They really did a remarkable job putting it through the ringer. Many thanks to ’em for all their help!

  22. What do you think are the main challenges facing the project in the near future?

    Funding diversity and promoting positive uses of Hidden Services are a couple that come to mind.

  23. What would you say, with regards to Tor, is the thing that you’re most excited about in the near future?

    Pity you didn’t ask me last month. I’d say the release of Nyx. Now brainstorming my next project so we’ll see.

  24. Final question – are you generally optimistic or pessimistic about the future for internet freedom?

    In the short term I’m optimistic. Response to the Snowden revelations showed tremendous public interest in defending digital civil liberties.

    But long term I’m worried. Not about government or three letter agencies, but advertising. Market forces and Moore’s law makes bulk surveillance both easier and more profitable every year. Maybe I underestimate the public’s desire for privacy, but when offered convenience in exchange for it I’m uncomfortable thinking where we might end up.

Seems I fail at sending these reports on time. Oh well, on with the show!


Nyx and Stem Release

As you may have guessed from last week’s announcement been busy with the launch of Nyx and Stem 1.6!

Very special thanks to…

  • Tommy and Stephanie for their delightful blog post.
  • All the relay operators that helped beta test Nyx.
  • George, Attila, and pascal for our new Stem and Nyx OpenBSD ports.
  • Derek, Juan, Anthony, Sjon, Alex, Markus, and Carlo for updating Stem and packaging Nyx for all our existing platforms. Damn you guys are fast!

Again, thanks all! Each of these made me do a little happy dance.


Montreal Developer Meeting

It’s the sign of a busy month if this isn’t my top highlight. As always Jon, Gunner, and Alison orchestrated a great meeting. Between discussions hit the sights and tried poutine for the first time.

Montreal dev meeting

As a final note the Tor office moved this month. Sadly this means we’re losing an absolutely fantastic mural done for us by Henry. However, we took some high definition photos. Here’s one I’ve rescaled to be usable as a wallpaper or poster…

Nyx

“When you do things right, people won’t be sure you’ve done anything at all.” -Futurama

After five years Nyx (previously known as arm) has gotten a long belated update. Under the covers our whole codebase has been rewritten from the ground up, but for users changes are subtle…

  • New website
  • Python 3.x support.
  • Bandwidth graph now prepopulates when you start up, so you have a graph right away.
  • Connections are now available without toggling DisableDebuggerAttachment in your torrc.
  • Support for showing IPv6 connections.
  • Dialog for picking tor events to log, rather than an arcane letter flag input.
  • Improved efficiency of log deduplication by multiple orders of magnitude. As such verbose logs no longer peg your CPU.
  • Richer control interpreter, including a python prompt like IDLE.
  • Removed features that frequently confused users such as the relay setup wizard and torrc validation.
  • Modernized dependencies, replacing TorCtl with Stem.

Hand-in-hand with Nyx I’m also pleased to announce Stem 1.6! A full year of improvements including descriptor creation support, ed25519 certificates, sizable performance tuning, and much, much more.

So how do I get started?

It’s easy! Just configure tor to accept a controller

% cat ~/.tor/torrc
ControlPort 9051
CookieAuthentication 1

… install Nyx…

% sudo pip install nyx

… and run!

% nyx

Nyx

Oh how I love Sol Duc. Hundreds of miles of trails, white birch groves, and of course sulfuric hotsprings. Rainforest moss lends the woods an unearthly charm that’s truly just delightful.

Highlight of this month for me was a vacation with my dad, first to Port Townsend then Sol Duc. But this has been an interesting month on other fronts too.


Nyx Performance

This month my chief focus has been tuning Nyx. CPU usage is now 20% lower, and far more responsive under verbose logging due to constant time log deduplication. Overall Nyx finally looks ready for release. I’ll probably move forward with a call for beta testers after the dev meeting.


Membership

This month tor ratified a policy for internal list membership. Much of it just formalizes procedures we’ve used for a while, but it also adds a requirement on continued involvement to stay subscribed. Naturally volunteers move on to other projects over time and this perfectly fine, but eventually their membership will be suspended with re-addition fast tracked if they return.

I’m now facilitating discussions with the Vegas leads and Council on how best to determine this. To give us a starting point I put together some fun stats, but numbers alone don’t come anywhere close to answering the question of “how has this person been involved in making tor better in the last six months?”. Discussions ongoing.

Hi all. This month been down in the engine room productionizing Nyx, with a special focus on memory usage. Dropped ~13%, but still not where I want so investigations are ongoing.


Nyx SQLite Cache

Main benefit came from moving cached consensus information from memory to SQLite. Besides the obvious memory benefits this allows the cache to persist between invocations, halving Nyx’s startup time (from 0.7 to 0.3 seconds).


Tor Manual Database

Stem provides easy programmatic access for Tor’s manual information. SQLite now backs this information, providing 8x faster reads (Manual.from_cache() dropped 16ms to 2ms), and now supports random access reads…

>>> import stem.manual
>>> print(stem.manual.query('SELECT description FROM torrc WHERE key=?', 'CONTROLSOCKET').fetchone()[0])
Like ControlPort, but listens on a Unix domain socket, rather than a TCP socket.  0 disables ControlSocket (Unix and Unix-like systems only.)

This further drops Nyx’s memory usage by allowing it to only fetch the manual information it needs.


Stem Multi-Processing

May’s test performance investigation has now led to a general purpose DaemonTask class to make Python multi-processing easy…

Threaded

import threading
import time

def fibonacci(n):
  if n < 2:
    return n
  else:
    return fibonacci(n-2) + fibonacci(n-1)

# calculate fibonacci sequences four times in parallel

start_time, threads = time.time(), []

for i in range(4):
  t = threading.Thread(target = fibonacci, args = (35,))
  t.setDaemon(True)
  t.start()

  threads.append(t)

for t in threads:
  t.join()

print('took %0.1f seconds' % (time.time() - start_time))
% python fibonacci_threaded.py
took 21.1 seconds

Multi-Process

import stem.util.system
import time

def fibonacci(n):
  if n < 2:
    return n
  else:
    return fibonacci(n-2) + fibonacci(n-1)

# calculate fibonacci sequences four times in parallel

start_time, threads = time.time(), []

for i in range(4):
  threads.append(stem.util.system.DaemonTask(fibonacci, (35,), start = True))

for t in threads:
  t.join()

print('took %0.1f seconds' % (time.time() - start_time))
% python fibonacci_multiprocessing.py
took 6.2 seconds

Presently this is only used for our tests, but soon I'll take advantage of this to make Nyx more performant on multi-core systems.

Hi all. This month pretended to be a web developer and made pretty things. Not an area I dabble in often but it was fun doing something artsy.


New Nyx Website

My eyes, they bleed! Made arm’s old site back in my college days and it shows…

old arm site

I’m delighted to announce Nyx’s new site is now live! New paint job aside our new site describes Nyx’s features, adds a FAQ, and better changelog.

new nyx site


Tor People Page

So many new people. Between newhires and new volunteers Tor is growing quite a bit right now and a common complaint I’ve heard is that our community makes it maddeningly hard to figure out who’s who. For some of us this is intentional, but others it’s not…

Who’s that arma guy I was talking with on irc? He seemed nice. Ok, thanks. Now which of these meeting attendees is Roger? Great. And, what does he do?“. Multiply that by ninety of us and it’s no wonder we drive our lovely hair-pulling newcomers to early baldness.

For those that would care to take part our people page now provides irc nicks, descriptions, pgp keys, twitter handles, and pictures. Hope this helps ya get a better idea of who those disembodied voices on irc are!

new nyx site

Hi all! Things I’m jugging are still in the air so this will be short. So what did I do this month besides buy a very, very manly puffer fish?

… sorry, those buggie eyes keep making me smile.


Migrate BridgeDB to Stem

Stem can now sign server and extrainfo descriptors! This was the last bit we needed for Leekspin parity. Isis and I are now swapping BridgeDB over.

After that’s been merged for a week or so I’ll likely cut the Stem 1.6 release. We have quite a few improvements accumulated so it’s about time…


Smaller things this month include…

Oh, and of course attended a local Retro Gaming Expo, Fremont Festival, great performance of Cabarat, and a coworker’s BBQ. Hope everyone else is having a similarly fun summer!

Hi all! Wanted to get my Pycon trip report out first but work keeps preempting that, so time to stop blocking. Lot of great stuff! But I suppose considering we’re already half way into June that’s no surprise…


Pycon and Stem Test Performance

Stay tuned for the full report, but highlights for me were discovery of mypy and multiprocessing.

Mypy is a static type checker with output similar to pyflakes or pycodestyle. Beginning with Python 3.5 core language syntax adds optional type information, and prior versions can provide this with comments. Mypy checks that function invocations match these types, providing type checks similar to what you’d get from static compilers like java.

I spent a bit integrating mypy with stem’s test suite but concluded it’s not ready for prime time quite yet. Mypy’s site cautions it’s still in early beta and indeed quite a few parts of stem (like enums) trip it up.

Multiprocessing on the other hand has been a big help.

Mulitprocessing is a python builtin that provides thread-style invocation of subprocesses. This is particularly interesting to me because for months I’ve struggled against the GIL for speeding up Stem’s tests. Multiprocessing is designed for just that, providing easy parallelization across cores.

Time since Pycon’s been spent integrating multiprocessing into our test suite, adding an @asynchronous decorator that invokes tests via subprocesses. On my antique dual core system this drops our test runtime from 53 to 22 seconds (58% faster). Should get even more on modern systems.

Evenings hit the Portland sights and tasty, tasty alcohol with my dad. Oh, and of course discussed Tor with Guido. It was a very fun trip.


Descriptor Creation

Stem and BridgeDB have always supported descriptor creation for their tests, with Isis even going the extra mile to make Leekspin. Taking the best of both Stem now supports a simple create() and content() function on descriptor classes…

RelayDescriptor.create({'router': 'demo 127.0.0.1 80 0 0'})

Leekspin’s still the only game in town for descriptor signing, but I hope to incorporate this soon too. You can read up more on this feature in our tutorials.


Couple other things this month include…

  • Stem now validates tor consensus signatures. Many thanks to Tyler Parks for all his help making this happen!
  • Passed GSoC administration on to Colin. Well past due we had fresh blood here.

Hi all. Emergent work encroached into my weekends a bit so less progress on my projects. On the nice side though had fun writing a PoE build and saw Cirque Du Soleil: Luzia with my family.

luzia

As always their performance was fantastic.


Tor Internal Bylaws

Alison’s Social Contract passed, and we’re presently in the process of electing a new Community Council. To this end helped conduct a candidate Q&A.

For a variety of reasons some folks would prefer not to vote. Some don’t have time, while others feel uncomfortable with voting due to their position in TPI or Tails. As such we’re now allowing folks to opt-out of counting toward the quorum requirement if they want.


tor-prompt –run

Stem’s tor-prompt command now supports a –run argument to run individual commands…

--run example

… listen to a space separated list of events with –run ‘SETEVENTS [event_types]’

--run example for events

… pipe its output to other destinations…

--run example for piping

… and invoke files with a series of commands…

--run example for files