Hey there wonderful world, ’tis been a year since my last post.

Yesterday my family had some local excitement. Amid a mighty crash a tree fell in front of our house. It blocked the road and pancaked our neighborhood’s mailboxes.

Fallen tree

I’m glad cuz it was heart-warming serendipity. Within ten minutes a dozen of us from the neighborhood were cleaning it up. Folks then rebuilt the mailboxes the next day. I love Vashon. It’s such a friendly place to live.

I’m taking a break from open source to selfishly dabble in writing fiction. Amazon managers used to compliment me on my writing skills but like my coding I’m meticulous. After a full year I’m only 12,000 words into a book. By obsessively polishing each paragraph I’m comically slow.

Is it time efficient? Nay. Fun? Absolutely. I spent a week reading Shakespeare to attempt archaic dialog (dost thou wot such uneath quoths?). I also whipped up a python script to improve word diversity. I’m such an engineer…

My last experience with Tor and Wikipedia were disappointing so I’ll continue to focus on non-technical projects for a while. I still volunteer at Vashon’s Food Bank and Granny’s Attic. But at some point I’d like to get back into coding. That’s my trained profession after all. But I’ve discovered the importance of being appreciated within a project so TBD on where it’ll be.

Happy summer everyone! Washington sizzled this month amid a historic heat dome. Willis Carrier, you are a national treasure.

I love AC

Pywikibot Scripts

To parallelize rampup with development I sunk this month into our scripts. I plan to refactor every script and expand their test coverage so this will be an ongoing project. This month included…




Deprecation Policy

In consultation with Xqt and JJMC89 I wrote a deprecation policy for pywikibot. From version 6.4.0 onward pywikibot will use symantic versioning, and assure backward compatibility for minor version bumps.

I also dropped our use of DeprecationWarnings since invisible notifications… aren’t particularly helpful.

Verbosity Reduction

Last, but probably my favorite work this month, I greatly reduced the verbosity of our test and sphinx build output.

My test reduction cropped our unit tests from 536 lines to ~20 (96% reduction). Documentation compilation dropped from 120 lines to 48 (60% reduction).

Hey everyone. This morning my grandma passed away peacefully in her sleep at 6am. She was ninety years old and ready to go. As the saying goes: “old age ain’t for sissies”.

These last few months supporting my family has been a full time job, but my schedule should now free back up.


CollecTor Support

The only technical thing to note is CollecTor support, a new module within Stem that makes gathering Tor’s network topology at prior points in time simple. For example…

import datetime
import stem.descriptor.collector

yesterday = datetime.datetime.utcnow() - datetime.timedelta(days = 1)

# provide yesterday's exits

for desc in stem.descriptor.collector.get_server_descriptors(start = yesterday):
  if desc.exit_policy.is_exiting_allowed():
    print('  %s (%s)' % (desc.nickname, desc.fingerprint))

Freedom! Sweet, delicious freedom! I went dark since March for crunch time at Amazon, but then decided one decade there was enough. Amazon was a great first job but time for a break.

Vacation aside, this summer I’m volunteering with our island food bank and Granny’s Attic to get more balance away from the keyboard. That said, knowing myself I’ll still need my open source fix.

Oh! And of course, most important of all: found a new puffy friend!

mini puffer

Damn I’m a sucker for puffers.

Tor/Stem CI

Stem’s tests were designed to verify Tor/Stem’s interoperability, as well as provide a tool to test Tor itself. This month teor integrated Stem into Tor’s Travis CI and we’ve been working out the kinks…

Smaller things these last few months include…

Ben Collier, PhD student with Edinburgh University recently conducted a survey with a number of Tor community members. I found a number of his questions interesting so with his permission sharing the interview here. Enjoy!

Tor office mural by Henry Ward
  1. How did you first get involved with the Tor Project, and with internet freedom activism more generally?

    I got involved with Tor through Google Summer of Code in 2009. Actually, think the application I wrote back then answers most of the questions for ‘why’.

    This application was not accepted. Tor took on several students, but unfortunately their top two picks (me and Runa) applied for the same task so they couldn’t accept us both. In the end though I’m glad for it since that’s how I first got involved authoring Nyx.

    Brenno Winter conducted an interview with me about this back in 2009.

  2. What are the main things you’ve worked on with the Tor Project? Which of these are you the most proud of? What are you working on at the moment, and what does an average day look like for you?

    Actually, Tommy recently wrote a blog post that answers this.

    In short my main involvement with Tor is Nyx (CLI monitor for tor relays) and Stem (Python library for Tor). I ran Tor’s GSoC program for six years but recently I turned in that hat, and now act as our membership secretary (onboarding new folks, maintaining parts of the site, etc).

    As for my average day, generally I do Tor stuff for a couple hours as I sip my morning coffee before work, and take on larger projects over the weekend. My day job is as a software engineer at Amazon.

  3. What would you say are your main motivations for the work you do?

    Tor is interesting in that there’s a wide range of interests in our community. For some its privacy, others counter-censorship or freedom of speech. For me my foremost interest is in Tor as an open source community.

  4. Do you see yourself as part of a Tor community? Do you think there is a strong community around Tor, and what are the main ways that the community interact with one another?

    Yup to both. Most common forms of communication are email, irc, and developer meetings.

  5. Do you think people in the Tor community are quite similar, or are there a lot of different views and perspectives?

    Good question. I’d say that our community is uniform in its libertarian bend. Tor is a privacy and counter-censorship tool after all, so everyone in our community tends to believe in privacy and free speech. That said, this still leaves quite a bit of room for differences. Everything from university academics to Iranian dissidents. And sometimes this leads to some healthy contentions.

    For instance, as mentioned above my foremost interest in Tor is as an open source community. I push heavily toward openness and doing all our work in public. This is somewhat antithetical though to how security and anonymity communities commonly operate. This tug-of-war is good though, with us meeting somewhere in the middle.

  6. What’s your process for doing development work on the tools you’ve worked on? Is it quite collaborative or do you tend to work on your own then feed back communally?

    Unfortunately the later. Tor has subcommunities (such as the core codebase, Guardian, Tails, etc) where multiple developers collaborate on a single codebase, but outside of that our person-to-project ratio is rather poor. I’m the sole developer on Stem and Nyx, which makes best practices such as code reviewing unfeasible.

  7. How do you organise development work in the Tor Project? Do you have a lot of autonomy to make decisions?

    Nick once called Tor a ‘do-ocracy’, which is to say that the person doing the work makes the decisions. For sections of tor where a single person does all-the-things (such as where I work) the person leading the charge has full autonomy. However, other sections where we have a larger developer population operate on different models.

    As for the internal community as a whole we’ve recently started using more formal voting procedures.

  8. What is the balance between feature development and maintenance? Where do new ideas tend to come from?

    This varies between projects and I can only speak to Stem and Nyx. Both of these projects were in the development two full years before their initial release, receiving a high degree of automated test coverage before seeing the light of day. As such maintenance has not been an especially large concern. This comes at the obvious cost though of a long development cycle.

  9. What are the main factors that you consider when making design decisions for your Tor Project work? Could you give me an example of a particularly important/interesting case where you had to make a decision, and how you made this?

    Hmmm, there’s quite a few different forms of design decisions (architecture, scalability, etc). For Stem the most relevant is API design so I’ll speak to that. To derive Stem’s API I dogfood the library (use it myself for projects), as well as keep an eye on how others are using it.

    I have quite a bit of experience when it comes to API design, but really the best way of discovering where rough edges are is to see a library used in practice, and asking yourself how differences in the library could make their code even better.

  10. When you’re making decisions about the projects you’ve worked on for Tor, at the design stages but also in your day-to-day work, do you find that your own personal values, and the values of the organisation, play a big role in these decisions? Do you think “values” are a useful way of talking about technology, and if so, what values do you think the technologies developed by the Tor Project represent?

    Hmmm. Can’t say it’s a foremost thought for me. Mostly I hack on code because I find it fun. This is a volunteer hobby for me, after all.

    I’ve noticed throughout my software engineering career that there’s a wide range of primary motivations. For some it’s impact – they want to change the world. For others it’s challenge. Personally I don’t lean toward those. My interests is in our community and doing quality work. The magnitude of impact isn’t a prime motivator for me – I don’t care overly much if my work greatly changes the world or not. Rather, I just care that the things I do are done well.

    I suppose that’s why I lean toward support and infrastructure roles.

  11. Do you find your other work (e.g. at Amazon) complements your work at Tor, or are they quite separate?

    The two synergize well in that tricks I pick up with Tor tend to benefit Amazon and vice versa. For instance, I first discovered the mocking framework Stem now uses during my dayjob, whereas an Amazon CLIs I develop benefited from my work on Nyx.

    That said, I do also keep a degree of separation. Tor Cloud was a project to provide Amazon cloud images to simplify relay setup. I made a point of not touching it with a ten foot pole. That said, honestly it’s never really been an issue. Work knows about Tor, Tor knows about work, and neither seems to care particularly much about the other.

  12. What are the challenges of onboarding new staff, especially when the organisation is going through a phase of expansion?

    Just the time to discuss 1:1 about what they’d like on the website, get them set up on irc, etc.

  13. The project is Open Source – why do you think this is particularly important, and what benefits and challenges do you find this poses? How does the Tor Project balance the competing views on this – between openness and more traditional approaches to security development?

    Open source is necessary for Tor as a matter of trust. Users depend on Tor to keep them safe, both in their private lives and even more critically in oppressive regimes. If Tor were an uninspectable black box would you trust it? I wouldn’t.

    Tor’s whole design is architected around distributed trust. No single relay knows your identity, no directory authority can mess with you, and by keeping the code open source we can’t impair your anonymity

    Generally speaking we error toward openness. Exceptions only arise when there’s a need for secrecy. For instance, tor-security@ where sensitive security issues are reported. Another is malicious relay detection so bad actors don’t learn how they’re being caught. However, even those become public eventually (security reports once a fix is available, and the bad relay blacklist is largely public).

  14. What would you say are the main ways that you’ve seen the Tor Project change as an organisation since you’ve been involved?

    Thanks to Shari (our executive director) Tor organizationally has greatly matured. Far less angst about job security and funding for folks employed the 501c3 side. As for the community side we’ve grown. One growing pain has been decision making as it turns out consensus doesn’t scale. Ever tried getting unanimous agreement from ninety people on a contentious topic? That… doesn’t work. As such we now have a formalized voting procedure for decision making.

  15. Do you see law enforcement as posing barriers to the work of the Tor Project? Do you think they understand Tor and its goals?

    Nope. I don’t see law enforcement as an enemy and hope they don’t see us as one either. Roger and others engage with the law enforcement community and we provide tools like Exonerator to make their lives (and lives of relay operators that don’t want kicked down doors) better.

    Just speaking for myself, I was glad to see the Silk Road takedown demonstrate that traditional policing (money trails, informants, etc) still work when it comes to bad actors on Tor. Criminal enterprises have always had IP level privacy through botnets. Our goal is to counteract bulk surveillance and provide individual privacy which I hopefully many (though understandably not all) in the law enforcement community can get behind too.

  16. Are you worried about the potential of governments cracking down on Tor and encryption technologies?

    Not my top concern. True, the Crypto wars of the nineties demonstrated that governments can take a laughably ill-conceived stance when it comes to encryption, but thankfully the Internet is global. Even if the US takes a backward stance in this regard EU jurisdictions don’t seem to be showing any sign of following suit.

  17. Does working on Tor mean you need to be more careful in your own day-to-day online security practices?

  18. Not in particular. I don’t involve myself with anything highly sensitive so don’t think I’m a particularly juicy target. Just about all my Tor involvement is public anyway.

  19. The media has in the past tended to focus on negative stories about Tor, and Hidden Services in particular. Does this bother you, and do you think how Tor is perceived is a problem? Do you think there are any ways to deal with “misuse” of the network that are appropriate, or possible?

    Good questions. While I can certainly see good uses for Hidden Services (such as dissident blogs) I can’t say it’s a feature I’ve found very compelling. As such my work doesn’t focus on it. Lot of others though see promise in it. You’re right that more than anything else within Tor it draws negative press. Kinda irked when that overshadows the rest of what Tor does but oh well, them’s the breaks.

  20. Tor has many uses – from protecting privacy-conscious citizens, to whistleblowing, to fighting censorship in repressive regimes, to fighting corporate surveillance. Which do you personally think are the most important?

    For me personally: privacy and free speech. Authoritarianism requires censorship to survive. It’s unsettling but unsurprising to see denouncement of our free press now that we’re getting our own little dose of extremism here in the US. Freedom of information both press and Tor support are necessary for an informed democracy.

  21. The relay operators I’ve spoken to had very diverse political views, but all had very positive views of the project and the main team. How do you keep the community engaged and happy? What challenges do you face in doing this?

    Glad to hear it, we love our wonderful relay community too!

    Relay operators are just as much a part of our community as developers, activists, and everyone else. By working in the open hopefully they feel included. Actually, just last month our relay community helped me beta test Nyx. They really did a remarkable job putting it through the ringer. Many thanks to ’em for all their help!

  22. What do you think are the main challenges facing the project in the near future?

    Funding diversity and promoting positive uses of Hidden Services are a couple that come to mind.

  23. What would you say, with regards to Tor, is the thing that you’re most excited about in the near future?

    Pity you didn’t ask me last month. I’d say the release of Nyx. Now brainstorming my next project so we’ll see.

  24. Final question – are you generally optimistic or pessimistic about the future for internet freedom?

    In the short term I’m optimistic. Response to the Snowden revelations showed tremendous public interest in defending digital civil liberties.

    But long term I’m worried. Not about government or three letter agencies, but advertising. Market forces and Moore’s law makes bulk surveillance both easier and more profitable every year. Maybe I underestimate the public’s desire for privacy, but when offered convenience in exchange for it I’m uncomfortable thinking where we might end up.

Hi all. Next weekend I’m returning to ye olde isle of Vashon to feast with family, so figured I might as well send this out early. I’ll be in a Thanksgiving food coma this time next week, after all…

My devious scheme for November was to finish rewriting arm’s graph panel and, while that certainly made progress, Stem, DocTor, and internal discussions stole the spotlight this month…


  • We now notify directory authority operators directly of issues in addition to the tor-consensus-health@ list. This should reduce duration of outages and other issues with the authorities.
  • We replaced turtles with longclaw. Sebastian deserves major kudos for orchestrating this.
  • Properly fixed DocTor’s OOM issues by not shelling out to send notification emails.
  • DocTor detected a burst of relays from Google App Engine. These relays lacked any contact information so we dropped them from the network as a potential Sybil attack. If you’re the operator of these then please let us know! We’d be delighted to add you back in once there’s a proper family and contact information.


A couple other quick things of note is that I spruced up our volunteer page and attended this month’s Seattle TA3M. The later had a nice bit of serendipity in that I met Anna, a UW PhD student who’s looking into our Bandwidth Authorities. Best of luck Anna! That’s certainly an area that could do with a lot of improvement.

"Yup, all done hacking on arm!" I told myself. I’m such a liar. My August was mostly spent adding features that didn’t make it into the 1.4.3 release. In particular…

  • a dialog with stats for exiting port usage (for exits) and client locales (for guards and bridges)
  • control socket support
  • torctl event parsing rewrite
  • descriptor dialog rewrite
  • expanded the projects listed on the tor front page

Google Summer of Code finished last week with all students passing. For me the real question of how we did will be answered in another few weeks when we discover which students stay and which evaporate. Most have expressed an interest in staying so that’s a good sign.

Finally, I’ve spent this last week writing a control port interpretor. Its purpose is to provide raw control port access (like a telnet session with the control port) but with usability improvements. In particular…

  • auto-negotiate authentication
  • tab completion for valid controller commands (which are fetched from the attached tor instance via the ‘GETINFO */names’ options)
  • up/down cycles through the history and ctrl+r provides history auto-completion
  • * nice formatting for the responses (context specific color/bolding)
  • * support for mutli-line controller commands and event listening
  • irc style interpretor commands…
    • /write [PATH] – saves interpretor backlog to the given path (PATH defaults to the last used location)
    • /find PATTERN – regex search through the backlog, highlighting matches
    • /quit – I’ll let you guess
    • * /help [OPTION] – provides usage information for both interpretor and tor commands
    • * /window [0-9] – switches between workspaces (like multiple telnet connections in screen sessions)
    • * /info RELAY – dumps consensus/descriptor entries for a relay by fingerprint or nickname (see the arm descriptor dialog for what this’ll look like)

* these are the todo items, everything else is done – ideas welcome for other features, especially if it’ll make your life easier!

This interpretor can both be a terminal prompt (by running “arm –prompt” or “arm -p”):

Interpretor Prompt

or used from the arm interface:

Interpretor Panel

They work from the same backend, but the curses/getstr vs prompt/readline frontends provide different capabilities…

  • Only the prompt provides line wrapping. I haven’t decided if I’ll do this in the panel or not since it’s a pita to code (many gory details due to scrolling) and not desirable for all commands…
  • Only the prompt provides suggested tab completion results or ctrl+r history search.
  • Only the panel can provide input syntax highlighting and nice scrolling keybindings.
  • Only the panel will be able to have a /window option.

Most of this next month will be spent polishing this new addition, then making the 1.4.4 arm release.