Oooh summer. Pollen scratching my eyes and a fiery ball of death hellbent on frying me. We oughta nuke the sun! Teach that lazy, freeloading ball of plasma who’s boss. Just think of all the savings on air conditioning!

In those rare moments not staving off heat stroke I spent much of this month visiting family, and just a tad on code…

• Stem 1.2.0 introduced a couple version compatibility regressions, the first with python 3.x and another with python 2.6. Both received hotfixes, bumping us now to version 1.2.2.

• Karsten and I discussed a Python rewrite of ExoneraTor at length. While this would be a pretty fun and interesting project, its relative impact compared to other things I have in the works right now isn’t especially high.

• Reviewed Stem usage in Kai’s DetecTor project.

Recently I shifted my focus back to arm. I’ve been truly impressed at the level of interest in it on IRC! New arm users are appearing almost daily, and I owe them a shiny new release!

Hi all, this month was focused on work leading up to today’s Stem 1.2.0 release. The control interpreter in particular turned out extremely well, so if you haven’t tried it yet then please check it out!

Besides implementing and writing a tutorial for the interpreter this involved addressing a host of other issues that had accumulated in trac. By far the most important was an issue where Stem spawned Tor instances would halt due to unread stdout content.

Last, I looked over the code of several new Stem users this month and added them to our examples page…

• Yawning’s or-applet
• Whonix
• Kostas’ torsearch

Hi all. This April’s status report is gonna be short and sweet. The bulk of my time went toward Google Summer of Code student selection. I’m delighted to announce that we have thirteen great students working with us this summer (twelve with Tor and one with the EFF)! This is twice as many as last year, so it will be interesting to see if we can manage it.

This concludes the bulk of my work as the org admin, so this next month I should be able to focus back on writing sweet, sweet code.

On April 26th I presented at LinuxFest Northwest. For those who haven’t seen it, this was a repeat of my TA3M ‘Tor Ecosystem’ presentation.

And last but not least, bit by bit I’ve been picking away at a feature branch for Stem. It’s not quite ready yet, but expect an announement and a new Stem release sometime in May. One improvement that *is* now available though is the following:

• The connection module’s connect_port() and connect_socket_file() functions have been deprecated in favor of a new, slicker connect().

This new function is quite a bit smarter about the error responses it provides, and is more intuitive to use.

Hi all. GSoC coordination and a couple particularly tenacious illnesses pretty well knocked me out of commission this March. The neatest changes I managed to slip in concerned Stem’s site…

• FAQ entry for how to interact with Tor’s controller interface directly. This comes up reasonably frequently on Stack Overflow. In the answer we discuss each connection and authentication scenario with examples.
• A few weeks ago Coderman made the great suggestion for a Stem Cookbook. I’ve put together a few scripts that demonstrate neat things. If you have any suggestions for other things to exemplify then please let me know!
• Code reviewed Martin’s munin-tor and Lunar’s exit-funding scripts. Both are now listed on our examples page, and I also made Exit Map more prominent (thanks Philipp for mentioning Stem in Spoiled Onions!).
• Handful of other minor improvements such as making FAQ entries more concise, syntax highlighting torrc snippets, and better module docs.

Stem now supports HS_DESC events, and fixed an issue with ExitPolicy pickle-ability thanks to Nick Hopper.

Final note for March is that as anybody who knows me can attest I make a lot of typos. I spend enough time on irc that I finally invested some effort into getting an Irssi spellchecker working.

I love Irssi to death but user friendly it is not. If you would like a similar setup then here’s my notes.

It was great meeting everyone in Reykjavík! February largely went toward GSoC prep. Thanks to everybody that sent me project ideas!

    Blog Post

This left a lot less time than I’d like for coding. The only noteworthy changes this month were…

• Added a new ‘stem.util.test_tools‘ module for easily running Pyflakes and PEP8 checks.

  Stem, ExitMap, and arm all do these checks as part of their tests so this let them shed quite a bit of boilerplate.

  Want these checks for your own project? It’s easy, so give it a try!

• Stem tutorial example for checking the Tor exits you’re using for your connections.
• Couple new Controller methods for checking where Tor is listening for connections of a given type…

  >>> for listener_type in stem.control.Listener:
  ...   ports = map(str, controller.get_ports(listener_type))
  ...   print "%s ports: %s" % (listener_type, ', '.join(ports))
  ...
  OR ports:
  DIR ports:
  SOCKS ports: 9050
  TRANS ports:
  NATD ports:
  DNS ports:
  CONTROL ports: 2779
  

• Gave Sphinx’s aafig module a try to see if it could improve our API
  overviews…
  • Before
  • After

  In the end I decided against it since this doesn’t add much, and it causes us to render the overview as a svg (so users can’t copy/paste function names).

• Sent Philipp some improvements for ExitMap.

Hi all. This month I decided to switch from a depth-first to breadth-first approach for overhauling arm. Changes included…

• Made arm PEP8 compliant. This took around a week during which sed and I were close friends. Like Stem, both pyflakes and pep8 checks are now included in arm’s test runs.

• Completely dropped tor_tools from the codebase. This is a neat milestone as tor_tools was a 2,700 line wrapper around TorCtl providing a friendlier API, thread safety, caching, etc. Stem provides all of these capabilities.

  Initially migration was simply switching tor_tools from TorCtl to Stem, but now arm works directly with a Stem controller allowing us to drop the tor_tools module completely.

• Finished arm’s new tracker module which provides tor resource usage, connections, and application names. It’s not just much nicer code, but now has unit tests!

Why am I sinking all this effort into arm? After all, it presently does pretty much what we want, right?

Well, yes. And I suspect user’s won’t see many obvious changes in the next arm release. Rather, my present focus on arm is partly for code quality but more importantly to make sure Stem is ready for prime time.

Arm and Vidalia are our two most complicated and demanding controller applications, and making a Stem-based arm release is a surprisingly good way of making sure Stem has all the pieces we need for tasks like a next-gen Tor GUI or a localhost status panel for relay operators.

As I’ve rewritten arm Stem has improved as well. Stem changes this month included…

• Added a port_usage() function for getting a description of a port’s common usage. This nicely complements our other connection utils…

  >>> from stem.util.connection import port_usage
  
  >>> port_usage(80)
  'HTTP'
  
  >>> port_usage(22)
  'SSH'
  
  >>> port_usage(9418)
  'Git'
  

• Couple new Controller methods to help with NEWNYM handling: is_newnym_available() and get_newnym_wait().

• Pyflakes and pep8 are python modules so why shell out to them? We now use their APIs instead reduced the runtime of Stem’s static checks by 60%.

• Greatly reduced the verbosity of stem’s test output. You can still get the previous, more detailed results with the ‘–verbose’ argument.

Last but not least, non-development tasks from this month included…

• Finally moved doctor to its new home on cappadocicum. (ticket)
• Submitted my Tor Ecosystem talk to LinuxFest Northwest. (presentation)
• Tickets for Iceland. Looking forward to seeing everyone there!

Hi all, hope you’ve enjoyed the holidays (and for a few of you CCC)! This December I finally swapped back to focusing on arm in earnest. Good news is that the codebase is rapidly improving, the focus for this month being…

• Finished overhauling the starter module.
• Expanded test coverage as more of arm gets rewritten.
• Integrated pyflakes and pep8 with our tests.
• Moved user facing strings to the internal configuration. I did this to improve our code, but it could also allow us to add internationalization support to arm in the future if folks want it.

Bad news is unfortunately that none of these improvements are user facing, and even by the rosiest estimates arm’s next release is still many months off.

This December I also finished with my efforts to revise our internal lists. None of the more substantial changes got much support, but this still resulted in some worthwhile improvements…

• A wiki with our lists, maintainers, and a brief summary of each: link
• We now have a public, documented policy for adding folks to our internal lists.
• The tor-assistants list is now a proper subset of tor-internal.
• Identified some unused lists we can drop. (ticket)

Hi all. Work and life is conspiring to keep me busier than usual. What time was left this November mostly went toward non-development activities. I miss coding…

Tor Ecosystem Presentation

On 11/18 I gave a presentation at Seattle’s TA3M overviewing projects in the Tor ecosystem, with an eye toward getting more developers involved…

• Presentation

• Slides

Preparing for this gobbled up a surprising amount of time (I don’t do much public speaking), but I’m really happy with how it turned out. I’ll likely give this presentation again at other local venues, such as LinuxFest Northwest in the spring.

Email List Management

Much of this last month went toward managing our lists and leading a discussion about what we’d like to do with our non-public lists. Work on this is ongoing, but at the very least we’ll finally get a public, documented process for adding folks.

I also sunk several hours into a frustrating GMail issue where it bounces Mailman’s moderator queue as spam. At one point this seems to have been responsible for unsubscribing everyone using GMail from tor-assistants@. I’ve changed Mailman’s configuration so this hopefully won’t happen again, but I’m out of ideas for the GMail bounces.

Other news this month includes…

• Fixed an important Stem ImportError under Windows. This resulted in a hotfix release of Stem (version 1.1.1). (ticket)
• Added Stem support for the new event types from proposal 218. (change)
• Code reviewed Philipp’s ExitMap project.
• Read over tbbscraper and added to Stem’s examples page.
• Helped provide stats for the impact of changing tor’s default exit policy. (ticket)

Hi all. Here’s a quick update of what I was up to this October…

Stem Release 1.1.0

After seven months of work Stem version 1.1.0 is finally out!

This month was primarily focused on pre-release performance and compatibility improvements in preparation for this…

• Handful of improvements that reduced the memory usage of our Descriptor classes by 20%.
• Replaced Stem’s custom caching with Python’s @lru_cache. This was added to the standard library in Python 3.2, so Stem includes a Python 2.x backport to our utilities.
• Dozens of compatibility fixes for Python 2.6 and 3.x.
• Added test coverage for Stem’s interpretor examples, courtesy of Python’s doctest module.

GSoC Mentor Summit

Nick, Moritz and I attended the GSoC Mentor Summit at Google’s main campus. By and large it was very similar to previous years, with many of the same faces and discussion topics. Unlike the prior couple years however I did not focus on keeping notes which, on reflection, was a mistake. Here’s the small bit I remember…

• Had a couple fun language discussions with developers from SciRuby and Scala. The former mostly concerned the advantages and disadvantages of Ruby with respect to Python (I use the former at work, so I’ve had some time to form an opinion on this). The later concerned Scala support for issuing database queries via list comprehensions.
• Couple discussions with Wikimedia developers regarding Tor abuse. Unfortunately this is a perennial discussion that never seems to go anywhere despite having plenty of technical solutions.
• Nick gave me a tour of Chutney’s codebase and tried to persuade me to take on development of it. Nearly worked too. But while Chutney, SoaT, TorBEL, and a dozen other projects would be a lot of fun to work on I only have so much I can squeeze in alongside a full-time job.
• A hackathon organizer with Random Hacks of Kindness discussed what they’ve found most successfully attracts contributors. The two most memorable tips were momentum and recognition. Regularly organizing events is necessary to maintain a core contributor group, and engaging local media doesn’t hurt as it provides recognition for the contributors.
• Shadowed Nick during discussions with a Mozilla developer concerning the upcoming Tor IM bundle. They certainly seem excited for OTR support…
• Visited with Terry and Arc from the PSF quite a bit. Still no ETA on when distributions will switch to Python 3, and Mailman 3 is still a ways off. Momentum on lots of projects though. Evidently they acted as an umbrella org for roughly sixty students. I would weep bitter, bitter tears if I had to be the admin for that…

Other news this month includes…

• Karsten, Andrew and I are now all moderators for the tor email lists. I’m taking point in this area, making daily sweeps through the moderator queue and tweaking Mailman to reduce friction.
• Handful of DocTor fixes and improvements. Karsten shut down the Java DocTor early in the month so our shiny, new Python DocTor is now flying solo!
• Attended events in the Seattle area including ‘Go for Python Developers’ at Google Fremont and the second TA3M meetup.
• Worked with authority operators to resolve brief outages with dizum and urras.

Hi all. Present status: gobbling birthday carrot cake. Before that, though, did a few things worth noting in September…

Stem: Connection Resolution Support

Stem can now provide you with the active connections belonging to a process. This is similar to arm, capable of doing lookups via seven *nix and FreeBSD resolution methods.

Unlike arm, however, this is a far cleaner implementation including both unit and integ tests. Unfortunately this doesn’t yet work with Tor’s DisableDebuggerAttachment feature, but I plan to look into workarounds for that soon.

‘How do I use this’, you ask? It’s easy!

Arm: Rewrote Starter Module

Arm is presently undergoing an overhaul, and this month I focused on its starter module. It’s now shorter, has unit tests, and is just generally damn better (before and after).

Other news this month includes…

• Google Summer of Code finals were last week. All the remaining students passed, having done great work – fingers crossed they stick around!
• DocTor’s python replacement is now happily chugging along on yatei and provides #tor-bots notifications.
• Attended a local TA3M meetup hosted by Lee Fisher. I’ll be presenting a ‘tor ecosystem’ talk at it in November.
• Notified outdates relays that will soon be dropped from the network if they don’t upgrade. (ticket)
• Investigated tor regression reported by stem’s jenkins tests. (ticket)
• Code reviewed Aaron’s TorPS.
• Fixed a stem caching bug regarding hidden service config options caught by wayzard.(ticket)
• My site (https://www.atagar.com, which hosts arm) finally has SSL support.