Code! Beautiful code! How I’ve missed thee!

As mentioned before I’m hip deep in the middle of an arm rewrite. Not only to overhaul the arm codebase, but to ensure Stem will be up to snuff for a Tor GUI or web dashboard (projects I’d love to jump into someday). I’m delighted to announce a nice bit of progress on this front.

September went toward rewriting its header panel (the top portion of arm’s interface)…

• Old zomg my eyes, they bleed!
• New hotness

As mentioned though this is really more about Stem than arm. Stem improvements
made along the way this month include…

• Far better handling for the ‘private’ keyword (the default prefix) and the default suffix in our ExitPolicy class. (ticket)
• New BaseController.connection_time() method that provides when our control connection was established or detached.
• New Controller.get_accounting_stats() method which reports accounting information for our relay.
• Controller now fetches our own descriptor if no fingerprint is provided to get_server_descriptor(), get_network_status(), or get_microdescriptor().
• New crop() and join() methods for the str_tools module.
• Dropped a redundant get_* prefix from most of our util function names. Old names still work as aliases.
• Our launch_tor_with_config() function raised an OSError if called too many times, caught thanks to RSenet. (ticket)

DocTor, the monitor for directory authority health, also got some love this month…

• Corrected suppression behavior when we have staggered alerts. This should greatly cut down on the amount of noise on the list.
• We now check if too many relays lack a bandwidth authority measurement… or we would if this wouldn’t perpetually be in alarm. (ticket 1, 2)
• Revised DocTor’s schedule as requested by Sebastian. (ticket)
• Expanded suppression for messages about turtles and kicked off a conversation to at long last sort this out. (ticket)

One final note is that I uploaded my Tor Ecosystem presentation to YouTube so volunteer page visitors can see a streaming presentation rather than downloading a 53 MB video.

Hi all, getting my status report out of the way a tad early. Deadlines for my day job at Amazon required working a weekend so April was mostly spent just getting caught up on code reviewing contributions…

• Nick Mathewson greatly improved Stem’s performance at parsing consensus documents. Thanks Nick!
• Rewrote Nick Hopper’s patch for verifying consensus signatures. I’d really love to merge it, but it needs tests first.
• Code reviewed federico3’s addition of better Controller methods for handling hidden service config options.
• Fixed stylistic issues cited by the newly released pep8 1.5.7, and also corrected a handful of python 3.x test failures.
• Code reviewed a BridgeDB change to take advantage of Stem for isis.
• GSoC finals are finished! This year had a lower pass rate than we usually do (four of our thirteen students didn’t pass), but I don’t think there’s necessarily any lessons to draw.

This concludes my backlog of things I owe other people, so next up I’ll finally be getting back to arm development!

For one brief, glorious moment in time I touched code. Really, it happened! House plants can vouch as my witness.

Or that is to say, I’ve coded a lot less than I’d like. Barring a long overdue vacation to Sol Duc this month went toward a lot of random distractions…

• Worked with Philipp on overhauling our procedure for flagging bad relays. Philipp’s taken the lead here and is doing a great job! While I find this space interesting I’m already spread too thin, so going forward I won’t be a co-maintainer for it after all.

• Looked over Nick Hopper’s consensus validation change, adding crypto blob type validation to simplify his work. I now owe him a more thorough code review.

• Handful of DocTor issues including OOM troubles, unpleasant amount of noise from consensus parameters, and a several minor revisions.

• Discussed adding a new X- extrainfo descriptor field with Virgil, suggesting revisions and helping him make it conformant with the spec.

• System I do development on died a puzzling death. Even after twenty hours investigating I’m still not quite sure what happened. Cleaning out dust, removing the RAID controller, and starting over seems to have done the trick.

• GSoC administrative work. We’re now less than two weeks from the final evaluations!

• Continued overhauling arm but as already mentioned focused on this a lot less on this than I’d like.

So little coding makes me grumpy, so like many of you I’m taking a hard look at where my time’s evaporating to. Maybe there’s something I can change or extricate myself from to better focus.

Oooh summer. Pollen scratching my eyes and a fiery ball of death hellbent on frying me. We oughta nuke the sun! Teach that lazy, freeloading ball of plasma who’s boss. Just think of all the savings on air conditioning!

In those rare moments not staving off heat stroke I spent much of this month visiting family, and just a tad on code…

• Stem 1.2.0 introduced a couple version compatibility regressions, the first with python 3.x and another with python 2.6. Both received hotfixes, bumping us now to version 1.2.2.

• Karsten and I discussed a Python rewrite of ExoneraTor at length. While this would be a pretty fun and interesting project, its relative impact compared to other things I have in the works right now isn’t especially high.

• Reviewed Stem usage in Kai’s DetecTor project.

Recently I shifted my focus back to arm. I’ve been truly impressed at the level of interest in it on IRC! New arm users are appearing almost daily, and I owe them a shiny new release!

Hi all, this month was focused on work leading up to today’s Stem 1.2.0 release. The control interpreter in particular turned out extremely well, so if you haven’t tried it yet then please check it out!

Besides implementing and writing a tutorial for the interpreter this involved addressing a host of other issues that had accumulated in trac. By far the most important was an issue where Stem spawned Tor instances would halt due to unread stdout content.

Last, I looked over the code of several new Stem users this month and added them to our examples page…

• Yawning’s or-applet
• Whonix
• Kostas’ torsearch

Hi all. This April’s status report is gonna be short and sweet. The bulk of my time went toward Google Summer of Code student selection. I’m delighted to announce that we have thirteen great students working with us this summer (twelve with Tor and one with the EFF)! This is twice as many as last year, so it will be interesting to see if we can manage it.

This concludes the bulk of my work as the org admin, so this next month I should be able to focus back on writing sweet, sweet code.

On April 26th I presented at LinuxFest Northwest. For those who haven’t seen it, this was a repeat of my TA3M ‘Tor Ecosystem’ presentation.

And last but not least, bit by bit I’ve been picking away at a feature branch for Stem. It’s not quite ready yet, but expect an announement and a new Stem release sometime in May. One improvement that *is* now available though is the following:

• The connection module’s connect_port() and connect_socket_file() functions have been deprecated in favor of a new, slicker connect().

This new function is quite a bit smarter about the error responses it provides, and is more intuitive to use.

Hi all. GSoC coordination and a couple particularly tenacious illnesses pretty well knocked me out of commission this March. The neatest changes I managed to slip in concerned Stem’s site…

• FAQ entry for how to interact with Tor’s controller interface directly. This comes up reasonably frequently on Stack Overflow. In the answer we discuss each connection and authentication scenario with examples.
• A few weeks ago Coderman made the great suggestion for a Stem Cookbook. I’ve put together a few scripts that demonstrate neat things. If you have any suggestions for other things to exemplify then please let me know!
• Code reviewed Martin’s munin-tor and Lunar’s exit-funding scripts. Both are now listed on our examples page, and I also made Exit Map more prominent (thanks Philipp for mentioning Stem in Spoiled Onions!).
• Handful of other minor improvements such as making FAQ entries more concise, syntax highlighting torrc snippets, and better module docs.

Stem now supports HS_DESC events, and fixed an issue with ExitPolicy pickle-ability thanks to Nick Hopper.

Final note for March is that as anybody who knows me can attest I make a lot of typos. I spend enough time on irc that I finally invested some effort into getting an Irssi spellchecker working.

I love Irssi to death but user friendly it is not. If you would like a similar setup then here’s my notes.

It was great meeting everyone in Reykjavík! February largely went toward GSoC prep. Thanks to everybody that sent me project ideas!

    Blog Post

This left a lot less time than I’d like for coding. The only noteworthy changes this month were…

• Added a new ‘stem.util.test_tools‘ module for easily running Pyflakes and PEP8 checks.

  Stem, ExitMap, and arm all do these checks as part of their tests so this let them shed quite a bit of boilerplate.

  Want these checks for your own project? It’s easy, so give it a try!

• Stem tutorial example for checking the Tor exits you’re using for your connections.
• Couple new Controller methods for checking where Tor is listening for connections of a given type…

  >>> for listener_type in stem.control.Listener:
  ...   ports = map(str, controller.get_ports(listener_type))
  ...   print "%s ports: %s" % (listener_type, ', '.join(ports))
  ...
  OR ports:
  DIR ports:
  SOCKS ports: 9050
  TRANS ports:
  NATD ports:
  DNS ports:
  CONTROL ports: 2779
  

• Gave Sphinx’s aafig module a try to see if it could improve our API
  overviews…
  • Before
  • After

  In the end I decided against it since this doesn’t add much, and it causes us to render the overview as a svg (so users can’t copy/paste function names).

• Sent Philipp some improvements for ExitMap.

Hi all. This month I decided to switch from a depth-first to breadth-first approach for overhauling arm. Changes included…

• Made arm PEP8 compliant. This took around a week during which sed and I were close friends. Like Stem, both pyflakes and pep8 checks are now included in arm’s test runs.

• Completely dropped tor_tools from the codebase. This is a neat milestone as tor_tools was a 2,700 line wrapper around TorCtl providing a friendlier API, thread safety, caching, etc. Stem provides all of these capabilities.

  Initially migration was simply switching tor_tools from TorCtl to Stem, but now arm works directly with a Stem controller allowing us to drop the tor_tools module completely.

• Finished arm’s new tracker module which provides tor resource usage, connections, and application names. It’s not just much nicer code, but now has unit tests!

Why am I sinking all this effort into arm? After all, it presently does pretty much what we want, right?

Well, yes. And I suspect user’s won’t see many obvious changes in the next arm release. Rather, my present focus on arm is partly for code quality but more importantly to make sure Stem is ready for prime time.

Arm and Vidalia are our two most complicated and demanding controller applications, and making a Stem-based arm release is a surprisingly good way of making sure Stem has all the pieces we need for tasks like a next-gen Tor GUI or a localhost status panel for relay operators.

As I’ve rewritten arm Stem has improved as well. Stem changes this month included…

• Added a port_usage() function for getting a description of a port’s common usage. This nicely complements our other connection utils…

  >>> from stem.util.connection import port_usage
  
  >>> port_usage(80)
  'HTTP'
  
  >>> port_usage(22)
  'SSH'
  
  >>> port_usage(9418)
  'Git'
  

• Couple new Controller methods to help with NEWNYM handling: is_newnym_available() and get_newnym_wait().

• Pyflakes and pep8 are python modules so why shell out to them? We now use their APIs instead reduced the runtime of Stem’s static checks by 60%.

• Greatly reduced the verbosity of stem’s test output. You can still get the previous, more detailed results with the ‘–verbose’ argument.

Last but not least, non-development tasks from this month included…

• Finally moved doctor to its new home on cappadocicum. (ticket)
• Submitted my Tor Ecosystem talk to LinuxFest Northwest. (presentation)
• Tickets for Iceland. Looking forward to seeing everyone there!