Between the normal October hubbub of baking peanut butter cookies and Halloween (the spiffiest US holiday, imho) I’ve been hacking a fair bit on our python projects.

Arm is now in maintenance mode, but has been getting plenty of love…

• Thanks to Carlo Strub arm now has a FreeBSD port!
• Sebastian and Robert spotted a couple substantial issues ([1], [2]), now fixed
• Jordi Espasa Clofent generously lent me an OpenBSD vm for arm testing. I fixed the issues that I could ([1], [2], [3]), but there’s still a couple bad ones outstanding…
• The control connection gets intermittent interrupt signals while arm starts. This one has me completely stumped. Wherever this fun-loving gremlin lives it’s deeper than I’d care to go (maybe a vm issue, OpenBSD quirk, or it’s just a conscientious objector of localhost socket connection – who knows).
• The uptime attribute for OpenBSD’s variant of ps is… er, difficult to parse. It’s in local time, has am/pm rather than being 24-hour time, and the whole format changes based on if the uptime is over a day or not. This whole platform has been scientifically designed to get on my nerves…

My main focus, however, has been on Stem. I’ve finished the ControlMessage class, a counterpart for TorCtl’s core sendAndRecv functionality which handles the base control protocol message parsing. From here it’ll be easy to implement counterparts for most of TorCtl’s functions (get_info, get/set_conf, etc), but that’s not really a high priority. Only a small fraction of my time has been spent working on the stem library – much more has been spent on the documentation and unit/integration testing which is what’ll give this library its worth. Besides being developer friendly and well tested, this will let us check when cutting new Tor releases if its changes will cause issues for stem’s users or not. I’ve also submitted a TorCtl change to take advantage of this but it’s looking kinda unlike that will happen.

At present the stem integration tests are a good basic verification test for Tor’s controller functionality, and will become better as I expand stem. If we become interested in testing for Tor then this will also give a very good starting point for writing those. However, while I’m happy to help with Tor testing I’m also tired of working alone on things that only I care about. If we expand testing to focus more on Tor then someone else will need to take a lead there.

Besides development, I did a code review for Tom’s torperf changes and attended the GSoC Mentor Summit where I met Mitar Milutinovic, David Fifield, the Umit developers, and took part in a counter-censorship discussion. We should follow up with Rodolfo Carvalho who’s developing Open Monitor (they have a skill set we lack and vice versa), but that’s up to others. From irc it sounds like we’re too overloaded right now to mentor for Google Code In – pity but maybe next year.

All in all a great month.

Hi all. For my part September was spent working toward the 1.4.4 arm release which tidies up the rough edges of the prior version and adds the control port interpretor (a pretty spiffy feature, imho). This feature was made with people either learning or commonly using the control port in mind so I’d really appreciate more feedback. Unfortunately Sebastian is the only dev so far to give it a try.

Besides that and contrary to my usual avoidance of fellow human beings, I’ve worked with others on a few things…

• Roger and others spotted an issue in Vidalia and TorCtl’s handling of control port authentication when there’s multiple authentication methods being used. I fixed this for TorCtl and arm.
• Juan Alcaine is helping with the arm RPMs, providing much needed testing and splitting arm from its dependencies. Next step is to get help from Erinn for uploading the arm/torctl rpms to the deb.tpo repos.
• Kamran has been working on a patch for exit locale selection in arm. It’s functional, but not quite done yet (I’m looking forward to seeing his finished version!).
• Met with Alasdair Young, another Seattleite who’s interested in checking Pidgin for leaks and hacking on MAT. I should get drinks with him again in a few weeks…
• Helped Rob Jansen a bit with a curses setup wizard for Shadow.

The 1.4.4 release is the finishing point for major arm development (for realz this time!), and next I’ll be shifting my focus to Stem (a fork of TorCtl) and our shiny, new django TorStatus site.

Hi all. For most of July I’ve been traveling. First along the ocean, visiting Ashland’s theaters and the Strawberry Festival, then ending with PETS in Waterloo. It was fun and great to see everyone, though I’m glad to finally have some time back at home.

During those trips I finished arm’s relay setup wizard and released version 1.4.3. At the dev meeting I also worked with Nick on refactoring TorCtl’s event parsing and Jake on a safe method for customizing Debian’s system wide torrc.

This release marks the end of my plans for major feature expansion of arm’s terminal interface. From here I’ll be shifting my focus to either Kamran’s gtk interface or the new TorStatus site (probably picking based on if Kamran wants to stay after GSoC or not).

June could have started a little better, beginning with a nasty flu bug that had me bedridden for the better part of a week. But once that was over with arm got several new features and is now tantalizingly close to its 1.4.3 release. Improvements include…

• Menu interface (thanks to Kamran for implementing its first version)
• TorCtl fixes for 2412, 2812, 2065, 1329, 2580, 3406, and 3409
• Newnym option
• Dependency auto-fetching via mirrors with signature checks (issue spotted by Sebastian and Robert)
• Relay setup wizard. This is still in the works and about a week away from completion, but it’s turning out very nicely.

Kamran has made some progress with the arm gui, porting the bandwidth graphs and nearly finishing the log panel. This has slipped quite a bit due to illness and family issues, though the parts that are done look great. For a description and screenshot of his work see his blog posting.

Finally, I dug into arm’s resource consumption and performance. I was able to reduce its memory usage by 12% and the shutdown time’s now instantaneous. However, besides this arm’s about as lean as I can reasonably make it…

• Memory:
  • 17.9 MB total memory usage
  • 3.0 MB (16.8%) is from the idle python interpretor
  • 7.5 MB (41.9%) is from importing the codebase
  • 7.4 MB (41.3%) is consumed at runtime, contribution from individual panels being negligible

• Startup time is 0.142 seconds. 0.123 is the baseline startup, with graphing contributing an extra 0.02 seconds (probably from reading the state file for bandwidth prepopulation). On the first startup there’s around an extra second, probably for importing the libraries.
• As for cpu usage, there’s spikes from connection and resource usage fetches but otherwise it’s flat (very little curses or controller activity due to caching and being smart with redraws). Individual panels don’t contribute noticeably to the baseline.

May was a beautiful, chaotic haze that began with the GSoC acceptance fallout…

• blog posting
• acceptance introductions
• last minute coordination to get a student for the EFF
• adding proposals, mirrors, and minor template additions for next year’s GSoC
• git repository and ldap discussions

… and somehow ended with me as the mentor (or co-mentor) for five students. Karsten, Norman, and I will be mentoring three students from Wesleyan college to work on a Django rewrite of the TorStatus site. I only have a little prior experience with Django so a good portion of this month was spent reading the Django book to get up to speed (only got to chapter ten – I’m a horribly slow reader).

Meanwhile Kamran Khan has been hacking on the arm codebase, finishing his first week with a functional menus prototype. The implementation details turned out to be trickier than I’d anticipated and will need more polish, but my hat’s off to him for being able to dive into a completely new codebase and develop such a substantial contribution in so little time (thanks, Kamran!). For more on this see Kamran’s blog.

The fifth student is ‘identity’ from irc. He’s doing his thesis on arm, writing both a paper on its future plans and implementing a minor feature (NEWNYM functionality). He, Kamran, and I will be having periodic sync-ups via Mumble to discuss their projects and issues they run into. If this is successful I’ll suggest it to the Wesleyan students too.

Besides all of this, I’m also keeping an interested eye on Julien’s Metadata Anonymisation Toolkit. I did a code review for an initial bit he implemented and might do more if I both have the time and he wants the help.

In terms of arm development, I finished a complete rewrite of the codebase that’s been almost a year in the works. Besides being a far saner implementation, this paves the way for the further performance enhancements and features discussed on its wiki. /me is still not quite done doing his happy dance about this

Other interesting developments include…

• arm’s now able to rename its process name (thanks to help from Jake)
• added unicode printing support (thanks to Robert for the idea)
• backported to Debian Squeeze by Dererk
• is almost backported for Ubuntu Lucid and Maverick thanks to Wendy, Kamran, and Evan Broder

Once again my month’s been split between GSoC and arm. The former’s been a quite juggling act (especially the admin role), but well worth it. When the dust settled arm got a fantastic student. Kamran Khan will be hacking on arm this summer, working on several enhancement including manual path selection, UPnP support, a newnym option, and best of all a GTK front end.

In other news, this month started with the 1.4.2 arm release followed by hotfixes and improvements to its deb. Thanks to Dererk, TorCtl has been properly packaged for Debian as python-torctl and arm uses this (rather than a bundled copy) for its debs. This release also had metadata fixes suggested by intrigeri (debian bugs 623311 and 623312).

Thanks both to Sebastian and my new team of git-fanatics at Amazon, arm has finally migrated to git. Besides some workarounds for svn:externals and ‘svn export’ this has been a painless transition and I’m definitely glad I made the move.

As for arm development, this month has included several notable fixes, performance improvements, and cleaning of the codebase…

• Improved arm’s startup time by 83% (from 0.84 seconds to 0.14).
• Thanks to Erinn and Andrew I finally have access to a Mac. PID resolution and several important issues for arm on that platform have been fixed.
• The deprecated connection panel and file descriptor popup have been dropped from the codebase (together over 1500 lines).
• Fixed a critical parsing error for circuit paths in older Tor versions thanks to asn.
• Investigated the work needed for Windows compatibility and process renaming. Unfortunately neither are likely to happen any time soon.
• … and many more (file descriptor warnings, using new ‘traffic/*’ getinfo options, etc).

Ok, my project’s released, applicants have responses, and at long last I stand triumphant over my inbox nemesis (for the moment, anyway – he’ll be back). No more excuses so here’s my status report for March.

This last month was spent juggling a few things. First and foremost I’ve been swapping between my mentor and admin hats for GSoC. By day I’ve been that annoying, nagging guy asking devs to talk to all these young upstarts that won’t get off their lawn. But by night I transform into a hideous slave driver, bent on demanding more and more from the poor applicants to my projects. I’m actually not sure which group is more likely to plot my assassination…

Time permitting I’ve continued to hack on arm, adding some sweet new features like expanded circuit paths and application identification to the connection panel. The 1.4.2 release (which has been in the works for the last three months) is finally done, for more on that see its blog post.

And finally, I’ve survived my last oncall with RCX Checkout (only got paged thirteen times!) and transferred to the Source team of Builder Tools. Yesterday was my first day and so far I’m loving it, though I’m suspecting my old team is trying to hint something since they left me in their oncall rotation…

I’m usually weary of randomization but this last month had too many interesting things to keep from branching out a bit. GSoC is coming up and Andrew has been drumming up interest among students from Wesleyan. In preparation for both I’ve been sprucing up the Tor volunteer page, prepping our GSoC application, and contacting new potential mentors like Tomás, Robert Ransom, Robert Hogan, and the T(A)ILS community for project ideas.

I’ve also been preparing my own project for possible contributors, moving arm’s development notes and revised project ideas to the Tor wiki. This will be the canonical place for arm development information and upcoming plans (I get enough of scrums and burn-down charts at work, and this should act nicely as an alternative for keeping people informed).

Arm development has stayed relatively on track, with the revised connection panel very nearly achieving parity with its predecessor (and in most respects surpassing it). Most of what remains are refinements and tasty new features. Arm has also been added to Debian (Sid) and Ubuntu (Natty) with backports pending. Many thanks to Peter for his help.

As with last month, I have another round of being oncall for work which will sap a chunk of my time (and leave me grumpy). That aside, the 1.4.2 arm release should be done by the end of the month and I’ll be keeping an eye on the channels for the first round of inquisitive students. Fingers crossed that we find some good ones that stick around this year.

This last month began with the release of arm 1.4.1, last minute features including better TBB compatibility and summarization/filtering of the options presented in the configuration panel. This was followed by three hotfixes for platform specific bugs in error handling and an improvement to help with Gentoo ebuilds (thanks to Fabian, Trystero, and Anthony).

Since then I’ve been diving into the connection panel source, splitting out and improving its functionality for mapping IP/ORPorts to fingerprints and identifying exit connections (this determines what information is publicly displayable or not). The pesky syshook concurrency error and among others have also been fixed.

I’ve been working weekends to make deadlines for Amazon and had a week long brush with the flu which ate a good portion of this month. I’m also scheduled for our oncall rotation next week which won’t help either. However, I’m hoping to have the connection panel rewrite finished this month and begin working on some tasty new features for it soon afterward.