Hi all. Between being ill, oncall for work, visiting with family over the holidays, and finally meeting a brilliant, wonderful girl named Megan I didn’t accomplish too much in December. This isn’t likely to change any time soon so the projects I maintain may get a little less attention – such a pity.

As normal I’ve mostly been split between developing stem and maintaining arm. Ideally I’d like to sink all my time into the former but arm had several issues that demanded attention this month…

• The ptrace change from ticket 3313 caused tor’s file descriptors to only be readable by root, breaking lsof, netstat, sockstat, ss, and some of arm’s proc based utilities. This does not break connection resolution itself, but rather the file descriptor to inode mappings used to associate connections to processes. Thanks to Jake we have a partial workaround which is to filter netstat results by the owner’s uid instead which, at least for the tor deb, should give the same results. For non-deb users I’ll need to just give a notice about why it’s broken or, in the case of Ubuntu users, suggest that they turn off ‘DisableDebuggerAttachment’.
• The latest arm release sometimes exhibits strange terminal glitches, caused by an interaction between the readline and curses modules. Thanks to Stephan Seitz for providing a reliable method for reproducing the issue.
• Kamran submitted a patch for UPnP support in arm which I spent a couple days reviewing. It’s a nice addition, though it’s gonna need some work before it’s merged.
• Tor Cloud use cases revealed a couple bugs with arm’s torrc validation including case sensitivity and an unexpected logging default for Debian. Thanks to koolfy for reporting the issues and Runa for the test system.
• Saving a snapshot of the log in arm had a couple issues, as reported by Jeff Bonner on the Debian bug tracker.
• Non-proc based connection resolution could fail due to terminal localization and an issue with the getProcessName() function. Issues caught thanks to Stephan Seitz.

The little time I’ve had for stem has mostly gone into improving and testing connection and authentication to the tor process. This module took a lot longer than I’d intended to finish but I’m really happy with the result. Also, both Sathyanarayanan and boerni have taken an interest in stem, making me a little hopeful that developing this library won’t be as lonely as arm was.

Hi all. I spent November chiefly focusing on stem, shoring up its testing and handling the connection/authentication handshake. The scope of the library is expanding very slowly but I’d argue that this is a good thing. Stem has almost compete code coverage with its tests exercising most use cases and edge situations I can think of. For instance, connection and authentication are run against configurations with…

• an open control port
• password authentication
• cookie authentication
• both password and cookie authentication
• control socket
• control socket with cookie authentication
• no method for controllers to connect

… and soon I’ll be adding tests for chroot setups (a use case where our projects traditionally have a lot of issues). If you’re making Tor changes that touch on PROTOCOLINFO or AUTHENTICATE then please run stem’s integ suite. It’s quick, it’s easy, and it’ll give your change a very, very good workout. To run it simply…

git clone git://git.torproject.org/stem.git
./stem/run_tests.py --integ --target CONN_ALL

Besides stem I’ve been involved with a smattering of other issues…

• Reviewed Filiol’s slides which, between the FUD, had a few reasonable concerns. This mostly concerned better monitoring so I filed a ticket which probably won’t get much traction until the next GSoC.
• Discussed a library for fetching consensus information directly from a variety of sources like cached descriptors and directory authorities/mirrors (ticket). I’ll be implementing this functionality in stem and Karsten plans to do the same with a java library.
• Discussed a Tor forum (ticket). Imho it will be doomed without Tor dev involvement and, since realistically we’ll give up on clicking through clunky web interfaces, we should have an email frontend too. Andrew disagrees so I’m taking my hands off of that project.
• Variety of small arm issues including ASC mishandling, torrc validation issues spotted by kookfy, and fixing an rpm dependency issue spotted by unspawn.
• Packaged the Tor posters and sent the ones to people in the US. I’ve had my fill of gigantic post office lines so the rest (and the stack of customs slips that go with ’em) will be waiting until after the holidays.
• Filed tickets (4629, 4630, 4627, and 3958) for TorCtl issues I’ve spotted while developing stem. I probably won’t keep doing this – it’s time consuming and pointless if stem replaces TorCtl. The last issue might also exist with Vidalia though Tomás hasn’t commented yet.
• Though not really Tor related, I submitted a patch to ReviewBoard for a weird XSS issue via comment fields. This next GSoC I’d like to set up a ReviewBoard instance for us. While I realize that shiny, ajax websites aren’t our style it would make code reviewing a lot nicer. code review, commit

All in all a good month.

Between the normal October hubbub of baking peanut butter cookies and Halloween (the spiffiest US holiday, imho) I’ve been hacking a fair bit on our python projects.

Arm is now in maintenance mode, but has been getting plenty of love…

• Thanks to Carlo Strub arm now has a FreeBSD port!
• Sebastian and Robert spotted a couple substantial issues ([1], [2]), now fixed
• Jordi Espasa Clofent generously lent me an OpenBSD vm for arm testing. I fixed the issues that I could ([1], [2], [3]), but there’s still a couple bad ones outstanding…
• The control connection gets intermittent interrupt signals while arm starts. This one has me completely stumped. Wherever this fun-loving gremlin lives it’s deeper than I’d care to go (maybe a vm issue, OpenBSD quirk, or it’s just a conscientious objector of localhost socket connection – who knows).
• The uptime attribute for OpenBSD’s variant of ps is… er, difficult to parse. It’s in local time, has am/pm rather than being 24-hour time, and the whole format changes based on if the uptime is over a day or not. This whole platform has been scientifically designed to get on my nerves…

My main focus, however, has been on Stem. I’ve finished the ControlMessage class, a counterpart for TorCtl’s core sendAndRecv functionality which handles the base control protocol message parsing. From here it’ll be easy to implement counterparts for most of TorCtl’s functions (get_info, get/set_conf, etc), but that’s not really a high priority. Only a small fraction of my time has been spent working on the stem library – much more has been spent on the documentation and unit/integration testing which is what’ll give this library its worth. Besides being developer friendly and well tested, this will let us check when cutting new Tor releases if its changes will cause issues for stem’s users or not. I’ve also submitted a TorCtl change to take advantage of this but it’s looking kinda unlike that will happen.

At present the stem integration tests are a good basic verification test for Tor’s controller functionality, and will become better as I expand stem. If we become interested in testing for Tor then this will also give a very good starting point for writing those. However, while I’m happy to help with Tor testing I’m also tired of working alone on things that only I care about. If we expand testing to focus more on Tor then someone else will need to take a lead there.

Besides development, I did a code review for Tom’s torperf changes and attended the GSoC Mentor Summit where I met Mitar Milutinovic, David Fifield, the Umit developers, and took part in a counter-censorship discussion. We should follow up with Rodolfo Carvalho who’s developing Open Monitor (they have a skill set we lack and vice versa), but that’s up to others. From irc it sounds like we’re too overloaded right now to mentor for Google Code In – pity but maybe next year.

All in all a great month.

Hi all. For my part September was spent working toward the 1.4.4 arm release which tidies up the rough edges of the prior version and adds the control port interpretor (a pretty spiffy feature, imho). This feature was made with people either learning or commonly using the control port in mind so I’d really appreciate more feedback. Unfortunately Sebastian is the only dev so far to give it a try.

Besides that and contrary to my usual avoidance of fellow human beings, I’ve worked with others on a few things…

• Roger and others spotted an issue in Vidalia and TorCtl’s handling of control port authentication when there’s multiple authentication methods being used. I fixed this for TorCtl and arm.
• Juan Alcaine is helping with the arm RPMs, providing much needed testing and splitting arm from its dependencies. Next step is to get help from Erinn for uploading the arm/torctl rpms to the deb.tpo repos.
• Kamran has been working on a patch for exit locale selection in arm. It’s functional, but not quite done yet (I’m looking forward to seeing his finished version!).
• Met with Alasdair Young, another Seattleite who’s interested in checking Pidgin for leaks and hacking on MAT. I should get drinks with him again in a few weeks…
• Helped Rob Jansen a bit with a curses setup wizard for Shadow.

The 1.4.4 release is the finishing point for major arm development (for realz this time!), and next I’ll be shifting my focus to Stem (a fork of TorCtl) and our shiny, new django TorStatus site.

Hi all. For most of July I’ve been traveling. First along the ocean, visiting Ashland’s theaters and the Strawberry Festival, then ending with PETS in Waterloo. It was fun and great to see everyone, though I’m glad to finally have some time back at home.

During those trips I finished arm’s relay setup wizard and released version 1.4.3. At the dev meeting I also worked with Nick on refactoring TorCtl’s event parsing and Jake on a safe method for customizing Debian’s system wide torrc.

This release marks the end of my plans for major feature expansion of arm’s terminal interface. From here I’ll be shifting my focus to either Kamran’s gtk interface or the new TorStatus site (probably picking based on if Kamran wants to stay after GSoC or not).

June could have started a little better, beginning with a nasty flu bug that had me bedridden for the better part of a week. But once that was over with arm got several new features and is now tantalizingly close to its 1.4.3 release. Improvements include…

• Menu interface (thanks to Kamran for implementing its first version)
• TorCtl fixes for 2412, 2812, 2065, 1329, 2580, 3406, and 3409
• Newnym option
• Dependency auto-fetching via mirrors with signature checks (issue spotted by Sebastian and Robert)
• Relay setup wizard. This is still in the works and about a week away from completion, but it’s turning out very nicely.

Kamran has made some progress with the arm gui, porting the bandwidth graphs and nearly finishing the log panel. This has slipped quite a bit due to illness and family issues, though the parts that are done look great. For a description and screenshot of his work see his blog posting.

Finally, I dug into arm’s resource consumption and performance. I was able to reduce its memory usage by 12% and the shutdown time’s now instantaneous. However, besides this arm’s about as lean as I can reasonably make it…

• Memory:
  • 17.9 MB total memory usage
  • 3.0 MB (16.8%) is from the idle python interpretor
  • 7.5 MB (41.9%) is from importing the codebase
  • 7.4 MB (41.3%) is consumed at runtime, contribution from individual panels being negligible

• Startup time is 0.142 seconds. 0.123 is the baseline startup, with graphing contributing an extra 0.02 seconds (probably from reading the state file for bandwidth prepopulation). On the first startup there’s around an extra second, probably for importing the libraries.
• As for cpu usage, there’s spikes from connection and resource usage fetches but otherwise it’s flat (very little curses or controller activity due to caching and being smart with redraws). Individual panels don’t contribute noticeably to the baseline.

May was a beautiful, chaotic haze that began with the GSoC acceptance fallout…

• blog posting
• acceptance introductions
• last minute coordination to get a student for the EFF
• adding proposals, mirrors, and minor template additions for next year’s GSoC
• git repository and ldap discussions

… and somehow ended with me as the mentor (or co-mentor) for five students. Karsten, Norman, and I will be mentoring three students from Wesleyan college to work on a Django rewrite of the TorStatus site. I only have a little prior experience with Django so a good portion of this month was spent reading the Django book to get up to speed (only got to chapter ten – I’m a horribly slow reader).

Meanwhile Kamran Khan has been hacking on the arm codebase, finishing his first week with a functional menus prototype. The implementation details turned out to be trickier than I’d anticipated and will need more polish, but my hat’s off to him for being able to dive into a completely new codebase and develop such a substantial contribution in so little time (thanks, Kamran!). For more on this see Kamran’s blog.

The fifth student is ‘identity’ from irc. He’s doing his thesis on arm, writing both a paper on its future plans and implementing a minor feature (NEWNYM functionality). He, Kamran, and I will be having periodic sync-ups via Mumble to discuss their projects and issues they run into. If this is successful I’ll suggest it to the Wesleyan students too.

Besides all of this, I’m also keeping an interested eye on Julien’s Metadata Anonymisation Toolkit. I did a code review for an initial bit he implemented and might do more if I both have the time and he wants the help.

In terms of arm development, I finished a complete rewrite of the codebase that’s been almost a year in the works. Besides being a far saner implementation, this paves the way for the further performance enhancements and features discussed on its wiki. /me is still not quite done doing his happy dance about this

Other interesting developments include…

• arm’s now able to rename its process name (thanks to help from Jake)
• added unicode printing support (thanks to Robert for the idea)
• backported to Debian Squeeze by Dererk
• is almost backported for Ubuntu Lucid and Maverick thanks to Wendy, Kamran, and Evan Broder

Once again my month’s been split between GSoC and arm. The former’s been a quite juggling act (especially the admin role), but well worth it. When the dust settled arm got a fantastic student. Kamran Khan will be hacking on arm this summer, working on several enhancement including manual path selection, UPnP support, a newnym option, and best of all a GTK front end.

In other news, this month started with the 1.4.2 arm release followed by hotfixes and improvements to its deb. Thanks to Dererk, TorCtl has been properly packaged for Debian as python-torctl and arm uses this (rather than a bundled copy) for its debs. This release also had metadata fixes suggested by intrigeri (debian bugs 623311 and 623312).

Thanks both to Sebastian and my new team of git-fanatics at Amazon, arm has finally migrated to git. Besides some workarounds for svn:externals and ‘svn export’ this has been a painless transition and I’m definitely glad I made the move.

As for arm development, this month has included several notable fixes, performance improvements, and cleaning of the codebase…

• Improved arm’s startup time by 83% (from 0.84 seconds to 0.14).
• Thanks to Erinn and Andrew I finally have access to a Mac. PID resolution and several important issues for arm on that platform have been fixed.
• The deprecated connection panel and file descriptor popup have been dropped from the codebase (together over 1500 lines).
• Fixed a critical parsing error for circuit paths in older Tor versions thanks to asn.
• Investigated the work needed for Windows compatibility and process renaming. Unfortunately neither are likely to happen any time soon.
• … and many more (file descriptor warnings, using new ‘traffic/*’ getinfo options, etc).

Ok, my project’s released, applicants have responses, and at long last I stand triumphant over my inbox nemesis (for the moment, anyway – he’ll be back). No more excuses so here’s my status report for March.

This last month was spent juggling a few things. First and foremost I’ve been swapping between my mentor and admin hats for GSoC. By day I’ve been that annoying, nagging guy asking devs to talk to all these young upstarts that won’t get off their lawn. But by night I transform into a hideous slave driver, bent on demanding more and more from the poor applicants to my projects. I’m actually not sure which group is more likely to plot my assassination…

Time permitting I’ve continued to hack on arm, adding some sweet new features like expanded circuit paths and application identification to the connection panel. The 1.4.2 release (which has been in the works for the last three months) is finally done, for more on that see its blog post.

And finally, I’ve survived my last oncall with RCX Checkout (only got paged thirteen times!) and transferred to the Source team of Builder Tools. Yesterday was my first day and so far I’m loving it, though I’m suspecting my old team is trying to hint something since they left me in their oncall rotation…

I’m usually weary of randomization but this last month had too many interesting things to keep from branching out a bit. GSoC is coming up and Andrew has been drumming up interest among students from Wesleyan. In preparation for both I’ve been sprucing up the Tor volunteer page, prepping our GSoC application, and contacting new potential mentors like Tomás, Robert Ransom, Robert Hogan, and the T(A)ILS community for project ideas.

I’ve also been preparing my own project for possible contributors, moving arm’s development notes and revised project ideas to the Tor wiki. This will be the canonical place for arm development information and upcoming plans (I get enough of scrums and burn-down charts at work, and this should act nicely as an alternative for keeping people informed).

Arm development has stayed relatively on track, with the revised connection panel very nearly achieving parity with its predecessor (and in most respects surpassing it). Most of what remains are refinements and tasty new features. Arm has also been added to Debian (Sid) and Ubuntu (Natty) with backports pending. Many thanks to Peter for his help.

As with last month, I have another round of being oncall for work which will sap a chunk of my time (and leave me grumpy). That aside, the 1.4.2 arm release should be done by the end of the month and I’ll be keeping an eye on the channels for the first round of inquisitive students. Fingers crossed that we find some good ones that stick around this year.