Squirrels! Five newborn squirrels are in our tree! And they… oh right, code. Talk about code, Damian, like a professional.

… awww, but damn they’re cute.

Stem Release 1.4.1

Though not as adorable as baby squirrels the new release of Stem is still pretty sweet. Ephemeral hidden services, substantially faster descriptor parsing, and a host of other improvements can all be yours with a simple upgrade!

Nyx Log Panel Rewrite

No, Nyx isn’t done but its log panel is! This is the third major curses component to be rewritten, and I’m delighted to say it now sucks less!

How did it suck, you ask? Though casual users may not have noticed arm had a laughably inefficient O(n³) approach to log deduplication. This was in fact so terrible arm simply turned off deduplication when it started bogging down the system too much.

This is just one of they myriad of improvements, but in Nyx deduplication is O(n²) and can easily be made linear (… though fuzzy matching makes this grosser than I’d like so first seeing if further improvement is warranted).

Few other noteworthy things were…

• Thanks to DonnchaC Stem’s tor message parsing is now a dramatic 300x faster when using python3!
• Wrote a simple bandwidth scanner to exemplify manual circuit construction for our tutorials.
• Sambuddha both made our tutorial examples downloadable and more easily testable!

Hi all! My April was split between SoP prep and Nyx. Changes for the later unfortunately aren’t yet ready so this is gonna be a shorter status report than usual.

Tor Summer of Privacy

It’s a thing! Of our twenty-two applicants sadly we only had funding for four, but all have great projects and we look forward to working with them this summer!

Personally I’m particularly excited since this will be the first time I’ve mentored since 2012. Cristobal’s working on a web dashboard for relay operators – think nyx, but ajax.

Few other noteworthy things were…

• Built on David’s hidden service descriptor work by adding a get_hidden_service_descriptor() method to Stem. Next up is support for Yawning’s ephemeral hidden services…
• Sambuddha revised Stem’s Twitter example to support Twitter’s v2 API.
• Stem test for installation via our setup.py.

Guten tag world! Time again for another status report…

Arm Graph Panel Rewrite

Really? Has it actually been four months?

Dusted off my work from November and finished rewriting arm’s graph panel. Besides untangling a mess of code that would make my family disown me, this leverages ‘GETINFO bw-event-cache’ to greatly improve graph prepopulation. Thanks, Nick, for adding that!

Coding aside, real challenge this month has been finding a better name. Thanks everyone who has helped (especially Mike!).

I’m now strongly leaning toward Nyx. Brevity aside, Nyx is the primordial goddess of the night with a description that’s positively delightful…

Her appearances are sparse in surviving mythology, but reveal her as a figure of such exceptional power and beauty, that she is feared by Zeus himself. She is found in the shadows of the world and only ever seen in glimpses.

Very curious to hear what the community thinks.

New Development Blog

Five years now I’ve dumped my status reports onto a single page. While this worked, clearly it lacked a certain… elegance. Moved to a proper blog…

• Before
• After

Personally I found it interesting how my style has evolved over the years. My current approach is to focus on a couple topics, followed by a brief list of additional things I’ve been up to. This is much easier to skim than the walls of text I once wrote.

Few other noteworthy things were…

• Stem can now read encrypted hidden service introduction-points.
• Support for the consensus’ new packages field and new HS extrainfo stats.
• You can now specify just a single test to run, rather than a whole module.
• clv expanded our tests to check if tor has capabilities Stem lacks.
• Sebastian sped up our integ tests by making them use trigger-able events.

Why can’t it always be winter? With murderous weather keeping you indoors life is so much more productive.

Hidden Service Descriptor Support

Stem now has the ability to read tor hidden service descriptors!

This is moot for the moment as there isn’t a method to get them, but with David’s ongoing work this is becoming a thing.

Google Summer of Code

Sadly we weren’t accepted into GSoC 2015. The program was trimmed back this year from 190 to 137 orgs, with several veteran orgs dropped to make room for newcomers. We’re in good company. Mozilla, Linux Foundation, Blender, and others were not included this year too. Friday we’re gonna get tips to improve our chances next year.

Pity, but c’est la vie. Regardless of GSoC we’re always happy to welcome newcomers, so if you’d like to get involved then don’t be shy!

Few other noteworthy things were…

• Gautham added connection resolution support for Windows!
• Foxboron added Tox support for testing multiple python versions.
• Notified relays prior to tor 0.2.4.18 that they should upgrade.
• Worked with Sebastian and Roger on a couple tor segfauts I ran into. (1, 2)
• Ported Tor’s argument handling tests to Stem.

Greetings wonderful world. January has been delightfully focused on a couple substantial feature branches that are now merged!

Descriptor Lazy Loading

When reading descriptors without validation (which is now the default) Stem is 25-70% faster!

This is because we now parse descriptors fields on-demand. Just need a server descriptor’s exit policy? That’s all we’ll bother reading. This is why the heavier the descriptor type the larger the benefit.

Unified Python 2/3 Codebase

Thanks to Foxboron Stem now has a unified python 2/3 codebase!

This means that Stem now runs directly with any version of python from 2.6 to 3.x. Prior to this Stem did a 2to3 export when you installed it to provide cross-version compatibility.

For users, the benefit of Foxboron’s overhaul is that python3 installation and test runs are now much, much quicker.

Beside these a few other noteworthy tasks were…

• Nick had a handful of suggestions for testing improvements, neatest of which was to include the command to re-run just individual tests when we have a failure.
• Our port_usage() function always returned None.
• Proc connection resolution could fail on busy systems.
• Fixed how the ‘strict’ argument works with exit policies.

Hi all! Tis that time of year coal appears in all my sox so guess it’s time for a status report. This month I set arm aside to polish improvements that accumulated in the Stem 1.3 release.

There were a lot of small tweaks, but the highlights for this month were…

Hidden Service Tutorial

By a bit of serendipity I came across a nice article by Jordan Wright for spinning up hidden services with Stem.

Federico Ceratto and Patrick O’Doherty have been improving our hidden service support, so I spruced up their contributions and wrote a tutorial of our own that demos it.

Gentoo Support

Thanks to Anthony Basile Stem is now packaged for Gentoo. Installing is as simple as…

  % emerge stem

I’ve been delighted at how many Gentoo people have jumped out of the woodwork to help. toralf worked with me on ticket to address testing issues, and Manuel took on Stem 1.3 packaging the same day as its release – thanks guys!

Couple other quick things of note is that I updated our OpenHub repository listing so it includes most of our newer projects, and trimmed membership of our internal lists.

I love the beginning of winter. Landscape a chilly tundra, no sun, yet still no snow to slip on. What I *don’t* love, however, is all the sicknesses that float around. This month has been uncommonly unlucky for me with three separate bugs making the rounds. No single one was a big whoop, but still can’t say I’m a fan of things making themselves at home in my warm, cozy respiratory track.

Oh well. Enough of that. This month was mostly focused on arm, specifically rewriting the graph panel. Work here is ongoing, but I hope to have this section done in November.

Other noteworthy things this month include…

• Stem now has a set of methods to more easily work with hidden services thanks to federico3 and Patrick. New methods include…

  • get_hidden_service_conf
  • set_hidden_service_conf
  • create_hidden_service
  • remove_hidden_service

• Addressed OOM issues with DocTor. DocTor uses quite a bit of memory, but the real trouble was that sending a notification at the end forked, doubling allocated memory. I addressed this by spawning the mail subprocess earlier, before we read in descriptor data.

• Couple DocTor patches from Tom Ritter – thanks Tom! (1, 2)

• Subteam discussions with Karsten and OONI folks accumulating in the VegasTeam3 wiki. I’m dubious that this will mean much (or anything) in practice, but we’ll see.

• Attended a couple local tech events including TA3M Seattle and Seagl.

• Got promoted at work! This was five years in the making, so I’m pretty delighted.

Code! Beautiful code! How I’ve missed thee!

As mentioned before I’m hip deep in the middle of an arm rewrite. Not only to overhaul the arm codebase, but to ensure Stem will be up to snuff for a Tor GUI or web dashboard (projects I’d love to jump into someday). I’m delighted to announce a nice bit of progress on this front.

September went toward rewriting its header panel (the top portion of arm’s interface)…

• Old zomg my eyes, they bleed!
• New hotness

As mentioned though this is really more about Stem than arm. Stem improvements
made along the way this month include…

• Far better handling for the ‘private’ keyword (the default prefix) and the default suffix in our ExitPolicy class. (ticket)
• New BaseController.connection_time() method that provides when our control connection was established or detached.
• New Controller.get_accounting_stats() method which reports accounting information for our relay.
• Controller now fetches our own descriptor if no fingerprint is provided to get_server_descriptor(), get_network_status(), or get_microdescriptor().
• New crop() and join() methods for the str_tools module.
• Dropped a redundant get_* prefix from most of our util function names. Old names still work as aliases.
• Our launch_tor_with_config() function raised an OSError if called too many times, caught thanks to RSenet. (ticket)

DocTor, the monitor for directory authority health, also got some love this month…

• Corrected suppression behavior when we have staggered alerts. This should greatly cut down on the amount of noise on the list.
• We now check if too many relays lack a bandwidth authority measurement… or we would if this wouldn’t perpetually be in alarm. (ticket 1, 2)
• Revised DocTor’s schedule as requested by Sebastian. (ticket)
• Expanded suppression for messages about turtles and kicked off a conversation to at long last sort this out. (ticket)

One final note is that I uploaded my Tor Ecosystem presentation to YouTube so volunteer page visitors can see a streaming presentation rather than downloading a 53 MB video.

Hi all, getting my status report out of the way a tad early. Deadlines for my day job at Amazon required working a weekend so April was mostly spent just getting caught up on code reviewing contributions…

• Nick Mathewson greatly improved Stem’s performance at parsing consensus documents. Thanks Nick!
• Rewrote Nick Hopper’s patch for verifying consensus signatures. I’d really love to merge it, but it needs tests first.
• Code reviewed federico3’s addition of better Controller methods for handling hidden service config options.
• Fixed stylistic issues cited by the newly released pep8 1.5.7, and also corrected a handful of python 3.x test failures.
• Code reviewed a BridgeDB change to take advantage of Stem for isis.
• GSoC finals are finished! This year had a lower pass rate than we usually do (four of our thirteen students didn’t pass), but I don’t think there’s necessarily any lessons to draw.

This concludes my backlog of things I owe other people, so next up I’ll finally be getting back to arm development!

For one brief, glorious moment in time I touched code. Really, it happened! House plants can vouch as my witness.

Or that is to say, I’ve coded a lot less than I’d like. Barring a long overdue vacation to Sol Duc this month went toward a lot of random distractions…

• Worked with Philipp on overhauling our procedure for flagging bad relays. Philipp’s taken the lead here and is doing a great job! While I find this space interesting I’m already spread too thin, so going forward I won’t be a co-maintainer for it after all.

• Looked over Nick Hopper’s consensus validation change, adding crypto blob type validation to simplify his work. I now owe him a more thorough code review.

• Handful of DocTor issues including OOM troubles, unpleasant amount of noise from consensus parameters, and a several minor revisions.

• Discussed adding a new X- extrainfo descriptor field with Virgil, suggesting revisions and helping him make it conformant with the spec.

• System I do development on died a puzzling death. Even after twenty hours investigating I’m still not quite sure what happened. Cleaning out dust, removing the RAID controller, and starting over seems to have done the trick.

• GSoC administrative work. We’re now less than two weeks from the final evaluations!

• Continued overhauling arm but as already mentioned focused on this a lot less on this than I’d like.

So little coding makes me grumpy, so like many of you I’m taking a hard look at where my time’s evaporating to. Maybe there’s something I can change or extricate myself from to better focus.