Damian's Dev Blog

Squirrels! Five newborn squirrels are in our tree! And they… oh right, code. Talk about code, Damian, like a professional.

… awww, but damn they’re cute.

Stem Release 1.4.1

Though not as adorable as baby squirrels the new release of Stem is still pretty sweet. Ephemeral hidden services, substantially faster descriptor parsing, and a host of other improvements can all be yours with a simple upgrade!

Nyx Log Panel Rewrite

No, Nyx isn’t done but its log panel is! This is the third major curses component to be rewritten, and I’m delighted to say it now sucks less!

How did it suck, you ask? Though casual users may not have noticed arm had a laughably inefficient O(n³) approach to log deduplication. This was in fact so terrible arm simply turned off deduplication when it started bogging down the system too much.

This is just one of they myriad of improvements, but in Nyx deduplication is O(n²) and can easily be made linear (… though fuzzy matching makes this grosser than I’d like so first seeing if further improvement is warranted).

Few other noteworthy things were…

• Thanks to DonnchaC Stem’s tor message parsing is now a dramatic 300x faster when using python3!
• Wrote a simple bandwidth scanner to exemplify manual circuit construction for our tutorials.
• Sambuddha both made our tutorial examples downloadable and more easily testable!

Hi all! My April was split between SoP prep and Nyx. Changes for the later unfortunately aren’t yet ready so this is gonna be a shorter status report than usual.

Tor Summer of Privacy

It’s a thing! Of our twenty-two applicants sadly we only had funding for four, but all have great projects and we look forward to working with them this summer!

Personally I’m particularly excited since this will be the first time I’ve mentored since 2012. Cristobal’s working on a web dashboard for relay operators – think nyx, but ajax.

Few other noteworthy things were…

• Built on David’s hidden service descriptor work by adding a get_hidden_service_descriptor() method to Stem. Next up is support for Yawning’s ephemeral hidden services…
• Sambuddha revised Stem’s Twitter example to support Twitter’s v2 API.
• Stem test for installation via our setup.py.

Guten tag world! Time again for another status report…

Arm Graph Panel Rewrite

Really? Has it actually been four months?

Dusted off my work from November and finished rewriting arm’s graph panel. Besides untangling a mess of code that would make my family disown me, this leverages ‘GETINFO bw-event-cache’ to greatly improve graph prepopulation. Thanks, Nick, for adding that!

Coding aside, real challenge this month has been finding a better name. Thanks everyone who has helped (especially Mike!).

I’m now strongly leaning toward Nyx. Brevity aside, Nyx is the primordial goddess of the night with a description that’s positively delightful…

Her appearances are sparse in surviving mythology, but reveal her as a figure of such exceptional power and beauty, that she is feared by Zeus himself. She is found in the shadows of the world and only ever seen in glimpses.

Very curious to hear what the community thinks.

New Development Blog

Five years now I’ve dumped my status reports onto a single page. While this worked, clearly it lacked a certain… elegance. Moved to a proper blog…

• Before
• After

Personally I found it interesting how my style has evolved over the years. My current approach is to focus on a couple topics, followed by a brief list of additional things I’ve been up to. This is much easier to skim than the walls of text I once wrote.

Few other noteworthy things were…

• Stem can now read encrypted hidden service introduction-points.
• Support for the consensus’ new packages field and new HS extrainfo stats.
• You can now specify just a single test to run, rather than a whole module.
• clv expanded our tests to check if tor has capabilities Stem lacks.
• Sebastian sped up our integ tests by making them use trigger-able events.

Why can’t it always be winter? With murderous weather keeping you indoors life is so much more productive.

Hidden Service Descriptor Support

Stem now has the ability to read tor hidden service descriptors!

This is moot for the moment as there isn’t a method to get them, but with David’s ongoing work this is becoming a thing.

Google Summer of Code

Sadly we weren’t accepted into GSoC 2015. The program was trimmed back this year from 190 to 137 orgs, with several veteran orgs dropped to make room for newcomers. We’re in good company. Mozilla, Linux Foundation, Blender, and others were not included this year too. Friday we’re gonna get tips to improve our chances next year.

Pity, but c’est la vie. Regardless of GSoC we’re always happy to welcome newcomers, so if you’d like to get involved then don’t be shy!

Few other noteworthy things were…

• Gautham added connection resolution support for Windows!
• Foxboron added Tox support for testing multiple python versions.
• Notified relays prior to tor 0.2.4.18 that they should upgrade.
• Worked with Sebastian and Roger on a couple tor segfauts I ran into. (1, 2)
• Ported Tor’s argument handling tests to Stem.

Greetings wonderful world. January has been delightfully focused on a couple substantial feature branches that are now merged!

Descriptor Lazy Loading

When reading descriptors without validation (which is now the default) Stem is 25-70% faster!

This is because we now parse descriptors fields on-demand. Just need a server descriptor’s exit policy? That’s all we’ll bother reading. This is why the heavier the descriptor type the larger the benefit.

Unified Python 2/3 Codebase

Thanks to Foxboron Stem now has a unified python 2/3 codebase!

This means that Stem now runs directly with any version of python from 2.6 to 3.x. Prior to this Stem did a 2to3 export when you installed it to provide cross-version compatibility.

For users, the benefit of Foxboron’s overhaul is that python3 installation and test runs are now much, much quicker.

Beside these a few other noteworthy tasks were…

• Nick had a handful of suggestions for testing improvements, neatest of which was to include the command to re-run just individual tests when we have a failure.
• Our port_usage() function always returned None.
• Proc connection resolution could fail on busy systems.
• Fixed how the ‘strict’ argument works with exit policies.

Hi all! Tis that time of year coal appears in all my sox so guess it’s time for a status report. This month I set arm aside to polish improvements that accumulated in the Stem 1.3 release.

There were a lot of small tweaks, but the highlights for this month were…

Hidden Service Tutorial

By a bit of serendipity I came across a nice article by Jordan Wright for spinning up hidden services with Stem.

Federico Ceratto and Patrick O’Doherty have been improving our hidden service support, so I spruced up their contributions and wrote a tutorial of our own that demos it.

Gentoo Support

Thanks to Anthony Basile Stem is now packaged for Gentoo. Installing is as simple as…

  % emerge stem

I’ve been delighted at how many Gentoo people have jumped out of the woodwork to help. toralf worked with me on ticket to address testing issues, and Manuel took on Stem 1.3 packaging the same day as its release – thanks guys!

Couple other quick things of note is that I updated our OpenHub repository listing so it includes most of our newer projects, and trimmed membership of our internal lists.

I love the beginning of winter. Landscape a chilly tundra, no sun, yet still no snow to slip on. What I *don’t* love, however, is all the sicknesses that float around. This month has been uncommonly unlucky for me with three separate bugs making the rounds. No single one was a big whoop, but still can’t say I’m a fan of things making themselves at home in my warm, cozy respiratory track.

Oh well. Enough of that. This month was mostly focused on arm, specifically rewriting the graph panel. Work here is ongoing, but I hope to have this section done in November.

Other noteworthy things this month include…

• Stem now has a set of methods to more easily work with hidden services thanks to federico3 and Patrick. New methods include…

  • get_hidden_service_conf
  • set_hidden_service_conf
  • create_hidden_service
  • remove_hidden_service

• Addressed OOM issues with DocTor. DocTor uses quite a bit of memory, but the real trouble was that sending a notification at the end forked, doubling allocated memory. I addressed this by spawning the mail subprocess earlier, before we read in descriptor data.

• Couple DocTor patches from Tom Ritter – thanks Tom! (1, 2)

• Subteam discussions with Karsten and OONI folks accumulating in the VegasTeam3 wiki. I’m dubious that this will mean much (or anything) in practice, but we’ll see.

• Attended a couple local tech events including TA3M Seattle and Seagl.

• Got promoted at work! This was five years in the making, so I’m pretty delighted.