Hi all! Wanted to get my Pycon trip report out first but work keeps preempting that, so time to stop blocking. Lot of great stuff! But I suppose considering we’re already half way into June that’s no surprise…
Pycon and Stem Test Performance
Stay tuned for the full report, but highlights for me were discovery of mypy and multiprocessing.
Mypy is a static type checker with output similar to pyflakes or pycodestyle. Beginning with Python 3.5 core language syntax adds optional type information, and prior versions can provide this with comments. Mypy checks that function invocations match these types, providing type checks similar to what you’d get from static compilers like java.
I spent a bit integrating mypy with stem’s test suite but concluded it’s not ready for prime time quite yet. Mypy’s site cautions it’s still in early beta and indeed quite a few parts of stem (like enums) trip it up.
Multiprocessing on the other hand has been a big help.
Mulitprocessing is a python builtin that provides thread-style invocation of subprocesses. This is particularly interesting to me because for months I’ve struggled against the GIL for speeding up Stem’s tests. Multiprocessing is designed for just that, providing easy parallelization across cores.
Time since Pycon’s been spent integrating multiprocessing into our test suite, adding an @asynchronous decorator that invokes tests via subprocesses. On my antique dual core system this drops our test runtime from 53 to 22 seconds (58% faster). Should get even more on modern systems.
Evenings hit the Portland sights and tasty, tasty alcohol with my dad. Oh, and of course discussed Tor with Guido. It was a very fun trip.
Descriptor Creation
Stem and BridgeDB have always supported descriptor creation for their tests, with Isis even going the extra mile to make Leekspin. Taking the best of both Stem now supports a simple create() and content() function on descriptor classes…
RelayDescriptor.create({'router': 'demo 127.0.0.1 80 0 0'})
Leekspin’s still the only game in town for descriptor signing, but I hope to incorporate this soon too. You can read up more on this feature in our tutorials.
Couple other things this month include…
- Stem now validates tor consensus signatures. Many thanks to Tyler Parks for all his help making this happen!
- Passed GSoC administration on to Colin. Well past due we had fresh blood here.