Hi all. Changes at work have me stressed so I’ll be skipping Rome this year, but none the less Tor has been a welcome anchor. Work may suck, but Tor? Well…

ORPort Protocol Support

As discussed yesterday Stem can now communicate over the ORPort protocol. Still lots of follow-up work to do, but thanks to Tim’s wonderful work prototyping how this is done with Endosome Stem can now download descriptors via the ORPort protocol!

import stem.client

with stem.client.Relay.connect('127.0.0.1', 12345, [3]) as relay:
  circ = relay.create_circuit()
  circ.send('RELAY_BEGIN_DIR', stream_id = 1)
  desc = circ.send('RELAY_DATA', 'GET /tor/server/authority
HTTP/1.0\r\n\r\n', stream_id = 1).data
  circ.close()

  print(desc)

Which then provides…

% python demo.py
HTTP/1.0 200 OK
Date: Wed, 07 Feb 2018 18:42:41 GMT
Content-Type: text/plain
Content-Encoding: identity
Expires: Fri, 09 Feb 2018 18:42:41 GMT

router Unnamed 97.113.177.53 12345 0 23456
identity-ed25519
-----BEGIN ED25519 CERT-----
AQQABm/qAazUltT1iUUbIMw8VNNhGb50FDHKJz6S94FLQNxL0LObAQAgBAAapbO9
iLFD0l9SEiEMFQWIT2VnbLyCZKvbrxTs5ULC1l1hQPoui6Y/lEd3yjrQhIs/vl6R
1S6FbwSFDmiXOzq47mFrse4C71ht3TpLOD0F3wiyjWtsqU1k7iPmmpejUgs=
-----END ED25519 CERT-----
master-key-ed25519 GqWzvYixQ9JfUhIhDBUFiE

Fallback Directory v2

Tim and I also worked together on a second iteration for our Fallback Directories. Expanded with additional data and a specification, Stem now supports the new format.

Happy New Years! Hope everyone’s holidays were delightful. Between candy canes and sugarplumbs December had some neat stuff…

Tutorials

Filled gaps within our tutorials, in particular multiprocessing and terminal styling…

Also worked with toralf on a demo for summarizing relay connections…

Packaging

To help us keep our various platforms up to date last month I put together a packages wiki. Sadly, the trouble with hand-edited wikis is that they get out of date.

To help with this we now have a daemon that notifies me when platforms have a new package available…

Hi wonderful world. Post-release followup and recent interview with Ben Collier has kept me pretty busy, but have a couple other fun things to report…

Packaging Community

Commonly major releases are followed by followup packaging work and Nyx’s recent release was no exception. But rather than simply work with our delightful packaging community as I usually do I decided instead to bring order to the chaos.

I’m delighted to say we now have a tor-packagers@ list where Tor developers can reach our packaging community, and packagers can subscribe to be notified of new releases. To go along with this I also made a wiki that gives an overview of our packages…

Tor Relay: Caer Sidi

In folklore Caer Sidi was an otherworldly fortress, unsuccessfully assailed by the Prydwin in Arthur’s quest for the holy grail. It’s a name I’ve always wanted to use for a relay.

Flavor text aside, while ago I got permission from Dreamhost to run a non-exit relay on their cloud infrastructure so I’d have a busy relay on which to test Nyx. I haven’t tried pushing the envelope, but I can say it’s been a nice low-cost ($6/month) hosting experience thus far. It’s now listed as a good experience on our ISP wiki.

Seems I fail at sending these reports on time. Oh well, on with the show!

Nyx and Stem Release

As you may have guessed from last week’s announcement been busy with the launch of Nyx and Stem 1.6!

Very special thanks to…

• Tommy and Stephanie for their delightful blog post.
• All the relay operators that helped beta test Nyx.
• George, Attila, and pascal for our new Stem and Nyx OpenBSD ports.
• Derek, Juan, Anthony, Sjon, Alex, Markus, and Carlo for updating Stem and packaging Nyx for all our existing platforms. Damn you guys are fast!

Again, thanks all! Each of these made me do a little happy dance.

Montreal Developer Meeting

It’s the sign of a busy month if this isn’t my top highlight. As always Jon, Gunner, and Alison orchestrated a great meeting. Between discussions hit the sights and tried poutine for the first time.

As a final note the Tor office moved this month. Sadly this means we’re losing an absolutely fantastic mural done for us by Henry. However, we took some high definition photos. Here’s one I’ve rescaled to be usable as a wallpaper or poster…

Oh how I love Sol Duc. Hundreds of miles of trails, white birch groves, and of course sulfuric hotsprings. Rainforest moss lends the woods an unearthly charm that’s truly just delightful.

Highlight of this month for me was a vacation with my dad, first to Port Townsend then Sol Duc. But this has been an interesting month on other fronts too.

Nyx Performance

This month my chief focus has been tuning Nyx. CPU usage is now 20% lower, and far more responsive under verbose logging due to constant time log deduplication. Overall Nyx finally looks ready for release. I’ll probably move forward with a call for beta testers after the dev meeting.

Membership

This month tor ratified a policy for internal list membership. Much of it just formalizes procedures we’ve used for a while, but it also adds a requirement on continued involvement to stay subscribed. Naturally volunteers move on to other projects over time and this perfectly fine, but eventually their membership will be suspended with re-addition fast tracked if they return.

I’m now facilitating discussions with the Vegas leads and Council on how best to determine this. To give us a starting point I put together some fun stats, but numbers alone don’t come anywhere close to answering the question of “how has this person been involved in making tor better in the last six months?”. Discussions ongoing.

Hi all. This month been down in the engine room productionizing Nyx, with a special focus on memory usage. Dropped ~13%, but still not where I want so investigations are ongoing.

Nyx SQLite Cache

Main benefit came from moving cached consensus information from memory to SQLite. Besides the obvious memory benefits this allows the cache to persist between invocations, halving Nyx’s startup time (from 0.7 to 0.3 seconds).

Tor Manual Database

Stem provides easy programmatic access for Tor’s manual information. SQLite now backs this information, providing 8x faster reads (Manual.from_cache() dropped 16ms to 2ms), and now supports random access reads…

>>> import stem.manual
>>> print(stem.manual.query('SELECT description FROM torrc WHERE key=?', 'CONTROLSOCKET').fetchone()[0])
Like ControlPort, but listens on a Unix domain socket, rather than a TCP socket.  0 disables ControlSocket (Unix and Unix-like systems only.)

This further drops Nyx’s memory usage by allowing it to only fetch the manual information it needs.

Stem Multi-Processing

May’s test performance investigation has now led to a general purpose DaemonTask class to make Python multi-processing easy…

Threaded

import threading
import time

def fibonacci(n):
  if n < 2:
    return n
  else:
    return fibonacci(n-2) + fibonacci(n-1)

# calculate fibonacci sequences four times in parallel

start_time, threads = time.time(), []

for i in range(4):
  t = threading.Thread(target = fibonacci, args = (35,))
  t.setDaemon(True)
  t.start()

  threads.append(t)

for t in threads:
  t.join()

print('took %0.1f seconds' % (time.time() - start_time))

% python fibonacci_threaded.py
took 21.1 seconds

Multi-Process

import stem.util.system
import time

def fibonacci(n):
  if n < 2:
    return n
  else:
    return fibonacci(n-2) + fibonacci(n-1)

# calculate fibonacci sequences four times in parallel

start_time, threads = time.time(), []

for i in range(4):
  threads.append(stem.util.system.DaemonTask(fibonacci, (35,), start = True))

for t in threads:
  t.join()

print('took %0.1f seconds' % (time.time() - start_time))

% python fibonacci_multiprocessing.py
took 6.2 seconds

Presently this is only used for our tests, but soon I'll take advantage of this to make Nyx more performant on multi-core systems.

Hi all. This month pretended to be a web developer and made pretty things. Not an area I dabble in often but it was fun doing something artsy.

New Nyx Website

My eyes, they bleed! Made arm’s old site back in my college days and it shows…

I’m delighted to announce Nyx’s new site is now live! New paint job aside our new site describes Nyx’s features, adds a FAQ, and better changelog.

Tor People Page

So many new people. Between newhires and new volunteers Tor is growing quite a bit right now and a common complaint I’ve heard is that our community makes it maddeningly hard to figure out who’s who. For some of us this is intentional, but others it’s not…

“Who’s that arma guy I was talking with on irc? He seemed nice. Ok, thanks. Now which of these meeting attendees is Roger? Great. And, what does he do?“. Multiply that by ninety of us and it’s no wonder we drive our lovely hair-pulling newcomers to early baldness.

For those that would care to take part our people page now provides irc nicks, descriptions, pgp keys, twitter handles, and pictures. Hope this helps ya get a better idea of who those disembodied voices on irc are!

Hi all! Things I’m jugging are still in the air so this will be short. So what did I do this month besides buy a very, very manly puffer fish?

… sorry, those buggie eyes keep making me smile.

Migrate BridgeDB to Stem

Stem can now sign server and extrainfo descriptors! This was the last bit we needed for Leekspin parity. Isis and I are now swapping BridgeDB over.

After that’s been merged for a week or so I’ll likely cut the Stem 1.6 release. We have quite a few improvements accumulated so it’s about time…

Smaller things this month include…

• Stem can now fetch connection data using fstat. This give us, and by extension Nyx, connection information on OpenBSD. Thanks cnidaria!
• Expanded our integ coverage of hidden services.
• Fixed a caching issue that caused perpetually stale source code in the ‘[source]’ links on Stem’s site.

Oh, and of course attended a local Retro Gaming Expo, Fremont Festival, great performance of Cabarat, and a coworker’s BBQ. Hope everyone else is having a similarly fun summer!

Hi all! Wanted to get my Pycon trip report out first but work keeps preempting that, so time to stop blocking. Lot of great stuff! But I suppose considering we’re already half way into June that’s no surprise…

Pycon and Stem Test Performance

Stay tuned for the full report, but highlights for me were discovery of mypy and multiprocessing.

Mypy is a static type checker with output similar to pyflakes or pycodestyle. Beginning with Python 3.5 core language syntax adds optional type information, and prior versions can provide this with comments. Mypy checks that function invocations match these types, providing type checks similar to what you’d get from static compilers like java.

I spent a bit integrating mypy with stem’s test suite but concluded it’s not ready for prime time quite yet. Mypy’s site cautions it’s still in early beta and indeed quite a few parts of stem (like enums) trip it up.

Multiprocessing on the other hand has been a big help.

Mulitprocessing is a python builtin that provides thread-style invocation of subprocesses. This is particularly interesting to me because for months I’ve struggled against the GIL for speeding up Stem’s tests. Multiprocessing is designed for just that, providing easy parallelization across cores.

Time since Pycon’s been spent integrating multiprocessing into our test suite, adding an @asynchronous decorator that invokes tests via subprocesses. On my antique dual core system this drops our test runtime from 53 to 22 seconds (58% faster). Should get even more on modern systems.

Evenings hit the Portland sights and tasty, tasty alcohol with my dad. Oh, and of course discussed Tor with Guido. It was a very fun trip.

Descriptor Creation

Stem and BridgeDB have always supported descriptor creation for their tests, with Isis even going the extra mile to make Leekspin. Taking the best of both Stem now supports a simple create() and content() function on descriptor classes…

RelayDescriptor.create({'router': 'demo 127.0.0.1 80 0 0'})

Leekspin’s still the only game in town for descriptor signing, but I hope to incorporate this soon too. You can read up more on this feature in our tutorials.

Couple other things this month include…

• Stem now validates tor consensus signatures. Many thanks to Tyler Parks for all his help making this happen!
• Passed GSoC administration on to Colin. Well past due we had fresh blood here.

Hi all. Emergent work encroached into my weekends a bit so less progress on my projects. On the nice side though had fun writing a PoE build and saw Cirque Du Soleil: Luzia with my family.

As always their performance was fantastic.

Tor Internal Bylaws

Alison’s Social Contract passed, and we’re presently in the process of electing a new Community Council. To this end helped conduct a candidate Q&A.

For a variety of reasons some folks would prefer not to vote. Some don’t have time, while others feel uncomfortable with voting due to their position in TPI or Tails. As such we’re now allowing folks to opt-out of counting toward the quorum requirement if they want.

tor-prompt –run

Stem’s tor-prompt command now supports a –run argument to run individual commands…

… listen to a space separated list of events with –run ‘SETEVENTS [event_types]’…

… pipe its output to other destinations…

… and invoke files with a series of commands…