Closing the decade I’m pleased to announce Stem 1.8, the final release in Stem’s 1.x series and with it Python 2.x support.

What is Stem, you ask? For those who aren’t familiar with it Stem is a Python library for interacting with Tor. With it you can script against your relay, descriptor data, or even write applications like Nyx.

https://stem.torproject.org/

So what’s new in this release?


CollecTor Downloader

Through our descriptor archive, CollecTor, Stem can now read Tor’s network topology at any prior point in time.

For example, listing today’s exits is as simple as…

import datetime
import stem.descriptor.collector

yesterday = datetime.datetime.utcnow() - datetime.timedelta(days = 1)
exits = {}

for desc in stem.descriptor.collector.get_server_descriptors(start = yesterday):
  if desc.exit_policy.is_exiting_allowed():
    exits[desc.fingerprint] = desc

print('%i relays published an exiting policy today...\n' % len(exits))

for fingerprint, desc in exits.items():
  print('  %s (%s)' % (desc.nickname, fingerprint))
% python demo.py 
1229 relays published an exiting policy today...

  MrExit (D628F6BB2330B3F78DBB4BED466B0A586D74782E)
  pangea03 (F21DFB7CCD5EEF3E021086EC96EF7CFCAA72F4F3)
  MacarenaValdes (5E3FD31B9DC279C06AD051D68BE08914F6CD3B46)
  TEMPORA (05EAA0696DCB694D6811042348DACD5059FE64AD)
  Quintex43 (1E5136DDC52FAE1219208F0A6BADB0BA62587EE6)

Bandwidth Metrics

Bandwidth Authorities generate the latency heuristics that govern Tor’s path selection. Guiding circuits to be fast, without overburdening individual relays.

With Stem you can peruse this information…

import stem.descriptor.remote
import stem.util.str_tools

bandwidth_file = stem.descriptor.remote.get_bandwidth_file().run()[0]

print('Bandwidth measurements are...\n')

for fingerprint, measurement in bandwidth_file.measurements.items():
  bandwidth = '%s/s' % stem.util.str_tools.size_label(1024 * int(measurement.get('bw', '0')))
  print('  * %s (%s) averaged %s' % (measurement.get('nick', ''), fingerprint, bandwidth))
Bandwidth measurements are...

  * DigiGesTor1e1 (0111BA9B604669E636FFD5B503F382A4B7AD6E80) averaged 23 MB/s
  * WonderWoman42 (E5AA85FA69CDC31900C86E6427C7E5DE11DE9E2D) averaged 37 MB/s
  * alterspalter (B6F0BC2B93CB3EFFFFF724CB4F5E025FB15EFB70) averaged 2 MB/s
  * blueberry (FE80E192AD48A1BEB02D88EBC7663061176E1A79) averaged 1 KB/s
...

Hidden Service v3 Descriptors

With George’s help Stem now reads, decrypts, and even creates HSv3 descriptors. For example…

from stem.descriptor.hidden_service import (
  HiddenServiceDescriptorV3,
  InnerLayer,
  IntroductionPointV3,
)

print(HiddenServiceDescriptorV3.content(
  inner_layer = InnerLayer.create(
    introduction_points = [
      IntroductionPointV3.create('1.1.1.1', 9001),
      IntroductionPointV3.create('2.2.2.2', 9001),
      IntroductionPointV3.create('3.3.3.3', 9001),
    ],
  ),
))
% python demo.py
hs-descriptor 3
descriptor-lifetime 180
descriptor-signing-key-cert
-----BEGIN ED25519 CERT-----
AQgABqvHAX8wXzJY+FqoJQPXNZ8u+SQGPZ1WN/r3hUna0R2AXQnEAQAgBAAuqibl
ALcKa/4nHtLZn2zKV8L4XIpkRyRm7btWPLpYN5Gseb03H5exL+I3SqfG3uNDw5QK
CmPlCQUy3usouSwhO/qWgdy0//bP5kRDma5GDXXWoi3+xTKM6Jez7TGxPAU=
-----END ED25519 CERT-----
revision-counter 1573695064
superencrypted
-----BEGIN MESSAGE-----
aDJodcMjhCvz1K7JCJEAH1H24hvoZ7gZw53AhPdvpHu+5d1Ogwio4qcIXEK1pEgy
QFF1fE6tnCzsk++eMa2WaKwIJYGLPoCnta78H5Ve6VoMj+Pyb5rE6wPTMTPSVm6M
UjllArr7DS8YcofloDxu3iwC3JZYFt/LB6ahq6lBKeot2BD/11pNggkZrZOCLgNQ
pUVyQau7K8ynagVlNNESnI3FccOBaBB4Xa5mObK2ylyiLQ08MqaImW7X2gxeZltT
/C/xtiJXGm2CzkjPpBpMWm09p7/a97GEWca5e8+fhpmGrN7zjAwjYInTvQHS5AyU
7eUFg8ItrRxAiRq4fbe/zepiq2vgfj1Pt7uxC0KCTcLWpd9O/FIvcFSk27Yrtniw
... etc...
-----END MESSAGE-----
signature VDDXXLvgU6qjRI4zfJR3GbQuVjz98qO0LI5gsI60LtGXK2POZ4E+3YVVWuVaEkvMsZaku5qCutIcu74/WQMxCQ

Happy turkey day everyone! George and I were busy with Stem’s HSv3 descriptor branch so I skipped last month’s report, but now have spiffy things to share.

Oh, and wrote a little guide for a mobile game I enjoy and finished unpacking my homely nest!

living room
office


HSv3 Descriptor Support

Stem can now create, parse, and decrypt hidden service v3 descriptors. For example…

from stem.descriptor.hidden_service import (
  HiddenServiceDescriptorV3,
  InnerLayer,
  IntroductionPointV3,
)

print(HiddenServiceDescriptorV3.content(
  inner_layer = InnerLayer.create(
    introduction_points = [
      IntroductionPointV3.create('1.1.1.1', 9001),
      IntroductionPointV3.create('2.2.2.2', 9001),
      IntroductionPointV3.create('3.3.3.3', 9001),
    ],
  ),
))
% python demo.py
hs-descriptor 3
descriptor-lifetime 180
descriptor-signing-key-cert
-----BEGIN ED25519 CERT-----
AQgABqvHAX8wXzJY+FqoJQPXNZ8u+SQGPZ1WN/r3hUna0R2AXQnEAQAgBAAuqibl
ALcKa/4nHtLZn2zKV8L4XIpkRyRm7btWPLpYN5Gseb03H5exL+I3SqfG3uNDw5QK
CmPlCQUy3usouSwhO/qWgdy0//bP5kRDma5GDXXWoi3+xTKM6Jez7TGxPAU=
-----END ED25519 CERT-----
revision-counter 1573695064
superencrypted
-----BEGIN MESSAGE-----
aDJodcMjhCvz1K7JCJEAH1H24hvoZ7gZw53AhPdvpHu+5d1Ogwio4qcIXEK1pEgy
QFF1fE6tnCzsk++eMa2WaKwIJYGLPoCnta78H5Ve6VoMj+Pyb5rE6wPTMTPSVm6M
UjllArr7DS8YcofloDxu3iwC3JZYFt/LB6ahq6lBKeot2BD/11pNggkZrZOCLgNQ
pUVyQau7K8ynagVlNNESnI3FccOBaBB4Xa5mObK2ylyiLQ08MqaImW7X2gxeZltT
/C/xtiJXGm2CzkjPpBpMWm09p7/a97GEWca5e8+fhpmGrN7zjAwjYInTvQHS5AyU
7eUFg8ItrRxAiRq4fbe/zepiq2vgfj1Pt7uxC0KCTcLWpd9O/FIvcFSk27Yrtniw
... etc...
-----END MESSAGE-----
signature VDDXXLvgU6qjRI4zfJR3GbQuVjz98qO0LI5gsI60LtGXK2POZ4E+3YVVWuVaEkvMsZaku5qCutIcu74/WQMxCQ

This branch took quite a lot of work. Many thanks to George and Paul for all the cryptography help!


Other things include…

Hey everyone. This morning my grandma passed away peacefully in her sleep at 6am. She was ninety years old and ready to go. As the saying goes: “old age ain’t for sissies”.

These last few months supporting my family has been a full time job, but my schedule should now free back up.

sunset


CollecTor Support

The only technical thing to note is CollecTor support, a new module within Stem that makes gathering Tor’s network topology at prior points in time simple. For example…

import datetime
import stem.descriptor.collector

yesterday = datetime.datetime.utcnow() - datetime.timedelta(days = 1)

# provide yesterday's exits

for desc in stem.descriptor.collector.get_server_descriptors(start = yesterday):
  if desc.exit_policy.is_exiting_allowed():
    print('  %s (%s)' % (desc.nickname, desc.fingerprint))

Behold, my glorious domain! Weep at its splendor!!1!

granny's attic

This month returned to the island I love, and having a blast volunteering locally. Both non-profits are amazing.

Granny’s is a second hand store that fundraises for our community. Volunteering there is a fun mixture of serendipity with the preeminent form of recycling. I really love matching excess stuff with homes that can make good use of it.

As for our food bank, it feeds fully a seventh of our island’s population. That’s ~150 people! Twice a week I’m unloading trucks and re-bagging groceries. After a decade at Amazon I gotta admit – there’s something deliciously cathartic in non-technical work.

Tor is taking a back seat right now, but I have a feature branch for CollecTor that should be ready to announce soon.

Aside from that lots of runs, plays, and other fun stuff. This is kicking off to be a very fun summer!

Freedom! Sweet, delicious freedom! I went dark since March for crunch time at Amazon, but then decided one decade there was enough. Amazon was a great first job but time for a break.

Vacation aside, this summer I’m volunteering with our island food bank and Granny’s Attic to get more balance away from the keyboard. That said, knowing myself I’ll still need my open source fix.

Oh! And of course, most important of all: found a new puffy friend!

mini puffer

Damn I’m a sucker for puffers.


Tor/Stem CI

Stem’s tests were designed to verify Tor/Stem’s interoperability, as well as provide a tool to test Tor itself. This month teor integrated Stem into Tor’s Travis CI and we’ve been working out the kinks…


Smaller things these last few months include…

Greetings wonderful tor community! This month I pretended to be a web development again for kicks and giggles.


Tor People Page

Hiro’s putting together a new website for Tor. As the maintainer of our people page I ported it over. Personally I think the new page looks a lot better.

Before

people page before

After

people page after

Hrio declined my patch, and that’s fine. Learning Lektor was fun. The new site will launch with Hiro’s version of the page with her as its maintainer.


Personal Website

Beside the Tor website I also put the finishing touches on my own! For comparison my old site is available here.

personal website

Happy January, onion enthusiasts! Made it through another spin of the globe. January I wrapped up a couple projects that were in flight…


Nyx 2.1 Release

My biggest news for users this month is Nyx 2.1! No features, but this release includes a full year’s worth of fixes that smooths Nyx’s rough edges.

If you run into any issues please let me know!


Bandwidth File Support

Bandwidth files are weighting heuristics used by Tor for relay selection. Stem can now read these documents. For example…

import stem.descriptor
import stem.util.str_tools

# Bandwidth files are not yet available, but soon will be...
#
#   https://trac.torproject.org/projects/tor/ticket/21377

with open('/home/atagar/Desktop/bandwidth_file_v1.2') as bandwidth_file:
  desc = next(stem.descriptor.parse_file(bandwidth_file, 'bandwidth-file 1.2'))

print('Median relay bandwidth measurements are...\n')

for fingerprint, measurement in desc.measurements.items():
  # Not quite sure what the units are. The spec seems to indicate KB/s?

  bandwidth = '%s/s' % stem.util.str_tools.size_label(1024 * int(measurement.get('bw_mean', '0')))
  print('  * %s (%s) averaged %s' % (measurement.get('nick', ''), fingerprint, bandwidth))
% python demo.py
Median relay bandwidth measurements are...

  * PompoRelay (319AC0126125DA6A557A416C1BB643E52E272AD0) averaged 187 MB/s
  * oberon (B4486871EA062BAF2354F070FECC344CF3C20880) averaged 196 MB/s
  * ICSIL1xDEDISx1 (7CBBAB7867F939763D94FC6C2819E129971CB0E4) averaged 54 MB/s
  * pollypurebred (EF37D150F3A851B32D9E843605D0EA8A1B938117) averaged 86 MB/s
  * Dementor (46D79CCB83639718177B4BDB8AED6BE9378B1D0F) averaged 158 MB/s
  * wormhole (8B863F5CE8676EFE5BB070CE3CAF54A76AE74D1A) averaged 90 MB/s

Hope everyone had a fun for the holidays! Like many of us December I glugged eggnog with family so this will be short.

  • Stem 1.7.1 hotfix release. Tor adjusted its bootstrap log format in a way that broke Stem, requiring a hotfix release for continued compatibility.
  • Nyx 2.1 incoming! No sexy features, but many long overdue stability fixes. January will be going toward this.
  • Stem failed to read large payloads from Tor’s ORPorts.
  • Writing a new homepage for myself. My current site dates back to my college days and it looks the part. Still in progress but this month I finished drafting its content, passed W3C validation, and made its github repo.
Old Homepage
old homepage
New Homepage
new homepage

Happy holidays everyone! This November Iain and I worked together on Stem’s descriptor capabilities. Though this won’t be of interest to a general audience, for metrics folks like us these are pretty neat.


Descriptor Digesting

Tor relay information is provided by multiple documents. Signed descriptors transitively validate others by inclusion of their digest. For example, our consensus references server descriptor digest, and server descriptors in turn cite extrainfo digests.

To illustrate, here’s a diagram from Iain…

descriptor digests

Stem can now calculate digests from server, extrainfo, microdescriptor, and consensus documents. For instance, to validate an extrainfo descriptor…

import sys

import stem.descriptor.remote
import stem.util.tor_tools


def download_descriptors(fingerprint):
  """
  Downloads the descriptors we need to validate this relay. Downloads are
  parallelized, providing the caller with a tuple of the form...

    (router_status_entry, server_descriptor, extrainfo_descriptor)
  """

  conensus_query = stem.descriptor.remote.get_consensus()
  server_desc_query = stem.descriptor.remote.get_server_descriptors(fingerprint)
  extrainfo_query = stem.descriptor.remote.get_extrainfo_descriptors(fingerprint)

  router_status_entries = filter(lambda desc: desc.fingerprint == fingerprint, conensus_query.run())

  if len(router_status_entries) != 1:
    raise IOError("Unable to find relay '%s' in the consensus" % fingerprint)

  return (
    router_status_entries[0],
    server_desc_query.run()[0],
    extrainfo_query.run()[0],
  )

if __name__ == '__main__':
  fingerprint = raw_input("What relay fingerprint would you like to validate?\n")
  print('')  # blank line

  if not stem.util.tor_tools.is_valid_fingerprint(fingerprint):
    print("'%s' is not a valid relay fingerprint" % fingerprint)
    sys.exit(1)

  try:
    router_status_entry, server_desc, extrainfo_desc = download_descriptors(fingerprint)
  except Exception as exc:
    print(exc)
    sys.exit(1)

  if router_status_entry.digest == server_desc.digest():
    print("Server descriptor digest is correct")
  else:
    print("Server descriptor digest invalid, expected %s but is %s" % (router_status_entry.digest, server_desc.digest()))

  if server_desc.extra_info_digest == extrainfo_desc.digest():
    print("Extrainfo descriptor digest is correct")
  else:
    print("Extrainfo descriptor digest invalid, expected %s but is %s" % (server_desc.extra_info_digest, extrainfo_desc.digest()))
python demo.py 
What relay fingerprint would you like to validate?
3BB34C63072D9D10E836EE42968713F7B9325F66

Server descriptor digest is correct
Extrainfo descriptor digest is correct

Detached Signatures

Somewhat obscure, detached signatures are exchanged among directory authorites to establish the next hour’s consensus. They reference the agreed upon digest the consensus should have, and are only available five minutes of every hour (minutes 55-60). Stem can now read these documents

import stem.descriptor.remote
    
detached_sigs = stem.descriptor.remote.get_detached_signatures().run()[0]
    
for i, sig in enumerate(detached_sigs.signatures):
  print('Signature %i is from %s' % (i + 1, sig.identity))

When available (minutes 55-60 of the hour)

% python demo.py
Signature 1 is from 0232AF901C31A04EE9848595AF9BB7620D4C5B2E
Signature 2 is from 14C131DFC5C6F93646BE72FA1401C02A8DF2E8B4
Signature 3 is from 23D15D965BC35114467363C165C4F724B64B4F66
Signature 4 is from 27102BC123E7AF1D4741AE047E160C91ADC76B21
Signature 5 is from 49015F787433103580E3B66A1707A00E60F2D15B
Signature 6 is from D586D18309DED4CD6D57C18FDB97EFA96D330566
Signature 7 is from E8A9C45EDE6D711294FADF8E7951F4DE6CA56B58
Signature 8 is from ED03BB616EB2F60BEC80151114BB25CEF515B226
Signature 9 is from EFCBE720AB3A82B99F9E953CD5BF50F7EEFC7B97

When unavailable (minutes 0-55 of the hour)

% python demo.py
Traceback (most recent call last):
  File "demo.py", line 3, in 
    detached_sigs = stem.descriptor.remote.get_detached_signatures().run()[0]
  File "/home/atagar/Desktop/stem/stem/descriptor/remote.py", line 476, in run
    return list(self._run(suppress))
  File "/home/atagar/Desktop/stem/stem/descriptor/remote.py", line 487, in _run
    raise self.error
urllib2.HTTPError: HTTP Error 404: Not found

Hello fellow onion lovers! Winter is here, and with it gloomy overcast days and seasonal depression. Yay!

Half kidding aside, September and October have been a great couple of months. Saw a great performance of Jane Eyre with Shari, Cirque du Soleil’s Volta with my family, took care of my grandma a few weeks, fun Amazon team outing that was littered in Henry’s art (the same artist that decorated our tor office)…

flatsticks art

… and of course tasty, tasty code.


Stem 1.7

No surprise the big ticket item of these last couple months has been Stem 1.7. In the works for almost a year, this release included numerous features and fixes.

Check it out!


Membership Audit

Tis that time of year again! Per our membership policy and my duties as our membership secretary I’ve sunk several weeks into reviewing what folks have been up to. Lotta neat things!

Nothing too notworthy to announce at the moment. The process is chugging along reasonably smoothly. I’ve attempted to tailor it the best I can as an opportunity to re-integrate fringe members of our community that are drifting away. In some cases folks indicate that work, life, and other hobbies have made it time to move on. In others we’re brainstorming fun projects they might enjoy.

Curious to see how it goes!


Smaller things these last couple months include…