Stem Release

Closing the decade I’m pleased to announce Stem 1.8, the final release in Stem’s 1.x series and with it Python 2.x support.

What is Stem, you ask? For those who aren’t familiar with it Stem is a Python library for interacting with Tor. With it you can script against your relay, descriptor data, or even write applications like Nyx.

https://stem.torproject.org/

So what’s new in this release?


CollecTor Downloader

Through our descriptor archive, CollecTor, Stem can now read Tor’s network topology at any prior point in time.

For example, listing today’s exits is as simple as…

import datetime
import stem.descriptor.collector

yesterday = datetime.datetime.utcnow() - datetime.timedelta(days = 1)
exits = {}

for desc in stem.descriptor.collector.get_server_descriptors(start = yesterday):
  if desc.exit_policy.is_exiting_allowed():
    exits[desc.fingerprint] = desc

print('%i relays published an exiting policy today...\n' % len(exits))

for fingerprint, desc in exits.items():
  print('  %s (%s)' % (desc.nickname, fingerprint))
% python demo.py 
1229 relays published an exiting policy today...

  MrExit (D628F6BB2330B3F78DBB4BED466B0A586D74782E)
  pangea03 (F21DFB7CCD5EEF3E021086EC96EF7CFCAA72F4F3)
  MacarenaValdes (5E3FD31B9DC279C06AD051D68BE08914F6CD3B46)
  TEMPORA (05EAA0696DCB694D6811042348DACD5059FE64AD)
  Quintex43 (1E5136DDC52FAE1219208F0A6BADB0BA62587EE6)

Bandwidth Metrics

Bandwidth Authorities generate the latency heuristics that govern Tor’s path selection. Guiding circuits to be fast, without overburdening individual relays.

With Stem you can peruse this information…

import stem.descriptor.remote
import stem.util.str_tools

bandwidth_file = stem.descriptor.remote.get_bandwidth_file().run()[0]

print('Bandwidth measurements are...\n')

for fingerprint, measurement in bandwidth_file.measurements.items():
  bandwidth = '%s/s' % stem.util.str_tools.size_label(1024 * int(measurement.get('bw', '0')))
  print('  * %s (%s) averaged %s' % (measurement.get('nick', ''), fingerprint, bandwidth))
Bandwidth measurements are...

  * DigiGesTor1e1 (0111BA9B604669E636FFD5B503F382A4B7AD6E80) averaged 23 MB/s
  * WonderWoman42 (E5AA85FA69CDC31900C86E6427C7E5DE11DE9E2D) averaged 37 MB/s
  * alterspalter (B6F0BC2B93CB3EFFFFF724CB4F5E025FB15EFB70) averaged 2 MB/s
  * blueberry (FE80E192AD48A1BEB02D88EBC7663061176E1A79) averaged 1 KB/s
...

Hidden Service v3 Descriptors

With George’s help Stem now reads, decrypts, and even creates HSv3 descriptors. For example…

from stem.descriptor.hidden_service import (
  HiddenServiceDescriptorV3,
  InnerLayer,
  IntroductionPointV3,
)

print(HiddenServiceDescriptorV3.content(
  inner_layer = InnerLayer.create(
    introduction_points = [
      IntroductionPointV3.create('1.1.1.1', 9001),
      IntroductionPointV3.create('2.2.2.2', 9001),
      IntroductionPointV3.create('3.3.3.3', 9001),
    ],
  ),
))
% python demo.py
hs-descriptor 3
descriptor-lifetime 180
descriptor-signing-key-cert
-----BEGIN ED25519 CERT-----
AQgABqvHAX8wXzJY+FqoJQPXNZ8u+SQGPZ1WN/r3hUna0R2AXQnEAQAgBAAuqibl
ALcKa/4nHtLZn2zKV8L4XIpkRyRm7btWPLpYN5Gseb03H5exL+I3SqfG3uNDw5QK
CmPlCQUy3usouSwhO/qWgdy0//bP5kRDma5GDXXWoi3+xTKM6Jez7TGxPAU=
-----END ED25519 CERT-----
revision-counter 1573695064
superencrypted
-----BEGIN MESSAGE-----
aDJodcMjhCvz1K7JCJEAH1H24hvoZ7gZw53AhPdvpHu+5d1Ogwio4qcIXEK1pEgy
QFF1fE6tnCzsk++eMa2WaKwIJYGLPoCnta78H5Ve6VoMj+Pyb5rE6wPTMTPSVm6M
UjllArr7DS8YcofloDxu3iwC3JZYFt/LB6ahq6lBKeot2BD/11pNggkZrZOCLgNQ
pUVyQau7K8ynagVlNNESnI3FccOBaBB4Xa5mObK2ylyiLQ08MqaImW7X2gxeZltT
/C/xtiJXGm2CzkjPpBpMWm09p7/a97GEWca5e8+fhpmGrN7zjAwjYInTvQHS5AyU
7eUFg8ItrRxAiRq4fbe/zepiq2vgfj1Pt7uxC0KCTcLWpd9O/FIvcFSk27Yrtniw
... etc...
-----END MESSAGE-----
signature VDDXXLvgU6qjRI4zfJR3GbQuVjz98qO0LI5gsI60LtGXK2POZ4E+3YVVWuVaEkvMsZaku5qCutIcu74/WQMxCQ

Peeking out from the engine room, I’m delighted to announce Stem 1.7. A full year’s accumulation of fixes and improvements.

What is Stem, you ask? For those who aren’t familiar with it Stem is a Python library for interacting with Tor. With it you can script against your relay, descriptor data, or even write applications like Nyx.

https://stem.torproject.org/

So what’s new in this release?


ORPort Descriptor Downloads

Stem can now download descriptors through ORPorts just like Tor!

reply = stem.descriptor.remote.get_server_descriptors(
  endpoints = (stem.ORPort('128.31.0.34', 9101),),
)

This is just the tip of the iceberg for the ORPorts capabilities we hope Stem will have. Whats next, python tor clients? Relays? Stay tuned!


stem.directory module

In collaboration with teor, Stem now provides authority and fallback directory information through our new stem.directory module

import stem.directory

COLUMN_FORMAT = '%-17s%-20s%-10s%-10s'

try:
  authorities = stem.directory.Authority.from_remote()
except IOError as exc:
  print('%s\n' % exc)
  authorities = stem.directory.Authority.from_cache()

print(COLUMN_FORMAT % ('Name', 'Address', 'ORPort', 'DirPort'))

for authority in authorities.values():
  print(COLUMN_FORMAT % (authority.nickname, authority.address, authority.or_port, authority.dir_port))
% python demo.py 
Name             Address             ORPort    DirPort   
maatuska         171.25.193.9        80        443       
tor26            86.59.21.38         443       80        
Bifroest         37.218.247.217      443       80        
longclaw         199.58.81.140       443       80        
dizum            194.109.206.212     443       80        
bastet           204.13.164.118      443       80        
gabelmoo         131.188.40.189      443       80        
moria1           128.31.0.39         9101      9131      
dannenberg       193.23.244.244      443       80        
Faravahar        154.35.175.225      443       80        


Descriptor Compression

Stem now supports Tor’s new ZSTD and LZMA compression. ZSTD and LZMA’s higher compression ratio comes at a CPU cost and are less available, so when using these to save bandwidth you should provide a fallback…

reply = stem.descriptor.remote.get_server_descriptors(
  compression = (
    Compression.LZMA,  # higher compression but often unavailable
    Compression.GZIP,  # decent compression and always available
  )
)

Damn this was long overdue. I’m delighted to announce Stem 1.5.2, the accumulation of seventeen months of improvements.

What is Stem, you ask? For those who aren’t familiar with it Stem is a Python library for interacting with Tor. With it you can script against your relay, descriptor data, or even write applications similar to Nyx and Vidalia.

https://stem.torproject.org/

So what’s new in this release? Short answer: a lot.


Improved Python 3.x Performance

Reading from tor’s control port 800x faster, Python 3.x users will find this release a dramatic improvement. By ‘dramatic’ I mean multiple orders of magnitude.

python3 performance difference


Tor Manual Information

Stem’s new stem.manual module provides programmatic access for Tor manual information. For example, say we want a little script that told us what our torrc options do…

from stem.manual import Manual
from stem.util import term

try:
  print("Downloading tor's manual information, please wait...")
  manual = Manual.from_remote()
  print("  done\n")
except IOError as exc:
  print("  unsuccessful (%s), using information provided with stem\n" % exc)
  manual = Manual.from_cache()  # fall back to our bundled manual information

print('Which tor configuration would you like to learn about?  (press ctrl+c to quit)\n')

try:
  while True:
    requested_option = raw_input('> ').strip()

    if requested_option:
      if requested_option in manual.config_options:
        option = manual.config_options[requested_option]
        print(term.format('%s %s' % (option.name, option.usage), term.Color.GREEN, term.Attr.BOLD))
        print(term.format(option.summary, term.Color.GREEN))  # brief description provided by stem

        print(term.format('\nFull Description:\n', term.Color.GREEN, term.Attr.BOLD))
        print(term.format(option.description + '\n', term.Color.GREEN))
      else:
        print(term.format("Sorry, we don't have any information about %s. Are you sure it's an option?" % requested_option, term.Color.RED))
except KeyboardInterrupt:
  pass  # user pressed ctrl+c

manual info demo


Fallback Directory Information

Relieving load fromt the directory authority, Stem can retrieve information about and use tor’s fallback directory mirrors

import time
from stem.descriptor.remote import DescriptorDownloader, FallbackDirectory

downloader = DescriptorDownloader()

for fallback_directory in FallbackDirectory.from_cache().values():
  start = time.time()
  downloader.get_consensus(endpoints = [(fallback_directory.address, fallback_directory.dir_port)]).run()
  print('Downloading the consensus took %0.2f from %s' % (time.time() - start, fallback_directory.fingerprint))

fallback directory fetching demo


As always this is just the tip of the iceberg. For a full rundown on the myriad of improvements and fixes in this release see our changelog.

Greetings wonderful carbon-based residents of the Internet. I’m pleased to announce the 1.4.0 release of Stem!

What is Stem, you ask? For those who aren’t familiar with it Stem is a Python library for interacting with Tor. With it you can script against your relay, descriptor data, or even write applications similar to Nyx and Vidalia.

https://stem.torproject.org/

So what’s new in this release?


Ephemeral Hidden Services and Descriptors

Tor’s 0.2.7.1 release is bringing with it new hidden service capabilities, most notably ADD_ONION and HSFETCH. Ephemeral hidden services let you easily operate a hidden service that never touches disk.

This latest Tor release also brought with it the ability to retrieve a hidden service’s descriptor information. Stem knows how to parse, validate, and decrypt these documents.


Faster Descriptor Parsing

When reading descriptors without validation (which is the new default), documents are now lazily parsed. This provides a very substantial speedup depending on the document’s type…

  • Server descriptors: 27% faster
  • Extrainfo descriptors: 71% faster
  • Microdescriptors: 43% faster
  • Consensus: 37% faster

Prefer to keep validation? No problem! Just include ‘validate = True and
you’ll be good to go.


As always this is just the tip of the iceberg. For a full rundown on the myriad of improvements and fixes in this release see our change log.

Greetings wonderful people of the world! After months down in the engine room I’m delighted to announce the 1.3.0 release of Stem.

For those who aren’t familiar with it, Stem is a Python library for interacting with Tor. With it you can script against your relay, descriptor data, or even write applications similar to arm and Vidalia.

https://stem.torproject.org/

So what’s new in this release?


Better Hidden Service Support

Now it’s easier than ever to spin up hidden services!

Thanks to contributions from Federico Ceratto and Patrick O’Doherty we now have a set of methods specifically for working with hidden services. Check it out in our new tutorial…

Over the River and Through the Wood


Faster Descriptor Parsing

This release dramatically improves the speed at which Stem can parse decriptors. Thanks to optimizations from Nick Mathewson and Ossi Herrala we can now read descriptors 40% faster!


This is just the tip of the iceberg. For a full rundown on the myriad of improvements and fixes in this release see our change log.

Hi all. After months of work I’m please to announce the release of Stem 1.2.0!

For those who aren’t familiar with it, Stem is a Python library for interacting with Tor. With it you can script against your relay, descriptor data, or even write applications similar to arm and Vidalia.

https://stem.torproject.org/

So what’s new in this release?


Interactive Tor Interpreter

The control interpreter is a new method for interacting with Tor’s control interface that combines an interactive python interpreter with raw access similar to telnet. This adds several usability features, such as…

  • Irc-style commands like ‘/help’.
  • Tab completion for Tor’s controller commands.
  • History scrollback by pressing up/down.
  • Transparently handles Tor authentication at startup.
  • Colorized output for improved readability.

For a tutorial to get you started see…

Down the Rabbit Hole


New connect() Function

This release of Stem provides a new, even easier method for establishing controllers. Connecting to Tor can now be as easy as…

import sys

from stem.connection import connect

if __name__ == '__main__':
  controller = connect()

  if not controller:
    sys.exit(1)  # unable to get a connection

  print 'Tor is running version %s' % controller.get_version()
  controller.close()


For a rundown on the myriad of improvements and fixes in this release see our change log.

Hi all. After seven months of work I’m pleased to announce Stem’s 1.1.0 release!

For those who aren’t familiar with it, Stem is a Python library for interacting with Tor. With it you can script against your relay, descriptor data, or even write applications similar to arm and Vidalia.

https://stem.torproject.org/

So what’s new in this release?


Remote Descriptor Fetching

The stem.descriptor.remote module allows you download current Tor descriptor data from directory authorities and mirrors, much like Tor itself does. With this you can easily check the status of the network without piggybacking on a local instance of Tor (or even having it installed).

For example…

from stem.descriptor.remote import DescriptorDownloader

downloader = DescriptorDownloader()

try:
  for desc in downloader.get_consensus().run():
    print "found relay %s (%s)" % (desc.nickname, desc.fingerprint)
except Exception as exc:
  print "Unable to retrieve the consensus: %s" % exc


Connection Resolution

One of arm’s most popular features has been its ability to monitor Tor’s connections, and stem can now do the same! Lookups are performed via seven *nix and FreeBSD resolvers, for more information and an example see our tutorials.


Numerous Features and Fixes

For a rundown of other changes see our change log.

Hi all. After eighteen months of work and a number of delays rivaling that of the Big Dig I’m pleased to announce the initial release of stem!

For those who aren’t familiar with it, stem is a python controller library for tor. With it you can write scripts and applications that interact with your tor client or relay. For some examples of what you can do see the tutorials on…

https://stem.torproject.org/

Stem is compatible with python 2.6 and higher (including the 3.x series), and is a near complete implementation of tor’s control and directory specifications. It has relatively high test coverage (~80% for most modules) and integration tests to check its continued interoperability with new releases of tor.

As always, if you encounter issues or have feature requests then please let me know! Also, if you write something that uses stem then please tell me, both so we can continue to improve our API and expand the tutorial’s list of examples.

Many thanks to everyone that helped make this initial release of stem possible, both with its development and packaging!