Archives

All posts by atagar

Peeking out from the engine room, I’m delighted to announce Stem 1.7. A full year’s accumulation of fixes and improvements.

What is Stem, you ask? For those who aren’t familiar with it Stem is a Python library for interacting with Tor. With it you can script against your relay, descriptor data, or even write applications like Nyx.

https://stem.torproject.org/

So what’s new in this release?


ORPort Descriptor Downloads

Stem can now download descriptors through ORPorts just like Tor!

reply = stem.descriptor.remote.get_server_descriptors(
  endpoints = (stem.ORPort('128.31.0.34', 9101),),
)

This is just the tip of the iceberg for the ORPorts capabilities we hope Stem will have. Whats next, python tor clients? Relays? Stay tuned!


stem.directory module

In collaboration with teor, Stem now provides authority and fallback directory information through our new stem.directory module

import stem.directory

COLUMN_FORMAT = '%-17s%-20s%-10s%-10s'

try:
  authorities = stem.directory.Authority.from_remote()
except IOError as exc:
  print('%s\n' % exc)
  authorities = stem.directory.Authority.from_cache()

print(COLUMN_FORMAT % ('Name', 'Address', 'ORPort', 'DirPort'))

for authority in authorities.values():
  print(COLUMN_FORMAT % (authority.nickname, authority.address, authority.or_port, authority.dir_port))
% python demo.py 
Name             Address             ORPort    DirPort   
maatuska         171.25.193.9        80        443       
tor26            86.59.21.38         443       80        
Bifroest         37.218.247.217      443       80        
longclaw         199.58.81.140       443       80        
dizum            194.109.206.212     443       80        
bastet           204.13.164.118      443       80        
gabelmoo         131.188.40.189      443       80        
moria1           128.31.0.39         9101      9131      
dannenberg       193.23.244.244      443       80        
Faravahar        154.35.175.225      443       80        


Descriptor Compression

Stem now supports Tor’s new ZSTD and LZMA compression. ZSTD and LZMA’s higher compression ratio comes at a CPU cost and are less available, so when using these to save bandwidth you should provide a fallback…

reply = stem.descriptor.remote.get_server_descriptors(
  compression = (
    Compression.LZMA,  # higher compression but often unavailable
    Compression.GZIP,  # decent compression and always available
  )
)

Greetings fellow onion enthusiasts! Like the rest of the summer August has been less tor-ry than usual. This month went toward eviscerating my home in an act of utter apartment seppuku.

apartment mess

Truly it’s amazing how much stuff closets accumulate. Dumped my every worldly possession in a pile and now dealing with the mess. I’ll be at this through September as well but so far so good. Eight full grocery bags of stuff to rid myself of and counting. Fun project!


Stem 1.7 Release Prep

As for the technical front… *sigh*. I told Matt I would release Stem 1.7 when he asked for it but six weeks later and still no release. Dealing with several pre-release issues…


Tor Summer of Privacy

This month Dave concluded his Tor SoP project. Honestly we didn’t get too far. I’ll probably advertise this project again in next year’s GSoC/SoP. That said, Dave sent quite a bit of code at the end so much of August went toward reviewing that.

PS. Already half way through September? Yikes. Between the tardiness of this report and spring cleaning I’m gonna skip my September update. I won’t be in Mexico but hope you all have fun!

Oh oops! Just realized I completely forgot last month’s status report. Probably just as well. Festivals and fun crowd tor out during the summer.

Strawberry Festival, Fremont Fair, Underground Tour, and spring cleaning aside honestly not much to report.

  • Lots of code reviewing. I might not be writing code but Dave is.
  • Membership and website adjustments. Rote work aside, rewrote our ‘welcome’ emails and worked with Tom on adding people page pronouns.
  • Revamped Stem’s equality checks and hashing. Hashes now account for attribute types, are consistent between interpreter runs, and cached by immutable classes.
  • Python 3.5 bundled a buggy lru_cache implementation. Stem now works around this.
  • Python 3.6’s new ‘async’ keyword broke Stem’s testing tools.

Hope everyone else is having a similarly great summer!

strawberries

Driver trained his whole life for this moment.

Decorated space marine marooned with his fellow starving colonists, the microfusion rifle in his hands hummed with unearthly firepower. Enough to vaporize a mountain. But tonight Driver stalked pray of a different kind. His nemesis. His dinner. A sleeping, injured turtle with dementia.

Just two feet from his quarry Driver raised his weapon and… missed. He missed eighty times in a row.

Ya know, I didn’t feel bad that night when Driver starved. Clearly I’m playing too much Rimworld.


stem.directory module

Courtesy of teor Tor authority and fallback information now reside in their own files. This is great, as it allows Stem to provide fresh directory information through our stem.directory module

import stem.directory

COLUMN_FORMAT = '%-17s%-20s%-10s%-10s'

try:
  authorities = stem.directory.Authority.from_remote()
except IOError as exc:
  print('%s\n' % exc)
  authorities = stem.directory.Authority.from_cache()

print(COLUMN_FORMAT % ('Name', 'Address', 'ORPort', 'DirPort'))

for authority in authorities.values():
  print(COLUMN_FORMAT % (authority.nickname, authority.address, authority.or_port, authority.dir_port))
% python demo.py 
Name             Address             ORPort    DirPort   
maatuska         171.25.193.9        80        443       
tor26            86.59.21.38         443       80        
Bifroest         37.218.247.217      443       80        
longclaw         199.58.81.140       443       80        
dizum            194.109.206.212     443       80        
bastet           204.13.164.118      443       80        
gabelmoo         131.188.40.189      443       80        
moria1           128.31.0.39         9101      9131      
dannenberg       193.23.244.244      443       80        
Faravahar        154.35.175.225      443       80        

May’s been quite a juggling act. Smaller things this month include…

"Mission accomplished!" I said. No aircraft carrier, but none the less you’d think I would learn from transnational snafus.

Much of March was still out of commission, but I’ve felt better a few weeks now so maybe this time it’ll hold? We didn’t get to a root cause so likely that whatever nailed me will crop back up someday. But we’ll cross that bridge when we come to it. After all, there’s only so many orifices a doctor can probe… right?   (he says hopefully)

What could be funner than that, you ask? Taxes! And dentists! And paperwork! Kidding aside, last couple months had some great tor stuff too.


ORPort Descriptor Downloads

Stem can now download descriptors through ORPorts just like Tor!

reply = stem.descriptor.remote.get_server_descriptors(
  endpoint = stem.ORPort('12.34.56.78', 90),
)

This came from a desire to health check my relay, but dovetails nicely with the ORPort capabilities that will be the big feature of Stem’s 1.7 release.

Stay tuned!


Descriptor Compression

Recently Tor added ZSTD and LZMA support and Stem now supports them too. ZSTD and LZMA’s higher compression ratio comes at a CPU cost and are less available, so when using these to save bandwidth you should provide a fallback…

reply = stem.descriptor.remote.get_server_descriptors(
  compression = (
    Compression.LZMA,  # higher compression but often unavailable
    Compression.GZIP,  # decent compression and always available
  )
)

Summer of Privacy

A very, very warm welcome to Dave Rolek! Dave will be expanding Stem’s ORPort and hidden service v3 capabilities this summer as part of Tor Summer of Privacy.

Congrats, Dave!


Smaller things these last couple months include…

  • Expanded Nyx’s FAQ to clarify how to get started and explain why apt-get often doesn’t work.
  • Further expunged Nyx’s old name. Thanks to Homebrew for deprecating its old port. One of these days I should reach out to the Wikipedia editors as well…
  • Stem now supplies a user agent when downloading descriptors, so relays can distinguish Tor and Stem clients from others.
  • Formalized the procedure we use when checking membership activity to better avoid surprises and encourage past volunteers to get back involved.
  • Finally got around to swapping this blog to https. Yeah, yeah, I know…

Hi all! Nasty stomach bug left me sleeping and running between doctors most of the month so sadly not much to report. That said, feeling much better now so picking steam back up on tasty, tasty code.

On the fun-things front this month I wrote my second build for Path of Exile, but less than usual for Tor.

poe


Tor Summer of Privacy

This year Google declined our GSoC application to make room for budding open source projects. Personally I think asking us to take a break every so often is applaudable (small projects need the boost!), but this left us asking: what next?

In 2015 when this happened we ran a smaller program of our own called Tor Summer of Privacy. Colin is now organizing a similar program for this year (thanks Colin!).

Stay tuned on Tor’s blog for the official announcement. Tim and I wrote a joint proposal, so if you wanna get involved with Tor this summer then check it out!


Tor Membership

This February the only other tidbit that comes to mind are a couple initiatives as our membership secretary. Kicked off the selection process for our next Community Council and began our bi-annual checkup for membership activity.

Hi all. Changes at work have me stressed so I’ll be skipping Rome this year, but none the less Tor has been a welcome anchor. Work may suck, but Tor? Well…

master plan


ORPort Protocol Support

As discussed yesterday Stem can now communicate over the ORPort protocol. Still lots of follow-up work to do, but thanks to Tim’s wonderful work prototyping how this is done with Endosome Stem can now download descriptors via the ORPort protocol!

import stem.client

with stem.client.Relay.connect('127.0.0.1', 12345, [3]) as relay:
  circ = relay.create_circuit()
  circ.send('RELAY_BEGIN_DIR', stream_id = 1)
  desc = circ.send('RELAY_DATA', 'GET /tor/server/authority
HTTP/1.0\r\n\r\n', stream_id = 1).data
  circ.close()

  print(desc)

Which then provides…

% python demo.py
HTTP/1.0 200 OK
Date: Wed, 07 Feb 2018 18:42:41 GMT
Content-Type: text/plain
Content-Encoding: identity
Expires: Fri, 09 Feb 2018 18:42:41 GMT

router Unnamed 97.113.177.53 12345 0 23456
identity-ed25519
-----BEGIN ED25519 CERT-----
AQQABm/qAazUltT1iUUbIMw8VNNhGb50FDHKJz6S94FLQNxL0LObAQAgBAAapbO9
iLFD0l9SEiEMFQWIT2VnbLyCZKvbrxTs5ULC1l1hQPoui6Y/lEd3yjrQhIs/vl6R
1S6FbwSFDmiXOzq47mFrse4C71ht3TpLOD0F3wiyjWtsqU1k7iPmmpejUgs=
-----END ED25519 CERT-----
master-key-ed25519 GqWzvYixQ9JfUhIhDBUFiE

Fallback Directory v2

Tim and I also worked together on a second iteration for our Fallback Directories. Expanded with additional data and a specification, Stem now supports the new format.

Happy New Years! Hope everyone’s holidays were delightful. Between candy canes and sugarplumbs December had some neat stuff…


Tutorials

Filled gaps within our tutorials, in particular multiprocessing and terminal styling

terminal styling demo

Also worked with toralf on a demo for summarizing relay connections

relay connections


Packaging

To help us keep our various platforms up to date last month I put together a packages wiki. Sadly, the trouble with hand-edited wikis is that they get out of date.

To help with this we now have a daemon that notifies me when platforms have a new package available…

package versions

Hi wonderful world. Post-release followup and recent interview with Ben Collier has kept me pretty busy, but have a couple other fun things to report…


Packaging Community

Commonly major releases are followed by followup packaging work and Nyx’s recent release was no exception. But rather than simply work with our delightful packaging community as I usually do I decided instead to bring order to the chaos.

I’m delighted to say we now have a tor-packagers@ list where Tor developers can reach our packaging community, and packagers can subscribe to be notified of new releases. To go along with this I also made a wiki that gives an overview of our packages

packages wiki


Tor Relay: Caer Sidi

In folklore Caer Sidi was an otherworldly fortress, unsuccessfully assailed by the Prydwin in Arthur’s quest for the holy grail. It’s a name I’ve always wanted to use for a relay.

Flavor text aside, while ago I got permission from Dreamhost to run a non-exit relay on their cloud infrastructure so I’d have a busy relay on which to test Nyx. I haven’t tried pushing the envelope, but I can say it’s been a nice low-cost ($6/month) hosting experience thus far. It’s now listed as a good experience on our ISP wiki.

Ben Collier, PhD student with Edinburgh University recently conducted a survey with a number of Tor community members. I found a number of his questions interesting so with his permission sharing the interview here. Enjoy!


Tor office mural by Henry Ward
  1. How did you first get involved with the Tor Project, and with internet freedom activism more generally?

    I got involved with Tor through Google Summer of Code in 2009. Actually, think the application I wrote back then answers most of the questions for ‘why’.

    This application was not accepted. Tor took on several students, but unfortunately their top two picks (me and Runa) applied for the same task so they couldn’t accept us both. In the end though I’m glad for it since that’s how I first got involved authoring Nyx.

    Brenno Winter conducted an interview with me about this back in 2009.

  2. What are the main things you’ve worked on with the Tor Project? Which of these are you the most proud of? What are you working on at the moment, and what does an average day look like for you?

    Actually, Tommy recently wrote a blog post that answers this.

    In short my main involvement with Tor is Nyx (CLI monitor for tor relays) and Stem (Python library for Tor). I ran Tor’s GSoC program for six years but recently I turned in that hat, and now act as our membership secretary (onboarding new folks, maintaining parts of the site, etc).

    As for my average day, generally I do Tor stuff for a couple hours as I sip my morning coffee before work, and take on larger projects over the weekend. My day job is as a software engineer at Amazon.

  3. What would you say are your main motivations for the work you do?

    Tor is interesting in that there’s a wide range of interests in our community. For some its privacy, others counter-censorship or freedom of speech. For me my foremost interest is in Tor as an open source community.

  4. Do you see yourself as part of a Tor community? Do you think there is a strong community around Tor, and what are the main ways that the community interact with one another?

    Yup to both. Most common forms of communication are email, irc, and developer meetings.

  5. Do you think people in the Tor community are quite similar, or are there a lot of different views and perspectives?

    Good question. I’d say that our community is uniform in its libertarian bend. Tor is a privacy and counter-censorship tool after all, so everyone in our community tends to believe in privacy and free speech. That said, this still leaves quite a bit of room for differences. Everything from university academics to Iranian dissidents. And sometimes this leads to some healthy contentions.

    For instance, as mentioned above my foremost interest in Tor is as an open source community. I push heavily toward openness and doing all our work in public. This is somewhat antithetical though to how security and anonymity communities commonly operate. This tug-of-war is good though, with us meeting somewhere in the middle.

  6. What’s your process for doing development work on the tools you’ve worked on? Is it quite collaborative or do you tend to work on your own then feed back communally?

    Unfortunately the later. Tor has subcommunities (such as the core codebase, Guardian, Tails, etc) where multiple developers collaborate on a single codebase, but outside of that our person-to-project ratio is rather poor. I’m the sole developer on Stem and Nyx, which makes best practices such as code reviewing unfeasible.

  7. How do you organise development work in the Tor Project? Do you have a lot of autonomy to make decisions?

    Nick once called Tor a ‘do-ocracy’, which is to say that the person doing the work makes the decisions. For sections of tor where a single person does all-the-things (such as where I work) the person leading the charge has full autonomy. However, other sections where we have a larger developer population operate on different models.

    As for the internal community as a whole we’ve recently started using more formal voting procedures.

  8. What is the balance between feature development and maintenance? Where do new ideas tend to come from?

    This varies between projects and I can only speak to Stem and Nyx. Both of these projects were in the development two full years before their initial release, receiving a high degree of automated test coverage before seeing the light of day. As such maintenance has not been an especially large concern. This comes at the obvious cost though of a long development cycle.

  9. What are the main factors that you consider when making design decisions for your Tor Project work? Could you give me an example of a particularly important/interesting case where you had to make a decision, and how you made this?

    Hmmm, there’s quite a few different forms of design decisions (architecture, scalability, etc). For Stem the most relevant is API design so I’ll speak to that. To derive Stem’s API I dogfood the library (use it myself for projects), as well as keep an eye on how others are using it.

    I have quite a bit of experience when it comes to API design, but really the best way of discovering where rough edges are is to see a library used in practice, and asking yourself how differences in the library could make their code even better.

  10. When you’re making decisions about the projects you’ve worked on for Tor, at the design stages but also in your day-to-day work, do you find that your own personal values, and the values of the organisation, play a big role in these decisions? Do you think “values” are a useful way of talking about technology, and if so, what values do you think the technologies developed by the Tor Project represent?

    Hmmm. Can’t say it’s a foremost thought for me. Mostly I hack on code because I find it fun. This is a volunteer hobby for me, after all.

    I’ve noticed throughout my software engineering career that there’s a wide range of primary motivations. For some it’s impact – they want to change the world. For others it’s challenge. Personally I don’t lean toward those. My interests is in our community and doing quality work. The magnitude of impact isn’t a prime motivator for me – I don’t care overly much if my work greatly changes the world or not. Rather, I just care that the things I do are done well.

    I suppose that’s why I lean toward support and infrastructure roles.

  11. Do you find your other work (e.g. at Amazon) complements your work at Tor, or are they quite separate?

    The two synergize well in that tricks I pick up with Tor tend to benefit Amazon and vice versa. For instance, I first discovered the mocking framework Stem now uses during my dayjob, whereas an Amazon CLIs I develop benefited from my work on Nyx.

    That said, I do also keep a degree of separation. Tor Cloud was a project to provide Amazon cloud images to simplify relay setup. I made a point of not touching it with a ten foot pole. That said, honestly it’s never really been an issue. Work knows about Tor, Tor knows about work, and neither seems to care particularly much about the other.

  12. What are the challenges of onboarding new staff, especially when the organisation is going through a phase of expansion?

    Just the time to discuss 1:1 about what they’d like on the website, get them set up on irc, etc.

  13. The project is Open Source – why do you think this is particularly important, and what benefits and challenges do you find this poses? How does the Tor Project balance the competing views on this – between openness and more traditional approaches to security development?

    Open source is necessary for Tor as a matter of trust. Users depend on Tor to keep them safe, both in their private lives and even more critically in oppressive regimes. If Tor were an uninspectable black box would you trust it? I wouldn’t.

    Tor’s whole design is architected around distributed trust. No single relay knows your identity, no directory authority can mess with you, and by keeping the code open source we can’t impair your anonymity
    either.

    Generally speaking we error toward openness. Exceptions only arise when there’s a need for secrecy. For instance, tor-security@ where sensitive security issues are reported. Another is malicious relay detection so bad actors don’t learn how they’re being caught. However, even those become public eventually (security reports once a fix is available, and the bad relay blacklist is largely public).

  14. What would you say are the main ways that you’ve seen the Tor Project change as an organisation since you’ve been involved?

    Thanks to Shari (our executive director) Tor organizationally has greatly matured. Far less angst about job security and funding for folks employed the 501c3 side. As for the community side we’ve grown. One growing pain has been decision making as it turns out consensus doesn’t scale. Ever tried getting unanimous agreement from ninety people on a contentious topic? That… doesn’t work. As such we now have a formalized voting procedure for decision making.

  15. Do you see law enforcement as posing barriers to the work of the Tor Project? Do you think they understand Tor and its goals?

    Nope. I don’t see law enforcement as an enemy and hope they don’t see us as one either. Roger and others engage with the law enforcement community and we provide tools like Exonerator to make their lives (and lives of relay operators that don’t want kicked down doors) better.

    Just speaking for myself, I was glad to see the Silk Road takedown demonstrate that traditional policing (money trails, informants, etc) still work when it comes to bad actors on Tor. Criminal enterprises have always had IP level privacy through botnets. Our goal is to counteract bulk surveillance and provide individual privacy which I hopefully many (though understandably not all) in the law enforcement community can get behind too.

  16. Are you worried about the potential of governments cracking down on Tor and encryption technologies?

    Not my top concern. True, the Crypto wars of the nineties demonstrated that governments can take a laughably ill-conceived stance when it comes to encryption, but thankfully the Internet is global. Even if the US takes a backward stance in this regard EU jurisdictions don’t seem to be showing any sign of following suit.

  17. Does working on Tor mean you need to be more careful in your own day-to-day online security practices?

  18. Not in particular. I don’t involve myself with anything highly sensitive so don’t think I’m a particularly juicy target. Just about all my Tor involvement is public anyway.

  19. The media has in the past tended to focus on negative stories about Tor, and Hidden Services in particular. Does this bother you, and do you think how Tor is perceived is a problem? Do you think there are any ways to deal with “misuse” of the network that are appropriate, or possible?

    Good questions. While I can certainly see good uses for Hidden Services (such as dissident blogs) I can’t say it’s a feature I’ve found very compelling. As such my work doesn’t focus on it. Lot of others though see promise in it. You’re right that more than anything else within Tor it draws negative press. Kinda irked when that overshadows the rest of what Tor does but oh well, them’s the breaks.

  20. Tor has many uses – from protecting privacy-conscious citizens, to whistleblowing, to fighting censorship in repressive regimes, to fighting corporate surveillance. Which do you personally think are the most important?

    For me personally: privacy and free speech. Authoritarianism requires censorship to survive. It’s unsettling but unsurprising to see denouncement of our free press now that we’re getting our own little dose of extremism here in the US. Freedom of information both press and Tor support are necessary for an informed democracy.

  21. The relay operators I’ve spoken to had very diverse political views, but all had very positive views of the project and the main team. How do you keep the community engaged and happy? What challenges do you face in doing this?

    Glad to hear it, we love our wonderful relay community too!

    Relay operators are just as much a part of our community as developers, activists, and everyone else. By working in the open hopefully they feel included. Actually, just last month our relay community helped me beta test Nyx. They really did a remarkable job putting it through the ringer. Many thanks to ’em for all their help!

  22. What do you think are the main challenges facing the project in the near future?

    Funding diversity and promoting positive uses of Hidden Services are a couple that come to mind.

  23. What would you say, with regards to Tor, is the thing that you’re most excited about in the near future?

    Pity you didn’t ask me last month. I’d say the release of Nyx. Now brainstorming my next project so we’ll see.

  24. Final question – are you generally optimistic or pessimistic about the future for internet freedom?

    In the short term I’m optimistic. Response to the Snowden revelations showed tremendous public interest in defending digital civil liberties.

    But long term I’m worried. Not about government or three letter agencies, but advertising. Market forces and Moore’s law makes bulk surveillance both easier and more profitable every year. Maybe I underestimate the public’s desire for privacy, but when offered convenience in exchange for it I’m uncomfortable thinking where we might end up.