Archives

All posts by atagar

Status Report for December 2013

Posted by atagar on January 1, 2014
Posted in: Status Report.

Hi all, hope you’ve enjoyed the holidays (and for a few of you CCC)! This December I finally swapped back to focusing on arm in earnest. Good news is that the codebase is rapidly improving, the focus for this month being…

  • Finished overhauling the starter module.
  • Expanded test coverage as more of arm gets rewritten.
  • Integrated pyflakes and pep8 with our tests.
  • Moved user facing strings to the internal configuration. I did this to improve our code, but it could also allow us to add internationalization support to arm in the future if folks want it.

Bad news is unfortunately that none of these improvements are user facing, and even by the rosiest estimates arm’s next release is still many months off.

This December I also finished with my efforts to revise our internal lists. None of the more substantial changes got much support, but this still resulted in some worthwhile improvements…

  • A wiki with our lists, maintainers, and a brief summary of each: link
  • We now have a public, documented policy for adding folks to our internal lists.
  • The tor-assistants list is now a proper subset of tor-internal.
  • Identified some unused lists we can drop. (ticket)

Status Report for November 2013

Posted by atagar on December 8, 2013
Posted in: Status Report.

Hi all. Work and life is conspiring to keep me busier than usual. What time was left this November mostly went toward non-development activities. I miss coding…

Tor Ecosystem Presentation

On 11/18 I gave a presentation at Seattle’s TA3M overviewing projects in the Tor ecosystem, with an eye toward getting more developers involved…

Preparing for this gobbled up a surprising amount of time (I don’t do much public speaking), but I’m really happy with how it turned out. I’ll likely give this presentation again at other local venues, such as LinuxFest Northwest in the spring.

Email List Management

Much of this last month went toward managing our lists and leading a discussion about what we’d like to do with our non-public lists. Work on this is ongoing, but at the very least we’ll finally get a public, documented process for adding folks.

I also sunk several hours into a frustrating GMail issue where it bounces Mailman’s moderator queue as spam. At one point this seems to have been responsible for unsubscribing everyone using GMail from tor-assistants@. I’ve changed Mailman’s configuration so this hopefully won’t happen again, but I’m out of ideas for the GMail bounces.

Other news this month includes…

  • Fixed an important Stem ImportError under Windows. This resulted in a hotfix release of Stem (version 1.1.1). (ticket)
  • Added Stem support for the new event types from proposal 218. (change)
  • Code reviewed Philipp’s ExitMap project.
  • Read over tbbscraper and added to Stem’s examples page.
  • Helped provide stats for the impact of changing tor’s default exit policy. (ticket)

Status Report for October 2013

Posted by atagar on October 27, 2013
Posted in: Status Report.

Hi all. Here’s a quick update of what I was up to this October…

Stem Release 1.1.0

After seven months of work Stem version 1.1.0 is finally out!

This month was primarily focused on pre-release performance and compatibility improvements in preparation for this…

GSoC Mentor Summit

Nick, Moritz and I attended the GSoC Mentor Summit at Google’s main campus. By and large it was very similar to previous years, with many of the same faces and discussion topics. Unlike the prior couple years however I did not focus on keeping notes which, on reflection, was a mistake. Here’s the small bit I remember…

  • Had a couple fun language discussions with developers from SciRuby and Scala. The former mostly concerned the advantages and disadvantages of Ruby with respect to Python (I use the former at work, so I’ve had some time to form an opinion on this). The later concerned Scala support for issuing database queries via list comprehensions.
  • Couple discussions with Wikimedia developers regarding Tor abuse. Unfortunately this is a perennial discussion that never seems to go anywhere despite having plenty of technical solutions.
  • Nick gave me a tour of Chutney’s codebase and tried to persuade me to take on development of it. Nearly worked too. But while Chutney, SoaT, TorBEL, and a dozen other projects would be a lot of fun to work on I only have so much I can squeeze in alongside a full-time job.
  • A hackathon organizer with Random Hacks of Kindness discussed what they’ve found most successfully attracts contributors. The two most memorable tips were momentum and recognition. Regularly organizing events is necessary to maintain a core contributor group, and engaging local media doesn’t hurt as it provides recognition for the contributors.
  • Shadowed Nick during discussions with a Mozilla developer concerning the upcoming Tor IM bundle. They certainly seem excited for OTR support…
  • Visited with Terry and Arc from the PSF quite a bit. Still no ETA on when distributions will switch to Python 3, and Mailman 3 is still a ways off. Momentum on lots of projects though. Evidently they acted as an umbrella org for roughly sixty students. I would weep bitter, bitter tears if I had to be the admin for that…

Other news this month includes…

  • Karsten, Andrew and I are now all moderators for the tor email lists. I’m taking point in this area, making daily sweeps through the moderator queue and tweaking Mailman to reduce friction.
  • Handful of DocTor fixes and improvements. Karsten shut down the Java DocTor early in the month so our shiny, new Python DocTor is now flying solo!
  • Attended events in the Seattle area including ‘Go for Python Developers’ at Google Fremont and the second TA3M meetup.
  • Worked with authority operators to resolve brief outages with dizum and urras.

Stem Release 1.1

Posted by atagar on October 14, 2013
Posted in: Stem Release.

Hi all. After seven months of work I’m pleased to announce Stem’s 1.1.0 release!

For those who aren’t familiar with it, Stem is a Python library for interacting with Tor. With it you can script against your relay, descriptor data, or even write applications similar to arm and Vidalia.

https://stem.torproject.org/

So what’s new in this release?

Remote Descriptor Fetching

The stem.descriptor.remote module allows you download current Tor descriptor data from directory authorities and mirrors, much like Tor itself does. With this you can easily check the status of the network without piggybacking on a local instance of Tor (or even having it installed).

For example…

from stem.descriptor.remote import DescriptorDownloader

downloader = DescriptorDownloader()

try:
  for desc in downloader.get_consensus().run():
    print "found relay %s (%s)" % (desc.nickname, desc.fingerprint)
except Exception as exc:
  print "Unable to retrieve the consensus: %s" % exc

Connection Resolution

One of arm’s most popular features has been its ability to monitor Tor’s connections, and stem can now do the same! Lookups are performed via seven *nix and FreeBSD resolvers, for more information and an example see our tutorials.

Numerous Features and Fixes

For a rundown of other changes see our change log.

Status Report for September 2013

Posted by atagar on September 29, 2013
Posted in: Status Report.

Hi all. Present status: gobbling birthday carrot cake. Before that, though, did a few things worth noting in September…

Stem: Connection Resolution Support

Stem can now provide you with the active connections belonging to a process. This is similar to arm, capable of doing lookups via seven *nix and FreeBSD resolution methods.

Unlike arm, however, this is a far cleaner implementation including both unit and integ tests. Unfortunately this doesn’t yet work with Tor’s DisableDebuggerAttachment feature, but I plan to look into workarounds for that soon.

‘How do I use this’, you ask? It’s easy!

Arm: Rewrote Starter Module

Arm is presently undergoing an overhaul, and this month I focused on its starter module. It’s now shorter, has unit tests, and is just generally damn better (before and after).

Other news this month includes…

  • Google Summer of Code finals were last week. All the remaining students passed, having done great work – fingers crossed they stick around!
  • DocTor’s python replacement is now happily chugging along on yatei and provides #tor-bots notifications.
  • Attended a local TA3M meetup hosted by Lee Fisher. I’ll be presenting a ‘tor ecosystem’ talk at it in November.
  • Notified outdates relays that will soon be dropped from the network if they don’t upgrade. (ticket)
  • Investigated tor regression reported by stem’s jenkins tests. (ticket)
  • Code reviewed Aaron’s TorPS.
  • Fixed a stem caching bug regarding hidden service config options caught by wayzard.(ticket)
  • My site (https://www.atagar.com, which hosts arm) finally has SSL support.

Status Report for August 2013

Posted by atagar on September 7, 2013
Posted in: Status Report.

Hi all, after a month of work I’m pleased to announce that the Stem based
replacement for Doctor is now live!

For those who aren’t familiar with it, Doctor is a service that pulls hourly
consensus information and checks it for a host of issues (directory authority
outages, expiring certificates, etc). In the case of a problem it notifies
tor-consensus-health@, and we in turn give the authority operator a heads up.

The new version of Doctor replaces Karsten’s java based counterpart and is
part of our ongoing scheme to eventually deprecate Metrics-lib in
favor of Stem.

Other news this month includes…

Status Report for July 2013

Posted by atagar on August 3, 2013
Posted in: Status Report.

Hi all. This month was mostly spent on non-tor work including a server migration, bad service outage at work, and a full week of cleaning my apartment. Still, plenty of spiffy news in stem land…

Remote Descriptor Fetching

Major feature for this month was the addition of a module to remotely fetch tor descriptors…

This works much like tor itself does, downloading descriptor content from directory authorities and mirrors. With it we can now easily script against the present state of the tor network without piggybacking on a live tor instance.

Curious what you can use it for? See our present monitors for some ideas.

This also included a little work with Nick on the spec and tor side…

  • Dropped the unimplemented microdescriptor query from the spec. (ticket)
  • Noting the max queryable fingerprints/hashes in spec. (ticket)
  • We’re getting a high failure rate on the downloads we make. A little more investigation is needed on my part to help narrow this down. (ticket)

Other news includes…

  • Revised the appearance of stem’s frontpage. The blue buttons were pretty jarring, so switched to something that matches the rest of the page. (before, after)
  • Added Slackware to our download page. Many thanks to Markus for adding us to SlackBuilds!
  • Worked with Sreenatha to port Tor Weather to stem. Unfortunately Weather does not presently have an active maintainer so I’m not sure how we will proceed on this front. (thread, ticket)
  • The Munich dev meeting has attracted quite a few potential volunteers. After discussing prospective projects with seros I tidied up our volunteer page. Changes included…
  • Our automated Jenkins test runs ran into another regression in tor that caused it to segfault. (ticket)
  • STREAM events mishandled IPv6 addresses. (caught and patched by soult, ticket)
  • Thread with Pierre about TorPylle which turned into a discussion with Nick regarding future language direction for the tor codebase. I’m looking forward to seeing where it goes!
  • We just finished with midterms for Google Summer of Code. Chang unfortunately did not pass, but the other projects are going well.
  • Sorted out travel arrangements for the GSoC mentor summit. Nick and I will be going, and Moritz is presently on the waitlist.
  • Code reviewed ra’s rttprober. He provided some great feedback for which I still owe him a reply.

Status Report for June 2013

Posted by atagar on June 27, 2013
Posted in: Status Report.

Hi all. Without much ado here’s my status report for June…

Stem: Migrated to the Mock module

Our homemade mocking framework has served us well, but over time it taught me one very important lesson: writing a mocking framework is hard. On the surface it seems pretty simple: apply and revert a set of monkey patches. But how do you monkey patch class methods? What about alias imports like the os module? And god forbid you want to mock python’s open() function.

I’m finally taking a lesson from one of my coworkers and using a library for this. Python has several options but the most common is PyPI’s mock module, which became part of the standard library in Python 3.3. (change)

Arm: Pruning the torTools utility

Arm’s torTools module was a wrapper around TorCtl that provided caching, thread safety, and a better API. Now that we’re using stem it’s obsolete, and my aim is to completely eliminate it from the codebase. This is easier said than done however. This month I pruned vast swaths of the module, reducing it from 2768 lines to 1020. (before, after)

Doing this required some expansions on stem’s part. Added functionality includes…

  • get_pid() and get_user() methods in the Controller
  • system functions for getting a process’ start time and FreeBSD jail path
  • the system module’s call() wasn’t respecting exit codes

Stem: Remote descriptor fetching

Throughout the month Karsten and I have been bouncing ideas back and fourth concerning a stem API for remote descriptor fetching. I have some s3krit ideas for improving this that will address both our use cases even better (spoiler: future style instances). We’re coming up on a four day weekend so hopefully I’ll be able to start implementing this soon!

Stem: Expanded FAQ with answers to common Stack Overflow questions

I took a pass over the tor related questions on Stack Overflow, answering fifteen that concerned controller scripting. The vast majority of those unfortunately were some variation of ‘how do I programmatically change my IP?’ which I answered with a Stem FAQ entry.

The only question I thought was especially interesting went along the lines of ‘How do I check the IPs of the exits I’m presently using?’ (link). I answered this with a FAQ entry too.

… and a handful of smaller tasks

  • Investigated the descriptor version provided via tor’s ‘GETINFO ns/*’ options. Contrary to to the spec it turns out these have been v3 all along, and stem now parses them as such. (#7953)
  • Our automated Jenkins test runs caught their first instance of tor regression. This concerned LOADCONF’s behavior after merging a branch for ticket #6752. (#9122)
  • Handful of GSoC tasks (welcome/sorry emails after acceptance was announced, and org admin prep for the program’s start)
  • Added msg_type argument to ControlMessage.from_str() (request by meejah, #8605)
  • Investigated cache consistency issue (thanks to Ravi, #7713)
  • Fixes for ONLINE target (patch by Jeremy, #8692)
  • Minor revisions to how consensus bandwidth-weights are validated (#6872)
  • Addressed an arm issue with certain TERMs (#9064)
  • Answered some questions from Sreenatha that came up while migrating Tor Weather to Stem.

Status Report for May 2013

Posted by atagar on May 26, 2013
Posted in: Status Report.

Hi all. This is a little early but considering the time sink GSoC has been I doubt I’ll get much more done, so here’s my status report for May. The few stem tasks I’ve snuck in included…

  • Added Ubuntu and Fedora to our download page, Fedora package is thanks to Juan Orti
  • Several testing fixes so our Jenkins tests runs now pass
  • The Controller mangled non-unicode descriptor content when using python 3.x (caught thanks to aj00200, #8755)
  • Expanded our client usage tutorial to use SocksiPy and include an example for polling Twitter (thanks to Ashish)
  • Integ tests now assert ownership on the tor process (patch by Jeremy Kushner, #8634)
  • The DescriptorReader mishandled relative paths (patch from Kostas, #8815)
  • Fix so the Controller cache is thread safe (patch by Akshit, #8607)
  • Running our integ tests littered the /tmp directory (patch by Ashish, #8622)
  • Improvements so we use the staticmethod decorator and new style exception catching (patches from Sean, #8824 and #8823)

Status Report for April 2013

Posted by atagar on May 6, 2013
Posted in: Status Report.

Hi all. Though April had a bit of stem work the month was mostly gobbled up by the chaos that is GSoC. I never cease to be amazed at how much time orchestrating it takes, but enough of that. Other tasks from last month included…

Improved Stem’s Site

General Stem Improvements

  • Overhaul for a large part of our testing framework, run_tests.py in particular was in need of a rewrite
  • Fix for broken process renaming (patch from ragwater, ticket)
  • We now have a custom :trac: and :spec: role for Sphinx (patch from ragwater, ticket)
  • ATTACHSTREAM provided an unexpected 555 response (caught thanks to a cypherpunks, ticket)
  • Added support for the ADDRMAP event’s new CACHED attribute
  • Looked into anomalous bridge-ip-transports lines in the consensus, turned out to be from an unmerged tor patch running on George’s relay