Damian's Dev Blog

Hi all, March has been a busy month! My fingers are in quite a few pies but they’re all still baking so this is gonna be a short one…

Google Summer of Code

Sweet, delicious chaos! Forty-five applications rolled in this year which needless to say is keeping us busy. Best of luck to everyone that applied!

Selection announcement is April 22nd.

Nyx Curses Usage

Still a work in progress but I’m abstracting direct Curses usage out of Nyx. Short term this provides better thread safety, and long term could allow direct Windows support through PDCurses. Work on this will continue all through April.

Overhaul aside, brainstormed project ideas with GSoC applicants including…

• Better dialog for event selection
• Hostnames and geoip information for connections

Also cleaned up our ticket queue and made Nyx a bugtracker page.

Few other noteworthy things…

• Stem now provides shorthand aliases for fetching descriptors. This makes scripting as simple as…
  

  import stem.descriptor.remote
  
  print 'Current exits are...\n'
  
  for desc in stem.descriptor.remote.get_consensus():
    if desc.exit_policy.is_exiting_allowed():
      print(" * %s: %s" % (desc.nickname, desc.exit_policy))
  

• Better descriptor validation of non-ascii content.
• Worked with Sebastian, who’s writing a guide for using Stem to better test Tor. He found some interesting testing issues on OSX..

Titan II, V2, SR-71 Blackbird, not to mention the Spruce Goose. If you’re ever in Oregon I’d highly suggest the Evergreen Aerospace Museum. It’s truly impressive. Bonneville Dam was fun too, but I’m kinda a sucker for big feats of engineering.

Fun trip aside February was a productive month!

Google Summer of Code

We’re in!

Org administration is taking a big chunk of my time and will continue to do so through March, but things are coming together nicely. Student applications are due March 25th. Many thanks to Isabela, Roger, and Sebastian for helping with the proposal!

Nyx Torrc Panel

Last panel’s done! In doing so we’re dropping torrc validation which has been the #1 point of confusion for arm users on irc. As always the rewrite came with deletion of what we in the biz call ‘a crap ton of code‘. Still lot more work before we release, but getting closer!

Few other noteworthy things…

• Stem now provides fallback directory information.
• Stem now understands ‘accpet6/reject6’ exit policy entries. This has been a big pain point for some users. Thanks to teor for the help!
• Wrote a DocTor script for David that notifies us when previously
  BadExited relays return to the network.

This was a great start to the year! Highlight was lunch with David White, author of Battle for Wesnoth who also gave me a tour of Valve. But January had lots of neat Tor stuff too…

Nyx Connections Despite DisableDebuggerAttachment

For years Tor’s DisableDebuggerAttachment has been the bane of Nyx. The feature wasn’t intended to effect us, but screws with proc permissions breaking every connection resolver we have.

Resolvers read /proc//fd to get connection inodes, then use that to determine what from /proc/net/tcp belongs to our process. Tor’s DisableDebuggerAttachment breaks that by making /proc//fd only readable by root. However, even without knowing the inodes we can identify Tor related connections by if they go to a relay or our Tor ports. This is exactly what Nyx already does to identify a connection’s type.

TL;DR. Connection resolution now works all the time. Only drawbacks are…

• This will blend connections when running multiple Tor processes.
• Connection resolution can’t work until we have consensus information. This can take a few seconds when starting up.
• We can’t show client or exit connections. Nyx already scrubbed these so no big loss, but means we now can’t even show that they exist.

Small drawbacks to have the connection panel work by default once again. User can still set ‘DisableDebuggerAttachment 0’ in their torrc for more reliable connection resolution.

IPv6 Connection Resolvers

Thanks to toralf Stem can now retrieve IPv6 connection information. More important for users, this means Nyx’s connection panel now works for IPv6 relays!

Few other noteworthy things…

• Working with Karsten and Philipp we now have benchmarks for how our descriptor parsing libraries perform.
• Nyx is faster! There’s still a lot of room to improve but quick pass over low hanging fruit reduced its load by an order of magnitude.

Happy holidays everyone! Between family and tenacious colds December is always a slow month, but I’m pleased to announce another milestone toward Nyx’s next release!

Simplified from a 611 line monstrosity to trivial 346 line panel, Nyx’s configuration editor is the latest part to get some love. Simpler codebase aside, this overhaul greatly improves startup time. In its last release arm required multiple seconds the first time it was run to bootstrap this panel. This is no longer the case.

And really… that’s it. Faster, simpler, and otherwise a drop-in replacement. Next up is the last (and simplest) panel which presents the torrc. Honestly we’re getting kinda close to being ready to release which is exciting. First time arm users have gotten a new toy since 2012!

Hi all! For years Nyx (aka arm) has done a neat trick where we describe what torrc options do and how they’re used. To do this Nyx had its own cobbled together parser for tor’s man page. Clearly a hack, but it worked.

That was all well and good, but we could clearly do better and now we have!

Stem manual information module

Besides filling Nyx’s needs the shiny new stem.manual module provides…

• Tor test coverage. This adds several integration tests to confirm tor can properly build a valid man page.

• Provides all Stem users with three methods for getting tor manual information…

  1. from_cache() – Retrieves information bundled with Stem. This is only as up to date as Stem itself, but the fastest and most reliable method.

  2. from_man() – Parses information from the local system by running ‘man tor’. Still fast, but obviously requires tor’s man page to be present.

  3. from_remote() – Retrieves the latest manual information from tor’s git repository. This is slow and shouldn’t be used without a fallback, but provides the most up-to-date manual information.

• Along with tor’s manual information we provide brief, more user-friendly descriptions of all tor’s configuration options.

• Parser is much improved over Nyx’s. In particular the stem.manual module has vastly improved performance, test coverage, and updated summary information.

Hi all! I’m heading home this next weekend for my favorite holiday (happy pumpkin day!), so sending this a tad early. Personally my favorite thing about dev meetings are the collaborative projects they inspire afterward. This month set aside Nyx in favor of working with the rest of ya.

Metrics Collaboration with Karsten

As part of our devious schemes to benchmark our various descriptor libraries I wrote a comparison of their capabilities.

Been a while since I used Eclipse. Got metrics-lib set up and sent a minor patch to note its dependencies. Next need to script up the benchmark scripts we discussed. Also discussed our CollecTor index and thanks to Karsten it’s now a thing!

List Administration

During our dev meeting we had several discussions around our email lists. This month set up tor-project@ to both…

1. Improve transparency a tad by providing a place for our internal community to discuss things that don’t need to be so secret.
2. Lessen the need to add everyone and their dog to tor-internal@!

Jury’s still out on if this’ll work, but I hope it will. Also followed up on the thread about trimming tor-internal@. The community seems pretty split on this so doubt anything will come of it.

DocTor and Directory Authorities

• DocTor check for network latency to help surface issues like Faravahar’s ongoing sluggishness.
• Took a quick stab at an IPv6 ping for authorities. Presently stumped on IPv6 support on cappadocicum.
• Sending the fingerprint change notifications to bad-relays@ and bumped its threshold to cut down on false alarms.
• Made a DirAuth wiki to help coordinate tickets. Was worth a shot but personally doubt this is gonna go anywhere.

---

Few other noteworthy things were…

• Tor SoP concluded this month with everyone passing! Worked with Cristobal to settle on a license for Erebus and now in the process of moving it to tpo infrastructure.
• Attended TA3M and SeaGL but sadly didn’t stick around late enough to hear Richard Stallman.

Damn this was a productive month! Berlin and a mad scramble at work were highlights. But those aside, I’m delighted to say my three month slog to rewrite Nyx’s connection panel is now done!

In the end I rewrote 2,700 lines of ‘oh god, what is this?’ into a simple 700 line module. As mentioned earlier this was the most sizable task left and, though there’s certainly a lot more work to, it’s all downhill from here.

Few other quick tasks of note…

• Thanks to dev meeting discussions with David, DocTor now checks for rapidly changing fingerprints. These often indicate malicious intent toward hidden services.
• Code reviewed some of Karsten’s metrics-lib work. He and I will soon work together on a comparison of our various parsing libraries.
• Few small Stem fixes including ‘use_mirrors = True’ under python3 and reading files with Windows newlines from Tor’s data directory.

End of the month already? Yikes that was quick! Half the time since my last report I’ve been on a very relaxing (… and very computer-free) vacation with family. Camel riding and rat circuses aside, not much to report on the Tor front. Stem now supports ed25519 descriptor content and just snagged plane tickets for Berlin – looking forward to seeing everyone soon!

Hi all! This month’s status report is gonna be brief. I’ve been head’s down on Nyx’s connection panel: probably the grossest, most tangled part of its interface. Lots of work done, but still lot to go.

I expect this section will take me another month, maybe two (this is just a hobby, after all!). But on the upside when it’s done we’ll finally be nearing sight of a release!

In other news…

• Stem got a couple anonymous improvements for its tests (1, 2) – thanks cypherpunks!
• Code reviewed Cristobal’s relay dashboard project. Looks good!

Oh, and this month ran in Vashon’s 5k. Yay, got in the top 10%. Boo, was beaten by a ten year old!

Guten morgen lovely people of the Internet. This June my grandpa passed away so spent quite a bit of time with family. Tor stuff didn’t get much attention, but I did revise a couple small parts of Nyx.

Nyx Installer and Descriptor Dialog Rewrite

This month I rewrote the installer of Nyx and its descriptor dialog. As usual this was mostly about cleaning and stabilizing the codebase. Only functional changes were…

• Our setup.py now supports –man-page and –sample-path arguments for customizing resource locations.
• Test coverage for installation.
• Installing and our tests are now python3 compatible. Sadly no, this doesn’t mean Nyx runs under python3 just yet but it’s a step in that direction.
• The descriptor dialog now shows microdescriptors (this part of the codebase predated them).