Archives

All posts by atagar

Hi all. My July was mostly spent traveling, both for Defcon and a couple visits with my family on Vashon (for a funeral and Strawberry Festival). August will be much the same. I’ll be gone…

  • 8/8 – 8/12 for Toorcamp
  • 8/17 – 9/3 for a family trip

Besides that and cat wrangling for the GSoC midterms, here’s the things I did this month…

Hi all. When tor developers go on trips we commonly write a trip report afterward. My recent trip to Defcon didn’t contain anything sekrit so sending the notes I took here. If any of these presentations sound interesting to you then videos should be available later on the Defcon site.

Cheers! -Damian

PS. I was new to many of the things being presented so my notes quite likely have some inaccuracies. Don’t quote me on anything – watch it for yourself if interested.

tl;dr Only presentations related to the privacy field were…

  • Can You Track Me Now?

    Panel discussion on mobile GPS tracking with lots of interesting stats.

  • Crypto and the Cops

    EFF presentation on the 5th amendment protections and how it applies to compulsory decryption.

Day 1

Las Vegas is… an unusual city. Like most people I knew I wasn’t in Kansas any more when I saw slot machines in the airport. By day the city’s not much to look at. Mostly just a bunch of dusty buildings baking in the 110 degree desert heat. But by night the whole city illuminates in neon lights making it quite a spectacle.

My first night I explored the strip and several of the larger casinos including Bally’s, Bellagio, Caesar’s Palace, and the Rio where the conference was taking place. They were… enormous. Caesar’s in particular took me almost a half hour simply to walk the length of it. Probably my favorite sight was the Bellagio fountains which had a magnificent water show choreographed to ‘Singing in the Rain’. It was very classy.

Day 2

The second day was the first of the conference and mostly consisted of two things: standing in line a couple hours for registration and introductory talks. The talks were just a single track and included…

  • Wireless Security: Breaking Wireless Encryption Keys

    Handshake process for WPA, WEP, and WEP2 among other things. I was still sorting out the material that I got during registration so I didn’t pay very close attention which was a pity since I could use a refresher.

  • Intro to Forensics: Tools & Tactics

    A very brief introduction to nmap, tcpdump, netcat, ntop, and Metasploit. Nothing new to me here though it was a nice reminder that I know less about them than I’d like. I haven’t played with them since grad school and should try them again when I get home…

  • Cerebral Source Code

    Presentation on social engineering. Sadly missed a large part of this one since I still hadn’t had breakfast. The QA portion that I saw was fun, mostly questions about how he’d extricate himself from various sticky situations. His answers were clever, as well as entertaining. My favorite…

    Q: What would it take to have more trust in this community?
    A: LSD?

  • DC101: Defcon Survival

    This was a panel discussion with general advice for enjoying Defcon. Mostly it boiled down to ‘you get what you put in’ which caught me a little off guard. I came here expecting to watch some talks, see Vegas, and schmooze with other conference attendees. Defcon, however, aims to be a lot more interactive than that and the organizers put a lot of effort into events that weave throughout the conference.

  • HF Skiddies suck, don’t be one. Learn some basic Python.

    This one made me twitch. The speaker both gave a very bad tutorial for getting started with python (who the hell fires up *Eclipse* every time they want to write python?!?) and seemed to be incoherently drunk for most of it. Mostly I spent this time reading the program.

  • Hacking the Hackers: How Firm is Your Foundation

    This was a long checklist of various topics the speaker thought the Defcon audience should know about, largely focused on hardware hacking since that was his specialty. Interestingly ‘TOR’ was the first thing he listed. I always wondered how that spelling persists despite our efforts to stamp it out. Considering that this conference had roughly 20,000 attendees, each of whom will pass it along I guess now I know…

  • Fun with the Lockpicking Village

    A few years back I read the MIT guide to lockpicking and spent a couple days practicing against my door. Since then I’ve fallen out of practice and this was a nice reminder about both the basics and a few things that I didn’t know (how bump keys work, defeating deadlatches, how to use a can of compressed air to defeat magnetic locks that allow egress traffic, etc).

The hallway track wasn’t anything too notable. Met three developers with Metasploit and someone from the security group I attended at WSU. But that was about it.

Day 3

Second day of the conference I ran into someone from Amazon Infosec. A talk I wanted to attend concerning the attack surface of near field communication (NFC) devices was full, but several of the others were interesting…

  • Can You Track Me Now?

    This was my favorite presentation of the whole conference. A panel of four presenters, including Chris Soghoian, discussed geolocational privacy of mobile devices including local threats, historical information, and requests for real time tracking.

    Why does this matter, you ask? Because when it takes a team of agents to track your every step surveillance is costly. But when a single agent can track you and your every acquaintance from the comfort of his desk we get a whole lot closer to a 1984 style surveillance state real fast.

    Sprint now has a department of 200 people to service wiretapping requests. They also provide a self-service portal to law enforcement that has been used six to seven hundred thousand times a year, for 8 million GPS coordinates. This includes your every move going back 6 months to 2 years depending on your provider. 7 years in the case of AT&T.

    And no, none of this requires probable cause or a warrant. In other words, beware if your ex is a cop.

    Another eye opening bit of the talk was phone encryption. Apple has a master skeleton key for all iPhone devices so, when requested, they can send law enforcement a DVD with all the contents of your phone. Android is a little different in that they remotely reset your password to something that they can give the law enforcement agent.

    Beside three letter agencies, ad networks and apps are also a concern. 47 of the top 100 mobile applications send locational information to 3rd parties. For instance, ‘Best Alarm Clock’ sends locational information in the clear to 3rd parties whenever the app is used (somehow I doubt they need to do that for the alarm clock to function…).

    On a policy front one slight glimmer of hope comes from a recent case which ruled that placing a GPS tracking device on your car *does* constitute a search and hence requires a warrant (which means they kinda sorta need evidence of wrongdoing, *gasp*!). Don’t read too much into this, though. The majority opinion ruled this way because it technically constituted trespassing on your property.

    Just because you’re paranoid doesn’t mean they’re *not* out to get you…

  • Crypto and the Cops

    Marcia Hofmann from the EFF discussed 5th amendment protections and how they apply to being compelled to reveal your passphrase for an encrypted device. There have been five cases in this area so far, some that ruled each way in terms of compulsory decryption. Though not intuitive, the main determining factor in those cases constituted a gap in knowledge. If law enforcement knows exactly what they’re looking for and where it is then there’s less chance of the amendment applying than if the search is a fishing expedition. Moral of the presentation: never volunteer information, even if you’re innocent. Ask for a lawyer first.

  • Making Sense of Static

    Walkthrough of GPS signal acquisition, mostly focusing on how receivers account for Doppler shift and determine the code phase. The speakers also presented libswiftnav, an open source implementation of a GPS receiver system.

  • Life Inside the Skinner Box

    Cautionary presentation about the automation of law enforcement and where it might someday lead. This was pretty high level, discussing the societal implications.

  • Anti-Forensics and Anti-Anti-Forensics

    Discussion by a forensic examiner of several tricks that can draw out the length of an investigation (and by extension cost, increasing the chance of settlement). He also covered how they can be mitigated. Some of them included…

    • using a non-standard RAID configuration
    • randomizing file crated/modified timestamps
    • altering file header types
    • using restricted windows filenames (CON, PRN, AUX, etc)
    • owning lots ‘o media
    • modifying files provided with your system so the checksums no longer match what a default install would include (and hence adds to the haystack of things to sift through)

Before heading back for dinner I also caught part of a Skytalk. I didn’t find the presentation to be too interesting (brainstorming about mass data aggregation), but I got a chuckle from the sign on the door: “Absolutely no cameras allowed. Violaters will be violated violently.”

Day 4

I had less luck on the third day of the conference, hopping between several talks that weren’t particularly interesting. The best I found were…

  • World War 3.0 – Chaos, Control, the Battle of the Net

    Policy talk about Dubai’s upcoming WCIT negotiations (aka, the thing that’s gonna politicize the Internet). Most of the presentation was surprisingly fluffy, the only interesting bit coming from Dan Kaminsky who made the point that there’s generally four interests involved…

    • privacy
    • security
    • anti-piracy
    • sovereignty

    … with ‘reliability’ being the golden goose that no one wants to cook. His point was simply that these four camps war and forge alliances, privacy and security for instance sometimes at odds over anonymity but thick as thieves on other obvious fronts. He also said something like “We have made a system optimized for moving pictures of cats. And it’s *really* good at it. Though we’d like it to do more.” (likely horribly misquoted)

  • Exploit Archeology: Raiders of the Lost Payphone

    Step by step process of reverse engineering and reprogramming an Exetel payphone. Twenty year old software from a company that imploded without a trace long, long ago. Ye gods each step turned out to be a trek to Mt. Doom and back…

  • Off Grid Communication with Android

    Mesh network by introducing a transparent proxy after going through the Java networking interface. He wrote something based on the ‘Wi-Fi Tether for Root Users’ app to put his phones into ad-hoc mode then tried both the OLSR and BATMAN proactive routing protocols. He also tried a reactive protocol that broadcasts messages, then left routing decisions to the receiver which can trivially pick the shortest path though this had issues scaling above 200 devices or so.

  • Back Ops

    My favorite talk of the day, given by Dan Kaminsky. It was originally slated for two hours, but got crunched at the last minute into a one hour slot so he went pretty fast over five different topics including…

    • Timing Attacks

      Attacks can distinguish 15-100 ms deltas over the Internet and 100 nanoseconds over a LAN. Making everything take an identical amount of time is generally unrealistic since it forces everything to worst case performance. His view was not to let the perfect be the enemy of the good, and simply introduce random delay into requests.

    • Bad Random Number Generation

      First a rant that for twenty years security engineers have worried about entropy sources, yet we still haven’t gotten it right. The common sources of entropy are: hardware RNG, keyboards, mouse, and disk rotation. However… most servers don’t have any of these, especially with the move to virtual storage devices. Roughly 1/200 keys on the Internet are poorly generated, and he proposed using idiosyncrasies in clock timing as an alternative source of randomness.

    • Security Aspects of Languages

      Starting with the question ‘why is PHP so damn popular?’, he discussed why secure coding practices like prepared statements often aren’t followed. His conclusion: SDEs are in charge and they just want their service to work. The more of a pita security makes itself the more it’ll be viewed as an obstruction and hence circumvented. He then proposed more usable security patterns, for instance making the languages smart enough to translate queries like “SELECT name FROM users WHERE username=$id” into the proper prepared statement counterpart.

    • Censorship Detection

      After a brief mention of OONI-Probe he discussed a censorship detection method of his via the use of favicons. I saw this in an earlier presentation of his and he mostly skimmed through it.

    • Vulnerability Scanning

      Trying to answer the question of ‘how much of the Internet is vulnerable to X’, he used *stateless* TCP connections to scan hosts far quicker than he usually could. The trick was to skip retaining any information about outbound connections, making fire-and-forget TCP handshakes and leaving it to the other end to remember the state of the connection.

Day 5

I didn’t expect much from the lineup on the last day, but some of them were surprisingly good. I also ran into a couple more people from Amazon and one from Tor.

  • SIGINT and Traffic Analysis for the Rest of Us

    Very good talk on APCO Project 25 (P25), two way radio communication used by police, fire departments, federal enforcement, the secret service, and the DoD. A drop-in replacement for analog FM, this was first introduced in the early 1990s. It can optionally be encrypted, with federal/DoD users commonly enabling it and police/fire leaving it off. If you see an earpiece or bulky walki-talky then this is what it’s probably using.

    The speaker discussed several vulnerabilities in P25. Firstly, it uses a forward correcting protocol that is easy to jam. Usually a jammer needs to be stronger than the signal it’s jamming, but with P25 it’s sufficient to selectively block a message’s header to cause the rest of the message to be ignored. How hard is this, you ask? To cheaply jam a P25 signal you just need a $15 GirlTech IMME toy (it cost less to buy the toy than the transmitter that comes in it). As an added bonus it comes in pink or purple and comes with ponies on the box!

    Second, P25 has no notion of authentication. Anyone can transmit and user ids can be trivially spoofed. The user id of transmitters are sent in the clear, even when encryption is enabled.

    Third, any P25 device that receives a malformed signal will reply with the user id (again, in cleartext). This is invisible to the user and means that you can ping all P25 devices and triangulate their positions. Think of it as the “Marauder’s Map” for the secret service.

    Fourth, P25 is a narrow band radio broadcast (12.5 Khz) which is fairly easy to intercept via a scanner (such as an Icom R-2500). This makes encryption a must for private communications. However, the encryption of P25 devices has several usability issues…

    • Encryption is enabled or disabled via a switch on the handset. The only indication of if encryption is enabled or not is a “1” or “(/)” symbol on the handset. Which means ‘encrypted’ and which means ‘cleartext’ is evidently a common point of confusion.
    • The beep that the device makes to indicate that it’s in encrypted mode is the same that’s used to indicate a low battery among other things.
    • Encryption is transparent to the receiver, so if an individual in a group is broadcasting in cleartext then it’s impossible for anyone else in the group to know and correct them.

    • Rekeyed P25 devices become incompatible while being rekeyed, forcing users to drop to cleartext if any of the devices haven’t yet been updated.

    As a result, about 30 minutes per day of federal communications are sent in the clear. In a multi-city study on what this included researchers were able to gather the names of informants, license places of vehicles being tailed, etc. These snippets were also enough to reasonably figure out what investigations were going on at a given time.

    When brought up with the federal field offices encryption rates improved for a week or so, then dropped back to their prior ratio of cleartext. The researchers also discussed these with the manufacturers, who insisted that it wasn’t their fault if their users couldn’t properly use their devices.

    Btw, the only federal service to have a spotless record in terms of encrypting their communications? The US Postal Service.

  • SCADA Strangelove or: How I Learned to Start Worrying and Love the Nuclear Plants

    This was a talk on the security of SCADA HMI software (or lack thereof), which is used throughout several minor things like nuclear power plants. Originally to prevent plant operators from installing pong on the kiosks, HMI applications hasn’t evolved much since those humble origins. This talk was funny as hell, going over issues with several widely deployed HMI systems…

    • Microsoft Bob: Early attempt at a friendly, ‘helpful’ interface. If you made three incorrect password attempts you got a dialog saying “I see you forgot your password. Please enter a new one here…”. It also stores passwords in plaintext.
    • InvisiLink: Stored passwords are an xor with a static key. What is that key, you ask? “0123456789”
    • KingView: Passwords simply have their last byte subtracted from 0xff.
    • Iconies Genesis32 9.2.2: The login dialog includes a ‘challenge code’ that can be used as an alternative method of getting local admin access. To use this code the user is supposed to call customer service, who then provides the password corresponding with the code. Alternatively, you can take the first eight characters of the challenge code’s md4 hash to do the same.

There were also a couple talks on ‘firesheep inspired’ frontends for various exploits. The first did NTLM relaying against Windows corporate networks to allow for a variety of not-so-nice things like reading through another person’s Exchange inbox.

The second was a UI to automatically ARP poison a network then use sslstrip to snag credentials. I felt a little embarrassed for the speakers in this talk since they first tried to sensationalize what sslstrip could do. After that they claimed to have a stealthy MITM, then described the noisiest attack I can think of. Not only did they ARP poison the network and do SSL downgrading, but they did an nmap scan of every host on the network and, just for added stealthyness, blocked all tunneling protocols (ssh/vpn). The later was on the theory that “they’ll just use the local network instead”. Uggg.

So more annoying push-button solutions for script kiddies. Yay.

Ending Conclusions

Fun and interesting conference, but *damn* vegas is pricey. The trip was 1.5k and only $200 of that was the conference registration. I enjoyed it and it was a good thing to do at least once, but way too costly to do again.

Hi all. This June I exchanged my developer hat to be a mentor instead, spending more time reviewing code than writing it myself. Fingers crossed that at least one or two of them stick around after the summer ends!

As such, this status report is more about other people than me. Apologies if I miss anything.

  • Ravi (GSoC Student)

    This month Ravi discovered a bug with Tor’s GETCONF method and wrote numerous features including SAFECOOKIE support and a GETCONF method for the controller. The GETCONF handling is complicated by the accursed HiddenServiceOptions so it has needed several iterations, but I plan to merge it this week. After that Ravi already has two more feature branches lined up for me to review (*sob*)…

  • Beck (Volunteer)

    Somehow I’ve never been able to bring myself to do development on Windows (if you haven’t seen Neal Stephenson’s Hole Hawg article then I recommend it). Fortunately Beck does, and has done a fantastic job of fixing stem and its tests to work there (1, 2). He also added get_version(), authenticate(), and protocolinfo() methods to the controller.

  • Erik and Megan (Wesleyan Students)

    Erik and Megan have been focusing on stem’s tests, first submitting a couple fixes for the mocking module (1, 2) then writing unit and integration tests for the proc utilities. Next they plan to implement the CSV export functionality suggested by naif.

  • Sathyanarayanan (Volunteer)

    Though work has kept him pretty occupied, Sathyanarayanan has been working on an ExitPolicy class and is presently at the dev meeting making plans to implement a python based Onionoo.

  • Karsten

    Though he isn’t hacking on stem itself, Karsten has been helping by reviewing its descriptor handling and suggesting improvements (1, 2).

Besides those, I implemented a few stem improvements too this month…

  • Sphinx Documentation

    At the start of June I rewrote stem’s documentation into reStructuredText so it could be compiled by Sphinx. The results are very pretty

  • Python 2.5 Compatibility

    Stem aims to support python versions 2.5 and above (in the 2.x series). However, most of our development has been on 2.7, letting backward incompatible changes slip in inadvertently. This ended up being a two week bug fixing odyssey, but now that it’s done stem and its tests should now work if users want an interpretor that harks back to 2006…

  • Test Freezing Issue on Mac OSX

    Both Sathyanarayanan and Karsten reported that stem’s integ tests freeze on Mac OSX. After a dozen hours of hair pulling I’ve narrowed it down to an issue where control sockets are left in a CLOSE_WAIT state when closed, and eventually we lose the ability to make new control sockets…
    https://trac.torproject.org/6235

    From what I can tell this is either an issue with Tor, Python, or Mac OSX (my money’s on the last). Help welcome if anyone has ideas.

Other random things from this month include going to the Fremont Fair, attending a SEAPIG meeting (local python developer group) and making travel arrangements to attend Defcon later in July.

Hi all. Spring is in the air, and with it many lovely things including the UW Street Fair, Folklife, and an iSec Open Forum. It also included a week of oncall but that doesn’t count in the ‘lovely things’ column. On the upside though, this time it was pleasantly light (first time I’ve gone a whole week without being woken up by a pager at 3am!).

As a GSoC admin I don’t have much to report besides a blog posting at the start of the month. However, as a mentor some neat things are in the works…

  • Ravi is adding SafeCookie support in stem. I’ve code reviewed the first couple iterations and it’s looking great. Given some tests and revising to fit with recent refactoring it should soon be ready to merge.

    After that he’ll start working on the general controller. This last weekend I refactored how response classes are organized and implemented GETINFO as an example, so this will hopefully be an easy project for him to start hacking on.

  • Beck added descriptor validation to check that the signing key’s hash matches its fingerprint. This is the first part of descriptor integrity validation that Karsten suggested, however this project has gotten stuck so he’s moving on to other stem tasks.
  • Investigated a couple issues brought up by Sathyanarayanan. (1, 2)
  • Two Wesleyan students will start working on stem soon!

Needless to say helping these projects has occupies much of my time, and will take even more once the Wesleyan students get started. But after years of trying in vain to attract developers to my projects I wouldn’t have it any other way.

Other stem tasks I finished this month includes…

  • ExtraInfo descriptor parsing. This took me a couple weeks since it contains so many attributes, but I’m glad it’s finished. Now all that remains before we can port Onionoo are the consensus’ network status entries. That’s now at the top of my todo list for descriptor work, but I’m setting it aside for now in favor of Sphinx and helping Ravi and Beck with controller work.
  • Improved the launch_tor() function, making the test instance easily configurable via a temporary torrc (similar to what Vidalia does) and adding integ tests. I also added a workaround so it’ll work on Windows. (1, 2)
  • Updated the stem development wiki so it’ll be easier for new volunteers to find tasks that interest them.
  • Discussed descriptor type annotations with Karsten and implemented stem’s side of it. We also discussed some changes to bridge descriptor sanatization which led to some minor stem changes too.

Ducks are awesome. Especially cute Indian Runners who waddle about upright like penguins. Besides this startling discovery, in April I wrote some code, released some code, and did a bunch of GSoC mentor/administration work.

Stem had a reasonably good month. Work included…

  • Finished and merged all of the outstanding todo items from my discussions with Karsten (merge diff). This included additional testing, support for bridge descriptors, re-discovering a tor bug that caused negative uptimes, and a variety of other things. I’m keeping an eye on metrics-lib tickets as they roll in so stem can improve from the issues that Karsten discovers (example), and discussion is ongoing about the addition of a descriptor header field.
  • Discussed stem’s copyright with Wendy and others. We now have a plan for contributors that’ll allow us to reuse stem code under other licenses when needed.
  • Ongoing discussions with Beck about potential stem projects. He sounds eager to work with Ravi and me on the general controller.
  • The whitespace conventions of my projects drive you guys nuts just as badly as yours annoy me. This was gonna be a continued pain point for accepting contributions from others so I’ve added a whitespace checker to stem’s tests that’ll yell at people if they start doing something funky.
  • Stole a trick from git and dropped the ‘–no-color’ argument from the test runner in favor of autodetecting if stdout is a tty terminal or not. This means that test output to your console will have pretty colors, but the ANSI escape sequences will be omitted if you’re piping the output to something else (less, a file, etc).
  • Merged a chroot testing target so we can ensure that stem plays nicely with those environments. This is a use case that traditionally causes problems for our controllers since they don’t account for a path prefix in cookie authentication, determining the data directory, etc.
  • More intermittent concurrency woes. Hopefully I fixed it for realz this time!

Arm also got some love, including a couple important fixes which were released in version 1.4.5

  • Fix for unrecognized authentication methods. I also filed a ticket with the fix for TorCtl which got a less-than-heartwarming thanks.
  • Added a notice when ptrace is disabled, which by extension causes some proc contents to only be readable by root (breaking arm’s connection panel). Only Jake has expressed an opinion that this is a good feature to have, but others don’t seem interested in discussing it so guess it’s something that I’ll just need to work around. The message tells users how to disable the feature and cites the ticket if they want to know more.
  • Helped arm users including Eric, LoneWolf, and MoPac.

Finally, I spent a good chunk of this month cat herding for GSoC. We survived the student selection process (yay!) and for the moment at least things are proceeding smoothly. Thanks to Sebastian for leading the student selection meeting and covering for the #gsoc deduplication discussions.

Cheers! -Damian

PS. The aforementioned ducks are in reference to an email thread dispensing free flightless avian waterfowl to the masses. Alas though, I couldn’t get any since my small apartment isn’t especially duck-friendly. *sob*

I never cease to be amazed at how fast a month sweeps by. This March I fell in love, fell out of love, got ill with a horrible stomach bug, and wrote a bunch of python code. My favorite was the last one and this is just about that.

My time developing stem this month was almost entirely dedicated to writing a python counterpart for metrics-lib. Most of the effort here went into reader concurrency, server descriptor validation, and lots of testing. For my part this project has the following goals…

  • provide the server descriptor, network status, and microdescriptor parsing needed by the controller
  • validate that new tor versions comply with the spec and don’t break our parsing
  • replace the java metrics-lib so we have a single codebase with multiple maintainers (in other words, persuade Karsten to hack on stem)
  • allow applications that just need descriptor data (such as the consensus tracker script) to use cached descriptor data so they don’t require an open control port

At present stem’s implementation just handles server descriptors. A lot more work will be needed to cover the rest of what metrics-lib does.

In other stem news Ravi Padmala, a contributor to several of our projects and a GSoC applicant, made multiple fixes to stem’s version parsing. I never cease to be amazed at how error prone something that sounds as simple as ‘parse the tor version’ can be. Guess that’s why we’re writing a library…

Sathyanarayanan also took the first stab at porting arm’s ExitPolicy class to stem, though more work is still needed there.

Besides stem, roughly an equal amount of my time has gone into this year’s GSoC (for anyone living under a rock we were accepted, yay!). With my org admin hat on I revised our application, made lots ‘o revisions to our volunteer page, and helped to respond to general GSoC inquiries.

With my mentor hat on I reviewed Ravi’s proposal and decided afterward that I dislike the fire-and-forget approach that we usually take with GSoC projects. We give students isolated projects where they can work independently because it is less work for us. On occasion we end the summer with a new core contributor or something we can use, but in general it fails on both counts. I want to try something a little different this year and actually work on the tasks with my applicant. Maybe it’ll work, maybe it’ll drive them mad. We’ll see…

Other random things that I did this month included…

  • Attending local presentations by Bruce Schneier and Dan Kaminsky.

  • Looking over meejah’s txtorcon, a python controller lib using twisted. It’s impressive that he got this up so quickly and it’s neat to see what a twisted implementation looks like. However, it is missing large and very basic controller functionality (such as parsing controller replies), and what parsing it does do is hacky (if "COOKIE" in protocolinfo_reply: do cookie auth which will obviously fail with replies like ‘SAFECOOKIE’). With work though this could be a nice alternative implementation. Meejah is obviously very capable and it’ll be interesting to see where he goes with it. (Correction: mistake on my part, txtorcon actually does have parsing for GETINFO and GETCONF responses)

  • Discussed with Norman and Karsten the possibility of Weslayan students working with us again this year on either Stem or Onionoo. It will be a smaller scope than last year’s project if it materializes (just a couple students) which in my opinion is a good thing. Large groups are hard to manage.

Oh, how todo lists never seem to get any shorter.

My first half of February was focused on stem development, most importantly the implementation and testing of the BaseController class. This is the foundation on which useful controller activity can be based, providing a parallel to TorCtl’s asynchronous controller communication (event handling) and sendAndRecv function. Good news is that the BaseController is also designed to be thread safe. Bad news is that getting the deadlocks worked out was a pain in the ass and consumed well over a week. Sometimes concurrency is hard. >:(

Other stem development included…

  • Simulated chroot setups for integration testing (ticket). This hasn’t yet been merged because I haven’t added a method for users to provide their chroot prefixes (and hence these integ tests for things like cookie authentication rightfully fail). Not hard, just haven’t gotten to it yet.
  • Gave some input on Robert’s Safe Cookie proposal and filed a ticket for supporting it in stem. Sathyanarayanan has offered to take the first pass at implementing it.
  • Discussions with people helping to make stem better. Sathyanarayanan put the finishing touches on configuration saving and Neena fixed an integration testing bug. Many thanks!

Later in the month I began making the arduous trek (half hour walk) to the Tor developer meeting. For everyone who could attend it was great to see you again! Highlights for me included…

  • Demoed stem to several people and schemed about its future plans.
  • Discussed a python metrics-lib with Karsten. Making a skeleton for that now holds the top slot on my dance card when I finally get some time to do development again.
  • Talked with potential mentors about ideas for GSoC. There was quite a bit of interest but not any concrete plans at the time.
  • Discussed the burden of proof needed for badexiting and resolved a ticket we had for an automated exit setup we’ve been seeing.
  • Brainstormed alternative names for the third incarnation of TorStatus with Karsten and Arturo. In the end we went with “Atlas”. I later filed tickets (1, 2, 3) to move it and Onionoo to tor’s infrastructure (tpo vm, git repos, trac, etc).
  • Talked with Runa and Karsten about the monitoring infrastructure project. It won’t be a GSoC project, but rather something that Runa plans to hack on later.
  • Organized for us to go on the Underground Tour. Note to future self: leaving with twice as much transit time as you need doesn’t work. Quadruple it.

Sadly as the month went on I’ve shifted more and more from development to helping others. Of late the little time I have has gone toward GSoC preparation…

  • Revised our GSoC landing page, rewriting a few of the sections.
  • Nagged lots of people for project ideas and added them to the ideas page.
  • Added a project idea for stem.
  • Dug up our application from last year. With only a few minor tweaks it should still be fine.
  • Discussions about if we’ll be filing a joint application with the EFF again or not. Conclusion was that it probably isn’t as vital to our acceptance as we once believed, but we’re still gonna do it because we like the EFF and they’ve pinky promised to communicate better this year.

… and then of course there were other things…

  • Code reviewed Karsten’s script for gathering obfsproxy statistics (ticket).
  • Several volunteer page changes, like adding Obfsproxy, Ooniprobe, and Shadow.
  • Discussions about trac with proper. On one hand I’m glad that he’s trying to help, but on the other I feel like there’s a growing need for us to include a banner on our wiki warning that it’s community maintained. With our logo and the official domain there’s a sizable risk that visitors won’t realize that we don’t review several of the pages at all.
  • Thanks to Sebastian for fixing an arm bug where reading tor logs from February 29th on leap years would crash arm. This problem was reported by dozens of people, which is actually really heart warming.

While I’d like to get back to my own coding, I doubt that these distractions will subside much any time soon. C’est la vie, I shouldn’t complain – it’s all good stuff.

Hi all. Performance reviews and oncall kept me occupied for much of January, and Megan had dibs on most of what remained. I’m still hacking on stem but progress isn’t as fast as I’d like. C’est la vie.

Stem’s development in January mostly focused on…

  • Writing a proper mocking module and refactoring the tests to use it. This will greatly improve the maintainability and ease of writing new tests going forward. Originally this began with the humble goal of ‘remove a built-in mocking hack from the system module’, then went down the rabbit hole of larger scale testing improvements. Still, I’m happy with the results.
  • Sathyanarayanan took on development tasks including integration tests for chroot setups, saving configurations, and troubleshooting test failures on OSX. Design discussions and code reviews take a fair bit of time but I’m thrilled to finally have someone to hack on the codebase with me. A couple other potential volunteers (piffey and blackpaw) showed interest but have since disappeared.
  • A large part of my discussions with Sathyanarayanan centered around making stem more developer friendly, both in terms of its utility APIs and easier collaboration. As it turns out keeping stem’s todo list in a text file on my netbook is not the most optimal location for other people. I’ve since moved it to a development wiki.
  • Expansion of the configuration utility. The most notable changes include multi-line configuration options and moving to a listener architecture. The former lets us move user facing strings out of the source (good if we ever translate) and the later greatly simplifies usage of this utility. It’ll also allow for runtime configuration editability later.
  • Additional options for running stem’s tests…
    • ‘–tor’ – Runs integration tests against a given tor binary (obviously needed to test during tor development).
    • ‘–no-color’ – Removes ANSI escape sequence formatting which is preferable when piping test output.
    • ‘–log’ – Makes stem provide its logging output with the test results. Hopefully by making log messages more visible during development we’ll get better, more user friendly logging for stem’s users. Actually, I’ve already rewritten most of stem’s log messages because of this option…

Non-development things I did include…

  • Sent tor posters to international people. The pile of customs slips was pesky, but worse was twiddling my fingers at the post office as they typed each form in one by one. Hunt and peck is not the fastest method for data entry…
  • The consensus tracker script had a couple interesting finds this month. The first was an oddly configured exit from the University of Waterloo and the second was 41 exits with what looks to be an auto-generated configuration.
  • Realized that my git-fu wasn’t up to par for some of the things we’re doing at work, so I read ‘Git from the Bottom Up‘. If you’ve ever been curious about git’s internal data model then this is the article for you. It’s short and gives a very well written overview starting with git’s most basic components (blobs) and building up from that. I’ve heard that Pro Git is also good so I might skim some of that next.

Looking forward to seeing most of you at the development meeting!

Hi all. Between being ill, oncall for work, visiting with family over the holidays, and finally meeting a brilliant, wonderful girl named Megan I didn’t accomplish too much in December. This isn’t likely to change any time soon so the projects I maintain may get a little less attention – such a pity.

As normal I’ve mostly been split between developing stem and maintaining arm. Ideally I’d like to sink all my time into the former but arm had several issues that demanded attention this month…

  • The ptrace change from ticket 3313 caused tor’s file descriptors to only be readable by root, breaking lsof, netstat, sockstat, ss, and some of arm’s proc based utilities. This does not break connection resolution itself, but rather the file descriptor to inode mappings used to associate connections to processes. Thanks to Jake we have a partial workaround which is to filter netstat results by the owner’s uid instead which, at least for the tor deb, should give the same results. For non-deb users I’ll need to just give a notice about why it’s broken or, in the case of Ubuntu users, suggest that they turn off ‘DisableDebuggerAttachment’.
  • The latest arm release sometimes exhibits strange terminal glitches, caused by an interaction between the readline and curses modules. Thanks to Stephan Seitz for providing a reliable method for reproducing the issue.
  • Kamran submitted a patch for UPnP support in arm which I spent a couple days reviewing. It’s a nice addition, though it’s gonna need some work before it’s merged.
  • Tor Cloud use cases revealed a couple bugs with arm’s torrc validation including case sensitivity and an unexpected logging default for Debian. Thanks to koolfy for reporting the issues and Runa for the test system.
  • Saving a snapshot of the log in arm had a couple issues, as reported by Jeff Bonner on the Debian bug tracker.
  • Non-proc based connection resolution could fail due to terminal localization and an issue with the getProcessName() function. Issues caught thanks to Stephan Seitz.

The little time I’ve had for stem has mostly gone into improving and testing connection and authentication to the tor process. This module took a lot longer than I’d intended to finish but I’m really happy with the result. Also, both Sathyanarayanan and boerni have taken an interest in stem, making me a little hopeful that developing this library won’t be as lonely as arm was.

Hi all. I spent November chiefly focusing on stem, shoring up its testing and handling the connection/authentication handshake. The scope of the library is expanding very slowly but I’d argue that this is a good thing. Stem has almost compete code coverage with its tests exercising most use cases and edge situations I can think of. For instance, connection and authentication are run against configurations with…

  • an open control port
  • password authentication
  • cookie authentication
  • both password and cookie authentication
  • control socket
  • control socket with cookie authentication
  • no method for controllers to connect

… and soon I’ll be adding tests for chroot setups (a use case where our projects traditionally have a lot of issues). If you’re making Tor changes that touch on PROTOCOLINFO or AUTHENTICATE then please run stem’s integ suite. It’s quick, it’s easy, and it’ll give your change a very, very good workout. To run it simply…

git clone git://git.torproject.org/stem.git
./stem/run_tests.py --integ --target CONN_ALL

Besides stem I’ve been involved with a smattering of other issues…

  • Reviewed Filiol’s slides which, between the FUD, had a few reasonable concerns. This mostly concerned better monitoring so I filed a ticket which probably won’t get much traction until the next GSoC.
  • Discussed a library for fetching consensus information directly from a variety of sources like cached descriptors and directory authorities/mirrors (ticket). I’ll be implementing this functionality in stem and Karsten plans to do the same with a java library.
  • Discussed a Tor forum (ticket). Imho it will be doomed without Tor dev involvement and, since realistically we’ll give up on clicking through clunky web interfaces, we should have an email frontend too. Andrew disagrees so I’m taking my hands off of that project.
  • Variety of small arm issues including ASC mishandling, torrc validation issues spotted by kookfy, and fixing an rpm dependency issue spotted by unspawn.
  • Packaged the Tor posters and sent the ones to people in the US. I’ve had my fill of gigantic post office lines so the rest (and the stack of customs slips that go with ’em) will be waiting until after the holidays.
  • Filed tickets (4629, 4630, 4627, and 3958) for TorCtl issues I’ve spotted while developing stem. I probably won’t keep doing this – it’s time consuming and pointless if stem replaces TorCtl. The last issue might also exist with Vidalia though Tomás hasn’t commented yet.
  • Though not really Tor related, I submitted a patch to ReviewBoard for a weird XSS issue via comment fields. This next GSoC I’d like to set up a ReviewBoard instance for us. While I realize that shiny, ajax websites aren’t our style it would make code reviewing a lot nicer. code review, commit

All in all a good month.