More Geekenfreude

Adding to the picture of the Obama tech-group’s cyber-campaign edge over Romney’s people, here are a few more details on the GOP’s Project ORCA  — you know, the GOTV system that failed more or less completely. From  Commentary:

The system had never been stress tested and couldn’t handle the crush of traffic all at once. Thousands of man-hours went into designing and implementing a program that was useful on one day and one day only, and on that day, it crashed. My source familiar with the campaign described it this way, “It was a giant [mess] because a political operative sold a broken product with no support or backup plan…”

Just to belabor the obvious.  Big data and robust software take a lot of time to get right…

…but the Romney side began only began to grasp the need for such a system well into the heat of the campaign [Powerline link]:

In the primary, we learned it was difficult to be working from Boston and really affect voter turnout in the states. It was disappointing to receive data later and realize if we had access to that data earlier, we could have done something differently and affected the outcome.

We have tweaked and improved Project ORCA throughout primary, so going into the general, we had several ideas and more time to incorporate those ideas into a system that would work nationally.

(Via Ars Technica, building ORCA took place over just seven months, leading up almost to the point of the general election)

By contrast, as the Michael Scherer’s piece I quoted yesterday describes, the Democratic cyber-team spent 18 months just to build the essential infrastructure of a usable meta-database and developing the software tools that would allow the Obama team to exploit that information for use in different settings throughout the active campaigning season.

And then there’s this, by Steve Lohron the NYT’s Bits Blog:

Another truly important change was in the technology itself. “Cloud computing barely existed in 2008,” Mr. Slaby said.

This time, the Obama campaign’s data center was mainly Amazon Web Services, the leading supplier of cloud services. The campaign’s engineers built about 200 different programs that ran on the Amazon service including Dashboard, the remote calling tool, the campaign Web site, donation processing and data analytics applications.

Using mainly open-source software and the Amazon service, the Obama campaign could inexpensively write and tailor its own programs instead of using off-the-shelf commercial software.

“It let us attack and engineer our own approach to problems, and build solutions for an environment that moves so rapidly you can’t plan,” Mr. Slaby said. “It made a huge difference this time.”

By contrast, the Romney development process, again, as reported by Ars Technica’s Sean Gallagher [h/t commenter dmislev]:

To build Orca, the Romney campaign turned to Microsoft and an unnamed application consulting firm..

[But there were] a series of deployment blunders and network and system failures. While the system was stress-tested using automated testing tools, users received little or no advance training on the system. Crucially, there was no dry run to test how Orca would perform over the public Internet.

Part of the issue was Orca’s architecture. While 11 backend database servers had been provisioned for the system—probably running on virtual machines—the “mobile” piece of Orca was a Web application supported by a single Web server and a single application server. Rather than a set of servers in the cloud, “I believe all the servers were in Boston at the Garden or a data center nearby…

Open source.  Build it yourself.  Don’t had over your wallet to a consultant and take (allegedly) turnkey delivery days or weeks before chequered flag goes down.

Lots of folks here have more experience with this kind of work than I ever will, but my friends in the open source camp always emphasize:  if you build the tool and know the tool, and do so in an environoment that’s easy for others to inspect, critique, and improve, you get good software.  You certainly can get fine software from conventional proprietary approaches — but not always, and you suffer most when you have a glitch:  fewer people know what’s going on, and the code itself can be much more opaque.  Commenters here can flesh that cartoon out with much more bitter experience, I’m sure — but I think we all know the eternal truth that you really, really don’t want to be testing critical new components on the night.

A last point:  One of the benefits of demanding extreme effort in our Presidential campaigns is so that they can serve as stress tests, a way to see how well each side handles pressure and complex tasks.  And here,  you can see a lot in the different approaches the two teams took to building technology intended to address essentially the same problem.  You get a sense of their respective management cultures, their analytical skills, their capacity to master their emotions and organize themselves against the specific tasks they face.

Or, as our friend John Hindrocket asked just a week ago,

Whom would you count on to organize anything, Mitt Romney or David Axelrod?


Image:  Titian, Allegory of Time Governed By Prudence, c. 1565.

Explore posts in the same categories: Romney, Stupidity, Technology

Tags: ,

You can comment below, or link to this permanent URL from your own site.

3 Comments on “More Geekenfreude”

  1. Strayer Says:

    What bothers me is the sheer arrogance of it. Not to ask questions or test a system before launching it shows Romney’s incompetence.
    It was obvious during the whole Romney campaign that he had no idea what he was talking about.
    I’ve seen this with rich or even affluent people. It’s the someone else will handle it attitude.
    So glad it didn’t work this time.
    It worked twice for Bush and that was the scary part of this campaign.

  2. Like a lot of geeks, you’re buying the line that what went wrong was ostensibly a lack of open source software or “Microsoft was to blame”. And yet every day on the Internet, enormous projects like Facebook or Twitter use their proprietary software to handle millions of complex interactions. The voting Americans in one party was a trivial data base by comparison.

    And you’re implying that you fly blind when you open your wallet and farm a project out to consultants. Even though anyone who has worked with “free as in free beer” Drupal knows the meter is running on the hired programmers for months on end in a balloon payment.

    So you then bless a funny amalgam — the software has to be open source to be cool, but it has to be kept inhouse under close wraps to be controlled, not farmed out and not just thrown up to the crowd as crowdsourcing, either.

    Now, was Obama’s software used for tracking polls and strike work in fact open source and transparently available for the crowd to follow it in real time? Maybe the software itself was open source and “in the cloud” and all the rest of it. But if it were open source it would still be used by a closed set of people because they couldn’t very well show their game hand. The idea that open is “better” and “really open” is very welded into the geek culture, but it may not be for every occasion.

    The other assumption you’re making is that Republicans are out of date, out of touch, can’t get good help, are stupid, not willing to accept science, blah blah.

    But the guy working on the Romney aps was Al Gore’s dev!

    Zac Moffat, the digital manager, used the consulting firm, Targeted Victory, where he used to work. That is the “unnamed firm” that Sean Gallagher refused to name — he tweeted to me that he is waiting for TV to confirm they worked on this. We’re still waiting…

    That’s why I ask outright if there wasn’t sabotage or simply casual neglect that would come with being an Obama voter just doing a gig for money. It’s not about out-sourcing or in-house; it’s about the nature of the devs’ political sympathies.

    Obama had enthusiastic coders working for him who probably went without sleep and drank Red Bull and ate pizza for days on end to help him. Romney didn’t have enthusiastic coders because I think they were the same kind of people — Obama voters.

    What will you all say if it turns out that Al Gore’s dev and various other Obama voters who had not real stake in the Romney campaign didn’t test this software, eh? It’s not clear how closely Zac Moffat oversaw their work, as several news accounts say he did not oversee it. The program manager in Targeted Victory would have likely done that — unless we’re going to find out that it was the moonlighting Googler or the Msft people — about which we don’t know anything.

  3. onkelbob Says:

    The physical infrastructure is always the easiest to overlook and under specify. That’s a ugly bottleneck having one web server and one app server. It’s rookie mistake to not think that through and it’s also one that most rookies make. One of the advantages of being in the biz from way back when (a time when physical facilities were not reliable) is that we greybeards always draw up the physical map first. That way you are protected from the single point of failure problem. You can’t let a power bump in one building or a back-hoe fade (call before you dig!) shut down your entire operation.
    This wasn’t a s/w failure, this was a planning failure. The s/w probably could have performed the task but it did not have CPU’s to do the actual work. From what I understand the servers did not crash or wedge, rather the latency was unaccounted for given the physical facilities. It takes time, albeit a small amount of time, to execute a computer command. However, as they learned, many small amounts add up to one very big amount, especially when many = millions. How long does it take you to pay the bridge toll? How long does it take you to reach the toll booth? Which time changes during rush hour? Rookies.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: