Adding to the picture of the Obama tech-group’s cyber-campaign edge over Romney’s people, here are a few more details on the GOP’s Project ORCA — you know, the GOTV system that failed more or less completely. From Commentary:
The system had never been stress tested and couldn’t handle the crush of traffic all at once. Thousands of man-hours went into designing and implementing a program that was useful on one day and one day only, and on that day, it crashed. My source familiar with the campaign described it this way, “It was a giant [mess] because a political operative sold a broken product with no support or backup plan…”
Just to belabor the obvious. Big data and robust software take a lot of time to get right…
…but the Romney side began only began to grasp the need for such a system well into the heat of the campaign [Powerline link]:
In the primary, we learned it was difficult to be working from Boston and really affect voter turnout in the states. It was disappointing to receive data later and realize if we had access to that data earlier, we could have done something differently and affected the outcome.
We have tweaked and improved Project ORCA throughout primary, so going into the general, we had several ideas and more time to incorporate those ideas into a system that would work nationally.
(Via Ars Technica, building ORCA took place over just seven months, leading up almost to the point of the general election)
By contrast, as the Michael Scherer’s piece I quoted yesterday describes, the Democratic cyber-team spent 18 months just to build the essential infrastructure of a usable meta-database and developing the software tools that would allow the Obama team to exploit that information for use in different settings throughout the active campaigning season.
And then there’s this, by Steve Lohron the NYT’s Bits Blog:
Another truly important change was in the technology itself. “Cloud computing barely existed in 2008,” Mr. Slaby said.
This time, the Obama campaign’s data center was mainly Amazon Web Services, the leading supplier of cloud services. The campaign’s engineers built about 200 different programs that ran on the Amazon service including Dashboard, the remote calling tool, the campaign Web site, donation processing and data analytics applications.
Using mainly open-source software and the Amazon service, the Obama campaign could inexpensively write and tailor its own programs instead of using off-the-shelf commercial software.
“It let us attack and engineer our own approach to problems, and build solutions for an environment that moves so rapidly you can’t plan,” Mr. Slaby said. “It made a huge difference this time.”
To build Orca, the Romney campaign turned to Microsoft and an unnamed application consulting firm..
[But there were] a series of deployment blunders and network and system failures. While the system was stress-tested using automated testing tools, users received little or no advance training on the system. Crucially, there was no dry run to test how Orca would perform over the public Internet.
Part of the issue was Orca’s architecture. While 11 backend database servers had been provisioned for the system—probably running on virtual machines—the “mobile” piece of Orca was a Web application supported by a single Web server and a single application server. Rather than a set of servers in the cloud, “I believe all the servers were in Boston at the Garden or a data center nearby…
Open source. Build it yourself. Don’t had over your wallet to a consultant and take (allegedly) turnkey delivery days or weeks before chequered flag goes down.
Lots of folks here have more experience with this kind of work than I ever will, but my friends in the open source camp always emphasize: if you build the tool and know the tool, and do so in an environoment that’s easy for others to inspect, critique, and improve, you get good software. You certainly can get fine software from conventional proprietary approaches — but not always, and you suffer most when you have a glitch: fewer people know what’s going on, and the code itself can be much more opaque. Commenters here can flesh that cartoon out with much more bitter experience, I’m sure — but I think we all know the eternal truth that you really, really don’t want to be testing critical new components on the night.
A last point: One of the benefits of demanding extreme effort in our Presidential campaigns is so that they can serve as stress tests, a way to see how well each side handles pressure and complex tasks. And here, you can see a lot in the different approaches the two teams took to building technology intended to address essentially the same problem. You get a sense of their respective management cultures, their analytical skills, their capacity to master their emotions and organize themselves against the specific tasks they face.
Or, as our friend John Hindrocket asked just a week ago,
Whom would you count on to organize anything, Mitt Romney or David Axelrod?
Image: Titian, Allegory of Time Governed By Prudence, c. 1565.