Iowa’s 2016 caucus app worked and everyone forgot about it

The 2020 mobile app is being blamed for the Iowa caucus failure. But four years ago, this other mobile app worked just fine.

The 2020 Iowa caucus has become a tech failure story: A flawed mobile app that was supposed to simplify reporting the caucus results ended up creating chaos and forced the Iowa Democratic Party to rely on paper ballot backups. Three days later, we still don’t know who won — and may never. But this wasn’t the first time mobile apps were used for the caucus. In fact, both parties used mobile apps with similar functions in 2016. We just didn’t hear much about them because they worked.

“This was my worst nightmare when I was watching this happen,” Rodney Guzman, the co-founder and co-CEO of the company that built the 2016 apps, told Recode. “I was thinking to myself, ‘oh my God, that could’ve been us.’”

But it wasn’t, because the 2016 app developers had resources that the 2020 developers sorely lacked: experience, time, and funding.

In 2016, both parties faced contested caucuses and the memory of 2012’s botched Republican caucus, when Mitt Romney was incorrectly declared the winner, was fresh in their minds. A mobile app that precinct chairs could use to calculate and report results — presumably faster and more accurately — was seen as a way to avoid repeating previous mistakes.

So the parties partnered with Microsoft, which contracted a company called InterKnowlogy to build the app and other digital caucus infrastructure on Microsoft’s technology. InterKnowlogy had nearly two decades of experience in app development, worked with Microsoft before, and a track record of success with “primetime apps,” as co-founder and co-CEO Rodney Guzman calls them. The Iowa Caucuses’ 2020 app, in contrast, was built by a relatively unknown new company called Shadow, which had never done something on this scale before.

Both the 2016 app and the 2020 app had similar requirements: While their functions weren’t complicated, there needed to be several versions of the same app for different operating systems, devices, and, in the case of 2016, caucusing systems. Most importantly — and this is unusual for app development — they all had to work perfectly for their debut (and only) use. There would be no time to roll out updates and improvements. The consequences of getting it wrong, as we now know, would be very public.

But the 2020 app developers had far fewer resources than 2016’s app team had. Guzman told Recode that his company had about a year of lead time, the first three months of which were spent on product design, meeting with both parties, and doing usability studies to anticipate precinct chairs’ needs — down to the size of the digital buttons they’d have to tap.

That three months is a longer amount of time than Shadow had in total, per multiple reports that say it only got two months to build its app. (Shadow CEO Gerard Niemira seems to dispute this, telling Vice that Shadow “began [its] engagement with the IDP in August” and “basically had the month of August, September, October, November, and December to do it,” though he didn’t specify when the actual coding work began.)

“To me, as an engineer, that just sounds nuts,” Michael Gramley, a software engineer who worked on InterKnowlogy’s app, told Recode. “You cannot make a stable platform in two months. We needed that much time just to test everything. If they really did that, then whoever the management was in that company set their engineers up to fail.”

The extra months InterKnowlogy had gave its engineers time to properly build and test the app. It also gave the product side time to properly prepare and train the precinct chairs who would be using it. Guzman and his team took into account that chairs may not own or use smartphones and had little experience with apps, let alone any familiarity with necessary security measures like two-factor authentication.

“We were really concerned that a lack of training was going to be the cause of most of the issues,” Guzman said. “We spent a lot of time and preparation to make sure that both parties were pushing out training material.”

This proved to be a good idea. The 2020 app’s main problem, according to the IDP, was a “coding issue” that caused partial results to be reported to the IDP headquarters. But there were widespread reports that the precinct chairs weren’t adequately prepared or instructed on how to use the app, and that they found it difficult to impossible to download, install, and boot on their smartphones.

One thing that shocked Guzman was that Shadow didn’t put its app in Apple’s or Google’s app stores, and instead forced users to “sideload” it — basically, download the app through a different channel. That’s a difficult process for experienced smartphone users, let alone someone who barely uses a smartphone.

“A lot of our jaws hit the floor when we saw that they didn’t even use the proper mobile app stores to deploy the apps,” Guzman said. “It had red flags all over it.”

The 2016 app, by contrast, was available in app stores for precinct chairs to download the way they would any other app by the beginning of October 2015 (you can still find both the Democrat and Republican versions in Android’s app store). Gramley guesses the difference here is that Shadow didn’t finish its app in time to get the app stores’ approval.

InterKnowlogy also built an automated phone system that precinct chairs could use to call in the results if they weren’t able to or didn’t want to use the app. Even with all of its work training precinct chairs on the app, Guzman said a “large percentage” of the results (he couldn’t remember the exact number) were reported through the phone. According to Gramley, there were “a couple disconnections,” but it was “not a major issue.”

This year’s caucus appears to have relied on human operators instead, and, the New York Times reported, there weren’t enough of them to handle a call load that really should have been anticipated after 2016.

Gramley joined the project about six months before the app’s debut, when the prototypes were done and the work building the actual apps began. He said there were about five to seven engineers on the project working full time at that point. They had a “code complete” version two months before the caucus and spent the rest of the time testing it out and fixing bugs.

Even with the comparatively longer lead time, Gramley said, InterKnowlogy’s engineering team was working 80-hour weeks leading up to the caucus.

“We pulled it off, but we had a lot of late nights leading up to it,” he said.

Had InterKnowlogy been offered the same tight deadline as Shadow apparently was, Guzman said he wouldn’t have accepted the project.

“There’s too much at risk,” he said. “I just wouldn’t have done it.”

Time was not the only thing Shadow was short on. Federal Election Commission records show that the IDP paid Shadow just $63,000 for its work on the app. Even if you combine that amount with the $58,000 that the Nevada State Democratic Party paid Shadow for a very similar (and, now, unused) version, that’s still far less than what an experienced company with the necessary talented engineers would charge for this kind of work. Gramley didn’t know how much InterKnowlogy was paid, but he estimated that the engineers’ salaries alone would have been between $1 million and $1.5 million for six months of work.

Guzman wouldn’t reveal how much InterKnowlogy was paid but said it was “at least” 10 times more than Shadow was.

“You get what you pay for,” he said.

According to the New York Times, only a quarter of the chairs in Iowa were able to use Shadow’s app. The rest relied on the phone system, which was overwhelmed with calls. Many chairs reporting hours-long hold times and hang-ups before they could report their results (including one while the chair was on CNN).

When all was said and done, Microsoft took a bit of a victory lap, posting about the “new technology” that “ushered in a new era for the caucuses.” PBS even cited the app and its speed as marking the beginning of “the smartphone election.” Ninety-five percent of the 2016 caucus results were in before 11 pm CT.

Four years later and more than 85 hours after the 2020 Iowa caucus began, the results are still not complete and the Democratic National Committee chair is calling for a recanvass. Asked for comment, Shadow pointed Recode to a statement on its website that said it “sincerely regrets” what happened on caucus night and “will apply the lessons learned in the future.”

The Iowa Democratic Party didn’t respond to request for comment on why it didn’t use the 2016 app for this year, and Microsoft wouldn’t comment beyond a tweet from a spokesperson that said it wasn’t involved in the 2020 app.

What frustrates Guzman and Gramley is that Shadow’s mistakes are being used to disparage the very concept of using mobile apps for elections, and InterKnowlogy’s success in 2016 has largely been erased. Several articles about 2020’s woes didn’t mention the 2016 app’s existence. For instance, the Wall Street Journal said “Iowa’s tally-by-app experiment failed.”

Despite pre-caucus concerns that the app could get hacked and the results altered, in the end its undoing appears to be something decidedly less nefarious: The people behind it didn’t know what they were doing. That disaster of a mobile app was the effect, not the cause. And that’s a mistake humans have been making for millennia — since well before apps and smartphones were even invented.

“My fear is that the lesson people will take from this is that apps are bad for this, not that bad processes are bad for this,” Gramley said. “It’s unfortunate because you can do this the right way. There is a right way to do this.”

Open Sourced is made possible by Omidyar Network. All Open Sourced content is editorially independent and produced by our journalists.