You — yes, you — can help AI predict the spread of coronavirus

Even if you know nothing about artificial intelligence, you can volunteer to forecast the future of Covid-19.

Roni Rosenfeld makes predictions for a living. Typically, he uses artificial intelligence to forecast the spread of the seasonal flu. But with the coronavirus outbreak claiming lives all over the world, he’s switched to predicting the spread of Covid-19.

It was the Centers for Disease Control and Prevention (CDC) that asked Rosenfeld to take on this task. As a professor of computer science at Carnegie Mellon University, he leads the machine learning department and the Delphi research group, which aims “to make epidemiological forecasting as universally accepted and useful as weather forecasting is today.” The group has repeatedly won the CDC’s annual “Predict the Flu” challenge, where research teams compete to see whose methods generate the most accurate forecasts.

Initially, Rosenfeld balked when the CDC asked him to predict Covid-19’s spread. He didn’t think his AI methods were up to the challenge. Yet he’s taking his best stab at it now — and you can help, even if you know nothing about AI.

When I called up Rosenfeld on March 18, he explained that one of the forecasting methods he’s using is called the “wisdom of crowds.” Those crowds are made up of regular people, who need nothing more than a bit of common sense, an internet connection, and a few spare minutes a week.

I talked to him about how he’s predicted the spread of the seasonal flu in the past, how he’s adapting his methods to predict the spread of the coronavirus, and how we can help. A transcript of our conversation, edited for length and clarity, follows.

Sigal Samuel

When did the CDC ask you to do coronavirus forecasting, and how did you feel about it initially?

Roni Rosenfeld

It was three or four weeks ago. I was very, very reluctant.

There are many people who do forecasting and many ways to do it. Our approach is very driven by machine learning, which basically means that we try to learn more from the past than from what we think the mechanism [of transmission] is. With mechanistic approaches, people try to build models that are based on an understanding of how epidemics spread. Our approach is non-mechanistic — it makes very few assumptions about how epidemics spread and it focuses more on past examples.

That worked quite well for the seasonal flu, because we have 20 years’ worth of data from many locations. The problem with forecasting the coronavirus pandemic is that there is no historical data to go on. So the machine-learning-based approaches are actually the worst here — they’re trying to learn something from almost nothing. That’s why I was extremely reluctant to engage in this, and I was about to turn the CDC down.

Sigal Samuel

What made you decide to say yes, ultimately?

Roni Rosenfeld

Well, in addition to machine learning, we have another methodology for forecasting, called the “wisdom of crowds.” That’s when you gather at least several dozen people and ask them each individually to make a subjective assessment of what the rest of the flu season will look like. What we’ve learned from experience is that any one of them on their own is not very accurate, but their aggregate tends to be quite accurate.

My feeling was that the wisdom of crowds method may be better than the machine learning method. They tap into different resources. Machine learning taps into past examples, and there aren’t many for coronavirus. But wisdom of crowds taps into the collective reasoning and common sense of many people, and people are actually pretty good at coming up with reasonable guesses about unusual circumstances.

I’m not terribly optimistic about either one of them making long-term predictions — as in, a month or two away — because what will happen depends greatly on what we do, from government measures to individuals’ decisions about social distancing.

But I became convinced that there are two things we can do well. One is to make very short-term forecasts — one to two weeks ahead. The second thing, which I think is even more important, is forecasting not the future but the present — I mean trying to estimate in real time the current prevalence of the disease. That’s called “nowcasting.” Without knowing the current prevalence, you can’t even begin to figure out where it’s going next.

Sigal Samuel

That seems potentially very useful, because the number of confirmed cases reported by public health authorities doesn’t reflect all the people who have the virus but haven’t yet been tested for it or noticed any symptoms. How do you go about nowcasting?

Roni Rosenfeld

There are a variety of data sources that can be brought to bear on the problem: social media mentions of the illness, frequency of Google queries with related terms, frequency of access to relevant Wikipedia pages and CDC pages, retail purchases for things like anti-fever medications and thermometers, and electronic health records.

We’ve put all these kinds of data to very good use when nowcasting the seasonal flu [using machine learning]. But we built models that were fit on historical data, namely on the relationship between these data sources and the actual prevalence of flu.

The challenge with Covid-19 is that these relationships can no longer be assumed to hold because the behavior of people and of systems has changed dramatically. There’s heightened anxiety, so people may look up information about the disease, not necessarily because they have the symptoms but because they’re concerned and curious.

Sigal Samuel

Yeah, last week I bought Tylenol and a thermometer, not because I had a fever or other symptoms, but because it seemed like a good idea to have those things on hand just in case. It seems like there’s going to be a lot of noise muddying your signal.

Courtesy of Roni Rosenfeld

Roni Rosenfeld

It’s actually even worse than noise. We know how to handle noise — we combine many sources to get rid of it. The problem here is systematic bias, meaning that the behavior of people is changing systematically.

To give you one example, many doctors’ offices are changing their practices and asking people to not come in if they suspect they have coronavirus. That means they will not be captured in the measures that the CDC’s surveillance system usually records.

That’s why many of these data sources can no longer be trusted or at least need to be retrained. That’s what we’re working on now.

Sigal Samuel

How are you going about making the needed adjustments?

Roni Rosenfeld

To start with, we’ve turned off all the data sources that have to do with social anxiety — Twitter, Google searches, Wikipedia page access. We’re down to short-term time series forecasting and wisdom of crowds focusing on just the first week. We find those are fairly adaptive.

Sigal Samuel

I’m curious about the wisdom of crowds methodology. The volunteers you get are not experts in AI or forecasting or epidemiology. They’re just regular people. So what are they going off of when they make their predictions?

Roni Rosenfeld

With the flu, we ask people to forecast for different regions of the country. We give them a map with lines showing the rise of the flu in previous seasons, and they’re supposed to click and draw a line showing how they think the current season will shape up — given what they see about previous seasons and a variety of links we give them to information about the flu.

You’re not likely to do very well, because you’re guessing and you’re not an expert. But you’re going to bring to bear your common-sense reasoning, what you know from current news sources, and what you know from friends and family about what’s happening in your area. And when we aggregate together dozens of these guesses, the prediction is actually quite good, at least for the flu.

Once you’re happy with your guess, you click save and we thank you very much. We even have a leaderboard to keep track of the accuracy of different people’s forecasts. You can see how well people did last week and how well they’re doing cumulatively.

Sigal Samuel

Do people get really competitive about this and take pride in their score?

Roni Rosenfeld

Absolutely! My wife is very competitive and she gets very frustrated whenever I beat her. She says, “I’m going to beat you this time!” And she does beat me on occasion, even though she’s not an expert.

We’ve found that being an expert is not necessarily helpful. What is helpful is paying attention to detail and being very conscientious about it. Some people take a minute to do each one of the regions, and they might do okay. But the people who do better are the ones who take their time and make fine adjustments. We ask you to do it not once, but to update it every week. If you’re lazy and say “What I did last week was good enough,” you will not do as well.

Sigal Samuel

Sounds like this really rewards the perfectionists and the obsessively detail-oriented among us, which I, for one, can get behind. Where can readers go if they want to volunteer? And is there anything else they should know?

Roni Rosenfeld

They can go to our Crowdcast platform. I think it’s worth doing, because it will give us some indication. But I can tell you in advance what the problem is. It’s not hard to get volunteers to do it the first time. And some of them might do it the second week. But as novelty wears off, they tend to lose motivation by the third week.

If we could get hundreds of people to do it consistently, we could do a lot more than we’re doing now. The CDC would like us to cover every state, but we need several dozen people to do each state. Scaling this up is really important, but with people who are in it for the long term. The important thing is to do it consistently, and you do get better the more you do it.

Sigal Samuel

I find this really appealing because we’re all feeling so powerless right now. It’s psychologically helpful to feel like there’s something useful we can do from our homes.

Right now, I’m sitting in my room in Washington, DC. At this point, do you feel able to predict when the coronavirus outbreak will peak in my region? What’s your own personal forecast, and what are you basing that on?

Roni Rosenfeld

My expectation is that sometime in April or May we’re going to see a peak — unless we clamp down really strongly as some other places have done by sheltering in place. This is based on the contagiousness of this virus and on an assumption of moderate social distancing.

In reality, I think that our country will not allow this to happen, because an epidemic wave of that magnitude will severely overwhelm the health care system and raise the death toll considerably. So most likely we will implement more severe mitigation measures and try to keep the wave down.

Sign up for the Future Perfect newsletter and we’ll send you a roundup of ideas and solutions for tackling the world’s biggest challenges — and how to get better at doing good.

Future Perfect is funded in part by individual contributions, grants, and sponsorships. Learn more here.