Dean Karlan explains his plan to get USAID to take evidence more seriously.
The US spends more, in absolute dollars, on foreign aid than any other rich nation. But a lot of development experts question whether the primary American aid institution, the US Agency for International Development (USAID), is spending its budget in a way that helps the most people, most effectively.
USAID relies heavily on a small number of well-connected contractors to deliver most aid, while other groups are often deterred from even applying by the process’s complexity. Use of rigorous evaluation methods like randomized controlled trials — where development programs are tested on a random subset of the target population to see if they work — are the exception, not the norm. If the goal is for the vast majority of USAID’s $41 billion-odd annual budget to go to proven, evidence-based programs implemented in a cost-effective way, a goal that its administrators have shared for decades, there’s still a long way to go.
One of the agency’s current leaders tasked with changing this status quo is its chief economist, Dean Karlan. At the time of his appointment last year, Karlan was already a giant in the field of development economics. He founded Innovations for Poverty Action, one of the most influential research groups conducting rigorous evaluations of anti-poverty interventions in the developing world, and has taught at Princeton, Yale, and most recently Northwestern. His papers have touched on everything from efforts to increase household savings in the Philippines to agricultural insurance in Ghana to entrepreneurship classes in Peru.
His appointment was perceived as a major victory for people in and around USAID who want its programs to rely more on rigorous evidence, and Karlan reached out to Future Perfect for his first public interview on his approach to the job. A transcript of our conversation, edited for length and clarity, follows.
Dylan Matthews
I’m curious about how one goes about integrating evidence into the USAID spending process.
What’s your model of how that works? How does the agency budget go from unallocated to allocated to specific projects? And where are the points where you can inject evidence into that?
Dean Karlan
There’s one punchline philosophy, which is to apply a bit of behavioral economics to the process. The mantra of applied behavioral economics is to make it easy. Make it easy for people to do the thing that they would say they want to do in a moment of deep reflection and full information.
That doesn’t actually tell you much, but it does tell you that we’re trying to understand the processes that are in place, and how to get information in the right way to the right people at that right point in time.
I was really overwhelmed with welcome emails, welcome notes, welcome sentiments. There’s a lot of like-minded people in USAID. I’m not saying it’s been perfect, but there’s been a lot of welcoming people who say, “I want to make these changes, here’s where the challenges have been.”
We have not produced in academia the kinds of “how-to” guides dialed into the kinds of things USAID does. It’s not the nature of what academics do. Some of what we need to do is more meta-analysis, more and more synthesizing of the existing research to the specific kinds of programs that USAID does.
It’s not just a collection of interesting papers, but more prescriptive. That’s part of what I mean by “make it easy.” Say you’re a really enterprising person in a [USAID country] mission, and said, “I’m going to go read Dean’s paper on financial inclusion.” My paper was not really dialed in to them in a way that would lend itself to saying, “What exactly do I stick in this request for proposals as an activity design?”
That’s one set of work. Some of it is about is about culture change and some of it is about education. It’s taking people who are super eager, but just not as exposed to what constitutes strong evidence and what’s weak evidence. One of the most important shifts is recognizing that when we talk about using evidence, we’re not talking about using USAID evidence. We’re talking about using the global evidentiary base.
There’s a kind of a cultural instinct, when you ask, “What’s the evidence we have on X,” to look inside USAID and what USAID has produced. In fact, evidence is evidence. Who cares who paid for it? The cash studies are a perfect example of this. Sure, USAID has some landmark projects, which are super exciting. But the fact is, that’s something like 5 or 10 percent of the evidentiary base of the impact of cash transfer programs. So if you want to know what to expect from giving out cash to people, you don’t just look at the things that USAID paid for.
Dylan Matthews
Sometimes what people mean by “effectiveness” versus “cost-effectiveness” versus “evaluation” versus “impact evaluations” can get a little muddled. There are subtle but very important distinctions between these things.
What’s the bar you’re setting? What kinds of evidence and information do you want and what are some examples of of evidence or information that would fall short of that standard?
Dean Karlan
So let’s take programs at the household or the community delivery level, where there’s some service — could be in-kind, could be cash, could be a training, could be a community meeting — but there’s some delivery of a service.
Dylan Matthews
Can you give an example of that kind of evaluation? Examples of “does it work” evaluations are easier to think of, at least for me. You imagine a graduation program, say, where recipients get cash or other assets and some training in hopes they “graduate” out of extreme poverty. We’ve had randomized trials testing if that works. What’s a trial that estimates how best to set up a given program?
Dean Karlan
One example you just named: graduation programs. Inside the evaluation, there was a test of group versus individual high-frequency meetings with households, to help with the income-generating activities that the program was trying to promote.
Say I have three goats. I want to someday have seven goats and then 10 goats. I’m building a plan to get there and having regular check-ins to help deal with issues that might be arising and help these households think about how to stay on track.
There were two competing ways of doing that. One is to hold individual meetings. The other is as a community. One thinking on individual meetings is that the households might get more customized, tailored information. They might also have things that are private that they don’t want to share publicly.
On the other hand, the group meeting might help build social capital. It might help people learn from each other’s issues. On the cost side, group meetings are cheaper because one field agent goes and has one meeting with many people at once.
So there’s a clear trade-off, and we didn’t know the answer. We’ve now seen this tested in two different instances on the same program. In both instances, it made absolutely no difference, which means “do groups” because those are cheaper to do.
Dylan Matthews
What are some of the biggest barriers to integrating evidence that USAID staff have brought up to you? What makes it not easy?
Dean Karlan
One answer is a lack of good synthesis. One of the biggest bottleneck issues is that there isn’t a step in the process for [evidence]. In the process of issuing an award, there’s no step that says, “And now check and see, of the proposed activities, what’s the cost-effectiveness estimate that we have?” That’s not an explicit step.
There’s are also bandwidth issues; there’s a lot of competing demands. Some of these demands relate to important issues on gender, environment, fairness in the procurement process. These add steps to the process that need to be adhered to. What you end up with is a lot of overworked people, and then you’re saying, “Here’s one more thing to do.”
It’s really important that we make that step, ideally, a negative cost step.
Dylan Matthews
A recent internal review suggested not just that the share of USAID projects getting a formal impact evaluation is low, but the share of impact evaluations rated high quality is very low — about 3 percent. What’s your diagnosis there? Is it a lack of training? Is it unclear expectations about what makes an evaluation high quality?
Dean Karlan
I think there’s some misinformation about what makes something high quality. But I also don’t think that’s the core problem we face. I do expect and want to see more impact evaluations done at USAID. Don’t get me wrong. That is a goal.
I don’t care what proportion of our awards get impact evaluations. That’s not a metric that’s important to me. What’s important to me is, are there evidence gaps where we, USAID, could help fill them?
If we are in a good position to learn more, then that is a great opportunity for us to have an even bigger impact than our award, by helping to produce knowledge in that area. That’s not measured by what proportion of our awards are we doing impact evaluations on.
Let’s take teaching at the right level in education as an example, or cash transfers would be another one. Cash transfers had 50, 100 or so randomized trials done on them. Teaching at the right level, not as many, but maybe a dozen. There are cases where we might be doing those, and there’s not a good argument for why we should do an impact evaluation. We should do a process check to make sure that we’re delivering what was delivered. But asking the big picture question about what the impact is, is just adding a drop in an already fairly full bucket of information about the impact of those activities.
So that’s a good example of where, you know, 3 percent is too high. I’m not saying three percent is high globally for USAID. I do think the number should be higher. But the point is, it should be guided by where we can be learning something that helps the world, not by just counting our awards and saying what proportion of them have impact evaluations.
