December 21, 2018Article

Vox: It’s hard to design good policies. This simple idea can help governments do it.

by Meagan Neal

Global/ Results for All/ 2018/

In 2014, the Zambian government knew its education system had a problem. The country had expanded school access, and most girls and boys were attending primary school.

But data from national assessments told a less encouraging story: Many of the kids in school weren’t actually learning. Only 33 percent of Zambian Grade 2 students were able to read a single word in their local language. The Ministry of General Education knew it had to try something different.

So the Zambian government decided to focus on helping kids catch up. But how? Some insight came from India, where the NGO Pratham had pioneered and scaled a remedial education approach known as Teaching at the Right Level.

Researchers from the Abdul Latif Jameel Poverty Action Lab (J-PAL) — a global research organization that conducts randomized evaluations of social programs — had evaluated versions of this approach for more than a decade. (Disclosure: I worked at J-PAL from 2015 to September 2018.) The researchers found that Teaching at the Right Level consistently helped kids improve their reading and math skills — particularly the children left furthest behind.

When J-PAL and Innovations for Poverty Action (IPA), a global research nonprofit, shared evidence with the Zambian government on this approach, the government was enthusiastic about trying it in Zambia. But adapting and scaling up a program in a new context isn’t easy. So to help make sure the program would be effectively implemented, J-PAL and IPA put in a bit of extra investment. They hired a staff member to work directly with the Ministry of General Education, sponsored exchanges to learn from the Indian program, and convened a large group of partners to design a pilot Teaching at the Right Level program for 80 Zambian schools.

Three years later, kids in 500 schools are benefiting from an evidence-based program to improve their math and reading skills — and the government is working to scale it up further.

The Zambian experience highlights a key opportunity in global development. Many conversations about development brush over governments, who are viewed — fairly or not — as sclerotic actors who can’t deliver services efficiently and effectively. But here’s the thing: Government interventions work at a scale that is absolutely unmatched. Investing in making them work better should be a priority if we want to lift up the world’s poor.

At the same time, there is more evidence on what helps improve education, health care, and social assistance than ever before — and governments like Zambia’s are increasingly eager to use it. Already, government ministries around the world are working to evaluate the impact of their programs, to make better use of their administrative data, and to draw ideas from outside research that might help improve their own services. But governments also face limited resources. While in theory many governments are keen to make their programs more evidence-based, in practice it’s a challenge to do so.

This means that if we want to improve the effectiveness of government programs, we need to make evidence-based policymaking more feasible in the halls of power. Which is why J-PAL and IPA’s simple investment — hiring someone to help the Zambian government use evidence in the design and implementation of a new program — is the kind of strategy we should think about applying elsewhere.

Making public spending more effective is one of the best tools we have to improve people’s lives

Governments provide crucial social services such as education and health care to billions of people around the world. And like most organizations, they’re not always doing it as effectively as they could be.

But the scale and reach of government programming means that even small improvements in their effectiveness can positively impact millions of people’s lives.

And yet, in discussions about development, the upsides of working with governments are frequently overlooked. Claire Walsh, who manages J-PAL’s Government Partnership Initiative (GPI) and with whom I worked when I was at J-PAL, says, “One of the big concerns people have is that governments are either ineffective or corrupt and can’t deliver good services to their people. A lot of money goes to people who aren’t eligible; there might be corruption; there might be problems with delivery.”

She says a common perception is that “we should be working just with the people to try to hold their governments accountable — working withgovernments isn’t really a good use of time or money.”

But although holding governments accountable is indeed vital, such thinking ignores a lot of potential impact. “Challenges like corruption or poor implementation are real, but that doesn’t mean that governments can’t improve,” says Walsh. “When you think about who’s ultimately going to be providing critical public services for people in low- and middle-income countries, as foreign aid becomes a smaller and smaller portion of the amount of social spending in low- and middle-income countries, it will be governments who are providing the bulk of these social services.”

And a lot of government agencies, she notes, are deeply dedicated to doing a good job for their citizens. “Improving what they do — even just a tiny bit — can have catalytic effects for millions of people because they’re already operating at a scale that reaches such a big part of the population,” she says.

If there were ever a time to deepen our focus on governments, it’s now. The amount of rigorous evidence on effective anti-poverty policy has grown tremendously over the past two decades, and this knowledge shouldn’t just sit on a shelf. Simultaneously, improvements in technology and administrative data have made it easier for governments to collect actionable information.

And increasingly, governments are eager to use this data. A 2017 report by Results for All found that over the past six to eight years, governments have adopted a wide range of new initiatives to increase the use of data and evidence in policy, including big data analytics, citizen engagement platforms, and trainings to build policymakers’ skills.

But government officials have limited time, constrained budgets, and a host of competing priorities vying for their attention. So while incorporating evidence in policymaking makes sense in theory, it ends up being a challenge in practice. Where to even begin? How to know whether a study is high-quality in the first place, not to mention relevant for their context? Who actually has the time to push forward a new initiative?

A simple idea: make it someone’s job to use evidence

It’s not enough for a government to commit to greater evidence-based decision-making — although it’s a start. Building a culture of using evidence in government is a long process. It requires not only relevant evidence and a willingness to learn and adapt, but also dedicated time, resources, and technical skills.

And that’s where the idea of making it someone’s job in government to help apply evidence comes in. When time and money are the main barriers preventing governments from using evidence — rather than, say, a lack of relevant research or a lack of interest — small investments in personnel can be powerful.

In Zambia, for example, a policy manager was hired to sit inside the Ministry of General Education. During the first year of the Teaching at the Right Level pilot, this staffer worked with the Zambian government to budget for the program, help officials understand the methodology behind the approach, plan trainings for teachers and school administrators, and develop a monitoring system to ensure the program was being effectively implemented.

This support helped the ministry understand why Teaching at the Right Level worked in India, and then, instead of simply copying a program from abroad, adapt it carefully for the Zambian school system. Now the staffer is helping the ministry apply the results of the pilot and prepare to scale up the program more widely.

And there are other approaches to this idea. A research assistant might organize a ministry’s administrative data to make it easier to use. Or researchers on contract might conduct an evidence review to figure out what studies are relevant to a government’s priority issue. A staff member might handle the logistics of pushing forward a new evidence-based reform, drafting presentations, and making sure key deadlines are met. Eventually, a government might work with researchers to develop an entire in-house laboratory for innovation and evaluation.

“It’s important to meet governments where they are,” says Walsh. “One of our government partners said to me once: ‘I wanted to build a culture of evidence and evaluation, but I realized we couldn’t even do basic data analysis. So I realized I needed to build a culture of data first.’”

While some governments are ready to launch into evaluating their own policies or implementing an evidence-based program, most aren’t there yet. Helping governments first make their own data more useful for day-to-day implementation decisions — Is the government reaching the people it wants to reach? If not, where do they need to deploy more resources? — can help build the foundation of evidence use necessary to get there.

Over the past three years, GPI has supported 28 partnerships, at least three of which have already contributed to scale-ups of evidence-based policies. An initial grant from GPI helped the J-PAL Africa team jump on the opportunity to work with the Zambian government. In India, GPI supported a J-PAL staff member sitting in the Ministry of Rural Development, where she lent support to the government’s work to scale up an evidence-based financial reform of a national social protection program reaching over 70 million beneficiaries. And in the Indian state of West Bengal, a small grant supported a monitoring system for the scale-up of a training program for informal health care providers.

The work won’t always yield results. The right circumstances need to be in place: a strong partnership, sufficient data, good timing, and government “champions” who are deeply committed to incorporating evidence. Even with more and more governments interested in using evidence, that’s a rare combination.

This also isn’t to claim more credit than is due. Skeptics of investing in work with governments will rightly point out that — in contrast to other high-priority causes like malaria prevention and deworming — you can’t claim a causal impact on people’s lives. In successful cases of evidence-based policy reform, there are so many actors involved that it’s difficult (if not impossible) to isolate the impact of one person or grant.

These are valid concerns, and those who want to be certain of maximizing the direct impact of their donations might justifiably prefer to invest elsewhere.

But if you take a “hits-based giving” approach — similar to a venture capital philosophy that one or two big wins can make an entire portfolio worth it — it’s reasonable to imagine how investing in evidence use could be a cost-effective way to improve policy. And indeed, a recent cause report on evidence-based policy by Founders Pledge suggests just that: the authors estimate that GPI’s work is roughly three to four times more cost-effective than direct cash transfers (an increasingly common benchmark).

Because with governments working on such a massive scale, helping them incorporate evidence can improve millions of people’s lives — even if only a few of the initiatives actually work.

So what happened in Zambia?

The Teaching at the Right Level pilot program was implemented well, according to monitoring data — and it worked. Most students had lacked basic reading and math skills at the start of the program. But a year later, many of the students at the bottom of the class had made big learning gains: the share of students who could not read a single letter fell from 33 percent to 8 percent.

Enthusiastic about the results of the pilot, the Ministry of General Education worked with its partners — the British Council, the Global Partnership for Education, IPA Zambia, J-PAL, Pratham, the UK Department for International Development, USAID’s Development Innovation Ventures, the USAID Zambia Mission, UNICEF, VVOB, and Zambia Education Sector Support Technical Assistance — to tease out lessons from implementation and to plan for scale. Currently, around 500 schools are running Teaching at the Right Level sessions. By 2020, the program will be implemented in around 1,800 schools — reaching approximately 280,000 students in grades 3 to 5.

And the success in Zambia also helped catalyze a larger movement across the African continent: education ministries in Botswana, Côte d’Ivoire, Kenya, and Nigeria are now developing their own Teaching at the Right Level pilots.

It’s a reminder of what’s at stake with government work: the world’s biggest development budgets and the widest reach of social programs. This means that governments are key players in the fight to eradicate poverty, and that where we can, we should be working with them — not working around them.

And where small investments of time and resources can help governments apply evidence, we can take advantage of research and data that already exists — and use it to create concrete policy changes that improve education, health care, and social assistance for millions of people. That’s potential for high-impact change that we shouldn’t ignore.