What Works Marketplace: Helping Leaders Use Evidence to Make Smarter Choices

EXECUTIVE SUMMARY

As a nation, we owe it to all citizens to invest our resources in the most effective solutions to the problems we face. This is particularly true when it comes to supporting vulnerable children, families, and communities—an area of both tremendous need and expense that is critical to the health and growth of our nation.

Unfortunately, we cannot say with confidence today that we are making the most of what we spend. Less than 1 percent of federal government spending is backed by even the most basic evidence of impact.1 It may be that many government programs are working. We just don’t know.

The truth is that it is not easy to identify the most effective solutions. Relatively few interventions have been rigorously evaluated; even fewer are proven to have positive results. In fact, most evaluations show mixed results. Therefore, determining whether an intervention works is not black and white and depends on how one plans to use it. Moreover, the needs and populations in our country are constantly changing, so solutions must continually evolve and consider local context.

But we still must embrace an approach based on meaningful data, quality evidence, and rigorous evaluation. A healthy market for evidence on effectiveness would give decision makers—policy makers, funders, and practitioners—the information they need to select the appropriate solution for their circumstances. It also would shed light on areas where there is currently not enough evidence on effectiveness, and thus more innovation or evaluation is needed.

The purpose of such a market is learning and continuous improvement. It is not the separation of interventions into two neat categories of those that work and those that don’t work. Our research shows that effectiveness is far more nuanced and constantly evolving, and that the required level of effectiveness depends on the application. Judging interventions could discourage innovation because of fear of negative repercussions.

To better understand what it takes for leaders to use evidence in making critical decisions, Results for America, in consultation with The Pew-MacArthur Results First Initiative (Results First)2, commissioned The Bridgespan Group to study this “market” for evidence on effectiveness—specifically, what is the current state of the market? What can be done to strengthen it? And who must lead it.

KEY FINDINGS

To better understand how well the market is working today and how it could be strengthened, we completed over 80 interviews with individuals on both
the supply and demand sides.3 On the demand side, we targeted our interviews mostly on two particular domains: child welfare and K–12 education. We then combined the insights from those interviews with secondary research and analysis of supply side information sources to deepen our understanding and test our findings and learnings.

When we began our research, it seemed the exclusive suppliers of information were the online clearinghouses, the demanders of that information were programmatic decision makers, and the commodity they were exchanging was information about interventions that have been proven to be effective— evidence-based practices.

What we found was a market significantly more complex. On the supply side, we found a long and fragmented supply chain with many more information suppliers than anticipated. On the demand side, we found several types of users with different needs, but among key decision makers demand for evidence is still limited. In fact, we determined that decision makers are only one subset of users for this type of information. Additionally, there are organizations like Pew and Results for America supporting both the supply and demand sides.

The good news is, both supply and demand for evidence on effectiveness appear to be growing. The bad news is, there are growing pains. We identified six gaps in the marketplace that prevent supply from effectively meeting the requirements of demand:

• Gap 1: Comprehensiveness. Decision makers want information on a broader range of interventions with varying levels of effectiveness. They also want to know which interventions have not been reviewed or rated.
• Gap 2: Implementation. Decision makers want information about interventions beyond evidence of impact—including peer experience implementing the intervention—to help them make informed decisions. Few clearinghouses provide this level of information.
• Gap 3: Guidance. Decision makers are looking for guidance and support in selecting and planning to implement the appropriate intervention. Clearing- houses, however, are not set up to provide this, and the intermediaries in this space are still relatively limited.
• Gap 4: Synthesis. Decision makers are looking for more than just interventions. They also are looking for information on policies and management decisions, as well as synthesized findings and best practices. This information is not available systematically and can be difficult to find, even where it does exist.
• Gap 5: Usability. Users do not find clearinghouses easy to use, nor do they understand the differences between them.
• Gap 6: Awareness. Decision makers receive information about interventions from purveyors and peers, but they do not receive information about evidence in a systematic or effective manner.

RECOMMENDATIONS

To address these gaps and strengthen the growing market for evidence on effectiveness, we developed three sets of recommendations. First, we need to strengthen the supply of evidence on effectiveness. Second, we need to build demand for this information. Finally, we need to develop infrastructure that will coordinate and support the interaction between supply and demand.

Within these broad categories, we offer nine specific recommendations:

Strengthen Supply:

1. Increase the number of studies on interventions available
2. Increase the amount of information on interventions available
3. Increase the types of review available, not only reviews of single interventions

Build Demand:

1. Increase awareness of sources for evidence on effectiveness, particularly through existing networks
2. Reduce barriers to use of clearinghouses
3. Guide decision makers through the selection process, including connecting them with advisers

Develop Infrastructure:

1. Establish common standards
2. Increase coordination among suppliers
3. Build a vibrant adviser market

These recommendations represent our initial conclusions on how to improve the market. Given that the market is still in an early stage of development, with many small actors and limited resources, it is not always readily apparent who should take on each recommendation. We hope these recommendations provide a starting point for discussion among the many actors who have a stake in making this market function well.

MOVING FORWARD

Implementing the recommendations in this report will require efforts from all involved in the market. Clearinghouses will need to play a central role as the primary aggregators of evidence on effectiveness. The sponsors and funders of clearinghouses will need to support improvements, such as through allocation of additional resources, revised mandates, or simply advice and encouragement. However, even an ideal set of clearinghouses will not be sufficient to change behavior and ensure the use of evidence on effectiveness. Other players in

the field must support and complement their efforts. In particular, the federal government will need to lead the national conversation about evidence on effectiveness and support the many other actors who play important roles in this market. Researchers and synthesizers will need to work with clearinghouses to make the right information about interventions available. They should also continue to use the available evidence to further improve the pool of effective interventions. Foundations need to direct and support the use of evidence on effectiveness through their grant making and advocacy. Finally, state and local leaders will need to work alongside all of these actors by actively engaging in the market for evidence on effectiveness and using this information to make informed decisions.

Given the large number of actors and relatively limited resources in this market, collaboration and coordination will be essential. It will be important to focus
on continuous improvement, and not on final judgments about what works. Most importantly, it will be critical to stay focused on our end goals: a healthy market for evidence on effectiveness, greater investment in the most effective solutions, and ultimately, better outcomes for vulnerable children, families and communities.