Leaders at all levels of government can and should harness the power of data and evidence to solve our nation’s greatest challenges. Over the past 10 years, policymakers from both parties have increasingly turned to evidence-based solutions to get better results and lower costs. This differs from the far too common approach where elected officials create programs and policies based on anecdote, politics, or well-intentioned ideas that may or may not be backed by evidence.
Over the last few years, Congress has passed bipartisan legislation that recognizes and advances the role of evidence, including the Workforce Innovation and Opportunity Act (WIOA), Every Student Succeeds Act (ESSA), and the Evidence-Based Policymaking Commission Act. Members of Congress have also recently considered other results-driven legislation, including the Social Impact Partnerships to Pay for Results Act, Juvenile Justice Reform Act, Strengthening Career and Technical Education for the 21st Century Act, and Families First Prevention Services Act.
But what exactly makes legislation evidence-based? And how can Congress strengthen this “What Works” approach when writing future legislation?
This policy brief outlines 9 ways Congress can ensure federal laws are evidence-based. Congress should take all 9 approaches whenever possible, though we recognize that some of the following proposals will be more relevant to some bills than others.
Recommendation #1: Evaluate the effectiveness of federal programs and practices.
- Set aside a percentage of program funds (1%) for federal agencies to conduct rigorous, independent evaluations of programs and practices.
- Authorize federal grant recipients to use their federal grant funds for evaluation.
Rationale: Congress rarely requires evaluations of federal programs and practices. According to a recent U.S. Government Accountability Report report, only 37% of federal program managers reported that an evaluation of their programs had been completed in the last 5 years. Thus, federal policymakers lack basic information about the effectiveness of federally-funded programs, let alone information on how to improve them. RFA believes that the federal government should invest at least 1 penny of every dollar to determine what works so that the other 99 cents are spent more effectively and efficiently. And while rigorous randomized control trials (RCTs) are the best way to determine impact, a range of evaluations and studies are needed to build the evidence base.
- The FY16 and FY17 appropriations laws permitted the Secretary of the U.S. Department of Labor to set aside up to .75% of program funds for evaluation.
- The Every Student Succeeds Act authorizes the U.S. Department of Education to set aside .5% of program funds for evaluation.
- The FY17 appropriations law authorized the U.S. Department of Health and Human Services (HHS) to set aside .33% of Temporary Assistance for Needy Families (TANF) block grant funds for evaluation, research, and technical assistance.
Recommendation #2: Define what “evidence-based” means.
- Include a rigorous, tiered definition of what “evidence-based” means to ensure federal funds are invested in the most effective approaches where the evidence base is strong, while allowing for flexibility where the evidence base is still being developed.
- Require federal grant applicants to review the body of evidence when identifying evidence-based approaches, rather than merely focusing on one or a few evaluation studies.
Rationale: Simply including the word “evidence-based” in federal legislation will not improve outcomes, since evidence can mean anything from an anecdote to the most rigorous randomized control trials, from qualitative research to quantitative evaluation. Recent legislation (described below) defines the range of studies that count as evidence to clarify how federal funds can or should be spent.
- The Juvenile Justice Reform Act (H.R. 1809) defines “evidence-based” programs as a program or practice that:
- is demonstrated to be effective;
- is based on an empirically supported theory;
- has measureable outcomes; and
- has been tested through randomized control studies or comparison group studies.
Promising programs are any program or practice that is:
- demonstrated to be effective based on any objective, valid evaluation; and
- will eventually be evaluated through a rigorous study.
- The Every Student Succeeds Act defines 4 tiers of evidence, directing grantees to spend funds on practices with higher levels of evidence where the evidence base is strong, while allowing grantees to spend funds on all levels where the evidence base is developing.
Recommendation #3: Apply the definition of “evidence-based” to how federal grant funds are allocated.
- When the evidence base is strong, require federal grantees to invest all or a portion of their funds in evidence-based practices.
- When the evidence base is still developing, require that an absolute priority or competitive preference be given to grant applicants that can demonstrate they will use competitive grant funds on evidence-based activities.
Rationale: The twin goals of evidence-based policymaking are to get better outcomes for citizens while more effectively investing taxpayer dollars. Directing federal grant funds toward evidence-based practices is a way to accomplish these purposes. Policymakers should be careful not to jump the gun, however, by requiring that all federal funds be invested in evidence-based activities if sufficient research and evaluation do not exist. But as the evidence base grows, so can the requirements to invest federal dollars in what works.
Examples: The Every Student Succeeds Act (ESSA) takes both approaches. First, ESSA requires all low-performing schools to use their school improvement (formula) funds to create improvement plans that include evidence-based interventions that meet the top three tiers of evidence (in the definition above). Second, ESSA requires the U.S. Secretary of Education to give a priority to federal grant applicants that propose to implement evidence-based practices in the top 3 tiers under 7 competitive grant programs (e.g., literacy education, family engagement).
Recommendation #4: Authorize a tiered-evidence innovation fund.
- Authorize a tiered-evidence grant program that: (A) allocates federal funding based on the level of evidence provided, with smaller awards made to test new and innovative strategies and larger awards made to scale strategies with stronger evidence; and (B) requires a rigorous, independent evaluation of each grant activity to further build the evidence base.
Rationale: This tiered-evidence design enables more dollars to be directed towards federal grant applicants that have demonstrated success and whose interventions are ready to be scaled, while directing lesser amounts toward interventions that need to be tested and proven. The tiered approach also requires evaluations to grow the evidence base, and to help grantees learn and improve. The chart below illustrates how the tiers work together. Federal legislation focused on large policy issues (e.g., higher education) should include an innovation fund that encourages innovation and builds evidence that ultimately inform how larger formula grant dollars are spent.
- Currently there are 5 authorized federal evidence-based innovation programs: the Social Innovation Fund; the Education Innovation and Research program; the Teen Pregnancy Prevention Program; the Maternal, Infant and Early Childhood Home Visiting Program; and Development Innovation Ventures.
- The Every Student Succeeds Act (ESSA) authorizes the Education Innovation and Research program with 3 types of grants: early-phase grants to fund the development, implementation, and feasibility testing of a promising program; mid-phase grants to fund implementation and a rigorous evaluation of a program that has been successfully implemented under an early-phase grant; and expansion grants to fund implementation and a rigorous replication of a program that has produced sizable, important impacts.
- The Strengthening Career and Technical Education for the 21st Century Act (H.R. 2353) would authorize a new innovation fund designed to create, develop, implement, or take to scale evidence-based, field initiated innovations, and to rigorously evaluate such innovations.
Recommendation #5: Provide Pay for Success authority.
- Authorize a new Pay for Success fund or include Pay for Success as an allowable activity within existing federal grant programs, including innovation funds.
Rationale: In most cases, Congress authorizes a program, sets the policy for how program dollars should be spent, and then directs awards to grant applicants that promise to follow the rules. Under Pay for Success, rather than pay for a service up front, a government agency enters into an agreement to pay only after a grantee delivers specified results. This approach uses evidence on the front end to select a proven intervention to tackle a specific challenge, and then uses evidence on the back end to determine if outcomes have been achieved.
- The Workforce Innovation and Opportunity Act (WIOA) includes provisions that: (1) increase the amount of WIOA funds states can set aside and distribute directly from 5-10% to 15% and permits states to invest these funds in Pay for Performance; (2) permits states to invest their own workforce development funds, as well as non-federal resources, in Pay for Performance; (3) permits local workforce investment boards to invest up to 10% of their WIOA funds in Pay for Performance; and (4) permits States and local workforce investment boards to award Pay for Performance contracts to intermediaries, community based organizations, and community colleges.
- The FYs 14-17 appropriations laws authorized the U.S. Department of Justice to invest up to $7.5 million of Second Chance Act funds in Pay for Success efforts.
- The FYs 15-16 appropriations laws authorized the Corporation for National and Community Service to invest up to 20% of Social Innovation Fund dollars in Pay for Success
- The Social Impact Partnerships to Pay for Results Act (H.R. 576) is a comprehensive Pay for Success bill that would direct $300 million in federal resources to states and local communities to support innovative Pay for Success arrangements.
Recommendation #6: Increase flexibility for federal grantees in exchange for using data and evidence to improve results.
- Authorize federal agencies to waive program requirements for federal grant recipients to allow them to implement innovative or evidence-based interventions in exchange for rigorously evaluating the new approaches.
Rationale: Congress usually includes a list of required and allowable activities for each federal program it authorizes. Congress could instead clarify the goals of each federal program and then provide flexibility to federal grantees to meet those goals using innovative and evidence-based strategies, while also requiring rigorous, independent evaluations of those strategies.
Example: The FY14-17 appropriations laws authorized 7 federal agencies to enter into up to 10 Performance Partnership Pilots for disconnected youth (P3). P3 pilots are agreements that give states, regions, localities, and/or tribal communities additional flexibility in using grant funds from 7 federal agencies: the U.S. Departments of Education, Housing and Urban Development, Justice, Labor, and Health and Human Services, along with the Corporation for National and Community Service and the Institute of Museum and Library Services. In exchange, P3 partners commit to achieving significant improvements for disconnected youth in educational, employment, and other key outcomes in exchange for this new flexibility.
Recommendation #7: Support appropriate sharing and use of data and evidence.
- Require federal agencies to disseminate and promote the utilization of evidence-based interventions (including all federally funded evaluations) in a publicly available, user-friendly format such as a What Works Clearinghouse.
- Authorize state and local government recipients of federal funds to share federal administrative data among their relevant agencies (consistent with strong privacy protections) to improve outcomes.
- Require federal agencies to provide technical assistance to its grantees on appropriate data sharing, data use, and data privacy.
Rationale: Rapid advances in technology and data tools have created enormous opportunities to understand the impact of government investments more quickly and at a lower cost. The revolution in big data, analytics, and rapid-cycle evaluation that is currently benefitting the private sector—such as the retail and medicine industries—could and should transform the impact of the public sector as well. At the same time, Congress must take great care to protect personally identifiable information and to guard against government overreach into the lives of individual citizens.
- The FY17 appropriations law directed the U.S. Department of Health and Human Services (HHS), with help from the U.S. Department of Labor, to create a “What Works Clearinghouse of Proven and Promising Projects to Move Welfare Recipients into Work.” This law also directed HHS to facilitate the sharing of information and best practices among states and localities.
- In 2014, the Administration for Children and Families within HHS created a Confidentiality Toolkit designed to promote appropriate, confidential data sharing across human service agencies. The toolkit had two goals: 1) to help state and local leaders provide more effective services, and 2) to provide greater clarity to the rules governing confidentiality. While not required by statute, such technical assistance could be included in legislation.
Recommendation #8: Repurpose federal funds away from practices, grantees, and programs that consistently fail to achieve desired outcomes.
- Use evaluation and outcomes data to identify low-performing federal grantees (e.g., the bottom 10%) and then require them to re-compete for future funding in order to continue receiving federal resources.
- Structure federal competitive grant programs such that grant recipients secure funding for a certain period of time (e.g., 3 years) but must show results in order to receive continuation funding (e.g., 2 additional years).
Rationale: Evidence-based policymaking involves: building evidence about the practices, policies, and programs that will achieve the most effective and efficient results so that policymakers can make better decisions; investing limited taxpayer dollars in practices, policies, and programs that use data, evidence, and evaluation to demonstrate how they work; and directing funds away from practices, policies, and programs that consistently fail to achieve desired outcomes. Federal investments in a practice, grantee, or program represent an opportunity cost, and those which are consistently low-performing should be improved over time or ultimately eliminated because they are not effectively serving populations in need.
- The Juvenile Justice Reform Act (H.R. 1809) would require states to subgrant delinquency prevention funds to local governments and nonprofits for 5-year periods. However, by the end of year 2, states must determine if their subgrantees have met the law’s requirements. If not, states must terminate the subgrant and reallocate those funds to other sub-grantees or use them in future subgrant competitions.
- The 2007 Improving Head Start for School Readiness Act requires grantees that receive poor ratings to compete to keep their grants. In FY12, the Administration for Children and Families within HHS established the Head Start Designation Renewal System requiring Head Start grantees to compete for grants moving forward if they failed to meet criteria related to service quality, licensing and operations, and fiscal and internal controls, including grantees scoring in the bottom 10% on certain outcomes measures.
Recommendation #9: Use federal funds to build state and local capacity.
- Require federal agencies to provide technical assistance to federal grantees (including states, local governments, nonprofits, and others) on building and using evidence. Since most federal agencies provide technical assistance through contracts, require that T.A. providers be proficient in building and using evidence.
- Require an absolute priority or competitive preference in federal competitive grant programs (including grants to states, local governments, nonprofits, and others) for applicants that demonstrate they have the capacity to build and use evidence.
- Authorize recipients of federal grants to use federal funds for rigorous program evaluation and evaluation capacity-building.
Rationale: The success of federal laws rises and falls on effective implementation at the state and local level. However, many states, local governments, nonprofit organizations, and others lack the capacity to evaluate their federally-supported work and to use those findings for program improvement. Technical assistance and targeted capacity building are critical to ensuring that federal evidence-based legislation succeeds.
- The Juvenile Justice Reform Act (H.R. 1809) would require the Office of Juvenile Justice and Delinquency Prevention (OJJDP) within the U.S. Department of Justice to provide training and technical assistance on evidence-based and promising programs or practices that promote the purposes of the legislation.
- The Every Student Succeeds Act allows the U.S. Department of Education to use evaluation funds to evaluate programs but also to “assist recipients of grants under such programs in collecting and analyzing data and other activities related to conducting high-quality evaluations.” ESSA also directs states to ensure that technical assistance providers to districts are proficient in evidence-based strategies.
Results for America (RFA) is committed to helping policymakers at all levels of government harness the power of data and evidence to solve our greatest challenges. We produced this discussion draft to help federal policymakers improve outcomes for children, families, and communities by drafting legislation that is evidence-based. We have also produced other relevant resources:
- RFA’s 2016 Federal Invest in What Works Index highlights the extent to which 7 federal agencies have built the infrastructure necessary to be able to use data and evidence when making budget, management, and policy decisions; and
- RFA’s upcoming Invest in What Works Congressional Index will highlight the extent to which targeted Congressional Committees included evidence provisions in the federal legislation they passed during the previous Congressional session.
About the Invest in What Works Policy Series
This policy brief is part of Results for America’s Invest in What Works Policy Series, which provides ideas and supporting research to policymakers to drive public funds toward evidence- based, results-driven solutions. Results for America is helping decision-makers at all levels of government harness evidence and data to make progress on great challenges. Our mission is to make investing in what works the “new normal,” so that when policymakers make decisions, they start by seeking the best evidence and data available, then use what they find to get better results.