What Works Toolkit Purpose

The purpose of the What Works Toolkit is bring together the best research in the field with examples of where these practices are working across the country to expand use among state and local government.  We believe that reforming government procurement to focus on outcomes can increase the impact of social services and improve people’s lives. The What Works Toolkit can be used by both government to guide implementation of an evidence-based procurement process that provides the necessary collaboration, focus on evidence and outcomes, and feedback loops, to ensure community problems can be solved at scale.  

Recommendation #1: Government should collaborate with human services providers to define desired objectives and establish outcomes.

Actions:

1. Government should engage with service providers and service recipients prior to the issuance of a concept paper/or an RFP to validate needs and collaboratively identify desired outcomes using available mechanisms such as pre-proposal conference with providers, letter of interest, or request for information.
2. Government should dedicate a person or team responsible for strategically managing the city’s portfolio of most important procurements that are due in the upcoming year.
3. Nonprofit human service providers should think holistically about how a potential government contract would achieve improved outcomes for the populations they serve and bring these overarching goals into government’s procurement process.  

Rationale:

To achieve better outcomes, both government and service providers need to change their normal way of doing business. An upcoming procurement can be used as an opportunity to revisit goals and align resources to meet the current priorities or point of view on service provision. Policy makers and practitioners should take time before the concept paper or RFP is issued to engage providers and community members to clearly define needs and definition of success.  By engaging nonprofit services providers,other community organizations, subject matter experts and funders, , the procurement cycle can leverage a collaborative sense of mission around what are the most significant goals and what success looks like,.

However, this also requires that nonprofit providers move away from simply complying with the requirement of various funding streams and focus on how a particular government initiative fits with their theory of change and expertise. In the end, this requires that each organization consider one another as valued partners to achieve outcomes for their residents and clients.

Examples:

• New York City Ecosystem: New York City has a variety of resources oriented around creating more collaborative relationship between government and nonprofits. The NYC Nonprofit Resiliency Committee provides for collaboration between New York City agencies and nonprofits. This collaboration has produced a variety of tools to increase cooperation between the two sectors including A Guide to Collaborative Communication with Human Service Providers, A Civic Design Guide (to improve the outcomes of government programs and contracting), a new procurement portal, and changes to other contracting policies (listed here).

• Bernalillo New Mexico Problem-Based Approach: Bernalillo County New Mexico, with support from the Government Performance Lab (GPL) at the Harvard Kennedy School, reformed its procurement practices to ensure the right services are reaching the right people. The county issued a problem-based RFP using the issue of “Adverse Childhood Experiences” (ACEs) as its motivating framework. This procurement approach is now being adopted more broadly throughout County government.     

Tool and Templates:

• Washington, D.C. Procurement Law

• Sample RFI

• Discussion Guide for Nonprofits and Government to Create a More Effective RFI and related Training Module

[In-development by RFA and Project Evident] for local government procurement officers to update their policies and practices to focus their contracts with human service nonprofit service providers on outcomes.  A request for information is one of the more efficient tools an agency has to collect information from potential providers. It could take the form of a survey or an open distribution of key questions that solicit written input.

Recommendation #2: Government should allow for funding flexibility and the braiding of  funding sources to permit human service providers to align programs and focus on holistic outcomes for their target populations.

Actions:

1. Governments should proactively remove budget silos across agencies and levels of government to meet residents’ needs more holistically.
2. Government should reflect these blended budget streams in contracts to allow providers to take a more holistic approach to providing services.
3. Government should review funding streams and ensure they are effectively meeting the needs of the target population.

Rationale:

A contract model that focuses on pay for outcomes attempts to align all stakeholders interests around creating the desired impact for the client. Focusing on outcomes, rather than outputs, requires that government provide flexible funding. Contract flexibility in turn, allows nonprofits to align programs and funding streams and focus on holistic outcomes for their target populations. Government can do this by allowing funds from multiple funding sources to be braided together, which reduces compliance technicalities. A further innovation would provide bonus payments to service provides when they exceed outcome goals.  Nonprofits also need to be prepared to use this flexibility to implement interventions that achieve the agreed upon outcomes and have the systems in place to measure their results towards these outcomes.

Examples:

• Houston Continuum of Care Programs Coordinate Funding: In Houston, the Continuum of Care (CoC) Program is designed to promote community wide commitment to the goal of ending homelessness. Integrated strategies among a group of 60 nonprofits and city and county agencies contributed to a 37 percent decline in the area’s homeless population between 2011 and 2014. Marilyn Brown, president of the Coalition for the Homeless of Houston and Harris County, which leads the area’s Continuum of Care, said “Integrated, community-wide strategies to prevent and end homelessness is the key to our success.’’

• San Francisco Breaks Down Silos: In 2017, the Government Performance Lab (GPL) at the Harvard Kennedy School provided pro bono technical assistance to help San Francisco improve its workforce development outcomes by better aligning services, funding, and contracting across departments. Details are highlighted on page 6 of the GPL ‘’Results Driven Contracting Solutions Book: How cities are improving the outcomes of high-priority procurements’’, but recommendations for improving system alignment included: conducting joint procurements, reducing service overlaps and gaps to make sure that the array of services corresponds to the needs of the population, and coordinating outreach to participants to match them to appropriate programs regardless of departmental entry point.

Tools and Templates:

• New York City Consolidated Funding Streams: In New York City, four city agencies (homeless, housing, health and public assistance) pool resources that formerly supported four separate contracts that were necessary to the successful implementation of a supportive housing project. A single contract issued on behalf of all agencies, consolidated into a single streamlined procurement all the components of the work.  The broader effort also aligned property disposition for development, and client assessment for placement into completed housing units. [Link TBD]

Federal Government Braided Funds through P3 Initiative: The federal government experimented with a braided funded approach in the Performance Partnership Pilot which allowed participating federal agencies to blend funds in support of localities which developed strategies to help disconnected youth, waiving many regulations and restrictions  on the use of the funds, in furtherance of achieving improved outcomes.

Recommendation #3: Government should develop RFPs that focus on outcomes, not outputs while incentivizing nonprofit human service providers to build and use evidence.

Actions:

1. Consider upcoming contracts
2. Define evidence and apply that definition to RFP requirements
3. Promote evidence based solutions and providers.
4. Update RFPs to include outcomes and other provisions, such as extra points for evidence, that shift dollars toward evidence-based, results-driven solutions and providers.
5. Community-based organizations should closely evaluate RFPs to develop program proposals that use (and implement with fidelity) evidence-based models.

Rationale:

Government should give service providers greater freedom and the tools necessary to meet outcomes, rather than focusing on reaching a narrow set of procedural milestones.. This requires contracts that are structured to pay for achieving long term outcomes rather than mandating specific service formulas.. In turn, nonprofits should focus on meeting outcome goals defined in collaboration with government (see #1 above). This requires that nonprofits build the programmatic and staff capacity to meet long term outcomes goals, rather than short term outputs. Striking the right balance here can be challenging.  The value of evidence based services, and the need to maintain fidelity to proven models suggest the wisdom of requiring proven program structures as part of a service agreement. However, local conditions vary and some freedom to adapt to reflect that knowledge can wisely incorporate flexibility into implementation.

To further improve outcomes, local governments should focus on funding interventions which have been shown to be successful at achieving the desired outcome  as well as supporting innovative ways to learn more about what works. This focus on interventions which have a track record of achieving outcomes will improve the impact of human service programs and ensure that taxpayer funds are achieving the greatest possible impact. In addition, government should incentivize nonprofits to build evidence of what works and use data to improve their models by giving preference points for proposals that use evidence-based strategies. To ensure ongoing innovation and an ever-growing evidence base, government should set aside funds in human service grants for data collection and evaluation. In sum, both nonprofits and governments should commit themselves to a culture of learning. In the long term this is the only way to improve the effectiveness of human services programs.

Examples:

• Requiring Evidence in Florida’s Department of Juvenile Justice: The Pew-MacArthur Results First Initiative highlights on page 4 of the ‘’How to Use Evidence in the Contracting Process Data and research can increase the efficiency and effectiveness of government programs’’, how Florida’s Department of Juvenile Justice requires the use evidence in the RFP process. The Office of Program Accountability monitors contracted providers using real-time data uploaded into its Juvenile Justice Information System, which shows whether a program is being implemented with fidelity to its model. DJJ also provides technical assistance for providers to support training on evidence-based program models.

• Santa Cruz, California Develops Shared Outcome Measures: The Santa Cruz Probation Department requires 100 percent of providers receiving grant funding to offer evidence-based programming. The Pew-MacArthur Results First Initiative highlights on page 7 of the ‘’How to Use Evidence in the Contracting Process Data and research can increase the efficiency and effectiveness of government programs’’, how the Probation Department works with service providers work to develop a common set of outcome measures which are tracked and reported quarterly along with submitting data for program evaluations and monitoring implementation to ensure fidelity.

Template/Tool:

• Sample RFPs

Future additions:

• Best practice sheet on evidence definitions [in-development with NFP]
• Model RFP [In-development through Smart Markets]

Recommendation #4:  Government should issue straightforward, effective contracts that align payment with outcomes.

Actions:

1. Government should  implement innovative contracting models, such as Pay for Success (PFS), outcomes rate cards, outcome bonus payments, that tie payments to the achievement of measurable outcomes.
2. Government should ensure that the contract reflects the evidence provisions, outcomes measures and identified community need.
3. Government should add provisions in contract to ensure appropriate and necessary data collection, integrate\ion and sharing across agencies and with service providers.
4. Government should consider multi-year contracts that incentivise working toward sustainable long term change.

Rationale:

Straightforward, fair contracts that align payment with achieving outcomes can increase the likelihood of achieving impact for residents and provide a clear framework for monitoring progress.  Government can do this by implementing innovative contracting models, such as Pay for Success (PFS), that ties payment for service delivery to the achievement of measurable outcomes. The contract should incentivize the long-term term change the government wants to see such as a  job training participant who finds and keeps a job, and experiences an increase in earnings rather that the number of students taught in a job training program. Learn more at Nonprofit Finance Funds’ Basic for Pay for Success site.

Both local governments and nonprofits can benefit from quicker processing of contracts and payment. For governments, completing contracting and payments in an efficient way allows staff to spend more time on monitoring for outcomes and avoids fiscal year expenditure deadlines that often do not align with programmatic cycles.  Further creating a central repository of grants, standardizing RFP language, implementing standardized contracts, using consistent contracting processes, and setting universal timelines for contracting would all lessen the bureaucratic burden for governments and nonprofits. One way to supercharge these efforts is to elevate the role of contract managers and procurement officers by providing them with standardized and user friendly tools, which allow them to spend more time on active contract management and less time on invoice review, data processing, and other technical compliance tasks.

Examples:

Contracting for outcomes Examples:

• Illinois Data-Driven Performance-Management: The Illinois Department of Children and Family Services is profiled on page 15 of the Pew Charitable Trust report, ‘’How States Use Data to Inform Decisions: A national review of the use of administrative data to improve state decision-making’’, for its use of a data-driven performance management approach. DCFS incorporates specific performance goals into service contracts with private providers that specify both incentives and penalties based on outcomes.

• Washington Education Collaborative: The Washington Education Research and Data Center brings together a dozen partners, including other state agencies and nonprofits, to compile education and workforce data to improve student achievement and workforce outcomes. The Center’s Memorandum of Understanding identifies how data will be collected and shared among partners with a strong focus on protecting individual privacy.

• The Pennsylvania Department of Corrections Incentive Structure: The Pennsylvania Department of Corrections is profiled on page 16 of the Pew Charitable Trust report, ‘’How States Use Data to Inform Decisions: A national review of the use of administrative data to improve state decision-making’’, for its use of data in the contracting processes. Community corrections programs that serve recently paroled or soon-to-be paroled inmates must meet performance targets based on the recidivism rates of their clients. Agencies with clients attaining a better-than-expected recidivism rate earn an increase of 1 percent in the departments per diem rate. Agencies with recidivism rates that are worse than expected for two consecutive contracting periods can have their contract terminated. Department officials credit this system for an 11.3 percent reduction in recidivism rates from 2014-15.

Tools and Templates:

• Sample Contracts

Recommendation #5: Government should use data and regular communication with providers to monitor implementation and progress (active contract management) to ensure contracts meet their outcome goals.

Actions:

1. Government should establish clear procedures for monitoring progress to meeting outcomes identified in contract, including at least quarterly data-driven conversations with providers to allow for an opportunity for course correction and program improvements.
2. Government should implement integrated data systems to power improved learning and provide consistent feedback to providers and policy makers.
3. The contract should ensure appropriate sharing and use of data, including data sharing agreements with nonprofit human service providers.
4. Government should share performance data of providers openly and providers should know how they are performing compared to their past performance and the performance of other providers under the same contract.

Rationale:

Collaboration is key in designing programs (as noted in number one above) but it is just as important for implementing effective programs. Closing the feedback loop, with governments soliciting and nonprofits providing frequent updates on program progress and interim goals, is the best way to improve coordination. Government contracts should include provisions for ongoing information sharing, including (but not limited to) data sharing agreements.  These data sharing agreements should outline the provisions for collecting, sharing and making decisions based on real time data. Governments and nonprofits can use dashboards, regular meetings, and site visits as tools to improve feedback mechanisms and improve their mutual efforts to ensure that human services reach desired outcome goals. To really power continuous learning and improve services government should create integrated data systems.

Example:

• Connecticut’s Data-Driven Conversations: The Government Performance Lab (GPL) at the Harvard Kennedy School provided pro bono technical assistance to help Connecticut’s Department of Children and Families (DCF) pilot improvements to the way the agency matches families to services. Connecticut’s DCF, among other innovative practices, is using collaborative, data-driven conversations between agency and providers to resolve problems with service delivery and identify opportunities for systems reengineering. This active contract management approach is being tested on intensive family preservation services across the state with the goal of expanding to other service types.

• Enhancing Workforce Development Services in Washington D.C:  To address the disproportionate employment rates across city wards, D.C., in partnership with Harvard Government Performance Lab as showcased in Harvard Kennedy School GPL: Results-Driven Contracting Solutions Book: “How cities are improving the outcomes of high-priority procurements”, employed results-driven contracting procedures to procure a new one-stop operator designed to connect unemployed residents with jobs. The city defined specific objectives for the one-stop operator but also invited vendors to explain how they were going to meet the identified goals and provide the outlined services (Rec #1). D.C. developed an RFP that defined key process and outcome metrics, rather than outputs, that will help them assess the performance of the one-stop operator and the overall workforce development system. Having established an active contract management approach, D.C. contract administrators will meet bi-monthly with vendors to review performance indicators and flag any coordination issues in real-time. These meetings will allow vendors and the District to brainstorm solutions together and foster a shared understanding of the program (Rec. #3 and #5). To incentive result rather than compliance, D.C. plans to renew contracts with vendors based on performance reviews, using A combination of process metrics and outcome metrics, and the vendor’s level of collaboration with the city.

Tools and Templates:

• Data Integration Toolkit
• Confidentiality Toolkit: In 2014, the Administration for Children and Families within HHS created a Confidentiality Toolkit designed to promote appropriate, confidential data sharing across human service agencies. The toolkit had two goals: 1) to help state and local leaders provide more effective services, and 2) to provide greater clarity to the rules governing confidentiality. While not required by statute, such technical assistance could be included in legislation. (seems like this should be in data sharing section.
• Data Sharing Across Agencies and Jurisdictions:Memorandum of Understanding (MOU) was signed between All Home, King County, and United Way of King County in 2016 committing the agencies to align the community priorities across the entire network and tie funding to outcomes that improve the effectiveness of the system.  The sharing of the data needed to facilitate this work was enabled by the Homelessness Partner Agency Privacy and Data Sharing Agreement. This agreement clarifies the rights and responsibilities of the Parties regarding access to and use of the HMIS data by the Partner Agency.  Seattle consolidated and streamlined the data collection process by using one system, the federally mandated Homeless Management Information System (HMIS) system, and collaborated with providers to increase data collection accuracy. This effort was enabled by the Washington Homeless Client Management Information System Law.
• Active Contract Management: Harvard GPL Active Contract Management has worked in NYC, Seattle, Rhode Island, and Illinois.
• Lab @DC Data Template Data Use Agreement April 2017-2
• Data Sharing Form: NYC created a simple form for internal data sharing
• AISP Resource: Legal Agreements and Supporting Documents

[Create example will be a sidebar]

Rhode Island Department of Children, Youth, and Families sidebar

Since March 2015, the Harvard Kennedy School Government Performance Lab (GPL) has provided pro bono technical assistance to Rhode Island’s Department of Children, Youth, and Families (DCYF) as part of a system-wide turnaround. To identify areas of budget overrun,  prior year invoices and service provider contracts were compiled into a single database and analyzed to project future cost and service trends. From there, more than 5,000 pieces of unique input were collected from DCYF frontline caseworkers, department supervisors, and direct care providers on service needs and historical gaps, referral and matching strategies, and potential contract structures for a new service array. Next, DCYF structured its service array around 15 outcome categories and linked those to specific performance objectives. DCYF completed a results-driven procurement process that resulted in 116 new contracts collectively representing approximately $90 million of services per year. Unlike conventional solicitations that request that vendors deliver a predetermined service model, the new procurement process asked providers to propose the services, supports, and resources that would best enable children and families to achieve any of the outcome categories prioritized by the department. The flexible nature of the solicitation leveraged the expertise of local experts and community providers to offer programs not previously considered by DCYF. New and innovative programs have proliferated in DCYF with results including a fifty-percent expansion of foster care resources for the most challenging adolescents, doubling the capacity of high quality family visitation and reunification services, and making start-up investments of $1.2M in nonprofit community organizations to support new and expanded programming. To institutionalize provider performance feedback loops, DCYF integrated small performance-based payment opportunities into all new contracts for family-based and residential services. Linking payment to DCYF measurement of administrative data ensures that throughout the duration of the multiyear contract providers will receive data from DCYF that will enable providers to learn what happens with their clients after they leave their care. This data will also help inform DCYF’s future referral and contracting decisions. Initially, DCYF piloted an active contract management system with four providers and has since expanded the process agency wide. Active contract management allows the DCYF program staff and service providers to track outcomes and quickly intervene if performance starts to drop. As a result, DCYF has reduced the number of children in group care by nearly twenty-percent, reduced the number of children entering state custody due to the improved performance of preventative services, dramatically expanded its portfolio of family-based services and supports, and strengthened the department’s financial controls and contract management practices.  

King County Case Study…Coming Soon

Seattle and King county partnership to reduce homelessness is a good example of how all these recommendations come together [Case Study Link TBD]