Why It Matters
Workforce agencies can evaluate service delivery strategies, learn which strategies are effective for their service population, and use that knowledge to shape future funding opportunities. As agencies learn which strategies are effective, they can require or encourage providers to use those strategies. This approach can be useful in cases where programs are newer or where few evidence-based strategies exist.
An evaluation is simply a systematic process to measure and understand the demonstrable effect of an action. Evaluation can occur before program launch (formative or theory-based evaluation), as a mechanism to understand the needs of the individuals who will be impacted. It can also occur during the program’s delivery to inform continuous process improvement (process/implementation evaluation). Finally, it can take place at program completion to understand the degree to which the program has met its intended goals (outcome or impact evaluation).
Overview
Any evaluation will begin by determining your research questions. Research questions can address both how the work was carried out, to inform future process changes or new program design efforts, as well as what results were achieved. The stage of the work and the target research questions will directly inform the type of evaluation chose
Process
- Was the intervention delivered as designed? Where there were deviations, what caused them?
- What are the strengths and weaknesses of the process?
- How accessible and accepted was the intervention approach?
- Which aspects of the intervention were most used?
- What considerations arose for organizational capacity, partnership or collaboration?
Outcomes and Goals
- What are the long-term effects of the intervention, both intended and unintended?
- What type of changes occurred in participant attitudes, beliefs, or knowledge as a result of their experience?
- What type of changes occurred in participant behavior?
- What was the cost relative to the benefits?
- To what extent did the intervention meet or exceed the stated goals?
- What external factors impacted the results for individuals?
Choose Your Approach
There are a variety of different types, or methodologies, of evaluations across a spectrum of complexity. A workforce agency should choose an evaluation approach that will help them answer the questions that are of greatest interest to the agency.
Timing is an important consideration for all evaluation types. If your intervention follows a specific set of steps each time, starts and ends on a specified date or requires a certain critical mass to meet its objective, it can be important to evaluate the intervention when these conditions have been met. If the service or intervention being evaluated is ongoing in nature (e.g., responding to customer queries) or providing differentiated services based on individual need, it can make sense to take a representative sample, both in terms of time and participants.
It’s important to note that the world of impact evaluation is continuously evolving. Workforce agencies should select the tool that best meets their needs, operational realities, available data sets and budgets. Over the course of a multi-year intervention, new evaluation approaches may become available or the additional data collected may make quantitative approaches more accessible. The most important consideration is to start incorporating evaluation into the procurement process as soon as possible. Agencies can add new approaches, or additional requirements, as they build evaluation capacity — both internally and among their service providers — and as the data is available. All of the evaluation approaches described here can provide valuable new insights into the work and shed light on opportunities for improvement.
Theory Based
- Realist Evaluations – Tests and refines the program theory before it’s launched, or while early in its delivery, based on initial data collection, needs assessment, or process evaluation. Generally looks at considerations around met and unmet needs of the target population, prioritization of the intervention and specific strategies that can be applied. Often focuses on determining if the approach is fit for purpose.
- Theory of Change (logic model) – Outlines a perspective or theory of how the project should work once operationalized. It describes a set of planned inputs and activities that should create a specific series of outputs, outcomes and impacts.
Such evaluations are often performed at early stages, while the work is still in progress. For example, Rapid-Cycle Evaluations (RCEs) allow an agency to test out tweaks to programs and services, offering data to iterate and improve programs more quickly instead of letting them run to their conclusion before making changes. For example, an agency might use an RCE to measure the impact of new outreach strategies on participant enrollment and might apply the findings of that RCE to ongoing recruitment efforts. This two-pager from Mathematica provides more information on RCE, including the types of questions it can help an agency answer and examples in workforce settings.
Qualitative Evaluation
Qualitative data from surveys, interviews or focus groups can help an agency understand how participants experience programs, and can offer valuable insights alone or in conjunction with quantitative data. For example, qualitative approaches can help shed light on the specific elements of a program that participants and/or employers found most helpful.
- Interviews provide an opportunity for one-on-one interaction and lend themselves to open-ended questions that can draw out details of the individual’s experience or perception. Their personal nature can help make individuals feel more at ease and enable an interviewer to address considerations such as English language fluency and cultural context.
- Focus groups provide small group conversations around a specific topic that are intended to create understanding, gather insights and foster connection. Generally focused on dialogue rather than specific data points. Participants can benefit from hearing from each other though care should be taken to ensure all participants have space to contribute ideas and that the focus group leader establishes trust and expectations for how information will be used up front.
- Surveys are useful to capture information from a larger group of individuals and can offer opportunities to understand their accompanying demographic makeup. Common types of surveys include (a) opinion and satisfaction surveys which measure views, attitudes and perceptions; (b) culture surveys which measure the point of view of employees/participants and are designed to assess whether it aligns with that of the program, the organization or its departments; and (c) engagement surveys which measure commitment, motivation, sense of purpose, and passion for experience or work.
- Case studies, often put together based on a series of interviews, focus groups, and research are another mechanism that can provide a more in-depth qualitative analysis of the intervention.
Urban Institute describes different approaches for evaluating workforce programs as part of its Local Workforce System Guide. Chicago Beyond’s Why Am I Always Being Studied? guidebook includes a chart (Page 37) showing different types of evaluation approaches and considerations for deciding which approach is right for an agency.
Quantitative Evaluation
- In experimental evaluation, a randomly selected group(s) receive an intervention or program and a randomly selected control group(s) does not. This approach, known as a randomized controlled trial (RCT), is a rigorous form of quantitative, causal evaluation that allows an agency to attribute outcomes to a particular intervention. RCTs can provide valuable information on how well a program is serving participants and can help advocate for more resources when it shows a program to be effective. For example, randomly assigning individuals who are experiencing homelessness to workforce services and comparing this group to those who do not receive services could shed light on the impact of workforce services in enabling an individual to obtain or retain stable housing. However, RCTs generally require high levels of financial investment and staff capacity, and the randomization needed for an RCT is not always possible due to funding, programmatic or policy considerations.
- Quasi- or non-experimental evaluation design includes a broad range of quantitative methods and are often used when randomization is not logistically feasible or ethical. These studies aim to establish a cause-and-effect relationship between a program and an outcome but without the use of randomization. Statistical or qualitative methods are used to account for potential differences between the group that benefited from the intervention and a similar group that did not. For example, because randomly assigning WIOA-enrolled participants to a control group that will not receive services might be politically or ethically problematic, other methods can be used to compare these participants with TANF participants who have similar characteristics but are not receiving WIOA services. Types of quasi-experimental designs include:
- Pre and post evaluation design is a simple technique that includes assessing the current state of the individual or process, applying the intervention, and then assessing the individual or process again to capture any changes. This approach can provide useful insights but care should be taken as it does not provide the necessary basis to determine cause and effect. For example, assessing an individual’s understanding of what makes a quality job, delivering a training on job quality elements and then assessing their understanding after the evaluation would create a pre and post evaluation.
- Interrupted time series design applies a set of repeated measurements before and after delayed implementation of the intervention as a means to help eliminate other explanations for the outcome. This can be applied to a single group or multiple groups. Taking multiple observations can improve the reliability of the results. Similarly, if an intervention includes multiple facets – education, transportation assistance, and childcare, for example – implementing one facet, allowing for a passage of time, and then adding in the next, can improve the evaluator’s ability to understand the impact of each facet of the intervention.
- Economic evaluation generally includes cost-benefit, cost-utility, cost-effectiveness, cost-minimization and/or cost-consciousness calculations to determine how to deploy resources to maximize desired impacts. For example, performing a Return on Investment evaluation on the cost of the training versus the financial benefit to the individual when entering the next job would provide an economic evaluation.
Mixed Method
Mixed methods are combinations of the other approaches listed.
- This approach can provide a more robust understanding of the intervention’s results.
- Can be performed in tandem or at different points during the process.
Resources
Examples
- The Colorado Department of Human Services’ (CDHS) 2018 ReHire Colorado RFP required all applicants to cooperate with the randomized controlled trial the department was using to evaluate the transitional jobs program: “In accordance with the Colorado Careers Act, the ReHire Colorado program will be subject to evaluation. This evaluation has taken the form of a randomized control trial. All selected vendors must agree to cooperate with this trial, at the discretion of the State, and ensure sufficient controls related to the evaluation model to protect the fidelity of the findings (see Exhibit F for an overview of the random control trial evaluation and anticipated responsibilities). Explicit guidance for the execution of the evaluation will be provided by the State.”
- The 2023 Supplemental Workforce Development Training Opportunity Grant RFP includes the following language: “In the event an evaluation of subaward outcomes is conducted, the subrecipient will be required to participate in the evaluation by providing requested data and information.”
- The Texas Workforce Commission (TWC) awarded a $200,000 contract to the Ray Marshall Center to conduct the evaluations in partnership with Building and Construction Trades program awardees. This initiative worked to build evidence by funding evaluations that assess participant outcomes, program implementation, and participant and employer feedback on the programs. TWC’s approach improves equity by providing access to evaluations that vendors otherwise may not be able to afford, leads to more opportunities for programs to show and improve their effectiveness, and creates pathways for evidence-building and advancing vendors up the evidence framework’s evidence tiers.
- Use these templates to embed evaluation requirements in your agency’s grants or contracts.
How to Incorporate What Works in Future Procurement Cycles
Once an evaluation has been completed, an agency will need to determine how to incorporate those findings into future grants and contracts. If an evaluation shows a program to be effective for a service delivery population, for example, an agency might require or encourage providers to use that approach in the future. They might gather providers, frontline staff, community groups, program participants, and other stakeholders in an agency and community to review the evaluation’s findings and determine a path forward. Urban Institute’s Data Walks guide can help to share evaluation findings with program participants, community members, agency staff, and providers.
Embedding Equity
Equity Considerations for Evaluation
Without an intentional focus on equity, evaluations may not provide a full and accurate picture of how a program is serving participants, and may perpetuate disparities. Workforce agencies can embed equity in evaluation by prioritizing evaluation approaches that empower workers and communities to be full partners in the research, including helping design the research, interpret data, and shape how findings are used. Using an equity lens in evaluation is critical to ensuring that evidence is built from comprehensive data that is appropriately collected and analyzed, that evaluations are inclusive of subpopulations, and that workforce agencies understand what works, for whom, and under what circumstances.
- Community-Based Participatory Action Research (PAR) is a research approach that engages community members as full partners at every stage of the research process, including shaping research questions and empowering community members to collect and analyze data. PAR disrupts historical power imbalances between researchers and communities, generating better research and knowledge. This approach recognizes that deep expertise lies within each community, making research done without community input incomplete and inaccurate. The Conservation Law Foundation’s PAR Field Guide includes guidance on setting up a PAR project, developing research questions and tools, and collecting and analyzing data.
- Chicago Beyond’s Why Am I Always Being Studied? aims to level the playing field between researchers and communities during evaluations, moving away from historical power imbalances and protecting against unintended bias in traditional research relationships. The guidebook includes questions and considerations for researchers, community-based organizations, and funders to ensure evaluations reflect community needs, goals, experience, and expertise.
- Urban Institute’s Community Voice and Power Sharing Guidebook offers practical advice on partnership building, community advisory boards, community-engaged survey development, and youth engagement. Urban has also created a guide for holding Data Walks, in which program administrators and service providers empower program participants and community members by sharing program outcome data or research findings with them. Data Walks allow participants and community members to ground the data in their lived experiences, shedding light on how and why programs and services are serving their community well or not.
Getting Started
How to Get Started
- Determine which grant or contact will be evaluated, the stage of the work (e.g. initial development, ongoing service delivery, completed project) and outline the research questions. Research questions may cover both process and outcome considerations.
- Select the desired evaluation approach, such as an RCT, interrupted time series analysis or regression discontinuity, that best fits the intervention, its maturity, the population receiving service and the available data.
- Assess whether in-house capacity exists or additional support will be needed to carry out the evaluation. If third-party support is required, initiate a procurement process to secure a vendor.
- For future contracts or grants, build evaluation requirements, as well as necessary funding, into the budget and the RFP. Set expectations for the role of the contractor and program participants, including required timing for data submission to facilitate the evaluation. Download these templates to embed evaluation requirements into your agency’s grants and contracts.
- Consider holding information sessions with contractors before launching the evaluation to address questions and set expectations for existing contracts that may not include explicit evaluation requirements,
- Determine how information from the evaluation will be shared with key stakeholders such as funders, service providers and participants, policymakers and the public. Keep in mind that this may require producing different versions or summaries of the evaluation results to speak to the respective stakeholder needs or interests.
- Establish desired feedback loops to gain insights from the evaluator while the effort is ongoing and to address potential issues that may arise, such as a lack of expected data or staffing challenges by the provider.
- Hire or deploy in-house evaluators to carry out the selected approach and produce the desired reports.
- Assess results and determine mechanisms for incorporating findings into ongoing or future projects. Consider how to incorporate necessary process changes to the way the evaluation is funded, timed or carried out.
Want more templates?
We have a full list of ready-to-use templates and language for each of the strategies