This brief is one in a series about the five key components of evidence-based policymaking as identified in “ Evidence-Based Policymaking: A Guide for Effective Government,” a 2014 report by the Pew-MacArthur Results First Initiative. The other components are program assessment, implementation oversight, outcome monitoring, and targeted evaluations.

Overview

For many years, policymakers and citizens have expressed concerns about the effectiveness of public programs. While efforts such as performance-based and zero-based budgeting have been helpful, these initiatives have often had only a limited impact on the way funding decisions have been made. Nonetheless, the importance of using taxpayer resources wisely has never been greater, particularly given the ongoing budget stresses facing many state and local governments.

The past decade has seen a dramatic increase in the availability of research that has rigorously tested the effectiveness of many social programs. Despite increased data on “what works,” many governments continue to allocate their resources based on anecdotes and inertia, funding the same programs year after year with limited knowledge of the outcomes they achieve. A better method, which some governments are beginning to implement, incorporates this information into an evidence-based approach to budgeting.

This issue brief, part of a series on evidence-based policymaking, highlights three key steps that governments can take to more effectively use data to inform the critical budget decisions they make each year:

1. Create a comprehensive inventory of funded programs and assess the evidence of each intervention’s effectiveness.
2. Require agencies to justify requests for new funding with rigorous research on program effectiveness.
3. Embed evidence requirements into agency contracts and grants to ensure that research guides program activities.

The challenges to creating a better budget process

The traditional budget process often frustrates policymakers. It typically focuses attention on incremental changes to existing spending, such as increasing or decreasing funding for all agencies by 2 percent, while the impact of the programs administered by these agencies receives little scrutiny. Policymakers interested in using evidence to inform their budget decisions face two main challenges.

Existing budget processes reinforce the status quo.

Most state and local governments use an incremental budgeting process, meaning that the activities funded in next year’s budget will be very similar to the current year’s, with a few exceptions for new initiatives or priorities. Budget rules and processes tend to reinforce this dynamic by requiring that agencies justify the amount of funding requested, rather than the results their program can be expected to achieve. Few governments ask agencies to support funding requests by providing evidence of program effectiveness. When they do, it is typically for new programs or additional funding only, so a large majority of programs are not subject to these requirements.

Governments lack key information about the programs they fund.

Most governments do not gather important facts about the programs that are included in the continuation base budget. For example, most lack a comprehensive list of these activities, and few compile comprehensive data on their cost and performance. In the absence of information about the outcomes delivered by individual programs, it is understandable that policymakers resort to across-the-board cuts when required to make budget reductions, as the dearth of data hinders their ability to target cuts to underperforming programs.

Three steps to creating an evidence-based budget

Policymakers in both the executive and legislative branches can make more informed decisions by incorporating rigorous evidence into the budget process. Key steps include developing a complete inventory of currently funded programs, requiring agencies to justify budget requests with rigorous research, and mandating that agencies incorporate evidence requirements in the contracts and grants they administer. Collectively, these actions can help ensure that funding is directed toward programs that are most likely to achieve desired outcomes.

Step 1: Create a comprehensive inventory of funded programs and assess the evidence of each intervention’s effectiveness.

The first step in developing an evidence-based budgeting process is to create a comprehensive list of funded programs and assess the level of evidence that exists about the effectiveness of each intervention. This enables policymakers to understand what programs are currently operating in the jurisdiction and what level of confidence they can have that investments in these interventions are likely to produce desired results.

Inventory current programs. Budget documents typically provide limited, if any, information about the individual programs that are being delivered by agencies and providers. An inventory enables policymakers to look inside the base budget and examine the specific interventions that are being funded. Governments typically assign responsibility for this inventory process to a central unit that can oversee agency data collection efforts. If possible, inventories should include a description of each program and its goals, services being provided, target population and number of clients served, and per-client and total cost. The inventory creates a baseline of information that can then be used to match programs to the evidence base.

Assess the likelihood that programs are successful. In a perfect world, governments would conduct periodic evaluations of all programs that they operate to provide comprehensive information about the effectiveness of each activity. However, such evaluations are expensive and time-consuming. Governments typically can conduct only a few such studies each year. Fortunately, the effectiveness of many interventions has been rigorously studied across the country in recent years. This information about what works can be used to assess whether local programs that are based on the same design are likely to generate positive outcomes. For example, a program that has been evaluated in multiple jurisdictions and consistently found to be successful is highly likely to produce good results if implemented locally with fidelity and under similar conditions.

Staff can begin this process by examining national research clearinghouses, which review and aggregate rigorous studies and rate programs by their level of evidence of effectiveness. This information can be found through the clearinghouses’ online portals (see “Using Research Clearinghouses to Match Local Programs to the Evidence Base”). Staff can then compare the programs their jurisdiction is operating (contained in the inventory) to those rated by the clearinghouses—considering factors such as target population and delivery setting—to determine if their local programs are evidence-based. Additionally, some governments rank programs by definitions of evidence that they have established through rule or law,¹ which can be helpful when classifying programs that are rated differently by clearinghouses.

Using Research Clearinghouses to Match Local Programs to the Evidence Base

Despite the tremendous growth of research on the effectiveness of human services programs, this information is often hard for policymakers to locate and difficult to interpret. Evaluation findings are frequently published in academic journals that are available only through paid subscriptions, or are presented in highly technical reports. Busy policymakers and their staff do not have time to search for and interpret thousands of studies to identify what is known about the effectiveness of their agencies’ programs.

To help address this problem, many research clearinghouses have been established in recent years that compile and curate findings in social policy areas. However, these entities do relatively little to publicize their activities. They also use differing nomenclature to report results, making it difficult to interpret and compare their findings. As a result, government leaders have underutilized this information on what works.

The Results First Clearinghouse Database provides a single access point to information from eight national research clearinghouses that conduct independent, transparent, systematic, and rigorous reviews of available research.² The online searchable database helps users find, view, and interpret results from these clearinghouses by reconciling the different rating systems and vocabularies, and providing data in a clear, accessible format. States and localities participating in the Results First Initiative receive technical assistance on how to match funded programs to those in the database.

Once completed, this evidence assessment enables leaders to categorize each program from their inventory by the level of available evidence about its effectiveness, identifying which ones are highly likely to work, which have been shown to be ineffective, and which need more research. This information can help governments as they analyze current spending and prioritize requests for new funding.

For example, in 2012, the Wyoming Department of Health contracted with the Wyoming Survey & Analysis Center (WYSAC) at the University of Wyoming to create an inventory of substance abuse prevention strategies focusing on changing aspects of the environment that contribute to the use of alcohol, tobacco, and other drugs. The inventory ranks each strategy according to the evidence supporting its effectiveness and highlights whether the strategy was implemented in the state.³ The report’s purpose was to provide information to community prevention professionals charged with implementing programs aimed at reducing alcohol, tobacco, and other drug use and abuse within their communities. “The county-level prevention workers are highly skilled but don’t have the time or resources to wade through all of the research on what’s effective and what isn’t,” said Humphrey Costello, an assistant research scientist at WYSAC and one of the authors of the report. “The goal with this report was to provide information on the evidence of program effectiveness in a straightforward way.”

The Wyoming program inventory has become a valuable tool for community coalitions engaged in making decisions about which strategies to implement to address critical needs in their communities. “The program catalogue gives us a framework for how decisions can be made,” says Brittany Ritter, a community prevention professional from Lincoln County and a member of the Prevention Management Organization of Wyoming. “We want to know that we are implementing the strategies that the research shows to work, and [the program inventory document] helps us do that.”

The Mississippi and Washington legislatures have passed laws requiring agencies to inventory their current programs and categorize each by defined standards of evidence. For example, in 2014, Mississippi mandated that the state’s corrections, health, education, and transportation agencies report program inventories, which staff then categorized into four groupings—evidence-based, research-based, promising practices, and lacking proof of effectiveness.⁴ Legislators plan to use these results to inform future budget decisions and reinvigorate the state’s performance budget system.

This approach has also been used effectively by local governments. Santa Barbara County in California surveyed its jail- and community-based service providers to develop a comprehensive list of the criminal justice treatment programs being delivered and to determine the extent to which evidence-based practices were being used. This analysis revealed that many of the interventions were not closely aligned with evidence-based practices, prompting the county to offer training and technical assistance to providers to help them improve their current activities or replace them with alternative evidence-based programs.⁵

Analyze whether programs’ benefits outweigh their costs. Cost-benefit analysis, which compares a program’s expense to the value of its expected benefits, can enable policymakers to assess whether programs achieve returns on investments that justify continued funding. While some cost-benefit analyses examine a single program, a growing number of governments are using the approach to compare the relative return on investment of portfolios of alternative interventions and target funding to those that are predicted to achieve the best outcomes. Several states, including Colorado, Iowa, Massachusetts, New Mexico, New York, Rhode Island, and Washington, have adopted this practice to analyze their investments in human services and rank programs by their predicted return on investment.⁶

For example, in 2014, Massachusetts completed an analysis of the predicted returns on investment and reductions in recidivism achieved by its corrections, probation, and parole programs. After examining this report, policymakers approved a competitive grant program to expand evidence- and research-based programming to reduce recidivism.⁷ In 2015, Colorado’s Office of State Planning and Budgeting released a report analyzing the benefits and costs of programs in adult criminal justice, juvenile justice, and child welfare. It plans to use these results to inform funding and policy decisions in the upcoming budget cycle.⁸

Use outcome data to determine whether programs are achieving expected results. Even the best evidence-based programs, if implemented poorly by agencies, will fail to deliver good results. Accordingly, governments should establish performance goals that measure the outcomes of all major programs and report this information during the budget process. For those that have been rigorously tested, outcome measures can gauge whether programs are achieving the results predicted by the research. For programs that lack rigorous research on their effectiveness, outcome data are still important for identifying trends, such as whether performance is getting better or worse. These trends can lead to questions during budget deliberations about whether alternative approaches should be considered.

Governments should make certain that measures are agreed upon by members of both the legislative and executive branches and that data are verified or audited to ensure accuracy and reliability. Maryland, for example, provides detailed requirements to agencies for developing and reporting outcome measures as part of the governor’s budget request, including developing controls and procedures to verify that data are accurate and reliable.⁹ New Mexico requires both the legislative and executive branches to agree on which outcome measures will be reported, as well as any changes to the measures, and has protocols for ensuring that data are verified for accuracy.¹⁰

State and local governments should also set performance targets as a way of determining whether programs are exceeding or falling behind in meeting their goals. Where feasible, research on expected results when a program is properly implemented should inform these targets. For example, an evidence-based home visiting program for new mothers—predicted to reduce the frequency by which children must be removed from the home by 10 percent—should be compared against baseline rates to determine its success rate. When research is not available, governments may utilize other benchmarks that allow for meaningful comparison, such as the performance in similar states or jurisdictions, or national standards. Virginia, for example, compares the performance of many state programs with the national average, the top-performing state in the nation, and three states similar to itself.¹¹

Step 2: Require agencies to justify requests for new funding with rigorous research on program effectiveness.

In addition to creating an evidence baseline on existing programs, policymakers can require requests for new or additional funding to be justified with rigorous evidence. Governments can also compel agencies to give priority to evidence-based programs when implementing their budget allocations.

Require agencies to incorporate evidence into budget requests. State and local governments can instruct agencies to justify requests for new or increased funding with rigorous evidence showing that the additional resources would produce positive results. Governments may choose to gradually phase in these requirements over time so that they apply to currently funded programs as well as to new programs. Two states are becoming exemplars in this approach.

In 2015, the Mississippi Legislature began requiring agencies seeking funding for new programs through the appropriations process to answer a series of questions to ensure that proposed initiatives adhered to key elements of high-quality design. For example, agencies must report whether a requested program has “an evidence-base, research base, promising practice or best practice type model”; must describe the monitoring system that will ensure implementation fidelity; and must explain how they will measure results.¹² During the fiscal year 2016 Regular Session, the newly created Performance Based Budgeting Committee of the House of Representatives expanded these requirements to include some existing programs operated by certain agencies. In some cases, vendors who contract with the state are also subject to this review.

As part of the annual appropriations process, state agencies in Mississippi submit their requests for funding to the Legislative Budget Office. During the 2016 Regular Session, staff from the Joint Committee on Performance Evaluation and Expenditure Review (PEER) reviewed agency requests for new program funding using the elements of quality design to determine whether they met the evidence standards. Preference was given to evidence-based programs that have been proved effective in addressing a particular problem.􀀀However, given the limited number of submissions for new funding that met the evidence standard, the Legislature provided limited funding for some non-evidence-based programs with the caveat that rigorous research on their effectiveness would be conducted during fiscal 2017. To facilitate this, PEER is developing standards for what constitutes a rigorous research evaluation.

“Developing skill for evidence evaluation—among state agencies and any other applicants for public funds—is an important next step in the process,” said Kirby Arinder, a PEER staff member. “As those parties become better at reviewing evidence in a disciplined fashion, they can help filter out all but the most evidentially supported, effective uses of public dollars.”

Utah has taken a similar approach by embedding evidence requirements into its state budget instructions. As part of Governor Gary Herbert’s effort to improve government performance by at least 25 percent, the Office of Management and Budget has required agencies to provide evidence of program effectiveness and performance. This may include recent performance data compared with industry standards or other relevant benchmarks, or more rigorous forms of evidence including program evaluations. Agencies must propose a business case for why they need new funding and describe whether the activity is an evidenced-based practice or otherwise supported by research, data, evaluation, or professional standards.¹³

“Agencies are coming from different starting points and have different experiences and levels of confidence in using data,” said Kristen Cox, executive director of the Governor’s Office of Management and Budget (GOMB) in Utah. “GOMB uses a consistent measurement approach that shows the relationships between cost, volume, and quality measures that provide more immediate feedback than some that take years to determine outcomes.”¹⁴

Gov. Herbert’s 2016-17 budget identified 35 programs that would receive one-time funding but would be required to provide additional evidence of effectiveness to receive continued support beyond 2017.¹⁵ “We turned down a lot of requests this year because agencies didn’t have the data to support their requests,” Cox said.¹⁶

Training Budget and Policy Staff to Use Evidence

Incorporating evidence into the budget process often requires additional training for both executive and legislative branch staff. A recent survey of 928 state and local government officials found that 76 percent of respondents said data-driven decision-making was a major challenge. Only 44 percent said they believe employees at their organization have the training needed to make data-based decisions.¹⁷ Such training may be particularly important for the budget, research, and audit staff who often are charged with supporting evidence-based budgeting efforts. When equipped to do so, these units can play a vital role in the process.

One example is the New Mexico Legislative Finance Committee (LFC), which regularly issues evaluation reports, cost-benefit analyses, and quarterly performance reports on programs in a variety of policy areas, including adult criminal justice, child welfare, and early childhood education. Budget guidelines issued by the LFC require that analysts, in developing their recommendations, utilize performance data and rigorous research to prioritize funding to programs shown to be effective or that achieve the best return on investment for taxpayers.¹⁸

“We use evaluations to identify the problem areas in the state and use benefit-cost analysis to show how much it is costing and what programs can deliver promising outcomes at higher returns,” said LFC Deputy Director Charles Sallee.¹⁹

Require agencies to prioritize funding to evidence-based programs. Policymakers can also use legislation or administrative policy to require agencies to give preference to evidence-based programs when implementing their funding allocations. These mandates are typically phased in over time, requiring agencies to increase the percentage of funding on evidence-based programs up to a certain threshold.

In 2003, Oregon passed S.B. 267, which required five state agencies to gradually increase the amount of funding allocated to evidence-based programs for drug and alcohol treatment, mental health treatment, adult recidivism prevention, and juvenile crime prevention. Every two years, the agencies were required to report the percentage of their total budgets allocated to evidence-based programs. Beginning in the 2005-07 biennium, the five agencies had to demonstrate that at least 25 percent of their funds were used to support evidence-based programs. That increased to 50 percent by the end of 2007-09, and 75 percent by the end of 2009-11. By 2012, all agencies reported that they had achieved these targets.²⁰

“I don’t think we would have gotten the engagement from the state agencies [to target funding to evidence-based programs] without the law,” said Paul Egbert, operations manager with the Oregon Criminal Justice Commission.²¹ “The legislation itself, judging on its own merits, drove practices that were evidence-based and that the state sorely needed.”

In Tennessee, the state passed law § 37-5-121 in 2007, mandating that by 2012-13 the Department of Children’s Services fund only evidence-based programs that have been proved to reduce juvenile delinquency.²² “There was a sincere desire to be good conservators of state dollars, providing the very best services that had research behind them,” said Debbie Miller, deputy commissioner for the Office of Juvenile Justice.²³ The state created definitions of evidence and partnered with the Peabody Research Institute at Vanderbilt University to evaluate the extent to which providers were implementing programs with fidelity to evidence-based principles.

Agency officials in Tennessee credit the law’s focus on program implementation—ensuring that programs are carried out according to research-based methods—for having a positive impact. “You can set up contracts with providers and say we want you to provide evidence-based services, but you must follow through and make sure those programs are being implemented with fidelity, and the law requires that,” said Elvie Newcomb, special projects manager with the Office of Juvenile Justice.²⁴

Finally, some states have created similar requirements through agency policies or directives, rather than through legislation. In New Mexico, the Corrections Department created an administrative policy in 2015 to prioritize funding toward evidence-based programs. The policy, which is intended to provide guidance and set expectations for the delivery of services across the state, requires the department to spend at least 70 percent of funds on programs with rigorous evidence of effectiveness.²⁵

Step 3: Embed evidence requirements into agency contracts and grants to ensure that research guides program activities.

Many decisions on how governments allocate funding to individual programs occur outside the formal budget process. Agencies often have significant flexibility in spending funds within their legislative budget allocation, and they frequently contract with private and nonprofit organizations to deliver services. State and local governments can use these grant and contracting processes to support evidence-based policymaking principles by requiring agencies to prioritize funding to programs that have strong proof supporting their effectiveness and to restrict funding of programs lacking that data.

Give preference in competitive grant programs to proposals that demonstrate evidence of effectiveness. State or local governments can refine contracting processes to incentivize providers to use research and evidence-based programs. Administrators can work with providers to educate them about these new requirements, and identify and resolve potential administrative barriers to using evidence-based programs. Administrators can also consider building in requirements for overseeing program implementation, outcome monitoring, and evaluation.

In 2014, New York’s Division of Criminal Justice Services (DCJS) awarded $5.1 million through a competitive request for proposal to support targeted, evidence-based alternatives to incarceration for offenders at a high risk of recidivism.²⁶ The grant program meant to generate long-term state and local budget savings by reducing prison and jail populations, reducing recidivism, and avoiding future victimizations. To be considered for funding, grantees were required to identify the specific evidence-based model that they would implement and agree to ongoing oversight and evaluation of outcomes. Since 2014, the competitive grant program has continued to grow, and the DCJS has also expanded provider requirements to increase program alignment with proven, evidence-based practices and has increased the agency’s capacity to perform research that gauges program effectiveness.

“The beauty of this approach is that it is very open and transparent,” said DCJS Executive Deputy Commissioner Michael C. Green. “Everything that we fund must meet a high standard, and providers receive the tools they need to meet that standard. There are no surprises.”²⁷

Federal Efforts to Support Evidence-Based Grant-Making

In recent years, the federal government has increasingly emphasized the use of evidence in funding decisions and has created several new evidence-based programs focused on improving outcomes for children, youth, and families. These include the Maternal, Infant, and Early Childhood Home Visiting program, which requires grantees to direct 75 percent of federal dollars to evidence-based programs and to evaluate the impact on key outcomes ($400 million in funding annually through fiscal 2017); the Investing in Innovation Fund, which prioritizes education programs that show strong evidence of effectiveness and evaluation of innovative programs ($120 million in funding for fiscal 2015); and the Social Innovation Fund, which provides competitive grants that focus on evidence-based interventions in areas of economic opportunity, health futures, and youth development ($70 million in funding for fiscal 2015).*

The federal government has also helped to support several pay for success initiatives through the Department of Labor’s Workforce Innovation Fund. With this innovative financing model, independent investors provide financial capital to cover program operating costs, disbursing payment only when those programs demonstrate that they have achieved the targeted outcomes. In 2013, the department issued two grants totaling $24 million to the New York Department of Labor and the Massachusetts Executive Office of Labor and Workforce Development to support programming that aims to increase employment and reduce recidivism among formerly incarcerated individuals.

Most recently, in March 2016, Congress passed legislation establishing a commission on evidence-based policymaking, which will conduct a comprehensive study of the data inventory, infrastructure, and statistical protocols related to federal policymaking and recommend how the data could be used in cost-benefit analyses, impact evaluations, and other policy-relevant research, all of which can be used to inform federal efforts to fund effective programs.^†

* Results for America, “Federal Evidence-Based Innovation Programs” (October 2015), http://results4america.org/wp-content/uploads/2015/10/Innovation-fact-sheet.pdf.

^† Justin Milner, “Everything You Need to Know About the Commission on Evidence-Based Policymaking,” The Urban Institute (April 2016), http://www.urban.org/urban-wire/everything-you-need-know-about-commission-evidence-based-policymaking.

Build evidence requirements into provider guidelines. Agencies can also incorporate policies requiring evidence-based programs into the performance standards they issue for program providers. These standards are included in service contracts and typically address a wide array of requirements, including defining and describing acceptable levels of services. The provider guidelines therefore provide an opportunity to disseminate expectations for the delivery of evidence-based services. Several states, including Hawaii and New York, have embedded requirements to use evidence-based programs or practices in their provider guidelines.

In 2012, Hawaii’s Department of Health created a set of performance standards for child and adolescent mental health services, which prioritizes the use of evidence-based practices for organizations that contract with the department.²⁸ The standards define evidence-based practices according to four tiers that providers can use to inform the types of services they offer, though providers are urged to give higher priority to more reliable or stronger forms of evidence in making treatment decisions. Contracted providers are required to comply with the guidelines as well as any additional requirements identified in their contracts, unless granted a written waiver from the Child and Adolescent Mental Health Division (CAMHD). “[Over the years] we’ve modified the performance standards to reflect the need to use evidence-based practices,” said Stan Michels, CAMHD administrator. “The standards help provide consistency in the quality of services provided across each island.”²⁹

In New York, the Office of Alcoholism and Substance Abuse Services (OASAS) has developed provider guidelines for organizations that contract with the state to supply prevention services, using the standards as a baseline to monitor program performance and contract compliance.³⁰ A key component of OASAS’ Prevention Framework is the utilization of evidence-based programs and strategies. The guidelines reinforce this by requiring providers to spend an increasing percentage of their grant funding on these proven strategies. (The minimum was 35 percent by 2011 and increases by 5 percent each year to 70 percent by 2018.) “The standard allows for a percentage of non-evidence-based programs as well, which gives agencies the flexibility to be creative in developing services that are responsive to specific community needs,” said Mary Ann DiChristopher, acting associate commissioner for OASAS.³¹

OASAS has also created a registry of evidence-based programs and strategies that is available to providers to offer guidance for selecting the appropriate option. An advisory panel of prevention scientists regularly reviews research on effective practices, including those submitted by providers to identify new programs or strategies that meet the criteria of the registry as evidence-based. “The science has demonstrated that the use of evidence-based programs and strategies has led to positive outcomes for youth and families … and New York wanted to be a leader in this area,” OASAS Commissioner Arlene González-Sánchez said.³²

Conclusion

Policymakers are increasingly interested in using data to guide important decisions on how to allocate scarce taxpayer dollars to achieve better outcomes. While previous efforts to reform the budget process were met with only limited success, the growing availability of rigorous evidence about what works holds the promise of enabling governments to design more effective budget processes and make more informed investment choices. To do this successfully, governments will need to develop a comprehensive baseline of information on the effectiveness of currently funded programs, re-engineer the budget development process to encourage agencies to use rigorous evidence, and provide guidance to agencies on how to incorporate evidence requirements into the contracts and grants they administer.

This brief is one in a series about the five key components of evidence-based policymaking, as identified in Evidence Based Policymaking: A Guide for Effective Government. The other components are program assessment, implementation oversight, outcome monitoring, and targeted evaluation.