This brief is one in a series about the five key components of evidence-based policymaking as identified in “ Evidence-Based Policymaking: A Guide for Effective Government,” a 2014 report by the Pew-MacArthur Results First Initiative. The other components are program assessment, implementation oversight, outcome monitoring, and targeted evaluations.
For many years, policymakers and citizens have expressed concerns about the effectiveness of public programs. While efforts such as performance-based and zero-based budgeting have been helpful, these initiatives have often had only a limited impact on the way funding decisions have been made. Nonetheless, the importance of using taxpayer resources wisely has never been greater, particularly given the ongoing budget stresses facing many state and local governments.
The past decade has seen a dramatic increase in the availability of research that has rigorously tested the effectiveness of many social programs. Despite increased data on “what works,” many governments continue to allocate their resources based on anecdotes and inertia, funding the same programs year after year with limited knowledge of the outcomes they achieve. A better method, which some governments are beginning to implement, incorporates this information into an evidence-based approach to budgeting.
This issue brief, part of a series on evidence-based policymaking, highlights three key steps that governments can take to more effectively use data to inform the critical budget decisions they make each year:
- Create a comprehensive inventory of funded programs and assess the evidence of each intervention’s effectiveness.
- Require agencies to justify requests for new funding with rigorous research on program effectiveness.
- Embed evidence requirements into agency contracts and grants to ensure that research guides program activities.
The challenges to creating a better budget process
The traditional budget process often frustrates policymakers. It typically focuses attention on incremental changes to existing spending, such as increasing or decreasing funding for all agencies by 2 percent, while the impact of the programs administered by these agencies receives little scrutiny. Policymakers interested in using evidence to inform their budget decisions face two main challenges.
Existing budget processes reinforce the status quo.
Most state and local governments use an incremental budgeting process, meaning that the activities funded in next year’s budget will be very similar to the current year’s, with a few exceptions for new initiatives or priorities. Budget rules and processes tend to reinforce this dynamic by requiring that agencies justify the amount of funding requested, rather than the results their program can be expected to achieve. Few governments ask agencies to support funding requests by providing evidence of program effectiveness. When they do, it is typically for new programs or additional funding only, so a large majority of programs are not subject to these requirements.
Governments lack key information about the programs they fund.
Most governments do not gather important facts about the programs that are included in the continuation base budget. For example, most lack a comprehensive list of these activities, and few compile comprehensive data on their cost and performance. In the absence of information about the outcomes delivered by individual programs, it is understandable that policymakers resort to across-the-board cuts when required to make budget reductions, as the dearth of data hinders their ability to target cuts to underperforming programs.
Three steps to creating an evidence-based budget
Policymakers in both the executive and legislative branches can make more informed decisions by incorporating rigorous evidence into the budget process. Key steps include developing a complete inventory of currently funded programs, requiring agencies to justify budget requests with rigorous research, and mandating that agencies incorporate evidence requirements in the contracts and grants they administer. Collectively, these actions can help ensure that funding is directed toward programs that are most likely to achieve desired outcomes.
Step 1: Create a comprehensive inventory of funded programs and assess the evidence of each intervention’s effectiveness.
The first step in developing an evidence-based budgeting process is to create a comprehensive list of funded programs and assess the level of evidence that exists about the effectiveness of each intervention. This enables policymakers to understand what programs are currently operating in the jurisdiction and what level of confidence they can have that investments in these interventions are likely to produce desired results.
Inventory current programs. Budget documents typically provide limited, if any, information about the individual programs that are being delivered by agencies and providers. An inventory enables policymakers to look inside the base budget and examine the specific interventions that are being funded. Governments typically assign responsibility for this inventory process to a central unit that can oversee agency data collection efforts. If possible, inventories should include a description of each program and its goals, services being provided, target population and number of clients served, and per-client and total cost. The inventory creates a baseline of information that can then be used to match programs to the evidence base.
Assess the likelihood that programs are successful. In a perfect world, governments would conduct periodic evaluations of all programs that they operate to provide comprehensive information about the effectiveness of each activity. However, such evaluations are expensive and time-consuming. Governments typically can conduct only a few such studies each year. Fortunately, the effectiveness of many interventions has been rigorously studied across the country in recent years. This information about what works can be used to assess whether local programs that are based on the same design are likely to generate positive outcomes. For example, a program that has been evaluated in multiple jurisdictions and consistently found to be successful is highly likely to produce good results if implemented locally with fidelity and under similar conditions.
Staff can begin this process by examining national research clearinghouses, which review and aggregate rigorous studies and rate programs by their level of evidence of effectiveness. This information can be found through the clearinghouses’ online portals (see “Using Research Clearinghouses to Match Local Programs to the Evidence Base”). Staff can then compare the programs their jurisdiction is operating (contained in the inventory) to those rated by the clearinghouses—considering factors such as target population and delivery setting—to determine if their local programs are evidence-based. Additionally, some governments rank programs by definitions of evidence that they have established through rule or law,1 which can be helpful when classifying programs that are rated differently by clearinghouses.