Step 2: Create policies and processes that support effective implementation and monitoring.
To scale up evidence-based programs, governments must develop the management infrastructure needed to facilitate effective program implementation. This includes creating standards or guidelines embedding these standards into contracts, and aligning them with administrative policies and processes to support effective implementation.
Develop implementation standards. Policymakers can establish common standards or guidelines for program implementation to ensure that providers meet a minimum level of competency in delivering services. Although some requirements will vary depending on the specific evidence-based intervention, other aspects of program implementation are universal and can be embedded into these standards. These may include minimum requirements for hiring and training staff, providing services to the target population specified by the evidence-based program provider, and ensuring that processes are in place to provide effective oversight of service delivery.
For example, leaders in Washington state developed standards to implement evidence-based juvenile justice programs after an evaluation found that sites where programs were not implemented with fidelity had poor results. These standards govern four key elements of quality assurance—program oversight, provider development and evaluation, corrective action, and ongoing outcome evaluation—and include protocols for hiring, staff training and assessment, and management and oversight of service delivery. Providers are required to complete an initial probationary period during which they receive training and feedback, and are then periodically evaluated. These implementation standards helped the state achieve greater reductions in crime and juvenile arrest rates, compared with the national average, and a decrease of more than 50 percent in the number of youth held in state institutions.15
Embed implementation standards or requirements into contracts. Agencies can build these standards into contracts to ensure that providers meet the required baseline levels of proficiency. In 2013, for example, New York state’s Division of Criminal Justice Services (DCJS) issued a request for proposal for alternatives to incarceration that required providers to identify the specific evidence-based interventions they planned to implement, provide detail on their screening and referral systems, and describe how they would adhere to the programs’ treatment protocols. The division now monitors providers’ fidelity to these requirements as part of a comprehensive process through which providers submit case-specific data to DCJS and undergo on-site reviews by third-party monitors contracted by the state. The reviews assess the degree to which programs are implementing principles of effective correctional interventions.16
Some agencies have embedded requirements related to implementation fidelity in their provider guidelines, which often cover a broad range of contracted services. In 2014, New York’s Office of Alcoholism and Substance Abuse Services updated its provider guidelines for prevention services, defining the strategies and activities necessary to reduce underage drinking, alcohol misuse and abuse, illegal drug abuse, medication misuse, and problem gambling.17 The guidelines require providers to implement programs with fidelity to the “core elements” of evidence-based services, including the target population, setting, and curricula content.
Align administrative policies and processes to support effective implementation. Implementing evidence-based programs often requires changes throughout service delivery networks. Existing administrative processes should be aligned with these delivery efforts. Otherwise, agencies and providers can face conflicting mandates or inflexible payment systems that make it difficult to effectively deliver critical services. Policymakers and agency leaders can help by creating feedback loops that enable administrators, providers, and technical assistance staff to regularly share information and solve unanticipated problems.18
For example, the Colorado Department of Corrections recently adopted a new integrated case management system to improve its planning and offender treatment services. In doing so, the department found that certain policies were not in alignment with the research on what works regarding low-risk offenders. Specifically, the research indicated that less contact with low-risk offenders leads to better outcomes. The department resolved the issue by facilitating changes to administrative regulation standards regarding the frequency of contact to better align policies and practices with the literature.
Step 3: Support service providers and staff through training and technical assistance.
Training and technical assistance are critical to implementing new interventions and practices. Program staff need to be trained on specific treatment protocols. Research shows that such training is most effective when delivered in multiple stages, including initial learning sessions followed by observation and feedback by experts, with subsequent ongoing in-service training and coaching once the program is up and running.19 Policymakers can support this process by funding and establishing systems that train staff on the delivery of evidence-based programs and practices; agency leaders can choose between several options for the delivery of this training.
It is particularly important for program administrators to ensure that staff are appropriately trained to use screening and assessment tools designed to help match participants with the appropriate interventions. Even the most widely replicated evidence-based programs are effective only when treating certain populations. Without the appropriate screening and assessment tools, agencies may refer participants to programs they do not need and that are not effective in addressing their problems. “We often hear frustration from agencies who tried evidence-based programs but still didn’t achieve the outcomes they sought because the programs weren’t delivered to the right population,” said Ilene Berman, senior associate with the Annie E. Casey Foundation’s Evidence-Based Practice Group.20
Determine the best vehicle for delivering training and technical assistance. Governments have several options for delivering training on evidence-based programs, such as using in-house personnel with expertise in these programs, contracting with program developers, or partnering with intermediary organizations. Some widely adopted programs, such as Multisystemic Therapy and Nurse-Family Partnership, offer training services to governments that implement them. Such program developers have deep expertise in their interventions and often have detailed training curricula. However, relying exclusively on program developers can limit an organization’s ability to develop its own expertise and may complicate training if agencies are implementing multiple evidence-based programs. Other options include leveraging the expertise of local researchers through a government-university partnership, or developing an evidence-based unit within a government agency.
Partner with a research university
Several states—including Maryland, Pennsylvania, and Washington—have established partnerships with research universities to provide training and technical assistance to staff and providers. These implementation centers can help support community readiness assessments, provide training and technical assistance on evidence-based programs, and oversee monitoring and quality improvement efforts.21 For example, the Institute for Innovation and Implementation at the University of Maryland was established in 2005 and is funded to provide training, implementation support, and evaluation services for select evidence-based programs across multiple policy areas, including juvenile justice and child welfare. The institute also provides technical assistance and project management support to state agencies engaged in statewide initiatives. “It’s important to have multiple state and local agencies on board as well as the provider community. … Collaboration across agencies is important in order to coordinate existing efforts, develop new strategies, and make sure that everyone is getting the same information,” said Jennifer Mettrick, director of implementation services at the institute.22
In Pennsylvania, the Evidence-based Prevention and Intervention Support Center, or EPISCenter, provides technical assistance to communities and service providers to support the implementation of evidence-based prevention and intervention programs. Since 2008, the center—a partnership between the state Commission on Crime and Delinquency and Pennsylvania State University, with funding from the commission and from the state Department of Human Services—has assisted in the establishment of nearly 300 evidence-based program replications in more than 120 communities throughout the state. Experts from the center provide technical assistance to local staff on implementation, evaluation, and sustainability, and help develop the infrastructure to monitor the program for fidelity to its original design.
Develop an evidence-based unit or division
Another option used by some states and localities is to establish specialized units within agencies that are charged with providing training as well as overseeing program implementation. These partnerships and units can help governments develop in-house expertise for a range of programs.
For example, Colorado’s Evidence-Based Practices Implementation for Capacity (EPIC) Resource Center is a collaborative effort of five agencies working in the state’s adult and juvenile justice systems. The center was created by the Colorado Commission on Criminal and Juvenile Justice in 2009 and formalized through legislation in 2013.23 Housed in the Division of Criminal Justice within the Department of Public Safety, the nine-person staff provides assistance to support effective implementation of evidence-based practices.