Implementation of Evidence-Based Practices: A Brief Background
In the past two decades, the field of children’s mental health has seen a shift from “care as usual” towards the use of evidence-based practices to treat mental health problems in youth. Often used alongside language such as “dissemination” and “implementation,” evidence-based practices, or EBPs, are defined as specific strategies, practices, or programs that have been empirically studied or evaluated using a rigorous research design. Results of such studies demonstrate the superiority of the EBP over “care as usual” in treating specific mental health problems in a target population (Drake et al., 2001; Hoagwood, Burns, Kiser, Ringeisen, & Schoenwald, 2001).
Developed and tested in academia, one of the biggest challenges facing EBPs, particularly in children’s mental health, is successfully transporting them from the university setting into routine community practice settings (Hoagwood & Olin, 2002). As the use of EBPs grew in popularity, so did the field of dissemination and implementation (D&I). Dissemination refers to the spreading of information about a particular practice, whereas implementation has been defined as “the use of strategies to introduce…evidence-based health interventions within specific settings” (U.S. Department of Health and Human Services, 2005). Both are necessary in the transportation of interventions, but the focus of this chapter is implementation – the work required to install an EBP into a community setting.
Research in the field has begun to identify specific strategies that appear to be commonly utilized across implementation efforts, including in-person trainings, ongoing consultation for training participants, fidelity monitoring to the EBP (or treatment adherence and competence), and evaluation of client outcomes (McHugh & Barlow, 2010). A number of projects are currently employing these strategies to implement psychological EBPs at both the state and national level. Since 2005, one such program – called Ambit Network – has been successfully implementing a trauma-focused EBP in community mental health agencies across Minnesota.
Implementation in Minnesota: Ambit Network
Housed in the College of Education and Human Development at the University of Minnesota, Ambit Network is community-university partnership of government, non-profit, and community mental health agencies throughout Minnesota. Funded in part by the Substance Abuse and Mental Health Services Administration’s (SAMHSA’s) National Child Traumatic Stress Network and the Minnesota Department of Human Services (MN-DHS), the mission of Ambit is to improve the systems of care that provide mental health services to children and families affected by traumatic stress through the implementation of EBPs into community agencies. To do this, Ambit selected EBPs on the basis of their appropriateness for highly-stressed families, including children experiencing homelessness and domestic violence, refugee and immigrant children, and children in military families. Using the universal implementation strategies identified by McHugh and Barlow (2010) to train the mental health workforce, Ambit Network is able to increase access to empirically-supported treatments and raise the standard of care provided to traumatized children in Minnesota.
Below, we review the implementation across Minnesota of one such EBP: Trauma-Focused Cognitive Behavioral Therapy (TF-CBT). TF-CBT was developed as a time-limited treatment for trauma and post-traumatic stress symptoms in children who have experienced trauma (Cohen, Mannarino, & Deblinger, 2006). A number of studies have documented the effectiveness of TF-CBT in treating not only internalizing symptoms (e.g., depression, anxiety), but also externalizing symptoms of trauma (e.g., anger, inappropriate sexual behavior) in children between the ages of three and 18 (Cohen & Mannarino, 1996, 1997, 1998; Cohen, Mannarino, & Knudsen, 2005; Deblinger, Lippmann, & Steer, 1996; Deblinger, McLeer, & Henry, 1990; Deblinger, Stauffer, & Steer, 2001; Deblinger, Steer, & Lippmann, 1999; King et al., 2000). When compared to “care as usual,” TF-CBT was found to be superior in reducing post-traumatic symptoms immediately post-treatment and in maintaining these improvements one- and two-years after treatment has ended.
To train providers in TF-CBT, Ambit adapted the Institute for Healthcare Improvement’s Breakthrough Series Learning Collaborative training model. The learning collaborative is a long-term training model in which trainees participate in in-person “learning sessions” and out of the classroom “action periods” (Institute for Healthcare Improvement, 2003). Over the course of six to 15 months, trainees participate in two to three learning sessions that alternate with action periods lasting between three to five months. By alternating learning sessions with action periods, trainees can learn the EBP during learning sessions lasting one to two days, and then return to their respective agency during action periods to implement and incorporate the EBP into their daily practice. By the end of the learning collaborative, trainees and their agencies have worked together to build capacity to sustain the EBP long-term and begin the process of implementing the practice across the agency.
The Challenges of Implementing a Long-Term, State-Wide Training Program
Most professionals would agree that learning a new practice or protocol, at the individual or even at the agency level, is difficult enough. From the beginning, Ambit faced the unique challenge of training and supporting large numbers of trainees in multiple agencies across Minnesota - all of whom belonged within different organizational structures that required individualized ways to implement TF-CBT into their organization. Since Ambit’s learning collaborative model only allows for five to six days of face-to-face time with trainees, the majority of the work involved with implementation – consultation, monitoring, and evaluation – had to be done remotely and communicated electronically, as trainees could be as close as a ten-minute drive to as far away as the Canadian border.
At its core, the learning collaborative training model for TF-CBT consists of the following three components to be delivered: in-person trainings, consultation calls, and a practicum period with technical support and assistance from Ambit.
In-person trainings with a certified TF-CBT trainer. These in-person trainings provided hands-on training in the TF-CBT treatment model and in conducting a trauma-informed assessment using standardized clinical assessments. Trainings held in the middle and end of the learning collaborative often included discussions on other important clinical topics (such as dealing with clients’ trauma-related avoidance symptoms).
Bimonthly consultation calls with a certified TF-CBT trainer. Content covered in each consultation call ranged from reviewing the TF-CBT treatment components to case presentations for consultation with the TF-CBT trainer. Trainees who presented on a case were expected to provide results of their client’s trauma-informed assessment and share with the group their client’s progress through (as well as the trainee’s adherence to) the treatment model.
Technical assistance during the follow-up and practicum period. Perhaps the most popular (and time-consuming) aspect of the technical assistance provided by Ambit during the learning collaborative is the help with baseline and follow-up assessments for clients receiving TF-CBT. A requirement of the learning collaborative is to complete TF-CBT treatment with a minimum number of clients, conducting an assessment at baseline, three-month follow-up, and at the end of treatment. To help fulfill this requirement, Ambit scored clinical assessments and monitored upcoming follow-up interviews for active clients, prompting trainees at three-month intervals to conduct another follow-up assessment until the client had ended treatment.
Agencies that were selected to participate in the learning collaborative entered into contracts with the Minnesota Department of Human Services Children’s Mental Health Division. These contracts allowed trainees to be reimbursed at a standard hourly rate for time spent on learning collaborative activities (excluding providing direct services). These contracts held trainees accountable for meeting the learning collaborative requirements. In addition, the primary source of funding for Ambit Network came from a federal SAMHSA grant – meaning either low- or no-cost services needed to be used to meet high programming demands while keeping grant expenditures low. Finally, the addition of the technical assistance and practicum requirements (i.e. to complete assessments) created unique training demands with no prototype for statewide delivery. Faced with these dilemmas, Ambit needed to find a way to efficiently – both financially and operationally – implement TF-CBT in the community.
Finding a Solution: The Innovative Use of Technology for Implementation
While in-person trainings did not present a problem, hosting bimonthly consultation calls and providing high-quality technical assistance during the follow-up and practicum period did. In its first year of implementation, Ambit encountered three major hurdles – primary among them, trainees’ limited knowledge of the clinical assessments required of them. Trainees were introduced to the clinical assessments at the first in-person training; the majority had little to no previous experience with the assessment tools. Although trainees quickly became comfortable with using the tools for a trauma-informed assessment interview, almost all trainees were unprepared for interpreting the results and using the information in treatment planning.
A second major hurdle identified in the first year of implementation was finding an easy-to-use consultation call system. A number of different systems were tested – including online phone conference call systems (e.g., FreeConference.com, and the University of Minnesota’s Gopher Conferencing) and videoconferencing via Skype. Most systems were easy-to-use with very little technical problems. However, the content covered in each call became an issue; trainees were doing case presentations and asking clinical questions to the TF-CBT trainer, but a disproportionate amount of time was spent conveying information (such as reading off assessment scores) instead of asking clinical questions and addressing the actual case.
The final hurdle was finding the most efficient way of providing technical assistance to trainees during the practicum period while also collecting data on the implementation process. Aside from supporting trainees learning to utilize and incorporate assessments into treatment planning for clients, the practicum period served to collect data and evaluate the process of implementation for key stakeholders. The most significant issue that came to light during the technical assistance portion was tracking assessment submissions and follow-up interviews for active clients. To provide technical assistance, Ambit had to develop a system for tracking and monitoring trainee and client outcomes over the course of the learning collaborative.
Beginning with an Excel spreadsheet, each assessment interview was entered as a new case. Baseline and end of treatment interviews were entered when the appropriate paperwork was submitted; follow-up interviews were entered when paperwork was submitted or if the trainee missed a follow-up and did not submit assessments. Each interview (submitted or missed, in the case of follow-up interviews) was entered and coded for the following:
- Basic client demographics
- Assessments submitted (sent, not sent)
- Interview status (complete, incomplete, late)
- Follow-up status (on-time, late, missed)
- Reason case closed
This Excel spreadsheet continued to be used into the second year of implementation. In an effort to report to partners the program’s progress, a number of charts were created within Excel that would update with each new entry. However, after a year and a half of tracking interview data on almost 50 trainees (and nearing 1,000 interviews), the limitations of Excel became clear: Excel did not have the power to perform the necessary functions needed for tracking and technical assistance. In addition, significant staff time was spent entering data and monitoring client files and interviews – meaning little time could be dedicated to planning and evaluation activities that would allow program growth and expansion. Near the end of the second year of implementation, program staff identified several tools available at no cost to University of Minnesota staff that addressed the above challenges – UMConnect, Gopher Conferencing, and the Microsoft Access database computer software.
UMConnect and Gopher Conferencing
Used in conjunction with one another, UMConnect and Gopher Conferencing created a web-enhanced consultation call system that was both user-friendly and freed time on consultation calls for the trainer to address clinical questions, provide instruction on assessments, and model treatment planning for trainees. Gopher Conferencing, the University of Minnesota’s phone conferencing system, provided a convenient online scheduling system and consistently clear phone calls, regardless of the number of individuals on the call. Trainees calling in long-distance incurred minor costs (less than $40 per year for 18 calls), but scheduling and moderating calls using Gopher Conferencing came at no cost for Ambit Network.
UMConnect is an online web conferencing system that can be used for sharing content in online meeting rooms; UMConnect is free to UMN faculty and staff and uses the Adobe Connect web conferencing software (
http://www.adobe.com/products/adobeconnect.html). During consultation calls, trainees would call into the conference using Gopher Conferencing; on their personal computers, trainees would enter the UMConnect meeting room using a URL provided by Ambit Network (see Figure 1). As trainees presented on active cases, Ambit Network staff would pull up clinical assessment reports and fidelity monitoring tools onto the meeting room screen. The trainer and trainees would be able to see all clinical scores and fidelity data, removing the need for the case presenter to commit time to reading through this information. This simple web-enhancement created an additional 5-10 minutes of consultation time per case presentation; with an average of two case presentations per call, this meant up to 20 minutes spent on consultation instead of conveying basic information.

Figure 1: Participant view of a fidelity monitoring tool during consultation call using UMConnect
Microsoft Office Access Database Computer Software
During the second year of implementation, a graduate assistant employed at Ambit Network designed and built a Microsoft Access database for tracking assessment submissions and follow-up interviews for trainees and their active clients. Microsoft Access had the advantage over Excel of being able to create a database that could be easily tailored to meet Ambit’s data management needs. Access was also a more powerful program in terms of data storage, provided a user-friendly data entry interface, and allowed for designing queries that could quickly pull data about a client, trainee, agency, or even an entire learning collaborative cohort. The current database used by Ambit has data stored across 23 tables; a number of queries are used on a monthly basis to update reports designed to monitor trainee progress towards meeting learning collaborative requirements (Figure 2). In turn, these reports are used during supervisor calls to assist in identifying strength and challenge areas in partnering agencies and create discussions about long-term sustainability of TF-CBT. The Access database is also used to provide information for federal quarterly reports and for reporting back to state partners.
Figure 2: Client status report created in Microsoft Access using queries to populate report fields
Impact in Minnesota and Future Directions
Since 2007, Ambit Network has trained almost 240 mental health providers from 43 agencies across Minnesota using this adaptation of the learning collaborative model. Trainees have screened almost 1,300 children for trauma and post-traumatic stress; these numbers alone demonstrate that successful implementation of an EBP developed for high-risk children into the community is possible. A little ingenuity and access to low-cost technology such as UMConnect allowed Ambit to develop an implementation model that could be easily replicated and adapted to meet the needs for future implementation efforts. It allows for the collection of both outcomes and process data – both of which are necessary for program planning, monitoring, and evaluation, particularly for grant funded programs. The success Ambit Network has had in implementing TF-CBT also highlights another crucial point - addressing a public health issue like childhood trauma requires a systems approach with enthusiastic partners. Over the past five years and numerous successful partnerships, Ambit Network has become a leader in the implementation field, leading the way in increasing access to quality mental health care for children and families across the state of Minnesota.
Special Acknowledgements:
Dawn Reckinger
Glenace Edwall, Minnesota Department of Human Services
Pat Nygaard, Minnesota Department of Human Services
Acknowledgements
Ambit Network (Gewirtz, PI) received funding support that made this project possible through a 2005, and 2009 Substance Abuse and Mental Health Services Administration (SAMHSA) grant from the National Child Traumatic Stress Network, and the partnership and financial support of the Minnesota Department of Human Services – Children’s Mental Health Division.
References
Cohen, J. A., & Mannarino, A. P. (1996). A treatment outcome study for sexually abused preschool children: Initial findings. Journal of the American Academy of Child & Adolescent Psychiatry, 35(1), 42-50.
Cohen, J. A., & Mannarino, A. P. (1997). A treatment study for sexually abused preschool children: Outcome during a one-year follow-up. Journal of the American Academy of Child & Adolescent Psychiatry, 36(9), 1228-1235.
Cohen, J. A., & Mannarino, A. P. (1998). Interventions for sexually abused children: Initial treatment outcome findings. Child Maltreatment, 3(1), 17-26.
Cohen, J. A., Mannarino, A. P., & Deblinger, E. (2006). Treating trauma and traumatic grief in children and adolescents: Guilford Press.
Cohen, J. A., Mannarino, A. P., & Knudsen, K. (2005). Treating sexually abused children: 1 year follow-up of a randomized controlled trial. Child Abuse & Neglect, 29(2), 135-145.
Deblinger, E., Lippmann, J., & Steer, R. (1996). Sexually abused children suffering posttraumatic stress symptoms: Initial treatment outcome findings. Child Maltreatment, 1(4), 310-321.
Deblinger, E., McLeer, S. V., & Henry, D. (1990). Cognitive behavioral treatment for sexually abused children suffering post-traumatic stress: Preliminary findings. Journal of the American Academy of Child & Adolescent Psychiatry, 29(5), 747-752.
Deblinger, E., Stauffer, L. B., & Steer, R. A. (2001). Comparative efficacies of supportive and cognitive behavioral group therapies for young children who have been sexually abused and their nonoffending mothers. Child Maltreatment, 6(4), 332-343.
Deblinger, E., Steer, R. A., & Lippmann, J. (1999). Two-year follow-up study of cognitive behavioral therapy for sexually abused children suffering post-traumatic stress symptoms. Child Abuse & Neglect, 23(12), 1371-1378.
Drake, R. E., Goldman, H. H., Leff, H. S., Lehman, A. F., Dixon, L., Mueser, K. T., et al. (2001). Implementing evidence-based practices in routine mental health service settings. Psychiatric Services, 52(2), 179-182.
Hoagwood, K., Burns, B. J., Kiser, L., Ringeisen, H., & Schoenwald, S. K. (2001). Evidence-based practice in child and adolescent mental health services. Psychiatric Services, 52(9), 1179-1189.
Hoagwood, K., & Olin, S. S. (2002). The NIMH blueprint for change report: Research priorities in child and adolescent mental health. Journal of the American Academy of Child & Adolescent Psychiatry, 41(7), 760-767.
Institute for Healthcare Improvement. (2003). The Breakthrough Series: IHI's Collaborative Model for Achieving Breakthrough Improvement. Retrieved May 5, 2012, from www.IHI.org
King, N. J., Tonge, B. J., Mullen, P., Myerson, N., Heyne, D., Rollings, S., et al. (2000). Treating sexually abused children with posttraumatic stress symptoms: A randomized clinical trial. Journal of the American Academy of Child & Adolescent Psychiatry, 39(11), 1347-1355.
McHugh, R. K., & Barlow, D. H. (2010). The dissemination and implementation of evidence-based psychological treatments: A review of current efforts. American Psychologist, 65(2), 73-83.