Lessons learned while building a trauma-informed public behavioral health system in the City of Philadelphia

Abstract

Exposure to traumatic experiences among youth is a serious public health concern. A trauma-informed public behavioral health system that emphasizes core principles such as understanding trauma, promoting safety, supporting consumer autonomy, sharing power, and ensuring cultural competence, is needed to support traumatized youth and the providers who work with them. This article describes a case study of the creation and evaluation of a trauma-informed publicly funded behavioral health system for children and adolescents in the City of Philadelphia (the Philadelphia Alliance for Child Trauma Services; PACTS) using the Exploration, Preparation, Implementation, and Sustainment (EPIS) as a guiding framework. We describe our evaluation of this effort with an emphasis on implementation determinants and outcomes. Implementation determinants include inner context factors, specifically therapist knowledge and attitudes (N = 114) towards evidence-based practices. Implementation outcomes include information on rate of PTSD diagnoses in agencies over time, number of youth receiving TF-CBT over time, and penetration (i.e., number of youth receiving TF-CBT divided by the number of youth screening positive on trauma screening). We describe lessons learned from our experiences building a trauma-informed public behavioral health system in the hopes that this case study can guide other similar efforts.

Exposure to traumatic experiences among children and adolescents is a serious public health concern (Gillespie et al., 2009). The majority of youth are exposed to at least one, and often multiple, traumatic experiences by the age of 17 years (Finkelhor, Turner, Ormrod, & Hamby, 2009). Chronic stress and trauma can compromise optimal brain development and negatively impact physical, emotional, behavioral, social, and cognitive development in youth (DeCandia, Guarino, & Clervil, 2014; Middlebrooks & Audage, 2008). For example, adults who experienced four or more adverse child experiences (ACEs) had poorer physical outcomes compared to adults who did not experience four or more ACEs (Felitti et al., 1998). These findings are consistent with a growing body of research that indicates a strong relationship between cumulative exposure to traumatic events in childhood and a wide array of health and mental health impairments in adulthood (Shonkoff, Boyce, & McEwen, 2009). This effect is likely more profound in urban inner city environments (The Research and Evaluation Group at the Public Health Management Corporation, 2013).

To address the needs of traumatized youth, strengthening the infrastructure of public mental health systems around trauma-informed principles is critical. Trauma-informed systems are built around core principles, including 1) understanding trauma and its impact; 2) promoting safety; 3) supporting consumer control, choice, and autonomy; 4) sharing power and governance; 5) ensuring cultural competence; 6) integrating care; 7) the belief that healing happens in relationships; and 8) the understanding that recovery is possible (Guarino, Soares, Konnath, Clervil, & Bassuk, 2009). This article describes the creation and evaluation of a trauma-informed publicly funded behavioral health system for children and adolescents in the City of Philadelphia that included the implementation of trauma-focused cognitive-behavioral therapy (TF-CBT), an evidence-based practice (EBP) for traumatized youth. Proven efficacious (Chaffin & Friedrich, 2004), TF-CBT incorporates many key trauma-informed principles, such as psychoeducation to help families understand trauma and its impact; safety planning to prevent re-traumatization; and focusing on recovery through the creation of the trauma narrative and cognitive processing of the traumatic event (Cohen, Mannarino, & Deblinger, 2006).

This case study reflects the shared perspectives of the community-academic partners who contributed to these efforts to build a trauma-informed system. The PACTS team includes policy-makers, leadership from community mental health agencies, and a university-based evaluation team.

The objectives of this manuscript are as follows:

1. Describe the context within which the trauma-informed system and the evaluation were developed.

2. Describe the implementation science framework that guides the evaluation.

3. Present data with regard to implementation determinants and outcomes.

4. Provide recommendations, based on lessons learned, for developing and evaluating a trauma-informed public mental health system that links to other youth-serving systems.

Context

Philadelphia is a large, diverse city of over 1.5 million people. Residents include African Americans (42%), Caucasians (37%), Hispanic/Latinos (13%), Asians (6%), and individuals of other origins (2%; The Pew Charitable Trusts, 2013). Philadelphia’s poverty rate is among the highest in the nation (The Pew Charitable Trusts, 2013); educational levels are well below national averages, and unemployment rates in low-income neighborhoods are over 20%. In 2014, the homicide rate was 16 per 100,000 residents (The Pew Charitable Trusts, 2015). These indicators demonstrate the high trauma risk for Philadelphia’s youth. Approximately 80% of Philadelphia’s youth (approximately 350,000 children and adolescents) are enrolled in Medicaid. Public behavioral health services are managed by Community Behavioral Health (CBH), a nonprofit managed care organization (i.e., ‘carve-out’) established by the City of Philadelphia that functions as a component of the Department of Behavioral Health and Intellectual disAbility Services (DBHIDS). DBHIDS oversees all public behavioral health service delivery in Philadelphia County.

In 2005, DBHIDS initiated a system transformation for people with behavioral health needs (Achara-Abrahams, Evans, & King, 2011). This transformation emphasizes the cultures, resilience and protective factors, and unique recovery processes of individuals and families. As part of this transformation, and given the challenges facing Philadelphia residents (e.g., poverty and community violence), DBHIDS simultaneously prioritized the need to create a trauma-informed system, particularly for children and families. Given that 40% of Philadelphia adults report witnessing violence when growing up, and over 37% report four or more ACEs (The Research and Evaluation Group at the Public Health Management Corporation, 2013), conservative estimates suggest that there are approximately 30,000 youth in Philadelphia in need of evidence-based trauma treatment. The full-scale effort to develop a trauma-informed behavioral health system began in 2011 with the ‘Trauma Initiative’ (see Figure 1 for a timeline of related activities). As part of this initiative, DBHIDS contracted with treatment developers to provide training for behavioral health agencies in both trauma-specific EBPs and trauma-informed models. These approaches include TF-CBT (Cohen, Deblinger, Mannarino, & Steer, 2004; Cohen & Mannarino, 1998) and Prolonged Exposure (PE; Foa et al., 2005), both evidence-based cognitive-behavioral therapies for the treatment of PTSD in children and adults (respectively). Additionally, DBHIDS provided training in the Sanctuary Model (Bloom et al., 2003), an evidence-informed approach to building trauma-informed organizations.

Figure 1.

Timeline of TF-CBT implementation

Note. The trauma initiative began in 2011; the Philadelphia Alliance for Child Trauma Services (PACTS) began in 2012. This is denoted by the vertical dashed line. For each year, we denote the core activities that occurred to visually depict the actions taken to build a trauma-informed public behavioral health system.

In 2012, DBHIDS was awarded a National Child Traumatic Stress Initiative Community Treatment and Service Center grant (Category III) from the Substance Abuse and Mental Health Services Administration (SAMHSA) to form PACTS. The mission of PACTS is to increase the number of children who receive evidence-based trauma treatments in Philadelphia. Consistent with this mission, this manuscript focuses on describing the implementation of TF-CBT (rather than PE and Sanctuary). TF-CBT was chosen based on the strong evidence of efficacy for youth (age four to eighteen years) who have experienced potentially traumatic events (see de Arellano et al., 2014). TF-CBT consistently receives the highest ratings in research reviews on treatment for children with PTSD symptoms (Chadwick Center for Children and Families, 2009; Saunders, Berliner, & Hanson, 2004; Silverman et al., 2008). Additionally, TF-CBT is the most widely disseminated treatment for youth with PTSD symptoms, including youth from diverse cultural backgrounds (Sigel, Benton, Lynch, & Kramer, 2013), and has a proven track record of dissemination and implementation (Cohen & Mannarino, 2008).

Implementation Science Framework

Both the implementation and evaluation of TF-CBT in the City of Philadelphia have been informed by leading implementation science frameworks (Aarons, Hurlburt, & Horwitz, 2011; Proctor et al., 2009) which specify the importance of both the innovation being implemented (i.e., TF-CBT), the fit of the innovation with the setting, and the inner (i.e., therapist and organizational) and outer context (i.e., system; see Figure 2) factors that influence the implementation process. Specifically, the EPIS framework (see Figure 2) is used to guide the understanding of contextual factors (or implementation determinants) which guide the implementation process and the stages of implementation (Aarons et al., 2011). The EPIS framework outlines the four phases of the implementation process (exploration, preparation, implementation, and sustainment) and identifies domains that are important to implementation, including the inner context (e.g., organizational and therapist characteristics) and outer context (e.g., the service environment, inter-organizational environment, and consumer support; Aarons et al., 2011). Within each phase, contextual variables relevant to the inner or outer context are posited. For example, during implementation, inner context variables hypothesized to be important to the implementation process include clinician attitudes toward EBP and their knowledge of EBP, whereas outer context variables include funding, engagement with treatment developers, and leadership. In this evaluation, we primarily focus on the implementation stage, although we provide background on the exploration and preparation phases. Future work will focus on the sustainment phase.

Figure 2.

The Exploration, Preparation, Implementation, and Sustainment (EPIS) framework

Note: Reproduced from (Aarons, et al., 2011). * = components that we did not measure as part of this evaluation. Our primary focus as part of this evaluation was the implementation phase. Specifically, we measured inner context implementation determinants, including therapist knowledge and attitudes, and implementation outcomes (i.e., penetration). In the manuscript, we report on implementation strategies used in each phase of the implementation process.

In addition to using EPIS to inform the implementation and evaluation, all of the work described within the case study has been strongly guided by a community-academic partnership (Drahota et al., 2016). The primary members of this partnership include policy-makers, leadership from community mental health agencies, and university-based researchers. The input of therapists, administrators, and consumers from community mental health agencies has also been solicited over the course of the efforts described.

Case Study: Creating a Trauma-Informed Behavioral Health System in the City of Philadelphia

To support TF-CBT implementation and consistent with leading implementation science frameworks used to guide the work, DBHIDS and CBH have used multi-faceted, multi-level implementation strategies including planning, educating, and financing (Powell et al., 2012; Proctor et al., 2009). Planning strategies include those that help stakeholders engage in pre-implementation strategies to increase engagement and buy-in during the exploration and preparation phases. Specifically, DBHIDS prioritized relationship building by bringing together multiple stakeholders from different organizations to attend to the barriers in implementing TF-CBT. Education strategies refer to those that are intended to change the knowledge of stakeholders implementing (e.g., training) and have occurred during all phases of the implementation process. DBHIDS has provided didactic training, ongoing clinical consultation, and technical assistance around both screening and assessment for trauma and the implementation of TF-CBT. Financial strategies refer to the use of incentives to increase use of EBPs. In 2012, CBH implemented an enhanced financial rate for therapists to incentivize use of TF-CBT during implementation. These implementation strategies are consistent with the broader approach that DBHIDS has taken to the implementation of EBPs (Powell et al., 2016).

A number of activities make up the efforts to create a trauma-informed behavioral health system. First, during the preparation phase, DBHIDS endeavored to develop a coordinated system of service providers (Grasso, Webb, Cohen, & Berman, 2013) given that historically, the approach to trauma treatment in Philadelphia was not well coordinated. Three agencies in close proximity, in the center of the city, provided specialty trauma treatment, but this was not enough to meet demand across a large metropolitan city (see Figure 3 for a map of trauma providers prior to PACTS). Now, following implementation of TF-CBT through PACTS, 14 behavioral health agencies (which comprise 16 programs) across the Philadelphia metropolitan area (see Figure 4) provide TF-CBT. It is noteworthy that the number of providers providing TF-CBT has greatly increased in both general outpatient and specialty settings (i.e., residential, Hispanic/Latino oriented programs). In addition to the 14 behavioral health agencies, during the preparation and implementation phases, linkages have been made between these agencies and pediatric hospitals, child advocacy centers, crisis response centers, the medical examiner’s office, juvenile justice, the child welfare system, and schools, in order to increase system capacity to screen and refer youth appropriately.

Figure 3.

Map of trauma providers prior to PACTS

Note. Prior to PACTS, there were three programs providing TF-CBT. Each circle in the figure represents one program which has multiple therapists within it providing TF-CBT.

Figure 4.

Map of trauma providers after PACTS (as of 2016)

Note. There are 14 organizations and 16 programs in the City of Philadelphia implementing TF-CBT through PACTS. Each circle in the figure represents one program which has multiple therapists within it providing TF-CBT.

Simultaneously as the referral network of behavioral health providers was built during the preparation phase, the PACTS team also began to provide training, consultation, and technical assistance to behavioral health and non-behavioral health staff in TF-CBT (see Figure 1 for a visual depiction of described activities; Hanson et al., 2014; Powell et al., 2012). Since 2011, six cohorts (182 therapists and 34 supervisors) have received training in TF-CBT. A seventh cohort will be trained in the Fall of 2016. As of 2016, only 46% of trained therapists (n = 83) and 44% of trained supervisors (n = 15) remain in their agencies, suggesting a high turnover rate. During the implementation phase, training includes two days of didactic training and ongoing consultation for eight months that consists of bi-weekly consultation calls with a TF-CBT certified master trainer. These calls average one hour in duration with up to ten participants in attendance. Each clinician presents twice on active cases during the 16 calls, and attendance is expected (at least 13 out of the 16 calls). In addition to the training and consultation, each agency designates a supervisor who provides weekly internal supervision to therapists. Annually, booster trainings are provided to keep therapists engaged in TF-CBT implementation on relevant topics (e.g., ACES). There have also been meetings with executive directors in 2013, 2014, and 2016 to keep leadership engaged in TF-CBT implementation in their agencies.

To increase access to evidence-based trauma services in the community, during the implementation phase, it became clear that trauma screening and assessment was necessary (Cusack, Frueh, & Brady, 2004; Frueh et al., 2002). Initially, few agencies implementing TF-CBT reported using a structured trauma screening at intake and/or a trauma symptom rating scale for ongoing assessment and progress monitoring. Clinicians and supervisors discussed a desire to utilize these measures; however, concerns were raised about the realities of implementing such tools, consistent with the broader literature (Jensen-Doss & Hawley, 2010, 2011; Osterberg, Jensen-Doss, Cusack, & de Arellano, 2009). During informal conversations with the PACTS team, clinicians reported the immense paperwork burdens that they faced and worries that client rapport could be impacted by adding more paperwork. Furthermore in these conversations, clinicians emphasized the importance of ensuring that these measures could be used in clinically meaningfully ways, and the need for support from their agencies around the use of these measures. Thus, in 2014, PACTS provided screening and assessment training to PACTS-trained clinicians (i.e., cohorts one through four), and subsequently provided consultation on screening and assessment to all organizations implementing TF-CBT as part of PACTS.

Methods

Evaluation of the implementation of TF-CBT is a core mission of PACTS and is led by the university-based evaluation group. To evaluate the effectiveness of the multi-faceted, multilevel implementation strategies used by DBHIDS (and consistent with the implementation science frameworks used to guide the evaluation), the evaluation team has used a hybrid approach (Curran, Bauer, Mittman, Pyne, & Stetler, 2012) that includes both implementation and client outcomes. Variables of interest include implementation determinants (inner context factors such as therapist knowledge and attitudes about EBPs) and implementation outcomes (i.e., penetration; Proctor et al., 2009). Client outcomes such as treatment outcome, symptoms, and functioning are also measured by the evaluation team but are not reported in this manuscript as collection of this data is ongoing.

Implementation Determinants

To understand the impact of the education implementation strategies used by DBHIDS (i.e., training and ongoing consultation) on implementation determinants, therapist knowledge of and attitudes towards EBP was evaluated in four cohorts (cohorts three through six).¹ Knowledge and attitudes were assessed at three time-points: pre-training, post-training (i.e., directly following the completion of the training workshop), and at six-month follow-up. The measurement strategy used was modified after cohorts three and four due to preliminary results suggesting that knowledge and attitudes did not change over time. Rather than use general EBP knowledge and attitude questionnaires, trauma-specific questionnaires were implemented because of the hypothesis that the general measures were not specific enough.

Knowledge

For cohorts three and four, therapist knowledge was measured using the Knowledge of Evidence-Based Services Questionnaire (KEBSQ; Stumpf, Higa-McMillan, & Chorpita, 2009), a 40-item self-report instrument that measures general knowledge of EBP, and demonstrates adequate psychometric properties (Stumpf et al., 2009). For cohorts five and six, the TF-CBT Knowledge Test (Woody, Anderson, D'Souza, Baxter, & Schubauer, 2015) and the TF-CBT Knowledge Questionnaire (Fitzgerald, 2012) were used to measure knowledge. The Knowledge Test provides clinicians the opportunity to apply their knowledge of TF-CBT to eight scenarios, while the Knowledge Questionnaire examines factual knowledge of principles of TF-CBT using 17 multiple-choice questions.

Attitudes

For cohorts three and four, therapist attitudes were assessed using the Evidence-Based Practice Attitude Scale (EBPAS; Aarons, 2004), a 15-item self-report questionnaire that assesses constructs related to the appeal of EBP, requirements to use EBP, general openness to new practices, and divergence between EBP and usual practice (Aarons & Sawitzky, 2006). The psychometric properties of the EBPAS have been well established and national norms are available (Aarons et al., 2010). For cohorts five and six, a modified version of the EBPAS specific to TF-CBT was utilized (Aarons, Fettes, et al., 2014). Specifically, rather than use the generic terminology in the question stems, TF-CBT was referenced (i.e., “I am willing to try TF-CBT even if I have to follow a manual” versus “I am willing to try new types of therapy/interventions even if I have to follow a treatment manual”).

Implementation Outcomes

Implementation outcomes are presented below and include administrative data. Implementation outcomes include: (1) rate of PTSD diagnoses in PACTS agencies over time, (2) number of youth receiving TF-CBT over time, and (3) penetration (i.e., number of youth receiving TF-CBT divided by the number of youth screening positive on trauma screening).

Data Sources

The data presented in this manuscript include both primary (i.e., self-reported from therapists) and administrative data (i.e., claims); primary data were collected following informed consent procedures. All data presented are quantitative; qualitative evaluation is ongoing.

Results

Implementation Determinants

Therapists trained in cohorts three and four (N = 54; 44% therapists trained) were 87% female. See Table 1 for means and standard deviations for each variable. Means at pre-training were consistent with means provided in a national sample of therapists in community mental health clinics (Aarons et al., 2010). Five repeated-measures analysis of variance analyses were conducted to ascertain change over time (pre-training, post-training, and six-month follow-up) in knowledge and attitudes. Knowledge and the majority of attitudes did not change significantly over time. However, a quadratic relationship with regard to openness to EBPs was observed (F (2, 50) = 6.68, p < .01), where openness to EBPs increased from pre-training (M = 2.98) to post-training (M = 3.23), (F (1, 25) = 14.889, p < .01) and decreased from post-training to six-month follow-up (M = 3.05), (F (1, 25) = 5.84, p < .05).

Table 1.

Therapist Evaluation Descriptive Statistics (Cohorts 3–4)

Variables	Pre Training (n = 58)		Post Training (n = 57)		6-Month Follow-Up (n = 41)		Repeated measures Analysis of Variance
	Mean	SD	Mean	SD	Mean	SD	df	F	p
EBPAS Requirementsa	2.86	1.05	2.80	1.09	3.11	0.95	50	0.47	0.63
EBPAS Appeala	3.03	0.77	3.21	0.76	3.26	0.74	50	1.63	0.20
EBPAS Opennessa	2.98	0.63	3.23	0.63	3.05	0.58	50	6.68	0.00
EBPAS Divergencea	1.06	0.47	1.17	0.63	1.04	0.56	50	1.81	0.17
KEBSQ Totalb	96.45	10.02	95.92	9.51	94.57	4.46	56	1.57	0.22

Note.

EBPAS = Evidence-Based Practice Attitudes Scale (Aarons, 2004)

KEBSQ = Knowledge of Evidence-Based Practice Scale Questionnaire (Stumpf, et al., 2009)

^aMeasured on a 5-point Likert scale (0-not at all; 1-slight extent; 2-moderate extent; 3-great extent; 4-very great extent). Higher scores are indicative of more positive attitudes, implementation climate, and implementation leadership.

^bMeasured on a scale ranging from 0–160, where higher scores are indicative of more knowledge of evidence-based services for youth.

Therapists in cohorts five and six (N = 60; 87% of therapists trained) were primarily female (83%). See Table 2 for means and standard deviations for each variable. Notably, means at pre-training on the TF-CBT specific version of the EBPAS appear higher than the general EBPAS scores measured in cohorts three and four. Six paired samples t-tests were conducted to ascertain change over time (pre- to post-training) in knowledge and attitudes. Once again, knowledge and the majority of attitudes did not change significantly over time. However, openness to TF-CBT (t (57) = − 2.72, p < .01) changed over time, where openness to TF-CBT increased from pre-training (M = 3.40) to post-training (M = 3.58). Data at six-month follow up is not available for cohorts five and six.²

Table 2.

Descriptive Statistics and Tests for Significance for Therapist Evaluation (Cohorts 5 and 6)

Variables	Pre Training (n = 60)		Post Training (n = 58)		Paired Sample t-test
	Mean	SD	Mean	SD	t	df	p
TF-CBT EBPAS Requirementsa	3.07	1.03	3.00	1.25	0.43	57	0.669
TF-CBT EBPAS Appeala	3.35	0.63	3.50	0.58	−1.85	57	0.069
TF-CBT EBPAS Opennessa	3.40	0.55	3.58	0.56	−2.72	57	0.009
TF-CBT EBPAS Divergencea	0.98	0.59	1.02	0.73	−0.59	57	0.557
TF-CBT EBPAS Totala	3.21	0.50	3.26	0.55	−0.85	57	0.399
TF-CBT Knowledge Questionnaire - Total Correctb	9.50	2.95	9.88	1.90	−1.72	57	0.175
TF-CBT Knowledge Test - Total Correct c	5.36	1.40	5.74	1.49	−1.38	57	0.092

Note.

TF-CBT EBPAS = Trauma focused cognitive behavioral therapy Evidence-Based Practice Attitude Scale (Aarons., 2014)

TF-CBT Knowledge Questionnaire (Fitzgerald, 2012)

TF-CBT Knowledge Test (Woody, et al., 2015)

^aMeasured on a 5-point Likert scale (0-not at all; 1-slight extent; 2-moderate extent; 3-great extent; 4-very great extent). Higher scores are indicative of more positive attitudes, implementation climate, and implementation leadership.

^bTotal correct out of 17, (1-correct; 0-incorrect). Higher scores are associated with higher levels of knowledge.

^cTotal correct out of 8, (1-correct; 0-incorrect). Higher scores associated with higher levels of knowledge

Implementation Outcomes

Rate of PTSD diagnoses in PACTS agencies over time. Rate of PTSD diagnosis in agencies participating in TF-CBT implementation was investigated using Medicaid claims data. Prior to the launch of the trauma initiative in 2011, approximately 4% of youth receiving treatment in PACTS agencies had a diagnosis of PTSD. In 2014, approximately 31% of youth in agencies implementing TF-CBT had a diagnosis of PTSD (Community Behavioral Health, 2015), suggesting a substantial increase in the number of youth diagnosed with PTSD which may be due to increased availability and/or access to trauma services. This finding provides initial support for efforts to increase trauma screening. This rate of PTSD diagnosis falls within the estimates provided for rates of PTSD in urban, inner-city youth (i.e., 14–67%; Horowitz, Weine, & Jekel, 1995; Lipschitz, Rasmusson, Anyan, Cromwell, & Southwick, 2000). These diagnoses are based on claims data rather than research instruments and may represent an over or underestimate of the actual PTSD diagnosis prevalence.

Number of youth receiving TF-CBT from behavioral health providers

The number of youth receiving TF-CBT from PACTS behavioral health providers was measured in two ways. First, administrative claims data were used to extract the number of youth who received TF-CBT as indicated by the TF-CBT modifier implemented in 2012 to allow agencies to bill for the enhanced rate. According to these data, 316 unique clients were treated with TF-CBT from 2012–2015. Interestingly, not all youth whose agencies billed for the enhanced TF-CBT rate had a PTSD diagnosis (see Table 3). In 2012, 50% (n = 2) of youth treated with TF-CBT had a PTSD diagnosis (N = 4). In 2013, 30% (n = 38) of youth treated with TF-CBT had a PTSD diagnosis (N = 125). In 2014, 34% (n = 43) of youth treated with TF-CBT had a PTSD diagnosis (N = 125). In 2014, 52% (n = 71) of youth treated with TF-CBT had a PTSD diagnosis (N = 137).

Table 3.

Diagnoses of youth receiving TF-CBT as reported in Medicaid claims data and billing

Diagnosis	Number of youth with diagnosis	Percentage
PTSD	146	35%
Acute reaction to stress	1	0.2%
Adjustment Disorder	108	26%
Anxiety Disorders	10	2%
Disruptive Behavior Disorders	73	18%
Mood Disorders	26	6%
Other Disorders	14	3%
Substance Use Disorder	31	7%
Autism Spectrum Disorder	5	1%
Total Diagnoses records	414	--

Note: The number of unique clients is 316. Claims data may have included more than one disorder per youth, thus the total number of diagnoses is greater than this number; however if youth had a PTSD diagnosis, then they were only counted once in the PTSD row. However, if they did not have a PTSD diagnosis, they could be counted more than once in all of the other diagnosis rows.

The second way to measure the number of youth receiving TF-CBT was through reports made by agency supervisors to the PACTS project manager. As evidenced by Figure 5, the number of clients starting and completing TF-CBT has grown each year, from approximately 80 youth completing treatment in 2012 to 330 youth in 2015 (the last year with complete data for the full year). Furthermore, about half of clients who have commenced TF-CBT have successfully completed all TF-CBT modules including the trauma narrative, a critical ingredient of TF-CBT (Deblinger, Mannarino, Cohen, Runyon, & Steer, 2011). The percentage of youth receiving all TF-CBT modules has been constant each year. Since 2012 when PACTS was launched, approximately 1335 children and adolescents have begun TF-CBT and about 776 youth have completed TF-CBT in the City of Philadelphia.

Figure 5.

Number of clients who started and completed TF-CBT from participating agencies implementing TF-CBT from 2012–2016

Note. Data from 2015–2016 only includes information from Quarters 1 and 2 because data is currently being collected. Opened cases refer to new cases that began TF-CBT in that year. Closed cases refer to cases that ended TF-CBT in that year. Completed with trauma narrative refers to those cases that were closed that completed TF-CBT and included the trauma narrative. All data presented here reflects unique cases.

The number of youth treated with TF-CBT as reported by supervisors is higher than the data identified through claims which can be explained in a number of ways. First, anecdotally, organizational leadership expressed reluctance about billing to the enhanced rate unless therapists were following TF-CBT with complete fidelity. Second, and also anecdotally, a number of the billing departments at the participating agencies were not aware that they could bill to the enhanced rate in early years. Third, participating agencies that were residential treatment facilities (two of the agencies) were not eligible for the enhanced rate. Fourth, three of the participating agencies already had negotiated an enhanced rate with CBH and did not use this modifier. Finally, therapists can only bill using this rate once they have completed all training requirements which takes approximately ten months.

Penetration

Penetration (i.e., number of youth receiving TF-CBT divided by the number of youth screening positive on trauma screening) has not been collected systematically across the course of the project. However, the project manager recently collected data during one month (March, 2016) to gain a better understanding of the number of youth screened for potentially traumatic events, the number of youth screening positive to experiencing a traumatic event, and the number of youth starting TF-CBT within participating agencies as reported by supervisors. During this month period, 232 youth were screened across the 14 agencies. Eighty-seven percent of these youth (n = 202) responded yes to at least one question on the Trauma History Questionnaire (THQ; Green, 1996), which is consistent with previous literature (Luthra et al., 2009). Twenty-two percent (n = 44) of eligible youth (i.e., screened positive on THQ) began TF-CBT during that month. This penetration rate suggests that about one-fifth of youth screening positive for a potentially traumatic event began TF-CBT in one month in PACTS agencies. Information about trauma symptoms at screening is not available, so it is difficult to interpret this penetration rate, particularly given that not all youth who endorse a traumatic event need TF-CBT. However, given that about 15% of youth who have experienced trauma go on to develop PTSD (Giaconia et al., 1995), a penetration rate of 20% would likely reach the majority of the youth within this sample who have clinically significant PTSD symptoms.

Discussion and Lessons Learned

This case study offers information about efforts to build a trauma-informed public behavioral health system for children and families. The results presented provide insight into the creation and initial evaluation of these efforts, and can hopefully serve as a guide to other systems undertaking similar efforts. A key aspect of developing a trauma-informed public behavioral health system was to establish a coordinated network of child providers located across the City of Philadelphia. In 2011, there were three programs providing trauma-specific services in one area of the city. As of 2016, there are fourteen providers, representing sixteen programs, providing evidence-based trauma treatments in all quadrants of the city. Second, linkages between locations where traumatized youth are identified and behavioral health providers have been strengthened, as consistent with recommendations to build trauma-informed health systems (Grasso et al., 2013; Ko et al., 2008). Third, and perhaps most importantly, there has been an increase in the number of youth identified and treated with TF-CBT. Specifically, the number of youth diagnosed with PTSD in agencies implementing TF-CBT has increased, suggesting an increased awareness and ability to screen and assess for trauma. Furthermore, these agencies have reported an over two-fold increase in the number of youth treated with TF-CBT since 2012, including youth receiving the entire treatment package, which is notable in an urban community setting. The ability to reach so many youth is partly attributable to the large footprint with regard to training behavioral health providers. Since 2011, 216 therapists and supervisors in community mental health settings have received training and consultation intended to increase implementation of TF-CBT. A number of lessons have been learned through the process of creating a trauma-informed behavioral health system, delineated below. These lessons are based upon the collective perspectives of the PACTS core team and the quantitative data presented above.

1. Agencies often need initial support in establishing trauma-based screening mechanisms

During the implementation phase, many of providers raised issues related to the screening and assessment of traumatized youth, consistent with the literature (Jensen-Doss & Hawley, 2010, 2011). Initially, providers reported difficulty finding clients who were candidates for TF-CBT because many agencies did not have established trauma-based screening mechanisms. To ameliorate this issue, each agency received training and tailored consultation (Nadeem, Gleacher, & Beidas, 2013) to assist with the implementing of screening and progress monitoring around trauma symptoms. This experience suggests that spending more time during the exploration and preparation phases of implementation on the proper identification of assessment tools, in partnership with those who will be implementing the EBP, is critical.

2. Engaging with leaders in agencies implementing TF-CBT is important

Executive leadership buy-in and involvement is critical to the implementation process (Aarons, Ehrhart, Farahnak, & Sklar, 2014) and was emphasized by annual meetings between the PACTS team and the executive directors in PACTS agencies. Equally as important, is the role of someone at each agency who can play the role of ‘boundary spanner.’ This refers to individuals who can serve as the “point-person” between the implementation team and the front-line implementers. Providing small incentives (e.g., a small stipend) and acknowledgement to those individuals proved critical to reinforce their efforts. This role was typically played by the clinical supervisor at each organization. These individuals were critical in communicating the PACTS message to front-line providers and to ensure that incentives (e.g., enhanced rate) for implementation were communicated to administrative staff.

3. Staff turnover is a critical issue that must be planned prior to training

Although a large number of clinicians have been trained in TF-CBT, there has been a high percentage of turnover in both therapists (46%) and supervisors (44%) over the past four years. This rate of turnover is roughly comparable to estimates in the literature (Mor Barak, Nissly, & Levin, 2001), and the annual average rate of turnover (25%) in Philadelphia community mental health clinics (Beidas, Marcus, Wolk, et al., 2015). Turnover can compromise continuity of care, destabilize agencies, diminish quality of services (Glisson & James, 2002; Knudsen, Johnson, & Roman, 2003; Mor Barak et al., 2001), and impact implementation of EBPs (Ganju, 2003; Isett et al., 2007; Woltmann et al., 2008), due to the substantial resources invested in training and supporting mental health workers.

The loss of almost half of the trained therapists is a threat to the sustainability of agencies providing TF-CBT (Stirman et al., 2012), although evidence from the Philadelphia system suggests that about 50% of therapists who turnover stay in the public system, suggesting some return on investment (Beidas, Marcus, Wolk, et al., 2015). Even more concerning is the loss of almost half of the supervisors, given that much of the literature assumes that turnover is not as insidious in supervisors, thus emphasizing train-the-trainer models (Nakamura et al., 2014). There are likely a number of reasons for turnover in PACTS agencies that are generalizable to community mental health and are perhaps specific to trauma-specific work. General reasons likely include professional opportunities (e.g., opportunities for promotion), personal reasons (e.g., familial responsibilities), organizational factors, and financial reasons (Beidas, Marcus, Wolk, et al., 2015). Specific reasons may include differing perspectives on how to treat trauma and vicarious traumatization. To ensure the viability of TF-CBT into the future, reducing turnover and identifying ways to support staff engaging in trauma-specific EBPs warrants future study. Strategies to reduce turnover include interventions to reduce burnout (Beidas, Marcus, Wolk, et al., 2015) and/or to improve organizational culture and climate (Glisson et al., 2008).

4. Training and consultation in evidence-based trauma treatments may be necessary but not sufficient to improve therapist knowledge and openness to them

In the therapist evaluation, change in implementation determinants hypothesized to be important to the implementation process was not observed, even after changing measurement methods, with one exception (i.e., openness to EBP). The first set of measures used in the therapist evaluation assessed knowledge and attitudes of EBP more broadly, rather than focusing specifically on TF-CBT. Given emerging evidence that therapists respond differently to the same questionnaire when it references EBP generally versus a specific EBP (Reding, Chorpita, Lau, & Innes-Gomberg, 2014), trauma-specific measures of attitudes (e.g., EBPAS-Trauma) and knowledge (see Woody et al., 2015) were implemented. However, findings remain consistent even with change in measurement strategy.

Null findings with regard to therapist knowledge change are inconsistent with the larger literature (Beidas & Kendall, 2010). It is not surprising that knowledge about general EBP did not change after therapists attended a TF-CBT workshop, particularly given that the measure used (KEBSQ) does not include questions related to trauma. However, the finding that TF-CBT specific knowledge also did not change is puzzling. The data suggests that there may have been a ceiling effect on the Knowledge Test – on average, prior to training, clinicians were able to answer more than 60% of the questions correctly. Alternatively, clinicians may have come to the training with a higher than average knowledge of TF-CBT (they were required to register for the online training; https://tfcbt.musc.edu/). A ceiling effect was not observed on the Knowledge Questionnaire. Potential explanations for the lack of findings may be because the measure was too difficult or did not represent workshop content.

The only finding in the therapist evaluation was that openness to EBP increased following training but decreased following consultation, which corroborates previous literature (Edmunds et al., 2014). Therapists who are initially open to new practices may become less open after they became aware of the significant investment needed to become competent in the EBP, making them less open to future new practices. It also may be that clinicians encountered challenges implementing TF-CBT after training and during consultation, which may have resulted in less openness (Edmunds et al., 2014). Unfortunately, it is unknown if this pattern was replicated in the TF-CBT specific EBPAS given that data at follow-up for cohorts five and six is not available. It is of note that initial attitudinal scores on the general EBPAS appeared lower than initial attitudinal scores on the TF-CBT specific EBPAS. This provides support to previous findings that EBP attitudes may differ by practice (Reding et al., 2014). Future work further specifying more precise measurement strategies in implementation science is needed. Cohort effects and other variables may have impacted these scores.

An alternative explanation for the findings observed in the therapist evaluation (i.e., lack of knowledge change; decrease in openness over time) suggests that implementation of TF-CBT did not meet the needs of therapists in the community. There are a number of explanations for why this may be the case. One explanation that is particularly germane based upon the PACTS team’s anecdotal experiences has to do with why clients come to treatment. Most youth present to community mental health clinics due to disruptive behavior problems, with many having underlying trauma histories. Therapists may need more modular approaches (Weisz et al., 2012) to allow them to treat the needs of the youth presenting to treatment; meeting the client where they indicate they need help (i.e., disruptive behavior) and then including elements of evidence-based trauma treatment. If therapists and organizations had been more involved during the exploration phase of treatment selection, a different EBP for trauma may have been selected. Additionally, therapist characteristics, such as the length of time in their agency and number of years practicing may have impacted their attitudes. This information was not collected as part of the evaluation but has been found to be predictive of knowledge and attitudes in previous work (Beidas, Marcus, Aarons, et al., 2015).

5. Youth who present in community mental health settings with trauma are heterogeneous, which has implications for how to assess and treat trauma

Youth receiving TF-CBT had heterogeneous diagnoses as reported in claims data. Approximately forty-two percent of youth had a PTSD diagnosis, while the remainder of youth had other diagnoses. The most common other diagnoses that were associated with claims data included adjustment disorder and disruptive behavior disorders. The variability in diagnoses has implications for decisions about which EBP to implement. Inclusion criteria have varied across TF-CBT treatment studies (Cohen, Mannarino, & Knudsen, 2005; Cohen, Mannarino, Perel, & Staron, 2007). Little is known about how to proceed when youth present with a history of trauma but do not meet full criteria for PTSD in community mental health settings (i.e., whether they still would benefit from TF-CBT, or if another EBP or prevention model would be more appropriate).

One important consideration that may explain the heterogeneous presentation of the client population may be that external forces (e.g., outer context factors such as fiscal and training pressures) may have resulted in the identification of youth for TF-CBT even if they did not meet full criteria for PTSD. Specifically, community mental health agencies were receiving an enhanced rate for each TF-CBT session. It is unlikely that this explanation best describes the scenario because agencies tended to not bill towards the enhanced rate because of concerns that they could only use the enhanced rate when they implemented TF-CBT with 100% fidelity. However, it is possible that participation in system efforts to implement EBP may have unintended consequences in the appropriate identification and subsequent treatment of youth in community settings. For example, therapist may treat all youth with TF-CBT even if that is not appropriate, given the desire to use what they have learned. This points to the importance of both including evidence-based assessment in all EBP efforts as well as measuring the potential unintended consequences of outer context factors on EBP implementation during all phases of the implementation process.

6. When implementing EBP in the community, taking a community-academic partnership approach is critical

The community-academic partnership supporting this work (Drahota et al., 2016) was the foundation of everything that was accomplished. Community partners, including policy-makers at DBHIDS, leadership at community mental health clinics, youth and families impacted by trauma, and the university-based evaluation team came together in various ways through PACTS in order to achieve the goal of improving access to TF-CBT in the City of Philadelphia. For example, all client-level outcome measures used in the client evaluation (not reported in this manuscript) were vetted from a youth community advisory board, who specifically selected the measures used in the evaluation. This allowed for the use of outcome measures that were both clinically relevant but acceptable to youth and families.

7. More work is needed to guide decisions on how to handle the agencies that struggle most with EBP implementation and sustainment

A big challenge has been the balance of sustaining the implementation of EBPs and dismissing agencies that are not meeting the criteria to be considered TF-CBT providers from the system perspective. Since 2011, three agencies that received training and consultation in TF-CBT are no longer formally involved in the system initiative to implement TF-CBT (i.e., no longer sustaining the implementation of TF-CBT). Two agencies were lost due to the closing of their behavioral health services. One agency was asked to leave due to inability to meet requirements, including identifying and treating youth with TF-CBT. Balancing decision-making around when to keep working with struggling agencies versus dismissing them from efforts to implement EBP is an area that warrants future research. Organizational readiness (Scaccia et al., 2015), or how ready an organization is to implement an innovation, can provide important insights into this process.

Moving forward with a trauma-informed behavioral health system

The public behavioral health system plays a primary role in supporting the behavioral health needs of children and youth in urban communities. Previous studies have highlighted the high percentage of children and youth experiencing traumatic events (Ford et al., 1999). Although the evidence base of robust treatment practices to address trauma has grown, “current policy and practice responses do not reflect the urgency, depth or quality required by the high level of need, low impact of many current efforts, and limited community-based service capacity,” (National Center for Children in Poverty, 2007, pp. 3).

Maintaining the use of EBP after the initial implementation period ends is a critical challenge in the public health sector, and comparatively less is known in the implementation science literature about how to enhance sustainability compared to initial implementation (Stirman et al., 2012). A benefit of Philadelphia’s payer (i.e., CBH) being integrated within DBHIDS is the ability to sustain initiatives beyond grant cycles. This has begun with the city provision of an enhanced rate for the use of TF-CBT. However, while DBHIDS and CBH will continue to support the initiative, the significant funding currently available from grant funding will not exist once the grant ends. Here again, it is useful to draw on the implementation science literature, which has highlighted nine core domains that affect capacity for program sustainability: political support, funding stability, partnerships, organizational capacity, program evaluation, program adaptation, communications, public health impacts, and strategic planning (Schell et al., 2013). Rather than viewing sustainability as a static end-point, newer models of sustainability emphasize continued learning and problem-solving among stakeholders, as well as the ongoing adaptation of interventions, with a primary focus on fit between interventions and context (Chambers, Glasgow, & Stange, 2013). Taken together, this suggests that in addition to outside support from DBHIDS and CBH, agencies implementing TF-CBT will need to work together to support each other as they move into the sustainment phase of implementation.

Several steps recently have been taken to move the responsibility from the grant personnel to agency leaders. First, a monthly web-based supervisors meeting to focus on the supervision of EBP and to allow for stakeholders to share challenges to successful implementation (e.g., turnover, lack of agency support, serving children and families who experience ongoing chronic stress) has been implemented to allow for collaborative problem-solving about how to best address these barriers. Second, meetings with agency leadership to discuss trauma work after the grant and to understand how to consolidate the gains made (e.g., maintaining the same number of agencies but increasing the depth of clinicians receiving training and holding new meetings with child-serving systems regarding cross-system support). It is likely that not all current agencies will continue to offer TF-CBT without ongoing support; however, many of the agencies have established strong relationships and will continue to collaborate in the future. Other future directions include expanding EBP offered to providers, including those for disruptive behavior disorders, a primary presenting concern for many traumatized youth, partnering with settings where youth spend their time, such as schools; and deliberately targeting the most vulnerable youth in the system (young children ages 2–6 years, LGBTQ youth, commercially sexually exploited children, and intentionally injured youth).

Conclusion

Building a trauma-informed behavioral health system in the City of Philadelphia has been a hugely informative learning experience. Preliminary findings and lessons learned presented here can help inform other large-scale efforts to create trauma-informed behavioral health systems (Grasso et al., 2013; Ko et al., 2008).

Highlights.

Many systems are moving towards incorporating a trauma-informed approach.

We describe a case study of building a trauma-informed public behavioral health system.

We present data on implementation determinants and outcomes.

We present lessons learned based on this experience.