Implementing trauma-focused cognitive behavioral therapy in Philadelphia: A 10-year evaluation

Abstract

Background: In 2012, Philadelphia's Department of Behavioral Health and Intellectual disAbility Services (DBHIDS) developed an initiative to implement an evidence-based treatment for posttraumatic stress disorder (PTSD), trauma-focused cognitive behavioral therapy (TF-CBT), across the city's behavioral health system. This report evaluates the initiative's 10-year implementation and effectiveness outcomes. Method: The Exploration, Preparation, Implementation, and Sustainment framework guided our implementation evaluation. The implementation outcomes include adoption, reach, and sustainment; these were obtained during regular evaluation data collection from publicly funded behavioral health agencies participating in the TF-CBT initiative. We analyze effectiveness outcomes (i.e., changes in PTSD symptoms) from a subset of patients receiving TF-CBT, which were collected in 6-month intervals by our research team between 2013 and 2021. Results: From 2012 to 2021, DBHIDS trained 478 clinicians in TF-CBT across 20 behavioral health agencies. During this time, 23,401 youths were screened for potentially traumatic events and PTSD symptoms, and 7,550 youths received TF-CBT. Through the TF-CBT initiative, the city expanded the network of TF-CBT providers from 3 to 20 agencies. DBHIDS sustained this network by maintaining the participation of 16 behavioral health agencies over the course of a decade. The subset of 202 youths who were evaluated to assess TF-CBT effectiveness was drawn from 94 therapists and 20 agencies across Philadelphia. All participating youths completed a baseline assessment, and 151 (75%) completed at least one follow-up assessment. Linear mixed-effects models accounting for observations nested within participants and nested within clinicians found that treatment significantly reduced PTSD symptoms. Conclusion: Between 2012 and 2021, DBHIDS successfully implemented and sustained TF-CBT across the city's behavioral health system. Adoption, reach, and sustainment of TF-CBT were high. Despite the considerable adverse experiences faced by youths seeking treatment in Philadelphia's behavioral health system, TF-CBT was effective. Future directions to improve TF-CBT implementation in the next iteration of the initiative are described.

Plain Language Summary

This practical implementation report describes a 10-year effort by the city of Philadelphia to develop a trauma-informed behavioral health system, making this report one of the longest evaluations of an implementation initiative in a large metropolitan area in the United States. In particular, the report describes the implementation and effectiveness outcomes of Philadelphia's Department of Behavioral Health and Intellectual disAbility Services (DBHIDS)'s implementation of the evidence-based treatment for posttraumatic stress disorder (PTSD), trauma-focused cognitive behavioral therapy (TF-CBT), across the city's behavioral health agencies. From 2012 to 2021, DBHIDS trained 478 clinicians in TF-CBT across 20 behavioral health agencies. During this time, 23,401 youths were screened for posttraumatic stress symptoms, and 7,550 youths received TF-CBT. A subset of 202 youths receiving TF-CBT from 94 therapists across 20 Philadelphia agencies were evaluated to assess the initiative's effectiveness. Linear mixed-effects models revealed that youths receiving TF-CBT from DBHIDS-trained clinicians saw their PTSD symptoms significantly reduce. The initiative's success in adoption, reach, sustainment, and effectiveness reveals the promise of sustained, multipronged, community-partnered implementation initiatives. In the future, researchers and policymakers must account for and address the structural and financial barriers that hinder these community-partnered implementation efforts from realizing their full potential in improving population health.

Introduction

Children and adolescents living in socioeconomically disadvantaged contexts are more likely to be exposed to traumatic events, are more likely to develop posttraumatic stress disorder (PTSD) in the wake of these exposures, and are less likely to recover from PTSD compared to youth from socioeconomically advantaged contexts (Copeland et al., 2007; Costello et al., 2002; McLaughlin et al., 2013; Peverill et al., 2021). Stress and adverse experiences compound over the lifespan and impose significant burdens for children and families who have the least access to material support and psychological services. Local and state governments have sought to improve the quality of trauma-informed behavioral health services for publicly insured youth to combat these vast social and health inequities. This practical implementation report describes Philadelphia's efforts to develop a trauma-informed child behavioral health system over the past decade to meet the high needs of the city's youth.

Philadelphia is a large, racially and ethnically diverse, and socioeconomically unequal city (Central Philadelphia Development Corporation, 2017). Almost a quarter of Philadelphians live below the poverty line, a large share of whom are youths (American Community Survey, 2021). Most Philadelphians under the age of 19 (61%) are Medicaid-insured (American Community Survey, 2020). In addition to experiencing material deprivation, Philadelphia youths are exposed to significant interpersonal and community violence (Beard et al., 2021; Cronholm et al., 2015; Turner et al., 2019).

In response to the high incidence of trauma, Philadelphia's Department of Behavioral Health and Intellectual disAbility Services (DBHIDS), which oversees the city's public behavioral health services, developed a large-scale initiative to improve the quality of trauma-informed services for the city's publicly insured youth. In 2012, DBHIDS applied for and was awarded a National Child Traumatic Stress Initiative Community Treatment and Service Center grant from the Substance Abuse and Mental Health Services Administration (SAMHSA) to establish the Philadelphia Alliance for Child Trauma Services (PACTS). DBHIDS charged its sole Medicaid-managed care organization, the nonprofit 501c(3) Community Behavioral Health (CBH), that oversees all federal and state Medicaid behavioral health service dollars in Philadelphia county, with spearheading the PACTS initiative (Community Behavioral Health, 2020). Since receiving the grant in 2012, DBHIDS has sustained the PACTS initiative through grant and a mix of city, state, and federal funding.

The overarching aim of PACTS is to create a trauma-informed publicly funded behavioral health system (Beidas, Adams et al., 2016). This multicomponent initiative involves (1) developing the infrastructure and technical capacity for trauma screening, assessment, and referrals; (2) building partnerships between behavioral health providers and other child-serving systems (e.g., schools, child advocacy centers, child welfare organizations); (3) providing training and consultation to clinicians in evidence-based treatments for PTSD, namely, trauma-focused cognitive behavioral therapy (TF-CBT; Cohen et al., 2010); and (4) enhancing clinician reimbursement rates to incentivize evidence-based treatment use. PACTS is designed to iterate and improve over time by collecting implementation outcomes.

The current investigation seeks to, first, provide an in-depth description of this implementation initiative and implementation outcomes from 2012 to 2021 using the Exploration, Preparation, Implementation, and Sustainment (EPIS) framework, a leading implementation science framework (Aarons et al., 2011). We describe the following implementation outcomes in depth: adoption, reach, and sustainability (Glasgow et al., 1999; Proctor et al., 2011). Second, we describe the effectiveness outcomes (i.e., changes in PTSD symptoms) for youth receiving TF-CBT over the period during which youth outcomes were collected (2013–2021). It is worth noting that the PACTS initiative is embedded in a broader multisectoral effort by DBHIDS to respond to the needs of families impacted by trauma and the social determinants of health. The current paper focuses on TF-CBT implementation; however, PACTS has been implemented alongside other important policy, practice, and program initiatives to strengthen the city's physical and behavioral health services.

Implementation Initiative and Outcomes

The implementation initiative's phases and corresponding implementation strategies and outcomes are depicted in Table 1.

Table 1.

Implementation Phases and Implementation Strategies According to the Exploration, Preparation, Implementation, and Sustainment (EPIS) Framework.

EPIS phase	Implementation strategies	Implementation outcome measure	Implementation outcome(s)
Exploration phase	Philadelphia's Department of Behavioral Health and Intellectual disAbility Services (DBHIDS) met with community and clinical partners in the city's public behavioral health agencies and the country's leading trauma treatment developers	Adoption	478 therapists trained in TF-CBT across 20 public behavioral health agencies
	DBHIDS piloted a small Trauma Initiative in three of the city's public behavioral health agencies
	DBHIDS applied for a Substance Abuse and Mental Health Services Administration grant
	DBHIDS partnered with academic researchers at the University of Pennsylvania to conduct the program evaluation
Preparation phase	DBHIDS invited its public behavioral health agencies to send clinicians to participate in ongoing training and consultation by a certified master trauma-focused cognitive behavioral therapy (TF-CBT) trainer	Reach	23,401 youths screened for posttraumatic stress disorder (PTSD) since 2012
	Clinicians participating in the training attended 2 days of didactics and ongoing biweekly TF-CBT consultation delivered remotely for 8 months
	DBHIDS staff and leaders met with agency leaders to keep them engaged in the TF-CBT initiative
	DBHIDS established the Evidence-Based Practice and Innovation Center in 2013 to provide technical and administrative support for the initiative
Implementation phase	DBHIDS built capacity for trauma treatment by training clinicians in evidence-based trauma screening and assessment	Reach	7,550 youths received TF-CBT since 2012
	DBHIDS developed several communication channels (e.g., weekly emails, a regular newsletter) to disseminate information about the TF-CBT initiative with behavioral health agencies and clinicians
	DBHIDS developed a Youth Advisory Board to consult on implementation efforts
	DBHIDS worked with Community Behavioral Health—the city's Medicaid-managed care behavioral health carve-out—to institute enhanced reimbursement rates for TF-CBT use
	DBHIDS partnered with community organizations and nonprofits across Philadelphia to expand the reach of the TF-CBT initiative and to co-develop trauma-informed programming
Sustainment phase	DBHIDS continues to apply for external funding to ensure the financial viability of the program and to build the internal capacity to sustain it	Sustainability	Between the first wave of Philadelphia Alliance for Child Trauma Services (PACTS) and the second wave of PACTS, the initiative grew by three agencies (with one agency no longer participating)
	DBHIDS uses practice-based evidence from clinicians, agency leaders, researchers, and the Youth Advisory Broad to increase the program's reach and acceptability

Implementation Initiative

Exploration Phase

In the early 2000s, DBHIDS began exploring the city's capacity for trauma-informed services and worked to identify the infrastructure necessary to build an integrated trauma-informed behavioral health system (Woll, 2013). This exploration phase involved gathering community and clinical partners to develop a plan to realize DBHIDS’ vision. As part of this planning, in 2010, DBHIDS invited the country's leading experts in trauma-informed interventions—including Judith Cohen, MD, TF-CBT co-developer; Steven Berkowitz, MD, Child and Family Traumatic Stress Intervention co-developer; Edna Foa, PhD, Prolonged Exposure developer; and Sandra Bloom, MD, co-developer of the Sanctuary Model, an organizational intervention to improve trauma-informed care—to devise a plan to disseminate and implement evidence-based trauma-informed interventions across the city's agencies. In 2011, the city piloted the Trauma Initiative, training organizations in the Sanctuary Model and training clinicians in evidence-based PTSD treatments in three of the city's public sector agencies that were already providing specialty PTSD treatment (Department of Behavioral Health and Intellectual disAbility Services, 2013). Recognizing the need to continue to expand these services and extend the geographic coverage of trauma-informed services, the city applied for and was awarded a $1.6 million SAMHSA grant in 2012 to form PACTS. DBHIDS also partnered with academic researchers at the University of Pennsylvania (Penn) to evaluate implementation and effectiveness outcomes.

Preparation Phase

To establish a network of clinicians trained in evidence-based PTSD assessment and treatments for youth, DBHIDS engaged behavioral health agencies to participate in the first wave of PACTS. The city focused its efforts on training clinicians in TF-CBT given its robust evidence base (Xiang et al., 2021). DBHIDS engaged this network of agencies and invited them to send clinicians to participate in free annual training and consultation by a TF-CBT-certified master trainer and to send nonclinical staff to receive TF-CBT technical assistance training.

Since 2012, DBHIDS has trained clinicians in TF-CBT annually in cohorts of 30–40 people. Clinician training involves 2 days of didactics and biweekly consultation with a TF-CBT-certified master trainer for 8 months. Consultation is delivered in a remote, group format of no more than 10 participants and involves clinicians presenting and receiving consultation on their TF-CBT cases. Clinicians commit to attending 13 of the 16 consultation calls and present on their cases twice. Each agency designates an internal supervisor who provides weekly TF-CBT supervision. In addition to providing annual TF-CBT training, DBHIDS offers annual booster trainings on TF-CBT and related clinical topics to keep clinicians engaged and learning. Since 2012, DBHIDS has met with PACTS agency leaders annually and has held research and clinical presentations to ensure that clinicians and agency leaders remain informed about PACTS’ success. In 2013, DBHIDS also established its own Evidence-Based Practice and Innovation Center, which provides implementation support for the city's implementation initiatives related to behavioral health (Beidas et al., 2019).

Implementation Phase

DBHIDS developed multilevel strategies to successfully disseminate and implement TF-CBT. To complement the TF-CBT training, the city began training clinicians in evidence-based PTSD screening and assessment to build service utilization capacity (Beidas, Adams, et al., 2016). DBHIDS regularly disseminates information about PACTS and engages partners by sending weekly emails, a regular newsletter, and ad hoc technical assistance to support agency and clinician TF-CBT implementation. Since 2012, DBHIDS has also held quarterly coordinating committee meetings to seek feedback from agency leaders, clinicians, and youths to iteratively improve the implementation of TF-CBT.

To ensure a patient-centered approach, the city created a PACTS Youth Advisory Board, designed to integrate the perspectives of youth receiving TF-CBT across the city's public agencies. Adolescents collaborated with PACTS leaders, agencies, and clinicians to ensure that TF-CBT implementation integrated their perspectives. The Youth Advisory Board had four members who had previously received TF-CBT from the city's agencies.

After the first 4 years (i.e., first wave) of PACTS from 2012 to 2016, the city reapplied for and received $2 million in grant funding from SAMHSA in 2016 and $1.6 million in 2022. In the second wave of PACTS, from 2016 to 2021, DBHIDS sought to expand the initiative and reach more vulnerable youth. The city focused its efforts on reaching younger children (ages 2–7), lesbian, gay, bisexual, transgender, and queer/questioning (LGBTQ+) youth, commercially sexually exploited children, unaccompanied minors, and intentionally injured youth. During the second wave of PACTS, DBHIDS continued its training and consultation efforts and sought to further develop community partnerships and integration of services among child-serving systems (advocacy groups, community organizations, schools, primary care practices, and the family court system). DBHIDS also worked with CBH to provide an enhanced reimbursement rate for clinicians using TF-CBT. DBHIDS developed a partnership with local immigrant and refugee-serving organizations to enhance the city's capacity to serve unaccompanied minors. DBHIDS also partnered with a local LGBTQ+ community-based organization to enhance TF-CBT implementation for LGBTQ+ youth and to disseminate information about the availability of TF-CBT for youths. DBHIDS worked with an anti-trafficking coalition to design a screening tool and decision tree to connect tracking survivors to resources. DBHIDS collaborated with a trafficking-specific family court program to develop a TF-CBT program for youth in detention.

Sustainment Phase

To ensure the sustainability of PACTS, DBHIDS has continued to apply for external federal funding to guarantee the initiative's viability as the system develops its internal infrastructure and capacity. DBHIDS has successfully received federal funding for PACTS since 2012; the focus of its most recent grant (awarded in 2022) is to develop financial and infrastructural strategies to sustain PACTS without needing to rely on external federal funding. In addition to developing strategies to internally support the PACTS initiative, PACTS has used practice-based evidence to refine and improve the initiative. The city has consulted with its Youth Advisory Board to increase demand for TF-CBT and to make TF-CBT services more patient-centered. The Youth Advisory Board developed several guides for clinicians working with adolescents exposed to trauma that were disseminated to PACTS behavioral health agencies and clinicians (see Supplemental Material for an example).

In addition to partnering with a team of Penn researchers to regularly collect evaluation data, the city has also developed collaborations with other academic researchers to improve the program. DBHIDS continues to direct several research and quality improvement projects that collect data from payers, agency leaders, clinicians, nonclinical staff, and youth to continue to identify barriers and facilitators to TF-CBT implementation.

Implementation Outcomes

Implementation outcomes include (1) adoption, as measured by the number of agencies and clinicians who have participated in PACTS from 2012 to 2021; (2) reach, as measured by the number of youth screened for PTSD and receiving TF-CBT through PACTS from 2012 to 2021; and (3) sustainability, as measured by the number of agencies and clinicians who remained in the PACTS initiative from the first wave (2012–2016) to the second wave (2016–2021). DBHIDS collected these data from PACTS agencies.

Implementation Outcome Results

Adoption

Since its inception in 2012, PACTS has provided TF-CBT training and support to 478 therapists from 20 public behavioral health agencies. Roughly 30–40 clinicians participate each year.

Reach

Since 2012, 23,401 youths have been screened for exposure to potentially traumatic events and PTSD symptoms. Across the two waves, 7,550 youths received TF-CBT. For context, each year, CBH serves approximately 30,000 children and adolescents under 18; in 2021, for example, CBH served 28,852 youth (Community Behavioral Health, 2022).

Sustainability

In the first wave of PACTS, 17 agencies participated in the initiative, and in the second wave, 19 agencies participated in the initiative. Between PACTS 1 and PACTS 2, three agencies were added, and one no longer continued to be part of PACTS. Over the course of a decade, the PACTS initiative sustained the participation of 16 agencies.

Effectiveness Evaluation Methods and Outcomes

We evaluated PTSD outcomes on a subset of youths who received TF-CBT through PACTS. Effectiveness outcomes of the first wave of PACTS have been previously published (Rudd et al., 2019).

Study Participants and Procedure

All youths between the ages of 3–21 receiving TF-CBT by a PACTS-trained clinician were eligible for the evaluation, led by the Penn evaluation team. The evaluation team attended biweekly meetings with PACTS leadership at CBH to co-develop recruitment strategies and ensure the project met evaluation goals. They also presented at quarterly meetings with the city's agency leaders to inform them about PACTS and to invite them to participate in the evaluation. The evaluation team sent weekly emails to clinicians and supervisors trained through PACTS inquiring about any new, potentially eligible patients for an evaluation.

Once a PACTS clinician informed the evaluation team of potential new participants, the team asked the clinician to acquire verbal permission from the patient and/or their caregiver (as applicable) to be contacted by the evaluation team. Once permission was granted, the evaluation team contacted the youth and/or caregiver to schedule a baseline assessment (i.e., a symptom assessment conducted within the first four sessions of treatment initiation). The evaluation team administered evaluation measures to youths (ages 11–21) and their caregivers (for youths ages 3–17) at the agencies where they were receiving treatment before the COVID-19 pandemic. Due to the coronavirus pandemic, in March 2020 behavioral health services shifted to teletherapy, and evaluations were conducted remotely. Evaluations occurred at treatment initiation (at baseline) and every 6 months until treatment termination. At treatment termination, a final assessment was conducted. All procedures were approved by the City of Philadelphia Institutional Review Board. All measures were determined in collaboration with the PACTS Youth Advisory Board.

PTSD Measures

Across both waves of data collection, participants’ PTSD symptoms were evaluated using measures that were psychometrically validated for their age group.

Wave 1 Measures

Trauma Symptom Checklist for Young Children (TSCYC)

Caregivers of youths ages 3–7 completed the TSCYC (Briere et al., 2001, 2005). The TSCYC is a 90-item symptom scale that prompts caregivers to rate the frequency of each symptom occurring in the past month on a 4-point scale (1 = not at all; 2 = sometimes; 3 = often; 4 = very often). The TSCYC produces eight clinical scales and a posttraumatic stress summary scale, which totals the arousal, avoidance, and intrusion symptoms, mapping onto the three symptom clusters of the Diagnostic and Statistical Manual of Mental Disorders, 4th ed. (DSM-IV) criteria for PTSD. Raw scores for the arousal, avoidance, and intrusion scales range from 9 to 36, and raw scores for the posttraumatic stress summary scale range from 27 to 108. Raw scores on the TSCYC are converted into T-scores manually. Higher T-scores (≥70) are considered clinically significant. The TSCYC clinical scales have good to excellent internal consistency (α = .81–.93; Briere et al., 2001, 2005). The test–retest reliability of the TSCYC (r = .79) is acceptable. The posttraumatic stress summary scale was the primary TSCYC outcome.

Child PTSD Symptom Scale-4 (CPSS-4)

Caregivers of youths ages 8–10 and youths ages 11–18 completed the CPSS-4 (Foa et al., 2001) in their preferred language (English or Spanish). The English CPSS-4 is a 24-item symptom scale that measures DSM-IV PTSD symptoms and functional impairment resulting from those symptoms. The first 17 items assess PTSD symptom severity and map onto DSM-IV PTSD symptoms. Respondents rate how often each symptom has bothered them in the past 2 weeks on the following scale: 0 = not at all or only at one time; 1 = once a week or less/once in a while; 2 = two to four times a week/half the time; or 3 = five or more times a week/almost always. These first 17 items are summed to calculate the PTSD Symptom Severity subscale, which ranges from 0 to 51. The CPSS-4 also includes seven functional impairment questions which prompt the participant to rate the areas of life disrupted by the previously described symptoms during the past 2 weeks. The Functional Impairment subscale involves summing the dichotomous (yes/no) responses to these items. Higher scores on both subscales suggest more PTSD symptom severity and impairment. Clinically significant PTSD is indicated by PTSD Symptom Severity subscale scores of 16 or greater. The CPSS-4 has good internal consistency on the Symptom Severity subscale and on the Functional Impairment subscale (both α = .89; Foa et al., 2001). The CPSS-4 Symptom Severity subscale also has excellent test–retest reliability (r = .84), and the Functional Impairment subscale has very good reliability (r = .70). The Spanish version of the CPSS-4 is also psychometrically reliable and valid, with excellent internal consistency on the Symptom Severity subscale (α = .88; Meyer et al., 2015). The PTSD Symptom Severity subscale was the primary CPSS-4 outcome.

Wave 2 Measures

TSCYC

The TSCYC as described above was administered to caregivers of youth ages 3–7. The posttraumatic stress summary scale was the primary TSCYC outcome.

Child PTSD Symptom Scale-5 (CPSS-5)

The CPSS-4 was updated to align with the DSM-5-defined criteria for PTSD (Foa et al., 2018) and was used for Wave 2 with caregivers of youth ages 8–10 and youth ages 11–18. As opposed to the CPSS-4, the CPSS-5-interviewer version (CPSS-5-I) prompts participants to rate their symptoms and functional impairment in the past month (as opposed to the past 2 weeks). The CPSS-5-I is composed of 20 symptom items that sum to the Symptom Severity subscale, which ranges from 0 to 80. Participants completing the CPSS-5-I rate how often each symptom has bothered them in the past month on the scale: 0 = not at all; 1 = once a week or less/a little; 2 = two to three times a week/somewhat; 3 = four to five or more times a week/a lot; or 4 = six or more times a week/almost always. The CPSS-5-I Functional Impairment subscale asks the same questions as the CPSS-4, but rather than using a dichotomous response scale (yes/no), it uses the same 0–4 rating scale as the Symptom Severity subscale. Scores on the functional impairment scale range from 0 to 28. The CPSS-5-I has excellent internal consistency (α = .92) and excellent test–retest reliability (r = .93; Foa et al., 2018). For the CPSS-5-I, a cutoff score of 31 on the Symptom Severity subscale indicates a probable diagnosis of PTSD; this scale was used as the primary CPSS-5-I outcome. The CPSS-5-I was not yet available in Spanish during the second wave of collection; Spanish-speaking youths or caregivers of youths ages 8–18 completed the CPSS-4. The Symptom Severity subscale was the primary CPSS-4 outcome.

PTSD Symptom Scale for DSM-5-Interviewer Version (PSSI-5)

Young people ages 19–21 completed the PSSI-5 (Foa et al., 2016). The PSSI-5 consists of 24 items, the first 20 of which measure PTSD symptoms and prompt youths to rate the frequency and severity with which they experience these symptoms on a scale of 0–4 (0 = not at all; 1 = once a week or less/a little; 2 = two to three times a week/somewhat; 3 = four to five times a week/very much; or 4 = six or more times a week/severe). The sum of the first 20 items yields a total PTSD Symptom Severity subscale (ranging from 0 to 80); a cutoff score of 23 on this subscale suggests a probable PTSD diagnosis. Participants are then prompted to answer four questions related to distress and interference as well as symptom onset and duration. The PSSI-5 has good internal consistency (α = .89) and test–retest reliability (r = .87; Foa et al., 2016). The PTSD Symptom Severity subscale was the primary PSSI-5 outcome.

Analytic Method

To evaluate the effectiveness of TF-CBT across both waves, a proportion of maximum (POM) scaling method was conducted to convert the primary PTSD outcome measures to the same scale so the scores could be used as repeated measures for individual participants (Little, 2013). POM scaling involves dividing the difference between the observed score and the minimum possible score by the difference between the maximum and minimum scores on a measure (i.e., POM = [observed – minimum]/[maximum – minimum]); scores range from 0 to 1. POM scaling is the preferred method of score transformation for longitudinal studies (Moeller, 2015). For the combined Wave 1 and 2 analyses, linear mixed-effects models with restricted maximum likelihood estimation, using random intercepts to account for observations nested within participants and within clinicians, were used to examine change in symptoms over time and to account for the missingness in repeated observations (Bates et al., 2015; Twisk et al., 2020). Age was selected as a covariate given some evidence that TF-CBT may be more effective for older adolescents (Hoogsteder et al., 2022). We did not select other covariates given that models failed to converge, though gender was considered as a covariate given research suggesting potential gender differences in TF-CBT treatment response (Ascienzo et al., 2022). Independent variables included timepoint and age, and the dependent variable was PTSD symptom severity (i.e., POM score). Level 4 models (participants nested within clinicians nested within agencies) were not possible due to the sample size and were inappropriate because the intraclass correlation coefficient among agencies was low (.03).

For these analyses, all assumptions of linear mixed-effects models were assessed. The lme4 package in R was used for analyses. To determine whether there were differences between PACTS Wave 1 and 2 participants, a two-proportion z-test assessed differences in race (i.e., the proportion of people of color), ethnicity, and gender; a two-sample t-test assessed age differences; and the nonparametric Wilcoxon rank-sum test assessed differences in youths per agency.

Results

Descriptive statistics of the samples are presented in Table 2. Overall, 202 youths completed baseline evaluations across both waves of data collection. During the first wave, 114 completed a baseline assessment, 79 youths (69%) completed at least one follow-up evaluation, and 47 (41%) completed a termination evaluation. Of the 88 who completed the baseline assessment in the second wave, 72 (82%) completed at least one follow-up evaluation, and 39 (44%) completed a termination evaluation. Across both waves, 151 (75%) completed at least one follow-up evaluation, and 86 (43%) completed a termination evaluation. No youth participating in the study had a follow-up evaluation beyond 18 months. For the demographics across waves, 20 agencies were represented, with a median of five youths per agency (interquartile range = 6, range = 1–44 youths per agency). The average age of youths receiving TF-CBT was 12.4 years old (SD = 3.95). Most youths identified as female (n = 124, 61%) and Black/African American (n = 114, 56%); over one-third (37%) identified as Hispanic/Latinx. No significant differences were found in demographics across waves including race (z = −0.99, p = .32), ethnicity (z = 0.03, p = .98), gender (z = −1.53, p = .13), age (t = 0.45, p = .66), and number of youths per agency (W = 114.5, p = .84).

Table 2.

Demographic Characteristics of PACTS Participants Across Both Waves of Data Collection.

Characteristic	PACTS Wave 1		PACTS Wave 2		PACTS waves combined
	n/M	%/SD	n/M	%/SD	n/M	%/SD
Sample	114		88		202
Agencies represented	15		16		20
Patients per agency (median, IQR)	5.00	11.00	5.00	6.50	5.00	6.00
Age	12.45	3.97	12.20	3.90	12.34	3.93
Race a
African American, Black	55	48%	59	67%	114	56%
American Indian	4	4%	4	5%	8	4%
Asian	0		4	5%	4	2%
White	17	15%	26	30%	43	21%
Two or more races	23	20%			23	11%
Other	15	13%	8	9%	23	11%
Ethnicity
Hispanic/Latinx	43	38%	33	38%	74	37%
Gender identity b
Female	64	73%	58	66%	124	61%
Male	50	44%	28	32%	80	40%
Genderqueer			1	1%	1	1%
Gender not listed			1	1%	1	1%
Sexuality c
Gay
Lesbian			1	1%	1	1%
Bisexual			10	11%	10	5%
Queer			5	6%	5	2%
Unsure/questioning			4	5%	4	2%
Straight/heterosexual			40	45%	40	20%
Sexuality not listed			2	2%	2	1%

Note. PACTS = Philadelphia Alliance for Child Trauma Services; IQR = interquartile range.

^a Participants endorsed multiple racial identifications, and as such, percentages do not add up to 100%.

^b Expansive gender identity options were not evaluated during the first wave of PACTS.

^c Sexuality was only asked of participants above the age of 11 during the second wave of PACTS. Many participants chose not to respond to these questions.

Results from combining the two waves of data revealed significant improvements in PTSD symptoms. At baseline, average PTSD symptom severity POM scores were .46 (SD = 0.23); at 6 months, M = 0.40 (SD = 0.25); at 12 months, M = 0.37 (SD = 0.29); and at termination, M = 0.33 (SD = 0.25; see Table 3). Linear mixed-effects models revealed significant improvements in PTSD symptoms between baseline (the reference group) and 6 months (β = −.09, p = <.001, 95% confidence interval [CI] [−0.13, −0.04]), 12 months (β = −.14, p = .03, [−0.26, −0.02]), and termination assessments (β = −.13, p < .001, [−0.17, −0.09]; see Table 4). Adjusting for age, this represents a 30% decline in PTSD symptoms from baseline to termination.

Table 3.

Posttraumatic Stress Disorder (PTSD) Symptom Severity Proportion of Maximum (POM) Scores Over Time.

Timepoint	Sample size (n)	PTSD symptom severity POM score
		M	SD
Baseline	201	0.46	0.23
6 months	58	0.40	0.25
12 months	7	0.37	0.29
Termination	86	0.33	0.25

Note. The total sample of participants who completed baseline interviews was 202 youths. However, one participant did not complete PTSD measures at baseline; thus, only 201 youths’ data were available for analysis. POM scores range from 0 to 1.

Table 4.

Linear Mixed-Effects Models for PACTS Wave 1 and 2 Posttraumatic Stress Disorder (PTSD) Measures

Fixed effects	Estimate	SE	t	df	95% CI		p
					LL	UL
Intercept	0.43	0.05	8.26	181.62	0.33	0.54	<.001
6-month follow-up	−0.09	0.02	−3.49	183.80	−0.13	−0.04	<.001
12-month follow-up	−0.14	0.06	−2.20	174.34	−0.26	−0.02	.03
Termination	−0.13	0.02	−6.40	179.83	−0.17	−0.09	<.001
Age	0.00	0.00	0.36	195.91	−0.01	−0.01	.72

Note. Outcome variable is the proportion of maximum score for all PTSD measures collected during both waves of PACTS; scores range from 0 to 1. PACTS = Philadelphia Alliance for Child Trauma Services; AIC = Akaike information criterion; BIC = Bayesian information criterion; SE = standard error; CI = confidence interval; LL = lower limit; UL = upper limit.

Discussion

Philadelphia's decade-long effort to develop a trauma-informed behavioral health system reveals the substantial resource investments, coordination, innovation, and partnerships needed for successful implementation. Using the EPIS framework, we provide an overview of how Philadelphia's DBHIDS began formulating the idea of training providers in screening for and treating PTSD in the early 2000s and executed these plans by partnering with treatment developers, federal funders, academic researchers, the network of public agencies and nonprofits in the city, and behavioral health treatment seekers. DBHIDS trained a large cadre of clinicians in evidence-based PTSD assessment and treatment. The city also leveraged its Medicaid-managed care carve-out to incentivize TF-CBT use by providing an enhanced reimbursement rate for TF-CBT delivery. Through this initiative, DBHIDS cultivated an integrated network of agencies and child-serving systems. DBHIDS also made concerted efforts to maintain engagement with partners through consistent communication, trainings, and events with agencies and youth.

Through these efforts, the city increased evidence-based PTSD assessment and TF-CBT adoption and trained 478 therapists since 2012. In this time, 23,401 youth have been screened for PTSD symptoms, and 7,550 youths received TF-CBT. While no data on youth PTSD prevalence rates exist, one study of adults seeking behavioral health treatment in Philadelphia's public agencies found that of the 10,260 individuals assessed, 58% met the criteria for PTSD—about seven times the national lifetime rate—suggesting a need for the availability of evidence-based PTSD treatments in Philadelphia, which PACTS has endeavored to meet (Pincus et al., 2022). Moreover, the city sustained the PACTS program between 2012 and 2021 by maintaining the participation of most agencies. Outcomes from a subset of PACTS patients over the course of 9 years (2013–2021) revealed that youths receiving TF-CBT saw significant PTSD symptom improvements. Our effectiveness findings add to the robust evidence base establishing TF-CBT's effectiveness for diverse and underresourced youth experiencing significant social stressors and ongoing adversity (Dorsey et al., 2017; Orengo-Aguayo et al., 2022; Patel et al., 2022; Thomas et al., 2022).

A major contributor to the success of PACTS is the unique public behavioral healthcare financing landscape of Philadelphia, which has a single managed care organization, CBH. CBH is the city's single payer, and this facilitates greater standardization, coordination, collaboration, and sustainment of evidence-based treatment implementation across the city's agencies. The city's management of its own funding also means that it can reinvest surplus into evidence-based practice initiatives, which the city has consistently done (Powell et al., 2016). Moreover, CBH's incentives are aligned with DBHIDS's imperative to improve the quality of behavioral health services while ensuring the financial viability of implementation initiatives. This is unlike other cities and states in the United States, which have delegated Medicaid management to several commercial insurers (Perez, 2018). The increased fragmentation and insurers’ focus on cost reduction in other city and state Medicaid systems make the kind of coordination and resource investments that were necessary in PACTS more challenging.

This case study of a decade-long implementation initiative, one of the longest evaluations in a large metropolitan area in the United States, reveals how essential it is to develop partnerships between various stakeholders. PACTS has been successful largely due to sustained coordination and investments from policymakers, payers, researchers, agency leaders, community organizations, clinicians, and patients. DBHIDS leadership identified implementation champions early in the process of developing PACTS and worked collaboratively with these agencies to develop and expand its network of providers. The public–academic partnership between researchers and the city has been especially crucial to the initiative's success. The model is uniquely suited for narrowing the research-to-practice gap due to the bidirectional translation and flow of knowledge (Pellecchia et al., 2018).

Our implementation strategies and outcomes are largely consistent with efforts undertaken by other municipal and state behavioral health systems, who have similarly partnered with academic institutions to implement TF-CBT, resulting in significant implementation successes (Amaya-Jackson et al., 2018; Connors et al., 2021; Kramer et al., 2015; Orengo-Aguayo et al., 2022; Sigel et al., 2013; Webb et al., 2014). The PACTS initiative joins the majority of states implementing TF-CBT that have (1) collaborated with treatment developers to design the TF-CBT initiative; (2) received SAMHSA grants to fund the initiative; (3) focused on agencies serving Medicaid-insured children and adolescents; (4) opted for the training and ongoing consultation model, compared to the learning collaborative model of TF-CBT dissemination; (5) provided ongoing consultation throughout the initiative; and (6) used outcome measures to assess effectiveness outcomes (Sigel et al., 2013). However, PACTS has trained more clinicians and served more youths than most TF-CBT initiatives, potentially due to the duration of Philadelphia's initiative and its unique single-payer behavioral health system (Amaya-Jackson et al., 2018; Connors et al., 2021; Sigel et al., 2013). PACTS has also faced similar barriers as other TF-CBT initiatives such as staff turnover, time and resource constraints, varied organizational support and capacity, and difficulty identifying and assessing appropriate clients for TF-CBT (Amaya-Jackson et al., 2018; Oliver & Lang, 2018; Sigel et al., 2013). To address the latter challenge, PACTS trained clinicians in PTSD assessment and regularly communicated with agencies and community organizations early in the implementation process to build capacity for TF-CBT implementation.

Limitations

Our practical implementation report does have limitations. First, the effectiveness study uses a pre–post design without a control group; clinical improvements in PTSD symptoms may therefore represent regression to the mean or other confounding factors unrelated to TF-CBT treatment. Relatedly, PACTS participation (for agencies and clinicians) was and continues to be voluntary. Not all public agencies or clinicians in Philadelphia participated in the PACTS initiative and not all PACTS agencies, clinicians, or youths participated in the TF-CBT effectiveness research evaluation. It is possible that the agencies and clinicians most invested in quality improvement self-selected into the PACTS initiative. There is evidence to suggest that Philadelphia clinicians willing to participate in the PACTS initiative are highly open to and knowledgeable about evidence-based treatments generally and TF-CBT specifically; they also report high intentions to use TF-CBT and see significant advantages to the treatment (Beidas, Adams, et al., 2016; Frank et al., 2021). Given that PACTS clinicians may be especially committed to improving their practice, the effectiveness outcomes may be partially driven by therapist effects (Lutz & Barkham, 2015; Saxon & Barkham, 2012; Saxon et al., 2017). Youths willing and able to participate in the evaluation may have also been more engaged in treatment than youths who did not participate. Thus, our effectiveness findings may overestimate treatment effects.

Finally, our investigation relies on self-reported data from agencies and clinicians on clinician screening and treatment delivery. Research conducted with Philadelphia clinicians has shown that clinicians’ self-reports overestimate cognitive behavioral therapy adherence (Becker-Haimes et al., 2022). Our data on TF-CBT delivery may therefore be inflated. However, PACTS TF-CBT training and consultation were rigorously conducted with TF-CBT master trainers, during which treatment integrity was emphasized. Thus, the limitations of self-report may be less pervasive for PACTS clinicians.

Future Directions

Lessons learned and future directions are described below and in Table 5.

Table 5.

Future Directions for Implementation Research.

Future direction theme	Implementation barrier	Future implementation research direction	Future policy direction
Implementation financing	Implementation efforts involve clinical and administrative tasks that are crucial, yet resource and time-intensive and not reimbursed by current public mental health financing models	Researchers can identify aspects of implementation that are necessary for successful implementation but not accounted for by current public behavioral health funding models	Sustained financial investments in public behavioral health funding will provide resources to cities, agencies, and clinicians to feasibly implement evidence-based treatments and narrow the research-to-practice gap
Workforce retention	Clinician turnover disrupts implementation efforts	Behavioral health service researchers can engage clinicians to identify predictors of turnover and the supports clinicians want to retain them	State and federal policymakers must improve the working conditions of clinicians to retain the current workforce and attract more clinicians
Multilevel support	Evidence-based behavioral health services cannot resolve the structural conditions that prompt patients to seek trauma services and that make recovery from posttraumatic stress disorder more challenging (e.g., continued exposure to violence and adverse childhood experiences)	Policy researchers can identify policies that prevent traumatic events and behavioral health disorders as well as policies that provide families with multiple levels of support in order for them to optimally benefit from evidence-based treatments	Evidence-based policies that target population behavioral health are necessary to prevent behavioral health disorders and to offer multilevel support to individuals in need of behavioral health services

Implementation Financing

Public behavioral health funding does not fully account for all of the costs of evidence-based treatment implementation, including costs to the system as well as to clinicians’ time for training, consultation, treatment and session planning, and administrative tasks (Bowser et al., 2021; Massatti et al., 2008; Okamura et al., 2018). Agency and DBHIDS leaders have shared that implementation initiatives are costly (Stewart et al., 2016). Consequently, the city has had to continually apply for several federal grants to successfully implement PACTS. In the third wave of PACTS, DBHIDS is exploring ways to develop the internal capacity and infrastructure to sustain the initiative, though funding remains a significant barrier. Researchers invested in partnering with public systems to implement evidence-based treatments can work to identify aspects of implementation that are not accounted for by current behavioral health financing models. Implementation researchers have proposed several ways to do this, including using a fiscal mapping process and evaluating the efficacy of disparate financing strategies (Dopp, Gilbert, et al., 2022; Dopp, Hunter, et al., 2022). Policymakers can then leverage this research to design strategies to comprehensively provide financial support to service systems for implementation initiatives.

Workforce Retention

Consistent with other TF-CBT initiatives, PACTS's most significant implementation barrier has been workforce turnover (Beidas, Adams, et al., 2016; Beidas, Stewart, et al., 2016; Oliver & Lang, 2018; Sigel et al., 2013). Turnover in Philadelphia's public behavioral health agencies is 41%, and in some agencies, it is significantly higher (Adams et al., 2019; Beidas, Marcus et al., 2016). Much like clinicians in other behavioral health systems, Philadelphia clinicians leave the workforce due to inadequate compensation as well as the demanding conditions of public behavioral health work, which generate burnout and secondary traumatic stress (Adams et al., 2019; Association for Behavioral Healthcare, 2022; Last et al., 2022). PACTS clinicians also report wanting additional clinical support (Last et al., 2021). Behavioral health service researchers invested in implementing evidence-based treatments, particularly PTSD treatments which can vicariously traumatize clinicians, must solicit strategies from clinicians to support their treatment delivery. Researchers must also focus on further understanding the predictors of workforce turnover. State and federal policymakers must improve the working conditions of clinicians by developing more peer and supervisory support, providing clinical and technical assistance, improving Medicaid reimbursement and clinicians’ wages, and developing caseload caps to prevent burnout.

Multilevel Support

Philadelphia's public behavioral health system has witnessed an increased demand for services as multiple social, political, and economic crises are affecting population mental health (American Academy of Pediatrics et al., 2021). Youths seeking PTSD treatment in Philadelphia face poverty, housing instability, and other adverse childhood experiences that synergistically compound (Briggs et al., 2021; Last et al., 2020). Youths’ continued exposure to adversity makes trauma recovery more challenging, revealing the need for multilevel supports. Policy researchers can identify and advocate for policies that prevent traumatic events, mitigate the negative psychological sequelae that follow trauma exposure, and provide children and families with resource and service access. Research demonstrates that policies such as Medicaid expansion and the Earned Income Tax Credit not only provide families material resources and access to health insurance but are also associated with lower rates of child maltreatment (Kovski et al., 2022; McGinty et al., 2022). Federal and state policymakers can leverage this evidence to design policies that address population-level behavioral health and provide multilevel support to families in need of a variety of services (Purtle et al., 2020).

Conclusion

Our practical implementation report describes Philadelphia's 10-year effort to implement TF-CBT across the city's publicly funded behavioral health agencies (2012–2021). This implementation effort has led to promising implementation outcomes, specifically adoption, reach, and sustainability. TF-CBT was effective at reducing PTSD symptoms in a subset of youth who participated in the initiative from 2013 to 2021. We outline the unique strengths of this implementation initiative, including the city's collaboration with various partners and unique financing structure. We also propose future directions for research and policy to improve implementation efforts based on lessons learned from the Philadelphia experience.