Provider fidelity and modifications to Cognitive Processing Therapy in a diverse community health clinic: Associations with clinical change

Abstract

Objective:

The purpose of this study is to examine associations between therapist adherence, competence, and modifications of an evidence-based protocol (EBP) delivered in routine clinical care and client outcomes.

Method:

Data were derived from a NIMH-funded implementation-effectiveness hybrid study of Cognitive Processing Therapy (CPT) for PTSD in a diverse community health center. Providers (n=19) treated clients (n=58) as part of their routine clinical care. Clients completed the PCL-S and PHQ-9 at baseline, after each CPT session, and post-treatment. CPT sessions were rated for treatment fidelity and therapist modifications.

Results:

Overall, therapist adherence was high, although it decreased across sessions suggesting potential drift. Therapist competence ratings varied widely. Therapists made on average 1.6 fidelity-consistent and 0.4 fidelity-inconsistent modifications per session. Results show that higher numbers of fidelity-consistent modifications were associated with larger reductions in posttraumatic stress and depressive symptoms. High adherence ratings were associated with greater reductions in depressive symptoms, whereas higher competence ratings were associated with greater reduction in posttraumatic stress symptoms.

Conclusions:

The results highlight the importance of differentially assessing therapist adherence, competence, and modifications to EBP in usual care settings. The findings also suggest that effective EBP delivery in routine care may require minor adaptations to meet client needs, consistent with previous studies. Greater attention to fidelity and adaptation can enhance training so providers can tailor while retaining core components of the intervention.

As one of several evidence-based treatments recommended for posttraumatic stress disorder (PTSD), the efficacy of Cognitive Processing Therapy (CPT; Resick, Monson & Chard, 2006) has been demonstrated both in rigorous controlled trials as well as in less rigorous designs across trauma-exposed populations including: rape and assault survivors, refugees, and torture survivors (Bass et al., 2013; Chard, 2005; Kaysen et al., 2011). The therapy has also been robust to co-occurring diagnoses and to other treatment delivery challenges such as comorbid substance abuse diagnoses, traumatic brain injury, and low levels of literacy (Ahrens & Rexford, 2002; Kaysen et al., 2014; Resick, Nishith, Weaver, Astin, & Feuer, 2002). CPT has excellent long-term outcomes, with treatment gains that may be sustained for over 5 years (Resick, Williams, Suvak, Monson, & Gradus, 2012) and improved health and quality of life outcomes (Galovski, Monson, Bruce, & Resick, 2009). It has been implemented and studied in large-scale mental health systems (e.g., Chard et al., 2012; Monson et al., 2018) but there has been limited study of its effectiveness in low income community-based settings in the United States, where frequently clients present with complex PTSD presentations, including extensive trauma exposures, multiple comorbidities, and reduced resources and capacity.

Clinicians in community mental health settings have thus reported a perceived need to modify protocol treatments to address the multiple contextual challenges experienced by their clients, such as stigma, cultural beliefs about mental health, the impact of poverty, language differences, low literacy, and ongoing violence (Cook, Dinnen, Thompson, Simiola, & Schnurr, 2014; Lau et al., 2017; Marques et al., 2016; Stirman et al., 2013a). Some of these changes, termed adaptations, occur in a planned or purposeful manner with the intention to retain fidelity to the fundamental elements or spirit of the intervention (Lee, Altschul, & Mowbray, 2008; Stirman, Gutner, Edmunds, Evans, & Beidas, 2015; Stirman, Miller, Toder, & Calloway, 2013b). Other forms of modification may be unplanned and occur less systematically due to clinician preferences, challenges that arise in the treatment setting, or individual client-level needs or challenges (Baumann, Cabassa, & Stirman, 2017; Stirman et al., 2015). Understanding which components of an intervention can be modified and which should be maintained is paramount to treatment implementation. Proposed modification frameworks suggest the possibility of adapting peripheral elements while adhering to core treatment elements (Cabassa et al., 2014; Chu & Leino, 2017). In contrast, studies have identified some adaptations in routine care that are not theoretically consistent with the treatment protocols, or that remove certain core treatment elements (Lau et al., 2017; Stirman et al., 2015), suggesting that clinician efforts to modify or adapt treatment may at times have a negative impact on treatment fidelity.

Modification and adaptation, while related to fidelity, are conceptually distinct constructs. Treatment fidelity comprises: (1) adherence, or the degree to which providers deliver the key components of interventions as intended, (2) competence, or skill with which the intervention is delivered, and (3) differentiation from other treatments (Gearing et al., 2011). Fidelity assessment, which focuses on the delivery of central aspects of the intervention (often via checklists), may fail to capture certain types of modifications or adaptations, including small changes to terminology or language, changes to the length of the session or protocol, or the use of elements that are neither prescribed nor proscribed (Stirman, Gamarra, Calloway, Bartlett, & Gutner, 2017). Additionally, many fidelity assessment instruments do not evaluate treatment differentiation, and thus they may not detect the integration of other treatment elements unless these elements displace key intervention elements. Thus, fidelity assessment provides a limited understanding of whether modifications to evidence based protocols (EBP) will enhance, erode, or leave unchanged the effects of the treatment protocol (Stirman et al., 2017).

While the impact of modification and adaptation remains largely under-studied in routine care settings, the impact of aspects of treatment fidelity on clinical outcomes has been considered in several studies, reviews, and a meta-analysis (e.g., Barber, Triffleman, & Marmar, 2007; Strunk, Brotman, & DeRubeis, & Hollon, 2010). While many studies have been conducted to understand the relationship between treatment fidelity and clinical change, a 2010 meta-analysis concluded that overall, fidelity did not appear to be predictive of symptom changes (Webb, DeRubeis, & Barber, 2010). Notably, though, much of the research on fidelity and treatment outcome has been conducted in clinical trials, where there is a restricted range of fidelity due to study procedures. Additionally, few studies have addressed potential temporal interactions between fidelity and symptom change where fidelity changes over the course of the protocol as a function of client characteristics, symptoms, or behaviors (Webb, et al., 2010).

Client characteristics may have complex relationships in particular with treatment fidelity. For example, Strunk and colleagues (2010) found evidence for an overall relationship between fidelity to CBT for depression and symptom change, but also found that competence was more important for clients with high levels of anxiety. Similarly, Hogue and colleagues (2008) found that moderate levels of adherence were associated with stronger outcomes for internalizing behaviors, but higher adherence was associated with better outcomes for adolescents with externalizing problems in multidimensional family therapy. These latter findings suggest that modification to protocols may not negatively impact clinical outcomes. In fact, competent treatment delivery may require not only skill at delivering specific treatment elements, but also subtle adjustments to ensure clients can understand and benefit from treatment (Chu & Leino, 2017; Roth & Pilling, 2008). In settings in which treatment is delivered to culturally diverse and diagnostically complex individuals with different levels of English fluency and literacy, adaptation is perceived as potentially necessary (Chu & Leino, 2017), but whether adaptations to language and content impact treatment outcome remains unknown.

In the absence of empirical evidence, modifications to EBP have been the subject of debate. Whereas some have argued that adaptation is to be expected and may improve treatment effectiveness and sustainability in routine care settings (Chambers, Glasgow, & Strange, 2013), others have cautioned that changes to evidence-based interventions, particularly unplanned or less systematically derived modifications, could result in diminished treatment response (Blakely et al., 1987; Cohen et al., 2008). Although modifications might facilitate implementation and sustainability by improving the fit between the intervention, the target population, and the context into which an EBP is introduced, they may also negatively impact treatment fidelity or compromise clinical outcomes (Stirman et al., 2017).

Although there is some evidence that clients experience clinical improvement when they receive adapted forms of CPT (Bass et al., 2013; Galovski, Blain, Mott, Elwood, & Houle, 2012; Schulz, Resick, Huber, & Griffin, 2006; Weiss et al., 2015), no studies have examined the association between provider modifications to EBPs delivered in routine care and clinical outcomes. Understanding the effectiveness of EBPs, as well as key implementation variables, such as the degree of EBP fidelity, the types of modifications made, and how modification and fidelity are related to clinical outcomes is essential to developing effective and sustainable programs in usual care settings. In this study, we examined associations between fidelity, adaptations, and clinical outcomes, using data collected during an open trial of CPT implementation (in English or Spanish) in a diverse low-income community care setting. We hypothesized that there would be associations between client outcome, as defined by changes in posttraumatic stress and depressive symptoms over the course of treatment, and (1) average number of fidelity-consistent modifications made by the therapist per session, (2) average adherence rating per session, and (3) average competence rating per session. Specifically, we hypothesized that more frequent fidelity-consistent modifications and higher competence ratings would predict larger reductions in symptoms, and that lower levels of adherence would predict less symptom change. Because there is little data available on CPT when not delivered in English, we also sought to explore whether different outcomes were associated with the language of treatment delivery.

Methods

Study Design

Data were derived from a National Institute of Mental Health-funded (NIMH K23 MH096029-01A1) implementation-effectiveness hybrid pilot study of CPT for PTSD in a diverse community health center (CHC) that was approved by the site’s Institutional Review Board. The parent study comprised 3 phases: Phase 1 (Pre-Conditions) employed a mixed-methods approach to identify, assess, and address organizational context and provider-level barriers and facilitators of implementation prior to the pre-implementation (piloting) of CPT (Resick, Monson, & Chard, 2016) within the treatment setting. The primary implementation result of the first phase was the need for a language adaptation (Kilbourne, Neumann, Pincus, Bauer, & Stall, 2007) of the CPT manual (Resick et al., 2008; CPT, cognitive only version without a written trauma account; CPT-C) which led to its translation to Spanish (Valentine et al., 2017).

Phase 2 (Pre-Implementation) piloted the CPT-C manual with providers and clients in English and Spanish. Implementation, provider, and client outcome data were collected to examine if adaptations to the manual were necessary. Through a process of formative evaluation (fully described in Valentine et al., 2017), data suggested the need for the following adaptations: cultural context, context of poverty and ongoing exposure to violence, and adapt the literacy level of the homework assignments and handouts. The key elements of CPT were retained. These results were similar to the factors that drove the manual adaptations made to the CPT-Congo manual, which was found to be efficacious in a clinical trial (Bass et al., 2013). As a result, in collaboration with D.K. (co-author), we modified the CPT-Congo manual to individual delivery mode, make the manual content more relevant to the cultural context, and translated it to Spanish. Additional language adaptations were made to the Spanish version based on results of Phase 2 trial, including simplifying concepts and removing jargon while retaining key principles of CPT. The final manual was named “CPT-Community” (Cognitive Processing Therapy for Community Mental Health Centers; Marques et al., 2016).

In Phase 3 (Implementation), providers were trained to deliver the CPT-Community protocol, and it was tested with providers and clients in English and Spanish. This study examines associations between treatment fidelity, modifications made in routine care, and treatment outcomes.

Participants

Providers were mental health clinicians working in a diverse community health center in the greater Boston area, providing counseling to at least one client with a primary diagnosis of PTSD in either English and/or Spanish, able to train in English, and who agreed to the study procedures (e.g., self-report measures, audio recording of sessions, participation in consultation).

Clients were eligible if they were at least 18 years of age, had a current primary diagnosis of PTSD indicated by their medical record and/or a Posttraumatic Symptom Checklist-Specific Version (PCL-S; Blanchard, Jones-Alexander, Buckley, & Forneris, 1996) score above the clinical cut-off (> 36; Wilkins, Lang, & Norman, 2011), agreed to have their sessions audio recorded for research purposes, and were able to participate in therapy in English or Spanish. Clients were excluded if they were acutely suicidal or homicidal, required hospitalization, had an unstable psychiatric regimen, evidenced current psychosis or mania, had a current primary diagnosis of substance dependence, were prescribed benzodiazepines, or reported concurrent cognitive-behavioral therapy (CBT) or previous CPT treatment.

Measures

Participant baseline information.

A demographic form included questions about gender, age, race, ethnicity, marital status, educational and occupational level, income, and employment history. In addition, providers were asked about the number of years spent working in mental health and at their current clinic, and the most common conditions they treat. Clients were asked about characteristics of work and social functioning, and when they experienced the traumatic event associated with their diagnosis of PTSD.

Post-traumatic Stress Disorder symptoms.

The PCL-S (Weathers, Litz, Herman, Huska, & Keane, 1993), a widely used 17-item self-report measure of Diagnostic and Statistical Manual of Mental Disorders-Fourth Edition (DSM-IV) criteria of PTSD symptoms, was completed by clients to assess PTSD symptoms. Using a 5-point Likert scale (1 = Not at all, 5 = Extremely), clients are asked to rate how much they are bothered by each PTSD symptom in the past week thinking about the index trauma they are working on in treatment, to generate a total score ranging from 17 to 85, with higher scores indicating greater symptom severity. The PCL-S has demonstrated good internal consistency (α > .75), test-retest reliability, convergent validity, and discriminant validity with measures assessing depression, other psychopathology, and physical pain (Wilkins, et al., 2011). For the current study, the PCL-S was translated to Spanish by a medical translation specialist from the Massachusetts General Hospital and back translated by two bilingual staff to ensure internal validity. Providers in the study were trained to review the PCL-S with their clients (as part of CPT treatment), including checking that the client had completed the PCL-S anchored to the index trauma. Cronbach’s alpha was 0.90 for the English version, 0.88 for the Spanish version, and 0.90 for both versions combined at baseline.

Depressive symptoms.

Clients’ depressive symptoms were assessed with the PHQ-9 (Kroenke, et al., 2001) a widely used 9-item self-report measure. Clients note how frequently they have been bothered by symptoms of depression over the last two weeks on a 4-point Likert scale (0 = Not at all, 4 = Nearly every day), generating a total score ranging from 0 to 27. The PHQ-9 has been shown to have strong internal consistency (Cronbach’s α = .89), construct validity and test-retest reliability (Kroenke & Spitzer, 2002; Kroenke et al., 2001). The publicly available Spanish translation of the PHQ-9 was used in this study, and it has been validated in Latino populations, and found to be an acceptable, culturally appropriate, and feasible measure to screen for depressive symptoms in Spanish-speaking populations (Wulsin, Somoza, & Heck, 2002). It has demonstrated strong internal consistency (Cronbach’s α = .84) and construct validity (Baader et al., 2012). For the current study, Cronbach’s alpha was 0.82 for the English version, 0.90 for the Spanish version, and 0.85 for both versions combined at baseline.

Therapist adherence and competence

(TAC; Nishith & Resick, 1994). TAC is an observer measure used to assess provider fidelity (i.e.: adherence and competence) to CPT. The 88 items assess provider adherence (0 = Not adherent; 1 = Adherent) and competence (0 = Not at all competent to 6 = Excellent competence) to specific CPT session components. An overall competency rating (average) is also calculated for each session. Previous studies have found high inter-rater reliability (97-100% agreement; Resick et al., 2002, Resick et al., 2008), and convergent validity between competence scores on the TAC and a validated cognitive therapy fidelity-assessment instrument (r=.82; Shaw et al., 1999; Stirman et al., 2011; Vallis, Shaw, & Dobson, 1986). For this study, the TAC was adapted to assess provider adherence and competence specific to the manuals used in the study (i.e., CPT-C vs. CPT-Community).

Modifications to the treatment protocol.

The Modification and Adaptation Checklist (MAC; based on Stirman, Miller, Toder, & Calloway, 2013b) is a session-level version of a measure previously used to assess adaptations made to treatment (Lau et al., 2017). Observers rate the presence or absence of 14 possible modifications made by providers (Stirman et al., 2013b) when compared to the assigned CPT protocol (e.g., Phase 2: CPT-C or Phase 3: CPT-Community). For this study, an additional modification type to capture changes to session format that deviated from typical CPT structure and style was added. This measure has demonstrated strong rater agreement (Cohen’s kappa = 0.87; Stirman et al., 2013b).

Modifications to evidence-based treatments were conceptualized to be either “fidelity-consistent” (modifications that do not change core elements of the treatment in such a way that reduces adherence to the intervention protocol, or make it challenging to differentiate between treatments), or “fidelity-inconsistent” (modifications that do alter the inclusion or delivery of core elements of the intervention protocol, and/or make it difficult to differentiate between treatments) (Stirman et al., 2015). Examples of fidelity-consistent modifications include “tailoring/tweaking/refining”, or modifications that include changes to the intervention but do not change the major intervention principles and techniques (e.g., modifying the language, cultural adaptations), and “lengthening sessions”, or modifications that extend session time to more than 60 minutes. Examples of fidelity-inconsistent modifications include “removing/skipping intervention modules or components”, where the modifications drop elements of the standard treatment’s intervention of interest, such as a CPT module, worksheet, or psychoeducation element, and “integrating other approaches”, or modifications that integrate techniques from other treatment approaches into the intervention¹.

Procedure

All 26 providers at the study site clinic were invited to participate through a presentation during a staff meeting. Interested providers met with a research coordinator to sign informed consent. Enrolled providers were randomly assigned to be part of the 1^st (n=10) or 2^nd (n=9) wave of training which corresponded to the CPT manual in which they were trained, and received a 3-day, one-time CPT training delivered by a certified CPT trainer and the study principal investigator. Treatment consisted of either CPT-C (cognitive-only, without written account; Phase 2) or the aforementioned adapted CPT-Congo manual (CPT-Community; Phase 3) protocols, delivered in English or Spanish, depending on client preference and provider ability². Providers participated in in-person weekly group consultation meetings during the study with the primary investigator. Group consultations focused on case discussion through reviewing of case conceptualization, and helping providers apply CPT to their individual patients using the Socratic method and modeling. As needed and possible, therapists brought in sections of recorded audiotape from sessions to review during the consultations meetings.

As part of routine clinical care, providers identified potential clients using the eligibility criteria (i.e., who were assessed as potentially having PTSD based on the provider’s clinical judgment and/or PCL-S scores, as well as those deemed to be clinically appropriate to be part of the study, such that they were not an immediate danger to self or others, not currently psychotic, etc), and invited them to participate in the study. Both new clients and those already engaged in treatment were eligible to participate, as long as they met study criteria. Interested clients met with a research staff member to sign informed consent. At baseline, after each CPT session, and at the end of treatment, clients completed the PCL-S anchored to their primary trauma and the PHQ-9. Providers audio recorded their sessions using a digital recording device, for research assessments and consultation purposes.

Fidelity (adherence and competence) and therapist modifications ratings were coded by 8 research assistants (two bilingual and bicultural English/Spanish speakers), trained by the principal investigator, a CPT expert (S.W.S.), and a postdoctoral fellow (S.V.) on the two versions of the CPT manuals utilized in the current study, CPT-C and CPT-Community, and the Therapist Adherence and Competence Protocol (TAC; Nishith & Resick, 1994). They were also trained to rate using the MAC by the developer of the measure (S.W.S.) and under the supervision of the principal investigator and postdoctoral fellow. The research assistants reviewed the audio recordings of all sessions in the study across phases (n=463), and fidelity ratings were completed based on the version of the CPT manual used in that session.

Fidelity (adherence and competence) and modification ratings were completed for each session. Four pairs of research assistants independently reviewed and double-coded CPT sessions until they achieved reliability for at least 3 consecutive clients (>80% coding agreement with CPT expert raters). Thirty percent of the research assistant codes were randomly spot-checked for consistency after reliability was achieved. In cases where there was disagreement, the research assistants and the supervisors discussed the ratings until a consensus was achieved. For both fidelity measures, agreement was calculated by dividing the number of agreed-upon items (items coded identically by both raters) into the total number of items on the measure. The ratings were reviewed during weekly coding meetings to enhance inter-rater consistency.

Statistical Approach

Latent growth curve modeling was used to assess the impact of treatment process variables on changes in weekly PCL-S and PHQ-9 scores from baseline to study week 12. Models were built progressively, from least to greatest complexity, and at each step model fit indices and plots of observed and estimated means were reviewed. We first modeled each outcome separately including intercept, slope and quadratic growth factors as random effects as well as time specific residuals. PCL-S and PHQ-9 models then were combined and the growth factors were allowed to correlate. However, correlations between PCL-S and PHQ-9 quadratic factors and the intercept and slope parameters from the other symptom type were dropped as they did not significantly contribute to the model (determined by likelihood ratio test) and led to estimation issues. Model fit was assessed using a model’s χ² and associated degrees of freedom (df), Comparative Fit Index (CFI) ≥ 0.95 (Hu & Bentler, 1999), root mean square error of approximation (RMSEA) ≤ .08³ (Chen, Curran, Bollen, Kirby & Paxton, 2008; MacCullum, Brown & Sugwara, 1996), and standardized root mean square residual (SRMR) ≤ .08 (Hu & Bentler, 1999). We used unstandardized parameter estimates (B) with 95% confidence intervals (CI) to determine the strength and precision of relationships. For comparing across constructs, we present standardized parameter estimates (β). To address missing data, we used an inclusive analysis strategy to identify auxiliary variables to be included in the multiple imputation model (see Enders, 2017). We used 100 multiply imputed data sets with parameter estimates averaged over the 100 sets. The large number of data sets was selected to stabilize the imputed values in the analyses stage, which is important in estimating treatment differences with longitudinal data (Lu, 2017). Imputation of missing data and most inferential tests were conducted using SPSS. SPSS uses Markov Chain Monte Carlo (MCMC) methods for estimation and Rubin’s rules to pool test results. Latent growth curve analyses were conducted in MPlus (v8, Muthén & Muthén, 1998 - 2017) using the 100 datasets generated in SPSS. Clinically meaningful change in symptom outcomes was determined using the criteria of 10 points for the PCL-S (National Center for PTSD, 2018; Monson et al., 2008) and 5 points on the PHQ-9 (Kroenke & Spitzer, 2002).

Because the original versions of both CPT manuals (CPT-C and CPT-Community) in the study had received empirical support in previous research (e.g., Resick et al., 2008; Bass et al., 2013), we first examined associations between the manual version and symptom change. The subsequent research questions were tested in two models. The first treatment model investigated the effects of four treatment factors on changes in PCL-S and PHQ-9 scores over the course of treatment. These were (1) total number of fidelity consistent modifications made by the therapist per session, (2) average adherence rating per session, and (3) average competency rating per session, and (4) language of treatment. Because of the temporal overlap in the measures of treatment variables and treatment outcomes, we developed a second treatment model. This model separated treatment variables into three sections: (1) baseline through week 4, (2) weeks 5 through 8, and (3) weeks 9 through 12. Time in regard to PCL-S and PHQ-9 scores was structured using a three-part spline model. Cross-lagged effects were included to assess the impact of treatment process variables and changes in symptoms (i.e., slopes) during early treatment periods on subsequent time blocks. This model is exploratory as power to detect a significant estimate was low. Finally, while the data include three levels of nesting (i.e. time points nested within clients and clients nested within therapists), we do not present analyses including therapist-level analyses. Because there were so few cases per therapists, model non-convergence was a significant issue. Analyses were conducted comparing scores on study-related variables across the 19 therapists and no significant differences were found.

Results

Table 1 includes sociodemographic information for providers and clients. Providers (n=19) were largely female (78.9%), white (78.9%), and social workers (73.7%). Mean age was 45.7 years (SD = 13.8), and providers had worked an average of 15.3 years (SD = 12.4) as a mental health provider at the study site. Almost half of the providers (42.2%) provided sessions in Spanish.

Table 1.

Sociodemographic Factors for Study Clients by Study Phase and for Therapists

	Clients		Therapists
	Phase 2	Phase 3
	n (%)	n (%)	n (%)
Gender - Female	19 (67.9%)	20 (69.0%)	15 (78.9%)
Primary Race a,b
Black or African American	1 (3.6%)	1 (3.4%)	1 (5.3%)
Latino / Hispanic	17 (60.7%)	11 (37.9%)	1 (5.3%)
White	10 (35.7%)	12 (41.4%)	15 (78.9%)
Other or biracial	0 (0.0%)	5 (17.2%)	2 (10.5%)
Marital Status a
Single	11 (40.7%)	12 (41.4%)	5 (26.3%)
Living with partner	2 (7.4%)	4 (13.8%)	3 (15.8%)
Married	8 (29.6%)	7 (24.1%)	7 (36.8%)
Divorced	3 (11.1%)	4 (13.8%)	3 (15.8%)
Separated	3 (11.1%)	2 (6.9%)	0 (0%)
Widowed	0 (0.0%)	0 (0.0%)	1 (5.3%)
Education a
Less than high school	4 (15.4%)	9 (30.9%)	0 (0%)
High school graduate	9 (34.6%)	11 (37.9%)	0 (0%)
Some college	8 (30.8%)	4 (13.9%)	0 (0%)
College graduate	4 (15.4%)	3 (10.4%)	0 (0%)
Graduate school	1 (3.8%)	2 (6.9%)	20 (100.0%)
Occupational Status a
Employed full-time	5 (19.2%)	5 (18.5%)	11 (57.8%)
Employed part-time	4 (15.4%)	7 (25.9%)	6 (31.6%)
Dependent on spouse or is a student	4 (15.4%)	0 (0.0%)	1 (5.3%)
Receiving of public/private assistance	6 (23.1%)	6 (22.2%)	1 (5.3%)
Not applicable	7 (26.9%)	9 (33.3%)	0 (0%)
Language of treatment (cases)
English only	19 (67.9%)	22 (73.3%)	11 (57.8%)
Spanish only	9 (32.1%)	8 (26.7%)	2 (10.6%)
Both			6 (31.6%)
	Mean (SD)		Mean (SD)
Age (years)	39.4 (15.11)	39.9 (13.39)	45.7 (13.8)
Years working as mental health provider at study clinic	n/a		15.3 (12.4)

Note.

^aResponses do not add up to 100% as people did not respond to the item or data are missing.

^bThere is a marginally significant difference between phases for clients race with more people reporting race as “Other” in the third phase, χ²(3) = 6.45, p = .092. All other comparisons of client characteristics between phases were not statistically significant with p > .282 for χ² tests and t(51) = 0.13 p = .899 for age.

Out of the total 98 clients who were identified and invited to participate in the study, 72 signed the study’s informed consent, while 26 were not enrolled in the study due to a variety of reasons (73% recruitment rate): 18 declined participation, 6 were not able to be contacted, and the providers commenced a different type of treatment with 2 of the patients prior to the informed consent appointment for the study due to clinical reasons. Fidelity and modification data were missing for 14 clients, with 7 due to not beginning treatment, and providers not returning audio recordings of the sessions for the remaining 7 clients. Thus, 58 of the 72 enrolled clients who had been randomized to using either the CPT-C or CPT-Community protocols were included in the current study. There were no differences between the use of the two protocols, CPT-C (n= 28, 48.3%) or the CPT-Community (n=30, 51.7%) in terms of client characteristics (Table 1). Also, manual version did not predict the slopes for either the PCL-S (β = −0.03, p = .720) or PHQ-9. (β = 0.13, p = .277). Similarly, treatment engagement did not differ between CPT-C (m=8.8 sessions) and CPT-Community (m=8.2 sessions), t(56) = 0.59, p = .556, nor did therapist-reported dropout rate⁴ between CPT-C (n=16) and CPT-Community (n=22), X²(1) = 1.68, p = .195. Therefore, data from both protocols were combined for remaining analyses. Session language did not predict any of the treatment outcomes investigated (i.e., number of weeks in the study, number of sessions, number of CPT sessions, final PCL score and final PHQ-9 score, all ps > .250). Thus, the direct effects of session language on treatment outcomes were not included in subsequent models.

Inspection of the descriptive statistics for the weekly PCL-S and PHQ-9 scores from baseline to week 12 indicate reductions in both symptom types over the course of treatment (see Table 2). As shown in Table 3, therapists made an average of 1.6 fidelity-consistent modifications per session. Over the course of treatment, the number of fidelity-consistent modifications made per session did not significantly differ between CPT-C clients (m = 1.8, SD = .75) and CPT-Community clients (m = 1.4, SD = .62), though the difference approached significance, t(106.1) = 1.90, p = .060. Also, adherence was high, with therapists being coded by raters as having completed about 90% of the items on the adherence scale and with limited variability among therapists; though adherence rates decreased over the course of the protocol. As seen in Table 3, competence ratings averaged around 3.5 points on a 5-point scale, indicating a “satisfactory to good” competence level, and also tended to decrease over the protocol. Neither the adherence nor competence ratings differed between protocols, both ps > .95. Table 4 provides correlations among fidelity-consistent modifications and the fidelity ratings (i.e., adherence and competence ratings). While not included in the models because of the strong overall correlation with adherence (r = −0.65, p < .001), therapists made an average of 0.4 (SD = 0.38) fidelity-inconsistent modifications per sessions. The number of fidelity-inconsistent modifications made per session did not differ between CPT-C clients (m = 0.36, SD = 0.01) and CPT-Community clients (m = 0.36, SD = 0.01; t(106.1) = 0.01, p = .992.

Table 2.

Descriptive statistics (Mean, SD, percent of scores missing) for therapeutic outcomes by study week

	Cognitive Processing Therapy Week Number
	Baseline	1	2	3	4	5	6	7	8	9	10	11	12
PCL-S	62.3	59.7	57.5	56.8	50.6	51.7	50.9	51.2	48.3	44.2	42.6	43.7	43.3
	(13.68)	(12.75)	(12.78)	(13.5)	(14.63)	(15.22)	(15.97)	(16.53)	(16.80)	(17.66)	(16.89)	(16.98)	(18.43)
	17.2%	12.1%	17.2%	22.4%	31.0%	37.9%	41.4%	48.3%	55.2%	58.6%	58.6%	60.7%	58.6%
PHQ-9	16.1	16.6	16.0	15.0	13.9	13.0	12.7	12.6	12.3	9.9	10.2	11.7	10.0
	(6.22)	(5.66)	(5.58)	(5.67)	(6.58)	(6.63)	(6.53)	(6.82)	(7.74)	(7.23)	(6.97)	(6.44)	(6.54)
	15.5%	22.4%	22.4%	31.0%	36.2%	44.8%	50.0%	55.2%	58.6%	62.1%	58.6%	60.3%	62.1%

Note. PCL-S= Posttraumatic Stress Disorder Checklist-Specific. PHQ-9 = Patient Health Questionnaire

Table 3.

Descriptive statistics for treatment fidelity ratings: therapist consistent modifications, adherence, and competence

	Overall	Time 1	Time 2	Time 3
		BL – Week 4	Weeks 5 - 8	Weeks 9 - 12
	Mean	Mean	Mean	Mean
	(SD)	(SD)	(SD)	(SD)
	n	n	n	n
Fidelity consistent modifications	1.58	1.55	1.57	1.52
	(0.73)	(0.85)	(0.81)	(0.88)
	58	58	43	29
Adherence ratings	0.89	0.92	0.87	0.82
	(0.13)	(0.12)	(0.22)	(0.23)
	58	58	37	20
Competence ratings	3.46	3.68	3.57	3.26
	(1.21)	(1.26)	(1.41)	(1.56)
	58	58	37	20

Note. BL = baseline. Fidelity consistent modifications = modifications made by therapist that do not alter the protocol substantially.

Table 4.

Pearson correlations among fidelity-consistent modifications, adherence ratings, and competence ratings

	Fidelity-Consistent Modifications	Adherence	Competence
Fidelity-Consistent Modifications	1.00	T1: 0.13, p = .324	T1: 0.54, p < .001
		T2: −0.17, p = .322	T2: 0.67, p < .001
		T3: −0.32, p = .163	T3: 0.67, p < .001
Adherence	−0.13, p = .350	1.00	T1: −0.09, p = .509
			T2: −0.22, p = .201
			T3: −0.22, p = .354
Competence	0.52, p < .001	0.09, p = .485	1.00

Note. T1 = time 1 (baseline to session 4). T2 = time 2 (sessions 5 – 8). T3 = time 3 (session 9 – 12). Values below the diagonal represent correlations between constructs representing the average across all sessions attended. Values above the diagonal are correlations of constructs within the same time period.

As seen in Table 2, rates of missing data for each week were 12.1% - 60.7% for PCL-S scores and 15.5% - 62.1% for PHQ-9 scores. Ratings were available for all clients for the overall scores for fidelity-consistent modifications, fidelity-inconsistent modifications and competence ratings. When these variables were divided into the three time segments missing data rates varied by time point. For fidelity-consistent and fidelity-inconsistent modifications, missing data rates were 0%, 25.9% and 50.0% at time 1, time 2 and time 3, respectively. For competence ratings, missing data rates were 0%, 36.2% and 65.5% at time 1, time 2 and time 3, respectively. As described in Enders (2017), we used an inclusive analysis strategy to identify auxiliary variables predicting missingness for each of the PCL-S weekly scores, PHQ-9 weekly scores and therapist fidelity and competence ratings. None of the client and therapist characteristics investigated consistently predicted loss of data over the course of treatment (p-value set at 0.01 to control for number of tests run). Client gender, age and study protocol each was related to a different PCL-S weekly score at p = .04, p = .05 and p = .002, respectively. These variables were included in the multiple imputation model, increasing the probability data are missing at random (MAR) rather than missing not at random (MNAR) (see Enders, 2017).

Reductions in both PCL-S scores and PHQ-9 scores over the course of treatment were derived from a baseline model estimating only changes in these outcomes over time. Except for SRMR = 0.117, the baseline model, with intercept, slope and quadratic growth factors, fit the data well, RMSEA = 0.00, CFI = 1.00, χ²(328) = 250.70, p > .05. PCL-S scores decreased by an average of 15.7 points (pre- to post-treatment standardized effect size d = 1.16) and PHQ-9 scores by an average of 5.3 points (d = 0.86). For PCL-S scores, both the slope estimate (β = −1.03, p < .001) and the quadratic estimate (β = 0.47, p = .023) were statistically significant. For the PHQ-9 scores, the slope estimate was statistically significant (β = −0.51, p = .008), but not the quadratic effect (β = 0.16, p = .517). As can be confirmed by reviewing the means in Table 2, PCL-S scores decrease over time until near the end of treatment, where the average stabilized. In contrast, PHQ-9 scores decreased at a consistent rate over all sessions. Based on imputed scores at baseline and at session 12, 67.9% of clients demonstrated clinically meaningful reductions on the PCL-S, 3.7% had clinically meaningful increases and for the remaining clients (28.4%) their imputed PCL-S scores at session 12 were within 10 points of their baseline scores. On the PHQ-9, rates 50.0% reported clinically significant reductions on the PHQ-9, 7.9% had clinically significant increases and 42.1% had scores at session 12 that were similar to their baseline scores.

Next, we investigated the impact of therapist modifications and fidelity ratings on changes in PCL-S and PHQ-9 scores. Parameter estimates for relevant treatment and symptom growth factors for the first treatment model (modeling symptom change over the course of treatment) are presented in Figure 1. The model fit the data well, except for SRMR = 0.125, RMSEA = 0.00, CFI = 1.00, χ²(427) = 304.99, p > .05. While this can indicate model misspecification, SRMR is strongly affected by small sample sizes (Iacobucci, 2010). In this model, we found that more fidelity-consistent modifications made by therapists were related to greater reductions in both PCL-S scores [B = −0.90 points per modification per session (−1.51, −0.29), β = −0.31) and PHQ-9 scores [B = −0.37 points per modification per session (−0.64, −0.10), β = −0.31]. Greater adherence to the CPT protocol was related to greater reduction in PHQ-9 scores [B = −2.20 points for every 10% increase in adherence (−3.88, −0.50), β = −0.25], but not for PCL-S scores [B = −2.68 points (−6.48, 1.13), β = −0.16]. In contrast, higher competence scores predicted larger reductions in PCL-S scores [B = −0.44 points (−0.83, −0.05), β = −0.20], but not for PHQ-9 scores [B = −0.08 points (−0.25, 0.08), β = −0.09). Ratings did not predict curvilinear trends (i.e., quadratic growth factors) and, thus, were not included in the final models for parsimony and ease of interpretation. Finally, language only predicted the frequency of fidelity-consistent modifications, with sessions conducted in Spanish having more modifications [B =0.81 modifications more in Spanish sessions than in English sessions (0.47, 1.14), β = 0.52]. Language did not predict significant differences in either adherence ratings, [B = −0.06 (−0.12, 0.01), β = −0.20] or competence ratings [B = 0.25, (−0.43, 0.92), β = 0.10].

Figure 1.

Partial modeling results for effects of language of intervention, fidelity consistent modifications, therapist competence ratings, and therapist adherence ratings on changes in posttraumatic stress and depressive symptoms during Cognitive Processing Therapy.

Note. β = standardized parameter estimates (95% confidence intervals of β). PCL-S = Posttraumatic Stress Disorder Checklist-Specific. PHQ = Patient Health Questionnaire. Consistent modifications = average number of treatment consistent modifications made by therapist per session. Competence = average competence rating across sessions. Adherence = average adherence rating across sessions. Language was dummy coded as English = 0 and Spanish = 1. Bolded lines represent statistically significant pathways. * p ≤ .05. ** p ≤ .01. *** p ≤ .001.

Figure 2 depicts the partial modeling results for our second, cross-lagged treatment model (that modeled symptom change over discrete time periods). Standardized pre- to post-treatment effect sizes were shown to be similar across the CPT discrete time periods. For the PCL-S, ds were 1.16, 1.33, and 1.59 for time 1 (baseline through session 4), time 2 (sessions 5-8), and time 3 (sessions 9-12), respectively. For the PHQ-9, ds were 0.86, 0.92, and 1.11 for time 1, time 2, and time 3, respectively. Significant parameter estimates were found for the effects of fidelity-consistent modifications and adherence ratings during time 1 on changes in both PCL-S and PHQ-9 symptoms during time 2. No other relevant model parameter was statistically significant. Similar to the first model, more fidelity-consistent modifications during time 1 predicted larger reduction in PCL-S scores [B = −1.38 points per modification per session (−2.46, −0.31), β = −0.36] and in PHQ-9 scores [−0.48 points per modification per session (−0.91, −0.05), β = −0.47] over time 2. In other words, the model suggested that if the average number fidelity-consistent modifications per session occurred consistently across the early treatment sessions (1.55 per session), there would be an estimated additional mean decrease of 8.6 points on the PCL-S across the middle of the CPT protocol over and above the change that could be expected for a patient for whom no fidelity-consistent adaptations were made. This difference falls short of cutoffs for clinically meaningful change (National Center for PTSD, 2018; Monson et al., 2008) but above a reliable change score of 6.70 derived based on Foa and colleagues’ (2002) method. Also, greater adherence predicted larger reduction in PCL-S scores [−14.28 points for 10% increased adherence (−26.24, −2.32), β = −0.51] and in PHQ-9 scores [−4.30 points for 10% increased adherence (−8.35, −0.25), β = −0.47]. Results related to adherence scores need to be interpreted cautiously as there was high adherence across therapists with limited variability during the first time period and the 95% CI for the estimates are quite large, indicating uncertainty regarding the strength of the effect.

Figure 2.

Partial modeling results for the cross-lagged effects of treatment consistent modifications, therapist competence ratings, therapist adherence ratings, and changes in PCL-S and PHQ scores.

Note. β = standardized parameter estimates (95% confidence intervals of β). T1 = time 1 (baseline and sessions 1 to 4). T2 = time 2 (sessions 5 to 8). T3 = time 3 (sessions 9 to 12). PCL-S = Posttraumatic Stress Disorder Checklist-Specific. PHQ = Patient Health Questionnaire. Fidelity-Consistent modifications = average number of treatment consistent modifications made by therapist per session. Competence = average competency rating across sessions. Adherence = average adherence rating across sessions. Bolded lines represent statistically significant pathways. * p ≤ .05. ** p ≤ .01. *** p ≤ .001. Parameter estimates are provided only for statistically significant pathways.

Discussion

This study is the first to examine associations between provider fidelity (adherence and competence), treatment modification, and treatment outcomes for an EBP delivered in a diverse, low-resource usual care setting. Overall CPT adherence was high (90%), although, similar to Boswell and colleagues’ (2013) findings in an RCT, adherence decreased across sessions. Competence was somewhat lower than that found in clinical trials with expert therapists. However, similar to Monson and colleagues’ (2018) findings in a community sample, overall protocol treatment effects were in the range found in clinical trials of CPT (Cusack et al., 2016), and 67.9% of the participants experienced clinically meaningful symptom reductions. We hypothesized associations between (1) average number of fidelity consistent modifications made by the therapist per session, (2) average adherence rating per session, (3) average competency rating per session, and (4) language of treatment delivery and changes in posttraumatic stress and depressive symptoms over the course of treatment. In cross-lagged analyses, we found that higher numbers of fidelity-consistent modifications during the first four sessions were associated with greater reductions in posttraumatic stress and depressive symptoms during the second four sessions of the protocol. Our findings suggested that for each additional fidelity-consistent adaptation made in a session during the early treatment phase, an additional 1.38 points decrease during the mid-therapy phase could be expected. This suggests that sufficient, appropriate adaptations that do not displace or interfere with the key elements of CPT consistently across the early sessions in the protocol could lead to incremental improvement across subsequent phases of treatment. In contrast, fidelity-inconsistent modifications, such as removal of aspects of the protocol intended to address trauma-related cognitions, substitution or integration of other treatment elements can detract from adherence to the elements of the protocol intended to address these key elements.

High adherence ratings were associated with greater reductions in depressive symptoms, but not posttraumatic stress symptoms. It is possible that all aspects of the protocol, including the behavioral assignments toward the end of the protocol and all aspects of cognitive restructuring are necessary to support recovery from depression, whereas competent delivery of elements that most closely target trauma-related cognitions are sufficient to address PTSD. Additionally, higher competence ratings were associated with greater reduction in posttraumatic stress symptoms, but not depressive symptoms. While there may be some overlap between competence and fidelity-consistent modifications, competence also captures the skill and mastery at delivering core CPT elements, which includes understanding which trauma-related cognitions to target at different points in the protocol and how to effectively challenge them with the client. Thus, the findings regarding competence and PTSD symptom change may be explained by a recent finding that competence in attending to assimilated beliefs related to the trauma and Socratic questioning skills are associated with symptom change in CPT (Farmer et al., 2017). Partial modeling results (cross-lagged) also indicated that fidelity-consistent modifications were associated with greater subsequent reductions in posttraumatic stress and depressive symptoms, suggesting that efforts to personalize the treatment without removing key session elements may enhance treatment effectiveness.

Our findings suggest that there may be important distinctions to draw between constructs of adherence, competence, and modifications, and they highlight the need for more research on their relation to clinical outcomes. Characterizing and understanding the nature and impact of adaptations in addition to continuous attention to fidelity may be necessary in research on delivery of EBPs in routine care settings, as fidelity ratings alone may not capture aspects of treatment delivery that are associated with clinical change. For example, skill in Socratic questioning and prioritization of assimilated automatic thoughts before over-accommodated thoughts (Farmer et al., 2017) have been shown to predict subsequent PTSD change. The fidelity-consistent modifications identified in this study did not alter these aspects of the protocol, and did not displace aspects of CPT that had been identified as “unique and essential” in the CPT fidelity measure. However, fidelity-consistent adaptations might not always fully align with therapist competence. For example, extending a session’s length may be indicative that a therapist has not paced a session efficiently or had difficulty explaining concepts or engaging in Socratic dialogue effectively. Thus, it is important to examine fidelity-consistent adaptations in conjunction with competence to determine whether and how each is uniquely associated with symptom change. As with previous findings on fidelity (e.g., Hogue et al., 2008), relationships between process variables of this nature and clinical change may be non-linear, and they may be moderated by client characteristics or clinical presentation.

Our findings are consistent with previous studies that suggest that effective EBP delivery may require at least minor adaptations, not typically captured in fidelity assessments, to meet client needs or to ensure that clients are able to understand and benefit from interventions (Chu & Leino, 2017; Roth & Pilling, 2008) without compromising fidelity or clinical effectiveness. These findings provide some support for Dynamic Sustainability Framework (Chambers et al. 2013), which suggests that in rapidly changing, diverse practice contexts, an ongoing, data-informed approach to adaptation of interventions can improve fit between interventions and dynamic treatment delivery contexts. Greater attention to multiple aspects of fidelity and treatment adaptation can enhance training and consultation to prepare providers to tailor interventions while retaining core components of the intervention.

Limitations and Future Directions

There are several limitations that should be noted for the current study. First, the majority of our providers and participants were women. Future studies should explore whether the current study’s results generalize to male therapists and clients. Second, the current study inclusion criteria used client’s PCL-S scores and/or medical record diagnoses, instead of conducting a comprehensive psychodiagnostic assessment, due to challenges inherent to conducting research in community clinic settings, and to maximize external validity. It should be noted however, that study providers received training on diagnosing PTSD and assessing for PTSD symptoms as part of their 3-day training workshop. While an observer coding system for adherence, competence, and modifications provides rich information regarding variables that are relevant to implementation in usual care settings, larger sample sizes, reductions in rates of missing data, and further replication with these measures are warranted. It will be particularly important to understand whether, and under what circumstances, adaptations may lead to clinically meaningful differences in symptom improvement. We hope that future studies provide opportunity to continue to study the implementation of these assessment tools, and develop implementation plans that speak to the need to monitor and address fidelity concerns in real time through training and consultation. In addition, while we took steps to address potential temporal confounds in our analyses, we were not able to examine moderating factors or interactions due to insufficient sample size. Differences in competence were not found across the sample of clinicians, suggesting that training and consultation may have been effective in reducing variations in fidelity. However, it may be more necessary—and more challenging—to provide treatment with competence for some clients or clinical presentations than others (Hogue et al., 2008; Strunk et al., 2010). Thus, moderation analyses are an important next step. Finally, more research is needed to delineate core versus peripheral components in EBPs to inform which adaptations may enhance versus diminish clinical effectiveness. Future studies could, for example, investigate whether greater latitude in selecting interventions for complex cases, or encouraging providers to make fidelity-consistent adaptations to protocols when their clinical judgement suggests that they are indicated, improves clinical or implementation outcomes.

Conclusion

Identification of treatment approaches that fit the needs of diverse and complex clients in routine care treatment settings are essential to improving treatment outcomes for PTSD. Using data from an open trial of CPT for PTSD in a diverse usual care setting, we found that fidelity in CPT was associated with treatment outcomes for posttraumatic stress and depressive symptoms. Furthermore, in models that included treatment fidelity, adaptations made by clinicians beyond those specified in the manual, but preserving the essential elements of CPT, were associated with greater improvements in PTSD. Assessing both fidelity and adaptation may therefore provide greater insight into the relationships between in-session delivery of EBPs and clinical outcomes.

Public Health Significance Statement:

Providers in routine clinical care settings have reported a perceived need to modify evidence-based protocol treatments in order to address their clients’ needs and contextual challenges. The results of the current study suggest that minor adaptations that are consistent with the treatment framework may aid effectiveness, and that higher fidelity is associated with better outcomes.