Changes in Implementation of Two Evidence-Based Psychotherapies for PTSD in VA Residential Treatment Programs: A National Investigation

Abstract

There has been little investigation of the natural course of evidence-based treatments (EBTs) over time following the draw-down of initial implementation efforts. Thus, we undertook qualitative interviews with the providers at 38 U.S. Department of Veterans Affairs’ residential treatment programs for posttraumatic stress disorder (PTSD) to understand implementation and adaptation of 2 EBTs, prolonged exposure (PE), and cognitive processing therapy (CPT), at 2 time points over a 4-year period. The number of providers trained in the therapies and level of training improved over time. At baseline, of the 179 providers eligible per VA training requirements, 65 (36.4%) had received VA training in PE and 111 (62.0%) in CPT with 17 (9.5%) completing case consultation or becoming national trainers in both PE and CPT. By follow-up, of the increased number of 190 eligible providers, 87 (45.8%) had received VA training in PE and 135 (71.1%) in CPT, with 69 (36.3%) and 81 (42.6%) achieving certification, respectively. Twenty-two programs (57.9%) reported no change in PE use between baseline and follow-up, whereas 16 (42.1%) reported an increase. Twenty-four (63.2%) programs reported no change in their use of CPT between baseline and follow-up, 12 (31.6%) programs experienced an increase, and 2 (5.2%) programs experienced a decrease in use. A significant number of providers indicated that they made modifications to the manuals (e.g., tailoring, lengthening). Reasons for adaptations are discussed. The need to dedicate time and resources toward the implementation of EBTs is noted.

In 2007 the U.S. Department of Veterans Affairs (VA) established national initiatives to provide training and consultation in two evidence-based treatments (EBTs) for posttraumatic stress disorder (PTSD; Karlin et al., 2010): prolonged exposure (PE; Foa, Hembree, & Rothbaum, 2007) and cognitive processing therapy (CPT; Resick & Schnicke, 1993). PE is a trauma-focused individual therapy covering 8 to 15 sessions. PE exposes patients to trauma-related situations that are objectively safe, but are avoided due to trauma-related distress (in vivo exposure) and to trauma memories through repeated recounting out loud of the details of the most disturbing event (imaginal exposure). CPT is a 12-session trauma-focused treat ment that can be delivered in group, individual, or combined format. CPT focuses on the relationship between unhealthy and distorted thinking patterns related to trauma by teaching new and adaptive ways of thinking.

A formative evaluation of services in VA PTSD residential treatment programs nationwide was conducted during the early years of the VA dissemination effort. At that time, a considerable proportion (38.7% and 65.9%, respectively) of eligible providers had received training in PE or CPT (Cook et al., 2013). Although many providers had received training, PE program implementation ranged from no usage to select patients receiving the treatment. CPT implementation ranged from no use, to use of only one aspect (e.g., specific worksheets), to strict manual adherence with all patients. Because these data were collected early in the dissemination effort, further training in and differing use of these EBTs may have occurred. For example, conditions that facilitate initial implementation may change with the passing of time (Stirman et al., 2012), particularly when structured dissemination efforts have been discontinued.

Little is known regarding what happens after initial EBT implementation (Landsverk, Brown, Rolls Reutz, Palinkas, & Horowitz, 2011). In a review of the literature on sustainability of EBTs across medical/health care (e.g., diabetes, coronary artery disease), public health/health promotion and mental health, l9 studies had lower levels of implementation, 17 studies increased use, and 3 indicated no change over time (Stirman et al., 2012). One large investigation examined implementation of EBTs for adults with severe mental illness in mental health organizations across eight states, finding a nonlinear progression with the most dramatic adoption in the first 12 months and relatively little change thereafter (Bond, Drake, Rapp, McHugo, & Xie, 2009).

There are several proposed influences on sustainability of EBTs, including organizational context, capacity, and processes (Stirman et al., 2012). Further, organizational or systems-level variables such as dedicated time and resources, number of adequately trained staff, and support from administration, may have an even greater influence on implementation than provider-level variables, such as knowledge of and attitudes towards EBTs (Aarons & Sawitzky, 2006).

Previous investigations have found that adaptation of EBTs is positively related to sustainability (Blasinsky, Goldman, & Unutzer, 2006; Scheirer, 2005; Tibbits, Bumbarger, Kyler, & Perkins, 2010). For example, organizations that adapted a depression intervention to their context were more likely to sustain its use (Blasinsky et al., 2006). Understanding providers’ perspectives on the delivery and adaptation of EBTs and their reasons for adaptation may ultimately improve implementation and help illustrate factors critical to sustainability.

In addition, there is limited research on the extent, nature, or impact of adaptations of EBTs for PTSD in community practice. Therefore, we conducted a follow-up of the 38 VA residential PTSD treatment programs across the United States to (a) present an update on the number of providers who received training in PE and CPT and the level of training attained; (b) document changes in the use of PE and CPT at the program level; (c) examine whether VA training in PE and CPT, as well as professional discipline, increased their implementation; and (d) detail the adaptations being made to these EBTs.

As the largest health care organization in the country, and with unprecedented federal funding and top-down administrative support for the EBT initiative, the VA affords an ideal opportunity in which to study implementation. Similarly, residential settings are ideal contexts in which to study implementation as the providers have the opportunity to deliver treatment over an extended period and patients are removed from external stressors that might interfere with engagement and participation in a trauma-focused EBT.

Method

Participants and Procedure

A formative evaluation of 38 VA residential PTSD treatment programs reporting patient outcome data to the VA’s Northeast Program Evaluation Center (NEPEC) was undertaken begin ning July 2008 through March 2011 (Cook et al., 2013). The same programs included in the formative evaluation were later recontacted between January 2010 and March 2012 to participate in a follow-up web-based survey and semistructured interview. These data are what are being newly reported here.

From 2008 to early 2011, 2-day program visits were conducted in which providers were interviewed regarding their use of various programming. Those data were used to create a program-level coding scheme, which will be discussed in the Measures section, with six possible levels of PE and CPT implementation. The results of that analysis can be found elsewhere (Cook et al., 2013) and constitute the baseline ratings here.

Beginning in January 2010 and ending in March 2012, we recontacted providers to participate in a web-based survey and telephone-based interview about the use of PE and CPT in their programs. Development of the survey and semistructured interview guide are described elsewhere (Cook et al., 2012). All providers new to the programs since the site visit (n = 19) were also approached to participate. We used the same adoption coding scheme for these data that was used at baseline. Interviews at both time points were voluntary, confidential, and recorded with written permission. A professional transcriptionist compiled verbatim deidentified interview transcripts. This study was exempted for review by the Yale Human Research Protection Program due to perceived low risk to human subjects. Nevertheless, the study was approved by the VA Connecticut Health Care System Institutional Review Board.

Of the 243 providers in these 38 settings, 190 (78.2%) provided complete data on the survey and qualitative interview and were included in the analyses. Descriptive information on the participating programs and follow-up providers is included in Table 1. Most providers were psychologists (n = 119, 62.6%) or social workers (n = 58, 30.5%), and most were female (n = 114, 60.0%) with a range in age of 26–75 years old (M = 45.57, SD = 11.13). Length of time working in residential programs ranged from less than 1 year to 30 years (M = 6.68, SD = 6.82).

Table 1.

Description of PTSD Residential Treatment Programs and Providers

Variable	n	%
Programs (N = 38)
Typea
PTSD and psychosocial residential   rehabilitation	18	47.3
Domiciliary	14	36.9
Other	6	15.9
Average LOS (days)
0–20	3	7.9
21–35	11	28.9
36–50	16	42.1
51–65	4	10.5
≥66	4	10.5
Staff (N = 190)
Type
Psychiatrist	6	3.2
Psychologist	119	62.6
Social worker	58	30.5
Nurse	7	3.7
Years on unit
<1	14	7.4
1–3	69	36.3
4–5	35	18.4
6–10	31	16.3
11–18	21	11.1
>19	20	10.5

Note. PTSD = posttraumatic stress disorder; LOS = length of stay.

^aIn brief the programs are defined as follows: PTSD residential rehabilitation programs: Lower intensity programming with access to acute services as needed, rehabilitative focus with average LOS of 30 days. PTSD Domiciliary: 24/7 care. May be part of a larger domiciliary serving multiple patient populations; LOS ranges from 38–102 days. Other: Specialized inpatient PTSD units, PTSD day hospital, evaluation and brief treatment PTSD units.

Measures

Two investigative team members (J. C., S. D.) independently extracted information from transcripts on provider and program use of PE and CPT. These team members then independently coded each program for six possible levels of PE and CPT implementation: (a) no adoption, (b) elements of treatment were used, (c) select patients receive the treatment, (d) patients in a trauma-processing track receive the treatment, (e) treatment is the core of programming, and (f) de-adoption. A summary document was made for each program including baseline level of adoption, follow-up level of adoption, a brief synopsis of program operations noting any major changes between the two dates of contact, and quotations supporting proposed adoption level. Program summary documents were compared between raters, amalgamated into one comprehensive document, and areas of discord were discussed until consensus was reached. Kappa coefficients for level of adoption between the two raters were .83 for PE and .87 for CPT. Overall κ between the raters for adaptations was .74.

Each provider was assigned to one of seven levels of training for PE and CPT: (a) did not complete any training, (b) completed informal training within the VA, (c) completed training outside of the VA, (d) read the treatment manual, (e) completed VA workshop training, (f) completed VA case consultation, or (g) became a national trainer.

An EBT adaptation coding tool (Stirman, Miller, Toder, & Calloway, 2013) was used to record modification into 12 possible categories: (a) tailoring/tweaking/refining, (b) integrating the intervention into another framework, (c) integrating elements of another intervention into the intervention of interest, (d) removing/skipping elements or components, (e) lengthening/extending, (f) shortening/condensing, (g) adjusting the order of the intervention, (h) adding modules or activities, (i) de parting from the intervention or “drift,” (j) loosening structure, (k) substituting, and (l) adaptations that abandon the principles of the intervention.

Multiple procedures were utilized at both baseline and follow-up to preserve internal validity including interview standardization through the use of a semistructured guide, audiotaping, professional transcription, and the use of qualitative data analysis software (Atlas.ti 6.0; Muhr, 2011).

Data Analysis

Analyses of variance were conducted to examine the relationship between level of training and professional discipline and implementation for both PE and CPT. We also examined changes from baseline and looked at the degrees of implementation as a function of baseline.

Results

As illustrated in Table 2, by follow-up, a substantial proportion had received VA training in PE (n = 87, 45.8%) or CPT (n= 135, 71.1%). Of those, 69 (36.3%) and 81 (42.6%) achieved certification, respectively. Eighty (43.1%) received some level of VA training in both PE and CPT.

Table 2.

Distribution of Completed Level of Training and Mode of Delivery of Providers in CPT and PE at Follow-Up

	CPT		PE
Variable	n	%	n	%
Training level
None	38	20.1	76	40.0
Informal within VA	5	2.6	2	1.1
Outside of VA	7	3.7	23	12.1
Read manual	5	2.6	2	1.1
Formal 2/4 days	54	28.4	18	9.5
Case consultation	81	40.0	69	32.5
Became trainer for VA	5	2.6	7	3.7
Method of delivery
None	40	21.1	138	72.6
Individual	18	9.5	44	23.2
Group	75	39.4	3	1.6
Both	57	30.0	5	2.6

Note. N = 190 eligible to receive PE and CPT training. CPT = cognitive processing therapy; PE = prolonged exposure; VA = Veterans Affairs.

As can be seen in Table 3 and Table 4, there was some change in PE and CPT implementation between baseline and follow-up. Although at baseline, 21 (55.2%) programs had not implemented PE in any form, nearly three fifths had some level of PE implementation by follow-up. Stated a different way, 42.1% of programs overall increased their implementation of PE, and 57.1% of those programs with no implementation of PE had at least partially implemented by follow-up. Of the 12 programs that had not implemented any form of CPT at baseline, more than half had increased their implementation by follow-up. Overall, 12 (31.6%) of the programs showed increased implementation of CPT. Ten of these programs, however, were already fully implementing CPT, so there was little room for improvement. In other words, among the 28 programs that had not fully implemented CPT at baseline, 49.8% showed some improvement. Notably, of the programs delivering these treatments to select patients only (n = 15) there was substantial variance in the number of select patients included, ranging from a low of 5% to a high of 75% with an average of 25.2%.

Table 3.

Level of Adoption for PE and CPT at 38 Programs at Baseline and Follow-Up

	PE				CPT
	Baseline		Follow-up		Baseline		Follow-up
Level of adoption	n	%	n	%	n	%	n	%
Not adopted	21	55.3	9	23.7	12	31.6	5	13.2
Elements adopted	5	13.2	8	21.1	3	7.9	2	5.3
For select patients only	8	21.1	15	39.4	7	18.4	15	39.4
For track patients only	2	5.3	5	13.2	5	13.2	8	21.1
Core of program	0	0	0	0	10	26.3	8	21.1
De-adopted	2	5.3	1	2.6	1	2.6	0	0

Note. PE = prolonged exposure; CPT = cognitive processing therapy.

Table 4.

Distributions of Level of PE and CPT Adoption at Baseline by Change at Follow-Up for 38 Programs

	PE						CPT
	No change (n = 22)		Improvement (n = 16)		Reduction (n = 0)		No change (n = 24)		Improvement (n = 16)		Reduction (n = 2)
Level of adoption	n	%	n	%	n	%	n	%	n	%	n	%
Not adopted	9	23.7	12	31.6	0	0	5	13.2	7	18.4	0	0
Elements adopted	3	7.9	2	5.3	0	0	0	0	3	7.9	0	0
For select patients only	7	18.4	1	2.6	0	0	6	15.8	1	2.6	0	0
For track patients only	2	5.3	0	0	0	0	5	13.2	0	0	0	0
Core of program	0	0	0	0	0	0	8	21.1	0	0	2	5.3
De-adopted	1	2.6	1	2.6	0	0	0	0	1	2.6	0	0

Note. PE = prolonged exposure; CPT = cognitive processing therapy.

Of those programs delivering neither PE nor CPT to at least select patients at baseline (n = 12), four (33.3%) focused primarily on skill-building, three (25.0%, respectively) delivered acceptance and commitment therapy (ACT; Hayes, Strosahl, & Wilson, 1999) or warzone trauma-focused groups, and two delivered process groups (16.6%). At follow up, five programs delivered neither PE nor CPT; instead, two (40.0%) offered war-zone trauma-focused groups or ACT and one (20.0%) offered a skills-based programming only. Of the programs delivering the full PE or CPT protocol to select patients (n = 25), 11 (44.0%) did so with 10%–25% of their patients, 3 (12.0%) with 26%– 50%, 7 (28.0%) with 51%–75%, and 4 (16.0%) with over 75% of their patients.

There was a significant relationship between mean provider level of PE training and its implementation at follow-up, F(4,33) = 4.80, p = .004. There was also a significant relationship between provider level of CPT training and its implementation at follow-up, F(4, 33) = 5.32, p = .002. There was no significant effect of profession on adoption of PE, F(3,186) = 0.21, p = .893, or CPT, F(3,186) = 0.72, p = .547. Although training was assessed at the site level, we examined the association between mean site training and professions of the respondents at each site. There was no significant effect of profession on training in PE, F(3,186) = 1.29, p = .280, or of CPT, F(3,186) = 0.94, p = .418.

There were numerous adaptations made to the treatments (Table 5). PE was most often adapted by solely offering elements of it (e.g., in vivo) or by offering elements in conjunction with other EBTs. Although there was not a significant overall relationship between PE adaptations and PE adoption, F(3,34) = 2.08, p = .122, post hoc comparisons revealed that those programs at follow-up that had patients in particular tracks receive PE treatment had usedmore adaptations (M = 1.00, SD=1.22) than those where the practice was not adopted (M=0.22, SD = 0.44) and than those where select patients received treatment (M = 0.27, SD = 0.59). There was a significant lack of over all relationship between CPT adaptations and CPT adoption, F(3,34) = 2.37, p = .725. Again, however, there were post hoc group differences; programs where at follow-up patients in particular tracks received treatment used more adaptations (M = 2.13, SD = 1.73) than those programs where the practice was not adopted (M = 0.40, SD = 0.55) and than those that included some elements of treatment (M = 0.00, SD = 0.00).

Table 5.

Provider Adaptations to Manualized CPT and PE Protocols

	CPT (n =31)		PE (n =20)
Adaptations	n	%	n	%
Tailoring/tweaking/refining	13	61.2	1	5.0
Integrating EBT into another framework	4	12.9	7	35.0
Integrating another framework into EBT	4	12.9	3	15.0
Removing modules/components	4	12.9	1	5.0
Lengthening/extending	8	25.8	3	15.0
Shortening/condensing	9	29.0	2	10.0
Adjusting module order	1	3.2	0	0
Adding modules/elements	2	6.5	0	0
Drifting/abandoning structure	2	6.5	0	0
Loosening structure	2	6.5	2	10.0
Repeating modules/elements	1	3.2	1	5.0
Substituting modules/elements	0	0	0	0
Format	0	0	3	15.0
Other	2	6.5	0	0

Note. CPT = cognitive processing therapy; PE = prolonged exposure; EBT = evidence-based treatments.

Tailoring, tweaking, or refining was the most common type of adaptation made to CPT. These typically involved adapting worksheets that providers perceived to be difficult to understand for the average literacy/education level of the patient population. Tailoring of worksheets also reportedly took place to accommodate cultural groups or those with traumatic-brain injury (e.g., completing all worksheets verbally). Shortening or condensing of treatments reportedly occurred due to length-of-stay restrictions with some programs delivering EBTs up to five times a week. Conversely, lengthening/extending was said to be necessary to individualize treatment and meet patient needs.

Discussion

Many providers in VA PTSD residential treatment programs reported greater degrees of implementation of PE and CPT over time. Although about half of the programs still reported no use of PE, there was an increase in its implementation for select patients. In addition, almost 70% of programs had implemented CPT as a full or partial protocol. Because it typically takes an average of 17 years to implement knowledge generated by randomized controlled trials into practice (Institute of Medicine, 2001), the study here as well as other recently published data from the training initiative (Eftekhari et al., 2013) can be viewed as an expedited transfer of knowledge from research to direct clinical service.

The most common level of implementation was “select patients receive the treatment” for both PE and CPT. Five programs did not deliver the full protocol of either PE or CPT to any patients. Instead, those programs delivered another EBT or warzone trauma process group. Adaptations to treatment were common; most programs made changes to content, number of sessions, and style of presentation. In particular, providers were most likely to report tailoring, tweaking, or refining elements of the treatment. More commonly, elements of PE (e.g., in vivo exposure) were utilized due to structural and resource constraints.

One controversy in implementation science is the tension between fidelity versus adaptation of EBTs. Some believe that fidelity is required to produce the best patient outcomes (Schoenwald, Sheidow, & Letourneau, 2004). Alternatively, others argue adaptation of EBTs is inevitable and necessary for implementation (Dearing, 2009; Palinkas et al., 2008). In a review of empirical studies of program sustainability in health care, partial sustainability was more common than full, even when full implementation had been initially achieved (Scheirer, 2005). Similarly, in a large-scale dissemination project in Pennsylvania, nearly three quarters of interventions instituted by community agencies and schools had been sustained at least 1-year postfunding, but a significant majority had been adapted or altered (Tibbits et al., 2010).

Our findings are in line with the “flexibility within fidelity” model (Kendell, Gosch, Furr, & Sood, 2008, p. 988) and other research on balancing clinical judgment and fidelity (Galovski, Blain, Mott, Elwood, & Houle, 2012; Levitt, Malta, Martin, Davis, & Cloitre, 2007). A recent meta-analysis similarly found that neither EBT adherence nor competence was related to patient outcomes (Webb, DeRubeis, & Barber, 2010), suggesting that other factors may more significantly influence outcome (e.g., fit between the innovation and setting, patient motivation or willingness to engage in EBTs).

A commonly cited reason for adaptation in this study was limitations related to established aspects of program structure (e.g., length of stay; rolling vs. cohort admission). One solution is to first learn EBTs with strict adherence and then use core ingredients (identified via dismantling studies) with strict fidelity, while adapting nonessential ingredients to local settings (Brekke, Phillips, Pancake, Lewis, & Duke, 2009). Another suggested approach to balance fidelity and adaptation is the use of transdiagnostic treatments (McHugh, Murray, & Barlow, 2009). In one recent investigation, modular treatment for childhood mental health problems produced significantly steeper trajectories of improvement than usual care and standard treatment on multiple clinical outcome measures (Weisz et al., 2011). These treatments involve a core treatment package with various modules or components that providers choose to implement and dose depending on patient characteristics and available resources (Mazzucchelli & Sanders, 2010). The use of a modular approach may allow providers greater flexibility in meeting the needs of their patients and setting, resolving issues of compatibility frequently identified as a barrier to adoption (Borntrager, Chorpita, Higa-McMillan, & Weisz, 2009).

Training in PE and CPT increased from baseline to follow-up; however, the numbers of providers trained appeared relatively constant. Possibly, more providers have achieved higher levels of training at follow-up compared to baseline (i.e., from attending a workshop at baseline to achieved certification at follow-up). Considered this way, level of training in EBTs appears to have improved between the two dates of contact through vertical (i.e., higher levels of expertise) rather than horizontal (i.e., numbers of providers newly trained) growth. Training results here are limited to residential PTSD programs and may not be reflective of training trends in the VA at large. There is some evidence, however, to suggest ongoing improvements to CPT dissemination efforts, such as the development of web-based refresher courses, have reduced perceived barriers to consultation and use postworkshop (Chard, Ricksecker, Healy, Karlin, & Resick, 2012).

It is important to note the distinctive attributes of the VA mental health system of care and specifically, residential PTSD treatment. The VA is a uniquely resourced institution guided by directives and mandates for treatment that help to standardize care across sites. Although programs in this study differed (e.g., number of patients, number of staff, length of stay) they all shared common assessment, treatment, and reporting requirements; such uniformity in the assessment and treatment of PTSD is less likely to be found in community-based care. As such, some findings may not generalize to other settings such as outpatient or community-based programming. There have been no recent systematic examinations of inpatient PTSD care outside of the VA however; thus, it is difficult to extrapolate.

There are several limitations of this investigation. A majority of the baseline assessments took place from 2008–2010, with all follow-up assessments taking place from 2010–2012. We made every effort to assess programs at similar intervals; however, select programs were interviewed at briefer intervals than others. A further shortcoming is the reliance on provider self-report, which is subject to memory bias, lack of knowledge, and demand characteristics. This is consistent with findings from a recent review, which found that few studies used an objective judgment of implementation quality or fidelity (Stirman et al., 2012). Providers may rate themselves significantly higher in their use of and adherence to EBTs than independent raters reviewing taped sessions indicating that self-report measures likely need verification (Carroll, Martino, & Rounsaville, 2010). Triangulation of provider reports, actual observation, or cross-check with patient records may provide greater credence to these findings.

It has been suggested that sustainability be assessed over several years rather than at just one time point (Stirman et al., 2012). Further data points are needed to document whether the changes described here are maintained or decrease. Aarons and Palinkas (2007) note that adaptability and sustainability may be related, such that organizations and providers that adapt EBTs to their context are more likely to be committed to their sustainability. Future research efforts should seek to systematically investigate the influence of adaptation on the sustainability of interventions postimplementation.

More research is needed to examine whether adaptations of PE and CPT decrease treatment effectiveness or allow for them to be sustained beyond what is achieved with strict fidelity, and why. Reasons for adaptations and whether these are made with intentionality or in reaction to an unexpected consequence of implementation, and to what effect, is another avenue for future investigation. Additionally, examining provider perceptions of barriers and facilitators to implementation of PE and CPT as well as their understanding of patient appropriateness for each treatment may further help to explain use.