A Formative Evaluation of Two Evidence-Based Psychotherapies for PTSD In VA Residential Treatment Programs

Abstract

Thirty-eight U.S. Department of Veterans Affairs (VA) residential treatment programs for posttraumatic stress disorder (PTSD) participated in a formative evaluation of their programmatic services, including evidenced-based treatments (EBTs), between July 2008 and March 2011. Face-to-face qualitative interviews were conducted with over 250 staff by an independent psychologist along with on-site participant observations. This evaluation coincided with a national VA dissemination initiative to train providers in two EBTs for PTSD, Prolonged Exposure (PE) and Cognitive Processing Therapy (CPT). A substantial proportion of eligible (based on professional background) residential treatment providers received training in PE (37.4%) or CPT (64.2%), with 9.5% completing case consultation or becoming national trainers in each therapy respectively. In semi-structured interviews, providers reported that their clinical programs had adopted these EBTs at varying levels ranging from no adoption to every patient receiving the full protocol. Suggestions for improving the adoption of PE and CPT are noted, including distilling manualized treatments to essential common elements.

Implementation of evidence-based treatments (EBTs) into community settings is a public health priority (U.S. Department of Health and Human Services, 2006). Despite evidence for the efficacy of specific psychotherapies, gaps exist between practices identified as effective by research and routine clinical care. Until recently much of the research concerning adoption of EBTs was anecdotal, based on case studies or involved highly controlled experiments (McHugh & Barlow, 2010). In addition, most research did not focus on multiple programs or whole institutions (Greenhalgh, Glenn, Bate, Macfarlane, & Kyriakidou, 2005). Greater understanding of processes supporting the adoption of EBTs at the organization, department or treatment team level is needed.

Formative evaluation, which assesses service delivery and fosters its improvement, can capture complex features of health care that may influence EBT adoption (Stetler et al., 2006). Qualitative methods, particularly those assessing provider attitudes and current practices, can provide unique insights on EBT adoption. This type of knowledge might be used to support the delivery of EBTs.

The U.S. Department of Veterans Affairs (VA) is among the largest and most comprehensive providers of health care services in the world. Although in some respects unique, the VA also represents an integrated laboratory for studying implementation. Lessons learned in VA may allow for the setting of realistic goals in other less uniformly managed and resourced health care settings.

In the past five years, the VA has instituted national initiatives to provide training and consultation in two EBTs for posttraumatic stress disorder (PTSD), Prolonged Exposure (PE: Foa, Hembree, & Rothbaum, 2007) and Cognitive Processing Therapy (CPT; Resick & Schnicke, 1993) (Karlin et al., 2010). In 2010, the VA and Department of Defense (DoD) used a published evidence grading system as the basis for recommending PE and CPT as first-line treatments for PTSD. These EBTs have undergone randomized controlled trials with both non-veteran (e.g., Foa et al., 2005; Resick et al., 2008) and veteran samples (Forbes et al., 2012; Monson et al., 2006; Schnurr et al., 2007). PE is an eight to 15-session trauma-focused manualized therapy based on emotional processing theory (Foa & Kozak, 1986). It involves four main components: education about reactions to trauma and PTSD; breathing training; exposure to trauma-related situations that are objectively safe but avoided due to trauma-related distress (in vivo exposure); and exposure to trauma memories through repeated recounting out loud by the client of the details of their most disturbing event (imaginal exposure). CPT is a 12-session trauma-focused manualized therapy that can be delivered in individual or group format. Adapted from cognitive techniques (Beck & Emery, 1985), CPT begins with the trauma memory and focuses on feelings, beliefs, and thoughts which directly emanate from the trauma.

The first VA trainings for CPT and PE took place in July and November of 2007 respectively (Karlin et al., 2010) and have continued to present. There is little information available on actual use of these therapies among the more than 2,000 providers trained (Karlin et al., 2010). Anecdotal reports have ranged from disapproval (Hagemoser, 2009) to enthusiasm (Zaffert & Westrup, 2008). Formative evaluation of the training in and use of PE and CPT in VA residential PTSD treatment settings provides an opportunity to understand the effects of national psychotherapy dissemination initiatives.

Residential PTSD treatment programs have been a cornerstone of VA for the past 40 years (Rosenheck, Fontana, & Errera, 1997). Typically, these programs serve veterans whose need for treatment exceeds what can be provided in an outpatient setting. For example, these veterans may be more likely to be in crisis, have more severe symptoms and long-standing problems and less community support, than outpatients. The increased intensity, frequency, and duration of residential treatment as well as the creation of a safe supportive treatment environment without interference and disruption of normal everyday stressors is viewed as essential for intensive treatment.

The network of VA PTSD residential programs provides formal mental health services within a therapeutic treatment community. Additional services, such as social, recreational, vocational and family counseling, are facility-specific and dependent upon site-specific resources and staff interests. Some programs specialize in treating PTSD populations with specific comorbidities (e.g., substance abuse) or specialized populations (e.g., women, traumatic brain injury). There are multiple types of residential programs with variance in intensity, length of stay and treatment goals. These include: Day Hospital offering four to eight hours of treatment with the option to board for one to three weeks; Evaluation and Brief Treatment Units that offer short-term inpatient intervention from 14-28 days; Residential Rehabilitation Programs that offer treatment for 28-90 days with a goal of community reintegration; Specialized Inpatient Units that provide 28-90 days of treatment in a more restricted hospital setting; and Domiciliaries that aim to transition the veteran into outpatient care.

This paper reports data from a larger formative evaluation of services in VA PTSD residential treatment programs (Cook et al., 2011). The larger study objectives were to: 1) identify a broad range of effective therapeutic program elements noted by providers, 2) facilitate sharing of therapeutic strategies, and 3) develop an ongoing learning community of providers and programs. We specifically report findings here on the initial adoption of PE and CPT in these settings during the initial stages of the system-wide dissemination. The number of providers who received training in PE and CPT and the level of training attained were documented along with reports of program use of PE and CPT and reasons for adoption or non-adoption. Lastly whether formal training increased the likelihood of adoption was examined.

Method

Participants and Procedures

From July 2008 through March 2011, the VA’s National Center for PTSD conducted a formative evaluation to identify favored services and facilitate inter-program exchanges in the residential PTSD treatment programs that have participated in a long-standing outcome monitoring initiative led by VA’s Northeast Program and Evaluation Center (NEPEC; N=38). The timing of this effort coincided with the initial stages of VA’s dissemination of PE and CPT. These 38 sites represent programs operating in each of the 22 Veterans’ Integrated Service Networks across the U.S. Seven programs not participating in NEPEC outcome monitoring were not included.

Two-day site visits to each program were conducted during which a clinical psychologist (CO) interviewed program directors, providers and staff. A semi-structured interview guide, which was refined during the first few visits, included questions regarding treatments offered (e.g., any VA or non-VA training in PE or CPT, use of EBTs, perceived effective treatment elements, etc.) and program organization (e.g., leadership, communication, etc.). Interviews were recorded on a voluntary basis with written permission. Additionally, when deemed appropriate, participant observations in treatment team meetings and groups were permitted. Immediately following each visit, field notes were taken and the investigative team (JC, NB) was debriefed. Interviews, participant observation and field notes were used as data sources for these analyses.

Data Analysis

Verbatim de-identified transcripts of interviews and field notes were independently reviewed. Two investigative team members (CO, SD) extracted information on provider and program use of PE and CPT. This data was abstracted, condensed, and converted into ratings of adoption levels through team consensus (with JC). CO and SD then independently coded each site for level of PE and CPT adoption (i.e., “quantitized” the qualitative data; Sandelowski, 2000). Initial kappa coefficients for level of adoption between the two raters were .68 for PE and .73 for CPT. Where discrepancies arose, the team (JC, SD, CO) discussed the ratings until consensus was reached. Procedures used to optimize internal validity included interview standardization, audio-taping and professional transcription, standardized data coding in the qualitative package Atlas.ti, and an iterative approach to analysis.

Because both level of training and level of adoption were ordinal, Kruskal-Wallis analyses were used to examine the relationship between the two variables for both PE and CPT.

Results

There were no significant differences between the 38 programs that currently report outcome data to NEPEC and the seven that do not in regards to number of beds (M = 18.24, SD=11.11; M =14.71, SD=9.95, respectively), number of full-time employees (=11.06, SD=6.81; M =9.53, SD=5.47) or type of program (χ² (2)=0.30).

Descriptive information on the participating programs and providers is included in Table 1. The majority were Residential Rehabilitation Programs followed by Domiciliaries. These two program types have become increasingly similar in recent years and represent a less intensively staffed level of care than inpatient units. The programs had a wide number of currently occupied beds from zero (day hospital, with optional lodging) to 48 (M=15.26; SD=10.65) and a targeted length of stay from five to 98 days (M =47.97, SD=21.15). Across the sites, 267 staff were interviewed. The most frequently reported profession was psychologist, followed by social worker, nurse and other, including psychiatrist, admission coordinator and chaplain. Staff reported a range of program service length from a few weeks to more than 25 years.

Table 1.

Description of PTSD Residential Treatment Programs and Providers

	Programs (N = 38)
	N	%
*Type
PTSD Residential Rehabilitation and Psychosocial Residential Rehabilitation	22	57.9
Domiciliary	8	21.1
Other	8	21.1
Average Length of Stay (Days)
0-20	5	13.2
21-35	5	13.2
36-50	16	42.1
51-65	8	21.1
66 and above	4	10.6
	Staff (N= 267)
	N	%
Type of Provider
Psychiatrist	18	6.7
Psychologist	109	40.8
Social Worker	56	21
Nurse	38	14.2
Other	46	17.3
	Length of Service on the Unit
In Years	N	%
<1	35	13.1
1-3	106	39.7
4-5	28	10.5
6-10	34	12.7
11-18	37	13.9
<19	27	10.1

^*In brief the programs are defined as follows:PTSD Residential Rehabilitation Programs: Lower intensity programming with access to acute services as needed, rehabilitative focus with average length of stay (LOS) of 30 days. PTSD Domiciliary: 24/7 care. May be part of a larger domiciliary serving multiple patient populations; LOS ranges from 38-102 days. Other: Specialized Inpatient PTSD Units, PTSD Day Hospital, Evaluation and Brief Treatment PTSD Units.

As shown in Table 2, data on PE and CPT training only include staff eligible per VA training requirements at the time of site visits (n =179) and included: psychologists, social workers, psychiatrists, and nurses with advanced degrees. A substantial proportion received training in PE (37.4%) or CPT (64.2%), with 9.5% and 9.5% completing case consultation or becoming national trainers respectively. Of these, 51 received training in both PE and CPT. Although the majority of residential staff (67%) was not eligible to receive the trainings, they played influential roles within the programs and thus were included in the analyses below.

Table 2.

VA Residential Providers Training in Prolonged Exposure and Cognitive Processing Therapy

	Prolonged Exposure (PE)		Cognitive Processing Therapy (CPT)
Training Level	N	%	N	%
Did not complete any training	110	61.3	61	34.1
Informal training within VA	1	.6	3	1.7
Training outside of VA	2	1.1	0	0
Read the manual	1	.6	4	2.2
Completed 2/4 day training	48	26.9	94	52.5
Completed case consultation	14	7.8	14	7.8
Became trainers for VA	3	1.7	3	1.7
Delivery Format	# of providers delivering PE		# of providers delivering CPT
	N	%	N	%
None	132	73.7	82	45.8
Individual	40	22.3	25	14.0
Group	5	2.8	47	26.3
Individual & Group	2	1.1	25	14.0

Note: This information is from 179 providers who were eligible to receive PE and CPT trainings due to their clinical responsibilities and advanced degrees in particular professions (e.g., psychologists, social workers, psychiatrists, and nurses with advanced degrees).

The following levels of PE and CPT adoption were identified: (1) treatment was not adopted; (2) some elements of treatment were offered; (3) treatment was offered on a selective individual basis; (4) different treatment “tracks” were developed and those in a particular track received treatment; (5) full treatment protocol was given to all patients; and (6) treatment was de-adopted. The adoption levels and providers’ perception for this choice are described below.

Treatment Not Adopted

Some programs reported no use of PE (n=21, 55.3%) or CPT (n=12, 31.6%). The most commonly reported reasons were structural incompatibility (e.g., short length of stay, low staff census), and insufficient time for training or consultation.

“This program is too short. We can’t effectively implement the EBT in the way it was meant to be done.”

“I would love to do individual PE … but where am I going to find time to do 90-minute sessions?”

Staff from some programs expressed a belief that not all veterans should engage in formal trauma processing due to the potential risk of symptom exacerbation, dropout and presence of other more pressing treatment needs. These programs delivered coping or skill-based treatments only. In addition, providers explained that it would take too much time to address potential patient ambivalence towards trauma processing particularly in programs with shorter lengths of stay. Likewise, administrative responsibilities surrounding admission, discharge planning and case management are reportedly time-consuming and interfere with the ability to deliver EBTs.

“We offer an alternative for people who cannot tolerate doing intensive trauma work.”

“There is the rare patient that comes through here who...it would often be cruel to put them through ... they have been beating their wife, they got arrested, they’ve gotten tazed by the police, they’ve been using methamphetamines, is that really the time to be doing PE?”

Concern was occasionally expressed about the effect of requiring delivery of manualized treatment in a therapeutic milieu because it might compromise provider autonomy or fail to accommodate veteran refusal.

“Clinicians are being told that what they have done for years is wrong.”

“Folks might wonder why we don’t use CPT or PE up front, because we already tried to do that and people stayed away from us in droves. They won’t do it.”

There are some programs that have not adopted PE or CPT as providers believed that the effective mechanisms of these treatments were already being delivered.

“I think we are doing a lot of things that CPT would have us do because we do focus on going back to that trauma and what their thoughts and emotions were at that trauma and try to change that whole mindset ... we just don’t have the proper name on it.”

“We’re doing exposure. But what’s really nice is it’s kind of not according to a prescribed methodology.”

In particular, seven programs offer a trauma or war-zone-focused group that might be viewed as a type of exposure where there is no or minimal repetition. This approach predated the official dissemination effort.

“It used to be in the 80’s, we did trauma-focused work which now is called PE ... A lot of it is kind of a little different spin.”

“I’ve been doing exposure work for 20 plus years. This is not new. It just got packaged.”

Some Elements of Treatment Were Offered

Some providers reported that their programs use PE (n=5, 13.2%) and CPT (n=3, 7.9%), without consistent protocol adherence. Rather, they integrate a portion of these treatments. For example, although as a full protocol PE was often viewed as structurally incompatible, in vivo exposure was one component more readily integrated. In some programs, in vivo exercises were formally given and monitored.

Similarly CPT was viewed by some as difficult to deliver due to the number of structured sessions, modules, and amount of homework assigned. Instead, specific modules, worksheets, or elements were utilized. This type of “off-label” integration of treatment elements was noted as appealing to providers who, for programmatic reasons or preference, did not implement the treatment in its entirety.

“I use the beginning part of CPT, the impact statement, just across the board, because I think that that’s a real nice shift for people when they first get here.”

Treatment Offered to Select Patients

Some programs offered PE (n=8, 21.1%) or CPT (n=7, 18.4%) on an individual basis to select patients, in part because providers reported there were limited time or resources and/or that not every veteran was ready for the EBT.

“We tend to be a demand-heavy/supply-poor environment, so my choice for who gets it is who I think is gonna benefit the most. PE, of course, is not for everyone. I think it’s a powerful therapy. And it’s extremely effective for those veterans who are ready for it. Everybody’s not ready.”

Selective use of EBTs reportedly affords providers an opportunity to be more discerning about which patients would most benefit. In this level of adoption, patient “readiness for treatment” (e.g., prior experience with CBT, level of literacy, stabilized substance use, housing, familial or other psychosocial needs) was factored into treatment decision-making.

Some programs reported adopting treatments selectively because of a lack of perceived advantage over other treatment programming (i.e., current practices were perceived as equally or more effective). In particular, Eye Movement Desensitization and Reprocessing (EMDR; Shapiro, 1999) is an EBT for PTSD recognized by both VA and DoD (2010). The use of EMDR in some programs predated the VA dissemination of PE and CPT. In these programs, EMDR has not been displaced by PE or CPT but is being offered as an alternative. EMDR is viewed by these programs as equally effective and less distressing.

“The repetition involved in PE is tough. EMDR doesn’t seem to be as grueling.”

Within this level of adoption, there were modifications made to the protocols - abbreviations, extensions or additional treatments added. According to these providers, patient needs often necessitate a change in session content and sequence.

“I don’t go in with the expectation that guys are gonna be done in nine to 12-sessions. It’s probably more like 15 to 20 for these folks and ... they have other treatment needs.”

Those in Particular Track Receive Treatment

Several programs developed treatment tracks offering PE (n=2, 5.3%) or CPT (n=5, 13.2%). Some offered a trauma-focused track that included the treatment, a separate skill-building track, and occasionally a third track that met a specific need (e.g., substance abuse). Within this category, modifications might be made, such as an addition of a full autobiography to CPT.

Treatment Was Core of Program

Some programs integrated the EBT (PE: 0; CPT: n=10, 26.3%) as the core of the program and offered it to every patient. Often treatment “language” was integrated across all programming (e.g., all staff use “stuck point” terminology), suggesting a true assimilation of the treatment into program culture.

“CPT’s become sort of the foundation of the program.”

PE was not identified by any program as the core of treatment, likely in part because it was not originally designed to be offered nor has it been tested in a group format.

Treatment Was De-adopted

In a few instances PE (n=2, 5.3%) or CPT (n=1, 2.6%) were de-adopted. Reasons include provider belief in greater effectiveness or compatibility of an alternative treatment or structural incompatibility. One program previously utilizing PE instead adopted Acceptance and Commitment Therapy (ACT; Hayes, Strosahl, & Wilson, 1999).

“We ultimately decided that when we started to look at ACT that would be a better fit in the fact that it allows for exposure...but it does it in such a way that the person can kind of titrate the amount of exposure that they’re willing to experience. It puts the patient more in control of the exposure.”

A second program de-adopted PE in favor of CPT after reporting PE to be less compatible with the treatment structure.

“We found that doing individual PE regardless of how ready the patient was, by the time they ended the program it got to be where we saw more symptoms and that’s where we decided that PE probably wasn’t a good thing to do in a residential program.”

Only one program de-adopted CPT, reportedly because trained providers left the setting. The program returned to delivering a war-zone focus group.

There was a significant relationship between provider level of PE training and its adoption, χ² (6)=16.03, p < .05, as well as a significant relationship between provider level of CPT training and its adoption, χ² (6)=36.50, p < .01.

Discussion

This is the first report on the initial training in and providers’ perspectives on adoption of PE and CPT within VA residential PTSD treatment programs. Though many providers reported being trained in these EBTs, adoption was by no means universal. Adoption generally occurred along a discernable continuum ranging from no adoption, to use of only one aspect (e.g., specific worksheets), and in the case of CPT only, strict manual adherence with all patients. These findings are encouraging and suggest an expedited transfer of knowledge from research to practice. Traditionally, dissemination of new knowledge generated by randomized controlled trials into practice takes an average of 17 years (Institute of Medicine, 2001).

In no instance was PE delivered to every patient. Providers were generally of the opinion that CPT was a better fit than PE for residential treatment. This may have been a pragmatic decision as there is an evidence-based manual for group CPT as well as the evidence-based option on whether to deliver the trauma narrative (Resick et al., 2008). To date there are no evidence-based group protocols for PE.

Although most providers reported that the barriers to PE adoption were primarily structural, others perceived difficulties in implementing exposure with chronic PTSD patients. This is consistent with other work noting potential barriers to PE such as marked psychological impairment, treatment non-compliance, and unresolved life crises (Becker, Zayfert & Anderson, 2004; Litz, Blake, Gerardi, & Keane, 1990). This is inconsistent with recent uncontrolled demonstrations of PE in veterans (e.g., Tuerk et al., 2010). For example, Rauch et al. (2009) found significant reductions in PTSD after PE use in an outpatient sample of ten veterans, many of whom had psychiatric comorbidities. However, veterans participating in residential treatment may include the most chronic, difficult to treat cases. Provider concern was also expressed that imaginal exposure might cause symptom exacerbation, although this has not been substantiated elsewhere (Foa, Zoellner, Feeney, Hembree, & Alvarez-Conrad, 2002). Indeed, no differences were found in dropout rates among exposure and other EBTs, suggesting that PE may not be less tolerable than alternate cognitive-behavioral approaches (Hembree et al., 2003).

A significant minority of the providers expressed concern regarding loss of autonomy and clinical judgment in the use of EBTs, a belief that the key elements of treatment were already in use, or that alternative treatments (e.g., ACT, EMDR) may be equally efficacious. These perspectives appeared to contribute to provider and site-level adoption. While it is generally preferential to use treatments that have sound empirical evidence, use of EBTs does not negate the possible benefits of aspects of treatment as usual, clinician judgment or other EBTs not selected as part of this national dissemination.

Within a majority of the identified adoption categories, protocols were typically modified from their original format. One key to adoption may be the separation of EBTs into components and promotion of only those elements that are congruent with residential settings (e.g., accommodating to length of stay, admission type). Indeed prominent investigators have argued against maintaining strict fidelity to original treatment manuals and favor a more flexible adaptation to settings and populations (Chorpita & Regan, 2009; McHugh, Murray, & Barlow, 2009). In a randomized controlled trial a modular treatment approach produced significantly steeper trajectories of improvement than usual care and standard treatment on multiple clinical outcomes (Weisz et al., 2012). Experiments to assist in streamlining PE and CPT by eliminating or de-emphasizing non-essential components may also help identify new combinations of effective ingredients. Adaptation may facilitate transportability as well as improve potential cost-effectiveness (McHugh et al., 2009). Hoge (2011) argued that there are five core components among all EBTs for PTSD: 1) narration, 2) cognitive restructuring, 3) in-vivo exposure, 4) stress inoculation, and 5) psychoeducation. Further, as long as these five components are applied sufficiently, the “packaging” of treatments is less important. Future investigations might examine if the various ways these components are packaged have comparable effects on patient outcomes. This approach may be a better fit for practical, evolving and adaptive clinical settings (Glasgow, & Chambers, 2012).

It is important to highlight that some providers in this study had not received formal VA training in PE or CPT, although all were familiar with both treatments, and some had been trained prior to the VA initiative, during graduate school, or in other professional settings. In interpreting these initial findings it is important to take into account a dynamic temporal perspective. The majority of this data was collected between April 2009 and December 2010 and thus this was in the early stages of the national dissemination effort. In addition, there was a significant relationship between provider level of EBT training and its site adoption. Thus training in and use of these treatments has likely evolved since.

One unexplored issue was the effect of dissemination strategies for the two therapies and subsequent rates of adoption. For example, training philosophies, strategy and timeline of the PE and CPT dissemination were different and may have affected uptake. Namely, CPT training occurred over two days and was open to all interested VA clinicians, whereas PE training was four days and required a nomination from VA regional networks. Additionally, the consultation and provider certification processes for these EBTs differed.

Relatively little attention has been paid to de-adoption of EBTs (Massatti, Sweeney, Panzano, & Roth, 2007). Factors affecting discontinuation are in need of investigation and may indicate to administrators and treatment developers better ways to facilitate continued practice or future dissemination efforts.

Another consideration for future dissemination efforts is whether the goal is to engage in a “zero-defect approach” or to have some portion of programs or providers implement the treatments. There is evidence that some patients who participate in PE or CPT either fail to make clinically significant gains or continue to experience PTSD symptoms (Alvarez et al., 2011; Hembree, Cahill, & Foa, 2004). In addition, some patients dropout or refuse to participate in these therapies (Schottenbauer, Glass, Arnkoff, Tendick & Gray, 2008). In veterans with chronic PTSD, goal-setting may need to be more modest to include aims such as broad psychosocial rehabilitation.

There are several limitations of this investigation that warrant mention. VA dissemination efforts were directed at all qualifying providers treating veterans with PTSD while this study involved only residential programs. While a number of the findings may apply to outpatient programs, residential programs are a unique microcosm of the many and diverse VA PTSD services. Additionally, although kappa coefficients for independent raters coding adoption level were modest, final ratings were based on consensus. The most frequent discrepancy in independent ratings came from those programs categorized as “some elements offered” and “not adopted.” Though direct assessment of the reliability of provider self- and site-assessment was not formally conducted, an independent consistent single observer conducted the site visits and interviews. Further triangulation of provider reports of practice use may have better substantiated these findings.

These data are derived from a formative evaluation, and allowed for a semi-prospective examination of the effects of dissemination rather than a purely retrospective analysis, which may be biased by recall and reconstruction. This work is now being extended by a systematic theory-driven empirically-based data collection to evaluate the process and outcome of implementation of PE and CPT in these settings. The information presented here will serve as a baseline for examining the maintenance and modifications of treatments over time.