Intensive Referral of Veterans to Mutual-Help Groups: A Mixed-Methods Implementation Evaluation

Abstract

Community-based support group participation protects against substance use disorder (SUD) relapse, but referrals during treatment are inconsistently delivered and may not acknowledge barriers facing rural patients. This formative evaluation of a rural intensive referral intervention (RAIR) to community-based support groups for veterans seeking SUD treatment surveyed patients (N = 145) and surveyed and interviewed treatment staff (N = 28). Patients and staff did not differ significantly on quantitative ratings of the helpfulness of, or satisfaction with, seven RAIR components, but staff did not deliver the intervention consistently or as designed, citing two themes: lack of commitment and lack of resources.

In 2015, the U.S. Government Accountability Office (GAO) notified Congress that the Department of Veterans Affairs (VA) had yet to implement a number of GAO recommendations for addressing the Veterans Health Administration’s (VHA) mission to provide timely health care (U.S. Government Accountability Office, 2015). Specific areas for improvement included having clear policies, consistent processes, and adequate staff training (U.S. Government Accountability Office, 2015). Exacerbating organizational problems is the increasing demand for VHA services. The number of veterans using VHA services has been increasing over the past two decades, and demand for VHA health care is expected to exceed supply through the end of this decade (RAND Health, 2016).

Several initiatives have been undertaken to address the unmet need for veterans’ health care, including passage of the Veterans Choice Act, which facilitates hiring and training additional VHA staff while leveraging existing non-VHA resources to provide health care closer to veterans’ homes (U.S. Department of Veterans Affairs, 2014). Whether (re)training staff or coordinating with community resources, a persistent challenge for a large organization like the VHA is implementing new evidence-based practices while clinicians and other staff members are struggling to keep up with demand for services (Atkins, Kupersmith, & Eisen, 2010). To facilitate the translation of research into evidence-based practice, the VHA in 1998 launched the Quality Enhancement Research Initiative (QUERI), whose mission is “to facilitate and support ongoing improvement in outcomes and in clinical care delivery” (McQueen, Mittman, & Demakis, 2004, p. 340).

The present study evaluated the implementation of an evidence-based practice in six substance use disorders (SUD) treatment facilities in the Midwest. The implemented intervention was an adapted intensive referral (RAIR) to mutual-help groups such as Alcoholics Anonymous (AA) and similar groups which provide social support shown to reduce relapse. Intensive referral includes additional mutual-help education, encouragement, meeting coordination, family outreach, and follow-up. As described elsewhere (Grant et al., 2017), our version of the intervention was tailored to the needs of patients attending treatment far from their rural homes and rural meetings. The intervention required both training of VHA staff and coordination with existing non-VA organizations. The results of the evaluation, therefore, not only identify what elements of this intervention were and were not implemented as designed, but what the VHA staff embraced or rejected also reveals opportunities and challenges for implementing other evidence-based practices.

The demand for VHA health care is particularly acute in mental health. Among returning Operation Enduring Freedom/Operation Iraqi Freedom (OEF/OIF) veterans, an estimated 36% meet criteria for alcohol misuse (Burnett-Zeigler et al., 2011). SUD typically requires multiple treatment attempts before permanent abstinence is attained, which further taxes VHA resources. Indeed, approximately 56% of people entering SUD treatment have received treatment previously (Substance Abuse and Mental Health Services Administration, 2009). In the VHA, 40% of veterans seeking specialty SUD treatment return to treatment within two years (Hawkins, Malte, Baer, & Kivlahan, 2012). This high relapse rate makes SUD treatment one of the most costly chronic conditions treated by the VHA (Yu et al., 2003).

Most veterans receive initial intensive SUD treatment which is followed by less intensive continuing care. Patients who are able to successfully navigate the transition from intensive treatment to continuing care dramatically improve their likelihood of remaining abstinent in the long-term. The relapse rate for patients discharged from intensive SUD treatment drops to nearly zero among those who attain 100 days’ abstinence (Kirshenbaum, Olsen, & Bickel, 2009). Thus, the transition from intensive treatment to the home setting is a particularly vulnerable time for relapse. While professional continuing care services are effective, participation in continuing care may be difficult due to a variety of barriers (McKay, 2009).

Specifically, while the Uniform Mental Health Services Handbook (Department of Veterans Affairs, 2008) states that “long-term management” for SUD “must be readily accessible to all veterans when clinically indicated,” access to both SUD intensive treatment and continuing care is especially problematic for rural veterans. Rural veterans have disproportionately served in OEF and OIF (Carr & Kefalas, 2009) and are returning to their communities with significant SUD and trauma-related symptoms. Rural veterans typically receive 3–5 weeks of initial SUD treatment in a residential VHA program distant from their homes. They return to their rural communities with little support for ongoing recovery. Additionally, rural veterans likely travel greater distances and may have less access than non-rural veterans to SUD continuing care (Wallace, West, Booth, & Weeks, 2007). This isolation is particularly salient given that rural veterans with co-occurring disorders such as Post-Traumatic Stress Disorder (PTSD) are at a further disadvantage, as relapse may happen more quickly and be more severe for these veterans (Bradizza, Stasiewicz, & Paas, 2006).

Especially for those with limited access to professional continuing care, support from family and friends may enhance their likelihood of recovery (Hunter-Reel, McCrady, & Hildebrandt, 2009). Though “significant-other” participation in SUD treatment is strongly associated with positive outcomes (Hunter-Reel et al., 2009), rural veterans are disadvantaged because distance often precludes treatment participation by friends and family. The importance of post-treatment support cannot be overstated. Social support is a protective factor for both SUD treatment relapse (Beattie, 2001) and post-deployment traumatic stress symptoms (Humphreys & Moos, 2007). Support provided by community self-help groups (e.g. 12-step programs such as AA) has consistently been shown to improve outcomes (Strickler, Reif, Horgan, & Acevedo, 2012). Further, promoting support group participation can begin during treatment and can serve to reduce continuing care costs (Hunter-Reel et al., 2009). A recent panel of SUD clinicians who work with veterans within and outside the VA concluded, “The integration of self-help (AA, NA, Gambler’s Anonymous [GA], Overeaters Anonymous [OA]) is an important dimension of the Veteran’s recovery plan, and though not mandatory, it is strongly encouraged” (Carroll et al., 2016, p. 357).

A growing body of research indicates explicit, consistent messages about the benefits of self-help group attendance (“Intensive Referral”) significantly increase attendance and reduce relapse rates among veterans (Timko, DeBenedetti, & Billow, 2006; Timko, Sutkowi, Cronkite, Makin-Byrd, & Moos, 2011). Timko and colleagues at the Palo Alto VA developed a protocol and training materials for the Intensive Referral intervention, which increased one-year abstinence rates among urban veterans by more than 24% compared to standard referral. We recognized that this intervention may be similarly effective in rural veterans given their distances from professionally-delivered VHA continuing care.

The objective of this study is an implementation-focused formative evaluation of the adapted intensive referral (RAIR) intervention to community-based support groups. We adapted the original intensive referral to meet the needs of veterans who were more likely to live in rural areas. RAIR’s reach and effectiveness have been reported in a separate study (Grant et al., 2017). Therefore, this evaluation focused on RAIR adoption, implementation, and maintenance, the final three elements in the RE-AIM evaluation framework (Glasgow, Vogt, & Boles, 1999). Specifically, this study used quantitative and qualitative measures to answer three research questions

• RQ₁: Are patients satisfied with RAIR and its components?

• RQ₂: Do treatment staff perceive RAIR and its components to be useful?

• RQ₃: What changes to RAIR do treatment staff recommend?

Methods

Design

The design of this study is an implementation-focused formative evaluation (Stetler et al., 2006) in which data were gathered during the implementation of the intervention, RAIR. This formative evaluation complements the data we gathered at three sites on patient behavioral health outcomes (Grant et al., 2017). The current study entailed training staff at six sites, and then evaluating the perceptions of site leaders who adopt the intervention and of staff members who implement it, in order to identify implementation barriers and facilitators.

The conceptual model is the RE-AIM framework, as expanded by Forman and colleagues for real-time program evaluation (Forman, Damschroder, Robinson, Heisler, & Kerr, 2010). The first two elements of the framework—reach and effectiveness—are comprehensively addressed with the outcome data we reported separately (Grant et al., 2017). Therefore, this complementary project focused on the three remaining RE-AIM dimensions: adoption, implementation, and maintenance. Adoption addresses the support developed to deliver RAIR and was measured using qualitative interview questions with staff who underwent RAIR training. Implementation addresses delivery of the intervention and was also measured using qualitative interview questions with staff, focused on RAIR’s components. Maintenance addresses the long-term viability of the intervention and was measured using qualitative interview questions with staff and with Likert-style satisfaction questions asked of both staff and patients. Thus, the multimethod study’s design incorporates both qualitative interview and quantitative survey responses.

Study conditions and participants

Human subjects participation in this study was approved and monitored by the Institutional Review Board of the Department of Veterans Affairs (VA) Nebraska-Western Iowa Health Care System, which disallows access to participant data by anyone outside the research team. Patients age 19 years and older and entering SUD treatment at three VA sites were invited to participate. Those who provided consent received either standard referral or RAIR. Of 195 patients enrolled at baseline, 89 (45.6%) were assigned to standard referral and 106 (54.4%) were assigned to RAIR. This implementation study was proposed and funded after the original three-site effectiveness trial had begun. It allowed RAIR training at an additional three sites and data collection from staff at all six sites, but did not allow for data collection from patients at the additional sites. Those staff participants included 10 site leaders, 8 addiction therapists, 8 peer support specialists, and 2 members of the research staff who provided support services for the implementation.

Standard referral

One addiction therapist (AT) at each of the first three sites was trained in RAIR. Patients included in the standard referral group included those who were assigned to an addiction therapist untrained in RAIR or who received no RAIR sessions from a trained AT or peer support specialist (PSS). Standard referral refers to the way patients are traditionally encouraged to use MHG resources during and following formal treatment, that is, they may be advised to attend meetings, find a home group, and seek out a sponsor, but would likely not receive consistent follow-up regarding their participation.

Rural adapted intensive referral

An addiction therapist (AT) at each site was trained in RAIR and the patients assigned to that addiction therapist received the intervention either from the AT or from a trained peer support specialist (PSS). The intervention consists of three sessions delivered approximately one week apart (Table 1). The first session presents the evidence for MHG effectiveness, schedules a meeting and attempts to locate a MHG member (or “buddy”) to escort the patient to the meeting, and requests permission to contact a family member to explain the importance of supporting MHG participation. The second session follows up on attendance or non-attendance, explores what is expected at meetings, and schedules another MHG meeting. The final session again follows up on (non)attendance, addresses barriers to meeting attendance, and encourages participation beyond meeting attendance (e.g. sponsorship).

Table 1.

Rural Adapted Intensive Referral (RAIR) Intervention

Session #1	Assess mutual help group (MHG) experience and introduce MHGs (handout #1) Plan meeting and buddy contact (self-help journal) Schedule next session Attempt or plan family outreach (release of information and handout #5)
Session #2	Check MHG attendance Discuss (lack of) attendance Check buddy contact (handout #2) Discuss buddy (lack of) contact Attempt or plan family outreach
Session #3	Check MHG attendance Discuss (lack of) attendance Check buddy contact (handouts #3 & #4) Discuss buddy (lack of) contact Discuss support and family outreach

The original intensive referral intervention included the introductory and follow-up sessions, handouts, a self-help journal, a designated meeting to attend, and an escort to the meeting. The current study adapted and expanded these components to address the needs of patients who may return to a rural area following treatment and who therefore may not be able to attend the same MHG meetings they attended in treatment. The handouts and self-help journal were updated. Research staff located rural MHG meetings and members in rural areas willing to escort patients to meetings in or near their homes. And those delivering the intervention were trained to reach out to family members in order to answer questions and to recommend they support patients’ MHG participation. Staff were encouraged to adapt the intervention delivery to fit the circumstances of the site and the patient(s). Therefore, later sessions could be delivered in person or via telephone and could be delivered one-on-one or in group format.

Staff training

In a series of on-site, half-day training sessions, staff were educated about the value of social support and specifically support groups in promoting recovery from SUD. The intervention flow chart, brochures, journal, medical record template and fidelity instrument were reviewed in detail. The research staff then performed RAIR with a volunteer patient with staff observing. Subsequently, staff members completed RAIR with a patient with the trainer observing. The trainer and staff member each completed and discussed the fidelity instrument as a means to provide feedback. The medical record template was then completed by the staff member and trainer. The staff were also trained in identifying community support group meetings and potential MHG buddies. The investigators reviewed a protocol for identifying such groups and buddies. Staff were asked to contribute to a list of meetings and MHG buddies which could be continuously updated and accessed by VHA staff. After the initial training period, research staff observed once monthly the trained Addiction Therapists performing RAIR. Both the Addiction Therapists and the research staff completed fidelity questionnaires.

During the first year of this intervention, we discovered ATs were not delivering it according to the protocol, citing having too much to accomplish in sessions with patients to fit it in. We therefore trained peer support specialists (PSS) at each site to deliver the intervention.

Quantitative outcome measures

Each interview schedule included 18 Likert questions and the remainder were open-ended. Staff were asked to rate the training and the overall intervention, and both staff and patients were asked to rate specific RAIR components including: the brochures, the interactive sessions, efforts to identify a support group meeting for the client to attend, efforts to identify a MHG buddy, client follow-up, and family outreach. For each of these components, the participant was asked to designate on a scale from 1 (not at all) to 5 (very much) how helpful the component was and how acceptable the component was. In addition, patients answered questions about their satisfaction with their treatment experience overall using the California Treatment Outcome Project (CalTOP) instrument. Questions address how satisfactory or helpful the patient found various treatment services (e.g. alcohol counseling or case management services). Patients who indicated they used the service responded using a Likert-style scale ranging from 1 (not at all) to 5 (very much).

Qualitative outcome measures

Qualitative interviews were conducted with site leaders, addiction therapists, peer support specialists, and research staff using semi-structured interview schedules of 46 to 48 questions, depending on the role of the interviewee. They progressed according to the RE-AIM’s adoption-implementation-maintenance elements, from RAIR training, to overall impressions of RAIR, to specific components of RAIR, to client and context issues, and concluded by soliciting input on changes to the intervention. Participants rated components on a Likert-type scale as described above, then were invited to explain what they liked and what they would change about the components and RAIR as a whole. In addition, the interview guide included questions about perceptions of differential effectiveness among women, ethnic minorities, rural residents, and groups (vs. individual sessions). The interview schedule for site leaders also solicited feedback on potential barriers to RAIR adoption and changes they would make to the way support group participation is addressed in the VHA.

Data collection

Patient data were collected between March, 1, 2013, and December 11, 2014. One hundred forty-one (72.3%) of the original 195 participants were successfully interviewed at 6-month follow-up, including 78 (73.6%) of those assigned to RAIR. However, only 49 (62.8%) of the RAIR patients followed-up actually received the first session, 35 (44.9%) received the second, and 26 (33.3%) received the third (final) session. The 28 qualitative interviews with staff lasted between 18 and 67 minutes, taking place between June 19, 2014, and May 13, 2015. The interviews were transcribed verbatim, resulting 15,223 lines of text in 484 single-spaced pages.

Analysis

Transcriptions were imported into NVivo 11 Plus for Windows (QSR International Pty Ltd., Burlington, Massachusetts). The semi-structured interviews following the RE-AIM framework allowed for deductive analysis of the responses to questions. The interview schedules included subheadings (e.g. adoption or implementation) which served as analytic codes or categories comprising specific responses (Richards & Morse, 2013). These sequenced interview sections were coded first. Next, during open coding, specific responses were assigned codes which were then divided and combined using grounded theory’s constant comparison techniques (Charmaz, 1995). Codes which endorsed the status quo were additionally coded as positive; codes which recommended change (elimination or modification) were additionally coded as negative. Following this data reduction process, the number of codes and categories remained too large to be of practical use, so the decision was made to retain and present the adoption, implementation, and maintenance codes which were mentioned at least three times by staff members representing at least two different roles in at least two different sites. The resulting analysis and recommendations therefore represent a consensus of independent perspectives.

Quantitative data were analyzed using independent samples t-tests. SPSS Statistics version 24.0 was used for quantitative analysis. Two-tailed tests were used and the threshold for a type I error was p < 0.05 for all tests.

Trustworthiness

The research team used multiple strategies to enhance the rigor of the qualitative research, employing methods and applying criteria suggested by Guba and Lincoln (1981). Credibility was addressed by conducting the interviews in teams, with research team members alternating interviewing and observing, and by team members debriefing after each interview by comparing and documenting their impressions. Member checking and negative case analysis were also used to ensure credibility. Dependability was enhanced by regular consultation among researchers working at different sites and by external monitoring by a data safety monitoring board and the institutional review board. Confirmability was ensured through multi-method and multi-researcher triangulation, as team members discussed the derivation and application of codes to ensure agreement on their appropriateness and accuracy (Berg & Lune, 2012). Transferability was the primary consideration in selecting representative excerpts to include in this article.

Results

Adoption

The adoption component of the RE-AIM framework addresses the effectiveness of planning and training in preparing organizational members to implement RAIR. On a 5-point Likert scale, with 1 being not at all, staff members rated high both training helpfulness (M = 4.50, SD = 0.72) and satisfaction with training (M = 4.06, SD = 0.83). Follow-up questions asked what elements of the training the staff members found most helpful and what advice they would give for enhancing helpfulness. Table 2 summarizes staff responses across all domains.

Table 2.

Adoption and Implementation Responses Identified from Interviews with Staff Members

	Positive Responses	Negative Responses
Preparation & Training	Are good overall. Specific strategies help: observing or delivering the intervention, role playing, and interacting with trainers.	Intervention possibly redundant. Not all staff were motivated for training.
Sessions	Enhance motivation and address resistance. Efficiently focus conversations. Foster relationships enjoyed by staff and clients. Leverage group dynamics. Address a deficit in treatment.	Sessions are too structured and scripted. Time required and scheduling are problematic. Sessions and PSSs do not adequately address resistance.
Handouts	Are easy to read and understand (style). Are educational and persuasive (content). Are thorough, allowing for elaboration and discussion. Amount and sequence of information is appropriate. Bullet points and questions are specific and helpful. Most helpful are #1 (topic), #2 (topic), and #5 (topic).	Are more appropriate for some clients than others. Require processing too much information. Are too focused on AA and 12-Step groups. May be (too) repetitive. Handout #2 (topic) is redundant. Handout #5 (for family members) is under-utilized.
Meeting Match	Is a helpful component, to rural veterans especially. Attending meetings during treatment is helpful. Matching to specific meetings or fellowships is helpful. Websites and apps help identify meetings. Client feedback on meetings benefits other clients. Networking helps identify appropriate meetings.	Is more challenging for clients from distant areas. A good meeting-client match is easier in local than in distant areas. May be more challenging in a group context than face-to-face.
Buddy Match	Is a helpful component. Buddy matching resources are helpful. Clients often know/meet someone who can be a buddy.	Is not often carried out. Does not improve on existing protocols. Is labor-intensive. Is easier for staff members in recovery.
Journal	Is a helpful component. Has useful elements: signature lines, phone numbers, and anticipated challenges. Appeals to some clients more than others.	Is not used as intended by staff or clients. Signature verification is problematic.
Follow-up	Is a helpful component.	May not happen regularly. Staff are ambivalent about following up because it may seem presumptuous, the client may have relapsed.
Family Outreach	Is a helpful component. Helps the family: makes them feel involved, addresses enabling behaviors, prepares them for client discharge, and directs them toward family support resources. Family who are contacted are generally responsive.	Some staff are ambivalent because outreach requires more time and effort or it is unpleasant for the staff. Clients are reluctant to identify a support person or sign the release, especially during an early session. May demotivate clients, because family members are unavailable, unwilling, or unable to become involved.

In keeping with the high numeric ratings, staff members were in general agreement that the training was effective.

I think the training is really great, the way that it’s set up now.

[PSS]

Beneficial training strategies mentioned included role playing and interacting with trainers (e.g. asking questions), but the most frequently-mentioned beneficial strategy was delivering the intervention or observing a trainer deliver the intervention with a patient on-site.

I thought the training was very helpful. We had, you know, cause we had, uh, we were able to identify a client that came in and we did the intensive referral. I think [name of addiction therapist] was the one that did the intensive referral with him and I thought that that was really helpful

[Site Leader (SL)]

The two negative evaluations of the training were related to each other: the intervention was perceived by some to be redundant and therefore some staff members were not motivated for the training.

In reality I think that we do a really good job with this part anyway, you know, ‘cause our program is a twelve-step facilitation program so, uh, you know. Was it redundant? Probably. Redundant in what we do in classes and groups and meetings, ‘cause we talk about relapse. We talk about the importance of sponsorship. We talk about, ah, attending meetings. Um, you know, so it’s something we were, we do, maybe not in the format of what the survey was dictating, uh, but pretty much all of the same things that are do there we do as a program overall.

[SL]

The above use of the term “dictating” offers insight into some staff members’ perception of the intervention as rigid, although it was designed to offer flexibility in adapting the components to the structure of the organization and to the patient’s MHG motivation and awareness. As the site leader noted, RAIR was designed to encourage consistency in the best practices of what staff were doing anyway, yet the staff may have perceived it as another (paper)work obligation.

I see a lot of things that are organized on the top levels and passed down, but when they do that the front line folks are the ones who do the work, I mean, and, and I think it’s important to involve the front line people, the people that actually do the work instead of having someone that’s so removed from the work design the programs.

[AT]

Implementation

The implementation component of the RE-AIM framework evaluates whether an intervention was delivered as it was designed. The RAIR had seven components and all staff members were asked to rate the helpfulness of each. Follow-up interviews with patients at the original three sites had already begun at the point the implementation study at six sites was undertaken, so the component satisfaction questions were asked of only 31 patients who were subsequently followed up and who had received any RAIR components. Results are reported in Table 3. The ratings of the two groups did not differ significantly on perceived helpfulness of any component and all components were rated above the midpoint of 3 on this 5-point scale, indicating general helpfulness. On average, patients found the sessions most helpful (M = 3.87, SD = 1.41) and staff found the meeting match component most helpful (M = 4.42, SD = 0.86). The self-help journal was rated as least helpful by staff (M = 3.41, SD = 1.44) and in patients’ evaluation the self-help journal (M = 3.17, SD = 1.68) nearly tied the MHG buddy match component (M = 3.13, SD = 1.75) as least helpful. The answer to RQ₁, therefore, is that patients were moderately satisfied with RAIR, and more satisfied with the sessions and meeting match efforts than with the journal and buddy match components.

Table 3.

Follow-up RAIR Patient and Staff Perceptions of Intervention Helpfulness

	Patients (NP = 31)		Staff (NS = 26)		MP – MS	t	p
Likert: very helpful (1) – not at all helpful (5)	MP	S.D.	MS	S.D.
Sessions	3.87	1.41	3.86	1.04	.01	0.02	.98
Handouts	3.70	1.34	4.33	1.09	−.63	1.87	.07
Meeting match	3.84	1.66	4.42	0.86	−.58	1.71	.09
Buddy match	3.13	1.75	3.93	1.24	−.81	1.82	.08
Self-help journal	3.17	1.68	3.41	1.44	−.24	0.55	.59
Follow-up calls	3.56	1.74	4.20	1.15	−.64	1.53	.13
Family outreach	3.52	1.77	3.75	1.21	−.23	0.55	.59

Staff members were asked to elaborate on the implementation of the intervention and their positive and negative evaluations of the components are summarized in Table 2. Because the positive responses indicate aspects of the intervention which were implemented as designed, the focus here will be on deviation from the design, or the negative responses.

Staff embraced the goal of the sessions, which was to systematically inform patients about the importance of social support and reduce the barriers to finding that support through MHGs. However, the low completion rate of the sessions indicates that they were not carried out as designed and the primary reason given was the time required to present the information and coordinate the meeting match, buddy, and family outreach.

I made a commitment, “Yeah, we’ll do this.” And then, and the staff were on board for it, and then a—in very short period of time it was like, “Oh my God. This is, this is taking awful lot more time than what we realized it was.” Uh, so, then I started getting a lot of feet in the sand kind of thing.

[SL]

Several staff members noted that the limited time they have with clients in one-on-one sessions is often devoted to problems which seem more urgent than long-term social support (e.g. transitional housing, family problems, etc.). To alleviate the time barrier, we trained peer support specialists (who typically do not address those more urgent problems) and encouraged them to deliver at least some RAIR sessions in groups. The peer support specialists, however, typically had little experience with recovery or with group processes, so some of them struggled to address resistance to MHGs generally, and AA particularly:

The way that the sessions are designed, like, if the veteran says that they don’t want to go in AA, and we have to keep, you know, kind of pushing them, I think sometimes, that’s kind of where I get even more resistance than, than uh – than support. Or like, they’re willing to attempt something, I kind of feel like they’re trying – they almost seem like I’m trying to force it.

[PSS]

The five handouts were praised overall for both content and structure (e.g. bullet lists and question/answer). The handouts, as well as the flowcharts provided to staff members, were designed and perceived as guidelines for the in-session conversations. Some redundancy of conversational topics was incorporated in order to reinforce over several weeks the importance of MHG participation, but some staff members perceived the redundancy as excessive.

They were good pamphlets. Um. The, the only thing would probably say is that they are a little bit redundant. But then, that may be, like ‘cuz it’s part of the idea is to just keep the idea going forward. And then it will make it a little bit redundant.

[PSS]

Because 12-step groups are the most common and accessible MHGs, particularly in rural areas, the handouts focused on 12-step norms and provided evidence for the effectiveness of AA. Staff members, however noted that many patients had long-term experience with 12-Step groups and either knew about norms and effectiveness or resisted efforts to promote AA:

I feel like it might be setting up barriers for the people who come in anti-Twelve-Step. I guess that some people have told us, “Oh I feel like you’re cramming Twelve-Step down our throats.”

[SL]

The meeting match component was perceived as helpful by both patients and staff because it reduces anxiety, increases likelihood of attendance, enhances commitment, and bridges the gap between treatment meetings and meetings near the patient’s home. Practically, however, staff had difficulty locating a meeting which would seem welcoming to a veteran in early recovery (who likely has other mental health issues).

When it’s out of town, we don’t have the information on which meetings are better to go to. Where they’re even located. The times. And, um, or contact people. So they’re pretty much on their own to go, um, take the initiative.

[AT]

The meeting match problems related to both logistics and patient fit. From the beginning, the research team offered logistical support by identifying veteran-friendly 12-Step meetings in distant locations (even out of state) and by coordinating with the outreach component of 12-Step fellowships, some of whom even attended on-site training sessions. During the implementation, meeting-finder smartphone apps became more available, although at least one site faced the challenge of poor mobile phone reception in their VA hospital. The patient fit issue was perhaps more problematic, as some staff were reluctant to send patients to a meeting or a fellowship they did not already know and trust.

For similar reasons of logistics and trust, the effort to identify an individual who would accompany a patient to a MHG meeting near their home was often not attempted or accomplished. The MHG buddy was akin to a (temporary) MHG sponsor, and the goals were to alleviate anxiety about entering a meeting alone and to establish a relationship with someone in recovery. Staff members approved the goals, but noted the challenges of accomplishing them.

I think most people recognize the importance of really connecting somebody to, um, their client to another client or somebody in the 12-step. Now, getting them to actually do that is, is another hurdle to get over.

[SL]

The tradition of anonymity in 12-step groups made locating a MHG buddy in a distant town especially challenging. Some local 12-step offices were more helpful than others in coordinating contact with an individual, and doing so was always labor-intensive, even with advanced communication technology. Further, some patients already had sponsors and/or did not need an escort for various reasons. Some staff members, including those in recovery, felt reaching out to an individual in recovery was the responsibility of the patient learning how to form sober relationships.

Like several other components, staff believed the self-help journal was useful in theory because it enhances accountability, motivates meeting attendance, structures session discussions, prompts reflection, and provides feedback to the counselor. It was, however, rarely used. From the beginning, the research team used a simplified version which incorporated brief checklists and short spaces for scheduling meetings, listing topics, and providing brief reflections on meetings and individual motivations. In general, the staff members reported the patients were unable or unwilling to complete the journal entries or to bring it to their next session:

When you’re looking at a special ops guy you know, coming off meth, and you say, “I want you do to a self-help journal.” Uh uh. (Shakes head no.)

[PSS]

As a means of accountability, the journal includes a line for a meeting chairperson to sign or initial that the patient attended. Staff members reported that the signature line evoked the attendance forms often required as a condition of probation and therefore carried negative connotations. At multiple sites, however, some staff members acknowledged that they did not emphasize the importance of the self-help journal.

Well we do it in the first group, and…and we give it to everyone, so they start them on that. We just need to do better at following up with it.

[SL]

Client follow-up is actually the final two sessions, but is considered a separate component because it distinguishes RAIR from standard referral, which typically encourages meeting attendance without subsequent contact to confirm attendance or address attendance barriers. Again, staff members approved of following-up because it motivates sustained engagement, recognizes and reinforces patient accomplishments, indicates the staff member cares, provides information or feedback to the staff member, is a component of the treatment or discharge plan, can prompt a reference to additional services, and provides an opportunity to address barriers. Although follow-up sessions could be accomplished with a telephone call, staff members often did not succeed either in phoning or in making contact.

I’m not sure how much follow-up there is.

[AT]

Certainly, follow-up is difficult if a patient has relapsed and less likely to respond to calls from a treatment center. Further, the high relapse rates can lead to staff ambivalence to reach out.

It’s also sad, too, because sometimes you go, “Oh, he went back out and is not in his room and blah blah blah.” It’s sad.

[PSS]

The lack of follow-up may also reflect staff ambivalence about asking whether a patient attended a meeting they said they would attend. This ambivalence is likely to be more acute for peer support specialists, given their status as peers rather than clinicians.

I’m not always sure if it’s, uh, it’s my place.

[PSS]

The final implementation component we measured was family outreach, which was the most substantial adaptation to the original intensive referral intervention. Staff endorsed the rationale behind family outreach, saying it reinforces recovery and support group engagement, provides encouragement, provides transportation, emphasizes the systemic nature of addiction, and helps the client understand the family’s perspective. Nevertheless, staff ambivalence about following up with clients echoed their ambivalence about discussing family relationships in the earliest session.

A lot of times they don’t have a relationship with me enough to share that type of information. Especially in session ONE. Um, but I think it’s good to ask, “What type of support do you have?”

[PSS]

In many cases, the patients’ relationship with family members is so strained or broken that some staff perceive family involvement could actually undermine progress in treatment. Others have no one left who will participate in treatment or even support meeting attendance. Or, if a family member is willing to offer support, the patient may not want them to do so.

I think at that point in their recovery things are so in turmoil, if there is a family, they don’t want to add another layer to it. Or, they don’t want us to know the truth, or whatever.

[PSS]

The staff indicated the lack of family involvement is a chronic problem in VA treatment. One PSS estimated only 20 percent of patients sign a release of information (ROI) to allow staff to contact family members, although he also noted that he successfully contacts most of the ones he is allowed to contact.

Maintenance

The maintenance component of the RE-AIM framework addresses the long-term viability of the intervention. At six-month follow-up, we administered to patients the California Treatment Outcome Project (CalTOP) satisfaction survey to determine whether those who received RAIR were significantly more or less satisfied with their treatment experience than those who received standard referral. Given that some patients in the randomized to RAIR did not actually attend any sessions, we opted to compare satisfaction scores of those who received no RAIR sessions to those who received at least one RAIR session. Table 4 displays comparisons on items relevant to the intervention. The results show no significant differences in satisfaction scores. Of note, however, the largest difference in mean scores indicated those receiving at least one RAIR session rated the provision of family services more highly at a significance level of p = .05, just over the acceptable significance level of p < .05. Overall, these results suggest RAIR could be maintained as it is without greatly affecting patient satisfaction.

Table 4.

Follow-up Patient Satisfaction with Treatment

Satisfaction with Service Provision	No RAIR Sessions (N0 = 95)		1–3 RAIR Sessions (NR = 50)		M0 – MR	t	p
	M0	S.D.	MR	S.D.
Alcohol counseling helpful	4.56	0.92	4.70	0.73	−0.14	0.84	0.41
Drug counseling helpful	4.40	1.01	4.66	0.90	−0.26	1.16	0.25
Family services helpful	3.39	1.44	4.29	1.07	−0.90	2.01	0.05
Mental health services helpful	4.07	1.26	4.07	1.13	0.00	0.01	0.99
Case mgmt. services helpful	3.79	1.50	4.00	1.57	−0.21	0.44	0.66
Agreement with treatment center counselor on goals	4.03	1.14	4.24	1.07	−0.21	1.08	0.28
Counselor desire to understand you	4.29	1.14	4.56	0.79	−0.27	1.65	0.10
Working together with counselor	4.21	1.10	4.28	0.88	−0.07	0.37	0.71
Satisfaction with treatment program	4.41	0.91	4.42	0.86	−0.01	0.09	0.93
Satisfaction with treatment so far	4.15	1.15	4.22	0.98	−0.07	0.39	0.70
Treatment matched expectations	4.08	1.03	3.96	0.93	0.12	0.72	0.48
Referral to mutual-help group helpful	3.97	1.37	3.86	1.44	0.11	0.44	0.66

^*p < .05; RAIR = rural adapted intensive referral; tx = treatment; Scores reflect means on CalTOP Likert item responses ranging from 1 (not at all) to 5 (very much).

The answer to RQ₂ is that staff rated the overall helpfulness of RAIR favorably (M = 3.98, SD = 1.15). Yet, they offered a number of specific recommendations to improve the intervention (Table 5). Taken together, the quantitative and qualitative feedback allow components to be assigned to two maintenance categories. First, those that can be retained with modifications include preparation and training (RE-AIM’s adaptation component), sessions, handouts, meeting match efforts, and follow-up calls. In answering RQ3, treatment staff identified 34 specific changes in RAIR. Two particular themes emerged from these recommendations: streamlining and flexibility. Reducing the number of sessions and the number of handouts would streamline and simplify the intervention while flexibility could be achieved by expanding the intervention’s scope to include MHG fellowships beyond 12-step groups (e.g. SMART Recovery and Celebrate Recovery).

Table 5.

Maintenance and Recommendation Codes Identified from Interviews with Staff Members

Component	Recommended Changes
Adoption: Preparation & Training	Cultivate staff members. Better match training time to the amount of information. Integrate the intervention into the schedule, and do so strategically. Ensure peer support specialists (PSSs) understand group facilitation, computers, 12-Step recovery, etc. Schedule regular progress meetings.
Sessions	Use PSSs (in recovery) to facilitate groups. Enhance client motivation and alleviate staff workload through scheduling. Strategically sequence group and individual sessions to enhance impact. Adapt to the background of the client(s). Combine sessions and/or integrate with Twelve Step Facilitation sessions.
Handouts	Address mutual help groups (MHGs) other than 12-Step programs. Consolidate (content of) handouts. Match handout(s) to knowledge and readiness of a client.
Meeting Match	Connect everyone to a meeting near the treatment facility. Contact local support group members. Use Internet and apps to locate appropriate in-person and online meetings. Solicit feedback from veterans to create a veteran-friendly meetings list. Attempt to match the person to the meeting. Emphasize meeting attendance early and consistently. Pre-screen meetings. Specify the day, time, and place of the client’s meeting.
Buddy match	Access protocols designed by MHGs. Develop a list of available buddies, sorted by location. Develop a HIPAA-compliant process for staff to screen buddies. Hold meetings on-site and/or invite treatment alumni members of MHGs. Network with treatment alumni to identify buddies. Encourage clients to find buddies at meetings. Shift the buddy matching effort to the client.
Journal	No consensus on specific changes.
Follow-up	Leadership should commit to follow-up via staff encouragement, allocation of time, delegation of responsibility, and staff training.
Family Outreach	Leadership should integrate the RAIR’s outreach with current practice. Leadership should hire or designate someone to focus on family outreach. Staff should encourage meeting attendance, especially if families cannot visit the treatment facility. Staff should involve family in family programming.

Second, the RAIR components which should be eliminated or made optional include the self-help journal, buddy match, and family outreach. These three components were perceived as problematic for both patients and staff members, even if the rationale for them was widely accepted. As communication technology continues to evolve, perhaps more efficient ways of accomplishing these goals will emerge.

To address the larger issue whether some version of RAIR could be maintained at sites and even expanded to other VA treatment facilities, staff were asked explicitly why sites failed to implement the intervention consistently or as designed and what sites would need in the future to successfully implement and maintain the intervention. A number of specific suggestions were offered and they comprised two themes: commitment and resources. First, commitment by the staff members and the leadership was variable across sites and within sites.

There was a meeting, um, in the past month where there were some darts being thrown my way, with, “Who, who signed on to this [intensive referral intervention]? And who’s idea was this anyway?” ‘cause there was this perception it was being done to them.

[SL]

The above quote was from a site leader at the site with the most staff resistance. Of note, the site also had three site leaders at one site, was undergoing a reorganization, and initiated the intervention several months after the training. Further, the interviews and some site leaders’ decisions at that site (e.g. refusing to assign staff in recovery to deliver the intervention) suggested some staff members undermined the implementation in order to retain the status quo. At the other sites, lack of commitment seemed to be the result of inertia rather than intention. The research team made an effort to enlist the support of at least one site leader at each location to champion RAIR and the training session emphasized the benefits in terms of long-term clinical outcomes. Yet, even enthusiastic staff members became less committed over time.

I don’t know that they bought into it as well as they needed to have. That, and they just, it was not seen as, um, I just never got the feeling that they were as invested in it.

[SL]

To remedy the decline in commitment over time, at least one site leader suggested regular staff reinforcement.

I think maybe weekly or every-other-week meetings about the intervention to discuss it further. Discuss questions, problems, you know, so we’re making sure on an ongoing basis that they’re completing the intervention.

[SL]

The commitment issues related to a second theme, the resources required to deliver the intervention as designed. Resources here refers less to material support like handouts and self-help journals, which were supplied by the research team, than to staffing levels, time, and effort. Resources issues were cited by the most resistant site, but the resistance may have influenced the perception of resource demands. It was the only site to claim their Medical Media office demanded to approve materials and to reproduce them, which they did after a delay and in a larger size, making portability a problem. The other five sites were less resistant overall, but the additional time and effort required by RAIR was challenging given the tight scheduling and the urgent client issues which frequently interfered with it.

There’s a lot of stuff going on and we don’t have the people to do it. You know, if you could have somebody completely do this intensive referral, be assigned to it, but you know, my whole thing is: our veterans don’t get enough time with the counselors, you know.

[PSS]

Staff differed on the remedy for time and effort shortages, with some recommending RAIR be an option during treatment and others recommending it be integrated into the schedule, perhaps as part of a relapse prevention group, which already covers some of the same concepts.

If you, you can put it into the actual schedule and program, you know, it’s there. And the people that’s gonna want to participate are gonna participate. And I think, you know, if it’s put into there, and some people are saying, “Well, I don’t wanna go” and this and that, you know, because they’ve got other things to do—make it part of.

[PSS]

Discussion

This study used survey and interview data and the RE-AIM framework to research the adoption, implementation, and maintenance of the first multi-site rollout of an adapted version of intensive referral to mutual-help groups for veterans in SUD treatment. Only 26% of those in the intervention group actually received all three sessions (Grant et al., 2017). The implementation barriers in this study suggest reasons why and hold implications for present and future attempts to address organizational improvements throughout the VA system.

The theme of commitment emerged as a challenge with two particular dimensions: leadership and consistency. The support of leadership was prioritized by the research team, which perceived, with few exceptions, general enthusiasm at all levels during the training sessions. Support from leadership, however, was not consistently reinforced throughout the implementation. Implementation challenges inevitably emerge in any procedural change and the challenges that emerged in RAIR related mostly to time, effort, staff resistance to change, and patient resistance to MHG referrals. To address these commitment issues, future implementation efforts must build in reinforcement by leaders and provide clear expectations for the commitment required to succeed.

Even with these efforts, staff may resist changes, particularly those which are perceived to impinge on their autonomy. Humphreys surveyed directors of VA SUD treatment programs and found that staff often used their own judgment in deciding whom to refer to MHGs rather than consistently referring everyone (Humphreys, 1997). Of note, in the present study, the site with the greatest staff resistance also reported the greatest patient resistance to MHG referral, even though the facility followed a twelve-step facilitation model. Thus, the staff may have been emphasizing patient resistance to 12-step programs as a way to account for their own resistance to change. A similar phenomenon was the staff member who mentioned patients did not complete the self-help journal, but also noted they did not consistently ask about it. Laudet found that staff and SUD outpatients reported patients’ lack of motivation to change was more of a barrier to successful MHG referral than resistance to specific components of 12-Step programs (Laudet, 2003). In short, both patients and staff members may cite patient rejection of MHGs when they actually are both resisting a change in the status quo.

Even if staff and leadership are fully committed to intensive referral and accepting of procedural changes, a lack of resources will continue to challenge successful implementation. The shortage of mental health workers in the United States is well documented and acutely felt in the VHA (Hoge et al., 2013). Staff reported the time and effort required to complete RAIR intruded on more immediate client demands on their time. Staff also reported RAIR duplicated what they already do. Yet these two claims are inconsistent, given that those who received all three RAIR sessions were significantly more likely than those receiving no sessions to be abstinent from alcohol at 6-month follow-up (Grant et al., 2017). More likely, the urgent demands placed on staff time result in less attention to any referral to post-treatment MHG involvement—either standard or intensive referral. This explanation illustrates the need for a standardized method of referring patients while simultaneously indicating why it is so difficult to accomplish: urgent needs like housing or treatment for PTSD will routinely eclipse attention to long-term recovery. If efforts to enhance post-treatment social support are to succeed, they must receive dedicated time in the treatment schedule. Because schedules are already full, adding an intervention will require elimination of something else or creative integration with a scheduled activity. Reductions in relapse rates might serve as a metric for comparing the relative value of schedule items.

Weisz and associates identified four dissemination and implementation challenges: the implementation cliff, relevance to practice, timeline mismatch, and goal tensions (Weisz, Ng, & Bearman, 2014). Our proposal addresses the implementation cliff (i.e. the documented reduced effectiveness in successive post-pilot implementations) and the relevance of research to practice by engaging leadership and staff in the development of what will be the third and fourth generations of RAIR. Future implementation will be adapted to the feedback we received from staff and patients. The timeline mismatch refers to the typical urgency of leadership and staff vs. the deliberation of researchers. In the case of the RAIR, we face the opposite challenge: to increase the perceived urgency among leadership and staff to adapt more effective referral methods than standard practice. Finally, goal tensions are described by Weisz and associates as the “implementation limbo” or leadership and staff’s efforts to reduce evidence-based treatments to their most essential (and least costly) elements. The specific recommendations we elicited will allow us to preempt goal tensions by identifying the essential and acceptable elements in future implementations.

Limitations and Future Research

A strength of this implementation study is that it was conducted under real-world conditions in sites that were diverse in leadership, SUD treatment services, veteran populations, and personnel. This diversity, however, was also a weakness, as there was variability across site and staff member characteristics. Further, we retained the content of the intervention, but the intervention may have been affected in undetectable ways when we offered alternative ways to deliver it using peer support specialists and groups rather than addiction therapists and one-on-one sessions. Finally, the modifications made to the intervention limiting direct comparison to previous implementations. Future research should refine RAIR in accordance with the suggestions of our participants and conduct formative and summative evaluations to determine whether the modification enhance effectiveness.

Conclusion

The full potential effectiveness of intensive referral to mutual-help groups will remain unrealized until implementation challenges relating to staff commitment and resource allocation are addressed. These challenges likely undermine many efforts to introduce new evidence-based practices at individual locations within the larger VHA system. Adequate preparation and staff training are prerequisites for even small-scale procedural changes in large health care systems like the VHA, but leaders must also consistently reinforce the commitment to change and accommodate additional obligations of time and effort.