Abstract
Purpose:
Many US military veterans experience anxiety, depression, and trauma-related disorders. A major goal of the Veterans Health Administration (VHA) has been to increase access to evidence-based psychotherapies (EBPs) such as cognitive-behavioral therapy to address veterans’ substantial health burden. However, despite widespread implementation of EBPs throughout the VHA, smaller clinics that often serve rural veterans face barriers to delivering these interventions. The Veterans Affairs Coordinated Anxiety Learning and Management (VA CALM) program aims to empower providers in rural areas with varying levels of training and experience in delivering EBPs to provide high-quality cognitive-behavioral therapy for anxiety, depression, and trauma-related disorders. The goal of this study was to better understand, through qualitative interviews, VHA community-based outpatient clinic providers’ perspectives on implementing VA CALM.
Methods:
Qualitative interviews with providers (N = 22) were conducted to understand implementation of VA CALM. Template analysis was used to organize and summarize responses.
Findings:
Providers noted several facilitators for implementing VA CALM in rural community clinics, including its perceived effectiveness, broad applicability, and structure. Barriers to implementation included scheduling problems and patient-related barriers.
Conclusions:
Incorporating providers’ perspectives on factors that affect implementing cognitive-behavioral therapy in this setting may inform future efforts to disseminate-implement EBPs in smaller, more remote VHA clinics.
Keywords: anxiety, depression, qualitative, rural, veterans
Anxiety and depressive disorders and posttraumatic stress disorders (PTSD) represent a substantial burden among US veterans.1-3 Several evidence-based mental health practices (EBPs), including cognitive-behavioral therapy (CBT), have been implemented within the US Department of Veterans Affairs (VA) to effectively treat mental health disorders across the health care system.4,5 Although the VA has made it a national priority to increase veterans’ access to EBPs, including CBT, across VA medical facilities,6 many veterans in rural areas continue to face obstacles to receiving these treatments.7-9 VA community-based outpatient clinics (CBOCs) were established to provide care closer to areas where veterans live, other than urban centers where large VA hospitals are typically located.10 CBOCs are smaller than VA hospitals, often with less specialty care, and often located in more remote geographical areas to serve veterans living in these and surrounding communities. Given that 4.7 million veterans live in rural areas,11 it is important to develop innovative methods for improving access to care, including CBT, for veterans living in these more remote geographical areas.
Mental health providers in VA CBOCs face numerous challenges to delivering EBPs in their practice. These challenges can include being the only mental health provider in a clinic, isolation from peers, and limited access to continuing education/training opportunities needed to ensure EBPs are delivered with fidelity. Further, these challenges may also contribute to the finding that rural (compared to urban) veterans are less likely to have access to needed mental health care, and that, when they do, they are also less likely to receive a dose of care that is sufficient to improve their mental health symptoms.12 It is important to understand factors that may affect the adoption of innovative strategies designed to support CBOC mental health providers in overcoming challenges to delivering EBPs.
The Coordinated Anxiety Learning and Management (CALM) program was developed to guide providers in delivering CBT to patients with anxiety disorders, depression, or PTSD.13,14 CALM is a computer-delivered program that is designed to deliver personalized CBT content to the provider and patient simultaneously, while guiding the patient through an effective “dose” of CBT.14 A randomized controlled trial (RCT) found that CALM results in reductions in anxiety and depression compared to a treatment-asusual control, with gains maintained over 18 months postrandomization.14,15
Given its efficacy with civilians, CALM was recently adapted for use within VA (henceforth referred to as VA CALM) CBOCs in the South and Mid-South regions of the United States.16,17 CALM was adapted for VA in the context of an RCT comparing 2 methods (computer or manual) of delivering CALM in VA CBOCs. In the present study, we examined factors that affected the adoption of the computer or manual version of CALM among VA CBOC mental health providers (N = 22) enrolled in the RCT. We conducted qualitative interviews, guided by the Consolidated Framework for Implementation Research (CFIR),18 to assess factors affecting the “real-world” adoption of CALM in CBOCs largely providing care to rural veterans. This article presents those findings, along with implications for implementing EBPs in smaller, more remote mental health clinics.
Method
Recruitment
Mental health providers (psychiatrists, psychologists, social workers, mental health counselors, N = 22) were recruited from an RCT comparing 2 methods of delivering CALM in VA CBOCs. Providers represented 20 CBOCs within Veterans Integrated Service Networks 16 (AR, LA, MS, TX) and 17 (TX).17 VISNs are geographically divided areas within the United States representing distinct administrative regions. According to the Rural-Urban Commuting Areas system,19 the percentage of veterans who live in rural areas at each facility ranged from 4.2% to 99.4%, M = 38.3, SD = 32.77. Of these, 7 facilities had rural populations of at least 50%, with 3 facilities at 99% or more. Providers enrolled in the RCT were randomized to deliver VA CALM using a manual or computer program consisting of the same information. Following participation in the RCT, enrolled providers received an email inviting them to participate in a qualitative interview, conducted over the telephone, to obtain feedback on their experiences using CALM (manual or computer) in their CBOC. A PhD-level qualitative researcher (TA) conducted all interviews.
Data Collection
To facilitate data collection, a semistructured interview guide (see the Appendix: Interview Guides below) was developed by 2 PhD-level researchers (MC and TA) with experience adapting interventions for the VA and conducting qualitative research. The aim of the interviews was to obtain data on barriers and facilitators to using VA CALM in clinical practice. Verbal consent and permission to audio record the interview were obtained from each participant over the telephone. Phone interviews lasted approximately 30 minutes. Audio recordings of the telephone interviews were de-identified and transcribed verbatim to facilitate rapid data analysis.
Data Analysis
Data collected from qualitative interviews were analyzed using template analysis, a method of team-based qualitative analysis.20,21 Template analysis is used to organize and summarize data into predefined domains. A qualitative researcher (TA) read all interview transcripts to identify broad domains reflecting provider experiences adopting the computer or manual version of VA CALM in their clinical practice. Prototype templates for the computer and manual conditions were created in electronic documents containing these broad domains of experiences implementing VA CALM. Throughout the analysis process, individual templates were combined into a summary template, and content was grouped into categories within each domain. The process of categorization allowed identification of barriers and facilitators encountered by providers across the 20 sites. To establish analytic validity, the researcher illustrated template content with verbatim quotations from providers. The quotations allowed verification of the accuracy of content labeling and grouping by other members of the study team (ie, Cresswell and Miller22).
CFIR18 was used to guide interpretation of qualitative data collected in the telephone interviews. CFIR is a framework of constructs, organized in a set of 5 domains, that can affect the success of adopting a new clinical intervention. Qualitative results were organized by 4 of the CFIR domains, including intervention characteristics (eg, characteristics unique to VA CALM), outer setting (eg, patients’ needs), inner setting (eg, culture and policy of an institution), and individual characteristics (eg, attributes contributing to an individual’s ability to engage in the target of implementation).
Results
Barriers and facilitators are summarized by each CFIR domain in Table 1.
Table 1.
Provider-endorsed Factors Affecting Implementation of the VA CALM Program in VA CBOCs
| Intervention Characteristics Attributes of VA CALM | |
|---|---|
| Barriers |
|
| Facilitators |
|
| Outer Setting Factors external to the institution (eg, patient preferences) | |
| Barriers |
|
| Inner Setting Factors inside the institution (eg, policies) | |
| Barriers |
|
| Facilitators |
|
| Characteristics of Individuals Attributes of providers | |
| Barriers |
|
| Facilitators |
|
Intervention Characteristics
Providers described the characteristics of VA CALM that served both as a barrier and facilitator to adopting this intervention in practice. In terms of facilitators, providers noted that the computer version of VA CALM allowed them to deliver CBT in a patient-centered manner. One provider noted, “It [VA-CALM] can be individualized and personalized for what the veterans were struggling with.” Providers in both the computer and manual conditions also stated that VA CALM helped them treat mental health disorders that they were underprepared to treat. One provider reported that, “I thought it [VA-CALM] was just perfect for what I needed because my other protocols don’t really address it [anxiety] as well as CALM did.” Finally, the option to treat disorders simultaneously (eg, anxiety with or without depression) was also cited as an advantage over existing CBT-based single-disorder protocols. Providers commented, “[VA-CALM] has a wide applicability for a lot of different patients…that was a wonderful asset.” This applicability is especially important for rural CBOC providers, who may have more limited access to referrals for specialty care and resource-intensive specialty trainings in EBPs (eg, CBT for anxiety disorders), allowing them to use an evidence-based intervention to treat multiple disorders simultaneously in a population of underserved veterans living in rural areas.8
Providers using both versions of VA CALM reported that the protocol’s structure helped them better deliver care to their patients. “I loved it,” one provider stated. “It gave you the structure, to keep following it [the CBT protocol] through each step.” Other providers reported that the structure of the computer program was helpful in “walking patients through the treatment process.” For example, one provider noted that, “I think the program is great with it being on the computer. That really helps guide you through the process.” Providers also reported that they found the theoretical framework of VA CALM helpful: “What attracts me to the CALM method is the enhancement of the behavioral health approach.” They also noted that using VA CALM had helped them gain proficiency in delivering CBT: “I think that it improved my skills.”
Providers also reported characteristics of VA CALM that served as barriers to using the intervention in clinical practice. Interestingly, providers with prior training in other EBPs for treating PTSD reported that they preferred to use existing protocols to the VA CALM protocol. One provider stated that, “I don’t think it [VA CALM] can compete with prolonged exposure when it comes to getting enough exposure therapy and getting through some of the more serious PTSD.” Providers using the manual or computer version of VA CALM also stated that they had difficulties covering or “getting through” all the material in each module during sessions with patients. One provider remarked, “There’s so much material to get into 1 session that it’s hard to manage all of it.”
Additional barriers affecting VA CALM adoption were related to the complexity of the computer program. Specifically, some providers suggested that the computer program was not as easy to use as they initially expected, “It was not as user friendly…having to sift through some of the pages or some of the pieces of the program that weren’t necessarily appropriate for that person.” Technical difficulties also created challenges in navigating the computer version of VA CALM. Providers stated that, “The bugs were still being worked out with the computer program, so I think that was a handicap.” Similarly, providers stated that the manual also was overly complex and not easy to use clinically (eg, identify appropriate sections or handouts).
Several providers described a steep “learning curve” to using the manual, such that even after completing many sessions of the protocol with patients, they did not believe that they had gained adequate mastery over the material to use it clinically.
Outer Setting
The CFIR domain of “outer setting” refers to the broader context within which an institution works, including its external policies and the care needs of patients. Outer setting barriers included patient preferences for receiving treatment that was “supportive” in nature rather than a more structured intervention such as CBT. The computer version of VA CALM was sometimes a “hard sell” for older veterans. One provider reported that, “…older veterans […] aren’t used [sic] to the computer interaction.” It is noteworthy that 2 of the 3 providers citing this concern were in CBOCs with at least 30% rural enrollment. However, this experience was not universal, as one provider reported, “They [older veterans] were kind of excited [about using the computer version of VA CALM], actually… .”
Providers also suggested that, for some veterans, life stressors or co-occurring mental health disorders (eg, substance use disorders) made it difficult for them to consistently attend treatment. This affected providers’ ability to use VA CALM as a viable treatment option for some veterans. One provider stated that, “I have had some veterans that want to participate in an evidence-based therapy, but they don’t have reliable transportation. …that’s a huge barrier.” Another provider who made home visits stated that not all patients’ homes had Internet access, which prevented the provider from using the computer version of CALM with these patients. Given that individuals in rural areas are less likely to have access to the Internet,23 providers using VA CALM in home visits may need to be equipped with a mobile device (eg, laptop and tablet) with Internet access.
Inner Setting
The CFIR domain of “inner setting” refers to an organization’s internal structures and culture, such as clinical roles and resources (eg, Internet access) available to providers. One of the most commonly cited barriers affecting adoption of VA CALM was providers’ inability to schedule weekly treatment sessions with their patients. Providers using the computer or manual version of VA CALM cited several reasons why scheduling weekly treatment appointments was problematic, including a large patient panel, clinic priorities (eg, prioritizing patients in crisis and/or with high risk for suicide), and being the only (or one of few) mental health provider(s) in the clinic. Although placing patients on a wait list was an option, one provider noted that, “The risk in waiting is that people lose that motivation [to seek treatment].” Interestingly, in the computerized group the providers who noted weekly sessions and limited staff as barriers were from CBOCs with at least 30% rural enrollment. One of these providers noted, “I would say that the biggest barrier would be using any type of protocol that is weekly just because of our location and frequency, not because of the theory or the method.” One provider working at a clinic with 99% rural enrollment also noted staffing as a critical issue: “They don’t replace docs after they leave so everybody has to do more and have more—they are not any time soon going to add staff here.” Other providers noted that they had limited time to complete VA CALM with a patient, given the nature of their position such as those providers who were delivering services in the primary care setting. Conversely, providers suggested that having their clinical supervisors’ and leadership support was an important facilitator to adopting VA CALM in their practice. For example, one provider stated that their supervisor “allowed me to go to the training and allowed for the time needed.”
Barriers to adopting VA CALM related to the inner setting centered on the availability of resources, particularly for providers using the computer version. Providers using the computer version reported difficulty finding an office with a computer for a private one-on-one session. The availability of other needed equipment to facilitate delivery of the computer version of VA CALM was also a challenge. One provider noted that office computer speakers did not work, which prevented them from playing important treatment-related videos. Another noted network connection problems, stating, “It’s [the network] not fast enough for some of the videos or loading some of the pages.”
Another barrier included the treatment setting. Some providers’ job duties included clinical care activities beyond conducting psychotherapy, such as managing psychiatric medications and providing case management to patients in their home. These providers noted that competing occupational demands conflicted with consistently using VA CALM to help their patients. In contrast, providers reported that having a peer, in the same clinic, also using the computer version of VA CALM served as an inner setting facilitator to using it clinically. Providers indicated that peers provided support to “help work out” problems or challenges (eg, locating the appropriate treatment section, printing direct-to-patient materials) using the program when they encountered difficulties.
Characteristics of Individuals
The CFIR domain of “characteristics of individuals” refers to characteristics of providers that can interfere with or facilitate adoption of a new clinical practice. In general, providers reported that they believed the program was helpful for patients to reduce their mental health symptoms, especially for the treatment of anxiety disorders. One provider using the computer version of VA CALM stated, “I just think it’s a really good program, and I do hope that it can get incorporated and rolled out… .” Providers noted that the computer version of CALM was especially helpful for patients with anxiety disorders: “I have had good feedback from pretty much every patient [with an anxiety disorder] that I have seen.” Another provider noted that the computer version of CALM may be helpful for clinicians with little training in CBT, stating, “I wish I would have gotten trained in CALM before I was trained in the other evidence-based therapies. When I didn’t have any evidence-based therapy, I was looking for something like this.” VA CALM may, therefore, be an important tool for increasing evidence-based care in rural areas, which face staffing barriers to treating mental health disorders.24,25
Providers with the ability to tailor VA CALM content to the specific needs of their patients tended to be more willing to use the intervention clinically. Also, several providers reported how tailoring VA CALM content to patient needs enhanced its acceptability to patients. In fact, all providers who noted this in the manual group were from clinics with 30%-90% rurality, highlighting the intervention’s ability to be individualized as particularly relevant for rural providers. For example, some providers described first reviewing the needs of patients prior to choosing session content (as opposed to treating all patients with depression using the same material), printing materials patients might find useful prior to each session, and selecting treatment modules specifically to address patient needs (eg, education about coping and the nature of depressive symptoms). Providers who did not report tailoring VA CALM to the specific needs of their patients were more likely to report having difficulties (eg, poor patient “buy in,” difficulty getting through treatment content) using the intervention in their practice.
Other facilitators to using VA CALM included making it easier (arranging the office so both can see the computer screen) for both the provider and patient to view VA CALM content on the computer. Providers also noted that having 2 computer screens allowed the provider and patient to view VA CALM computer content more easily in session.
Discussion
In light of the interview results, VA CALM may benefit mental health providers located in smaller, more remote VA clinics. Veterans with mental health disorders (depression, anxiety, and PTSD) living in rural areas often do not receive adequate doses of psychotherapy. Interviews with providers suggested that VA CALM enabled clinicians to provide evidence-based psychotherapy for a range of mental health concerns, sometimes simultaneously, that may have otherwise gone untreated or undertreated, especially anxiety disorders. Although VA has prioritized dissemination of training in evidence-based care for depression and PTSD, there is no such effort for anxiety disorders. VA CALM may, therefore, be especially helpful in these areas for bolstering evidence-based treatment options, especially for anxiety disorders, in rural areas where such treatment may not be readily available.
Data from qualitative interviews with VA CBOC mental health providers described barriers and facilities to using VA CALM (computer and manual version) in clinical practice. Regarding barriers, one concern raised frequently by providers was the amount of material providers were asked to cover, especially in earlier sections (eg, psychoeducation about mental health symptoms) of the VA CALM treatment program. Providers reported that they commonly had less than 50 minutes to cover intervention content and were unable to extend the length of sessions. Multiple types of support and training are often needed for success when learning new EBPs.26-28 Although training in VA CALM was provided (along with supervision and external facilitation), it is likely that for some providers, additional support in tailoring the administration (eg, how to split up or shorten session content) of VA CALM in their practice is needed to optimize its adoption in this setting. Regarding facilitators of using VA CALM, providers highlighted the benefit of having co-located peers also using the VA CALM protocol in their practice as helpful to overcoming challenges administering this intervention through group problem solving and information sharing. This is consistent with literature showing that a community of practice (eg, a group of providers that share resources, information, and experience implementing a new care practice) can help providers learn and use a new clinical treatment.5,29,30 Indeed, communities of practice have been developed to support the adoption of other EBPs that are based on CBT, to good effect.5,29 Interestingly, some CFIR-related factors were reported by CBOC providers as both barriers and facilitators to using VA CALM. For example, regarding the computer modality, some providers found the computer version easy to use and helpful in efficiently navigating CALM (CBT) content. Providers with access to a high-speed Internet connection and appropriate technology (eg, functioning audio, 2 computer screens, and access to a printer) found that the computer version helped facilitate treatment delivery. Ensuring that the resources needed for clinical practice are available to rural providers (eg, mobile network access) is important in implementing and sustaining new clinical practices, and it is also important for being able to sustain such practices.31-33
In addition, only providers in the computer (and not manual) group indicated that they would continue using VA CALM in their practice at the completion of the RCT. Further, even after using the manual for several sessions with patients, only providers in the manual group reported that they believed they lacked mastery over the material. Although this does not preclude the possibility that some providers in the manual group might also continue using the protocol, these data suggest that a well-organized computer program, which carefully “walks” providers and/or patients through treatment content, is a particularly promising method for promoting the longer-term adoption of EBPs in VA CBOCs. Given that rural veterans are less likely to receive outpatient treatment and adequate doses of psychotherapy,8,34 the structure and support provided to clinicians by the VA CALM program may be especially important to increase access to mental health care among rural veterans.
VA CALM can enable CBOC mental health care providers, including those with less training in CBT, to treat common mental health conditions (eg, depression and anxiety disorders) using high-quality, evidence-based care. Given that mental health services in CBOCs are often understaffed,35 using CALM in these settings may be especially beneficial. VA CALM also fills a critical gap in current VA EBPs, given that no current treatment packages specifically target anxiety disorders and that anxiety disorders are undertreated in the VA.36 Further, VA CALM may have enhanced the efficiency of CBT treatment delivery by enabling providers to treat a range of mental health disorders using one program. Rural areas face shortages of mental health providers,37 and one quarter38 of veterans nationwide live in a shortage area. Given that providers working in clinics with high percentages of rural patients in our sample expressed limited staffing as a barrier, matching programs like VA CALM to providers’ needs and schedules may be an important next step. Efforts have been made in VA to make community mental health care more available (eg, by reimbursing services) when VA mental health care cannot be provided (or geographical distance to such care poses a burden for a veteran) in rural areas. However, such mental health care may not be available in the community, leaving rural veterans with few or no options for treatment.39 VA CALM may be an important workforce development tool for allowing providers with differing levels of training in psychotherapy and EBPs to provide high-quality care in the context of existing challenges to delivering care for common mental health disorders.
Limitations and Conclusions
The findings of this study should be interpreted in light of limitations. Although a strength of qualitative methods is an in-depth understanding of participants’ experiences and the context of those experiences, our findings may not be generalizable beyond VA mental health providers, in the South and Mid-South regions of the United States, working in VA CBOCs. Also, this study presented data collected from provider interviews only and did not include interviews from other important stakeholders (eg, veterans receiving VA CALM and clinic managers) who may have important opinions on factors that affect the adoption of VA CALM in VA CBOCs.
Despite these limitations, this study extends prior work on the clinical effectiveness of CALM in non-veteran populations,13,14 as it obtained VA CBOC providers’ perspectives on barriers and facilitators to adopting VA CALM in their practice. These findings may help inform future efforts to implement EBPs in smaller, more remote VHA clinics to better serve veterans, in need of mental health care, living in rural communities.
Funding:
This work was funded by a Department of Veterans Affairs grant CRE 12–314 (Cucciare) and is partly the result of the use of resources and facilities at the Houston VA HSR&D Center for Innovations in Quality, Effectiveness and Safety (CIN13-413) and the VA South Central Mental Illness Research, Education and Clinical Center. The funding bodies did not play a role in study design; in collection, analysis and interpretation of data; in writing the report; or in the decision to submit the article for publication. The opinions expressed are those of the authors and not necessarily those of the Department of Veterans Affairs, the U.S. government, or Baylor College of Medicine.
Appendix
Appendix: Interview Guides
Individual interviews with Department of Veterans Affairs Mental Health Providers: Computerized Version
I’d like to hear about your experiences using the CALM program with veterans. How did it go?
- To what extent do you think the face-to-face training prepared you to use Coordinated Anxiety Learning and Management (CALM) with your patients?
- Was there anything that you really liked about the training?
- Was there anything that you didn’t like?
- Could it have been improved in any way?
- Next, I’d like to talk a little about your experiences using the CALM program during sessions with veterans.
- Was there anything that you really liked about it? That you didn’t you like?
- Did you have any difficulty getting veterans comfortable with using CALM? If so, what did you do to help them engage with the program?
- Would anything improve your patients’ experiences with the program?
- What was your comfort level with using CALM to guide cognitive-behavioral therapy (CBT) sessions?
- Is there anything that would have improved your experiences?
- How do you think you would have fared at providing CBT without the program? What, if anything, did CALM add to the treatment process?
- Do you think that you will continue to provide CBT using the CALM program?
Did your supervisor support your participation in this study? If yes, how?
Did you have any Internet technology problems with the program?
What kinds of things get in the way of providing CBT with the CALM program at your clinic?
Is there anything that would help you to continue providing CBT using the CALM program after the study is over?
Do you think that the program would help community-based outpatient clinic mental health providers deliver CBT in routine care?
Do you think that these providers would be more willing to delivery CBT if they had the program?
Individual Interviews with VA MH Providers: Manualized Version
I’d like to hear about your experiences using the CALM manual with veterans. How did it go?
- To what extent do you think the face-to-face training prepared you to use CALM with your patients?
- Was there anything that you really liked about the training?
- Was there anything that you didn’t like?
- Could it have been improved in any way?
- Next, I’d like to talk a little about your experiences using the CALM manual during sessions with veterans.
- Was there anything that you really liked about it? What didn’t you like?
- Did you have any difficulty getting veterans comfortable with the manual? If so, what did you do to help them engage during therapy?
- Would anything improve your patients’ experiences with the manual?
- What was your comfort level with using the manual to guide CBT sessions?
- Is there anything that would have improved your experiences?
- How do you think you would have fared at providing CBT without the manual to guide sessions? What, if anything, did it add to the treatment process?
- Do you think that you will continue to provide CBT using the manual?
Did your supervisor support your participation in this study? If yes, how?
What kinds of things get in the way of providing CBT with the CALM manual at your clinic?
Is there anything that would help you to continue providing CBT using the manual after the study is over?
Do you think that the manual would help community-based outpatient clinic mental health providers deliver CBT in routine care?
Do you think that these providers would be more willing to deliver CBT if they had the manual?
References
- 1.Milanak ME, Gros DF, Magruder KM, Brawman-Mintzer O, Frueh BC. Prevalence and features of generalized anxiety disorder in department of veteran Affairs primary care settings. Psychiatry Res. 2013;209(2):173–179. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Seal KH, Bertentha D, Miner CR, Sen S, Marmar C. Bringing the war back home: mental health disorders among 103 788 US veterans returning from Iraq and Afghanistan seen at Department of veterans Affairs Facilities. Arch Intern Med. 2007;167(5):476–482. [DOI] [PubMed] [Google Scholar]
- 3.Seal KH, Metzler TJ, Gima KS, Bertenthal D, Maguen S, Marmar CR. Trends and risk factors for mental health diagnoses among Iraq and Afghanistan veterans using Department of veterans Affairs health care, 2002–2008. Am J Public Health. 2009;99(9):1651–1658. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Karlin BE, Cross G. From the laboratory to the therapy room: National dissemination and implementation of evidence-based psychotherapies in the U.S. Department of veterans Affairs Health Care System. Am Psychol. 2014;69(1):19–33. [DOI] [PubMed] [Google Scholar]
- 5.Karlin BE, Ruzek JI, Chard KM, et al. Dissemination of evidence-based psychological treatments for posttraumatic stress disorder in the veterans Health Administration. J Trauma Stress. 2010;23(6):663–673. [DOI] [PubMed] [Google Scholar]
- 6.US Department of Veterans Affairs. VHA Handbook 1160.01: Uniform Mental Health Services in VA Medical Centers and Clinics. Washington, DC; US Department of Veterans Affairs; 2008. [Google Scholar]
- 7.Buzza C, Ono SS, Turvey C, et al. Distance is relative: unpacking a principal barrier in rural healthcare. J Gen Intern Med. 2011;26(Suppl 2):648–654. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Cully JA, Jameson JP, Phillips L, Kunik ME, Fortney JC. Use of psychotherapy by rural and urban veterans. J Rural Health. 2010;26(3):225–233. [DOI] [PubMed] [Google Scholar]
- 9.Fischer EP, McSweeney JC, Wright P, et al. Overcoming barriers to sustained engagement in mental health care: Perspectives of rural veterans and providers. J Rural Health. 2016;32(4):429–438. [DOI] [PubMed] [Google Scholar]
- 10.Congressional Research Service veterans Health Administration. Community-Based Outpatient Clinics (R41044). 2010. [Google Scholar]
- 11.VA Office of Rural Health. Rural veterans. 2019. Available at: https://www.ruralhealth.va.gov/aboutus/ruralvets.asp. Accessed August 24, 2020.
- 12.Mott JM, Grubbs KM, Sansgiry S, Fortney JC, Cully JA. Psychotherapy utilization among rural and urban veterans from 2007 to 2010. J Rural Health. 2015;31(3):235–243. [DOI] [PubMed] [Google Scholar]
- 13.Craske MG, Rose RD, Lang A, et al. Computer-assisted delivery of cognitive behavioral therapy for anxiety disorders in primary-care settings. Depress Anxiety. 2009;26(3):235–242. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Craske MG, Stein MB, Sullivan G, et al. Disorder-specific impact of coordinated anxiety learning and management treatment for anxiety disorders in primary care. Arch Gen Psychiatry. 2011;68(4):378–388. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Roy-Byrne P, Craske MG, Sullivan G, et al. Delivery of evidence-based treatment for multiple anxiety disorders in primary care: a randomized controlled trial. JAMA. 2010;303(19):1921–1928. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Abraham TH, Marchant-Miros K, McCarthur MB. Adapting Coordinated Anxiety Learning Management (CALM) protocol for veterans Affairs community-based outpatient clinics: an iterative approach. JMIR Ment Health. 2018;5(3):e10277. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Cucciare MA, Curran GM, Craske MG, et al. Assessing fidelity of cognitive behavioral therapy in rural VA clinics: design of a randomized implementation effectiveness (hybrid type III) trial. Implement Sci. 2016;11:65. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.Damschroder LJ, Aron DC, Keith RE, Kirsh SR, Alexander JA, Lowery JC. Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implement Sci. 2009;4(1):50. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.United States Department of Agriculture. Rural-urban commuting codes; 2020. Retrieved from https://www.ers.usda.gov/data-products/rural-urban-commuting-area-codes.aspx. Accessed August 24, 2020.
- 20.Hamilton A Qualitative methods in rapid turn-around health services research. Veterans Affairs Health Services Research & Development CyberSeminar: US Department of Veterans Affairs; 2013. [Google Scholar]
- 21.Taylor B, Henshall C, Kenyon S, Litchfield I, Greenfield S. Can rapid approaches to qualitative analysis deliver timely, valid findings to clinical leaders? A mixed methods study comparing rapid and thematic analysis. BMJ Open. 2018;8(10):e019993. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.Creswell JW, Miller DL. Determining validity in qualitative inquiry. Theory Pract. 2000;39(3):124–130. [Google Scholar]
- 23.Nguyen A, Mosadeghi S, Almario CV. Persistent digital divide in access to and use of the Internet as a resource for health information: Results from a California population-based study. Inform J Med Informatics. 2017;103:49–54. [DOI] [PubMed] [Google Scholar]
- 24.Cooper CD, Helfrich CD, Thielke SM, et al. Association between mental health staffing level and primary care mental health integration level on provision of depression care in veterans Affairs medical facilities. Adm Policy Ment Health. 2018;45:131–141. [DOI] [PubMed] [Google Scholar]
- 25.Grubbs KM, Fortney JC, Kimbrell T, et al. Usual care for rural veterans with posttraumatic stress disorder. J Rural Health. 2017;33(3):290–296. [DOI] [PubMed] [Google Scholar]
- 26.Davis DA, Thomson MA, Oxman AD, Haynes RB. Changing physician performance. A systematic review of the effect of continuing medical education strategies. JAMA. 1995. 274(9):700–705. [DOI] [PubMed] [Google Scholar]
- 27.Stirman SW, Crits-Christoph P, DeRubeis RJ. Achieving successful dissemination of empirically supported psychotherapies: a synthesis of dissemination theory. Clin Psychol (New York). 2004;11(4):343–359. [Google Scholar]
- 28.Weingardt KR, Cucciare MA, Bellotti C, Lai WP. A randomized trial comparing two models of web-based training in cognitive-behavioral therapy for substance abuse counselors. J Subst Abuse Treat. 2009;37(3):219–227. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 29.Landes SJ, Matthieu MM, Smith BN, et al. Dialectical behavior therapy training and desired resources for implementation: Results from a national program evaluation in the Veterans Health Administration. J Trauma Stress. 2010;23(6):663–673. [DOI] [PubMed] [Google Scholar]
- 30.Schectman G, Stark R. Orchestrating large organizational change in primary care: the Veterans Health Administration experience implementing a patient-centered medical home. J Gen Intern Med. 2014;29(2):550–551. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 31.Gallo KP, Barlow DH. Factors involved in clinician adoption and nonadoption of evidence-based interventions in mental health. ClinPsychol (New York). 2012;19(1):93–106. [Google Scholar]
- 32.Powell BJ, McMillen JC, Proctor EK, et al. A compilation of strategies for implementing clinical innovations in health and mental health. Med Care Res Rev. 2011;69(2):123–157. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 33.Bradley EH, Webster TR, Baker D, et al. Translating research into practice: Speeding the adoption of innovative health care programs. Issue Brief (Commonw Fund). 2004;July (724):1–12. [PubMed] [Google Scholar]
- 34.Teich J, Ali NM, Lynch S, Mutter R. Utilization of mental health services by veterans living in rural areas. J Rural Health. 2017;33(3):297–304. [DOI] [PubMed] [Google Scholar]
- 35.Chapko M, Borowsky S, Fortney J, et al. Evaluation of the Department of Veterans Affairs community-based outpatient clinics. Med Care. 2002;40:555–560. [DOI] [PubMed] [Google Scholar]
- 36.Barrera T, Mott JM, Hundt N, et al. Diagnostic specificity and mental health service utilization among veterans with newly diagnosed anxiety disorders. Gen Hosp Psychiatry. 2014;36(2):192–198. [DOI] [PubMed] [Google Scholar]
- 37.Merwin E, Hinton I, Dembling B, Stern S. Shortages of rural mental health providers. Arch Psychiatr Nurs. 2003;17:42–51. [DOI] [PubMed] [Google Scholar]
- 38.Doyle JM, Streeter RA. Veterans’ location in health profession shortage areas: Implications for access to care and workforce supply. Health Serv Res. 2017;52(Suppl 1):459–480. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 39.Ohl ME, Carrell M, Thurman A, et al. Availability of healthcare providers for rural veterans eligible for purchased care under the Veterans Choice Act. BMC Health Serv Res. 2018;18:315. [DOI] [PMC free article] [PubMed] [Google Scholar]