A Qualitative Evaluation of Veterans Health Administration’s Implementation of Measurement-Based Care in Behavioral Health

Abstract

Measurement-based care (MBC) in behavioral health involves the repeated collection of patient-reported data that is used to track progress, inform care, and engage patients in shared decision-making about their treatment. Research suggests that MBC increases the quality and effectiveness of mental healthcare. However, there can be challenges to implementing MBC, such as time burden, lack of resources to support MBC, and clinician attitudes. The Veterans Health Administration (VHA) is currently undertaking a multi-phase MBC roll-out, the first phase of which included 59 sites across the country. The present study examined implementation of this initiative in an effort to learn more about the process of implementation, including best practices, challenges, and innovations. Semi-structured interviews were conducted with 20 MBC site champions and 60 staff members from 25 VHA medical centers across the country. Qualitative data analysis was conducted to identify key themes related to MBC implementation. Results were described for three components of MBC implementation: preparing for implementation, administering measures, and using and sharing data. Training and staff buy-in were key to the preparation phase. Staff members reported a variety of methods and frequencies for the collection of MBC data, with many staff members identifying a need to streamline the collection process. Staff members reported using data to track progress and adjust treatment with patients. Efforts to use data on a programmatic level were identified as a next step. Innovative solutions across clinics and sites are described in an effort to inform future MBC implementation, both within and outside of VHA.

Introduction

In recent years, there has been increasing recognition of the role of measurement-based care (MBC) in the provision of mental health services. Defined as “the systematic administration of symptom rating scales and [use of] the results to drive clinical decision-making at the level of the individual patient,” (Fortney et al., 2017), MBC has three main components: (1) collecting patient-reported data repeatedly throughout treatment; (2) using the information to inform treatment decisions; and (3) sharing the information with patients, other members of the treatment team, and/or administrators/organizations (U.S. Department of Veterans Affairs, 2016), in an effort to promote data-driven care and improve the effectiveness of treatment. However, MBC is currently not the norm in U.S. mental health care (Fortney et al., 2017). Fewer than one in five clinicians report using standardized measures to assess patient progress (Hatfield, McCullough, Frantz, & Krieger, 2010; Zimmerman & McGlinchey, 2008).

The U.S. Veterans Health Administration (VHA) began a multi-year effort to increase use of MBC in behavioral health care throughout its national treatment system (U.S. Department of Veterans Affairs, 2016). The first phase of this implementation effort (October 2016 – September 2017) was intended to be a “learning by doing” process that would inform efforts to further expand use of MBC. This article details qualitative evaluation findings from the initial phase of this effort, highlighting both potential challenges and solutions in implementing MBC in a public sector organization.

Potential Benefits of MBC

Studies have suggested that MBC can increase the quality and effectiveness of mental health care in several ways. First, MBC provides timely feedback to clinicians about the progress their patients are making in treatment (Arbuckle et al., 2013; Harding, Rush, Arbuckle, Trivedi, & Pincus, 2011). This allows clinicians to better identify patients at risk for deteriorating or not improving during the course of care (Boswell, Kraus, Miller, & Lambert, 2015; Boswell, 2017). This is essential, as clinicians may have trouble identifying these patients using clinical judgment alone (Walfish, McAlister, O’Donnell, & Lambert, 2012). MBC also supports clinician decision-making by facilitating data-driven treatment decisions (Fortney et al., 2017; Lavin, Berry, & Williams, 2017). If a patient is not improving, the clinician may consider adjusting the nature of treatment (e.g., adjusting a medication dose, considering a different medication or evidence-based psychotherapy, making a referral to address persistent symptoms) (Fortney, Sladek, & Unutzer, 2015; Kearney, Wray, Dollar, & King, 2015). In fact, MBC has been associated with positive treatment outcomes and reduced likelihood of deterioration (Kearney et al., 2015; Knaup, Koesters, Schoefer, Becker, & Puschner, 2009; Shimokawa, Lambert, & Smart, 2010). MBC can also benefit patients who are improving as expected, as it can inform discussions and decisions regarding termination or transfer to a lower level of care.

Clinicians can share feedback with their patients about their level of symptoms and treatment progress (Boswell et al., 2015). This allows patients the opportunity to be more engaged as active partners in their treatment (Fortney et al., 2015; Fortney et al., 2017). For example, MBC can be used as tool for engaging patients in shared decision-making, consistent with the principles of patient-centered care (Boswell et al., 2015; Santana & Feeny, 2014). Moreover, MBC has the benefit of being transtheoretical, such that it can be used by therapists across treatment approaches (Fortney et al., 2015; Fortney et al., 2017; Scott & Lewis, 2015).

In addition to the benefits for making decisions about individual patients, MBC data provides a metric that could be analyzed at a programmatic or organizational level. For example, a given clinic or health system could aggregate MBC data across clinicians to evaluate the effectiveness of their care or develop quality improvement initiatives (Fortney et al., 2015; Kearney et al., 2015; Lavin et al., 2017).

Given the evidence for MBC, the Joint Commission recently updated its standards to require the use of MBC in behavioral health (Lavin et al., 2017).

Challenges to Implementing MBC

Despite the benefits, there remains a large number of mental health clinicians who do not use MBC (Hatfield et al., 2010; Jensen-Doss et al., 2018; King et al., 2017; Landes et al., 2015; Zimmerman & McGlinchey, 2008). One key challenge is the time burden of MBC (Arbuckle et al., 2013; Hatfield & Ogles, 2004; Scott & Lewis, 2015). This includes the time needed to administer measures – especially if administered during a treatment session – as well as time to score, interpret, and provide feedback (Boswell et al., 2015). A prior study of VHA mental health clinicians suggested that lack of time and competing tasks were among the more salient barriers to using MBC (Landes et al., 2015). A related challenge is a lack of resources needed to support MBC, such as technology that would streamline the administration, entry, and scoring of measures; support from supervisors, colleagues, or support staff; and established procedures or tools providing guidance as to how MBC data should guide treatment decisions (Arbuckle et al., 2013; de Jong, 2016; Fortney et al., 2017; Hatfield & Ogles, 2004; Scott & Lewis, 2015).

Another challenge is clinician attitudes toward MBC (de Jong, 2016). Given the relative recency of MBC implementation efforts, many clinicians may not be trained in the use of or recognize the potential benefits of MBC. Some clinicians perceive MBC as a barrier to developing therapeutic alliance with their patients (Boswell et al., 2015), perhaps because they are concerned the outcome measures will be seen as too reductionistic or “clinical” in nature (Arbuckle et al., 2013; Hatfield & Ogles, 2004). These attitudes persist despite evidence suggesting that patients view the use of MBC instruments favorably (Connolly Gibbons et al., 2015; Malpass, Shaw, Kessler, & Sharpe, 2010). Clinicians may also have concerns about the validity of these instruments. Although there are validated brief measures for many conditions (Beidas et al. 2015), clinicians may have concerns that existing measures could be too brief or too non-specific to provide useful clinical data (Hall et al., 2014). Finally, clinicians may be concerned about the ways organizations and payers will use the data (e.g., as part of performance reviews, to determine reimbursement) (Boswell et al., 2015), and thus reluctant to incorporate MBC into clinical practice. This highlights the importance of understanding the process by which health systems implement MBC and the factors that facilitate or hamper implementation.

Implementation of MBC in the Veterans Health Administration

In 2016, VHA began undertaking a multi-phase effort to implement MBC at VHA medical centers across the country. The initiative was shaped by several contextual factors. The first was information technology. VHA has an integrated electronic health record for all of its patients which facilitates sharing of patient data across clinicians. Yet VHA also has stringent data security requirements that limit how externally acquired data can be imported to the medical record. Moreover, the Department of Veterans Affairs’ Office of Information and Technology is separate from the VHA, making integration more challenging. For example, VHA has developed tools allowing patients to remotely view information from their medical records online (Klein et al., 2015), but has yet to develop methods allowing patients to remotely enter self-reported outcomes into the medical record from their own devices. Thus, assessment data needed to be entered into the VHA electronic medical record, usually by hand entry, or through specific tablets compatible with VHA requirements. Moreover, the VHA electronic health record has little capability for customizing reports or graphs of assessment data.

The second factor considered was the decentralized nature of VHA. The VHA mental health care system encompasses a diverse range of treatment settings, including primary care, general mental health, addiction, PTSD clinics, residential programs, and inpatient programs, which might have different relevant timelines for assessment. Clinics also had to decide how to best integrate MBC implementation into staff members’ existing duties. The initiative therefore allowed individual sites some latitude in picking relevant outcomes from a list of validated measures, deciding how data would be entered, and determining frequency of assessments. Phase 1 of the initiative also began prior to the Joint Commission setting standards for assessing behavioral health outcomes (Lavin et al., 2017).

The first phase of this effort began in October 2016 and included 59 sites with 185 clinics, including outpatient substance use disorder clinics (34), outpatient PTSD clinics (33), general mental health clinics (42), primary care mental health integration (22), residential rehabilitation treatment programs (32), and other programs such as medical psychology, inpatient programs, and vocational programs (22). These Phase 1 sites volunteered to participate in the VHA implementation effort. Each participating site had a site champion whose role was to support MBC implementation locally.

All sites participating in Phase 1 of the initiative committed to using at least one core MBC measure, which included the PTSD Checklist for DSM-5 (PCL-5; Weathers et al., 2013), Patient Health Questionnaire-9 (PHQ-9; Kroenke, Spitzer, & Williams, 2001), Generalized Anxiety Disorder-7 (GAD-7; Spitzer, Kroenke, Williams, & Löwe, 2006), or Brief Addiction Monitor (BAM; Cacciola et al., 2013), administered at intake and at repeated predetermined intervals as clinically appropriate (generally not less than once every 30 days) for patients entering a new episode of care. Other measures, including those measuring functioning, quality of life, or working alliance, were encouraged but not required. Clinicians were required to enter the data into the electronic health record, which allows data to be tracked and shared among treatment clinicians. Assessment results were to be shared with the veteran at the time of the visit and with other treatment clinicians as needed to support treatment planning, assess progress, and inform shared decision-making regarding changes to treatment.

To support implementation, all participating programs were encouraged to complete an implementation plan. The implementation plan outlined key decision points including which measures would be used, method of administration, which staff would administer the measures and enter the data into the electronic health record, the frequency of assessments, clinical use of MBC, plan and timeline for getting staff trained, implementation start date, success indicators, and sustainment plan. Champions were offered live virtual trainings on how to complete the implementation plan. To support buy-in, champions were encouraged to complete these plans with input from everyone on the team implementing MBC. Additionally, champions could sign up for one or two one-hour individualized coaching sessions with an MBC subject matter expert (“Coach”) for support developing an implementation plan or navigating barriers encountered during implementation. By the end of the twelve-month Phase 1 period, 82 coaching calls were held and 56 implementation plans were voluntarily submitted for review by MBC Coaches.

Additional supports provided to Champion Sites included webinar trainings on MBC; monthly training sessions featuring specific MBC topics; monthly Community of Practice conference calls providing implementation support from Coaches and peers, organized by program type; and an online forum with support materials created by subject matter experts (e.g., brief videos demonstrating clinical interactions between veterans and clinicians, training slides, and assessment instruments) as well as content by initiative participants (e.g., completed implementation plans). Specific attitudinal barriers and perceptions of MBC were addressed via training (e.g., a webinar training addressed myths about how MBC data could negatively impact veterans’ benefits). Although Champion Sites were encouraged to take advantage of available training and implementation supports, all aspects of participation were voluntary.

Present Study

The VHA initiative was an early attempt to make MBC standard practice in behavioral health settings. As described, the VHA MBC initiative was designed to rollout in multiple phases. Phase 1 provided an opportunity to learn more about the process of implementation, including the best practices, challenges, and innovative solutions, so that lessons learned could be integrated into future phases. Initial surveys of site champions highlighted some of the implementation challenges, and indicated that sites varied in how they progressed in implementing MBC, but provided only limited information on how different clinic teams addressed emergent challenges. We therefore conducted additional qualitative interviews with both site champions and front-line staff to: 1) clarify barriers to implementing MBC; 2) identify local solutions to these implementation challenges; and 3) identify factors that were credited as contributing to success in sites that made greater progress in implementing MBC.

This evaluation adds to prior literature on MBC implementation efforts in several ways. First, the present study focused on clinicians serving an adult population across a variety of clinical settings, including general outpatient mental health, specialty mental health (e.g., posttraumatic stress disorder treatment, substance use disorder treatment), and primary care/behavioral health integration. Second, though VHA developed core MBC standards for Phase 1 sites with regard to specific measures and frequency of administration, many of the specific aspects of implementation were left to local VHA medical centers – which vary substantially with respect to size, location, and available resources. Each facility determined its own processes. Therefore, this study provides a unique opportunity to understand the different factors that shape MBC implementation, from the initial planning phase through implementation, including identifying common best practices that emerged across sites; understanding challenges faced, and how these differed by setting or population; and identifying innovative solutions designed to address implementation challenges.

Method

We conducted interviews with 1) 20 site champions, and 2) 60 VHA mental health staff involved in MBC, including leadership, clinicians, and support staff that participated in Phase 1 of the VHA MBC initiative. All participants provided verbal consent and interviews were audio recorded. This project was approved by the RAND Human Subjects Protection Committee.

We first reviewed MBC implementation data collected from the 59 VHA facilities involved in Phase 1 of the VHA MBC initiative. VHA facilities were categorized on three dimensions: 1) size, based on the number of patients served in each health care system (classified as large, medium, or small); 2) total number of patient-reported outcome measures completed by new patients beginning treatment at each facility across clinics, based on administrative data (categorized as higher, average, lower); and 3) self-reported level of MBC implementation based on a prior survey of site champions (full, partial, none). These data were used to select sites that represented a diverse range of contextual characteristics.

Interviews with Site Champions

Twenty VHA facilities were selected for participation in site champion interviews. We selected sites to represent variation with respect to the three dimensions identified above (size, total number of patient-reported outcome measures, and self-reported level of MBC implementation). Our selection process ensured that there were at least five sites from each category of these three dimensions, and our sample slightly over-represented sites with higher levels of MBC implementation based on available data (i.e., higher average patient-reported outcome measure administration, higher self-reported levels of implementation). This was to ensure that we had a range of experience in MBC implementation and included some sites that had successfully overcome barriers. Selection also accounted for variation in MBC data collection methods used. Sites that were unresponsive to or declined invitation to participate (n = 3) were replaced with VHA sites with similar characteristics. We conducted one-hour telephone interviews with site champions at 20 selected VHA facilities. The semi-structured interview included questions across the following domains: how collection of outcome data occurs; how data are used in clinical practice; how implementation is monitored at the site; and perceptions of benefits, challenges, and facilitators to implementation. These domains reflect in part stages of MBC implementation described in the literature (as noted below), as well as our efforts to understand the specific conditions that might hamper or facilitate MBC implementation.

Interviews with Staff

Ten VHA facilities were selected for VHA staff interviews. These sites were also selected to represent each category within the three dimensions described above (facility size, number of outcome measures administered, self-reported level of implementation), with at least two sites in each category of each dimension. Facilities were chosen in a separate process from those selected for site champion interviews, but there was some overlap (n = 5). Selection was weighted towards larger facilities with higher levels of implementation. Geographic region was also considered. We conducted 60 interviews, lasting 30–45 minutes, with site-level and clinic-level administrators (n = 12), clinicians (n = 43) and support staff (n = 5) across the ten VHA facilities. Interviews were either conducted in person (n = 28) or via phone (n = 32) and assessed participants’ perspectives on the processes, barriers, and facilitators related to local MBC implementation. Domains for VHA staff interviews were similar to those for the site champion interviews, but the questions had a greater focus on individual-level practices and perceptions.

Data Analysis

Data from the site champion and staff interviews were analyzed using a Rapid Assessment Process (Beebe, 2014), a team-based approach that involves iterative data collection and analysis and allows researchers to more quickly gain an understanding of the perspective of the target population. The Rapid Assessment Process used in this analysis involved organizing detailed notes from each interview into structured summaries, rather than using transcripts and direct quotes for analysis. Interviewers reviewed summary notes to ensure all relevant information was captured. Notes were coded using iterative thematic analysis and analyzed in Dedoose (v8.0.42; 2018), a qualitative data analysis software. To code data, we developed a codebook for each interview type (site champion, staff) in two phases. Codes were initially created deductively, based on a hierarchical list of topics, subtopics, and concepts drawn from the research questions, interview guide, and MBC implementation literature. Some topics were developed a priori based on prior findings from surveys on VHA implementation of MBC (e.g., processes of preparing for implementation, how data were collected, how outcomes data were used in care). Then we used an inductive approach to augment the codebook with themes that emerged during an initial review of multiple sets of notes. The project team reviewed the notes independently, then collaborated on modifying the codebook to include emergent concepts. The codebook included an initial set of parent codes and sub-codes. Site champion interviews were coded by a single project team member. Staff interviews were coded by two project team members, who began by coding three staff interviews independently. They then met to reconcile differences in code application to ensure coding accuracy. One of these project team members coded the remaining staff interviews. After the interviews were coded, results were analyzed within the framework of three stages of MBC implementation identified in the literature: preparing for MBC implementation, data collection, and use and sharing of the data. Some codes mapped directly onto one of these stages (e.g., MBC data collection, MBC data use). Other codes, such as challenges to implementation, addressed issues across these stages. For these codes, we analyzed the excerpts for both sub-codes (e.g., specific challenges experienced) and relevant stages of MBC implementation.

Results

Preparing for Implementation

Staff members reported varying degrees of familiarity with MBC prior to the initiative. About half of the staff members (34 of 60 staff members) reported having extensive experience with MBC, while a third (n = 21) reported moderate levels of experience. Only four staff members reported that they had limited familiarity with MBC, including one administrator, one clinician, and two support staff.

Choice of measures.

The VHA MBC initiative required that assessments include at least one of four core measures: PHQ-9, PCL-5, GAD-7, BAM. The majority of champions mentioned implementation of a version of the PCL-5 (n = 17) or the PHQ-9 (n = 16), followed by the BAM (n = 12) and GAD-7 (n = 9). Although the Beck Depression Inventory-II (Beck, Steer, & Brown, 1996), Alcohol Use Disorder Identification Test (AUDIT-C; Bush, Kivlahan, McDonell, Fihn, & Bradley, 1998), and quality of life measures (e.g. World Health Organization Disability Scale [WHODAS]; Ustun, Kostanjesek, Chatterji, Rehm, & World Health Organization, 2010) were not included in the official rollout, a small number of champions mentioned implementation of these measures at their sites (n = 3, n = 4, respectively).

Preparation activities.

Site champions used a variety of resources when preparing for MBC implementation. Several champions reported attending “Community of Practice” calls (15 of 20 champions) and working with VHA MBC implementation coaches who provided remote, individual consultation (13 of 20 champions). Some reported using an implementation planning worksheet (6 of 20 champions) or online resources (9 of 20 champions), such as online trainings or materials available within VHA. All of the champions reported using standing staff or team meetings to discuss MBC and provide training. In turn, a third of staff members interviewed (n = 20) reported receiving the bulk of their introduction and orientation to MBC through staff meetings. A small number described making use of other training resources, including obtaining guidance from site champions (n = 5) and using VHA online resources (n = 5).

Of the staff interviewed, about half (n = 32) explicitly described receiving training from VHA. This included training resources developed to support this initiative (e.g., online VHA trainings, Community of Practice calls that provided less structured implementation support specific to setting or context), training and communication directly from the site champion, as well as more informal on-the-job training. In large part, training provided was described as adequate. However, a small number of staff members (n = 8) identified unmet training needs or additional training tools that would have been valuable. For example, one administrator suggested the need for training slides that provide detailed information about available measures and how to introduce them to veterans, and staff members indicated that it would be helpful to have training with more explicit guidance on how to use MBC data to adjust treatment plans.

Barriers during the preparation phase.

A lack of staff or clinician buy-in was described as a barrier to MBC implementation by about half of the champions (n = 11) and staff members (n = 27). Some interviewees noted that other staff members perceive MBC as yet another requirement for clinicians without a tangible benefit, and that some staff members do not understand the rationale or benefits of MBC – although, of note, few of the staff members interviewed expressed a lack of buy-in on their own part. A small number of staff members noted that it has been more challenging to implement MBC among certain disciplines, including psychiatry (potentially due to the brevity and low frequency of visits). Some site champions noted that buy-in was particularly a challenge in clinics that were not routinely using MBC or among clinicians who made less use of evidence-based practices.

Some site champions (n = 7) noted that another barrier was lack of leadership buy-in. For example, one site champion said that staff were aware that leadership were not strongly encouraging MBC, which contributed a lack of motivation on the part of staff. Lack of leadership support also meant that some site champions had fewer available resources to support the initiative. A small number of site champions also noted that without leadership support, they had little power to enforce the MBC standards established at their sites.

Staff members also expressed concerns about the specific measures selected for the roll-out. Some described concerns about measure validity (n = 14). In some cases, this included more general concerns about brief symptom measures – for example, they capture only a snapshot in time and may not be reflective of a patient’s actual level of symptomatology, or may reflect general distress rather than symptoms of a specific diagnosis. Others cited limitations of the specific core MBC measures chosen for this initiative, including the face validity of the measures and associated concerns about response style (e.g., the face validity of items could make it easy for patients to underreport or overreport symptoms). A small number of clinicians also described a need for measures assessing other types of symptoms or quality of life.

Factors associated with successful preparation.

Site champions made several recommendations for sites preparing to implement MBC. About half of site champions (n = 9) recommended involving staff members in the implementation process, including clinicians and front-line and support staff who may be involved in the collection of MBC data. For example, champions recommended using a team-based approach to MBC implementation and having designated staff members to complete each step of MBC. They also suggested the importance of designing an MBC implementation plan with the input of staff members in more of a “bottom-up” than “top-down” approach. A small number of site champions (n = 6) also recommended securing leadership support for MBC implementation.

Relatedly, about one-quarter of staff members (n = 14) highlighted staff buy-in as key to success. In addition to involving staff members in implementation planning, staff members suggested that another way to garner buy-in might be to provide ongoing training and education. For example, staff members suggested developing a basic MBC training for new staff joining a clinic; requiring refresher courses to support ongoing implementation; and providing time for clinicians to practice MBC under supervision. One staff member described a particularly valuable training led by her clinic supervisor, in which the supervisor did a live feedback session with a veteran for clinicians to demonstrate how MBC could be discussed in session, and how to use the results for treatment planning. (For a summary of facilitators and barriers across MBC components, see Table 1.)

Table 1.

Summary of Facilitators and Barriers across MBC Components

MBC Component	Facilitators	Barriers
Preparing for implementation	• Site champion supporting implementation • Staff engagement in planning and implementation	• Lack of staff or clinician buy-in • Lack of leadership buy-in • Concerns about brief symptom measures • Concerns about specific selected measures
Administering measures	• Availability of functioning tablet or kiosk for data collection • Support staff for administration • Providing rationale for MBC for patients	• Limited time and competing demands • Technology limitations (e.g., no tablet/kiosk, software crashes) • Patient concerns (e.g., privacy, use of data) • Telemental health-related challenges
Using and sharing data	• Patient engagement in discussion/use of data • Clinical support tools	• Technology limitations (e.g., lack of straightforward way to graph progress) • Limited time and competing demands • Lack of way to aggregate data

Administering Measures

According to staff members, clinicians are most commonly responsible for administering MBC measures, including treating clinicians and case managers. Only a small number (n = 6) said that measures are generally administered by support staff. Regarding method of administration, about half the staff members reported administering measures via paper and pencil (n = 34), with a small number administering via electronic methods only (n = 5) or using some mix of paper and pencil or electronic measures (e.g., depending on the treatment modality or preferences of the veteran) (n = 10) (note that not all staff members reported on method of administration). Of those using electronic administration, some used tablets to administer measures. Others stated that they open the measures using one of the specialized mental health software options in the electronic medical record (Mental Health Assistant) or other software designed to support MBC (Behavioral Health Lab), and read items out loud and enter responses directly into the software. Though most staff members reported administering measures via paper and pencil at least some of the time, more than half of staff interviewed (n = 38) reported that they ultimately enter the data into the medical record via the Mental Health Assistant or Behavioral Health Lab.

Though support staff and administrators had some knowledge of clinicians’ practices, clinicians had the most direct insight about the frequency with which they administer MBC measures. Among the 43 clinicians, there was substantial variability in the frequency with which measures are administered. The most common administration schedules included weekly or every session (n = 14); monthly (n = 13); or on some other schedule (n = 12). A small number (n = 6) reported administering measures every other week or at the beginning of treatment, end of treatment, and once or twice during treatment. Note that these categories are not mutually exclusive; some clinicians specified different schedules for administration based on the treatment or patient. For example, one clinician reported administering MBC measures every four weeks with most patients, but adjusting the frequency to weekly when providing cognitive processing therapy. Others said they administer MBC measures more often with patients during the initial phases of treatment, and transition to less frequent administration for their longer-term patients or those in maintenance phases. For some clinicians, the schedule for measure administration was determined by their program; for example, clinicians in substance use clinics at multiple sites stated that it is standard for the BAM to be administered every 30 days.

Barriers to administration of measures.

Interviewees described a number of implementation barriers they had encountered. Both site champions (n = 12) and staff members (n = 22) identified limited time and competing demands as key challenges. Most champions noted that there is not staffing support for the administration and entry of MBC data at their sites, which is why the onus generally falls to clinicians. In turn, clinicians must balance MBC with other clinical demands. Some clinicians noted that they have brief sessions with patients (e.g., 20–30 minutes), and administering measures in session takes time that could be spent on other clinical matters. Others said that patients sometimes present to sessions with more pressing concerns (e.g., suicidal ideation), leaving little opportunity to administer MBC measures.

Because clinicians generally administer MBC measures, they noted that streamlining data collection with technology (e.g., with a tablet) might ease some of the time-related concerns. However, about half of the staff members (n = 29) reported technological obstacles related to data collection. Many do not have access to tablets or dedicated kiosks. Those who do have access to tablets described other issues, such as difficulty having the tablets communicate with the medical record. Other challenges related to limitations of the electronic medical record and MBC software. For example, clinicians reported that the different systems do not always communicate seamlessly, some programs crash without saving data, and interfaces are not user-friendly and can be cumbersome. Finally, staff members noted that not all measures of interest are available within certain programs.

A small number of staff members cited patient-related factors as a barrier. For example, three staff members reported that certain patients express concerns about privacy associated with MBC technology, and two noted that patients with cognitive or physical difficulties sometimes have trouble completing the measures. Some staff members indicated that veterans do not find MBC useful or do not understand the rationale, but that this may reflect situations in which clinicians have not taken time to describe the purpose of MBC and how data are used. Four staff members stated that some veterans are also concerned that the data will affect their service connection or disability benefits.

Finally, a small number of staff (n = 5) highlighted the challenges to collecting MBC data when providing telemental health services. Some patients receive telemental health services at local community clinics. In these cases, support staff may be available to provide a measure to the patient, but the clinician must ensure that the support staff has all needed measures in advance of the appointment. If the patient is running late, there may not be time to administer the measures as other clinicians share the equipment. Other patients receive telemental health services at home. In this case, clinicians sometimes have patients read their responses out loud or ask patients to scan the measures, but there is no streamlined way to collect the data.

Factors supporting administration of measures.

Some of the most common implementation recommendations suggested by staff members involved facilitating the collection and entry of MBC data (n = 23). Nearly half of the staff members (n = 25) specifically indicated that having a tablet or other e-screening method would reduce the burden of data collection. Six clinicians suggested that other staff members (e.g., support staff) could administer the measures rather than the treating clinicians, though others noted the importance of garnering buy-in from support staff to make this feasible. One support staff member described efforts made to facilitate data collection on behalf of clinicians by creating batteries of measures that are specific to clinician and treatment. These packets include the clinicians’ preferred measures and other clinical measures required to be completed in the electronic medical record on a regular basis. This staff member provides these to patients on the schedule specified by each clinician.

Almost all staff members (n = 53) reported that MBC implementation was more successful when patients understood the value of MBC. They suggested that most patients are willing to complete the measures if time was spent providing a rationale for the importance of the data. Some staff members suggested that veteran reactions are strongly influenced by the attitudes of their clinicians toward MBC. Regarding ways to educate and gain buy-in from veterans, one clinician reported developing a patient handout explaining MBC and using this with patients who are reluctant to complete the measures.

Using and Sharing Results

Clinician-level use and sharing of MBC data.

Both site champions and staff were asked about the ways MBC data are used in practice. In reporting the results of staff interviews, we largely focus on the reports of the 44 clinicians (rather than administrators or support staff), as they are most directly engaged in using these data in clinical work.

Some clinicians reported using data as part of the initial diagnostic process. In most cases, these clinicians indicated that MBC instruments were part of a larger diagnostic assessment (e.g., to get a broad sense of symptoms, identify potential referrals or treatment options), but were not generally used in isolation. More than half the clinicians (n = 28) indicated that they use the measures to help identify high risk patients. Most of these clinicians (n = 19) specifically noted that they use the PHQ-9 self-harm question as an initial indicator of risk.

Most clinicians (n = 33) indicated that they use MBC data to track patient progress. For example, they reported using data to determine if veterans are improving, identify individuals who may have plateaued or regressed, and assess the effectiveness of treatment or progress toward specific treatment goals. Moreover, almost all clinicians (n = 39) described using the data to adjust their treatment plan or inform treatment decisions. Specific examples included:

• recommending more intensive care (e.g., moving from outpatient to residential substance use treatment, extending the length of a residential treatment program);

• making referrals for other needed services (e.g., determining if a patient would benefit from trauma-focused treatment in addition to substance use treatment);

• adjusting medication types or dosage;

• informing the discharge planning process.

Almost all clinicians (n = 39) said that they use MBC data as a tool for discussing symptoms and treatment with their patients. Some clinicians reported having patients elaborate on specific symptoms they endorsed on a given measure, such as what they believe may account for the presence or change in certain symptoms, or exploring any discrepancy between what patients reported on a measure and what they are experiencing. Others reported that they discuss current symptom severity or describe whether there has been treatment progress. When there has not been progress, clinicians reported using the data to explore potential causes with patients – for example, if they are not fully engaging in treatment or may need additional services to address a certain set of symptoms.

Many clinicians (n = 31) reported that they use MBC data to communicate with members of the treatment team or other clinicians. In some cases, clinicians indicated that they communicate symptom severity to clinicians to justify a referral. Others reported sharing MBC data during treatment team meetings, including team meetings attended by patients.

Program-level use and sharing of data.

Aggregation of data and use of data for program evaluation purposes was deliberately de-emphasized in Phase 1 of the initiative so that the focus could be placed on the individual patient level. That said, in addition to using MBC data in clinical care, we learned of ways that clinics use MBC data to monitor clinic or program outcomes. About one-third of staff members (n = 18), including clinicians, support staff, and administrators, reported that they knew of ways data were being used to monitor outcomes at their clinics. Some reported that their clinic maintains a local database that is used to track data across clinicians. Others said that they pull data from patient medical records, though some of these staff members also described challenges to accessing data in this way. Staff members also shared unique ways the data are being used by their clinic to inform programming or quality improvement initiatives. For example, one clinic has used the data to identify factors associated with treatment engagement and drop-out. Others described using the data to determine that a group they started was not achieving the intended outcomes, or that a certain intervention was more effective when delivered in individual rather than group therapy.

Barriers to using and sharing MBC data.

About one-third of staff members (n = 21) described ways that technological limitations impeded their ability to use MBC data with patients. One of the most common issues was difficulty using existing software to generate meaningful graphs of patient progress. Staff members described the graphs as difficult to read, containing too much detail in some cases (e.g., total scores in addition to cluster scores for the PCL), or having y-axis scales that make interpretation difficult.

A small number of staff members (n = 7) described the ways that limited time and competing demands can interfere with efforts to use and share MBC data. For example, one clinician described working in a short-staffed clinic in which therapists also serve as case managers. This clinician indicated that it would be easier to generate and share graphs of patient progress if clinicians did not already have so many responsibilities to fulfill. Another indicated that when MBC results are shared with patients during treatment team meetings, this discussion can take up too much time during the 30-minute meeting. This clinician also noted that updated treatment plan information is not always available at the treatment team meeting, making it difficult to discuss MBC in the context of the treatment plan.

Some staff members described barriers to using and sharing data at a programmatic level (n = 10). Many of these staff members (n = 6) specifically noted that it is difficult to aggregate data using the existing software, and potential workarounds used by some clinics (e.g., keeping a separate Excel spreadsheet with clinic-level data) require extra time or manpower.

Facilitators of using and sharing MBC data.

As with the collection of data, patient buy-in was identified as a key facilitator of the use and sharing of MBC data. Some staff members described patients as invested in learning about their treatment progress, with several staff members highlighting graphs of scores over time as particularly valuable tools for sharing results and garnering buy-in from patients.

Of note, many staff members (n = 41) reported that MBC is valuable in that it provides important information about patient progress and/or can be used to adjust patient treatment. Some staff members shared unique strategies they use to ensure they are effectively using MBC data in their practice. For example, one clinic developed a treatment planning system that helps to connect specific BAM responses to treatment elements. Another staff member reported overlaying information about the patient’s treatment (e.g., prescription refills) with the graph of symptoms over time, and then using this to understand the impact of treatment.

Discussion

There is increasing research supporting the effectiveness of MBC, describing best practices, and identifying its benefits. In recognition of these benefits, and in an effort to make MBC the standard of care, VHA developed an initiative designed to roll-out MBC in phases to VHA medical centers and community clinics across the country. Though some MBC parameters were designated by VHA leadership (e.g., the core measures to be used), participating sites had flexibility to determine the way that MBC would be implemented. This study represented an opportunity to examine the successes of the VHA initiative, including any best practices; identify any challenges; and develop innovative solutions to these challenges.

Preparing for Implementation

In preparing to implement MBC, one of the most important activities is the selection of relevant, valid measures (Fortney et al., 2017; Harding et al., 2011; Kearney et al., 2015). VHA MBC leadership specified the measures to be used in this initiative, ensuring that selected measures were valid and appropriate to the target population. At the same time, clinicians were given flexibility to administer other measures, including quality of life measures. Our discussions with site champions suggested that most sites focused on the core VHA measures, but some staff members expressed an interest in using other measures as part of their clinical practice. As clinicians become more comfortable with the practice of MBC, it is possible that they will begin to identify and incorporate supplemental measures relevant to the patient population they serve.

Another key element of the VHA MBC initiative was the identification of an MBC champion at each implementation site. This is a practice recommended in the MBC literature as a way to increase the “adoption, implementation, and sustainability” of MBC efforts (Boswell et al., 2015). These site champions were important facilitators of MBC, in that they worked to garner leadership support, develop and disseminate training materials, and engage staff members in the planning process.

We also found that staff buy-in across levels was key to implementation success. Site champion interviews suggested that there are many ways leadership can demonstrate support, including providing words of encouragement, allowing protected time for site champions to work on MBC implementation, and incorporating expectations for MBC into local site policies. Clinician buy-in was also described as important to MBC implementation. Champions and staff members suggested incorporating MBC training into the on-boarding of new staff members, engaging staff in the implementation planning process, helping clinicians understand the clinical utility of MBC, and sharing positive feedback from patients about the use of MBC.

An interesting finding that emerged is that some disciplines, such as psychiatrists, may experience additional challenges to MBC implementation. Psychiatry visits are typically only scheduled as often as is needed to make treatment decisions (i.e., maintain or change medications) and visits can be brief, making it challenging for psychiatrists to find time to integrate MBC into their practice. That said, psychiatrists often serve as the formal care manager for patients, which could make these visits an especially appropriate time to use MBC. Therefore, finding ways to promote MBC in this group of clinicians – such as engaging support staff to assist in the collection of data or providing additional education about the importance of MBC – could be valuable.

Administering Measures

The MBC literature emphasizes the importance of collecting data at regular intervals (U.S. Department of Veterans Affairs, 2016); however, there remains limited guidance as to the ideal frequency of data collection. Clinicians in this study reported substantial variability in how frequently measures were administered. That said, our interviews also provided insight into the types of considerations that determined how frequently clinicians administered MBC measures. For example, some clinicians adjusted the schedule of administration based on the phase of treatment (e.g., assessing more often during the acute phase vs. less often during the maintenance phase), treatment modality (e.g., administering more or less often for group versus individual therapy), or specific therapy being implemented (e.g., administering more often for evidence-based trauma treatments, which may have a specified administration schedule). Others defaulted to the schedule prescribed by their clinic (e.g., monthly BAM administration). The ideal frequency of data collection may in fact vary based on many of these considerations; however, research suggests that timely feedback may be the most effective (e.g., Kearney et al., 2015; Shimokawa, Lambert, & Smart, 2010). For example, providing clinicians with session-by-session feedback about patient progress has been associated with improved treatment outcomes (Harmon et al., 2007). Another study found that when clinicians received more prompt feedback about their patients’ progress (i.e., provided one week vs. two weeks after a measure was administered), their patients achieved similar clinical outcomes but in fewer sessions of care, suggesting that feedback allowed for more efficient provision of care (Slade, Lambert, Harmon, Smart, & Bailey, 2008).

Our interviews revealed a number of implementation barriers related to the collection of data, including lack of an easy mechanism for data collection and limited time. Certainly, these barriers are not unique to the VHA setting and have been identified in other MBC implementation studies (e.g., Arbuckle et al., 2013; Boswell et al., 2015; de Jong, 2016; Fortney et al., 2017; Hatfield & Ogles, 2004; Scott & Lewis, 2015). Many staff members noted that having a simple way for veterans to electronically complete the measures would reduce the burden of data collection on the clinician and increase their buy-in for MBC. This approach has been used by another large governmental healthcare system, which now collects patient-reported outcome data on over 70% of all outpatient encounters, through the development of a parallel data collection system that captures patient-reported outcomes (separate from the electronic medical record) (Hoge et al, 2015; Ivany et al., 2019). This type of technology-based solution also has the potential to be more scalable than relying on support staff or clinicians to manually tabulate and record data.

Because MBC is designed to engage patients as active participants in their treatment (Fortney et al., 2015; Fortney et al., 2017), patient buy-in is also key. Most staff members indicated that patients are generally amenable to the collection of MBC data. When patients do express concerns about the completion of MBC measures, staff members noted that this can often be successfully addressed by providing a thorough explanation for the purpose and importance of MBC.

Using and Sharing Results

At the patient level, best practices in MBC include using the data to monitor progress; sharing feedback with patients about their symptoms and treatment; and engaging patients in data-driven, shared decision-making about their treatment (Arbuckle et al., 2013; Boswell et al., 2015; Fortney et al., 2017; Harding et al., 2011; Lavin et al., 2017; Santana & Feeny, 2014). The majority of clinicians interviewed reported using MBC data in all of these ways. This suggests that, in large part, clinicians and other staff understand the rationale for MBC. However, given the barriers to optimal MBC data collection, there may be related limitations to optimizing the use of the data. If barriers to measure administration are successfully addressed, additional challenges to using and sharing the data may arise. For example, in a related study of 26 clinician-patient dyads, we found that clinicians varied in how well they integrated MBC into care and used it to support shared decision-making (Brooks Holliday et al., under review). This will be important to monitor as VHA MBC implementation expands.

Staff members also identified certain areas for growth. Although there has been increased measurement of patient symptoms, clinicians are still learning how to use that information to improve shared decision-making and guide better treatment decisions. This is critical, because more appropriate and timely treatment decisions are the main mechanism through which MBC can potentially improve patient outcomes (Kearney et al., 2015; Shimokawa, Lambert, & Smart, 2010). Some participants suggested the development of normative feedback on patient progress, and resources that guide clinicians in how to adjust a patient’s treatment plan if data show they are not showing sufficient progress (Boswell et al., 2015; Kearney et al., 2015).

On a programmatic level, MBC creates opportunities to evaluate the effectiveness of programming and develop quality improvement initiatives (Fortney et al., 2015; Kearney et al., 2015; Lavin et al., 2017). Our interviews suggested that, because VHA’s current approach is intentionally focused on decision-making for individual patients, there are no straightforward way to aggregate data at the treatment team, clinic, hospital or enterprise levels, and no easy procedures for adjusting outcomes for patient case-mix, which has limited the use of the data for programmatic initiatives. As MBC becomes more standard across VHA medical centers and if a standard technical program is disseminated, this important next step will become possible.

Limitations

Our study has certain limitations. First, sites volunteered to participate in Phase 1 of this initiative. Therefore, participating sites likely had some level of buy-in to MBC before the initiative began. There may be different implementation challenges at sites with less experience implementing MBC or sites that are required to implement MBC, rather than volunteer, or certain challenges may be more pervasive (e.g., lack of clinician buy-in). In addition, our interviews were limited to a subsample of these pilot implementation sites. Though we aimed to represent VHA medical centers across a range of characteristics, there may be ways in which our results do not generalize to other medical centers. The second phase of MBC implementation (October 2017 to September 2018) in VHA expanded MBC to all residential treatment programs and compensated work therapy programs (which are covered by the new Joint Commission standards for measuring behavioral outcomes) and to all addiction treatment programs throughout the country; other programs could choose whether to adopt MBC. The process of mandating implementation of MBC as a required standard may differ from the experience of early adopters who volunteered to implement MBC.

Second, we relied on semi-structured interviews to collect our qualitative data, which included several prompts. This may have led champions and staff members to endorse more knowledge and use of MBC than is reflective of their care (Ross, Ionita, & Stirman, 2016). This, combined with the fact that our sample was not a representative sample, contributed to our decision not to report percentages with our data. However, use of detailed prompts also allowed us to probe the various ways clinicians use patient-reported data, even if they may not have initially conceptualized it as an aspect of MBC (e.g., monitoring patient risk).

Finally, our study was aimed at understanding challenges in implementation, not explaining variation in implementation success across clinics or sites. Without a definitive measure of successful implementation that includes how data were used in care, we could not conduct an explanatory sequential mixed-methods study in which we explicitly selected and compared sites with high and low implementation success. We conducted some exploratory analyses by site, but found few themes that seemed to vary by site or clinic characteristics, suggesting that many of the implementation barriers and facilitators may be consistent across sites. Determining which implementation approaches led to more successful adoption of MBC is a potential avenue for future research.

Strengths and Future Directions

This study also has several strengths. First, our use of qualitative data allowed us to gather rich detail about the VHA MBC implementation efforts. As described above, initial surveys had suggested certain implementation challenges and site-to-site variability in implementation efforts; our study explored these in further detail and provided additional context for those findings. In addition, our use of qualitative data allowed us to highlight innovative strategies being implemented at certain VHAs or by certain staff members. Finally, this method allowed us to understand the substantial variability that can occur even within a single VHA medical center, including variability on the clinician and clinic levels. In addition, the results of this study will help VHA to identify what types of implementation resources medical centers may need, and develop plans for the ongoing implementation and sustainability of MBC within the health system. Ultimately, this will support continuing VHA efforts to improve the quality, outcomes, and patient-centeredness of care provided to veterans.

On a broader scale, the VHA MBC initiative represents the efforts of a large health system to make MBC the standard of care across its facilities. This study suggests that there are certain core elements that should be considered by health systems interested in implementing MBC, which are consistent with previous research. This includes ensuring buy-in from staff at all levels, including leadership; disseminating standard technology to collect and display MBC data for clinicians; and ensuring that staff have access to the resources needed to implement MBC effectively (see Table 1). This study found that most clinicians reported using MBC data in ways that are consistent with best practices, including monitoring progress, sharing feedback with patients, and engaging with shared decision-making. To the extent possible, health systems should make tools available to support these functions for all clinicians. This could include necessary technology, but also decision-guides.