Evaluation of an Electronic Health Record (EHR) Tool for Integrated Behavioral Health in Primary Care

Abstract

Background:

Integrating behavioral health into primary care can improve care quality; however, most electronic health records are not designed to meet the needs of integrated teams. We worked with practices and behavioral health (BH) clinicians to design a suite of electronic health record tools to address these needs (“BH e-Suite”). The purpose of this article is to examine whether implementation of the BH e-Suite changes process of care, intermediate clinical outcomes, and patient experiences, and whether its use is acceptable to practice members and BH clinicians.

Methods:

We conducted a convergent mixed-methods proof-of-concept study, implementing the BH e-Suite across 6 Oregon federally qualified community health centers (“intervention clinics”). We matched intervention clinics to 6 control clinics, based on location and patient panel characteristics, to assess whether process of care (Patient Health Questionnaire-9 [PHQ-9] and Generalized Anxiety Disorder-7 screening) and intermediate outcomes (PHQ-9, Generalized Anxiety Disorder-7 scores) changed postimplementation. Prepost patient surveys were used to assess changes in patient experience. To elucidate factors influencing implementation, we merged quantitative findings with structured observations, surveys, and interviews with practice members.

Results:

Implementation improved process of care (PHQ-9 screening). During the course of the study, change in intermediate outcomes was not observed. Degree of BH e-Suite implementation varied: 2 clinics fully implemented, 2 partially implemented, and 2 practices did not implement at all. Initial practice conditions (eg, low resistance to change, higher capacity), process characteristics (eg, thoughtful planning), and individual characteristics (eg, high self-efficacy) were related to degree of implementation.

Conclusions:

Health information technology tools designed for behavioral health integration must fit the needs of clinics for the successful uptake and improvement in patient experiences. Research is needed to further assess the effectiveness of this tool in improving patient outcomes and to optimize broader dissemination of this tool among primary care clinics integrating behavioral health. ( J Am Board Fam Med 2018;31:712–723.)

Each year, 26% of the United States population experiences an emotional, mental, or behavioral health problem, and the majority of these individuals are seen in primary care settings^.1,2 Here we define behavioral health as encompassing “any behavioral problems bearing on health, including mental health and substance abuse conditions, stress-linked physical symptoms, patient activation, and health behaviors”.³ Research supports the inseparability of behavioral health care and medical care^4–6, and strong evidence shows that integration, bringing behavioral and medical care together, improves clinical outcomes, improves patient experiences of care, and reduces health care costs (ie, the “Triple Aim” of health care).^7–9

Less well known is how to effectively deliver integrated care in routine primary care practice. At many practices beginning to integrate care, a new professional joins the team, a behavioral health clinician (BHC). BHCs are embedded in primary care teams to provide brief, targeted treatment to patients with mild-to-moderate behavioral health conditions. Although most primary care practices now routinely use electronic health records (EHRs),¹⁰ these EHRs are not tailored to the specific tasks or workflows of BHCs embedded in primary care clinics. Technological improvements are needed to enable BHCs to use the EHR for integrated care delivery.^11–13

In Turning EHRs into Assets for Mental Health and Uniting Practice (TEAM-UP), we collaborated with OCHIN, a not-for-profit health information network that provides health information technology (IT) solutions to a collaborative of 510 community health centers (CHCs) on a single instance of Epic EHR. In the United States, CHCs comprise the country’s health care safety-net by providing primary care and public health services to under or uninsured, low-income patients. A detailed description of OCHIN can be found else-where.^{14, 15} With input from practicing BHCs, we developed a suite of user-centered tools (the BH e-Suite) to address the information needs of BHCs practicing in integrated primary care practices. The BH e-Suite is incorporated into Epic and enables integrated care delivery by organizing the tools BHCs use into a single tab, with fast links to those tools. In addition, we added functionality that supports psychosocial assessment, information-gathering, goal-setting, documentation, monitoring, and tracking, by using a range of point-and-click functions and drop-down menus. It also includes templates for commonly used screeners (eg, Patient Health Questionnaire-9 [PHQ-9]; Generalized Anxiety Disorder-7 [GAD-7]) that auto-calculate and auto-populate progress notes and show changes in scores over time.

This study had the following aims:

1. Examine whether implementing the BH e-Suite changes process of care, intermediate clinical outcomes, and patient experiences postimplementation;

2. Assess acceptability of the BH e-Suite to BHCs; and

3. Identify clinic and practice member factors influencing the implementation of the BH e-Suite.

We hypothesize that BHCs will find the BH e-Suite acceptable to use. We also hypothesize that implementing the BH e-Suite will improve process measures (eg, rate of screening with PHQ-9) and patient experience of care. However, we do not expect improvements in patient depression symptoms (PHQ-9 scores) during the short time frame of this study.

Methods

Design

We used a convergent, iterative, mixed-methods design to accomplish the main aims of this study.

Clinic Sample

Six OCHIN CHCs (intervention clinics) were recruited to implement and use the BH e-Suite. A detailed description of the intervention clinics is provided in Table 1. Clinics were offered minimal support to implement the tool (ie, training manual, one 13-minute video) and no financial incentives. Although the functionalities we developed are organized into a single “suite” for BHCs, this functionality and the information documented were available to other users practice-wide. BHCs, however, were the primary users of the BH e-Suite in this study. Propensity scoring methods were used to match the intervention clinics to control clinics not using the behavioral health tool, based on 7 criteria: (1) geographic location (rural vs urban), (2) race (% White, log transformed), (3) ethnicity (% non-Hispanic White, log transformed), (4) insurance (% Medicaid), (5) percent of adults with serious mental health disease (log transformed), (6) number of active primary care providers, and (7) number of active patients. Thus, 6 control clinics were similar to intervention clinics on the above factors but did not use the BH e-Suite.

Table 1.

Description of Integration and Implementation Approach by Practice

Clinic 1 (partially implemented) is a primary care Federally Qualified Health Center (FQHC). This practice is partnering with a local nonprofit, community-based prevention, mental health, and addiction agency to colocate one licensed clinical psychologist in the practice 5 days a week. The psychologist provided brief, targeted behavioral health care to patients, with a particular focus on providing behavioral health support to patients with diabetes. Primary care physicians in the practice referred patients to the psychologist, with the front desk scheduling these appointments, and when needed, they engage the psychologist during a patient’s visit for an introduction or warm-handoff, or to seek this professional’s advice regarding the best treatment path for the patient (eg, see psychologist, see mental health provider, seek specialty treatment outside the practice). The psychologist had a private office in the practice that is located in close proximity to some of the primary care pods and farther from others, as this is a large practice. In addition, this practice also contracted with the same organization to colocate a mental health professional. This professional was an unlicensed social worker and was supervised by a licensed professional located at the mental health organization. This person provided traditional mental health services for patients with more serious and persistent mental illness, typically meeting with patients for 50-minute visits and for 12 weeks or more, as needed. This practice had written protocols in place for introducing the psychologist and mental health provider to patients and for making care transitions/referrals to these professionals. Not long after implementation of the BH e-Suite, the practice lost a critical team member who knew the tool well and was a consistent user. When this individual left, so did the institutional knowledge on tool usage. Although new behavioral health clinicians (BHCs) used the tool, use was superficial (eg, for documentation purposes only).

Clinic 2 (fully implemented) is a primary care FQHC that employs two licensed clinical social workers (LCSW) to provide brief behavioral health care to their patients. The LCSWs shared an office that is adjacent to the primary care team’s location in the practice. They did not have a private office to see patients, but generally did so in a medical examination room. Typically, the LCSWs were engaged by a primary care physician when the patient was in for a medical visit; the physician or medical assistant would go find the LCSW, who conducted a brief assessment and intervention and established a plan for following up with the visit. LCSWs would see patients, as needed, for brief visits. These visits are scheduled by the LCSW with the patient. The LCSWs played a central role in screening and brief intervention for substance use, as this had been a practice focus, and had developed protocols and a committee that reviews prescriptions for controlled substances. The practice had clear protocols for these workflows. Prior to expansion of the LCSW workforce in this practice, there was a colocated mental health professional, but this role has been phased out, and LCSWs assisted in transitioning patients to care at a local community mental health center (CMHC) for patients with more serious and persistent mental illness and substance use care need. Tool uptake was initially slow, due in part to internal billing negotiations. Once implemented, additional BHCs joined the staff and all used the tool extensively and fully.

Clinic 3 (did not implement) is a FQHC-CMHC hybrid. It is a county health department health center that includes primary care, mental health care, developmental disabilities services, environmental health, and other health-related community services. The units in this health center functioned autonomously, with the primary care practice located on the 1st floor and the mental health practice located on the 2nd floor. The CMHC served the county and took referrals from other organizations in the county, and primary care physicians in the building referred patients with mental/behavioral needs to the mental health unit in the same way other organizations in the community do. This health center does not currently employ BHCs to work closely with the primary care team. This practice had prior experience embedding BHCs with primary care, but just prior to implementation of the BH e-Suite these clinicians left the clinic and were not replaced. In addition, this health center does not have written protocols or other documentation in place to describe how medical, behavioral, and mental health care might be integrated for patients. The clinic experienced a great deal of turnover during the study.

Clinic 4 (partially implemented) is a primary care FQHC. This practice partnered with a local nonprofit, community-based prevention, mental health, and addiction agency to colocate one licensed clinical psychologist in the practice 5 days week to provide patients with brief, targeted behavioral health care, with a particular focus on providing behavioral health support to patients with diabetes. The psychologist had a private office and treated patients there, but s/he also had a workstation that is within close proximity to the primary care team. In addition to accepting referrals from the primary care physicians in the practice, physicians engaged the psychologist via warm-handoff so that s/he could provide brief services to patients in the medical examination rooms, as needed. The physicians also sought this professional’s advice regarding the best treatment path for patients (eg, in-clinic psychologist, in-clinic mental health provider, out-of-clinic specialty treatment), as needed. The practice also employed a colocated mental health provider who cares for patients with more serious and persistent mental illness who need longer visits (50-minutes) and a longer treatment course. This role was filled by a psychology practicum student from a local university where s/he received supervision for this work. This practice developed workflow for their integrated approach, specifying the roles for the front desk, medical assistants, physicians, psychologists, and mental health provider. At the time the BH e-Suite was launched, the practice experienced turnover among clinical staff that was quite disruptive. Operations did stabilize and the BH e-Suite was used by BHCs, but these professionals did not use all functionality.

Clinic 5 (fully implemented) is a primary care FQHC that employed one unlicensed social worker (MSW) to provide brief behavioral health care to their patients. The MSW shared an office with the practice’s Epic specialist. This professional did not see patients in their office but in medical examination rooms. Typically, the MSW was engaged by a primary care physician when the patient was in for a medical visit; the physician or medical assistant will go find the MSW, who will do a brief assessment and intervention and establish a plan for following up. MSW also helped when patients need to be referred to outpatient mental health and substance use care. MSW would see patients, as needed, for brief visits. These visits were scheduled by the MSW with the patient. In addition, the MSW played a central role in screening and brief intervention for substance use, as this has been a practice focus. The practice had clear protocols for these workflows. The BHC role was new to this practice. Tool implementation was limited in part due to access issues (the BHC had difficulty in accessing certain functionality). Documentation was the primary focus of tool usage.

Clinic 6 (did not implement) is a FQHC primary care branch of the county health department. This practice hired one LCSW to work in this practice to deliver integrated care. This was the practice’s first BHC, and this person was hired a few months prior to the start of the study. The LCSW provided both brief therapy to patients as well as more traditional mental health care to patients who need it. To accommodate both types of patients, the LCSW took referrals and was engaged in warm-handoffs by the primary care physicians when the LCSW was available. The LCSW shared an office with a diabetes counselor and this office was located next to the medical examination rooms and a couple rooms away from the primary care provider office. The LCSW saw patients in his/her office or in a medical examination room, depending on space availability. This practice did not have written protocols or workflows for care integration. This BHC was excited about the BH e-Suite but practice leadership was not.

Patient Sample

Eligible patients were frequent attenders (defined as those who were in 75th percentile for the number of outpatient visits), over 18 years old, diagnosed with at least one behavioral and one medical condition, and seen for a behavioral health visit 12 months before and after implementation of the BH e-Suite.

Measures and Data Collection

Process of care measures included the proportion of patients screened for PHQ-9 and GAD-7. Intermediate clinical outcome measures included PHQ-9 and GAD-7 scores. These measures were extracted for eligible patients from the 6 intervention and 6 control clinic EHRs 12 months before and after implementation.

Patient experience-of-care measures included previously validated patient-reported measures of care coordination (Picker Institute Scale, scores range 1 to 3 and 1 to 5),¹⁶ satisfaction with care (scores range 1 to 5), integration of care (Primary Care Assessment Tool, scores range 1 to 3),¹⁷ and care communication (Mercer Connecting, Assessing, Responding, and Empowering [CARE] Tool, scores range 1 to 5; see Appendix).¹⁸ These measures were collected by surveying a sample of eligible patients among the intervention clinics at baseline and again at 6 months postimplementation. Surveys were administered in the waiting room by either a research assistant or by a front desk staff person and placed in a secure lockbox after collection to ensure anonymity. An alert within the EHR informed clinic administration that the patient was eligible for a survey. At baseline, 337 patients responded to surveys and postimplementation, 302 patients responded to surveys.

Measures of BH e-Suite acceptability included domains from the Technology Acceptance Model (TAM),¹⁹ a 16-item survey completed by BHCs at baseline and postimplementation that measured the following: (1) Perceived usefulness, including the degree to which the EHR aligned with BHCs’ current workflow and tasks and also improved job performance, ability to manage patient care (eg, assess and diagnose patients), organize patient information, and coordinate care with integrated care team members; (2) perceived ease of use, including the perceived burden, time-efficiency, learnability, and interface qualities (ability to navigate and enter information); and (3) percent usability, including the frequency at which the BHCs found themselves abandoning the BH e-Suite, and the extent to which they used all the available BH e-Suite features.

Factors Influencing Implementation

Practice capacity for change was measured using the validated 23-item Adaptive Reserve questionnaire,^20–22 which was collected among intervention clinics via a practice member survey at baseline. Practice member surveys were distributed at monthly all-staff clinic meetings at baseline and had a greater than 75% response rate across practices. Qualitative methods (observations and inter-views) were used to assess factors influencing implementation of the BH e-Suite. Observations were conducted by a field team experienced in qualitative research, primary care delivery, informatics, and human factors research. This team conducted a 2-to-4-day site visit at the 6 intervention clinics over a 6-month period, November 2013-April 2014, before implementation of BH e-Suite. The length of visit depended on practice size, and each visit involved intensive observation of the integrated care team’s EHR use, including observing individual work areas, team work areas, and encounters with patients. In addition, we shadowed primary care clinicians, BHCs (ie, licensed clinical psychologists and social workers), and other key members of the clinical care team, and sat in on their visits with patients when permitted. Among all 6 participating clinics, we accumulated a total of 184 single-spaced pages of field notes from approximately 150 person-hours of observing all aspects of clinical care.

Semistructured interviews were conducted at clinic site visits with 2 to 4 practice members representing different roles in the practice (eg, BHC, primary care providers, medical assistants) to understand their approach to integrating care, the workflows and tasks involved in integration, and how they used the EHR for documentation and information sharing. By observing and interviewing different roles in the practices, we gained a robust understanding of how care for behavioral health conditions was delivered at baseline and the environment in which the BH e-Suite would be implemented. We also conducted 6-month postimplementation interviews with BHCs to ascertain the level of implementation, experience with the tool, and changes in clinical practice connected to tool use. Interviews followed a semistructured guide²³ that assessed integrated care workflows and tasks, experience with the current EHR, practice structure, and organizational culture. We conducted 29 interviews (11 BHCs, 4 medical assistants, 7 mental health clinicians [MHCs], and 7 primary care clinicians), each lasting between 45 minutes and 1 hour. In intervention practices, MHCs provide care for patients with more severe and persistent mental health issues, while BHCs provide care to patients with mild to moderate mental health problems over a shorter period of time.

Data Analysis

Descriptive statistics (tabulations, percentages, means, and standard deviations) were used to describe EHR-derived process of care and intermediate clinical outcome measures, stratified by case and control clinics at baseline. We used SAS PROC MIXED²⁴ to model 2-level, mixed effects (random intercept) regression to account for the clustered sampling design (patients clustered within clinics). Linear regression models were designed to evaluate the interactive relationship between pre- and postimplementation and intervention/control in regard to patient PHQ-9 and GAD-7 scores (dependent variable) across intervention and control clinics. Practice member and patient surveys were descriptively analyzed by examining response frequency distributions. Among intervention clinics, t tests were computed to examine the mean change after implementation in patient-reported outcome measures (eg, care coordination, care communication with primary care physician, integration of care, and satisfaction with care). When specific survey items were not answered, respondents’ items were excluded from analyses. Missing data ranged from 1% to 12% among patient-reported constructs.

Our multidisciplinary team used a 4-step approach to analyze qualitative data.²⁵ First, we collectively read field notes and interview data collected from each practice to develop an understanding of each practice, as a case, with particular attention to factors influencing implementation of the tool, BHC workflows, tasks, and tool ease-of-use. Through this process, we created a codebook to label text. We used these codes in group analyses until we reached stability; code definitions were standardized and analysts were trained to code data reliably. Individuals completed the data analysis, meeting as a group to compare work and come to consensus when analytic differences emerged.

Second, we analyzed findings across clinics to identify common themes. We grouped emerging findings into categories of themes by using an immersion-crystallization approach.²⁶ Third, we connected our findings to the existing literature.²⁵ We conducted a second in-depth comparative analysis by using the Consolidated Framework for Implementation Research (CFIR)²⁷ constructs to understand differences in implementation across intervention clinics. After considering all 39 CFIR constructs, 7 relevant CFIR constructs that mapped to 3 domains (inner setting, process characteristics and individual characteristics) were identified and an in-depth analysis was conducted to identify how these factors influenced implementation of the BH e-Suite.

Qualitative and quantitative analyses were first conducted independently to ensure unbiased interpretation. After completing analyses, qualitative and quantitative findings were integrated at the practice level by using data-triangulation techniques.²⁵ The Institutional Review Board at Oregon Health & Science University approved this study protocol (No. 9366).

Results

The distribution of patient characteristics was similar between intervention and control clinics (Table 2).

Table 2.

Patient Sample Description of Intervention and Control Clinics at Baseline

Description	Intervention, N = 4,377	Control, N = 3,628
	N (%)	N (%)
Age, mean, SD	49.1 (14.2)	49.1 (14.6)
Sex, female	1,308 (30)	1,113 (30)
Race/Ethnicity
White, Non-Hispanic	2,964 (69)	2,302 (65)
Black, Non-Hispanic	36 (1)	101 (3)
Hispanic	1,216 (28)	990 (28)
Other	86 (2)	170 (5)
Language, English	3,280 (75)	2,792 (77)
Household income, $
<14,999	2,254 (56)	2,219 (63)
15,000 to 29,999	1,339 (33)	835 (24)
30,000 to 44,999	319 (8)	277 (8)
45,000 to 59,999	77 (2)	113 (3)
60,000	36 (1)	71 (2)
Number of comorbidities
0	455 (10)	356 (10)
1	1,008 (23)	791 (22)
2	1,107 (25)	910 (25
3+	1,807 (41)	1,571 (43)
Depression diagnosis	1,951 (45)	1,678 (46)
Diabetes diagnosis	1,175 (27)	1,122 (31)

SD, standard deviation.

Intervention clinics implemented the BH e-Suite to varying degrees: two practices used all the features and functions of the BH e-Suite EHR tool (“fully implemented”); 2 practices implemented some, but not all, of the features of the tool (“partially implemented”); and 2 practices did not use the tool at all (“did not implement”).

Clinical Care and Patient Perception of Care

Overall, there was a significant increase between intervention and control clinics in process of care, specifically the proportion of patients screened with PHQ-9 (Table 3). There was no change in intermediate clinical outcome measures (ie, PHQ-9 and GAD-7 scores). When examined by degree of implementation, significant but small differences were observed in outcome measures between intervention and control clinics. Table 3 also displays the change in patient-reported experience of care postimplementation. Patient perceptions of integration-of-care improved overall and among clinics that fully implemented the tool. Patient perceptions of communication with primary care providers significantly improved only among clinics that partially implemented the tool.

Table 3.

Clinical Care and Patient Experience of Care Outcomes across Degree of Implementation

	Overall	Fully Implemented	Partially Implemented	Did Not Implement
Process of Care Outcomes*
PHQ-9 screens, OR	1.06‡	1.10‡	1.01	1.11‡
GAD-7 screens, OR	1.01	1.01	0.85§	1.03§
Intermediate Clinical Outcomes*
PhQ-9 scores, β	0.16	−0.64	0.23	1.98‡
GAD-7 scores, β	1.38	2.59	4.42	−4.55 ‖
Patient Experiences of Care†, mean (SD)
Satisfaction with care	0.06 (0.07)	0.09 (0.11)	0.08 (0.12)	−0.03 (0.08)
Care communication with PCP	0.15 (0.08)	−0.02 (0.13)	0.39§ (0.13)	0.04 (0.13)
Care communication with BHC	0.06 (0.13)	0.10 (0.24)	0.12 (0.23)	−0.10 (0.20)
Integration of care	0.20§ (0.07)	0.44‡ (0.13)	0.09 (0.13)	0.10 (0.13)
Care coordination	0.06 (0.04)	0.06 (0.06)	0.11 (0.06)	−0.02 (0.07)

BHC, behavioral health clinician; GAD-7, Generalized Anxiety Disorder-7; PCP, primary care provider; PHQ-9, Patient Health Questionnaire-9; OR, odds ratio; SD, standard deviation.

^*Pre/post change among intervention clinics compared to control clinics.

^†Pre/post change among intervention clinics.

^‡P value<.001.

^§P value<.01.

^‖P value<.05.

Acceptability of BH e-Suite

Clinicians working in clinics that fully or partially implemented the BH e-Suite reported that the tool was acceptable and easy to use. They reported the tool added 1 to 2 minutes to the initial visit but saved a significant amount of time during follow-up visits by automatically populating the history of the presenting illness and patient instructions at subsequent visits. As a BHC from clinic 5 who fully implemented the tool reported:

“[Tool adoption was] actually really easy because we were a small, very small core group of people when we started, and as we brought on new hires the system adopted it. Our IT people adopted it, our Epic people adopted it and it is not like you have a choice. This is just what we’re doing. We’re using this. As we’ve hired new folks, this has just been what they have been introduced to and it is been great.”

BHC reported finding the tool beneficial, too: “I would say the biggest gain for me has been in follow-up visits so I am not having to copy paste from previous visits. It is really streamlined the process for follow-up in terms of mental status examination and history of the presenting illness. Those parts have helped immensely” (Clinic 2).

Clinics that fully or partially implemented the tool found it easy to use (mean Technology Acceptance Model score, 3.75 and 3.5, respectively). Among clinics that partially implemented the BH e-Suite, clinicians reported that they would use the tool more often if they were given more training in its use. For instance, a newly hired BHC at Clinic 1 who only used the tool for documenting encounters, explained that she was just learning to use Epic and picking up the BH e-Suite as well as learning to practice in a new setting was difficult: “I just stopped using it [all] because… it just seemed like a lot… it just takes time and practice, but some days my schedule is just really full, and I just have to go quickly.” All clinics that implemented the tool to some degree reported an increase in perceived usability from preintervention (mean, 1.87) to postintervention (mean, 3.78) (data not shown).

Factors Influencing Implementation of the BH e-Suite

Clinics that fully implemented the BH e-Suite had higher adaptive reserve scores, indicating greater capacity for change (Table 4). They also had leadership buy-in and support, more resources dedicated to implementation, processes and workflows developed in advance of implementation, staff of all levels engaged in implementation, and BHCs who were formally trained in use of the BH e-Suite. BHCs noted that the tool simplified EHR use, and they welcomed it as a way to help address their health IT needs. The lead BHCs championed tool implementation and supported staff use of the tool by creating step-by-step guides, allocating time and space for training and fostering collaboration among BHCs and Epic specialists. Clinics that partially implemented had less adaptive capacity and needed more time, training, and support to use the tool. They also experienced important disruptive events (turnover in BHC staff), which was a barrier to implementation. Clinics that did not implement the BH e-Suite had the lowest adaptive capacity and also lacked leadership buy-in and support for use of the BH e-Suite. Importantly, although these clinics agreed to participate in the study, saying they would have an embedded BHC on staff, they were unable to fulfill this promise and, instead, continued to refer patients to a colocated MHC who had different information and documentation needs; MHCs at these clinics used a different EHR tool to meet their unique documentation needs.

Table 4.

Implementation Degree across Practices and Consolidated Framework for Implementation Research Domains

Domain	Definition	Fully Implemented	Partially Implemented	Did Not Implement
Inner Setting
Baseline Adaptive Reserve (mean, SD)	Practice capacity for change	0.63 (0.08)	0.56 (0.07)	0.55 (0.08)
Implementation climate	Commitment to Implement change	• BHCs interested in improving BH documentation in EHR	• BHC interested in improving BH documentation in EHR	• BHCs satisfied with current system and resistant to new tool
		• Upper management advocated for using a new tool tailored to BHC needs	•Interest in tool from upper management	• Little buy-in from upper management
		• Epic specialist available to assist with implementation.	• Epic site specialist did not assist with implementation	• Integration model did not facilitate use of the tool, as these clinics focused on long mental health appointments, rather than brief interventions with BHCs
Readiness for implementation	Organizational capacity to implement change	• Step-by-step guides produced by BHC leadership on how to use the tool	• Unstable BHC workforce/turnover; no written BHC workflows	• No time allocated for training and educating BHCs about tool although Epic site specialist available
		• Stable BHC workforce, written BHC workflows in place
Process Characteristics
Planning	Tasks and methods conducted to prepare for implementation	• Had introductory implementation meeting before rolling out the tool	• No implementation plan aside from one introductory implementation meeting before rollingout the tool	• Had introductory implementation meeting before rolling out tool
		• Lead BHC acted as opinion leader and champion of tool implementation	• No follow-up by clinics to implement	• MH manager felt the tool was not useful to them and declined implementing the tool
Engaging	Involvement of staff in the change process	• Familiarized other clinical staff, including front desk staff and PCPs, with the new Tool	• BHC supervisor left this role; loss of champion derailed implementation	• No clinic champion or formally appointed internal implementation leader
Reflecting and evaluating	Team debriefing about process and experience	• Dedicated agenda time at BHC monthly meeting to discuss the tool and provide feedback to tool developer	• Feedback given to tool developers, but no follow through by practice	• No reflection or evaluation because tool was not implemented
Individual Characteristics
Knowledge/belief about intervention	Individuals’ attitudes toward and value placed on intervention	• Believed tool enhanced EHR use (simplified it)	• Newly hired BHCs reported receiving limited (1 hour), and inadequate training in the tool	• Did not find the tool to be useful
		• Consensus view: tool was a step of progress	• BHCs did not know how to use tool
Self-efficacy	Belief in one’s ability to execute implementation tasks	• BHCs felt they received sufficient training to use tool

BH, behavioral health; BHC, behavioral health clinician; MH, mental health; PCP, primary care provider; EHR, electronical health record; SD, standard deviation.

Discussion

Policy reforms have been implemented across the United States in support of the integration of behavioral health services into primary care clinics,²⁸ and there are an increasing number of initiatives aimed at integrating care across diverse settings.^29–31 This momentum to change practice and policy is crucial because integrated care is specifically designed to identify and address patient medical and behavioral health care needs together and to improve quality, patient experience, and reduce costs.^4,6,32–35

Systems that have integrated, or are planning to integrate, behavioral health and medical care must consider an investment in health IT. Documentation requirements for BHCs are different from other members on the integrated team. Therefore, clinics must think about workflows, tasks, and health IT functionality to effectively use the EHR to deliver high quality integrated primary care. Our study showed that a user-centered suite of EHR tools tailored for BHCs was feasible to use and acceptable to embedded BHCs practicing in integrated CHCs. Importantly, use of the BH e-Suite increased the perceived productivity of BHCs and improved process of care. At the outset of this study, we were sensitive to the unintended effects of having BHCs use the EHR during patient visits, as this might negatively impact the therapeutic relationship, patient experience of care, and intermediate clinical outcomes. With regard to patient experiences of care, we found the opposite to be true; use of the BH e-Suite improved patient perceptions of communication with their clinicians and BHCs, as well as their perceptions of receiving integrated care. Through in-depth observation, interviews, and the intentional engagement of users in the design process, we successfully developed a tool that was acceptable to patients and BHCs and avoided a common error of overcomplicating documentation for short BHC visits. We believe that this careful design work may have contributed to the BH e-Suite’s ability to create care efficiencies and patient connection, without compromising the therapeutic relationship.

Our study has important lessons for clinics planning to use health IT tools for integration. In order for clinics to invest the time to implement the BH e-Suite, these tools must fit their needs. Clinics that had a “colocated” approach to care (patients internally referred to specialty MHCs for longer-term care instead of to an integrated BHC) did not find the BH e-Suite fitting. In fact, these clinics had a competing set of tools, specifically designed by and for specialty MHCs; not surprisingly, these clinics preferred the other tools. This suggests that motivation for implementation and perceived usefulness of a tool are aligned and must be carefully considered during health IT implementation.

Two study clinics with high motivation and adaptive capacity implemented these tools on their own and subsequently demonstrated improvement in patient experience. Two clinics that had midlevel adaptive capacity were only able to partially implement the BH e-Suite, despite recognizing its utility. These latter 2 clinics likely represent the norm among primary care clinics in the United States, which suggests the need for evidence-based implementation support strategies, such as practice facilitation or expert consultation, to assist with tool implementation. This finding has implications for clinics and researchers considering a health IT implementation; regardless of the improvements using a new health IT tool may bring, successful implementation may require external assistance.

Limitations

Our results should be considered in light of several limitations. First, the sample size of clinics was small, and generalizability of the study findings is limited. However, even with the small number of clinics, there was considerable variability in the integration and implementation approach, and we found that use of the BH e-Suite significantly enhanced patient experiences of care. Second, although we recognized the social and technical aspects of the change process when introducing a new tool within primary care,³⁶ this study did not have the resources to provide implementation support to clinics, and we did not test how to best implement the BH e-Suite. Our qualitative findings provide formative insights into the factors that influence implementation, but more work is needed to study the effectiveness as well as the wider dissemination and implementation of this tool among CHCs. Third, some may consider it a limitation that we developed this tool in Epic, as this EHR system may not be affordable for all clinics. However, the partnership between researchers, OCHIN Epic developers, and CHCs made development of this tool possible. OCHIN serves over 500 CHCs across the nation, and they all now have access to the BH e-Suite. We have published details about the functionality of these tool elsewhere, allowing for this suite of tools and features to be developed by others²³ and streamlining behavioral health integration.

Integrated primary care clinics need EHR tools that support the delivery of whole-person care. EHRs are seldom designed with behavioral health in mind; to date, few EHR-based tools have been developed and tested to support the work of BHCs and integrated teams. The BH e-Suite may fill this need, as it was acceptable to BHCs, feasible to implement by motivated clinics with some training, and has the promise to improve outcomes. More research is needed to assess the effectiveness of this tool in improving process and/or patient outcomes that are not just limited to depression symptoms but also to manage chronic diseases (such as type 2 diabetes) where integrated care has been shown to have an impact. Furthermore, future research should investigate how to disseminate and implement this tool widely among integrated primary care clinics.

APPENDIX:

Patient-Reported Measures

Measure	Survey Questions	Response Items
Care coordination using the Picker Institute Scale	1) Did you know who was in charge of your care for each of your health problems?	Questions 1 to 4: [1]= No, [2]= Yes, sometimes, 3]= Yes, most of the time; or
	2) How often were the doctors, nurses and other health care providers who cared for you familiar with your most recent medical history?	Questions 5 to 8: [1]=Never, [2]=Rarely, [3]=Occasionally, [4]= Often, [5]= Very often
	3) How often were your providers aware of changes in your treatment that other providers recommended?
	4) Do you think your providers had all the information they needed, such as test results, to make decisions about your treatment?
	5) How often did you know who to ask when you had questions about your health problems?
	6) How often were you given confusing or differing information about your health or treatments?
	7) How often did you know what the next step in your care would be?
	8) How well did your health care providers work together?
Satisfaction with care	How satisfied have you been with…	[1]= Very dissatisfied, [2]=Dissatisfied, [3]= Neutral, [4]= Satisfied, [5]= Very satisfied
	1) the overall quality of care at your primary care clinic;
	2) being able to get an appointment in your clinic as soon as you think you need it;
	3) being able to get a telephone call or email answered from your primary care clinic as soon as you think you need it.
Integration of care using the Primary Care Assessment Tool	1) Did your primary care doctor suggest you go to the behavioral health provider?	[1]= Probably Not, [2]= Probably, and [3]= Definitely
	2) Did your doctor discuss with you different places you could have gone to get help with that problem?
	3) Did your doctor or someone working with your doctor help you make the appointment for that visit?
	4) Did your doctor provide any information for the behavioral health provider about the reason for the visit?
	5) Did the doctor know you made this visit to the behavioral health provider?
	6) Did your doctor know what the results of this visit were?
	7) After going to the behavioral health provider, did your doctor talk with you about what happened at the visit?
	8) Does your doctor seemed interested in the care you get from the behavioral health provider?
Care communication using the Connecting, Assessing, Responding, and Empowering (CARE Tool)	How was the doctor at…	[1]= Poor, [2]= Fair, [3]= Good, [4]= Very good, [5]= Excellent
	1) making you feel at ease
	2) letting you tell your story
	3) really listening
	4) being interested in you as a whole person
	5) fully understanding your concerns
	6) showing care and compassion
	7) being positive
	8) explaining things clearly
	9) helping you take control
	10) making a plan of action with you