Evaluation of the Role of Training in the Implementation of a Depression Screening and Treatment Protocol in 2 Academic Outpatient Internal Medicine Clinics Utilizing the Electronic Medical Record

Abstract

Systematic approaches to depression identification and management are effective though not consistently implemented. The research team implemented a depression protocol, preceded by training, in 2 faculty-resident practices. Medical assistants used the Patient Health Questionnaire (PHQ)-2 for initial screening; providers performed the PHQ-9. These were documented in the electronic medical record. Logistic regression was performed to assess the association of provider type, clinic site, and training attendance with documentation of PHQ-9 after positive PHQ-2s, and with repeat PHQ-9s after positive PHQ-9s. In logistic regression analysis, training attendance was positively associated with documentation of PHQ-9 after a positive PHQ-2 (odds ratio [OR] = 2.4 [confidence interval (CI) = 1.3–4.3]) and repeated documentation of a PHQ-9 after a positive PHQ-9 (OR = 2.5 [CI = 1.1–5.3]). This study describes the successful implementation of a stepped-care approach to depression care. The positive association of training with compliance with protocol procedures indicates the importance of training in the implementation of practice change.

Mental illness is pervasive and costly in the United States, with one quarter of the population suffering from some form of mental illness¹ and up to one third of primary care patients suffering from depression alone.² Mental illness significantly affects outcomes in patients with other chronic medical illnesses.^3,4 For example, patients with both coronary artery disease and depression have increased mortality rates when compared with patients with no mental illness.^5,6 Patients with both diabetes and depression have increased morbidity and mortality when compared with patients with diabetes alone.^4,7

Mental illness is commonly treated in primary care settings, with 43% to 60% of treatment in primary care and 17% to 20% in specialty mental health care.^8,9 However, depression remains underdiagnosed in primary care. In one study, only 41% of those diagnosed with mental illness (including anxiety, mood, impulse control, and substance abuse disorders) in face-to-face interviews had received any treatment in the prior year.⁸ Furthermore, many patients struggle to access care with mental health specialists, largely because of payment systems that separate care for mental and physical illnesses.^10,11 Indeed, mental health specialists are cited as the most difficult specialty to access.¹² Therefore, improving the diagnosis and treatment of mental illness in primary care is critical to reducing poor outcomes among patients with psychiatric illness.

Evidence-based strategies for the screening and treatment of depression have been developed that could be used in the primary care setting.^13,14 Specifically, the Collaborative Care Model, which utilizes a stepped-care approach to the treatment of depression, has proven effective in managing depression in primary care.^13–15 The stepped-care approach to depression treatment involves initiating less-intensive treatment for depression initially. Treatment intensity is based on close monitoring of response to treatment. In stepped care, this close monitoring of response to treatment and adjustments to treatment regimen are based on treatment algorithms.

However, the best approach for implementing these evidence-based strategies into practice in primary care clinics remains unclear. Implementation of stepped care and collaborative care approaches to the treatment of depression has been highly variable in different clinical sites.^16,17 Often, the protocols are adopted incompletely, and providers do not use standardized tools to assess depression.¹⁸ Adoption of protocols varies by both primary care provider and clinic site.¹⁹

From 2012 to 2013, the research team implemented a depression care protocol, which included universal screening and a stepped-care approach to treatment, in the internal medicine clinics at the University of Colorado Hospital (UCH). With implementation of the protocol, providers were offered training in the rationale for the protocol and in the care processes associated with the protocol. This study evaluates the effectiveness of the training with both a retrospective analysis of documentation of protocol processes in the electronic medical record (EMR) and a survey of providers. The research team sought to evaluate the training session and to identify factors that contribute to effective protocol use.

Methods

Design

The research team conducted a retrospective data analysis and a provider survey to assess the effectiveness of a 1-hour training session in preparing providers to perform the care processes involved in the depression screening and treatment protocol. According to the policy activities that constitute research at the Colorado Multiple Institutional Review Board (COMIRB) the retrospective data analysis reported here met criteria for operational improvement activities exempt from ethics review. The survey component of the evaluation was COMIRB approved.

Study Population and Study Setting

In February of 2012, a universal depression screening and treatment protocol (depression protocol) was implemented at the University of Colorado Anschutz Internal Medicine Clinic (Anschutz) and subsequently, in October of 2012, at the Lowry Internal Medicine Clinic (Lowry). Both clinics are academic, combined resident and faculty practices. The depression protocol adapted the collaborative care model,¹⁵ which utilizes a stepped-care treatment algorithm based on Patient Health Questionnaire (PHQ)-9 scores.²⁰ The algorithm was originally developed by educators at the University of North Carolina for use in their resident continuity clinics.²¹ This stepped-care algorithm was built into the EMR.

Depression Protocol Procedures

Figure 1 illustrates the clinic flow for the depression protocol. The primary protocol procedures are as follows: (1) medical assistant (MA) administers the initial depression screening, the PHQ-2, to all patients at new-patient or annual visits; (2) MA gives a paper PHQ-9 to patients with a positive PHQ-2; (3) provider documents the PHQ-9 into the EMR, so that the score can be tracked; (4) provider treats the patient based on the PHQ-9 score and his or her own clinical judgment; (5) patients with positive PHQ-9 scores are reassessed in person in 1 to 3 months depending on score and treatment strategy; (6) the PHQ-9 is administered at the follow-up visit to assess treatment effectiveness and determine the next step in treatment; (7) when a psycho-tropic medication is started or changed, the provider initiates a 1- to 2-week follow-up call by a member of the provider’s team (MA or nurse).

Figure 1.

University of Colorado Hospital depression protocol algorithm.

Abbreviations: MA, medical assistant; PHQ, Patient Health Questionnaire; PCP, primary care provider.

Local Context and Approval Process

Initially, the research team approached the physician practice director of the Anschutz clinic with the idea of implementing the protocol in all clinic patients. Although the practice director of the clinic was supportive of the protocol, implementation required approval from hospital administration. The Anschutz clinic is run by a UCH practice manager who is responsible for regulatory compliance of procedures in the clinic. Ultimately, any changes to clinical practice for ambulatory staff require approval from the Director of Nursing and Clinical Standards for Ambulatory Services for UCH. Because the MAs played a vital role in the protocol, the research team worked closely with both the clinic practice manager and the Director of Nursing and Clinical Standards for Ambulatory Services on all MA procedures in the Protocol. The MA “call-back” (protocol item number 7) was the most challenging procedure to reconcile with the MAs’ scope of practice. Triage calls requiring clinical decision making were considered beyond the scope of work for an MA. This issue with scope of work was resolved by developing a script for MAs to use for the patient “call-back.” If the patient deviated from concrete yes/no answers to the questions in the script, the patient was transferred to the team nurse for appropriate triage. After receiving approval for the protocol from clinic and hospital leadership, it was built into the EMR.

Building the Protocol Into the EMR

UCH and all ambulatory clinics use the EpicCare EMR (Epic Systems Corporation, Verona, WI). The technical build for the protocol was developed in collaboration between the Epic Ambulatory Physician Champion and the UCH Epic analysts. The build involved getting larger Clinical Content Build Out Committee approval to avoid negative downstream impact to the integrated system. The following concerns were addressed: (1) The primary concern involved ensuring that providers were alerted to the MA documentation in the EMR. This concern was mitigated using a best practice alert (BPA). When the MA documented a positive PHQ-2, the system alerted the provider of record with a BPA, which prompted the provider to fill out the PHQ-9. (2) A second concern was “real estate” in the main navigation screen of the EMR. The Epic analyst reported receiving frequent requests for pilot projects for which prime screen space is utilized. As a clinically important condition for National Committee for Quality Assurance accreditation, depression screening for primary care was felt to take priority over other projects. For the MA screening, the PHQ-2 was given the prime space of the Visit Navigator for improved ease of use. (3) A third concern was provider efficiency with regard to use of the protocol. To improve ease of use, the BPA took providers directly to the PHQ-9 flow sheet, the location in the chart for documentation of the PHQ-9. Processes were developed so providers could easily import the PHQ-9 results into their note. Finally, a template decision aid was developed, which could be easily recalled and imported into the plan portion of the note.

Training

The provider training took place on-site the week prior to implementation of the depression protocol at each clinic site. The training consisted of a 1-hour participatory lecture with 2 primary components: (1) an educational component focused on the rationale for universal screening for depression, the stepped-care approach to depression treatment, and the collaborative care model, and (2) a procedural component focused on the details of the depression protocol algorithm and documentation in the EMR. At the Anschutz clinic the training was mandatory for residents but optional for faculty. At the Lowry clinic, the training was targeted to faculty, who were responsible for training residents. The MAs were trained separately.

Measures

The PHQ-9 is a 9-question version of the PHQ that has been validated for the detection of depression,²² changes in severity over time,²³ and monitoring of treatment outcomes.²⁴ The PHQ-2 is a 2-question version of the questionnaire, with slightly higher sensitivity and lower specificity, that is used as an initial screening tool for depression.²⁵ Both the PHQ-9 and the PHQ-2 have been validated for use in the primary care setting.²⁶ PHQ-2 and PHQ-9 data were extracted from the medical record from the time of implementation of the protocol at each site through March 2013.

The 22-question survey instrument was developed by the research team to assess provider comfort with depression protocol procedures, provider perception of the impact of training on competency in managing depression, provider opinion of the value of the training, training attendance, and provider demographics. Questions were answered using a 4-point Likert scale. The survey was administered online to all providers at Anschutz and Lowry in January 2013. An e-mail with a link to the survey was sent to providers 3 times over 3 weeks. Consent was implied with completion of the survey, and responses were anonymous.

Statistical Analysis

Survey Analysis

Two scales were created: (1) comfort with protocol procedures (Procedures) and (2) improvement in depression management competency (Competency) by summing the Likert scale responses for the 5 questions related to each of these outcomes. The research team performed t tests (2-tailed) to test for differences in these 2 outcomes by provider type (resident/faculty) and training attendance and calculated a Cohen’s d effect size for the effect of training on the outcomes.

Retrospective Data Analysis

The research team examined 2 primary outcomes in the retrospective analysis of provider documentation of protocol processes in the EMR: (1) documentation of PHQ-9 at visits for patients with positive PHQ-2s and (2) PHQ-9 monitoring at subsequent visits for patients with positive PHQ-9s. The team performed descriptive analyses of these outcomes for Anschutz faculty, Anschutz residents, Lowry faculty, and Lowry residents. Then, a multiple logistic regression analysis was performed for the 2 outcomes. PHQ-2 screening was nested within providers. Therefore, a generalized estimating equation²⁷ method was used with a binomial distribution to control for correlation between documentation within the same provider. The research team examined differences by 3 binomial independent variables: provider type (resident/faculty), clinic site (Anschutz/Lowry), and training attendance (yes/no). Odds ratios and their 95% confidence intervals were generated. All statistical analyses were performed using SAS release 9.2 (SAS Institute Inc., Cary, NC). A 2-sided P value <.05 was considered statistically significant.

Results

Table 1 shows provider distribution by provider type and clinic site, percentage of providers who attended the training, and summary outcomes for each group of providers. Because faculty members were responsible for training residents at Lowry, very few Lowry residents attended. Also, because the protocol was in place for 6 months at Lowry compared to 13 months at Anschutz, the number of charts with documented PHQ-9s and repeated PHQ-9s are lower at Lowry. In the multiple logistic regression, only training attendance significantly affected documentation of a PHQ-9 at visits with positive PHQ-2s and documentation of a subsequent visit PHQ-9 for patients with positive PHQ-9s (Table 2). Neither clinic site nor provider type significantly affected the odds of either outcome. Notably, in the unadjusted analyses, Lowry residents have lower documentation rates for both initial PHQ-9s and subsequent visit PHQ-9s. There was a 62% response rate to the survey. In all, 58 providers answered the survey—22 residents and 31 faculty; 5 did not identify provider type. Of survey participants, 40 attended the training, 16 did not attend the training, and 2 did not respond. Table 3 shows association of the 2 scales with provider type and training attendance. Faculty reported higher scores on both the Procedures and Competence scales. Although training attendance was not significantly associated with scores on either scale, the effect size of training attendance on the Procedures outcome met Cohen’s convention for a moderate (d = 0.4) effect size. The effect size on the Competence indicated no effect (d = 0.04).

Table 1.

Provider Characteristics, Training Attendance, and Documentation of PHQ-9 Results.

Characteristic (n)	In-Person Training (%)	PHQ-9 Documented Patient Charts, n (%)	PHQ-9 Repeated Patient Charts, n (%)
Anschutz
Faculty (26)	73	211 (66.6)	60 (31.4)
Residents (35)	91	72 (70.6)	17 (27.4)
Lowry
Faculty (18)	83	112 (60.9)	34 (37.0)
Residents (21)	10	20 (48.8)	3 (20.0)

Abbreviation: PHQ, Patient Health Questionnaire.

Table 2.

Multiple Logistic Regression for PHQ-9 Documented for Positive PHQ-2 and Repeated PHQ-9 for Positive PHQ-9.

Characteristic	Adjusted ORa	Confidence Interval
Documented PHQ-9 for positive PHQ-2
Provider (attending)	0.8	0.5–1.5
Clinic (Lowry)	0.6	0.3–1.2
Training (attended)	2.4	1.3–4.3
Repeated PHQ-9 for positive PHQ-9
Provider (attending)	1.2	0.6–2.3
Clinic (Lowry)	1.2	0.7–2.1
Training (attended)	2.5	1.1–5.3

Abbreviations: PHQ, Patient Health Questionnaire; OR, odds ratio; GEE, generalized estimating equation.

^aRepeated documentation by providers has been accounted for using the GEE model. All the variables (provider type, clinic type, and training attendance) were added simultaneously as covariates.

Table 3.

Survey Association Between Mean Scores for Procedures and Competence by Provider Type and Training Attendance.

Scale	Mean Score (SD)	t Test (df)	P Value
Procedures
Residents	13.5 (1.8)	3.4 (48)	.002
Faculty	15.5 (2.2)
Attended	15.0 (2.3)	1.3 (52)	.2
Not attended	14.1 (2.0)
Competence
Residents	12.4 (2.3)	112 (60.9)	.34 (37.0)
Faculty	12.8 (3.0)
Attended	12.6 (3.1)	0.1 (53)	.9
Not attended	12.4 (2.5)

Abbreviation: SD, standard deviation.

Discussion

The research team conducted a mixed-method evaluation of a provider training used in the implementation of a primary care depression screening and treatment protocol. In retrospective data analysis, attendance at the training session was associated with 2.4 times increased odds of documenting a PHQ-9 when the patient had a positive PHQ-2 and 2.5 times increased odds of repeating the PHQ-9 after a positive PHQ-9. In the provider survey, provider attendance was associated with a non–statistically significant trend toward increased comfort with the protocol procedures (Procedures). However, the Cohen’s d effect size was moderate, indicating that the increase in comfort was meaningful on a practical level. Furthermore, providers reported no improvement in their competency in managing depression (Competence) in association with the training.

This study adds to the building research on implementation of stepped-care approaches to depression screening and treatment. In addition to its focus on the role of training, this study adds a nuanced view of the use of EMRs in the implementation of these programs. Two studies of the implementation and sustainability of depression care practice improvements across a primary care practice network found that most practices were able to implement and sustain the use of the PHQ-9 for diagnosis and monitoring of depression. However, tracking of the PHQ-9 was particularly difficult to implement and sustain.²⁸ Another large-scale quality improvement study in the Netherlands assessed the facilitators and barriers to the implementation of a stepped-care approach to depression. In qualitative interviews, participants reported poor information systems as a primary barrier to implementation.²⁹ Likewise, in an assessment of primary care clinics participating in the Institute for Healthcare Improvement’s Breakthrough Series for Depression, difficulty with technology or technical systems was noted as a primary barrier to practice change.³⁰ This study describes the successful implementation of a tracking system within the EMR, although its usefulness depends on provider compliance with documentation of initial and follow-up PHQ-9s within the EMR.

The findings that the training was associated with increased odds of compliance with the depression protocol procedures and a moderate effect size for comfort with protocol procedures points to the importance of training in successful implementation of the depression protocol. The training was designed to transmit practical knowledge of the processes of care involved in the protocol. The success of this approach is consistent with a 2010 qualitative study by Olson of the methods that health care teams used to acquire knowledge.³¹ The study analyzed 3 successful efforts to improve clinical practice in the area of antibiotic resistance in an effort to better understand effective methods of practice change. The authors use a differentiation made by philosopher Gilbert Ryle³² to differentiate between “knowing how” and “knowing that.” They refer to understanding how to actually implement practice change rather than purely understanding that a particular practice change should occur. This analysis showed that practical training in how to perform the protocol played a significant role in the “knowing how” aspect of practice change.

The training was associated with statistically significant increased odds of performance of protocol procedures, although provider assessment of the training was more equivocal. These findings highlight the need for objective assessment measures for educational interventions. They are consistent with prior studies on evaluating practice change interventions.³³ The use of process of care measures is particularly important when evaluating educational interventions aimed to implement evidence-based practice.

Furthermore, this study points to the need for multi-faceted approaches to the implementation of evidence-based practice. These survey responses indicate that training attendance did not correlate with the reported increase in competency in treating depression, consistent with evidence that didactic education about clinical guidelines alone does not effectively lead to practice change.^34,35 In this study, the training session included didactic background, rationale for the protocol, and practical training in protocol procedures.

The depression protocol evaluated in this study utilized EMR technology to provide clinical decision support. Although clinical decision support is a valuable resource available with an EMR,^36,37 there is recognition of the overuse of this support. When clinical decision support is overused, providers often ignore or circumnavigate the cues developed to guide workflows.³⁸ Critical evaluation of which pieces of clinical decision support will add value to patient care while also allowing the provider to efficiently navigate patient encounters is necessary.³⁹ In this initial ambulatory clinical decision-support project at UCH, successful balance with the depression protocol was achieved by choosing a brief, recognized screening tool and incorporating this tool into clinical workflows that were efficient for staff and providers while providing patients with evidence-based care.

This study has multiple limitations. First, the study was a quality improvement project in one institution. Therefore, it is not generalizable to other health care settings. Second, because the participants were not randomized, this study suffers from selection bias. As a quality improvement initiative rather than a clinic study, all participants were offered the training. Providers who chose to attend the training were more likely to comply with documentation in the depression protocol. Similarly, participants in the survey also were more likely to be compliant with the protocol. Thus, the results may overestimate the effect of training. Third, because the process measures evaluated in this study are not rare events, the odds ratios cannot be interpreted as measures of relative risk because they would overestimate relative risk. Fourth, because the survey was anonymous, survey data cannot be linked with performance of process measures. Although the research team collected process measure data on all providers in the clinics, only 62% of providers responded to the survey. Thus, definitive conclusions cannot be drawn regarding the comparison of the 2 different evaluation techniques used in the study. Fifth, this study examines process measures associated with implementation of the depression protocol and, therefore, did not examine the clinical impact of the protocol. Follow-up studies are planned to assess this clinical impact. Despite the limitations, this study offers insights into the “real-world” implementation and evaluation of a systematic approach to the screening and treatment of depression and, thus, has relevance for practices or groups that wish to pursue implementation of similar protocols.

Conclusions

In conclusion, this study highlights the importance of practical knowledge and multifaceted approaches to the implementation of evidence-based care into practice. The Collaborative Care Model,^13–15 as well as other integrated care models, improves management of psychiatric illness within the primary care setting. These team-based models of care have gained broad acceptance as the preferred approach to caring for patients with chronic disease, including those with concomitant psychiatric illness. The movement toward team-based care has been adopted on a national policy level through incentives in the Affordable Care Act for primary care practices to adopt the Patient-Centered Medical Home model.⁴⁰ Health care administrators and researchers need to consider multifaceted approaches that incorporate practical training as they implement these new models of care in the primary care setting.