Effectiveness of Quality Improvement Coaching on Process Outcomes in Health Care Settings: A Systematic Review

Abstract

Background

A culture of improvement is an important feature of high-quality health care systems. However, health care teams often need support to translate quality improvement (QI) activities into practice. One method of support is consultation from a QI coach. The literature suggests that coaching interventions have a positive impact on clinical outcomes. However, the impact of coaching on specific process outcomes, like adoption of clinical care activities, is unknown. Identifying the process outcomes for which QI coaching is most effective could provide specific guidance on when to employ this strategy.

Methods

We searched multiple databases from inception through July 2021. Studies that addressed the effects of QI coaching on process of care outcomes were included. Two reviewers independently extracted study characteristics and assessed risk of bias. Certainty of evidence was assessed using GRADE.

Results

We identified 1983 articles, of which 23 cluster-randomized trials met eligibility criteria. All but two took place in a primary care setting. Overall, interventions typically targeted multiple simultaneous processes of care activities. We found that coaching probably has a beneficial effect on composite process of care outcomes (n = 9) and ordering of labs and vital signs (n = 6), and possibly has a beneficial effect on changes in organizational process of care (n = 5), appropriate documentation (n = 5), and delivery of appropriate counseling (n = 3). We did not perform meta-analyses because of conceptual heterogeneity around intervention design and outcomes; rather, we synthesized the data narratively. Due to imprecision, inconsistency, and high risk of bias of the included studies, we judged the certainty of these results as low or very low.

Conclusion

QI coaching interventions may affect certain processes of care activities such as ordering of labs and vital signs. Future research that advances the identification of when QI coaching is most beneficial for health care teams seeking to implement improvement processes in pursuit of high-quality care will support efficient use of QI resources.

Protocol Registration.

This study was registered and followed a published protocol (PROSPERO: CRD42020165069).

Supplementary Information

The online version contains supplementary material available at 10.1007/s11606-021-07217-2.

INTRODUCTION

High-quality health care is a priority for both patients and clinicians. In 2001, the Institute of Medicine (now the National Academy of Medicine) outlined a strategy to improve the quality of health care in the USA anchored on six aims: safety, effectiveness, patient-centeredness, timeliness, efficiency, and equity [1]. The pursuit of these aims is the process of quality improvement (QI), which can be defined as “a framework we use to systematically improve the ways care is delivered to patients” [2]. QI is one aspect of the science of improvement, or “an applied science that emphasizes innovation, rapid-cycle testing … and spread in order to generate learning about what changes, in which context, produce results” [3]. Improvement science offers rigorous approaches to the attainment of high-quality care through clinic-level process refinement and the uptake of evidence-based practices [4, 5]. One approach to promote the pursuit of high-quality health care is the provision of longitudinal, expert support to help individuals and health care teams identify and implement areas of practice change [6–8]. QI coaching [9] is a commonly used strategy for the provision of longitudinal, expert support to clinical teams seeking to engage in QI processes.

A quality improvement (QI) coach supports an interdisciplinary health care delivery team in their pursuit of achieving sustained change and the improvement of clinical processes. Quality improvement coaches assist with goal setting and attainment, connect teams to system-level resources for change, and improve efficiency and team dynamics around improvement processes utilizing a variety of strategies [9]. The coach role can be agnostic to the clinical content area and does not require topical expertise. QI coaching is similar to other approaches that encourage the systematic adoption of high-quality, evidence-based practices such as facilitation. While there are multiple definitions, facilitation can generally be thought of as a “process of working with groups to support participatory ways of doing things.” [10] There are multiple scholarly fields that promote a coach-like role to support the optimal improvement of clinical care delivery (e.g., QI, implementation science, systems redesign), each with its own terms to describe the coaching-like processes (Supplementary Information) [11]. The effects of the coaching intervention can be measured at multiple levels including the level of care delivery such as provider behaviors or practice activities and policies (process outcomes) or at the level of patient care (clinical outcomes) [9].

To address the current gap in the literature, we investigated the effect of QI coaching on practice- or clinical team-level behaviors and process outcomes and found that QI coaching is a complex intervention that has the potential to improve the capacity for improvement activities at the team and practice level. Specifically, this review was conducted to support a type of QI coaching (i.e., transformational coaching) used in the Department of Veterans Affairs (VA) which has not been specifically described in the literature. Thus, we used a broad search strategy to identify interventions that shared the essential components that must be maintained to ensure fidelity to the VA’s QI coaching intervention. Components considered essential for the QI coach for this review include the following: (1) the coach is content-agnostic (not required to be an expert in the specific clinical topic or intervention that is the focus of the QI project). (2) The coach is external to the target of coaching (i.e., not a member of the health care delivery team being coached). (3) The coach aims to catalyze and/or build capacity for sustained change and improvement through activities such as assisting with goal setting, goal attainment, connection to system-level resources for change, and/or improving efficiency and team dynamics around change/improvement processes.

METHODS

Study Design

This work is part of a larger Veterans Health Administration (VHA)–funded report (www.hsrd.research.va.gov/publications/esp), which addresses the effects of QI coaching on practices, providers, patients, and processes. We established an a priori published protocol (PROSPERO: CRD42020165069) and followed PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidance [12].

Data Sources and Searches

In collaboration with a reference librarian, we searched MEDLINE® (via Ovid®), Embase (via Elsevier), and CINAHL Complete (via EBSCO) from inception (since the database began indexing journal content) through July 2021 (Supplementary Information). As there is no MeSH term existing for QI coaching and there are multiple terms for similar interventions, we identified the most commonly used terms and pseudonyms for a person (or persons) who potentially shared the essential components based on our operationalized definition of QI coaching listed above (e.g., practice facilitator, outreach visitor, QI coach). Specifically, we incorporated related terms from the fields of QI, improvement science, and implementation science, which themselves employ overlapping terms and methods pertaining to the support of clinical teams and practices in the uptake and improvement of evidence-based clinical processes. We also screened references from high-quality systematic reviews and studies identified by stakeholders during topic development.

Study Selection

Our inclusion and exclusion criteria are listed in Supplementary Information. Relevant terms identified after execution of the literature search were searched independently, and any references meeting our inclusion criteria were imported into two electronic databases (for referencing, EndNote®, Clarivate Analytics, Philadelphia, PA; for data abstraction, DistillerSR; Evidence Partners Inc., Manotick, ON, Canada). Citations classified for inclusion by at least one investigator at title and abstract were reviewed at full text by two investigators according to a priori established eligibility criteria. All articles meeting eligibility criteria at this level were included for data abstraction.

Data Extraction and Quality Assessment

One investigator abstracted data into a customized DistillerSR database; a second investigator reviewed data for accuracy. Data elements included descriptors to assess applicability, quality elements, intervention details, and all measured outcomes. Multiple reports from a single study were treated as a single data point, prioritizing results based on the most complete and appropriately analyzed data. Key features relevant to applicability included the match between the sample and target populations (e.g., age, large health care system). Two investigators independently assessed study quality using the Cochrane Effective Practice and Organisation of Care (EPOC) Risk of Bias (ROB) Tool [13]. We assigned summary ROB scores (low, unclear, or high) to individual studies (Supplementary Information).

Data Synthesis and Analysis

We collected all outcomes reported by studies meeting eligibility criteria and organized them by the level at which they produced potential changes. Specifically, we grouped outcomes by the level at which a process occurred: practice (e.g., processes requiring collaboration and simultaneous participation of multiple providers or clinical teams in a practice setting) or provider level (e.g., processes conducted by individual clinicians at the point of care such as ordering labs for a given condition). Other measures targeted clinical outcomes at the patient level (e.g., improved individual health outcomes). We described key study characteristics of the included studies using summary tables. Because complexity of targeted behavior change predicts intervention success, we grouped outcomes by the complexity of desired clinical practice or provider behavior promoted by the QI coach [14]. Specifically, those behaviors that required multiple steps or those requiring the agreement or collaboration of multiple individuals were considered more complex (e.g., adherence to multi-step guideline recommendations for asthma-related care) and those that could be completed individually less complex (e.g., ordering a lab). Then, we grouped outcomes by clinical care delivery similarity (e.g., ordering a lab, improving documentation). Within these groupings, we organized findings by study-level ROB.

Across included studies, we identified intervention activities employed by coaches to support interdisciplinary teams and matched them to the Expert Recommendations for Implementing Change (ERIC) strategies [15]. ERIC was chosen because it is widely cited and incorporates relevant QI ideas. Given the conceptual heterogeneity in process of care outcomes assessed, the measure used to assess a given outcome, and the selection and dosing of coaching strategies employed, we described the specified outcomes narratively rather than calculating a summary effect.

We organized the adoption of targeted process of care activities according to the complexity of the specific behavior required by the relevant QI activity; specifically, we used the following eight categories: composite outcomes of multiple clinical processes of care, organizational processes of care, documentation, medication prescription, counseling, provider exams and procedures, lab tests, and vital signs. Heterogeneity, primarily of outcome measurement, precluded pooled assessment of the effect of coaching across these categories.

To support synthesis across the included studies, we employed a vote-counting method based on direction of effect [16, 17]. Following this approach, we categorized the intervention effect as harmful or beneficial based on the direction of effect without consideration for magnitude or statistical significance [16, 17]. We calculated the overall proportion of beneficial findings and obtained the exact 95% confidence interval (CI) for the true proportion of beneficial findings. We employed an exact binomial probability test to test the hypothesis that the intervention was truly ineffective, and provided the resulting p-value (i.e., the probability of observed or more extreme proportion if, in fact, the proportion of beneficial studies is truly 0.5). Exact CIs and p-value were calculated using “binom.test” function in the R statistical package. The certainty of evidence was assessed using Grading of Recommendations Assessment, Development and Evaluation (GRADE) [18]. Certainty of evidence assessment conveys the level of confidence in effect estimates supporting a given conclusion [18].

Role of Funding Source

The US Department of Veterans Affairs was not involved in the design, conduct, or analysis interpretation.

RESULTS

From 1983 screened citations, we reviewed 116 full-text articles and identified 23 unique studies (all cluster-randomized trials [CRTs]) (Fig. 1) [19–41]. All studies were conducted across North America, Europe, and Australia. All but two trials were conducted within the primary care setting and one study was conducted in the VA [25]. Among the included studies, two were in children and twenty-one were in adults. The targets of the coaching included diabetes (n = 3), mental health (n = 1), and preventive services (n = 15). Details of each of the 23 included studies are in Table 1.

Fig. 1.

Literature flow diagram. Search results from MEDLINE (1,231), Embase (503), CINAHL (241), and identified from relevant articles (8) were combined

Table 1.

Study Characteristics

Study Country # enrolled # arms Funding source Companion Paper	Eligibility Clinical context	Intervention duration Name of coaching role Intervention description	# team members Team composition: N (%) VA-based?	Primary outcomes Outcomes type	Risk of bias
Carroll, 2018 [20] USA 42 practices 2 arms NIDDK	Non-hospital based, ambulatory primary care practices with at least 1 physician and a minimum of 2000 patients seen in the past year Chronic kidney disease	Duration: 36 months Coaching role: practice facilitator Practice facilitation (PF) arm received site coordination, identified a local physician champion (had an academic mentor); audit and feedback; creation of QI team; and education via academic detailing. PF delivered virtually, to assist with goal setting, help QI teams strategize/test/ implement change, facilitate meetings and foster continuous QI culture, liaison for data and performance feedback and share best practices and linking intervention practices	# team members: NR Team composition: NR VA: no	Annualized loss of eGFR (protocol paper gives “patient-level score based on % of goals achieved” as primary outcome) Outcome type: process of care activities	Objective: high Self-reported: NA
Chinman, 2017 [25] USA 69 teams 2 arms VA HSR&D QUERI	The 3 HUD-VASH teams were selected based on their willingness to participate and similarity to each other in terms of Veteran composition. Within the 3 HUD-VASH teams, all case managers were invited to participate HUD-VASH	Duration: 12–23 months Coaching role: technical assistance A 10-step process to build capacity for implementation of evidence-based practices was used, called Getting to Outcomes (GTO). This involved a manual, 6-h training, and ongoing technical assistance, which consisted of bi-weekly phone calls to help sub-teams implement GTO practices. Meetings included goal setting, tailoring of performance targets, additional training to address gaps, reviewing performance data and troubleshooting	# team members: NR Team composition: case manager: 100% VA: yes	NR (2 outcomes noted in Abstract; Adoption and Reach of MISSION-Vet) Outcome type: process of care activities	Objective: unclear Self-reported: NA
Dickinson, 2014 [23] USA 40 practices 3 arms NIDDK; NIMH	Small to midsize community health centers and independent mixed-payer primary care practices in Colorado Diabetes	Duration: 6, 12, or 18 months, depending on arm Coaching role: practice facilitator Practice facilitator met with practices over 18 months an average of 9.7 times. Practices in the CQI group received practice facilitation based on the Model for Improvement. The CQI facilitators provided a structure and process for quality improvement using CQI tools that particularly focused on sequential PDSA cycles guided by quality measurement data	# team members: NR Team composition: NR VA: no	Diabetes process of care Outcome type: process of care activities	Objective: high risk Self-reported: unclear risk
Dickinson, 2019 [19] USA 36 practices 3 arms NIDDK	Family medicine or general internal medicine practices with at least 80 patients with type 2 diabetes, all clinicians agreeing to participate Diabetes	Duration: NR Coaching role: practice facilitator Short-term PF by a “trained” practice facilitator; 4 meetings to assist with Connection to Health (CTH) adoption plan, followed by monthly PF calls to review data with practice on CTH use; optional booster session; control arm included self-management support education (SMS ED)	# team members: NR Team composition: • CTH + PF intervention clinicians: mean 6.1 (SD 4.3) • CTH intervention clinicians: mean 7.3 (SD .41) • SMS ED control clinicians: • Mean 7.4 (SD 3.4) VA: no	HbA1c, systolic and diastolic blood pressure, body mass index Outcome type: process of care activities	Objective: unclear Self-reported: NA
Due, 2014 [31] Denmark 186 practices 2 arms Danish Research Foundation; Health Insurance foundation; Research Foundation for Primary Care	Consecutively included Danish general practices that signed up for facilitation visits and completed a baseline questionnaire Chronic obstructive pulmonary disease; diabetes	Duration: 9 months Coaching role: facilitator Facilitators helped define goals and suitable means for achieving, support a process of change, demonstrate instruments, standardized visit reports	# team members: NR Team composition: NR VA: no	Change in # of annual chronic disease checkups per 100 patients affiliated with the practice Outcome type: process of care activities	Objective: unclear Self-reported: high
Engels, 2006 [32] Netherlands 49 practices 2 arms Netherlands Organisation for Health Research and Development	Primary care practices that were on a list for a practice assessment using the national Dutch Visitation Instrument for Practice management were contacted Primary care	Duration: 1 year Coaching role: outreach visitor After initial assessment, the practices in the intervention group undertook a CQI process with the help of an “outreach visitor.” Outreach visitors were all experienced practice assistants who had also participated in a 3-day training program to learn how to organize the QI meetings, guide the practice team through the steps of the CQI model and deal with group processes in general	# team members: NR Team composition: NR VA: no	NR Outcome types: process of care activities; goal attainment	Objective: NA Self-reported: low
Goodwin, 2001 [26] USA 79 practices 2 arms NCI; Family Practice Research Center from American Academy of Family Physicians Companion paper: Stange, 2003 [45]	Members of the Ohio Academy of Family Physicians practicing in northeast Ohio were invited to participate General primary prevention	Duration: 12 months Coaching role: nurse facilitator Nurse facilitator worked with individual practices during 1-day practice assessment to inform strategy tailoring, complete a practice environment checklist then during 1.5 h meetings using peer data comparison, supported strategy choices, planed generation for change, identified a practice leader, provided a manual and conducted several follow-up visits	# team members: NR Team composition: NR VA: no	Rate of patients being up to date on USPSTF recommended prevention services by medical record review = # preventive services up to date by age/sex eligible/total eligible Outcome type: process of care activities	Objective: unclear Self-reported: NA
Harris, 2015 [37] Australia 32 practices 2 arms National Health and Medical Research Council; Australian National Heart Foundation	Primary care practices with use of EHR, which could be audited as well as employment of a practice nurse Chronic vascular disease prevention	Duration: 6 months Coaching role: practice facilitator Facilitation included a training workshop, 3 practice visits with GP, practice nurse, and possibly office manager; 3 follow-up phone calls; clinical audit provided to practices; goal setting, local resource provision, problem solving	# team members: NR Team composition: • MD: 83 • PN: 40 VA: no	Change in proportion of patients aged 40–69 years with smoking status, alcohol intake, body mass index (BMI), waist circumference, blood pressure recorded and for those aged 45–69 years with lipids, fasting blood glucose and cardiovascular risk in the medical record Outcome types: process of care activities; self-efficacy	Objective: high Self-reported: low
Hogg, 2008 [35] Canada 54 practices 2 arms CIHR	Primary care practices (solo or group) in Eastern Ontario with 6 or fewer physicians General primary prevention	Duration: 11.5 months Coaching role: outreach facilitator One of 2 nurses would make monthly visits to a practice. Practice facilitation included feedback from an initial audit, discussion of the use of tools such as prevention flow sheets, chart flags, sticker reminders, electronic reminders, patient care records, etc., and developing a plan for improvement with physicians. Periodic follow-up visits (every 3–6 weeks) involved monitoring progress on plan and making any adjustments	# team members: NR Team composition: • Intervention physicians: mean 3.5 • Control physicians: mean 2.6 VA: no	Composite index of preventive performance, defined as the number of appropriate preventive maneuvers done minus the number of inappropriate maneuvers done, divided by the total number of eligible preventive maneuvers Outcome type: process of care activities	Objective: low Self-reported: NA
Gold, 2019 [38] USA 29 practices 3 arms National Heart, Lung, and Blood Institute of the National Institutes of Health	Community health clinics that provided primary care to adults and were members of OCHIN (non-profit health information technology organization) Diabetes	Duration: 30 months Coaching role: practice facilitator Practice facilitator (PF) met with sites ≥ 2 times in 1st year and ≥ 3 times over 2nd and 3rd years. PF role was to address barriers related to implementing intervention bundle. Strategies used include: meeting with clinicians, understanding clinic work flow, training on intervention bundle, monthly emails, webinars, and technical assistance. (Arm 3)	# team members: NR Team composition: staff, providers, others not otherwise specified VA: no	Proportion of patients with diabetes who were indicated to have ACE/ARBs and statin prescriptions Outcome type: process of care activities	Objective: Self-reported:
Lemelin, 2001 [36] Canada 46 practices 2 arms Ontario Ministry of Health	Community primary care practices with a payment system based primarily on capitation. HSOs located in remote areas were excluded because of cost, and the HSO in which investigators worked were also excluded General primary prevention	Duration: 18 months Coaching role: facilitator Nursing prevention facilitators met with up to 8 practices in-person and via email. They used 7 intervention strategies: audit and ongoing feedback, consensus building, opinion leaders and networking, academic detailing and education materials, reminder systems, patient-mediated activities, patient education materials	# team members: NR Team composition: • Intervention: MD: mean 2.91; RN: mean 1.16 • Comparator MD: mean 2.70; RN: mean 1.64 VA: no	Overall index of preventive performance (calculated by subtracting the proportion of patients receiving inappropriate preventative maneuvers from the proportion of patients who received the 8 recommended preventive maneuvers) Outcome type: process of care activities	Objective: unclear Self-reported: NA
Liddy, 2015 [34] Canada 84 practices 3 stepped-wedge clusters Ontario Ministry of Health; Pfizer Canada (indirectly); CIHR; Ottawa Hospital Companion paper: Deri Armstrong, 2016 [46]	Eligible practices provided general primary care and were in operation for at least 2 years Cardiovascular disease	Duration: 2 years Coaching role: practice facilitator Practice outreach facilitation (audit and feedback, consensus building, and regular meetings to focus on goal setting, planning and implementation via PDSA cycles, interactive collaborative meetings [a series of half-day]) with chronic care model (decision support, community resources, self-management support ad delivery system redesign)	# team members: 182 providers Family physicians: 100% VA: no	“Quality of care composite score” = patient-level score intended to reflect adherence to recommended guidelines for cardiovascular disease Outcome type: processes of care	Objective: unclear Self-reported: NA
Lobo, 2002 [33] Netherlands 124 practices 2 arms Netherlands Heart Foundation	Primary care practices with the presence of a computer system, ancillary staff present, and no major changes planned during the course of the project Cardiovascular preventive care	Duration: 21 months Coaching role: outreach visitor Coach was an “outreach visitor,” met with teams for 15 visits (first 8 visits were dedicated to organization of preventive care, last 7 visits were dedicated to clinical decision making), coaching interactions followed theoretical model of change intervention allowed practice members to draw up and prioritize their own list of gaps and planned changes. The intervention focused on six aspects of practice organization: availability of instruments and materials, involvement of the practice assistant in preventive tasks, presence of separate preventive clinics, teamwork within the practice, record-keeping and follow-up routines	# team members NR Team composition: • Intervention: GP: 57 (% of practices with 1 GP); practice assistants 27 (% with only one practice assistant) • Comparator: GP: 55 (% of practices with 1 GP); practice assistants: 32 (% with only one practice assistant) VA: no	Difference between the deficiency scores in each aspect of organizing preventive care before and after the intervention; this enabled consideration of the ratio of baseline score and postintervention score Outcome type: process of care activities	Objective: low Self-reported: low
Margolis, 2004 [28] USA 44 practices 2 arms AHRQ; US Bureau of Maternal and Child Health; NC Division of Medical Assistance; NC AHEC; RWJF	Primary care practices near UNC Chapel Hill and Charlotte AHEC; sufficient newborns enrolled, not an academic affiliate or publicly funded center, annual Medicaid billing > $50,000 General primary prevention	Duration: 2 years Coaching role: project staff Practices form teams and review chart abstractions, academic detailing, selection of goals and strategies; project staff (coach in this case) provide tools and help with customizing; help teams run PDSA cycles, spread of positive outcomes to other staff	# team members NR Team composition: • Intervention: clinicians: mean 5.6 (range: 1 to 12); staff: mean 17.0 (range: 1 to 56) • Control: clinicians: mean 4.4 (range: 1, 12); staff: mean 14.1 (range: 3 to 31) VA: no	Change over time of proportion of children in each practice who received all four services (immunizations, screening for anemia, screening for lead, screening for TB) Outcome type: process of care activities	Objective: low Self-reported: NA
Meropol, 2014 [22] USA 30 practices 2 arms Medicaid Technical Assistance and Policy Program; Center for Child Health and Policy at Rainbow Babies	Primary care practices were identified through 2 PBRNs; practices had at least 15% of patients 10 years of age or younger and at least 20% of pediatric patients covered by Medicaid insurance, and agreed to provide at least 2 of 3 targeted services and participate in educational meetings and chart reviews Well-child visits at age 24–30 months	Duration: 6 months Coaching role: practice facilitator Practice coaching and rapid-cyclefeedback/change to improve delivery of recommended pediatric preventive services in 3 domains. During weekly visits, the facilitator reviewed a small convenience sample of charts from the previous week and documented whether targeted services were performed; plotted each week’s results on “run charts”; and “huddled” briefly with available practice members to review run charts, assess what had worked, brainstorm solutions for further improvement, and select new tools/procedures to implement during the coming week	# team members NR Team composition: • Intervention clinicians per practice: mean 3.5 (SD 2.34); non-clinician staff: mean 4.74 (SD 3.97) • Control clinicians per practice: mean 3.64 (SD 2.27); non-clinician staff: mean 3.14 (SD 1.67) VA: no	NR Outcome type: process of care activities	Objective: low Self-reported: NA
Mold, 2014 [21] USA 45 practices 4 arms NHLBI	Primary care practices were members of 1 of 3 practice-based research networks in Oklahoma or New York Asthma	Duration: 6 months Coaching role: practice facilitator Assistance from practice facilitator during visits either half- day weekly or a full day every other week to assist practice with meeting goals	# team members: NR; # practices with mid-level practitioners: 27 (63%) VA: no	NR explicitly; appears to be adherence to 6 guideline recommendations Outcome type: process of care activities	Objective: unclear Self-reported: NA
Ornstein, 2004 [29] USA 20 practices 2 arms AHRQ; DHHS	Primary care practices that are community-based family or general internal medicine practices with the same electronic medical record Cardiovascular preventive care	Duration: 2 years Coaching role: NA (coaching by a team of people) Multimethod quality improvement intervention that included 6–7 site visits, audit and feedback as well as 2 network meetings. The site visits were led by one of the coauthors and included engaging clinicians and staff in the project, general education and group discussion. Teams also identified specific clinical indicators that they wished to work on	# team members: NR Team composition: • MD: 45 • Mid-level providers: 17 VA: no	Primary practice-level outcome was the percentage of performance targets achieved; primary patient-level outcome was the percentage of patients for whom the recommended process measures had occurred or the recommended outcome measure had been achieved Outcome type: process of care activities; goal attainment	Objective: high risk Self-reported: NA
Parchman, 2013 [24] USA 40 practices 2 arms NIDDK; Audie L. Murphy Veterans Hospital, Veterans Health Administration Companion paper: Noel, 2014 [47]	Small, autonomous primary care practices in South Texas Exclusion criteria: • Multi-specialty practices • Practice owned by a large vertically integrated health care system • Practices with five or more physicians Diabetes	Duration: 12 months Coaching role: practice facilitator Coach was a practice facilitator who coached practices to implement changes of delivery of care to improve diabetes care, primary care teams consisting of providers and non-providers. Practice facilitators held a minimum of 6 1-h team meetings within each practice over a 12-month period of time. PF efforts, baseline chart audit, and feedback, as well as interactive consensus building and goal setting, were incorporated into the intervention	# team members: NR Team composition: • MD or DO: 15.4% • NP: 3.6% • PA: 2.9% • RN/LVN: 5.4% • Medical Assistant: 31.8% • Receptionist: 12.1% • Office manager: 7.5% • Other: 21.4% VA: no	Certified case manager score Outcome type: process of care activities	Objective: NA Self-reported: low
Rask, 2001 [27] USA 4 practices 2 arms Aetna Inc. through the Quality Care Research Fund	Community-based clinics that are part of a larger primary care center located in Atlanta, Georgia. Clinics were selected for the study because of their high patient volume and relatively large populations of diabetes patients Diabetes	Duration: 1 year Coaching role: nurse facilitator Nurse facilitator oriented the clinics to the performance improvement activity, conducted in-services with new office staff, attended monthly operations meetings, and visited the clinics weekly to answer questions about the study. The nurse facilitator also distributed materials and a summary of the ADA clinical practice recommendations. The facilitator also created and distributed a patient reminder form and conducted monthly medical record reviews then provided site-specific feedback to the physicians and medical directors. Control arm included feedback only	# team members: NR Team composition: • Internal medicine physicians: 22 • Family practice physicians: 6 VA: no	NR Outcome type: process of care activities	Objective: unclear Self-reported: NA
Ruud, 2021 [39] Norway 39 practices 2 arms Southeastern Regional Health Authority in Norway	Norwegian clinical sites providing routine public mental health services to adults or adolescents with psychosis Mental health clinics	Duration: 18 months Coaching role: implementation facilitator The facilitators were mostly mental health nurses. They met with clinics every other week for 6 months and then once per month for 12 months. The facilitator’s role was to help the clinics use QI procedures to implement one of the four evidence-based practices	# team members: NR Team composition: clinic managers and clinicians VA: no	Evidence-based practice fidelity to each of four EBPs (physical health care, antipsychotic medication management, family psychoeducation, illness management and recovery) Outcome type: process of care activities	Objective: Self-reported: NA
Shelley, 2020 [40] USA 257 practices 4 waves AHRQ (EvidenceNOW initiative)	Small independent practices (≤ 10 full time equivalent clinicians) in NYC that were using one of two eligible electronic health record (EHR) systems with no plans to change EHR in next 18 months; also no immediate plans to begin other cardiovascular-related QI projects Cardiovascular disease	Duration: 1 year Coaching role: practice facilitator Practice facilitator did 13 on-site visits over 12 months which consisted of: how to optimize EHR to monitor data, provide intervention updates and patient education materials, and redesign workflows to facilitate integration of evidence-based practices	# team members: NR Team composition: NR VA: no	Number of at-risk patients who reached clinical goals for each of the four ABCS guidelines: aspirin use, BP control, cholesterol management, and smoking cessation Proportion of smokers who received counseling Composite measure for patients with a history of ischemic vascular disease who met treatment targets for aspirin, BP, and cholesterol measures Outcomes type: process of care activities	Objective: Self-reported:
van Bruggen, 2008 [30] Netherlands 1640 patients 2 arms AGIS Insurance Center	Patients with diagnoses of type 2 diabetes in 1 of 30 primary care clinics agreed to participate from the broader population of 70 clinics solicited. Exclusions included the inability to complete a questionnaire, severe mental illness, unwillingness to attend the practice regularly, a limited life expectancy, or current treatment in the outpatient clinic of the local hospital Diabetes	Duration: 1 year Coaching role: nurse facilitator Two nurse facilitators interviewed practice staff, analyzed barriers, discussed means to overcome barriers and handed out abstracts of guidelines for diabetes care. These trained facilitators visited all intervention practices 2 times per month for approximately 3 h. They trained the GPs, practice assistants and nurses in the guidelines, encouraged the introduction of structured diabetes care, emphasized the need for 3-monthly control and gave assistance in managing people with type 2 diabetes. Performance feedback was given at 6 months	# team members: NR Team composition: NR VA: no	Percentage of people with poor glycemic control at baseline that achieved an HbA1c of < 8% Outcome type: process of care activities	Objective: unclear Self-reported: unclear
Zgierska, 2020 [41] USA 26 practices 3 waves Pfizer Independent Grants for Learning and Change	Academic adult primary care clinics in Wisconsin that were not involved with any other opioid QI initiatives Primary care	Duration: 4–6 months Coaching role: practice facilitator Practice facilitator delivered six, 1-h facilitation sessions over 4–6 months that consisted of helping each clinic optimize their workflows to support clinician adherence to guideline recommendations with measureable QI outcomes selected by clinic team preference	# team members: NR Team composition: Volunteer clinic staff (prescribers, nurses, and others) VA: no	Clinic-level percentage of target patients with opioid treatment agreement in place Outcomes type: Process of care activities	Objective: Self-reported: NA

Abbreviations: ADA American Diabetes Association; AHEC Area Health Education Center; AHRQ Agency for Healthcare Research and Quality; CIHR Canadian Institutes of Health Research; CQI continuous quality improvement; CTH Connection to Health; DHHS Department of Health and Human Services; DO Doctor of Osteopathy; eGFR estimated glomerular filtration rate; EHR electronic health record; GP general practitioner; GTO Getting to Outcomes; HSO Health Standards Organization; HSR&D Health Services Research and Development; HUD Housing and Urban Development; LVN licensed vocational nurse; NA not applicable; NCI National Cancer Institute; NHLBI National Heart, Lung, and Blood Institute; NIDDK National Institute of Diabetes and Digestive and Kidney Diseases; NIMH National Institute of Mental Health; NR not reported; NP nurse practitioner; PA physician assistant; PBRNpractice-based research network; PDSA Plan, Do, Study, Act; PF practice facilitation; QUERI Quality Enhancement Research Initiative; QI quality improvement; RN registered nurse; RWJF Robert Wood Johnson Foundation; SD standard deviation; SMS EDself-management support education; USPSTF US Preventive Services Task Force; VASH Veterans Affairs Supportive Housing

Terms used for the QI coach–like role included practice facilitator, practice outreach facilitation, practice coach, nurse facilitator, nurse prevention facilitator, and outreach visitor. Interventions varied in duration from 4 to 48 months. Coaches employed varied combinations of 13 distinct implementation strategies. Studies reported a median of 5.56 implementation strategies (range 3 to 9) delivered by the coach-like role. The four most used coach-delivered implementation strategies were to develop a formal implementation plan (19/23 studies), audit and provide feedback (18/23), develop/distribute educational materials (17/23), and conduct educational outreach visits (17/23). The least used strategies were organizing clinician team meetings (3/23) and developing stakeholder interrelationships (2/23). Table 2 details the transformational coaching activities used in the included studies.

Table 2.

Transformational Coaching Activities

Coach-delivered implementation strategy	Operationalized definitiona	ERIC strategy category	Examples from included studies
Baseline local need assessment (8 studies)	Collect and analyze data before the start of coaching intervention to assess local needs related to QI project	Use evaluative and iterative strategies	Performed a multimethod practice assessment, including assessment of practice communication, change and work culture, and level of implementation of the Chronic Care Model.[23]
Develop a formal implementation plan (19 studies)	Develop a formal implementation plan that includes clear goals and strategies	Use evaluative and iterative strategies	Group discussion to reflect on findings and identify priorities for improvement.[24]
Educational outreach visits (17 studies)	Coach meets with providers in their practice settings to educate about the clinical innovation	Train and educate stakeholders	Training: study staff conducted an in-person, 6-h training with each subteam on how to use Get To Outcomes plan, implement, evaluate.[25]
Develop/distribute educational materials (18 studies)	Provide manuals, toolkits, and other supporting materials to teams	Train and educate stakeholders	Coaches introduced the concept of the Chronic Care Model and presented an evidence-based “toolkit” comprised of 5 activities to improve diabetes outcomes.[24]
Teach and support implementation/QI tools (9 studies)	Introduce and train teams on QI techniques and tools appropriate to the innovation or QI project being implemented	Use evaluative and iterative strategies	Education on “fostering a continuous QI culture.”[20] Used the Chronic Care Model: the QI approach.[34]
Revise professional roles (10 studies)	Shift and revise roles among professionals who provide care, and redesign job characteristics	Support clinicians	A “lead physician” for liaising with the facilitator was identified in the practice.[35
]Technical assistance (9 studies)	Provide technical assistance (e.g., data support) focused on QI project needs	Provide interactive assistance	MISSION-Vet service data was collected with a Computerized Patient Record System note template that was developed for each team. Data from the notes were extracted to create feedback reports.[25]
Develop resource sharing (4 studies)	Develop partnerships with organizations that have resources needed to implement the innovation	Support clinicians	Enhanced community linkage; “community resources.”[34]
Create a learning collaborativeb (5 studies)	Facilitate the formation of groups of providers or provider organizations and foster a collaborative learning environment to improve implementation of the clinical innovation	Train and educate stakeholders	The learning sessions provided an opportunity for practice members to share successes and challenges with other practices [23]
Organize clinician team meetings (3 studies)	Develop and support team meetings to structure protected time to reflect on the implementation effort, share lessons learned, and/or support one another’s learning	Develop stakeholder interrelationships	All practices were encouraged to initiate or increase routine staff meetings.[24]
Partner with local leadership (2 studies)	Create and engage a formal group of multiple levels of stakeholders (e.g., local leadership) to provide input and advice on QI/implementation efforts and to elicit recommendations for improvements	Develop stakeholder interrelationships	Get administrative buy-in.[20] Work with opinion leaders and encourage networking.[36]
Audit and feedbackc (18 studies)	Collect and summarize clinical performance data over a specified time period and provide it to clinicians and administrators to monitor, evaluate, and modify provider behavior	Use evaluative and iterative strategies	Written feedback and practice-based discussion of clinical record audit of recording and levels of behavioral and physiological risk factors.[37]
Ongoing consultation (12 studies)	Provide ongoing consultation to support maintenance of QI project or innovation	Train and educate stakeholders	The facilitator gradually transfers various tasks to an interested member of the team. The practices also meet without the facilitator to further customize their work.[32]

^aOperationalized definitions were modified from the ERIC strategy taxonomy

^bStudies with a learning collaborative were only included if the collaborative was not longitudinal and was only a minor part of the overall coaching-like intervention

^cAudit and feedback are considered 2 separate strategies, [48] though in many included studies they were described together

Abbreviations: QI quality improvement

Across the included studies, four studies measured practice-level outcomes, nineteen measured provider-level outcomes, and eight measured patient-level outcomes (Supplementary Information). Nine studies evaluated composite measures of process of care activities (i.e., outcomes which included the conduct of multiple disease-specific or clinical care approach actions). Overall, interventions typically targeted multiple simultaneous processes of care activities requiring disparate clinical behaviors (e.g., ordering a lab test, complicated patient counseling), but which were usually linked by a common goal (e.g., improving management and outcomes for a specific disease).

Of the nine trials that assessed the composite process of care outcomes, six were low or unclear ROB and two were high ROB. Six of eight low or unclear ROB trials favored the intervention (75%; 95% CI 35 to 97%). The probability of observing 75% or more trials with a beneficial effect, assuming the proportion of beneficial studies is truly 0.5, is p = 0.29. For the organizational process of care outcomes, four of five trials (including the two low ROB studies) favored the coaching interventions (80%; 95% CI 28 to 99%; p = 0.38). Of the five studies (1 unclear and 4 high ROB) that assessed the effect of coaching on appropriate documentation, three included outcomes that favored the interventions (60%; 95% CI 15 to 95%; p = 1). Six of seven studies (2 unclear and 4 high ROB) testing the effect of coaching on appropriate medication prescriptions contributed to the analysis. Four of these six studies included at least one outcome that favored the coaching intervention (66%; 95% CI 22 to 96%; p > 0.69). The three trials (1 unclear and 2 low ROB) that assessed the effect of coaching on counseling provision favored the intervention (100%; 95% CI 29 to 100%). Four trials assessed the provision of appropriate exams or procedures, and three out of those four included at least one outcome that favored the interventions (75%; 95% CI 19 to 99%). Of the six trials that assessed the effect of coaching on ordering of labs or vitals, all but one included at least some outcomes that favored the intervention (83%; 95% CI 36 to 100%; p = 0.22). Figure 2 shows a high-level summary of these results.

Fig. 2.

Summary of results

Two trials measured the effect of coaching on QI process goal attainment. One unclear ROB study found a significant increase in the number of QI projects per practice in the intervention versus the comparator arms with a mean of 3.9 QI projects per practice versus 2.6 (p < 0.001). In a high ROB trial, there was no significant difference between the intervention and control practices in the percentage of mean QI indicators at or above target (p > 0.2). No studies directly addressed the self-efficacy of team members related to QI method skills or a specific QI project activity. No trials addressed the effect of QI coaching or similar roles on team member knowledge.

Certainty of Evidence

Overall, our assessment of the certainty of evidence based on GRADE ranged from very low to low across outcomes (see Table 3). Downgrading, or causes for lower certainty, included imprecision, inconsistency, and a high risk of bias.

Table 3.

Certainty of Evidence Ratings

Outcome	Number of studies (N)	Range of effects	Certainty of evidence (rationale)
Adoption of targeted process of care activities
Composite process of care outcomes	9 randomized trials (677 practices and health service organizations)	6 of 9 trials (75%; 95% CI 35 to 97%) with at least 1 outcome favoring the intervention; 5 trials with statistically significant findings	Moderate certainty that coaching probably has a beneficial effect on composite process of care outcomes (rated down for serious risk of bias)
Organizational processes of care	5 randomized trials (471 practices)	4 of 5 trials (80%; 95% CI 28 to 99%) with at least 1 outcome favoring the intervention; 3 trials with statistically significant findings	Very low certainty that coaching possibly has a beneficial effect on organizational processes of care (rated down for serious risk of bias, inconsistency, indirectness, and imprecision)
Appropriate documentation	5 randomized trialsb (168 practices)	3 of 5trials (60%; 95% CI 15 to 95%) with at least 1 outcome favoring the intervention; 3 trials with statistically significant findings	Very low certainty that coaching possibly has a beneficial effect on appropriate documentation (rated down for very serious risk of bias, serious inconsistency, and imprecision)
Appropriate medication prescription	7 randomized trialsb (452 practices)	4 of 6 trials (66%; 95% CI 22 to 96%) with at least 2 outcomes favoring the intervention; none statistically significant	Very low certainty that coaching probably does not have a beneficial effect on appropriate medication prescription (rated down for very serious risk of bias, serious inconsistency, and imprecision)
Appropriate counseling	3randomized trials (412 practices)	3 of 3 trials (100%; 95% CI 29 to 100%); all statistically significant	Very low certainty that coaching possibly has a beneficial effect on appropriate counseling (rated down for serious risk of bias, indirectness, and imprecision)
Appropriate provider exams and procedures	4 randomized trials (255 practices)	3 of 4 trials (75%; 95% CI 19 to 99%) with at least 1 outcome favoring the intervention; 2 trials with statistically significant findings	Very low certainty of uncertain effect of coaching on improvement of provider exams/procedures (rated down for serious risk of bias, inconsistency, and imprecision)
Ordering of lab tests and vital signs	6 randomized trials (146 practices)	5 of 6 trials (83%; 95% CI 36 to 100%); 4 trials with statistically significant findings	Very low certainty that coaching probably has a beneficial effect on ordering of labs/vitals (rated down for serious risk of bias, inconsistency, and very serious imprecision)
QI process goal attainment (e.g., the number of QI projects reaching completion)
Mean # of QI projects initiated	1 randomized trial (49 practices)	3.9 QI projects per practice (intervention) vs 2.6 (comparator);p < 0.001	Low certainty that coaching possibly has a beneficial effect on number of the projects initiated (rated down for serious inconsistency and imprecision)
% mean indicators at target	1 randomized trial (23 practices)	Not significanta	Very low certainty that coaching has no effect on the number of indicators at target (rated down for serious risk of bias, inconsistency, and imprecision)
Improved team member knowledge
No trials addressed this outcome	–	–	–
Improved team member self-efficacy
Confidence in management	1 randomized trial (26 practices)	Mean confidence (pre) = 3.36 (SD 0.82); (post) = 3.89 (SD0.79); p value 0.000	Low certainty that coaching possibly has a beneficial effect on team member self-efficacy (rated down for serious risk of bias, and inconsistency)

^aAuthors only reported not significant results for comparison of relevance

^bOnly 3 trials provided valid information on direction of effect

Abbreviations: CI confidence interval; QI quality improvement

DISCUSSION

QI coaching is a complex intervention that has the potential to promote high-quality care through effective and efficient implementation of improvement activities at the team and practice level. We identified 23 trials that addressed the effects of QI coaching primarily in the context of primary care practices. We found that coaching interventions seemed to have more of an impact on less complex tasks like documentation possibly due to fewer implementation barriers. Process tasks that were more complex like medication prescription have lower confidence ratings due to the imprecision of the outcome measurement. While our confidence in these findings was found to be low to very low due to imprecision, inconsistency, and high risk of bias, the results suggest that clinical teams may be able to preferentially identify types of QI activities most likely to benefit from coaching support and thus help to facilitate the efficient use of process improvement resources around the implementation of evidence-based guidelines within busy clinical practices.

Our findings are largely consistent with and build on recently conducted reviews of roles like QI coaching, specifically external change agents and practice facilitation. Baskerville and colleagues [42] conducted a systematic review of 23 articles looking at the impact of practice facilitation on evidence-based practice behavior. Their approach differed from ours in that they considered the adoption of evidence-based guidelines to be a conceptually common outcome measure and did not distinguish between high and low complexity guidelines. They calculated standardized mean differences across studies and combined them for a pooled estimate. With this approach, they reported an effect size of 0.56 (95% CI 0.43 to 0.68) favoring practice facilitation in the adoption of evidence-based guidelines. However, we considered the adoption of evidence-based processes of care by the complexity of the care activity and noted that there appears to be variation in the effect of coaching-type roles on different types of processes of care. Additionally, our review differed from Baskerville et al. [42] in that it included a more expansive definition of the coaching intervention and focused on process outcomes.

A more recent review by Wang and colleagues [43] examined the impact of an intervention similar to QI coaching (i.e., practice facilitation) on chronic disease management in primary care. They grouped outcomes by type of outcome (e.g., lab vs diagnosis) within disease group (e.g., cervical cancer process of care measures vs chronic kidney disease process of care). This approach is consistent with the way interventions are often designed, specifically around the management of a particular disease; however, it could mask differences in effect by the complexity of the process of care. Across 25 studies, Wang et al. [43] concluded that process measures improved 8.8% with screening, and diagnosis improved the most. In contrast, we found the best evidence for a likely effect on the composite process of care outcomes (which were usually disease-specific and more general like preventive guidelines), organizational processes of care, counseling, and simple tasks like ordering of labs and vital signs. We found an uncertain effect on documentation (including documentation of diagnoses) and likely no effect on the prescription of disease-appropriate medications.

While there has been increased awareness about what coach-like interventions consist of, as well as their effectiveness, [42] QI coaching utilization remains uneven. One reason for this variability could be the barrier of establishing and sustaining funding for these roles, especially from smaller, independent practices. However, the benefits of using a coach to implement improvement activities could outweigh this initial cost. Prior reviews have also looked at which aspects of coach-like roles are likely contributors to an overall effect. Alagoz and colleagues [44] explored the role of external change agents in promoting changes in health care organizations in small primary care clinics across 21 included studies. They concluded that clinic-level individualized follow-up via practice facilitation is the most effective approach; however, the most commonly employed approaches are academic detailing and audit and feedback. Similarly, we found that audit and feedback (89% studies) and academic detailing, or educational outreach visits (68% studies), were among the most commonly used implementation strategies along with developing a formal implementation plan (95%) and distributing educational materials (74%), and that only 10 of 19 studies employed ongoing consultation (53%).

Limitations of the identified literature included loss of significant data when an entire practice (or cluster) dropped out of a study; inadequate descriptions of both the team members and patients; a lack of statistical consideration of clustering; and a lack of clearly identified primary outcomes. In addition, there was notable heterogeneity across study intervention core components, outcome measures, and the practice setting in which these studies took place. These factors along with inconsistency and imprecision of results led to downgrading the certainty of evidence. The uncertainty of these results could be addressed by more high-quality studies of QI coaching interventions, as well as investigation of coaching on very specific and consistently identified outcomes. Limitations of our approach to this review include potentially introducing heterogeneity by including literature from multiple fields of study and the loss of relevant information due to exclusion of studies with co-interventions, which prevented isolation of the coaching effect.

We also identified multiple gaps in the literature. First, few coaching interventions employed the strategies we identified as being most helpful in combination (e.g., stakeholder/leadership engagement and technical support). Second, most coaching interventions focused on predetermined QI projects rather than on the capacity for QI more generally. Third, all but one of the included interventions were conducted in primary care settings, so the effect of coaching in other clinical settings (e.g., inpatient, subspecialty clinics) is unknown.

CONCLUSION

QI coaching is a complex intervention that has the potential to improve the capacity for improvement activities at the team and practice level. QI coaching, and other interventions with similar characteristics (i.e., facilitation, outreach visitors), may have an effect on certain processes of care activities including composite process of care outcomes, ordering of labs and vital signs, and possibly on changes in the organizational process of care and delivery of appropriate counseling. Differences among studies in the description and dosing of implementation strategies employed by coaches, as well as outcome measurement, precluded a more definitive estimate of effects. Future research that standardizes and provides more detail about when coaching interventions are most effective will better support future comparisons and implementation efforts.

Supplementary Information

Below is the link to the electronic supplementary material.

Funding

This project was funded by the Department of Veterans Affairs, Veterans Health Administration, Office of Research and Development, Quality Enhancement Research Initiative (ESP 09–010). This work was also supported by the Durham Center of Innovation to Accelerate Discovery and Practice Transformation (ADAPT) (CIN 13–410) at the Durham VA Health Care System. The views expressed are those of the authors and do not reflect the official views or policy of the Department of Defense or its Components. Dr. Ballengee’s effort was supported by the Department of Veterans Affairs Quality Scholars Program. Dr. Goldstein’s effort is supported by VA HSR&D CDA award #13–263. Dr. Lewinski’s effort is supported by a VA HSR&D grant #18–234. Dr. Ramos’s effort is supported by the National Cancer Institute (R01CA2011790), and the National Institute on Aging (P30 NIA 12734132; U24-AG058556-03); Dr. Rushton receives funding from HRSA Primary Care Training and Enhancement Program (TOBHP29992), which is not related to this work. Dr. Brahmajothi’s effort is supported by DoD grant (W81XWH1810454).

Declarations

Conflict of Interest

Dr. Lewinski reports receiving funding from Otsuka and the PhRMA Foundation. Dr. Zullig reports receiving research funding from Proteus Digital Health and the PhRMA Foundation as well as consulting with Novartis, none related to the current work.