Shadow Coaching Improves Patient Experience with Care, But Gains Erode Later

Abstract

Background.

Healthcare organizations strive to improve patient care experiences. Some use one-on-one provider counseling (“shadow coaching”) to identify and target modifiable provider behaviors.

Objective.

We examined whether shadow coaching improves patient experience across 44 primary-care practices in a large urban Federally Qualified Health Center.

Research Design.

Seventy-four providers with “medium” (i.e., slightly below average) overall provider ratings received coaching and were compared to 246 uncoached providers. We fit mixed-effects regression models with random effects for provider (level of treatment assignment) and fixed effects for time (linear spline with a knot and “jump” at coaching date), patient characteristics and site indicators. By design, coached providers performed worse at selection; models account for the very small (0.2 point) regression-to-the-mean effects. We assessed differential effects by coach.

Subjects.

46,452 patients (from 320 providers) who completed Clinician and Group Consumer Assessment of Healthcare Providers and Systems (CG-CAHPS®) surveys.

Measures.

CAHPS overall provider rating and provider communication composite (scaled 0-100).

Results.

Providers not chosen for coaching had a non-significant change in performance during the period when selected providers were coached. We observed a statistically significant 2-point (small-to-medium) jump among coached providers after coaching on the CAHPS overall provider rating and provider communication score. However, these gains disappeared after 2.5 years; effects differed by coach.

Conclusions.

Shadow coaching improved providers’ overall performance and communication immediately after being coached. Regularly planned shadow coaching ‘booster’ sessions might maintain or even increase the improvement gained in patient experience scores, but research examining additional coaching and optimal implementation is needed.

Introduction

Good provider communication is crucial to the doctor-patient relationship and influences patient adherence to care plans and clinical outcomes.^1–9 Most measure patient experience using the Consumer Assessment of Healthcare Providers and Systems (CAHPS®) surveys to,^10–12 the national standard for collecting, tracking and benchmarking patient care experiences across settings including ambulatory care.^13–29 With CAHPS scores publicly available for individual providers and at the group level, provider groups can identify specific, modifiable provider behaviors that they can target to improve patient care experiences.^30–33

Healthcare organizations generally use one or more of six methods of changing physician behavior: education/training, feedback, engaging physicians as leaders in change efforts, administrative changes, financial incentives, or financial penalties.^34,35 Substantial time and resources are invested in communication training for clinicians (such as continuing medical education); however, little research has been done to test the effect of such training on patient experience scores and those studies that have provide little evidence of its effectiveness.^36,37

Some groups and practices use individualized feedback or one-on-one provider counseling (“shadow coaching”) based on patient experience scores to target modifiable provider behaviors.^38–44 Shadow coaching refers to the process of commissioning coaches to observe providers in real-time encounters at a point-of-care and provide structured, specific feedback (vs. general feedback) to encourage targeted behaviors.⁴³ Although there are various coaching models and philosophies,^40,41,44 shadow coaching is usually conducted during a half or full day to observe several patient encounters.^42,45Prior work has examined coaching,^46–52 compliance training for nursing staff,^53–55 and coaching using simulated patient encounters.⁵⁶ Witherspoon and White (1996)⁵⁷ suggest coaching has four different functions: coaching for skill enhancement, increased performance, development, and strategic planning. The first three functions are relevant to our study. The fourth, coaching for strategic planning, is executive coaching and not addressed. Also, three terms are often associated with coaching, but are different from coaching: 1) Managing- To manage people is to make sure that they do what they already know how to do. 2) Training - When people need to learn something new, training is introduced. 3) Mentoring – This involves advising, guiding and counseling by an expert and can involve a component of coaching.⁴⁹ Coaching is slightly different than managing, training or mentoring, as its optimal use leads to the increased utilization of a person’s current skills and resources without counseling or advising.

Shadow coaching, a type of collaborative or peer-assisted learning,^58,59 is a distinct type of coaching in which peers, who often have a similar knowledge level to those they coach,^60,61 enter an equal, non-competitive voluntary relationship, establishing goals or preferred outcomes, observing, and providing feedback to improve task performance and support changes that amplify participants’ strengths and capacity.^58,62–65 Sessions usually occur in dyads.⁶⁶ Mutual trust between recipients and coaches is essential for successful peer-coaching relationships.^67–70

Some studies on shadow coaching have tracked the impact of coaching on outcome measures, providing evidence that coaching, shadow or otherwise, helps build and maintain competencies among physicians, nurses and other staff, and increases compliance with practice guidelines.^47,52,56 Despite the primary shadow coaching objective to improve individual provider behaviors and the associated patient interactions and experiences, research on its effectiveness and its specific impact on patient experiences is relatively sparse.^71–73 We examine whether shadow coaching improves patient experience scores and if improvements are sustained.

Methods

Setting.

The study was conducted in a large, urban Federally Qualified Health Center (FQHC) in California with nearly 1 million patient visits annually. Six years prior to this study, the FQHC’s chief medical officer implemented a company-wide quality monitoring system based on the overall provider rating and provider communication composite of the Clinician and Group CAHPS (CG-CAHPS) visit survey 2.0: https://www.ahrq.gov/cahps/surveys-guidance/cg/visit/index.html.¹⁴ Provider communication was chosen because it has the highest correlation of the CAHPS composites with the overall rating of care.²¹

Shadow coaching was introduced as part of quality monitoring with the goal of improving patient care experiences. Every 6 months, in January and July, the FQHC calculated every provider’s average 6-month score on the CAHPS overall provider rating (scored with a 0-100 possible range, higher score is better). Providers with a 6-month average score between 45-89 in the 6 months prior to the coaching were identified as “medium performers” and selected for coaching.

Intervention.

Eight full-time, high-performing providers at the FQHC, based on patient experience and other performance indicators, were selected to shadow other providers during 4 or more patient encounters during a half-to-full day. Coaches attended a one-day coaching seminar by the SullivanLuallin group.^42,74–76 Provider assignments were made based on geography; coaches were essentially assigned regions to minimize their commuting time. Medical director coaches were not permitted to coach providers that reported to them. These shadow coaches observed providers and after the observation provided verbal feedback about strengths and areas of improvement with a focus on patient-provider interactions. Coach feedback was based on their own experiences as high-performing physicians and broader insights into what makes for high-quality patient-provider communication derived from the coaching seminar. This initial feedback was followed by a written coaching report from the coach to the provider summarizing the comments and recommendations from the coaching session.⁴² The primary goal of the shadow coaching session was to identify and target areas of patient-provider interaction that a provider could improve when interacting and caring for their patients, with a focus on provider communication. Coaching occurred from March 2015 to August 2018.

The FQHC in our study implemented shadow coaching with the afore-mentioned critical components as their program established that having CG-CAHPS scores over 90 was the goal for all providers, participation in shadow coaching was voluntary, and coaching included both (1) self-evaluation in an immediate feedback conversation with the coach after the observation of patient encounters and (2) a written coaching report that focused on both encouraging current behaviors that were participants’ strengths and recommendations of beneficial new behaviors.

Measures.

The CAHPS surveys that include the overall provider rating and the communication items were completed either by adult patients or by parents of children who are patients. The communication composite (4 items) assesses the frequency of explaining things in a way that is easy to understand, listening carefully to the patient, showing respect for what the patient says, and spending enough time with the patient⁵ using a 4-point Never/Sometimes/Usually/Always response scale.

Data and Analysis.

The sample included 320 providers from 44 practices with 46,452 completed CG-CAHPS visit surveys. We used a pre-post design to compare the performance trajectories of providers who were coached vs. those who were not during the interval between January 2012 to June 2019. The dependent variables were the CG-CAHPS overall provider rating and provider communication composite.

We compared trends for coached and uncoached providers before and after coaching. The actual coaching date was used for the coached providers; the mean coaching date was assigned for uncoached providers. We fit mixed-effects regression models with random effects for provider (the level of treatment assignment) and fixed effects for time (a linear spline with a single knot and a possible “jump” at the coaching date),^77,78 patient characteristics (adult/child, age, gender, race/ethnicity, language, general health status, education), and practice indicators. This spline model allows the slope to change at the time of coaching (i.e., the knot), for a gradual change in scores over time, and also allows for a possible vertical discontinuity or a jump in the measured scores instantaneously after the date of the intervention. Allowing the trajectory to change at the time of coaching independently for the coached and uncoached groups addressed the possible threat of regression to the mean from performance-based treatment assignment (very small effect of 0.2 point).

The spline model allows us to detect two different forms of intervention effects, each of which represents a departure in the intervention group from what would have been expected in the absence of an intervention. The first effect, referred to as “Differential immediate change (i.e., jump) at coaching for coached,” captures an immediate change in scores in the intervention group (coached providers), relative to the control group (uncoached providers), at the time of the intervention (coaching for the coached providers). This shift or jump at the time of coaching is at the spline knot in the model (the coaching date). The null hypothesis of the test of this effect is no differential change in score between the coached at uncoached providers at the time of coaching, i.e., no instantaneous effect of the intervention at the time of the intervention. A significant positive value for the coefficient indicates an instantaneous positive change for the coached provider relative to any change for uncoached providers at coaching. The second intervention effect in our study, labeled as “Differential slope change at coaching for coached,” captures any change in the slope of the coached providers at the time of coaching relative to any change in the slope of the uncoached providers. The null hypothesis of the test of this effect corresponds to no differential slope change and hence no gradual effect of the intervention. A significant positive value for this effect indicates the trajectory of the outcome after coaching for the coached providers increases relative to that of the uncoached providers.

Including practice fixed effects and provider random effects in the models allowed mean performance to vary by provider and practice. As a sensitivity analysis, we fit two additional models per dependent variable to see if there was a differential intervention effect in our intervention variables by two provider characteristics: specialty (5-category) and provider type (2-category). In these models, we interacted our intervention variables (and all supporting lower-order interactions) with the provider characteristic and then conducted joint tests of whether there was variation in our intervention effects across the provider characteristic categories. We found no evidence that there were differences in intervention effects by specialty or provider type in our main study variables for either outcome.

Lastly, we assessed differential effects by coaches. Of the eight coaches, four coached more than one provider for whom we had patient experience data. Indicator variables for these four coaches were created and interacted with our main study variables of interest in place of a single coached indicator (and all supporting lower-order interactions). We conducted joint tests to determine whether our main study variables of interest differed among the four coaches, with a null hypothesis of no difference among coaches in the jump at coaching nor in the slopes after coaching.

Results

The FQHC identified 74 “medium-performing” providers (of 320) based on CAHPS scores and shadowed their interactions with patients (i.e., coached providers). About half (52%) of the FHQC’s providers specialized in family medicine, followed by pediatrics (25%). The majority of providers were doctors with MD, DO or DDS degrees (63%). Similar proportions of coached and uncoached providers were medical doctors (vs. nurse practitioners/medical assistants) and specialties were similar, except that more coached providers specialized in internal medicine (22% vs. 10%) and more uncoached providers were pediatricians (28% vs, 15%). The median patient age was 35-44, with 18% under 17 and 9% over 64.; 37% were male; 69% were Hispanic. Half (50%) of the adult patients and 71% of the parents of child patients had not attended college (Table 1).

Table 1.

Provider and Patient Characteristics, Overall and By Coached Status

	All Providers		Coached Providers		Uncoached Providers
	Provider Characteristics
Total Providers (N)	320		74		246
Provider Type	%	N	%	N	%	N
MD/DO/DDS	62.5	200	63.5	47	62.2	153
Nurse Practitioner/Medical Assistant	27.5	88	36.5	27	24.8	61
Missing	10	32	0	0	13	32
Provider Specialty
Family Medicine	51.9	166	51.4	38	52.0	128
Internal Medicine	12.8	41	21.6	16	10.2	25
Infectious Disease	2.5	8	2.7	2	2.4	6
Pediatrics	25.0	80	14.9	11	28.1	69
Women’s Health	7.5	24	9.5	7	6.9	17
Missing	0.3	1	0	0	0.3	1
Total Patient Surveys (N)	46,452		20,720		25,732
	Mean	SD	Mean	SD	Mean	SD
Number of patient surveys per provider	145.2	157.4	280.0	157.7	104.6	133.1
	Patient Characteristics
Age (years)	%	SE	%	SE	%	SE
0-17	18.0	0.18	14.3	0.24	21.0	0.25
18-24	6.3	0.11	7.3	0.18	5.5	0.14
25-34	14.2	0.16	15.6	0.25	13.1	0.21
35-44	13.5	0.16	14.3	0.24	12.9	0.21
45-54	17.6	0.18	17.3	0.26	17.8	0.24
55-64	21.2	0.19	21.6	0.29	20.8	0.25
65+	9.2	0.13	9.8	0.21	8.7	0.18
Male	37.4	0.22	35.7	0.33	38.8	0.30
Race, ethnicity, and language of survey
Hispanic and surveyed in Spanish	29.1	0.21	27.2	11.9	28.0	17.6
Hispanic and surveyed in English	40.2	0.23	41.8	0.34	38.9	0.30
Non-Hispanic White	12.6	0.15	11.2	0.22	13.7	0.21
Asian/Pacific Islander	6.6	0.11	7.7	0.19	5.6	0.14
Other race or ethnicity	11.6	0.15	11.0	0.22	12.1	0.20
General health status
Excellent	19.1	0.18	18.0	0.26	20.1	0.24
Very good	26.6	0.20	26.3	0.30	26.8	0.27
Good	30.1	0.21	33.1	0.32	31.2	0.28
Fair	18.5	0.18	18.8	0.27	18.3	0.24
Poor	3.8	0.09	3.9	0.13	3.7	0.11
Education of adult patients (N=38,120)
8th grade or less	14.2	0.17	13.6	0.25	14.7	0.24
Some high school	12.8	0.16	13.0	0.24	12.7	0.22
High school diploma	23.2	0.21	23.6	0.31	22.8	0.28
Some college or 2-year degree	31.6	0.23	31.3	0.34	31.8	0.31
4-year college degree	10.9	0.15	11.0	0.23	10.9	0.21
More than 4-year college degree	7.3	0.13	7.5	0.19	7.2	0.17
Education of parent for child patients (N=8,332)
8th grade or less	12.8	0.34	12.6	0.57	12.9	0.43
Some high school	13.1	0.35	13.2	0.59	13.1	0.43
High school diploma	25.8	0.45	27.4	0.77	24.9	0.55
Some college or 2-year degree	31.7	0.48	30.7	0.80	32.2	0.60
4-year college degree	10.3	0.31	10.4	0.53	10.2	0.39
More than 4-year college degree	6.3	0.25	5.6	0.40	6.6	0.32

NOTE: SD= standard deviation. SE= standard error. MD/DO/DDS= Medical Doctor/Doctor of Osteopathic medicine/ Doctor of Dental Surgery

Patient Experience Trends Before and After Coaching.

Model results (Table 2) indicate that uncoached providers (n = 246), had a non-significant change in performance during the period that the selected providers were coached, as expected. Among coached providers, we identified a statistically significant two-point jump on the 0-100 scale for the CAHPS measures (overall provider rating 2.0 with standard error (SE) 0.6 and communication score 1.9 with SE 0.6) at the time of coaching (labeled as “Differential immediate change (i.e., jump) at coaching for coached” in Table 2). Differences of 1, 3, and ≥5 points for CAHPS measures are considered small, medium, and large, respectively.⁷⁹ The change in scores for the uncoached providers at the mean time of coaching of coached providers (labeled as “Immediate change at coaching for uncoached” in Table 2) was non-significant for both the overall provider rating and communication score, −0.3 and −0.8, respectively. Slopes from the spline model before and after coaching for both the uncoached and coached providers are also shown in Table 2. There was a significant increase in overall provider rating in the time before coaching for both the uncoached and coached providers (slope before coaching was equal to 0.4 (SE 0.1) for uncoached and 0.6 (SE 0.2) for coached); however, these slopes after coaching were not significantly different from zero (i.e., a flat line). For provider communication, the slope before coaching for the coached providers (but not for the uncoached providers) was significant, the estimate of the slope before coaching was equal to 0.4 (SE 0.2).

Table 2.

Linear Spline Model Results with a Single Knot and Possible “jump” at Coaching Date, By Case-Mix Adjusted Measure

	Case-mix Adjusted Overall Provider Rating (n=46,089)				Case-mix Adjusted Provider Communication (n=46,440)
	Estimate	SE	p-value		Estimate	SE	p-value
Independent Variables (Fixed Effects)
Intercept	80.4	1.3	<.0001	***	79.5	1.4	<.0001	***
Coached indicator	−0.8	0.8	0.287		−0.3	0.8	0.7414
Pre-coaching slope for uncoached providers	0.4	0.1	0.0033	**	0.3	0.1	0.0667
Difference in pre-coaching slope between coached and uncoached providers	0.1	0.2	0.5151		0.1	0.2	0.6762
Immediate change at coaching for uncoached	−0.3	0.4	0.4686		−0.8	0.5	0.1084
Differential immediate change (i.e., jump) at coaching for coached	2.0	0.6	0.0007	***	1.9	0.6	0.0022	**
SLOPE CHANGE AT COACHING FOR UNCOACHED	−0.1	0.2	0.7822		0.0	0.3	0.8722
Differential change in slope AT COACHING FOR COACHED	−0.8	0.3	0.0185	*	−0.7	0.4	0.0358	*
POST ESTIMATION
Uncoached providers slope in pre-coaching period	0.4	0.1	0.0033	**	0.3	0.1	0.0667
Uncoached providers slope in post-coaching period	0.3	0.2	0.0777		0.3	0.2	0.1309
Coached providers slope before coaching	0.6	0.2	0.0003	***	0.4	0.2	0.0235	*
Coached providers slope after coaching	−0.3	0.2	0.0738		−0.3	0.2	0.0635
Random Effects
Variance Components
Provider	20.9	2.3	<.0001	***	24.5	2.6	<.0001	***
Residual	257.7	1.7	<.0001	***	290.0	1.9	<.0001	***
Square Root of variance component
Provider	4.6				4.9 (0.0)
Case mix adjusters (Fixed Effects)
Patient Age (Ref: 18–24)
0-17	0.5	0.5	0.2523		1.5	0.5	0.0021	**
25-34	0.5	0.4	0.1463		1.2	0.4	0.0013	**
35-44	2.2	0.4	<.0001	***	3.1	0.4	<.0001	***
45-54	4.1	0.4	<.0001	***	5.2	0.4	<.0001	***
55-64	5.25	0.4	<.0001	***	6.6	0.4	<.0001	***
65+	5.58	0.4	<.0001	***	7.2	0.5	<.0001	***
MALE	0.01	0.2	0.9532		0.4	0.2	0.0361	*
Patient race/ethnicity/language (Ref: White, English Survey)
Hispanic, Spanish survey	2.3	0.3	<.0001	***	−0.0	0.3	0.9343
Hispanic, English survey	1.5	0.3	<.0001	***	1.1	0.3	<.0001	***
API	2.0	0.4	<.0001	***	3.0	0.4	<.0001	***
Other	0.6	0.3	0.069		0.0	0.3	0.9834
Patient General Health Status (Ref: Poor)
Excellent	12.5	0.4	<.0001	***	13.8	0.5	<.0001	***
Very good	9.2	0.4	<.0001	***	11.3	0.5	<.0001	***
Good	7.2	0.4	<.0001	***	9.1	0.4	<.0001	***
Fair	5.3	0.4	<.0001	***	6.1	0.5	<.0001	***
Respondent education (Ref: 8th grade or less)
Some high school	0.0	0.3	0.9115		0.1	0.3	0.743
High school diploma	−0.3	0.3	0.2607		−0.1	0.3	0.6359
Some college or 2-yr degree	−1.7	0.3	<.0001	***	−1.5	0.3	<.0001	***
4-year college degree	−2.6	0.4	<.0001	***	−2.2	0.4	<.0001	***
More than 4-year college	−3.1	0.4	<.0001	***	−2.4	0.4	<.0001	***
Site (coefficients not shown)				**				**
Omnibus test (43 df)			0.0011				0.0037

NOTE:

^*p<0.05,

^**p<0.01,

^***p<0.001.

SE= standard error. The slope before coaching describes change in effect from 2.5 years prior to coaching up to coaching date. The slope before coaching describes change in effect from day after coaching date to 2.5 years post coaching.

Figure 1, Panel A shows the adjusted overall provider rating results before and after coaching (in 6-month intervals) for the uncoached providers, coached providers and estimated trend if the coached providers had not been coached—that is, we predicted what their patient experience trend would have been if they had not received coaching. Figure 1, Panel B shows the same results for the adjusted provider communication composite. Note that after coaching, the improvement gains in patient experience scores faded significantly, by ~40% a year, disappearing after 2.5 years. That is, we calculate [(coached indicator*years since coaching*post-coaching indicator)/ (coached indicator*post-coaching indicator)], which is [(−0.8/2) *100], equaling 40% for overall provider rating and 37% for provider communication.

Figure 1.

Adjusted Over Time Trend, Before and After Coaching, By Measure

In the models controlling for individual coaches, we found providers coached by Coach B and D both had over a 2-point jump in provider rating at the time of coaching, (labeled as “Coach B’s (or Coach D’s) immediate change (i.e., jump) at coaching compared to uncoached” in Table 3) with jumps equal to 2.6 (SE 0.9) and 2.8 (SE 0.8), respectively. This was also seen in provider communication, where again providers coached by Coach B and D had a significant jump at the time of coaching compared to providers of other coaches and the non-coached providers. Coach B’s providers improved by 2.7 (SE 0.9) at the time of coaching and Coach D by 3.2 (SE 0.9) in provider communication scores. The joint (3df) test of whether the jumps among the four coaches differed was not significant for the provider rating, (F=2.46, (p = 0.06) but was for provider communication F=4.09 (p=0.007). There were no significant differences among the coaches in their slopes after coaching (p>0.05 for each); therefore, we do not report these slopes by coach. In Figure 2, Panel A shows estimated regression lines for the two high-impact coaches (Coach B and D) and Panel B for the low-impact coaches (Coach A and C). A significant jump at coaching is seen in provider rating for providers coached by Coach B (p = 0.005) and Coach D (p < 0.001). These differences hold up to one and half years after coaching for providers coached by Coach D (p = 0.002 at six months; p=0.04 at one and half years), and for up to six months after coaching for Coach B (p = 0.03). For provider communication (data not shown), a significant jump is also seen after coaching for providers coached by Coach B (p = 0.006) and Coach D (p < 0.001). These differences hold up to one and half years after coaching for providers coached by Coach D (p< 0.001 at six months; p=0.008 at one and half years), and for up to six months after coaching for Coach B (p = 0.02).

Table 3.

Linear Spline Model Results with a Single Knot and Possible “jump” at Coaching Date including Differential Coach Effects, By Case-Mix Adjusted Measure

	Global Rating (N=45,735)	Provider Communication (N=46,085)
Independent Variables (Fixed Effects)	Estimate (SE)	Estimate (SE)
Intercept	80.5 (1.3) ***	79.3 (1.4) ***
Coach A	0.1 (1.2)	0.7 (1.3)
Coach B	0.2 (1.4)	0.5 (1.6)
Coach C	0.0 (1.7)	1.9 (1.8)
Coach D	−2.5 (1.2) *	−2.5 (1.3)
Pre-coaching slope for uncoached providers	0.4 (0.1) **	0.3 (0.1)
Coach A’s difference in pre-coaching slope compared to uncoached providers	0.6 (0.3)	0.6 (0.4)
Coach B’s difference in pre-coaching slope compared to uncoached providers	0.4 (0.4)	0.0 (0.4)
Coach C’s difference in pre-coaching slope compared to uncoached providers	0.3 (0.4)	1.1 (0.4) **
Coach D’s difference in pre-coaching slope compared to uncoached providers	−0.4 (0.3)	−0.6 (0.3)
Uncoached provider immediate change at coaching	−0.3 (0.4)	−0.7 (0.5)
Coach A’s immediate change (i.e., jump) at coaching compared to uncoached	0.1 (0.9)	−0.1 (1.0)
Coach B’s immediate change (i.e., jump) at coaching compared to uncoached	2.6 (0.9) **	2.7 (0.9) **
Coach C’s immediate change (i.e., jump) at coaching compared to uncoached	1.4 (1.1)	0.0 (1.1)
Coach D’s immediate change (i.e., jump) at coaching compared to uncoached	2.8 (0.8) ***	3.2 (0.9) ***
Slope change at coaching for uncoached	−0.1 (0.2)	0.0 (0.3)
Coach A’s slope change at coaching compared to uncoached	−0.7 (0.6)	−0.7 (0.6)
Coach B’s slope change at coaching compared to uncoached	−1.3 (0.5) **	−1.0 (0.5)
Coach C’s slope change at coaching compared to uncoached	−0.8 (0.6)	−1.0 (0.7)
Coach D’s slope change at coaching compared to uncoached	−0.5 (0.5)	−0.2 (0.5)
Post-estimation	0.3 (0.4)	1.1 (0.4) **
Uncoached providers slope in pre-coaching period	−0.4 (0.3)	−0.6 (0.3)
Uncoached providers slope in post-coaching period	−0.3 (0.4)	−0.7 (0.5)
Coach A’s slope before coaching	0.1 (0.9)	−0.1 (1.0)
Coach B’s slope before coaching	2.6 (0.9) **	2.7 (0.9) **
Coach C’s slope before coaching	1.4 (1.1)	0.0 (1.1)
Coach D’s slope before coaching	2.8 (0.8) ***	3.2 (0.9) ***
Coach A’s slope after coaching	−0.1 (0.2)	0.0 (0.3)
Coach B’s slope after coaching	−0.7 (0.6)	−0.7 (0.6)
Coach C’s slope after coaching	−1.3 (0.5) **	−1.0 (0.5)
Coach D’s slope after coaching	−0.8 (0.6)	−1.0 (0.7)
Random Effects
Variance Components
Provider	21.2 (2.3) ***	24.8 (2.7) ***
Residual	256.5 (1.7) ***	288.7 (1.9) ***
Square Root of variance component of provider	4.6	5.0
Case Mix Adjustors (Fixed Effects)
Patient age (Ref: 18–24)
0-17	0.6 (0.5)	1.5 (0.5) **
25-34	0.6 (0.4)	1.3 (0.4) ***
35-44	2.2 (0.4) ***	3.1 (0.4) ***
45-54	4.1 (0.4) ***	5.2 (0.4) ***
55-64	5.3 (0.4) ***	6.6 (0.4) ***
65+	5.6 (0.4) ***	7.2 (0.5) ***
Male	0.0 (0.2)	0.4 (0.2) *
Patient race/ethnicity and survey language (Ref=White, English survey)
Hispanic, Spanish survey	2.2 (0.3) ***	−0.1 (0.3)
Hispanic, English survey	1.5 (0.3) ***	1.1 (0.3) ***
API	1.9 (0.4) ***	3.0 (0.4) ***
Other	0.5 (0.3)	0.0 (0.3)
Patient General Health Scale (Ref: Poor)
Excellent	12.4 (0.4) ***	13.7 (0.5) ***
Very good	9.1 (0.4) ***	11.3 (0.5) ***
Good	7.2 (0.4) ***	9.0 (0.4) ***
Fair	5.3 (0.4) ***	6.1 (0.5) ***
Respondent education (Ref: 8th grade or less)
Some high school	0.0 (0.3)	0.1 (0.3)
High school diploma	−0.4 (0.3)	−0.2 (0.3)
Some college or 2-year degree	−1.7 (0.3) ***	−1.5 (0.3) ***
4-year college degree	−2.6 (0.4) ***	−2.2 (0.4) ***
More than 4-year college degree	−3.1 (0.4) ***	−2.5 (0.4) ***

NOTE:

^*p<0.05,

^**p<0.01,

^***p<0.001.

SE= standard error. The slope before coaching describes change in effect from 2.5 years prior to coaching up to coaching date. The slope after coaching describes change in effect from day after coaching date to 2.5 years post coaching.

Figure 2.

Adjusted Over Time Trend Before and After Coaching, By Impact

Discussion

Studies consistently show that effective communication between clinicians and patients is a critical determinant of overall patient experience.^21,80–86 Organizations engage in several ways to change providers behavior; many provide communication training for clinicians. Generally, provider training includes 4 to 8 hours of instruction and takes place during regular office hours,^87–91 requiring clinicians to be relieved of their responsibilities. However, very little research has studied the effect of this type of communication training, with limited evidence of its effectiveness in improving patient experiences.^{36,88,91–93} Often, any initial improvement from such communication training dissipates as providers return to overloaded schedules and variable patient expectations.⁴³

Shadow coaching, on the other hand, has proven more effective, with some studies providing evidence that coaching, shadow or otherwise, helps build and maintain competencies among physicians, nurses and other staff, and increases compliance with practice guidelines.^47,52,56,89 Ravitz et al 2013⁵⁶ studied 26 physicians and found improvements to physician-patient communication 4-6 months after an intervention that included four weekly videotaped sessions with feedback and coaching. Yusuf et al 2018⁵² reported significant improvement for nurse-physician dynamic after 3 months of observation and coaching with results sustained at 12 months. Our study is consistent with these studies in finding an immediate improvement in providers’ overall performance and communication immediately following shadow coaching, as captured by the CAHPS measures.

Our study is the first to our knowledge to evaluate improvement sustainability 2.5 years after shadow coaching. Although the peer shadow coaching used in our study incorporates many features that are consistent with the literature on successful behavior change (a learner-centered approach, immediate feedback, written recommendations on what skills to practice/behaviors to engage in), we found that the gains in patient experience scores for coached providers faded significantly (i.e., about ~40% a year) and disappeared after 2.5 years. This may correspond to providers gradually slipping back into their previous habits of interacting with patients, as there is some evidence that clinician behavior lapses when feedback ends.⁹⁴ More generally, erosion related to physician behaviors and coaching⁹⁵ is a specific instance of a general phenomenon, and has been found in other behavioral change interventions (e.g., smoking cessation, alcohol use, anger management, weight loss programs). Engaging providers in the coaching process is critical^96,97 but its program implementation also needs a “booster” component to sustain desired behaviors.^95–99 Structural and environmental factors, which may have led to the need for the coaching in the first place, may also have been at least partly responsible for the erosion of improvement. Organizational structure may obstruct providers seeking to change their behaviors.^100,101 Having less than full engagement by providers involved in the organization-wide mission to provide high-quality care experiences could stifle improvement efforts. Making structural changes that allow physicians to have more time with patients could foster and encourage physicians to focus on improving provider-patient interactions. Targeted quality improvement efforts aimed to ease providers overloaded schedules may allow for more time for physicians to try out and incorporate new behaviors. Addressing such systemic changes could allow providers to maintain and even further improve their interactions with patients.

We also considered whether coaches differed in effectiveness. We found evidence that two coaches were more effective than the other two, initially and 18 months later, but without clear evidence of differences in the rate at which gains eroded. These two effective coaches provided additional follow-up at 3-and 6-months to review CAHPS scores with the provider and touched base with them on what was working or not working based on the recommended/suggested behavioral modifications; the other coaches did not provide any follow-up. Such reinforcement after coaching may solidify new learning and lead to more enduring change, consistent with the limited research that states providing skill reinforcement post training helps change physician performance.^36,102,103 Regularly planned annual shadow coaching ‘boosters’ might maintain or even increase the improvement in patient experience scores that otherwise erodes.

These findings have practical implications for implementing a coaching program. First, the two most effective coaches spontaneously added follow-up engagement with those they coached, suggesting that such follow-up should be added as a required component. Second, evidence that gains erode suggest the importance of adding “refresher” booster coaching sessions. Booster sessions have been shown to be important for other behavioral interventions such as drug use prevention programs.^104,105 These additions would ensure a more consistent and persistent coaching effect on provider behaviors. Finally, future research should examine the most effective pairing of coaches with those coached, considering supervisory relationships, medical specialty, and other factors. Research examining the impact of additional coaching and work that investigates and identifies mechanisms that sustain changes in provider behavior are also both needed.

Limitations.

We studied one large FQHC using CAHPS data as the basis for eligibility to provide shadow coaching to providers, so our findings may not generalize to all settings, but they are instructive, given limited research evaluating the effect of shadow coaching on patient experiences. We also were unable to evaluate all providers who were coached, as patient experience surveys were not available for some providers before and after coaching. Our findings support the current literature on the effectiveness of shadow coaching, but also extend the literature by investigating the sustainability of improvements and the differential effects of coaches.

Conclusion

Practices use patient experience scores as a metric for patient-centeredness and for improving provider-patient interactions. Overall ratings of providers and provider communication scores can be improved by implementing peer shadow coaching that targets modifiable provider behaviors. Shadow coaching ‘booster’ sessions might maintain or increase the improvement in patient experience scores that otherwise erodes. Attention should also be paid to the faithful implementation of all aspects of shadow coaching to ensure its effectiveness.