Evidence relating healthcare provider burnout and quality of care: A systematic review and meta-analysis

Abstract

Background:

Whether healthcare provider burnout contributes to lower quality of patient care is unclear.

Purpose:

To estimate the overall relationship between burnout and quality of care, and to evaluate if published studies provide exaggerated estimates of this relationship.

Data Sources:

MEDLINE, Embase, and Web of Science, with no language restrictions, from inception through May 28, 2019.

Study Selection:

Peer-reviewed publications published in any language that quantified healthcare provider burnout in relation to patient quality of care.

Data Extraction:

Two reviewers independently selected studies, extracted measures of association of burnout and quality of care, and assessed potential bias using the Ioannidis (excess significance) and Egger (small-study effect) tests.

Data Synthesis:

We screened 11,703 citations, selecting 123 publications with 142 study populations encompassing 241,553 healthcare providers. Relations between burnout and quality of care were highly heterogeneous (I² 93.4%−98.8%). Only 4 studies specified a primary outcome. Of 114 unique burnout and quality combinations, 58 indicated burnout related to poor quality of care, 6 indicated burnout related to high quality of care, and 50 showed no significant effect. Excess significance was apparent (73% observed vs 62% predicted studies with statistically significant results, P = 0.011), with this indicator of potential bias most prominent for the least rigorous quality measures of Best Practices and Quality and Safety.

Limitations:

Primarily observational studies; unable to determine causality or directionality.

Conclusion:

Healthcare professional burnout frequently associates with poor quality of care in the published literature. The true effect size may be smaller than reported. Future studies should pre-specify outcomes to lower the risk of exaggerated effect size estimates.

Funding Sources:

Stanford Maternal and Child Health Research Institute (PI: Tawfik), Eunice Kennedy Shriver National Institute of Child Health and Human Development (R01 HD084679–01, PIs: Profit and Sexton)

INTRODUCTION

Healthcare providers face a rapidly changing landscape of technology, care delivery methods, and regulations that increase risk for professional burnout. Studies suggest that nearly half of healthcare providers may have burnout symptoms at any given time.(1) Burnout has been linked to adverse effects at the provider and organizational levels, including suicidality, broken relationships, decreased productivity, unprofessional behavior, and turnover.(2–6)

Recent attention has focused on the relation between healthcare provider burnout and reduced quality of care, with a growing body of primary literature and systematic reviews reporting associations between burnout and adherence to practice guidelines, communication, medical errors, patient outcomes, and safety metrics.(7–11) Most studies in this field employ retrospective observational designs, and use a wide range of burnout assessments and analytic tools to evaluate myriad outcomes among diverse patient populations.(12) This lack of a standardized approach to measurement and analysis increases the risk of bias of evidence, potentially undermining scientific progress within a rapidly expanding field of research by creating difficulty deciphering which of the apparent clinically significant results represent true effects.(13) The present analysis sought to appraise this body of primary and review literature, developing an understanding of true effects within the field using a detailed evaluation for reporting biases.

Reporting biases take many forms, each contributing to overrepresentation of “positive” findings in the published literature. Study publication bias occurs when studies with negative results are published less frequently or less rapidly than studies with positive results.(14) Selective outcome reporting occurs when multiple outcomes of potential interest are evaluated, but outcomes with positive results are selectively presented or emphasized.(13) Selective analysis reporting occurs when multiple analytic strategies are employed, but those that produce the largest effects are selectively presented. Overall, these biases result in excess statistically significant results in the published literature, threatening reproducibility of findings, promoting misappropriation of resources, and skewing study designs for assessments of interventions to reduce burnout or improve quality.(13)

METHODS

We conducted a systematic literature review and meta-analysis to provide summary estimations of the relation between provider burnout and quality of care, estimate study heterogeneity, and explore the potential of reporting bias in the field. We followed the PRISMA and MOOSE guidelines for methodology and reporting.(15, 16)

Data Sources and Searches

We searched MEDLINE, PsycINFO, Health and Psychosocial Instruments, Mental Measurements Yearbook, Embase, and Web of Science from inception through May 28, 2019, with no language restrictions. We used search terms for burnout and its subdomains (emotional exhaustion, depersonalization, and reduced personal accomplishment), healthcare providers, and markers of quality of care as shown in Appendix Tables 1–3.

Study Selection

We included all peer-reviewed publications reporting original investigations of healthcare provider burnout in relation to an assessment of quality of patient care. Providers included all paid professionals providing outpatient, pre-hospital, emergency, or inpatient care to patients of any age, including medical, surgical, and psychiatric care. We chose an inclusive method of identifying studies of burnout, considering assessments to be related to burnout if the authors defined them as such and used any inventory intended to identify burnout, either in part or in full. We similarly chose an inclusive approach to identifying quality of care metrics, including any assessment of processes or outcomes indicative of care quality. We included objectively measured and subjectively reported quality metrics, originating from the provider, from other sources within the healthcare system, or from patients and their surrogates. We considered medical malpractice allegations a subjective patient-reported quality metric. Although patient satisfaction is an important outcome, it is not consistently indicative of care quality or improved medical outcomes, suggesting that it may be related to factors outside the provider’s immediate control such as facility amenities and access to care.(17–20) Thus, for the purposes of this review we excluded metrics solely indicative of patient satisfaction to reduce bias from these non-provider related factors that may affect satisfaction.

We included peer-reviewed indexed abstracts if they reported a study population not previously or subsequently reported in a full-length manuscript. For study populations described in multiple full-length manuscripts, we included the primary result from the manuscript with the earliest publication date as the primary outcome, with any unique outcomes from subsequent manuscripts as secondary outcomes. We supplemented the database searches with manual bibliography reviews from included studies and related literature reviews.(7–9, 21–24) In line with our reporting bias aim, we did not expand our search beyond peer-reviewed publications and did not contact authors for unpublished data. If insufficient data were presented in a publication to calculate an effect size, we supplemented with data from subsequent peer-reviewed publications when available, yet retained attribution of these effect sizes to the initial report. We excluded any purely qualitative studies.

All investigators contributed to development of study inclusion and exclusion criteria. The literature review and study selection were conducted by two independent reviewers in parallel (authors DT and either AS or KA), with ambiguities and discrepancies resolved by consensus.

Data Extraction and Quality Assessment

We extracted data into a standard template reflecting publication characteristics, burnout and quality metric assessment methods, risk of bias, and strength of reported relationship. Data were extracted by two independent reviewers (DT and AS), with discrepancies resolved by consensus. We estimated effect sizes and precision using Hedges’ g and standard errors (SE), respectively. Hedges’ g estimates effect size similarly to Cohen’s d, but with a bias correction factor for small samples. In general, 0.2 indicates small effect, 0.5 indicates medium effect, and 0.8 indicates large effect.

We classified each assessment of burnout as Overall Burnout, Emotional Exhaustion, Depersonalization, or Low Personal Accomplishment. We also identified burnout assessments as Standard if defined as Emotional Exhaustion score ≥ 27 or Depersonalization score ≥ 10 on the Maslach Burnout Inventory, or as the midpoint and higher on validated single-item scales. We categorized quality metrics within five groups: Best Practices, Communication, Medical Errors, Patient Outcomes, and Quality and Safety, and reverse-coded any “high quality” metrics such that positive effect sizes indicate burnout’s relation to poor quality of care.

For publications with multiple distinct (non-overlapping) study populations reported separately, we considered each population separately for purposes of analysis. For publications with multiple outcomes for the same study population, we decided to perform analyses that use only one outcome per study, ideally the specified primary outcome. Whenever there was no clear primary outcome, we chose the first listed outcome, consistent with reporting conventions of presenting the primary outcome first. We considered other outcomes secondary, excluding them from the primary analyses to avoid bias from intercorrelation but including them in selected descriptive statistics and stratified analyses where appropriate.

Data Synthesis and Analysis

We calculated Hedges’ g from odds ratios (dichotomized data) using the transformation $\text{log}(OR)*\frac{\sqrt{3}}{\Pi }$, or from correlation coefficients (unscaled continuous data) using the transformation $\frac{2*r}{\sqrt{1-r^2}}$, both multiplied by a bias correction factor $\sqrt{\frac{N-2}{N}}$, consistent with published norms.(25, 26) Further details are provided in Appendix Methods.

The majority of studies reported burnout as a dichotomous variable or with unscaled effect size estimates, facilitating transformations as above. We scaled effect sizes accordingly for the 6 studies reporting burnout only as a continuous variable in order to maintain comparability, adapting our methods from published guidelines.(27, 28) Based on known distributions of burnout scores among providers,(29–31) we calculated the difference between the mean scores of providers with and without burnout to average 47.6% of the span of the particular burnout scale used. We thus converted effect sizes from continuous scales to the corresponding effect size reflective of a 47.6% change in scale score when needed to extrapolate to dichotomized burnout. We also performed sensitivity analyses excluding these few scaled effect sizes. More detailed description of this process is in Appendix Methods.

We initially intended to primarily perform a random effects meta-analysis including all primary (or first listed) effect sizes, with secondary meta-analyses stratified by quality metric category and by each unique burnout-quality metric combination. However, due to high heterogeneity in the pooled meta-analyses, we only report summary effects from the unique burnout-quality metric combinations. We also performed sensitivity analyses limited to studies with Standard burnout assessments and those with independently-observed or objectively-measured markers of quality of care. We used the empirical Bayes method with Knapp-Hartung modification to estimate the between-study variance tau².(32) We evaluated study heterogeneity using I². Details on this meta-analysis approach are in Appendix Methods.

We performed the Ioannidis test to evaluate for excess significance(33) by identifying the study population with the highest precision (1/SE) among those with the lowest risk of bias (studies using a full validated burnout inventory with an objective quality metric). We then calculated the power of all studies to detect the effect size of this study and compared observed vs. expected number of studies with statistically significant results using paired T-tests. Secondarily, we stratified excess significance testing by outcome category.

Because small studies may carry increased risk for bias, we performed the Egger test to evaluate for small-study effects.(34) We regressed standard normal deviate (Hedges’ g/SE) on precision (1/SE), using robust standard errors due to clustering of effect sizes at the study population level.

We used Stata 15.0 (StataCorp LLC, College Station, TX) for all analyses. All tests were two-sided. For summary effects, we considered two different thresholds of statistical significance, P < 0.05 and the newly proposed P < 0.005.(35, 36) We made no further corrections for multiple testing.

Role of the Funding Source

This study was performed in accordance with the Institutional Review Board requirements of Stanford University, and was classified as research not involving human subjects. This study was funded by the Stanford Maternal and Child Health Research Institute and the Eunice Kennedy Shriver National Institute of Child Health and Human Development. The funders of this study had no role in study design, data collection, analysis, interpretation, or writing of the report. The lead author had full access to all data in the study, and affirms that the manuscript is an honest, accurate, and transparent account of the study; that no important aspects of the study have been omitted; and that any discrepancies from the study as originally planned have been explained.

RESULTS

The search identified 11,703 citations, screening produced 313 potentially eligible publications retrieved in full text, and 120 were included, plus 3 additional publications identified by bibliography review (Figure 1). Overall, we included 123 publications from 1994 through 2019,(37–159) encompassing 142 distinct study populations detailed in Appendix Table 4. The median sample size was 376 (interquartile range 129 – 1417). The 142 study populations included physicians (N=71, 50%), nurses (N=84, 59%), and other providers (N=18, 13%), for a total of 241,553 healthcare providers evaluated. Quality metrics covered inpatient (N=122, 86%), outpatient (N=62, 44%), adult (N=134, 94%), pediatric (N=93, 65%), medical (N=135, 95%), and surgical (N=89, 63%) patients. Only 4 studies explicitly specified a primary outcome. Six studies did not provide sufficient data to derive an effect size from the original publication, but provided usable data published in a subsequent review.(39, 66, 69, 107, 115, 117) One research group reported results from a single study population in two publications—the first published effect was considered primary, with results from the later publication date considered secondary effects.(112, 160)

Figure 1.

Study selection process

Overall Burnout, Emotional Exhaustion, and Depersonalization were the primary predictors for 56, 75, and 11 study populations, respectively, from a variety of survey instruments as outlined in Appendix Table 5. The 50 distinct quality metrics included 10 Best Practices, 8 Communication, 10 Medical Errors, 10 Patient Outcomes, and 12 Quality and Safety measures (26 measured provider perception of quality, 15 used independent or objective measures of quality, and 9 included both types of assessments).

As illustrated in Figure 2, 38 (33%) of the 114 distinct burnout/quality combinations were reported 3 or more times. The most frequently reported effect related Emotional Exhaustion to Low quality of care (N = 41), with the majority of the reported effect sizes in the Quality and Safety and Medical Errors categories. Although all 5 categories of outcomes had estimates more frequently relating burnout in the direction of poor quality of care (denoted in red), 7 of the 16 estimates pointing in the opposite direction were found in the Communication category. Results were similar when limited to primary (or first listed, when primary was not specified) effect sizes only (Appendix Figure 1).

Figure 2.

Summary of all included burnout and quality metric combinations, showing frequency of effect size reporting and value of summary effect size.

Meta-analyses combining burnout and quality metrics within quality categories revealed I² values of 93.4 to 98.8% indicating extremely high heterogeneity, and therefore summary effects are provided only at the level of the 114 distinct burnout/quality combinations, 46 of which included primary effect sizes. Meta-analyses of these 46 combinations revealed 24 (52%) with statistically significant summary effects > 0 (burnout related to poor quality of care), 1 (2%) with statistically significant summary effects < 0 (burnout related to high quality of care), and 21 (46%) with no difference at the P < 0.05 threshold. When the P < 0.005 threshold was used, the respective numbers were 18 (39%), 1 (2%), and 27 (59%). Results are summarized in Table 1, and primary effect sizes from all included studies are shown in Appendix Figure 2.

Table 1.

Number and direction of summary effect sizes for each combination of burnout and quality metric. Summary effect sizes obtained via empirical Bayes meta-analysis.

	Na	P < 0.05 threshold			P < 0.005 threshold
		g > 0b	g < 0c	No effectd	g > 0b	g < 0c	No effectd
Primary effects only	46	24 (52%)	1 (2%)	21 (46%)	18 (39%)	1 (2%)	27 (59%)
Primary and secondary effects	114	58 (51%)	6 (5%)	50 (44%)	47 (41%)	6 (5%)	61 (54%)
Standard burnout definitions	24	15 (62%)	1 (4%)	8 (33%)	14 (58%)	1 (4%)	9 (38%)
Independent/Objective quality metrics	48	14 (29%)	2 (4%)	32 (67%)	9 (19%)	2 (4%)	37 (77%)

^aNumber of distinct burnout/quality combinations represented

^bSummary effect Hedges’ g > 0 indicates burnout related to poor quality of care

^cSummary effect Hedges’ g < 0 indicates burnout related to high quality of care

^dSummary effect Hedges’ g not significantly different from 0 at the specified P-value threshold

Results were similar with inclusion of secondary effect sizes. Of the 114 distinct burnout/quality combinations, 58 (51%) had statistically significant summary effects > 0, 6 (5%) had statistically significant effects < 0, and 50 (44%) showed no difference at the P < 0.05 threshold. When the P < 0.005 threshold was used, the respective numbers were 47 (41%), 6 (5%), and 61 (54%). Results from all burnout and quality metric combinations are shown in Appendix Figure 3. Our findings were similar when limited to studies explicitly using Standard burnout definitions, but the observed relationships were attenuated when limited to independent/objective quality metrics, as shown in Table 1.

The most precise study with low risk of bias(143) reported a small effect size (Hedges’ g=0.26, analogous to an odds ratio of 1.5–1.6). Using this estimate, the Ioannidis test found an excess of observed versus predicted statistically significant studies (73% observed vs 62% predicted at the 0.05 significance threshold, P = 0.011) (Table 2). When stratified by quality metric category, an excess of statistically significant studies was seen in Best Practices and Quality and Safety. Results were similar for the P < 0.005 significance threshold.

Table 2.

Predicted significance vs observed significance for primary (or first listed, when primary was not specified) effect sizes among all included studies, and stratified by quality metric category.

		P < 0.05 threshold			P < 0.005 threshold
	Number of studies	Predicted significance	Observed significance	P-value	Predicted significance	Observed significance	P-value
Full cohort	142	62%	104 (73%)	0.011	46%	96 (68%)	<0.001
Best Practices	14	12%	9 (64%)	0.001	2%	8 (57%)	0.001
Communication	5	43%	3 (60%)	0.67	40%	3 (60%)	0.63
Medical Errors	32	50%	20 (62%)	0.169	33%	15 (47%)	0.182
Patient Outcomes	17	64%	9 (53%)	NPa	54%	9 (53%)	NPa
Quality & Safety	74	65%	62 (84%)	<0.001	50%	60 (81%)	<0.001

^aNP: not pertinent, observed smaller than predicted

The Egger test did not show small-study effects (Intercept −1.32, 95% CI −3.48 to 0.85) indicating smaller studies did not systematically overestimate effect sizes (Figure 3). Funnel plot relating effect size to standard error is shown in Appendix Figure 4.

Figure 3.

Standard normal deviate (Hedges’ g / standard error) in relation to precision (1 / standard error)

DISCUSSION

This overview extends previous work in the field, by including a comprehensive evaluation for reporting biases in the healthcare provider burnout literature, encompassing 145 published study populations that quantified the relation between burnout and quality of care over a 25-year period for 241,553 healthcare professionals. The majority of evidence suggests a relationship between provider burnout and impaired quality of care, consistent with recent reviews of various dimensions.(7–10, 22) Although the effect sizes in the published literature are modestly strong, our finding of excess significance implies that the true magnitude may be smaller than reported, and the studies which attempted to lower the risk of bias demonstrate fewer significant associations compared to the full evidence base. The fact that only 4 studies specify primary outcomes further supports the possibility of reporting bias causing exaggerated effects.

From a 2015 MEDLINE, Web of Science, and CINAHL search, Salyers et al(9) reported effect sizes of r=−0.26 (Hedges’ g=0.54) and r=−0.23 (Hedges’ g=0.47) for the relationship between burnout and quality and safety outcomes, respectively. These effect sizes are somewhat larger than those observed in the present study. However, the prior meta-analysis also included markers of patient satisfaction, and only included 82 studies through March 2015. More recently, a 2017 all-language search of MEDLINE, Embase, and CINAHL by Panagioti et al.(10) identified 47 physician studies and reported a more similar summary odds ratio of 1.96 for patient safety incidents (approximate Hedges’ g=0.37). However, that review included 42,473 physicians (less than 20% of the number of subjects represented here), and did not include diverse healthcare professionals.

The observed relationships between burnout and quality of care are likely multifactorial. Individuals experiencing burnout may have less time or commitment to optimize the care of their patients, take more unnecessary risks, or lack ability to pay attention to necessary details or recognize the consequences of their actions.(71) Conversely, exposure to adverse patient events or recognition of poor quality of care may result in emotional or other psychological distress to the providers. This phenomenon is often referred to as secondary trauma, particularly in relation to sentinel events or important safety incidents, but it is plausible that it could also arise from repeated minor incidents.(161) The true effect sizes relating burnout and quality of care in both directions are important to understand, in order to make sound decisions on resource allocation and study design of interventions both to improve quality of care and to diminish burnout.

Recent concerns have arisen regarding variability in burnout assessment methods, and this variability was evident among the body of literature compiled here as well.(12) In this regard, the subset of studies in our analysis using the most widely-accepted “Standard” burnout assessment methods demonstrated similar to slightly increased frequency of significant associations when compared to the full evidence base. This finding suggests that the relationship between burnout and quality of care in the published literature is not due to suboptimal measures or variability in the definition of burnout used.

Excess significance in the published literature was specifically noted for adherence to best practice guidelines and for quality and safety metrics. Investigations of burnout in relation to these outcomes are typically retrospective studies of routinely-collected outcome metrics in existing datasets without pre-registered protocols. The relative ease of defining and evaluating many outcomes in many ways with these datasets increases risk for selective outcome reporting and selective analysis reporting, which may have contributed to excess significance. We found slightly lower effect sizes, but without excess significance for the Patient Outcomes subgroup, possibly reflecting the fact that these studies more commonly employed quality metrics with little or no flexibility in their definition and measurement (e.g. mortality or length of stay).

In direct assessment, studies employing independent or objective quality metrics demonstrated less frequent significant effects. This finding is not surprising, as prior research suggests that current methods for objectively measuring quality of care are not able to reliably identify certain events such as errors in judgment, technical procedural mistakes, or near-misses.(10, 162) Objective metrics are also costly to measure and difficult to connect to an individual due to the team-based nature of most clinical care, limiting their application to smaller studies and those in which a quality metric can be connected reliably to a provider. On the other hand, subjective quality metrics may be more sensitive and comprehensive, but more bias-prone (e.g., experiencing burnout may create recall bias). Further research is needed to determine the appropriate balance between the insensitivity of objective quality metrics and the potential for recall bias with subjective quality metrics.

Our analysis found no evidence specifically for small-study effects, in which small (more imprecise) studies report larger effects than large studies. These findings are consistent with those of prior meta-analyses, which traditionally evaluate for small-study effects as a surrogate for all forms of reporting bias.(9, 10) The discrepancy between our findings of overall excess significance without evidence of small-study effects may highlight the insensitivity of the latter test as a marker of all forms of bias. Moreover, smaller studies in this field are more likely to have objective measurements, while larger studies are more likely to have subjective measurements. This would dilute the ability of the small-study effect test to show a typical bias pattern.

Our study should be viewed in light of its design. Although most included studies are cross-sectional, observational and unable to determine the directionality of a causal relationship, longitudinal studies suggest bi-directional causality.(62, 149, 151, 152) We conducted extensive searches by two independent reviewers, but some relevant studies may be missed. Burnout has a number of important outcomes beyond effects on quality of care which were not the focus of our analysis.(2–6) Finally, excess significance may be due to genuine heterogeneity of effects across studies rather than reporting bias.(33) The effects reported here represent the results of heterogeneous studies, and so we do not report a single summary effect size. Rather, we report frequencies of significant summary effect sizes within burnout/quality combinations to provide a quantitative framework for interpretation while acknowledging that a distribution of true effect sizes is expected in this field-wide assessment, in contrast to a traditional meta-analysis.(163)

We avoided scoring quality assessments of the included studies, choosing instead to analyze key aspects of study quality as suggested by the proposed reporting guidelines for meta-analyses of observational studies.(16) Judging the quality of mostly cross-sectional observational studies is notoriously difficult and there are no widely accepted tools. Salyers et al.(9) created a 10-item tool to assess quality aspects in 82 burnout and quality of care studies and they did not identify any relationship between study quality score and effect size.

Our findings carry several important implications for future intervention trials and observational studies. For intervention trials, the potential for exaggerated published effects should be considered in power calculations to lower the risk of false negative results (Type II error). In addition, future studies should attempt to reduce the risk of reporting biases. Standardization and consensus on core outcomes may be useful for future studies if appropriate targets can be identified.(164) Such standardization can improve comparability among studies, facilitating traditional meta-analysis estimates of the relevant effect sizes. Some outcomes such as self-reported medical errors, low quality of care, and low patient safety score were particularly prevalent in the literature, suggesting that researchers already consider these outcomes either important or feasible to measure. However, if core outcomes are to be widely accepted, they must be both important and feasible to measure. Thus, in addition to this “popular vote” approach, expert consensus is needed to curate an appropriate list of core outcomes for this field. Other outcome evaluations could then be discouraged except if a unique justification is present.

Study registration may further reduce the risk of study publication bias and improve transparency surrounding unpublished studies. By registering a study publicly at its outset, researchers can reduce the likelihood that a study is conceived and conducted, but remains unpublished due to undesirable or lackluster results.(165) In a similar manner, protocol pre-specification may reduce the risk of selective outcome reporting and selective analysis reporting within published studies, allowing easier identification of any post-hoc analyses. Published analyses that deviate from the pre-specified protocol would require justification from the authors, and this approach would alert the readers that those results may be more susceptible to bias. These mechanisms are currently used rarely in any field of medicine outside of clinical trials, but could become widely adopted with sufficient advocacy by researchers, publishers, funders, and other stakeholders.

In conclusion, healthcare provider burnout is frequently associated with reduced quality of care in the published literature. However, few rigorous studies exist and the effect size may be smaller than reported, and it may be particularly smaller for objective quality measures. It is not yet known whether curtailing burnout improves quality of care and/or whether improving quality of care reduces burnout, and adequately powered and designed randomized trials (91, 166, 167) will be indispensable to answer these questions.