The Doctors’ Effect on Patients’ Physical Health Outcomes Beyond the Intervention: A Methodological Review

Abstract

Background

Previous research suggests that when a treatment is delivered, patients’ outcomes may vary systematically by medical practitioner.

Objective

To conduct a methodological review of studies reporting on the effect of doctors on patients’ physical health outcomes and to provide recommendations on how this effect could be measured and reported in a consistent and appropriate way.

Methods

The data source was 79 included studies and randomized controlled trials from a systematic review of doctors’ effects on patients’ physical health. We qualitatively assessed the studies and summarized how the doctors’ effect was measured and reported.

Results

The doctors’ effects on patients’ physical health outcomes were reported as fixed effects, identifying high and low outliers, or random effects, which estimate the variation in patient health outcomes due to the doctor after accounting for all available variables via the intra-class correlation coefficient. Multivariable multilevel regression is commonly used to adjust for patient risk, doctor experience and other demographics, and also to account for the clustering effect of hospitals in estimating both fixed and random effects.

Conclusion

This methodological review identified inconsistencies in how the doctor’s effect on patients’ physical health outcomes is measured and reported. For grading doctors from worst to best performances and estimating random effects, specific recommendations are given along with the specific data points to report.

Video abstract

Point your SmartPhone at the code above. If you have a QR code reader the video abstract will appear. Or use:

https://youtu.be/rvHjVIEPVhI

Introduction

A fundamental question in medical research is whether medical practitioners have an effect on patients’ health beyond the intervention, patient risk, and hospital variables. Previous research has revealed that when a treatment is delivered by a doctor (ie surgeon or medical physician), patient outcomes may vary systematically by medical practitioner.^1,2 It is well known that hospitals can have an influence on patients’ health outcomes, with wide variation between hospitals.^3–7 Such outcomes include adverse events,⁴ prescribing errors,⁴ hospital readmission,^5,6 and mortality.^7–9 Comparing hospitals requires a sound methodology and reliable estimates that take into account the multiple variables involved.^8,10 In contrast to the substantial research on hospital effects, there is minimal research on the effect of doctors.

The influence of doctor-patient communication has been investigated as a “doctor effect” on patients’ health outcomes,^1,11,12 including symptoms,^13,14 readmission rates in the emergency department,^13,15 health-related quality of life,¹⁶ and improved diabetes control.¹⁷

Research on the therapist effect in psychotherapy has shown significant effects of therapists on patient outcomes beyond the therapy technique or modality applied.^18,19 This wide variation among practitioners has been acknowledged and incorporated into the training material for psychotherapists.^20,21 In surgery, outcomes associated with procedure volume, seniority, level of experience, or doctor specialty, include mortality rate,²² length of hospital stay,^23,24 postoperative complications,²⁵ and readmission.^26,27 While research on the doctors’ effect in non-surgical specialties is limited, there is evidence from studies in primary care,^1,28 intensive care,²⁹ acute care,³⁰ and obstetrics,³¹ where medical practitioners had an effect on patients’ health outcomes.

Given the significant therapist effect in psychotherapy, and the known wide variation in patient outcomes across hospitals, but unclear effect of individual doctors on patient outcomes, we conducted a systematic review of the effect of doctors on patients’ physical outcomes. We aimed to assess whether doctor effects vary with specificity, outcome and intervention. However, in conducting the review, we found substantial variation in the way a doctor effect is measured and reported, therefore making data synthesis challenging and meta-analysis impossible. This has led to the present study where we have conducted a methodological review of studies that measure and report on doctors’ effect on physical patient outcomes. The focus of the methodological review is on the method of measurement of the doctors’ effect as well as how it is reported. The data source for the review is the included studies from our systematic review.³²

Objective

To conduct a methodological review of studies reporting on the effect of doctors on patients’ physical health outcomes and to provide recommendations on how this effect could be measured and reported in a consistent and appropriate way.

Materials and Methods

Design

The present study is a methodological review where the focus is on statistical analysis and reporting.³³ The search strategy, data collection, and extraction are explained in detail in a previous report of a systematic review of the surgeons’ effect on patients’ physical health outcomes.³²

Search Strategy

Three databases were searched initially: PubMed, Embase, and PsycINFO; and over 10,000 publications were screened. For each of the studies identified that met the inclusion criteria, a citation analysis on Scopus was conducted to identify further eligible studies. The full search strategy and keywords can be found in the Supplementary Material.

Study Selection and Eligibility Criteria

The studies selected in the initial electronic search and the studies added through the citation analysis were independently reviewed by two researchers with a third reviewer acting as an arbitrator if required. This process resulted in 79 included studies, all of which are included in the present study. Any physical patient health-related outcome was eligible for inclusion. Studies that fulfilled any of the following criteria were excluded: (1) studies that only described a doctors’ effect on particular doctor-related variables (such as specialty of doctor), (2) studies with fewer than 15 doctors, (3) cross-sectional studies, and (4) studies that mention fixed or random effects but did not list them either graphically or in numerical form.

Data Extraction and Quality Assessment

CS extracted the relevant information for assessing doctor effects from each included study, and the extracted data was then reviewed by a second researcher. The data items extracted can be found in Table 1. For quality assessment, the Newcastle-Ottawa Scale (NOS) was used, with the majority of studies scoring between 8 and 9 (9 being the maximum total).^34–36

Table 1.

Data Items Extracted

Data Item	Comment
Publication	First author, year
Surgeon or Other Medical Specialty	Surgeon, Other
Practitioner Type	Surgeon, GP, Cardiologist, etc.
Medical Specialty of Doctor
Detailed Intervention
General Outcome
Specific Outcome	Often same as General Outcome
Type of Study	Cohort or Randomized Controlled Trial
Newcastle Ottawa Scale Score	0–9
Count of Doctors in Study
Count of Patients
Count of Institutions
Doctor ICC	Intra-class correlation coefficient, here a measure of the strength of the effect on patients’ physical health
Multivariate Data Analysis used	Y/N
Percentage of Doctors that are Outliers	Positive and Negative Outliers
Country of dataset analyzed

Methodological Review

We planned to describe the methods used to estimate and report the doctors’ effect on patients’ physical outcomes including the statistical model used, types of confounding variables adjusted for (patient variables, hospital/institution variables, doctor variables), and the method of reporting the doctor effect.

Results

Of the 79 included studies, 62 used a multivariable multilevel regression model to estimate the doctors’ effect, 72 studies included patient variables in their model, 41 studies included hospital or institution variables in the model, 60 studies included doctors’ volume, and 24 studies included other doctor variables. There were two different ways that the doctors’ effect was reported: fixed effects and random effects,^37,38 with 54 studies reporting fixed effects and 34 studies reporting random effects.

Table 2 provides details for each included study, presenting in part the wide variety of statistical methods used.

Table 2.

Detailed Results for Each Study

Publication	Doctors	Patients/Procedures	Institutions	ICC%	Neg Outlier%	Pos Outlier%	Country	MLR*	MV**	Statistical Analysis	PV^	HV^^	DVo#	ODV##	Confidence Interval Calculation
Anderson, 201622	NS	2880	35		Other	Other	US	Y		“Gaussian Kernel Densities were constructed to show the relative distributions of the effects of individual institutions and surgeons”	Y	Y	Y	N	None
Aquina, 201551 pg e163	NS	158,596	NS		Other	Other	US	Y		“Mixed Effects Multivariable Logistic Regression”, conference abstract	Y	Y	Y	N	95% CI given, but not method
Aquina, 201552	223	14,875	99		13.0	28.0	US	Y		“Bivariate and hierarchical logistic regression with further multivariable analysis” R 3.1 SAS 9.3	Y	Y	Y	N	95% CI given, but not method
Aquina, 201653	3481	125,160	210	24.3	Other	Other	US	Y		“Three-level mixed-effects logistic regression analyses were performed” R 3.2.0 SAS 9.4	Y	Y	Y	Y	None
Aquina, 201754	1572/2012	124,416/78,267	260/256	40.5/14			US	Y		“Mixed-effects Cox proportional hazards analyses” R 3.2.1 SAS 9.4	Y	Y	Y	Y	95% CI given, but not method
Arvidsson, 200555	25	1068	7		Other	Other	Sweden	N	Y	SAS 8.2 NL Mixed model	Y	Y	N	N	None
Becerra, 201756	1503/814	12,332	187	7.9			US	Y		“Multilevel logistic regression”, “multilevel competing-risks Cox models” SAS 9.3 R	Y	Y	Y	Y	95% CI given, but not method
Beckett, 201830	22	21,570	1				UK	N	N	Analysis based on r-square	N	N	Y	N	None
Begg, 200257	159	10,737	72		8/13/9	3/14/3	US	Y		Correlation-adjusted and GEE logistic regression	Y	Y	Y	N	None
Bianco, 200558	159	5238	NS		7.5	2.5	US	Y		Logistic regression, binomial distribution, histograms, extra-binomial variation	Y	Y	Y	N	None
Bianco, 201059	54	7725	4		9.3	13.0	US	Y		“[M]ultivariable, parametric random-effects regression survival-time model, using a log-logistic survival distribution to model hazard over time” Stata 9.2	Y	N	Y	N	95% CI given, but not method
Bolling, 201060	1088	28,507	639		6.6	7.4	US	Y		GEE logistic regression SAS 9.2 GENMOD	Y	N	Y	N	Funnel plot with 95% CI
Bridgewater, 200361	23	8572	4		0.0	0.0	UK	Y		Unspecified, using SAS	Y	N	Y	N	95% CI given, but not method
Bridgewater, 200562	25	1097/9066	4		0.0	0.0	UK	Y		Unspecified, using SAS	Y	N	Y	N	Clopper-Pearson 95% CI
Brown, 201663	133	14,033	84		6.0	6.8	US	Y		Bayesian hierarchical logistic regression	Y	N	Y	Y	95% CI given, but not method, f did better
Burns, 201164	1557	246,469	156		0.7	4.5	UK	N	Y	Logistic regression	Y	Y	Y	N	“We constructed funnel plots using exact Poisson control limits by means of the web tool available at www.erpho.org.uk/topics/tools/funnel.aspx.”
Cirillo, 202065	32	19,824	1		3.1	0.0	Italy	Y		Logistic regression, random effects meta-analysis	Y	N	Y	N	95% CI given, but not method
Cromwell, 201366	490	1194	126/129		0.0	0.0	UK	N	Y	Stata Funnel plot, Wilcoxon extended by Cuzick	Y	Y	Y	N	Binomial distribution 95% CI
Dagenais, 201967	19	1461	1	14.4	10.5	10.5	US	Y		Hierarchical logistic regression	Y	N	Y	Y	95% CI given, but not method
Davenport, 202068	55	25,596	1				US	Y		SAS 9.4 inference testing	Y	N	Y	Y	95% CI given, but not method, though not relevant for mortality
Duclos, 201269	28	2357/2904	5	10/32			France	Y		Mixed effects logistic regression	Y	Y	Y	Y	Binomial distribution 95% CI
Eastham, 200370	44	4629	2		Other	Other	US	Y		Logistic mixed model	Y	Y	Y	N	None
Eijkenaar, 201371	447/537	26,684/37,832	N/A	2.5/0.6			Netherlands	Y		Generalized Linear Multilevel Models using SAS 9.2 GLIMMIX	Y	N/A	Y	N	95% CI given, but not method
Eklund, 200972	48	1275	>1		2.1		Sweden	N	Y	RCT Pearson Chi2, Fisher’s exact, Cox regression, “Z-test for heterogeneity”	Y	N	Y	Y	None
Faschinger, 201173	17	36,329	1		Other	Other	Austria	N	Not specified	Correlations calculated	Y	N/A	Y	N	None
Fountain, 200474	43	876/504	28	7.4			UK	Y		SAS NLMIXED, dealing with convergence issues	Y	N	N	N	Standard error calculated
Gani, 201526	56	22,559	1	2.8			US	Y		“[M]ultilevel multivariable logistic regression” Stata 12.1 GLLAM	Y	N/A	Y	Y	95% CI given, but not method
Glance, 200675	138	51,750	33	5.9	5.1	8.7	US	Y		Stata 8.2 SAS GLIMMIX	Y	Y	Y	Y	“Quality outliers were identified using 1) the ratio of observed-to-expected mortality rates (O/E ratio) and confidence intervals (CIs) calculated using both parametric (Poisson distribution) and nonparametric (bootstrapping) techniques; and 2) shrinkage estimators.”
Glance, 201645	420/241	55,436	40	0.5/1.8	0.0/3.3	0.0/1.7	US	Y		Hierarchical logistic regression	Y	Y	N	N	95% CI given, but not method
Goodwin, 201342	1099	131,710	268	0.75	0.6	1.5	US	Y		“[H]ierarchical general linear model”	Y	Y	N	N	95% CI given, but not method
Gossl, 201376	21	8187	3		0.0	4.8	US	N	Y	Logistic regression	Y	N/A	N	N	Deviation from normal distribution
Grant, 200877	31	14,637	4		0.0	0.0	UK	N	Y	SAS 8.2 Logistic regression	Y	N	Y	N	95% CI given, but not method
Gutacker, 201843	212–3760	24,505–405,671	30–152	0.4–12.7			UK	Y		“Three-level hierarchical generalised linear mixed models”	Y	Y	Y	N	None
Hannan, 201778	403	27,560	60	12.0	18.6	12.7	US	Y		Hierarchical logistic regression	Y	N	Y	Y	95% CI given, but not method
Harley, 200531	143	NS	Multiple		6.3	2.1	UK	N	Y	Multivariate Analysis	N	N	N	N	95% CI given, but not method
Healy, 201725	97	3118/2078	46		10.3/9.3	7.2/4.1	US	Y		“Multi-level mixed-effects logistic regression” Stata 13	Y	N	Y	N	95% CI given, but not method
Hermanek, 199979	43	1121	7		9.3	16.3	Germany	N	Y	“Multiple logistic regression analyses”	N	N	N	N	95% CI given, but not method
Hermann, 200280	20	16,443	1		Other	Other	Austria	N	N	Chi-square, Brandt and Snedecor contingency tables for binomial distributions	N	N	N	N	None
Hofer, 199981	232	3642	3	1.0			US	Y		“hierarchical regression for general linear models”	Y	N	N/A	N	None
Hoffman, 201782	1128	183,283	601	6.2			US	Y		“Generalized linear mixed effects models”	Y	Y	Y	Y	Conference abstract ICC CI not specified how
Holmboe, 201083	236	22,526	13 states	12.0			US	Y		SAS 9.1.3 NLMIXED	Y	N	Y	Y	Delta method for 95% CI
Huesch, 200984	398	221,327	75		1.2	Other	US	N	Y	Using SEMA by SEMATECH	Y	Y	N	N	Binomial distribution 95% CI
Hyder, 201385	575	1488	298	0.3			US	Y		Multilevel Models SAS 9.3	Y	Y	Y	N	95% CI given, but not method
Jemt, 201686	23	8808	1		8.7	Other	Sweden	N	N	Chi-square	N	N/A	N	N	None
Johnston, 201087	404	55,515	12		Other	Other	UK	N	N	Funnel plots	N	N	Y	N	None
Justiniano, 201988	345	1251	118		Other	Other	US	Y		Bayesian hierarchical regression	Y	Y	Y	Y	95% CI given, but not method
Kaczmarski, 201989	5337	291,065	NS		17.5	3.7	US	Y		Hierarchical logistic regression SAS 7.1	Y	N	Y	Y	95% CI given, but not method
Kaplan, 200990	210	7574		33.0/30.6	27.6	43.8	US	Y		Binary mixed models SAS NLMIXED	Y	N	N	N	Standard error calculated
Kerlin, 201891	345	11,268	104	1.8	22.9	25.2	US	Y		Bayesian hierarchical regression Stata 14.2	Y	Y	Y	Y	Bayesian 95% credible intervals of odds ratios
Kissenberth, 201892	57	1703	NS	44.0			US	N	Y	Linear regression	Y	N	N	N	Conference abstract, no CI
Krein, 200293	258	12,110	9	1.0			US	Y		Multilevel analysis MLwiN 2000	Y	Y	N	N	None
Kunadian, 200940	261	149,888	48		1.6	1.1	US	Y		Multivariate Logistic Regression	Y	Y	Y	N	Binomial distribution 95% CI
Landercasper, 201994	71	3954	NS		5.7	4.3	US	Y		Mixed effects multivariate model SAS 9.4	Y	N	Y	Y	95% CI given, but not method
LaPar, 201495	93	4194	17		Other	Other	US	N	Y	[M]ultivariable, mortality risk-adjusted models with restricted cubic splines	Y	Y	Y	N	None
Likosky, 201296	32	11,838	8		Other	Other	US	N	Y	Multivariate Logistic Regression	Y	N	N	N	None
Luan, 201997	38	1277	21		2.6	15.8	US	Y		Multivariate Mixed Effects Logistic regression Stata 15	Y	Y	Y	N	Bonferroni corrected 95% CI, no further details
Martin, 201398	298	6091	43	2.5	Graph too small	Graph too small	US	Y		Logistic regression	Y	Y	Y	N	Bayesian 95% coverage intervals, surgeon performance assumed normally distributed
McCahill, 201299	54	2206	4		11.1	31.5	US	Y		Logistic regression	Y	Y	Y	N	95% CI given, but not method
Navar-Boggan, 2012100	47	5979	1		6.4	12.8	US	Y		“Multilevel multivariable random-effects logistic regression” Stata 9	Y	N/A	Y	Y	95% CI given, but not method
O’Connor, 2008101	120	2589	18	0.8			US	Y		“Multivariate hierarchical models” MLwiN	Y	Y	Y	Y	None
Orueta, 2015102	1479	2,207,175	130	4.2			Spain	Y		“Four-level mixed effect models” inc district SAS 9.2 GLIMMIX	Y	Y	Y	Y	95% CI given, but not method
Papachristofi, 2014103	24/18	18,426	1	0.1/2.8	0.0/16.7	0.0/0.0	UK	Y		“Logistic random effects regression” with random effects	Y	N/A	Y	N	95% CI given, but not method
Papachristofi, 201639	190/127	110,769	10	0.25/4.0	0.0/15.0	0.0/6.3	UK	Y		“[L]ogistic random-effects regression analysis” using R 3.01	Y	Y	Y	N	95% CI given, but not method for practitioners, comment why no 95% CI for ICC
Papachristofi, 201723	190/127	107,038	10	0.19/2.8	2.1/11.8	0.5/14.2	UK	Y		“Logistic mixed effects models” using R 3.2.2	Y	Y	Y	N	95% CI given, but not method
Quinn, 2018104	2724	123,141	51	2.2	0.2	0.2	US	Y		“3-level crossed random effects logistic regression models” Stata MP 14.2, SAS 9.4	Y	Y	Y	N	“Ninety-five percent CIs were calculated according to Agency for Healthcare Research and Quality methods for risk-adjusted rates.”
Rudmik, 2017105	43	2168	Multiple		16.3	4.7	Canada	Y		Logistic regression	Y	N	Y	N	Binomial distribution 95% CI
Selby, 2010106	1005/1,049	169,156/232,053	35	1.9/1.9			US	Y		“Multilevel linear and logistic regression”	Y	Y	N	N	Standard error calculated
Shih, 2015107	345	5033	24	14.0			US	Y		“Hierarchical logistic regression”, Stata 12.0	Y	N	Y	N	None
Singh, 201515	525	48,883	143	15.0	12.8	12.5	US	Y		“[M]ultilevel, multi-variable models”	Y	Y	Y	Y	95% CI given, but not method
Singh, 201824	3987	39,884	NS		10.0/0.1	7.2/0.0	US	Y		Mixed models, SAS GLMM	Y	Y	Y	Y	95% CI given, but not method
Singh, 201928	4230	565,579			0.0	0.1	US	Y		“Multilevel logistic regression” SAS 9.4 GENMOD, GLIMMIX, Stata 15.1 margins	Y	N	Y	N	Formula for 95% CI given and bootstrapping
Thigpen, 201827	34	995	1		5.9	8.8	US	N	Y	“Linear regression model”	Y	N	Y	N	Efron’s bootstrap for 95% CI
Tuerk, 2008108	42	1381	1	2.0			US	Y		“Hierarchical linear models” HLM6	Y	N	N	N	ICC as per Bryk Raudenbusch, 95% CI not calculated
Udyavar, 2018109	2149	569,767	224	2.3			US	Y		“Multilevel random effects modelling” Stata 14 MELOGIT	Y	Y	Y	Y	95% CI given, but not method
Udyavar, 2018110	175	65,706	31	8.7			US	Y		“[M]ultilevel random effects models” Stata 14	Y	Y	Y	Y	ICC 95% CI not calculated
Udyavar, 2019111	5816	215,745	198	27.3			US	Y		“[M]ultilevel mixed effects modeling”	Y	Y	Y	Y	Odds ratio 95% CI given, but not method
Verma, 2020112	135	103,085	7		18.5	14.8	Canada	Y		Six different multivariable regression analyses R 3.5	Y	Y	Y	N	95% CI given, but not method
Xu, 2016113	276	2525	44		3.3	0.0	US	Y		“Logistic regression and post-estimation”	Y	Y	Y	Y	None
Xu, 2019114	14,598	1,884,842			Other	Other	US	Y		“Multivariable logistic regressions” Stata MP 14	Y	N	Y	Y	95% CI given, but not method

Abbreviations: *MLR, Multi-level regression; **MV, If no MLR, was multivariate regression used? ^{^}PV, Patient variables; ^{^^}HV, Hospital variables; ^#DVo, Doctors’ volume of procedures used; ^##ODV, Other doctor variables than volume used.

Fixed Effects – Grading Doctors by Their Effect

Fixed effects are represented by the range of patient outcomes that doctors are responsible for after all available confounding variables have been accounted for. They are shown visually using a caterpillar plot, which ranks doctors by outcomes from lowest to highest, or a funnel plot, which shows each doctor as a dot and indicates whether doctors are outside a 95% or 99% confidence interval. For example, Papachristofi et al³⁹ showed caterpillar graphs with an ICC of 4.0% (surgeons) and an ICC of 0.25% (anesthetists) (Figure 1), while Kunadian et al⁴⁰ showed a funnel plot with an ICC of 6.5% (Figure 2), redone at a higher resolution by the authors (Figure 3) and the same data as a caterpillar plot (Figure 4). Measuring fixed effects allows identification of high and low outliers and how heterogeneously doctors perform. They also show whether variation in performance is consistent with chance or whether the variation is more significant than that. Fixed effects are calculated through “modelling fixed provider effects”.⁴¹

Figure 1.

Estimated probability of in-hospital death within three months of surgery for a patient with average Euro-SCORE risk: (a) surgeons adjusted for centre and anaesthetist; (c) anaesthetists adjusted for centre and surgeon. The horizontal line is average probability (1.8%) for the study cohort. Error bars = 95% CI.

Notes: Reproduced from: Papachristofi O, Sharples LD, Mackay JH, Nashef SAM, Fletcher SN, Klein AA. The contribution of the anaesthetist to risk-adjusted mortality after cardiac surgery. Anaesthesia. 2016;71(2):138–146. doi:10.1111/anae.13291.³⁹ © 2015 The Authors. Anaesthesia published by John Wiley & Sons Ltd on behalf of Association of Anaesthetists of Great Britain and Ireland. Creative Commons CC BY (https://creativecommons.org/about/cclicenses/).

Figure 2.

A funnel plot with each cardiologist represented by a black dot with 95% and 99% confidence intervals. The grey horizontal line is the average mortality for percutaneous coronary intervention (PCI) in New York State 2002–2004.

Notes: Reproduced/used with permission of John Wiley & Sons - Books, from: Kunadian B, Dunning J, Roberts AP, Morley R, de Belder MA. Funnel plots for comparing the performance of PCI performing hospitals and cardiologists: demonstration of utility using the New York hospital mortality data. Catheter Cardiovasc Interv. 2009;73(5):589–94. doi:10.1002/ccd.21893.⁴⁰ Copyright © 2009 Wiley‐Liss, Inc. Permission conveyed through Copyright Clearance Center, Inc.

Figure 3.

This figure was created by the authors and is a higher resolution version of Figure 2 using the same data. It is a funnel plot with each cardiologist represented by a dot with 95% and 99% confidence intervals. Cardiologists whose mortality confidence interval is above the 95% line are marked in red, those below marked in green.

Notes: Adapated/used with permission of John Wiley & Sons - Books, from: Kunadian B, Dunning J, Roberts AP, Morley R, de Belder MA. Funnel plots for comparing the performance of PCI performing hospitals and cardiologists: demonstration of utility using the New York hospital mortality data. Catheter Cardiovasc Interv. 2009;73(5):589–94. doi:10.1002/ccd.21893.⁴⁰ Copyright © 2009 Wiley‐Liss, Inc. Permission conveyed through Copyright Clearance Center, Inc.

Figure 4.

A caterpillar plot created by the authors. It uses the same data as Figures 2 and 3. Beige (on left) and brown (on right) confidence intervals have an upper limit above 10%. Green confidence intervals are wholly below average mortality, red confidence intervals wholly above.

Notes: Data from this publicly available source¹¹⁷ which is the same one as used by Kunadian et al.⁴⁰

Random Effects – Estimating the Variation Due to the Doctor

Random effects represent a percentage of the total variation in outcomes between patients that the doctors are responsible for. They are estimated via the intra-class correlation coefficient (ICC), which is the proportion of the total variation in the patient outcome attributed to doctors. There are many different ways to describe this effect.³⁷ The ICCs measured and reported in the studies ranged from 0% to 47% with a median of 3%.

Discussion

This methodological review of studies that report a doctors’ effect on a patient's physical outcomes has identified wide variations in how researchers measure and report a doctors’ effect. However, there were 2 broad methods identified: fixed effects that allow doctors to be ranked; and random effects where the proportion of variance attributed to unexplained differences between doctors is estimated. The most common statistical model used in the analyses was a multivariable multilevel regression where the types of confounding variables adjusted for included those assessing patient risk, known doctor attributes, and, to a lesser degree, differences between hospitals or institutions.

Glance et al³⁸ discuss in some detail three approaches of provider profiling for binary outcomes, namely conventional logistic regression, hierarchical logistic regression, and fixed-effects logistic regression. They conclude that hierarchical logistic regression is generally preferred, except in the case where providers have low case volume, where hierarchical logistic regression understates the provider effect. We agree that hierarchical logistic regression is an acceptable method for provider profiling as it allows measurement of the strength of the providers’ effect on physical patient health.

This review identified substantial heterogeneity in how the percentage of the variation due to the doctors is reported. For example, Goodwin et al⁴² reported the percentage of the variation for the null model as the “ICC” and the variation calculated after taking all available information into account as “partitioned variance”. It is helpful to calculate the variation of the null model as, if there is negligible or no variation, there is no need to include additional levels in the analysis. In both cases, the null and adjusted models, the ICC was calculated. In contrast, Gutacker et al⁴³ referred to the random effect measure as the “variance partition coefficient”.

A crucial element of reporting fixed effects is the calculation of the confidence intervals of each doctors’ performance. Glance et al^38,44,45 provide a detailed technical discussion of the respective advantages of using fixed (grading doctors from worst to best) and random effects (calculating the percentage of outcome variation due to the doctor). One pertinent issue discussed is that the smaller the cluster is, ie the fewer patients the doctor has, the greater the shrinkage towards the mean,^46,47 reducing the calculated ICC, and leading to an underestimate of the difference in performance between doctors.

Interpreting the Doctor’s Effect

The clinical importance of the findings from the studies assessed in this methodological review depends on how common the outcome assessed is and how varied the doctors’ effect is among practitioners. The more common and the more varied, the more the finding is clinically important. The choice between a doctor with an above or below average effect will have implications for the patients’ health outcomes at different levels of how common the outcome is and how strong the doctors’ effect is. The stronger the doctors' effect and the more important the outcome, the more the choice of doctor matters for the individual. The more common the outcome is, the more the choice of doctor matters for population health.

Table 3 by Baldwin et al,²¹ originally from Wampold et al,⁴⁸ and augmented by Kraemer et al,⁴⁹ shows effect sizes for different ICCs. The intra-class correlation coefficient (ICC) can measure the percentage of the variation in patients’ physical health outcomes due to each component of a medical interaction,²¹ which is typically the patient, the doctor, the hospital, and the intervention. Table 3 shows a scenario where 50% of the patients recover from an intervention when there is no doctors’ effect, ie for an ICC of zero. However, an ICC of 5.9% is reported to produce a medium-sized effect (Cohen’s d) with a Number Needed to Treat (NNT) of 3.6. Under such circumstances, an ICC of 5.9% can mean that doctors have a clinically significant effect that is greater than many interventions.

Table 3.

Relationship Between ICC, Cohen’s d, Success Rates and NNT

ICC	Cohen’s d50	Proportion of Untreated Controls Below Mean of Treated Persons	Success Rate of Untreated Persons	Success Rate of Treated Persons	NNT – Numbers Needed to Treat49
Small
0.0%	0.0	0.500	0.500	0.500	∞
0.2%	0.1	0.540	0.475	0.525	17.7
1.0%	0.2	0.579	0.450	0.550	8.9
Medium
2.2%	0.3	0.618	0.426	0.574	6.0
3.8%	0.4	0.655	0.402	0.598	4.5
5.9%	0.5	0.691	0.379	0.621	3.6
8.3%	0.6	0.726	0.356	0.644	3.0
10.9%	0.7	0.758	0.335	0.665	2.6
Large
13.8%	0.8	0.788	0.314	0.686	2.3

Notes: Cohen’s d’s aim is to describe the magnitude of response to treatments between two groups, for example, a treatment and a control group. More technically, “The difference between the Treatment and Control group means, divided by the within-group standard deviation”.⁵⁰ The Number Needed to Treat (NNT) is defined as the number of patients one would expect to treat with Treatment to have one more success (or one less failure) than if the same number were treated with Control.⁴⁹

Recommendations

How to Report a Doctors’ Effect

If researchers wish to report a doctors’ effect that has been estimated, we recommend the following:

• Including “doctors” effect’ or “physicians” effect’ in the keywords and optionally in the title or abstract

• Using multivariable multilevel regression for the analysis with adjustments for patient risk, doctor experience, hospital effects, and any other potential confounding variable

• For describing fixed effects – grading doctors from worst to best, showing individual results for each doctor in a Table or a Figure

• For describing random effects, calculation of the intra-class correlation coefficient (ICC), describing the variation with 95% confidence interval and whether the outcome is a binary or continuous variable

• Making the dataset used for the analysis available for other researchers to conduct their own analyses.

What to Report

Observational Studies

We recommend reporting doctor effects after all available confounding variables have been taken into account, either by (a) the percentage of variation in the patient outcomes which is attributed to the doctor but unexplained by known attributes such as their experience, or (b) the ordering of doctors by their patients’ physical health outcomes, or (c) ideally both.

Reporting this data offers the potential to identify both low and high outliers among doctors, as well as how much of an unexplained doctors’ effect there is on patient outcomes.

Data Points to Report

Table 4 lists the data points that are recommended to report. Table 5 shows a specific example of those reported data points employing the dataset used in Kunadian et al.⁴⁰

Table 4.

Data Points to Report

Data Points to Report	Description
1. Intervention	Type of intervention
2. Type of study	We do not recommend using cross-sectional studies (surveys), as response rates can introduce a selection bias. This does not concern patient-reported data recorded by the doctor, like levels of pain or mobility.
3. Count of doctors	Count of doctors overall. For randomized controlled trials, the count of doctors for each arm.
4. Count of patients or procedures	If available, both patients and procedures.
	Randomized controlled trials: In addition to the above: number of arms to the study. If the trial is not too large, a matrix showing how many patients of each arm were served by each doctor.
5. Count of higher aggregation, if any – hospital, practices, counties, states	If there are more than two levels, ie not just patients/procedures and doctors, but also hospital, or medical practice, or county, or state, reporting their number could be useful. As there is a well-known hospital effect, distinguishing between hospital and doctors’ effects will be useful.
6. Outcome type	The patients’ physical health outcomes measured, for example mortality, length of stay, complications, pregnancies, blood pressure or HbA1c levels under control/ not under control. Definitions for each outcome. For example, with mortality, whether it is in-hospital, 30-days, or five years. Whether the outcome is binary, ordinal, or continuous. If feasible, all 30-days, in-hospital, and longer times, if they are available.
7. Percentage of patients/procedures with this outcome	For binary outcomes, the percentage of patients by doctor with that outcome – lowest percentage, highest, mean, and median. For ordinal or continuous outcomes, lowest, highest, average, mean, and median outcome by doctor.
8. Multivariate analysis (Y/N)	Has there been a multivariate analysis, and which variables were considered for exclusion in the analysis, and which were included in the final analysis?
9. Volume effect Y/N/NS (NS=’not stated’)	Was the number of patients/procedures per doctor included in the analysis? Was the effect, if any, reported as being substantial, not substantial, or not stated?
10. Observed vs expected recorded Y/N/NS	Were investigations done to identify low and high outliers among the doctors, and their count, or proportion recorded? Were funnel plot(s) provided, pointing out 95% and optionally, 90% and or 99% outliers? Alternatively, a caterpillar plot, ie a fixed effect chart showing the patient outcome for each doctor, together with the individual doctor’s 95% CI, sorted by patient outcome, showing outliers among doctors.
	Confidence interval options:115 Binomial (normal distribution in patient outcomes) Delta method – what are the details, and how is it done? Other – bootstrap, simulation116
11. Percentage variation number/NS	The variation due to the doctors in the patients’ physical health outcome as a percentage of the total variance of all investigated levels, with 95% confidence levels. Optionally, absolute variance and total variance as well.
12. ICC calculated during multilevel, multivariate analysis	As the percentage of the total variance of all investigated levels is the definition of the ICC, reporting of the ICC (intra-class correlation coefficient) as such with 95% confidence intervals as a more detailed alternative to reporting only the variation.
13. Pre-shrinkage ICC calculated through simulation	The ICC calculated in multilevel analysis is often reported as lower than it really is due to shrinkage.46,47 In order to find the pre-shrinkage ICC, the following approach can be taken: Simulated datasets that have the same distribution as the doctor/patient clusters in the data investigated can be generated using increasing ICCs until a generating ICC is found that has the same post-shrinkage ICC as the dataset investigated. Reporting this pre-shrinkage ICC can be valuable, as it can be much larger than the post-shrinkage ICC when, for example, the patients’ physical effect is not common (under 10%).

Table 5.

Data Points Reported by Kunadian et al

Data Points to Report	Kunadian et al:40 an Example*
1. Type of intervention	Percutaneous Coronary Interventions (PCI) in New York State 2002–2004, also known as angioplasty.
2. Type of study	Cohort study from medical records.
3. Count of doctors	261
4. Count of patients or procedures	149,888 patients, procedures not stated.
5. Count of higher aggregation, if any – hospital, practices, counties, states	48 hospitals
6. Outcome type	30-day and 3-year mortality following PCI.
7. Percentage of patients/procedures with this outcome	Overall, 944 deaths out of 149,888 PCI procedures. After excluding patients listed as “All Other doctors in this hospital”, 912 deaths in 146,781 procedures.
8. Multivariate analysis (Y/N)	Yes. Risk-adjusted mortality rate.
9. Volume effect Y/N/NS (NS=’not stated’)	Yes. Neither the downloadable paper nor Kunadian state whether there is a volume effect for cardiologists. Kunadian states there is no significant relationship between hospital volume and risk of in-hospital death from these data.
10. Observed vs expected recorded Y/N/NS	Yes.
Were funnel plot(s) provided?	Yes, provided in Kunadian as Figure 2.
Were caterpillar plots provided?	Not by Kunadian et al40. See Figure 4 as provided by authors.
Were confidence intervals calculated?	Neither the downloadable document nor Kunadian state how the confidence interval was calculated.
11. Percentage variation Number/NS	NS
12. ICC calculated during multilevel, multivariate analysis	ICC was calculated by the authors of this paper to be 6.54%, 95% CI (4.32%, 9.79%).
13. Pre-shrinkage ICC calculated through simulation	Using simulated data with the same number of doctors, cases per doctor, and deaths per doctor, resulted in an average ICC of 6.48%, 95% CI (4.47%, 9.32%) after 550 simulations. Therefore, there is no substantial shrinkage at work, which is not unexpected as the mean number of cases per doctor is high at 558.

Notes: *Kunadian et al's 2009 paper⁴⁰ refers to a version of the original dataset¹¹⁷ that can be freely downloaded and is sufficiently detailed for our purposes.

Strengths and Limitations

This is the first methodological review on the reporting of doctors’ effect on patient outcomes. The clarity and simplicity of how doctors’ and surgeons’ effects are described here and the suggested standardization of such reporting should allow meta-analysis to be conducted, allow robust identification of outliers, and make the re-analysis of much existing data feasible. However, a limitation is that, as all of the included studies were conducted in North America or Europe, it is unclear whether the findings can be generalized to other regions, particularly in developing nations.

Conclusion

A doctors’ effect on patients’ physical health can be measured and reported in two ways:

Firstly, by calculating the percentage of variation in patients’ physical health outcomes due to the doctor in the form of the intra-class correlation coefficient (ICC). Secondly, by grading doctors from best to worst patients’ physical health outcomes, assigning a confidence interval to those outcomes, and reporting how many doctors’ confidence intervals fall wholly above or below the overall average. Ideally, both should be reported.

Funding Statement

This research received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors.

Ethical Approval

As this is a methodological review, no ethical approval was required.

Author Contributions

Both authors made a significant contribution to the work reported, whether that is in the conception, study design, execution, acquisition of data, analysis and interpretation, or in all these areas; took part in drafting, revising or critically reviewing the article; gave final approval of the version to be published; have agreed on the journal to which the article has been submitted; and agree to be accountable for all aspects of the work.

Disclosure

All authors have completed the ICMJE uniform disclosure form at www.icmje.org/coi_disclosure.pdf and declare no support from any organization for the submitted work, no financial relationships with any organizations that might have an interest in the submitted work in the previous three years, and no other relationships or activities that could appear to have influenced the submitted work. The lead author affirms that the manuscript is an honest, accurate, and transparent account of the study being reported, that no important aspects of the study have been omitted, and that any discrepancies from the study as originally planned (and, if relevant, registered) have been explained.