Are providers overconfident in predicting outcome after cardiac arrest?

Abstract

Aim:

To quantify the accuracy of health care providers’ predictions of survival and function at hospital discharge in a prospective cohort of patients resuscitated from cardiac arrest. To test whether self-reported confidence in their predictions was associated with increased accuracy and whether this relationship varied across providers.

Methodology:

We presented critical care and neurology providers with clinical vignettes using real data from post-arrest patients. We asked providers to predict survival, function at discharge, and report their confidence in these predictions. We used mixed effects models to explore predictors of confidence, accuracy, and the relationship between the two.

Results:

We completed 470 assessments of 62 patients with 65 providers. Of patients, 49 (78%) died and 9 (15%) had functionally favourable survival. Providers accurately predicted survival in 308/470 (66%) assessments. In most errors (146/162, 90%), providers incorrectly predicted survival. Providers accurately predicted function in 349/470 (74%) assessments. In most errors (114/121, 94%), providers incorrectly predicted favourable functional recovery. Providers were confident (median confidence predicting survival 80 [IQR 60 – 90]; median confidence predicting function 80 [IQR 60 – 95]). Confidence explained 9% and 18% of variation in accuracy predicting survival and function, respectively. We observed significant between-provider variability in accuracy (median odds ratio (MOR) for predicting survival 2.93, 95%CI 1.94 – 5.52; MOR for predicting function 5.42, 95%CI 3.01 – 13.2).

Conclusions:

Providers varied in accuracy predicting post-arrest outcomes and most errors were optimistic. Self-reported confidence explained little variation in accuracy.

Introduction

Most patients resuscitated from cardiac arrest are initially comatose.¹ Despite advances in care, many die in the hospital.^2–4 Most common proximate cause of death is withdrawal of life-sustaining therapy (WLST) because of perceived poor neurological prognosis.^5–7 Consensus guidelines recommend using multiple prognostic modalities and delaying WLST for several days after return of spontaneous circulation (ROSC).^8–10 Even when precautions are taken, neurological prognostication after cardiac arrest is challenging.

Failure to prognosticate accurately has significant consequences. Inaccurate or premature WLST can increase avoidable mortality when patients who might recover if life-sustaining therapies were to have been continued instead die.^5,11 Simultaneously, delayed prognostication can prolong intensive care provided to patients with unrecognized irrecoverable injury, leading to emotional and financial burden to families and surrogates. Because there is no gold standard to predict outcomes,^12,13 physicians must rely on clinical judgement to interpret and synthesize available data when formulating an overall estimate of potential to recover.

Physicians’ accuracy predicting outcomes after cardiac arrest is poorly characterized. Similarly, their ability to judge when available data are sufficient to allow an accurate prognostic estimate is unknown. In this study, we quantified accuracy of providers’ predictions of vital status and function at hospital discharge in a prospective cohort of patients resuscitated from cardiac arrest. We hypothesized there would be significant between-provider variability in accuracy and tested whether measurable provider-level factors explain this variation. Finally, we tested whether self-reported confidence in their predictions was associated with increased accuracy and whether this relationship varied across providers.

Methods

The University of Pittsburgh Human Research Protection Office approved all aspects of the study. We performed a single center, prospective cohort study from November 2018 to October 2019. We enrolled clinical providers at a single academic medical center who were involved in post-arrest patient care as part of their usual clinical duties. Clinical providers could be attending physicians, fellows or residents. Simultaneously, we prospectively identified patients resuscitated from in- or out-of-hospital cardiac arrest cared for by our Post-Cardiac Arrest Service. For these patients, we reviewed the electronic medical record to create a 1–2 sentence clinical synopsis summarizing major historical features. Such a synopsis, for example, might be “This is a 72-year-old gentleman resuscitated from out-of-hospital cardiac arrest yesterday. He was initially transported to another facility, then transferred here about 60 minutes after ROSC.”

We only interviewed providers who were not involved in the clinical care of a given test patient. Initially, AS presented a clinical vignette to the study participant and then the study participant could ask for additional available clinical data at the time of their participation. This included any available historical information, vital signs, physical exam findings, laboratory results, medications, imaging, electroencephalographic tracings or reports, and any other information requested by the respondent that would be available to the treating clinical team.

We blinded study participants to the patient’s identity and requested they not seek additional information outside the context of the study. We queried providers in real-time, while the test patients were still undergoing treatment and while ultimate outcome was unknown. Each provider completed sequential assessments on patient hospital days 1, 2, 3 and 5. If the patient died or was discharged earlier, we stopped asking providers for further predictions since real-time data no longer existed. We collected data for multiple initial days because many patients die within the first few days of admission. For each daily response, enrolled providers completed a brief web-based survey (Table 1) that included their prediction of the patient’s prognosis, self-reported confidence in this prediction, whether the provider would recommend WLST based on the available data that day, and what factors influenced their prediction. We asked providers to predict whether the patient will die despite maximal medical therapy. We defined favorable functional outcome as discharge from the hospital to home or acute rehabilitation. In addition to daily responses, we collected basic demographic information from each enrolled respondent, including current training level, years since receiving MD/DO, number of post-arrest patients cared for annually and primary medical specialty.

Table 1.

Summary of Questions and Answer Format for Daily Real-Time Assessments

Question	Answer Format
1a. Will the patient survive to hospital discharge?	Yes or No
1b. How confident are you in your prediction?	0 – 100% (0% not, 100% extremely)
2a. Will the patient be discharged to home or acute rehabilitation? (favorable function outcome)	Yes or No
2b. How confident are you in your prediction?	0 – 100% (0% not, 100% extremely)
3. Would you recommend withdrawal of life sustaining therapy (WLST) today?	Yes or No
4. What 3–5 factors helped you make your prognostic assessment?	Open ended question

When soliciting factors that influenced providers’ assessments, we allowed free-text responses. We planned a priori to classify these responses as: neurological factors; non-neurological factors; time since arrest; and patient or family wishes. Neurological factors consisted of exam findings, electroencephalogram (EEG), computed topography (CT) head, magnetic resonance imaging (MRI) brain, and somatosensory evoked potentials (SSEPs). Non-neurological factors included premorbid conditions, arrest history, multisystem organ failure and shock (including specific mentioning of elevated lactate, use of vasopressors or acidosis). After completion of coding, we identified the following additional common themes from factors previously coded as “other”: respiratory status, clinical trajectory, sedatives/paralytics, and presence of mechanical circulatory support.

Statistical Analysis

We used descriptive statistics to summarize patient and provider characteristics. We considered accurate predictions to be those concordant with the observed outcome (i.e. provider does not think the patient will survive to discharge and patient does not survive; or, provider thinks the patient will survive to discharge and patient survives). We calculated the overall proportion of accurate responses by day. Because many patients whose data we presented had the potential to die after WLST, our main outcome labels (survival and functionally favorable survival) might not reflect an objective ground truth. In other words, some providers’ predictions of recovery might be inaccurate because of inappropriate WLST, family’s representation of a patient’s values and preferences, or other intangible factors rather than objectively irrecoverable injury. We performed an sensitivity analysis to eliminate potential bias introduced from “mis-labeled” ground truth outcomes. For instance, if a patient had unclear prognostic clinical features and the family opted for WLST based on the patient’s values, then possibly the provider’s prediction would misclassified as an inaccurate. For the sensitivity analysis, we reviewed each case in which a patient did not survive to hospital discharge. In each case, we differentiated between unequivocally irrecoverable illness and potentially recoverable cases. To do this, we applied a set of highly specific markers of irrecoverable brain injury or multisystem organ failure to define unequivocal cases: death by neurological criteria (brain death); rearrest or death from overwhelming multisystem organ failure despite maximal support; or with a combination of diffuse severe cerebral edema on brain imaging;¹⁴ highly malignant EEG (sustained generalized suppression beyond 48 hours or burst suppression with identical bursts);^15–17 bilaterally absent N20 cortical responses on SSEPs obtained at least 72 hours post-arrest without artifact or concomitant sedation;¹³ or persistently absent pupillary light reflex at 72 hours.¹⁸ After identifying potentially mislabeled (i.e. equivocal) poor outcomes, we repeated our analyses by excluding any equivocal cases. Then, we re-assigned the equivocal patients a counterfactual favorable outcome label (survive and favorable functional outcome at discharge), and once again repeated our analyses.

We used random effects models to quantify between-provider variation in accuracy predicting survival or function and report median odds ratios to summarize this variability when statistically significant. We then used fixed effect models with Huber White sandwich estimators to account for clustering within-providers to test whether experience (dichotomized as attending physician versus trainee), medical specialty, or day of assessment predicted accuracy. We used Kruskal-Wallis tests to determine if each of these three factors were associated with confidence. Finally, we used mixed effects models to test whether confidence predicted accuracy, whether this relationship varied across providers and whether provider-level factors explained this variability. Individual patient assessments were clustered both within-patient (i.e. over time) and within providers, but not perfectly nested into a three-level hierarchy. Mixed effects models cannot account for all potential correlations between observations in this data structure. Since between-provider variability was the focus of our work, we focused on this source of correlation and treated serial assessments within patient as independent. We performed all analyses using Stata version 14 (StataCorp, College Station, TX).

Results

We completed 470 assessments from 65 different providers, half of whom (33, 51%) were attending physicians with mean 8 ± 10 years of clinical experience (Table 2). Assessments used data from 62 patients (Table 3), of whom (49, 78%) survived to hospital discharge and (9, 15%) had a favorable function at discharge. Most deaths (32, 67%) occurred after WLST for perceived poor neurological prognosis. Providers accurately predicted survival in 308/470 (66%) assessments (Table 4a). In most errors (146/162, 90%), providers predicted survival for a patient who died. Providers accurately predicted function in 349/470 (74%) assessments (Table 4b). In most errors (114/121, 94%), providers incorrectly predicted a favourable outcome. Accuracy improved after removing potentially equivocal deaths (Table 4) (accuracy 281/373 (75%) and 306/373 (82%) predicting survival and function, respectively). Most errors (76/92, 82% and 59/66, 89% for survival and functional outcome, respectively) were still from over-optimism but decreased. Reassigning favorable outcomes in equivocal cases did not substantially alter results compared to removing these cases (accuracy 351/470 (75%) and 361/470 (77%), respectively).

Table 2.

Demographic characteristics of providers

Characteristics	Providers (n = 65)
Current Level of Training
Attending	33 (51)
Fellow	20 (31)
Resident	10 (15)
Other	2 (3)
Medical Specialties
General critical care	32 (49)
Neurology	17 (26)
Neurocritical care	6 (9)
Emergency medicine	5 (7)
Other	5 (7)
Average years since graduation	8 [5 – 16]
Number of post arrest patients annually	15 [5 – 25]
Mean confidence for survival	80 [60 – 90]
Mean confidence for function	80 [60 – 95]

Data are presented as raw numbers with corresponding percentages or median with interquartile range

Table 3.

Demographic characteristics of patients

Characteristics	Patients (n = 62)
Age	60 [51 – 69]
Female gender	21 (34)
Initial arrest rhythm
Pulseless electrical activity	22 (35)
Ventricular tachycardia or fibrillation	10 (31)
Asystole	17 (27)
Unknown	10 (16)
Out of hospital arrest	53 (85)
Pittsburgh Cardiac Arrest Category
I	5 (8)
II	18 (29)
III	9 (15)
IV	21 (33)
Unknown	9 (15)
Transferred from another hospital	40 (65)
Survived to discharge	14 (22)
Disposition of survivors
Home or inpatient rehabilitation	9 (15)
SNF or LTAC	5 (8)
Cause of Death
WLST – Neurologic	32 (67)
WLST – Non-neurologic	11 (23)
Brain death	6 (13)

Data are presented as raw numbers with corresponding percentages or median with interquartile range.

SNF=skilled nursing facility, LTAC= long-term acute care facility, WLST = withdrawal of life-sustaining therapy

Table 4a.

Accuracy of survival predictions. A table of providers’ prediction of survival based on the actual survival of the patient, for all assessments, n (%).

		Actual Outcome (all assessments, n = 470)		Actual Outcome (excluding equivocal cases, n=373)		Actual Outcome (reassign outcome in equivocal cases, n=470)
		No	Yes	No	Yes	No	Yes
Prediction	Yes	146 (31)	134 (29)	76 (20)	134 (36)	76 (16)	204 (43)
	No	174 (37)	16 (3)	147 (39)	16 (4)	147 (31)	43 (9)

Table 4b.

Accuracy of functional outcome Predictions. A table of providers’ prediction of functional outcome (discharge disposition from hospital) based on the actual functional outcome of the patient, for all assessments, n (%).

		Actual Outcome (all assessments, n = 470)		Actual Outcome (excluding equivocal cases, n=373)		Actual Outcome (reassign outcome in equivocal cases, n=470)
		No	Yes	No	Yes	No	Yes
Prediction	Yes	114 (24)	83 (18)	59 (16)	82 (22)	59 (13)	137 (29)
	No	266 (56)	7 (1)	224 (60)	7(2)	224 (58)	49 (10)

Data are presented as raw numbers with corresponding percentages.

Overall, providers were confident in the predictions, with median confidence for survival 80 out of 100 [IQR 60 – 90] and for functional outcome 80 [IQR 60 – 95]. Confidence did not differ between attending physicians and trainees. Accuracy predicting survival and functional outcome were both predicted by confidence (both P<0.001) (Figure 1). However, confidence explained little of the observed variability in accuracy (9% survival and 18% functional outcome). When providers were 100% confident, they were accurate in 89% of assessments predicting survival and 97% of assessments predicting function. In comparison, when providers reported 50% confidence, they were accurate in 62% of assessments predicting survival and 74% of assessments prediction function.

Figure 1.

Distribution of providers’ confidence, stratified dy accuracy, when predicting (a) survival to hospital discharge, and (b) functionally favorable recovery at discharge.

Accuracy of outcome predictions varied across providers. There was substantial between-provider variability in accuracy of predictions for survival (P<0.001 for likelihood ratio test vs a fixed effects model; median odds ratio (MOR) 2.93, 95% confidence interval (CI) 1.94 – 5.52 and for functional outcome (P<0.001 for likelihood ratio test; MOR 5.42, 95%CI 3.01 – 13.2). Substantial between-provider variability also existed in the relationship between confidence and accuracy (P<0.001 for the likelihood ratio test vs a fixed effect model). Accuracy in predicting survival did not change with experience but attending physicians were less accurate predicting functional outcomes at discharge compared to trainees (P=0.003). We observed greater between-provider variability among attending physicians (MOR 4.00, 95% CI 2.17 – 12.1) than trainees (MOR 2.07, 95% CI 1.34 – 5.83) when predicting survival to discharge. Results predicting function were similar. Medical specialty and post-arrest day were not associated with accuracy of either survival or functional outcome predictions. Self-reported confidence in predicting survival and function did not change by experience, medical specialty or days since arrest.

If a provider indicated they would be recommended WLST for a given case, 11% survived to discharge but none had favorable functional outcome. Providers were more confident when they were willing to recommend WLST (90 [IQR 72.5 – 100] for WLST yes vs. 78 [IQR 60 – 90] for WLST no, P<0.001). Providers used a variety of factors to make prognostic assessments (Table 5). Neurological factors were most commonly used in 420 (89%) assessments, specifically including exam (81%) and EEG (37%). Non-neurological factors were next commonly used 344 (73%).

Table 5.

Factors used for prognostication

Factor	Frequency (n = 470)
Neurological	420 (89)
Physical Exam	379 (81)
EEG	173 (37)
CTH	77 (16)
Evoked potentials	19 (4)
Sedatives/paralytics	25 (5)
MRI	3 (1)
Non-neurological	344 (73)
Arrest history	157 (33)
Premorbid condition	113 (24)
MSOF	104 (22)
Shock	97 (21)
Respiratory status	24 (5)
Mechanical circulatory support	7 (1)
Clinical trajectory	14 (3)
Days since arrest	29 (6)
Family wishes	1 (0.2)

Data are presented as raw numbers with corresponding percentages.

EEG = electroencephalogram, CTH = computed tomography head, MRI= magnetic resonance imaging, MSOF= multisystem organ failure

Discussion

In this prospective, observational study, we found providers to be rather inaccurate in predicting survival and functional outcome of post cardiac arrest patients. Most errors were optimistic, even when potentially equivocal outcomes were removed from analysis. Providers were more accurate when they were more confident, but confidence explained little in the variation of accuracy predictions. To our knowledge, this is the first study to explore the accuracy of providers predicting outcomes after cardiac arrest and the role of self-reported confidence. We also observed that accuracy, confidence and the relationship between the two varied significantly across providers and that this variability was not explained by measured provider-level characteristics such as experience or medical specialty. Interestingly, attending physicians were less accurate in their predictions in functional outcome at discharge compared to trainees.

There may be unmeasured intrinsic provider factors that contribute to between-provider variability. Currently, no gold standard tests exist for post arrest prognostication.^12,13 A recent survey found self-reported neuroprognostic approaches to be varied and often inconsistent with evidence-based guidelines,¹⁹ indicating that knowledge gaps may contribute to prognostic variability seen in our study. Other intrinsic provider factors leading to inaccurate prognostication could include unconscious biases and inherent optimism. Additionally, likely providers rely on heuristics when making decisions on what to do for their patients.²⁰ Heuristic judgements are rapid solutions based on pattern recognition and mental shortcuts.^21,22 Providers develop good heuristics when they perform the same task repeatedly or when they receive feedback on performance. When prognosticating after cardiac arrest, providers rarely receive unbiased feedback because of self-fulfilling prophecies,^23–25 resulting in poorly calibrated decisions. The self-fulfilling prophecy and poorly calibrated decisions may explain the results of our study that more experienced providers are worse at predicting functional outcomes. We found greater between-provider variability in accuracy among attending physicians than trainees. Development of heuristics may explain this result, since more experienced providers may become entrenched in their thought processes over time.

We also observed a tendency towards overly optimistic predictions. Recently, we showed that providers desire near certainty for their neuroprognostic tools, with most viewing a false positive rate (FPR) of <0.1% as acceptable for recommending WLST in a patient who would have otherwise recovered.²⁶ By contrast, providers tolerate greater error when continuing care in a patient with irrecoverable recovery.²⁶ This asymmetric view of the cost of errors may explain a bias towards continuing life-sustaining therapy or a hesitancy among providers to make nihilistic predictions. Given the optimistic errors seen in our study, providers’ may act more cautiously and err on the side of continuing life sustaining therapy when uncertain, which is a reasonable approach. However, optimistic errors can have consequences including financial to family and society. Post-arrest care is expensive,^27–29 particularly when a patient is frail or undergoes mechanical circulatory support.^30,31 Optimistic prognostic errors may also limit palliative interventions at the end of life,³² potentially prolonging patients’ suffering. Additionally, surrogates of critically ill patients experience negative psychological outcomes, especially when making decisions about end of life.^33,34 This emotional burden may be worsened by inaccurate prognostication and prolongation of critical care for patients destined to have unfavorable outcomes.

Prognostic variability and its effects on treatment recommendations have also been observed in prior studies in other brain injured populations. After intracerebral hemorrhage (ICH) and traumatic brain injury, cohort studies have found overly pessimistic prognostication.^35,36 In a prospective, observational study, a third of providers were inaccurate when making predictions about 90-day Modified Rankin Score of ICH patients,³⁷ similar to the rate of inaccuracy seen in our study. Other studies used clinical vignettes of patients with either ICH or traumatic brain injury and found significant variability in providers’ prognostic assessments and their clinical decision making for treatment recommendations.^38,39

Our results parallel those in general critical care patients as well. One study asked providers to predict long-term prognosis of general ICU patients by asking them to choose from four options (good prognosis to death) and showed that providers were wrong a third of the time.⁴⁰ Similar to our work, most errors were overly optimistic predictions and providers’ experience did not affect results.⁴⁰ In a study of medical ICU patients, providers were asked to make binary predictions about survival. Healthcare providers were once again inaccurate and were only correct about half of the time.⁴¹ Similar to our study, when medical and surgical ICU physicians were asked to predict patients’ survival and functional status, providers were more accurate when they were more confident.⁴² These inaccurate predictions have shown to have significant consequences, as poor outcome predictions has been associated with withdrawal of mechanical ventilation, and more strongly predicted withdrawal than age, severity illness or organ failure.⁴³

Our study has several limitations. Principle among these is the single-center design, which limits the generalizability of the results. Our hospital has a Post-Cardiac Arrest Service, which consists of clinicians and researchers with significant experience in post-arrest prognostication. Over time, the service may have influenced other providers’ optimism and beliefs. Beyond the specific institution, it is also unclear whether our results would generalize to other nations, cultures or patient populations. For example, patients enrolled in our study were relatively young, which may have influenced providers’ predictions. Another limitation is the main cause of patient death being WLST for neurologic reasons. WLST-N occurs because of surrogates’ views on patient’ wishes, based on information provided to them by the primary treating team. This creates unknowns about the true natural history of post cardiac arrest patients where care is withdrawn. The design of our study resulted in more assessments being conducted using data derived from patients with unclear prognostic outcomes. Patients with particularly poor prognostic signs (i.e. diffuse cerebral edema and malignant burst suppression/myoclonus status epilepticus) were less likely to survive to hospital day 5 and so fewer assessments were conducted. This limitation may explain the lack of observed improvement in accuracy with time. Overall visual appearance or clinical gestalt may influence providers’ predictions. Since we provided only electronic information, we cannot draw conclusions about the roll of this gestalt on accuracy or confidence. Finally, assessments were clustered both within-provider and within-patient. Hierarchical regression models cannot account for the variance-covariance of multi-level clustering when one level is not completely nested within a higher level. Since between-provider variability was our scientific interest, we accounted for this in our models. However, within-patient correlations between serial assessments may have biased our results.

In conclusion, we quantified the accuracy of a diverse cohort of providers’ in predicting survival and functional outcome of post-arrest patients. We found providers were inaccurate in about a third of their assessments and most of the errors were optimistic. We found that accuracy and confidence were correlated, but confidence explained little in variation in accuracy. Most importantly, our study showed that there was substantial between-provider variability of accuracy, confidence and the relationship of accuracy and confidence, which cannot be explained by our measurable provider-level factors or day since arrest. Future work will explore sources of error and modifiable patient and provider factors that could explain the variability seen in accurate prognostication.