Time to awakening and self-fulfilling prophecies after cardiac arrest

Abstract

Objective:

Withdrawal of life-sustaining therapies for perceived poor neurological prognosis (WLST-N) is common after resuscitation from cardiac arrest and may bias outcome estimates from models trained using observational data. We compared several approaches to outcome prediction with the goal of identifying strategies to quantify and reduce this bias.

Design:

Retrospective observational cohort study

Setting:

Two academic medical centers (“UPMC” and “UAB”)

Patients:

Comatose adults resuscitated from cardiac arrest

Intervention:

None

Measurements and Main Results:

We considered as predictors clinical, laboratory, imaging and quantitative electroencephalography (EEG) data available early after hospital arrival. We followed patients until death, discharge or awakening from coma. We used penalized Cox regression with a LASSO penalty and 5-fold cross-validation to predict time to awakening in UPMC patients then externally validated the model in UAB patients. This model censored patients after WLST-N, considering subsequent potential for awakening to be unknown. Next, we developed a penalized logistic model predicting awakening, which treated failure to awaken after WLST-N as a true observed outcome, and a separate logistic model predicting WLST-N. We scaled and centered individual patients’ Cox and logistic predictions for awakening to allow direct comparison, then explored the difference in predictions across probabilities of WLST-N. Overall, 1,254 patients were included and 29% awakened. Cox models performed well (mean area under the curve 0.93 in the UPMC test sets and 0.83 in external validation). Logistic predictions of awakening were systematically more pessimistic than Cox-based predictions for patients at higher risk of WLST-N, suggesting potential for self-fulfilling prophecies to arise when failure to awaken after WLST-N is considered as the ground truth outcome.

Conclusions:

Compared to traditional binary outcome prediction, censoring outcomes after WLST-N may reduce potential for bias and self-fulfilling prophecies.

Introduction

More than 125,000 Americans are hospitalized after resuscitation from cardiac arrest annually.¹ Most are initially comatose, of whom approximately 70% die in the hospital.^2,3 Differentiating patients who will awaken from those with irrecoverable brain injury is a focus of neurological prognostication. Awakening from coma is a key step in recovery and may not occur for many days after initial resuscitation.^3-6 Most non-survivors succumb after withdrawal of life-sustaining therapy for perceived poor neurological prognosis (WLST-N), which often occurs earlier than awakening is typically observed.^7,8 While early limitations in aggressive care are consistent with some patients’ preferences, it is possible many patients who undergo early WLST-N have unrecognized recovery potential.^7,8

The goal of most prognostic models is to predict recovery for individual patients given their clinical characteristics. Models are trained using existing data with the goal of informing future clinical care. Outcomes in the training data are affected not just by clinical characteristics but also by treatment decisions. WLST-N leads almost invariably to death, a relationship that holds regardless of affected patients’ characteristics. Thus, clinical factors that led to WLST-N will be learned to predict poor outcome regardless of the true causal relationship between those factors and outcomes. Using output from these models to guide clinical care prospectively can create self-fulfilling prophecies when biased model predictions lead clinicians to replicate potentially flawed past treatment decisions.

Machine learning (ML) might improve post-arrest prognostication by providing outcome predictions that are more accurate and/or timely than otherwise possible. Like other approaches trained using observational data, ML is susceptible self-fulfilling prophecies. Prior studies have applied ML to post-arrest electroencephalography (EEG) with promising results, but have considered as candidate features only a subset of clinical data routinely available.^9-14 We built on this prior work to consider both quantitative EEG (qEEG) features and a wider range of clinical characteristics. Importantly, we revisited the standard approach of predicting recovery as a binary outcome. Instead, we used a survival approach to predict time to awakening. We hypothesized that survival regression would yield more optimistic patient-level predictions than logistic regression for patients at high risk of WLST-N. If confirmed, this approach could be useful to detect and mitigate the risk of self-fulfilling prophecies.

Methods

Patients and setting

We performed a retrospective, observational cohort study including comatose patients hospitalized at one of two academic medical centers (UPMC Presbyterian hospital and University of Alabama Birmingham (UAB) hospital) after cardiac arrest between September 2010 and September 2019 who underwent at least 6h of continuous EEG. We excluded patients who were awake on presentation or within 6h of EEG initiation; those who arrested due to neurological or traumatic etiologies; and those were transferred or had EEG initiated >24h post-arrest. The University of Pittsburgh Human Research Protection Office (HRPO) approved this study (STUDY19020205) on 3/11/2019. All study procedures were followed in accordance with the ethical standards of the HRPO and with the Helsinki Declaration of 1975. We followed reporting requirements of the Transparent Reporting of Multivariable Prediction Model for Individual Prognosis or Diagnosis (TRIPOD) statement.¹⁵

We previously described systems of post-arrest care at both participating sites.¹⁶ Briefly, at both centers patients were cared for by a specialized team of physicians. Standard care included goal-directed early resuscitation, structured neurological examination on hospital arrival and daily thereafter, active temperature control to 33°C or 36°C, initial brain computerized tomography (CT), early coronary angiography, and neurological prognostication consistent with American Heart Association and European Resuscitation Council guidelines current during the study period. Comatose post-arrest patients at both institutions underwent continuous EEG monitoring, except in cases with rapid awakening; limitations in care due to prior advanced directives; refractory multisystem organ failure; or non-survivable primary anoxic brain injury on initial CT, defined as loss of grey-white differentiation (ratio of grey-to-white matter attenuation <1.10) with effacement of the sulci and/or basal cisterns. We used all available data in existing cohorts from each center so did not perform an a priori sample size calculation or power analysis.

Covariates and outcomes

We considered clinical covariates available at hospital arrival (or on return of spontaneous circulation for in-hospital arrests) until 6h after EEG initiation (Supplemental Digital Content - Appendix). Both centers maintained prospective registries from which we extracted baseline clinical and demographic information including age, sex, presenting rhythm and arrest location. At UPMC, we also recorded arrest etiology, which for this analysis we simplified as confirmed cardiac vs non-cardiac etiology.¹⁷ We reviewed presenting brain CTs to calculate gray-white ratio (GWR) at the level of the basal ganglia¹⁸ and determined presence or absence of sulcal and basal cistern effacement. We summarized initial neurological examination findings using Full Outline of UnResponsiveness subscales and separately coded presence or absence of individual brainstem reflexes.¹⁹ At UPMC, we extracted details of medication administration and laboratory values from the electronic health record from arrival (or arrest for in-hospital arrests) until 6h after EEG initiation. From this, we calculated cumulative doses of intravenous medications for two epochs, hospital arrival or arrest until EEG start and during EEG, including medications administered to at least 5% of patients. We determined highest and lowest values of laboratory test results acquired in at least 50% of patients. We calculated Sequential Organ Failure Assessment (SOFA) subscales and Pittsburgh Post-Cardiac Arrest Category.²⁰ In total, this yielded 146 predictors describing initial post-arrest clinical status.

We generated qEEG features over the first 6h of monitoring. EEG was recorded at 256Hz using 22 gold-plated cup electrodes applied in standard 10-20 International System of Electrode Placement positions. We used Persyst version 13 (Persyst Development Co., Prescott, AZ) to generate over 5,000 standard and custom qEEG features at 1Hz across individual electrodes. We previously found qEEG summarized hourly optimizes computational efficiency with minimal information loss.²¹ Thus, we generated hourly summaries of each qEEG, including maximum, minimum, mean, median, 25^th percentile, 75^th percentile, standard deviation and sum, resulting in 289,680 qEEG features per patient.

Our primary outcome was awakening, defined as following verbal commands. Trained providers evaluated patients at least daily until death, awakening or hospital discharge. We recorded time of awakening and calculated time from collapse to awakening in hours. A secondary outcome was WLST-N. At UPMC, proximate cause of death was prospectively recorded as one of the following mutually exclusive categories: WLST-N; withdrawal of life-sustaining therapies for non-neurological reasons (e.g., preexisting advanced directives); brain death; rearrest or death due to refractory multisystem organ failure.

Statistical analysis

We performed analyses using R version 4.0.1.²² We summarized baseline characteristics and outcomes. We used the missForest package to perform 10 random forest imputations of missing data (<0.1% of total data elements, with no variable missing in >1%),²³ then used glmnet to build Cox regression models with a LASSO penalty to predict time to awakening (Figure 1).²⁴ We censored observation at death after WLST-N.^7,20

Figure 1:

Conceptual design of the study

First, we evaluated model performance within a single center (UPMC). To do so, we randomly split the cohort into 5 folds for training and testing. We used nested cross-validation within each training set to tune the shrinkage parameter (lambda) to maximize cross-validated predictive log partial likelihood, then evaluated test set performance at the lambda that minimized cross-validation error. We predicted individual patient hazards in the test sets, which we used as classifiers to predict awakening. We evaluated test set model fit using C-index and pairwise log-rank P values by consecutive quartiles. We calculated fold-specific areas under the receiver operating curve (AUROC), then summarized overall test set performance. We next performed external validation, training the model using all UPMC patients (“complete UPMC data Cox model”) and testing using all UAB patients (“external validation model”). We again evaluated test set performance. We examined calibration of test set predictions using decile calibration plots. As a secondary analysis, we assigned an arbitrarily long observation time to patients who succumbed despite maximal medical support (brain death or rearrest) and repeated modeling procedures. The motivation for this was that ground truth outcomes were theoretically known with certainty in these cases, making awakening impossible even with prolonged observation.

Next, we evaluated the potential usefulness of survival analysis to mitigate self-fulfilling prophecies. We repeated training and test procedures in the UPMC cohort using LASSO logistic regression to predict awakening as a binary outcome, determined individual predicted outcome probabilities in the test sets, and pooled test set results. We repeated training and testing procedures to develop a separate model predicting WLST-N. For each patient, we scaled and centered the logistic regression-generated outcome estimates to create standardized prediction scores and again evalued calibration using decile calibration plots. We explored the difference between survival and logistic standardized prediction scores across WLST-N probabilities using linear regression. Logistic regression (which treats the observed outcome as “ground truth”) would be expected to be excessively pessimistic compared to survival regression (which learns from patient observation to the point of WLST-N, then censors that patient’s potential for future awakening) in cases where WLST-N is most likely, since WLST-N may cause death in patients who otherwise would have awakened.

Finally, we predicted individual patients’ median time to awakening. We used test set risk estimates from Cox regression to construct conditional Kaplan Meier survival estimators using observation weights from a Gaussian kernel.²⁵ Using this method, any individual patient’s survival curve is constructed from a weighted combination of similar patients. We derived quantile predictions and examined calibration against actual time to awakening.

Results

Overall, 2,389 patients were treated after cardiac arrest at UPMC of whom 1,002 (42%) were included, and 252 were treated at UAB of whom 166 (66%) were included (Supplemental Digital Content - Figure 1). Median age was 59 [interquartile range (IQR) 48 – 69] years, 440 (38%) were female, and most patients treated at UPMC were white while most treated at UAB where Black or African American (Table 1). Overall, 29% awakened from coma a median of 46 [IQR 33 – 88] and 65 [IQR 46 – 123] hours post-arrest at UPMC and UAB, respectively. EEG was initiated a median of 9.5 [IQR 7.6 to 12.0] hours post-arrest.

Table 1:

Baseline clinical characteristics and outcomes, stratified by participating site.

Characteristic	UPMC Presbyterian (n = 1,002)	UAB Hospital (n = 166)
Age, years	59 [48 – 69]	60 [48 – 68]
Female sex	377 (38)	63 (34)
Race
White	746 (74)	109 (43)
Black or African American	105 (10)	134 (53)
Asian	6 (1)	1 (0)
Other	5 (1)	0 (0)
Unknown	140 (14)	8 (3)
Ethnicity
Hispanic	2 (0)	5 (2)
Not Hispanic	723 (78)	0 (0)
Unknown	218 (22)	161 (98)
Arrest out-of-hospital	844 (84)	133 (80)
Presenting rhythm
VT/VF	324 (32)	58 (35)
PEA	381 (38)	52 (31)
Asystole	323 (32)	14 (8)
Unknown	57 (6)	42 (25)
Arrest duration, minutes	19 [11 – 29]	15 [10 – 30]
Confirmed cardiac etiology	283 (28)	--
Awakened from coma	298 (30)	44 (27)
Time at risk, hours
Awakened	46 [33 – 88]	65 [46 – 123]
Did not awaken	77 [47 – 128]	148 [91 – 248]

Data are presented as raw number with corresponding percentages or median [interquartile range].

Abbreviations: VT/VF – Ventricular tachycardia/fibrillation; PEA – pulseless electrical activity.

Across training sets, survival models selected generally consistent feature sets (Figure 2 + Supplemental Digital Content – Table 1), albeit with expected minor variations due to multicollinearity among correlated EEG predictors (e.g., spatially adjacent electrodes). LASSO-selected features included clinical covariates known to predict outcome (initial shockable rhythm, cardiac etiology of arrest, favorable neurological examination on presentation); parasagittal suppression ratio; delta-band rhythmicity; amplitude-integrated EEG; frontal alpha-band power; and burden of epileptiform discharges. These models were well calibrated (all log-rank P values <10⁻⁵; C-indices 0.72 – 0.78 with all P values < 0.001 vs the bootstrapped null). Mean AUROC was 0.93 for UPMC test sets and 0.83 in the UAB test set (Figure 3). Models trained using arbitrarily long observation times for patients who died despite maximal support had similar performance (data not shown). Mean AUROC was 0.93 in the logistic model trained to predict awakening and 0.81 in the logistic model trained to predict WLST-N, and both models’ predictions were well-calibrated.

Figure 2:

Predictors and normalized coefficients selected in penalized Cox regression from the complete UPMC data Cox model.

Figure 3:

(A) Individual set performance (grey) and average performance (black) in the UPMC cohort predicting awakening after penalized Cox regression; (B) Performance in external validation in the UAB cohort (green), with UPMC performance (light grey) for reference.

Standardized scores predicting awakening from logistic regression were more optimistic than scores from Cox regression for patients at low risk of WLST-N (Figure 4). Conversely, for patients at high risk of WLST-N, logistic regression was more pessimistic than Cox regression. In other words, censoring outcomes for patients exposed to WLST-N (rather than considering failure to awaken after WLST-N to be a true outcome) resulted in systematically more favorable outcome predictions when WLST-N was common. Linear regression quantifying the magnitude of this effect showed an intercept of −0.015 (P = 0.020) and slope of 0.033 (P = 0.006).

Figure 4:

Pooled test set performance from the UPMC cohort showing the difference in individual patients’ standardized prediction scores derived from Cox and logistic regressions predicting awakening across predicted probability of withdrawal of life-sustaining therapy for perceived poor neurological prognosis. Positive values on the Y axis indicate Cox-based estimates were more optimistic than logistic-based estimates. Linear modeling used to quantify the magnitude of this effect showed an intercept of −0.015 (P = 0.020) and slope of 0.033 (P = 0.006), demonstrating systematically more optimistic predictions for patients at high risk of WLST-N when their potential for awakening after WLST-N is treated as censored using Cox regression rather than absent using logistic regression.

Performance predicting time to awakening was modest. Median predicted survival time was strongly associated with actual survival time (P <0.0001) but only explained 21% of the variation in actual survival time. Predictions were well calibrated (slope = 1.13, P value vs a null of 1.0 not significant; P value for intercept also not significant). Interquartile ranges for individual patients were broad (mean 8.5 (SD 5.2) days) and contained the observed awakening time in 64%. Considering awakening delayed >72 hours as a binary outcome, predictions were 64% (95% CI 52 – 72%) sensitive and 72% (95% CI 65 – 78%) specific.

Discussion

We identified a risk of self-fulfilling prophecies to result when binary outcome models are used to predicting awakening from coma after cardiac arrest. The threat of self-fulfilling prophecies is pervasive in medicine. WLST-N is widespread,²⁶ and in critical illness almost invariably leads to death. Conventional models trained using observational datasets where WLST-N occurred learn to replicate past clinical decision making. Factors that prompted clinicians or families to choose WLST-N are associated with poor outcome regardless of whether this relationship would hold in the absence of WLST-N.²⁷ Insofar as this may inform future care, whether directly as decision support tools or simply by reinforcing clinicians’ prior beliefs, these spurious correlations will be perpetuated or even amplified over time.²⁸

Current approaches to minimize self-fulfilling prophecies in analyses of observational data are lacking. Seemingly minor limitations such as “do not resuscitate” orders independently predict worse hospital survival even among those who do not require cardiopulmonary resuscitation, and these factors are inconsistently included in available datasets. Choices to escalate intensity of care (renal replacement therapy, a second vasopressor, etc.) are challenging to control for when they do not occur.²⁹ Although approaches like causal inference aim to account for these and other sources of bias statistically,³⁰ it is impossible to fully reconstruct counterfactual patient outcomes. Excluding patients exposed to WLST-N at any point or early post-arrest limits generalizability, and so is also not an ideal solution. A growing body of prognostication research is published from regions where WLST-N is either prohibited or rarely occurs. The extent to which these data generalize internationally is uncertain. Moreover, treatment decisions short of WLST-N affect outcomes. For example, discharge of a patient with persistent unconsciousness and complex medical needs into the care of their family may still result in avoidable death.

Survival analysis allows models to learn from variable durations of patient observation while treating potential for recovery subsequent to WLST-N as unknown, offering a theoretical advantage in cases where quantifable treatment decisions prevent observation of ground truth outcomes. Time-to-event prediction has been implemented using many ML classification tools including random forest,³¹ naïve Bayes,³² and k-nearest neighbors.³³ We used LASSO regression because feature selection and a parsimonious model aid in global interpretability and clinical acceptability. We compared results to more traditional binary classification to explore the extent to which WLST-N might introduce bias. Consistent with the hypothesis that at least some patients exposed to WLST-N might otherwise have gone on to awaken, we found logistic regression led to more pessimistic outcome predictions among patients at high risk of WLST-N (Figure 4). This highlights the insidious risk of self-fulfilling prophecies in cases where biased modeling approaches result in pessimistic outcome estimates.

We also demonstrated the ability of early markers of post-arrest illness, particularly qEEG, to predict awakening. We included a wide array of traditional predictors of hypoxic-ischemic brain injury severity, as well as markers of shock, hepatic and renal dysfunction, sedative and analgesic infusions, and processes of care that may delay awakening irrespective of brain injury severity. Notably, virtually all predictors selected in each training set (Supplemental Tables) and the pooled UPMC model (Figure 2) were based on qEEG and models performed well even though awakening occurred on average 2 to 3 days later. Unsurprisingly, higher amplitudes (amplitude integrated EEG (aEEG), peak envelope) and conversely lower suppression ratio were strong predictors of awakening. After cardiac arrest, burst suppression may result from neuronal energetic failure due to primary and secondary brain injury, selective vulnerability of inhibitory cortical interneurons, and deafferentation, among other mechanisms.^34-38 A continuous cortical background on initial EEG implies the absence of these processes and potential for favorable recovery despite early coma.^16,21 Mild ischemia decreases presynaptic transmission, particularly in hypoxia-sensitive α-generating cortical layers IV and V.^39,40 Under sedation or anesthesia, the anterior shift in alpha power requires thalamocortical connectivity. Thus, preserved frontal alpha power suggests intact function of both cortical and subcortical networks.⁴¹ Spike count and standard deviation in aEEG tracing likely reflect epileptiform activity and paroxysmal high amplitude activity such as burst suppression with identical bursts, both negatively associated with awakening.^16,42 Rhythmicity in the low-delta range (<0.5Hz) and higher ranges are indirect measures of generalized suppression and more continuous cortical background activity, respectively.

Our analysis has important limitations. Operationalizing outcomes of post-arrest patients in clinical research is complex and awakening from coma is a crude measure. We focused on awakening because it is well-defined, a necessary first step in recovery, strongly protective against subsequent WLST-N,⁴³ and assessed daily at least in all included patients. However, awakening overlooks key domains of recovery including survival, function and health-related quality of life, all of which vary over time.⁴⁴ Long-term function (for example, modified Rankin Scale score at 6 months) may be of greater clinical relevance, but has limitations including loss to follow-up. Longer-term outcomes are also influenced by access to and quality of post-acute rehabilitation, social support and other factors that can act as unmeasured confounders. Based on our study aims, we accepted the limitations of awakening as an outcome to avoid obscuring a potential signature of self-fulfilling prophecies with these unmeasured confounders.

Regarding choice of candidate predictors, our focus was specifically on early clinical information. As such, we did not consider the full range of data acquired during post-arrest care. Together with our inclusion criteria, this ensured all predictors were ascertained before awakening or consideration of WLST-N. Prognostic value of both EEG and other factors varies over time, and we did not consider specific markers of poor outcome including somatosensory evoked potentials or pupillary light reflex at 72 hours. We included data within 6 hour of EEG initiation, rather than within a fixed window post-arrest, to reduce the impact of missing data from variable timing of clinical EEG acquisition. Since EEG evolves dynamically in the first 24 hours post-arrest, this may have increased uninformative variability in our qEEG predictors. On aggregate, these decisions may have reduced model performance.

Other aspects of our results also limit the immediate clinical applicability of our findings. Discrimination in our external UAB test set was lower than that observed internally, highlighting the potential for treating hospital to affect model results and need for broader external validation. It is unclear what specific factors were responsible for this difference. While patient hazard estimates were quite useful as predictors of eventual awakening, kernel-weighted survival curves did not yield clinically useful estimates of time to awakening. Although well-calibrated, precision was poor and interquartile ranges were broad. This may be due to unmeasured sources of between-patient variability or limited sample size. The moderate imbalance in our data, whereby only 29% of patients awakened, may lead to overly optimistic estimates of discrimination because even poorly performing models may learn to predict the more likely outcome class.

Finally, while comparing estimates from logistic and survival models provided indirect evidence of potential for bias introduced by WLST-N, this cannot be definitively confirmed. Moreover, while survival analysis may constitute a significant improvement in accounting for the risk of self-fulfilling prophecies, it is still prone to misestimation in cases where the positivity assumption is violated. Put differently, in cases that systematically undergo WLST-N, model predictions may not be reliable.

Neurological prognostication after resuscitation from cardiac arrest is challenging. While ML approaches are powerful, they may obfuscate biases introduced by WLST-N under the veneer of methodological rigor. Simply choosing not to study existing data sets out of concern for bias is unrealistic and would hobble scientific progress. We offer one alternative that relies on existing tools and has theoretical advantages. The potential for bias we identified in conventional binary outcome prediction highlights the need for ongoing development of rigorous approaches to detect and eliminate the risk of self-fulfilling prophecies in neuroprognostication and algorithmic prediction.

Key points.

Question:

In comatose survivors of cardiac arrest, does censoring observation after withdrawal of life-sustaining therapies for perceived poor neurological prognosis (WLST-N) affect the predicted probability of awakening from coma.

Findings:

We developed parsimonious models predicting time to awakening in comatose post-arrest patients. Among patients at high risk of WLST-N, censoring observation after WLST-N resulted in systematically more optimistic outcome estimates that standard logistic regression.

Meaning:

We identified risk of self-fulfilling prophecies using standard models to predicting awakening after cardiac arrest.