Accuracy of Frailty Instruments in Predicting Outcomes following Perioperative Cardiac Arrest

Abstract

Background:

Frailty is associated with increased 30-day mortality and non-home discharge following perioperative cardiac arrest. We estimated the predictive accuracy of frailty when added to baseline risk prediction models.

Methods:

In this retrospective cohort study using 2015–2020 NSQIP data for 3,048 patients aged 50+ undergoing non-cardiac surgery and resuscitation on post-operative day 0 (i.e., intraoperatively or postoperatively on the day of surgery), baseline models including age, sex, ASA physical status, preoperative sepsis or septic shock, and emergent surgery were compared to models that added frailty indices, either RAI or mFI-5, to predict 30-day mortality and non-home discharge. Predictive accuracy was characterized by area under the receiver operating characteristic curve (AUC-ROC), integrated calibration index (ICI), and continuous net reclassification index (NRI).

Results:

1,786 patients (58.6%) died in the study cohort within 30 days, and 38.6% of eligible patients experienced non-home discharge. The baseline model showed good discrimination (AUC 0.77 for 30-day mortality and 0.74 for non-home discharge). AUC-ROC and ICI did not significantly change after adding frailty for 30-day mortality or non-home discharge. Adding RAI significantly improved NRI for 30-day mortality and non-home discharge; however, the magnitude was small and difficult to interpret, given other results including false positive and negative rates showing no difference in predictive accuracy.

Conclusions:

Incorporating frailty did not significantly improve predictive accuracy of models for 30-day mortality and non-home discharge following perioperative resuscitation. Thus, demonstrated associations between frailty and outcomes of perioperative resuscitation may not translate into improved predictive accuracy. When engaging patients in shared decision-making regarding do-not-resuscitate orders perioperatively, providers should acknowledge uncertainty in anticipating resuscitation outcomes.

INTRODUCTION

Optimizing surgical care for older adults is a critical challenge.¹ In response to geriatric surgical patients’ needs, professional societies have prioritized preoperative risk stratification², tailored multidisciplinary care³, and decision-making that aligns treatment with patients’ goals^4–6.

Up to 50% of older surgical patients meet criteria for frailty⁷, a condition of decreased physiologic reserve. Frailty is associated with worse outcomes after surgery^8–16 as well as cardiopulmonary resuscitation (CPR) for in-hospital cardiac arrest^17–19. Moreover, frailty improves predictive accuracy for outcomes following non-cardiac surgery²⁰, emergency general surgery²¹, and abdominal surgery²². Such data support use of frailty for risk stratification and shared decision-making in those settings.

While evidence suggests that frailty is associated with worse outcomes after cardiac arrest in the perioperative period^23,24, the predictive accuracy of frailty in perioperative CPR is unknown. Quantifying the predictive value of frailty is necessary to guide shared decision-making regarding do-not-resuscitate (DNR) orders and appropriateness of perioperative CPR.²⁵ To determine the clinical significance of frailty as a predictor of perioperative CPR outcomes, we analyzed the predictive accuracy of frailty indices in predicting mortality and non-home discharge when added to a baseline prediction model.

METHODS

Data

We analyzed de-identified 2015–2020 American College of Surgeons National Surgical Quality Improvement Program (ACS-NSQIP) data, described previously.^10,12,23,26 Analysis was performed April 1, 2023 through January 12, 2024. This study adhered to the Strengthening the Reporting of Observational Studies in Epidemiology reporting guideline²⁷ and was deemed exempt from review and informed consent by the Mass General Brigham Institutional Review Board.

Inclusion Criteria

We included patients age ≥50 undergoing non-cardiac surgery who underwent CPR on postoperative day (POD) 0, i.e., intraoperatively or postoperatively on the day of surgery.^23,28 ACS-NSQIP²⁹ defines cardiac arrest as “the absence of cardiac rhythm or presence of chaotic cardiac rhythm…which results in a cardiac arrest requiring the initiation of CPR”. Patients were excluded only if data were missing for one or more fields necessary to determine frailty, establish outcome, or perform multivariable analyses.

Measurement of Frailty

Frailty was measured using the revised Risk Analysis Index (RAI)^30,31 and the 5-factor Modified Frailty Index (mFI-5)³². The revised RAI is designed for retrospective use with ACS-NSQIP variables, including age, sex, functional status, dyspnea, weight loss, malignant neoplasms, and other comorbidities. Its development, validation, accuracy, and feasibility have been described previously.^30,31,33,34 Missing cognition data was addressed by scoring all patients as having no cognitive decline, in order to conservatively bias results towards the null.³⁵ mFI-5 incorporates congestive heart failure, diabetes, chronic obstructive pulmonary disease or pneumonia, hypertension requiring medication, and partially or totally dependent functional health status.³²

Outcomes

The primary outcome was 30-day mortality following cardiac arrest. The secondary outcome was non-home discharge among survivors to discharge.

Baseline Model

The baseline model used multivariable logistic regression and incorporated exposures routinely available preoperatively and associated with increased mortality following perioperative CPR: age, sex, ASA physical status, preoperative sepsis/septic shock, and emergent (vs. urgent or elective) surgery. These variables were prespecified based on their clinical importance and existing literature.^28,36,37

Statistical Analysis

To assess the predictive value of RAI and mFI-5 for the outcomes, these frailty instruments were separately added to the baseline model using multivariable logistic regression. Frailty instruments were treated as continuous variables; restricted cubic splines were used for RAI, while mFI-5 was entered linearly. Predictive accuracy for the resulting models were compared to that of the baseline model. Predictive accuracy was characterized using discrimination as measured by the area under the receiver operating characteristic curve (AUC-ROC)³⁸, calibration as measured by the integrated calibration index (ICI)³⁹, and event reclassification as measured by the continuous net reclassification index (NRI)⁴⁰. 95% confidence intervals (CIs) were calculated using bootstrapping with 1000 replicates.

Predictive accuracy was additionally evaluated in subgroups by urgency of surgery and operative stress score (OSS)¹¹.

Sensitivity analyses parameterized RAI and mFI-5 as categorical instead of continuous variables, using RAI ≥40 and mFI-5 ≥2 as cutoffs.²³

All statistical hypotheses were two-sided, with no correction for multiple testing. Statistical analyses were performed using R version 4.3.0.

RESULTS

Among noncardiac surgeries for patients aged 50 or older in 2015–2020 ACS-NSQIP data, there were 3149 cases of cardiac arrest requiring CPR on POD 0. After excluding 101 patients due to missing data, 3048 patients remained in the study sample (Figure A.1).

Patient Characteristics

Patient characteristics are presented in Table 1. RAI scores ranged from 14 to 71 with mean 37.73 and SD 6.19. mFI-5 scores ranged from 0 to 5 with mean 1.19 and SD 0.92. RAI distribution by mortality is shown in Figure 1.

Table 1.

Patient characteristics by mortality outcome

	Overall n (%)	Dead at 30 days n (%)	Alive at 30 Days n (%)
Overall	3048	1786	1262
Age, Median [IQR], y	71 [63, 79]	73 [65, 81]	68 [60.25, 76]
50–64	912 (29.9)	442 (24.7)	470 (37.2)
65–74	982 (32.2)	565 (31.6)	417 (33.0)
75–84	764 (25.1)	497 (27.8)	267 (21.2)
≥ 85	390 (12.8)	282 (15.8)	108 (8.6)
Sex
Female	1346 (44.2)	806 (45.1)	540 (42.8)
Male	1702 (55.8)	980 (54.9)	722 (57.2)
Preoperative Frailty Score
RAI, mean [SD]	37.73 [6.19]	38.31 [6.46]	36.92 [5.70]
mFI-5, mean [SD]	1.19 [0.92]	1.23 [0.94]	1.13 [0.90]
Frail (binary category vs. non-frail)
Defined by RAI > 40	791 (26.0)	533 (29.8)	258 (20.4)
Defined by mFI-5 > 2	930 (30.5)	586 (32.8)	344 (27.3)
Functional status
Independent	2716 (89.1)	1549 (86.7)	1167 (92.5)
Partially dependent	265 (8.7)	184 (10.3)	81 (6.4)
Totally dependent	67 (2.2)	53 (3.0)	14 (1.1)
Congestive Heart Failure	244 (8.0)	161 (9.0)	83 (6.6)
Weight loss	130 (4.3)	86 (4.8)	44 (3.5)
Diabetes	418 (13.7)	223 (12.5)	195 (15.5)
Dyspnea	81 (2.7)	56 (3.1)	25 (2.0)
Renal Failure	351 (11.5)	239 (13.4)	112 (8.9)
Cancer	158 (5.2)	102 (5.7)	56 (4.4)
Preoperative Sepsis
None	2106 (69.1)	1050 (58.8)	1056 (83.7)
Sepsis/SIRS	567 (18.6)	403 (22.6)	164 (13.0)
Septic Shock	375 (12.3)	333 (18.6)	42 (3.3)
ASA Physical Status
1 and 2	292 (9.6)	76 (4.3)	216 (17.1)
3	1158 (38.0)	516 (28.9)	642 (50.9)
4	1135 (37.2)	769 (43.1)	366 (29.0)
5	463 (15.2)	425 (23.8)	38 (3.0)
Procedure Urgency
Emergency Surgery	1189 (39.0)	971 (54.4)	218 (17.3)
Urgent Surgery	616 (20.2)	360 (20.2)	256 (20.3)
Elective Surgery	1243 (40.8)	455 (25.5)	788 (62.4)
Operative Stress Scorea
Low (1–2)	493 (17.1)	207 (12.3)	286 (23.8)
Moderate (3)	1469 (50.8)	838 (49.6)	631 (52.5)
High (4–5)	929 (32.1)	643 (38.1)	286 (23.8)

^aN for operative stress score subgroups sum to 2,891 (5.2% missing)

Abbreviations: RAI = Risk Analysis Index; mFI-5 = 5-factor Modified Frailty Index; ASA = American Society of Anesthesiologists; SIRS = systemic inflammatory response syndrome

Figure 1.

Distribution of Risk Analysis Index (RAI) frailty scores by mortality outcome

Figure 1 presents a box-and-whisker plot of Risk Analysis Index (RAI) frailty scores in our study sample by mortality status at 30 days post-perioperative cardiac arrest. The box displays Q1 (1^st quartile/25^th percentile), the median (2^nd quartile/50^th percentile), and Q3 (3^rd quartile/75^th percentile). The upper whisker extends from Q3 to the largest data point no further than 1.5*IQR from Q3 (where IQR=Q3-Q1), and similarly, the lower whisker extends from Q1 to the smallest data point no further than 1.5*IQR from Q1. The individual points plotted beyond the whiskers represent outliers.

Mortality

1,786 patients (58.6%) died within 30 days following CPR.

Non-home Discharge

Among 1,160 patients admitted from home and who survived to discharge, 448 (38.6%) were discharged to a non-home location.

Predictive Accuracy of Frailty

Age, septic shock, ASA class, and emergent surgery were associated with worse outcomes before and after adding frailty (Table A.1).

Table 2 presents the changes in predictive accuracy after adding frailty. The baseline model showed good discrimination, with an AUC of 0.77 for 30-day mortality and 0.74 for non-home discharge. Neither AUC-ROC nor ICI were affected by addition of frailty to the baseline model for either mortality or non-home discharge. NRI improved with addition of RAI to the baseline model for both 30-day mortality and non-home discharge, but the magnitude of these differences was small. Moreover, false positive and false negative rates did not differ much after adding frailty, and numbers of reclassification errors after adding frailty were similar between events and non-events.

Table 2.

Changes in predictive accuracy for 30-day mortality and non-home discharge with addition of frailty indices to base prediction model

	Base	Base + RAI	Base + mFI-5
30-day Mortality
AUC	0.77 (0.76, 0.79)	0.78 (0.76, 0.79)	0.77 (0.76, 0.79)
ICI	0.01 (0.00, 0.02)	0.01 (0.00, 0.02)	0.01 (0.00, 0.02)
Reclassification
Events (%)		−40	−10
Non-events (%)		57	21
NRI		0.17 (0.06, 0.35)	0.09 (−0.09, 0.19)
TPR (%)	74	75	74
FPR (%)	35	34	35
TNR (%)	65	66	65
FNR (%)	26	25	26
Non-home Discharge
AUC	0.74 (0.71, 0.77)	0.75 (0.72, 0.78)	0.75 (0.72, 0.77)
ICI	0.02 (0.01, 0.04)	0.03 (0.01, 0.05)	0.02 (0.01, 0.04)
Reclassification
Events (%)		−32	−12
Non-events (%)		56	31
NRI		0.24 (0.10, 0.36)	0.18 (−0.02, 0.31)
TPR (%)	45	45	46
FPR (%)	12	12	12
TNR (%)	88	88	88
FNR (%)	55	55	54

Abbreviations: RAI = Risk Analysis Index; mFI-5 = 5-factor Modified Frailty Index; AUC = area under the receiver operating curve; ICI = integrated calibration index; NRI = continuous net reclassification index; TPR = true positive rate; FPR = false positive rate; TNR = true negative rate; FNR = false negative rate.

Subgroup Analyses

Results were similar when stratified by surgical urgency or OSS (Table 3).

Table 3.

Change in area under the receiver operating characteristic curve (AUC) with 95% confidence intervals for 30-day mortality with base prediction model and with addition of frailty indices, by surgical subgroup

	Overall n	Mortality n (%)	Base	Base + RAI	Base + mFI-5
Surgical Urgency
Elective	1,243	455 (36.6)	0.63 (0.60, 0.66)	0.64 (0.61, 0.67)	0.63 (0.60, 0.66)
Urgent	616	360 (58.4)	0.67 (0.62, 0.71)	0.68 (0.63, 0.72)	0.67 (0.63, 0.71)
Emergent	1,189	971 (81.7)	0.74 (0.70, 0.77)	0.74 (0.71, 0.78)	0.74 (0.71, 0.77)
Operative Stress Score
Low (1–2)	493	207 (42.0)	0.78 (0.74, 0.82)	0.79 (0.75, 0.83)	0.79 (0.74, 0.83)
Moderate (3)	1,469	838 (57.0)	0.77 (0.74, 0.79)	0.77 (0.75, 0.79)	0.77 (0.75, 0.79)
High (4–5)	929	643 (69.2)	0.78 (0.74, 0.81)	0.78 (0.75, 0.81)	0.78 (0.75, 0.81)

Abbreviations: RAI = Risk Analysis Index; mFI-5 = 5-factor Modified Frailty Index; AUC = area under the receiver operating curve.

Sensitivity Analyses

Results were not sensitive to adding frailty indices as binary, instead of continuous, variables (Table A.2).

DISCUSSION

In this cohort study of patients who underwent perioperative CPR, we found that addition of frailty indices did not increase the predictive accuracy of models predicting 30-day mortality and non-home discharge. These results add to evidence on frailty’s significance in the perioperative period^20,21,23 and have implications for risk stratification, decision-making regarding perioperative CPR, and future research to support goal-concordant management of complications of surgery and anesthesia.

Given the growing focus on delivering care tailored to patients’ unique vulnerabilities and priorities, the predictive performance of clinical characteristics warrants careful study to support evidence-based risk stratification and personalized decision-making. Although a prior study²³ identified a population-level association between frailty and worse outcomes following perioperative CPR, current analysis suggests frailty does not augment predictive accuracy when added to recognized risk factors including ASA status, age, sepsis, and emergency surgery.⁴¹ Our findings suggest frailty should not in itself be considered an important basis for predicting who is likely to survive or return home following perioperative cardiac arrest.

There are several possible explanations for these results. One is methodological. Whereas prior explanatory models excluded age and sex due to collinearity with frailty^10,12,23, the current investigation included these variables in the baseline risk model given their clinical salience and incorporation in other similar risk prediction models^20,21. Second, “cardiac arrest” is nonspecific, encompassing many physiological etiologies with differing prognoses that may vary by time of arrest.⁴² And finally, “resuscitation” is no less ambiguous given the range of interventions that may qualify as CPR and their variable association with complications. It is plausible that frailty is predictive of worse outcomes in specific circumstances (echoing variable associations between frailty and surgical outcomes depending on surgical risk and urgency^9–11,14,16), but that such relationships are masked by a preponderance of events in which outcomes are driven by other factors. Overall, these considerations suggest a need for providers to acknowledge uncertainty in anticipating outcomes following perioperative CPR and to avoid overestimating the predictive accuracy of frailty in this setting.

Our findings build on prior studies demonstrating favorable predictive performance of frailty in predicting outcomes following noncardiac surgery and emergency general surgery.^20,21 Although evidence suggests frailty does offer value in predicting outcomes following non cardiac surgery and emergency general surgery^20,21, our findings indicate that frailty may not have the same predictive value in anticipating clinical trajectory following a major complicationsuch as cardiac arrest. Risk stratification following major complications is an important goal for guiding shared decision-making in high-risk surgical patients4 and guiding implementation of primary prevention strategies^23,33.

This study has several strengths. Building on prior work, this cohort is the largest to evaluate the relationship between frailty and CPR outcomes in any setting, enabling us to control for confounders and perform subgroup analyses by level of surgical stress and urgency. We used a national database including over 700 hospitals and the most thoroughly validated NSQIP-based frailty measure, thus optimizing external validity.

There are also several limitations. AUC-ROC in our results were only in the range of “good”, highlighting a need for improved risk prediction in this context. AUC-ROC is independent of prevalence and misclassification costs and can be skewed when outcomes are rare^38,43; however, mortality in our cohort was 58.6%. NRI is difficult to interpret with poorly-defined clinical utility.^40,44 ACS-NSQIP does not include arrest features, and the possibility that frailty is predictive of worse outcomes in specific arrest circumstances cannot be ruled out. We likewise cannot rule out the possibility that other frailty indices (e.g., the Clinical Frailty Scale or Frailty Index) may augment predictive accuracy in this context. Our sample, by definition, did not include those with active DNR orders or those who were deemed ineligible for surgery, so it is possible that frailty is predictive of arrest or resuscitation outcomes in other patient samples. It is also possible that frailty is predictive of neurological function or other, more patient-centered outcomes (e.g., functional trajectory and days alive at home), and these should be studied.

In this cohort study of patients who underwent perioperative CPR, we found that the addition of frailty indices did not increase the predictive accuracy of models predicting 30-day mortality and non-home discharge. Further research is needed to inform individualized risk stratification as a basis for goal-concordant management of complications of surgery and anesthesia.

APPENDIX A

Figure A.1.

Study Flow Diagram

Abbreviations: RAI = Risk Analysis Index; ASA = American Society of Anesthesiologists; POD = postoperative day

Table A.1.

Odds ratios (ORs) demonstrating association between model variables and outcomes of 30-day mortality and non-home discharge

	Outcome: Mortality OR (95% CI)			Outcome: Non-home Discharge OR (95% CI)
	Base Model	Base Model + RAI	Base Model + mFI-5	Base Model	Base Model + RAI	Base Model + mFI-5
Age 50–64	Ref	Ref	Ref	Ref	Ref	Ref
Age 65–74	1.39 (1.13, 1.70)	1.43 (1.16, 1.77)	1.38 (1.12, 1.70)	1.79 (1.31, 2.45)	1.79 (1.30, 2.48)	1.76 (1.29, 2.42)
Age 75–84	1.94 (1.56, 2.42)	2.05 (1.63, 2.58)	1.93 (1.54, 2.41)	2.22 (1.56, 3.15)	2.21 (1.54, 3.18)	2.19 (1.54, 3.12)
Age 85+	2.82 (2.13, 3.75)	2.97 (2.21, 4.00)	2.79 (2.11, 3.71)	5.44 (3.29, 9.17)	5.09 (3.02, 8.73)	5.27 (3.18, 8.90)
Female	Ref	Ref	Ref	Ref	Ref	Ref
Male	0.90 (0.76, 1.06)	0.93 (0.78, 1.11)	0.90 (0.76, 1.06)	0.66 (0.50, 0.86)	0.62 (0.47, 0.82)	0.66 (0.51, 0.86)
No sepsis/SIRS or septic shock	Ref	Ref	Ref	Ref	Ref	Ref
Sepsis/SIRS	1.26 (1.00, 1.59)	1.21 (0.96, 1.54)	1.24 (0.99, 1.57)	2.89 (1.87, 4.52)	2.69 (1.73, 4.23)	2.82 (1.82, 4.42)
Septic Shock	2.90 (2.01, 4.26)	2.80 (1.93, 4.12)	2.88 (2.00, 4.23)	4.22 (1.74, 11.46)	3.53 (1.44, 9.65)	4.06 (1.66, 11.06)
ASA 1 or 2	Ref	Ref	Ref	Ref	Ref	Ref
ASA 3	1.96 (1.46, 2.64)	1.93 (1.44, 2.61)	1.92 (1.43, 2.60)	2.45 (1.65, 3.70)	2.34 (1.58, 3.54)	2.21 (1.48, 3.36)
ASA 4	3.40 (2.51, 4.65)	3.27 (2.39, 4.48)	3.28 (2.39, 4.53)	3.68 (2.38, 5.77)	3.07 (1.96, 4.89)	3.04 (1.92, 4.87)
ASA 5	11.42 (7.26, 18.30)	11.38 (7.23, 18.26)	11.22 (7.12, 18.01)	1.61 (0.65, 3.97)	1.56 (0.63, 3.86)	1.50 (0.61, 3.70)
Elective or Urgent Surgery	Ref	Ref	Ref	Ref	Ref	Ref
Emergent Surgery	2.28 (1.83, 2.85)	2.36 (1.89, 2.95)	2.32 (1.86, 2.90)	1.85 (1.19, 2.86)	1.96 (1.26, 3.06)	1.91 (1.23, 2.97)
RAI	-	1.04 (0.87, 1.24)	-	-	1.45 (1.08, 1.95)	-
(90th percentile vs. 10th)
mFI-5 (90th percentile vs. 10th)	-	-	1.09 (0.90, 1.31)	-	-	1.55 (1.13, 2.13)

Abbreviations: RAI = Risk Analysis Index; mFI-5 = 5-factor Modified Frailty Index; ASA = American Society of Anesthesiologists; SIRS = systemic inflammatory response syndrome.

Table A.2.

Changes in predictive accuracy for 30-day mortality and non-home discharge with addition of frailty indices as binary variables to base prediction model

	Base	Base + RAI	Base + mFI-5
30-day Mortality
AUC	0.77 (0.76, 0.79)	0.77 (0.76, 0.79)	0.77 (0.76, 0.79)
ICI	0.01 (0.00, 0.02)	0.01 (0.00, 0.02)	0.01 (0.00, 0.02)
Reclassification
Events (%)		−38	−33
Non-events (%)		56	44
NRI		0.18 (−0.14, 0.24)	0.10 (−0.09, 0.17)
TPR (%)	74	74	74
FPR (%)	35	34	34
TNR (%)	65	66	66
FNR (%)	26	26	26
Non-home Discharge
AUC	0.74 (0.71, 0.77)	0.75 (0.72, 0.78)	0.75 (0.72, 0.78)
ICI	0.02 (0.01, 0.04)	0.03 (0.01, 0.04)	0.02 (0.01, 0.04)
Reclassification
Events (%)		−37	−28
Non-events (%)		68	58
NRI		0.30 (0.18, 0.41)	0.29 (0.09, 0.42)
TPR (%)	45	46	46
FPR (%)	12	13	13
TNR (%)	88	87	87
FNR (%)	55	54	54

Abbreviations: RAI = Risk Analysis Index; mFI-5 = 5-factor Modified Frailty Index; AUC = area under the receiver operating curve; ICI = integrated calibration index; NRI = continuous net reclassification index; TPR = true positive rate; FPR = false positive rate; TNR = true negative rate; FNR = false negative rate.