Assessment of Racial Bias within the Risk Analysis Index of Frailty

Abstract

Objective:

Our objective was to assess potential racial bias within the Risk Analysis Index (RAI).

Background:

Patient risk measures are rarely tested for racial bias. Measures of frailty, like the RAI, need to be evaluated for poor predictive performance among Black patients.

Methods:

Retrospective cohort study using April 2010–March 2019 Veterans Affairs Surgical Quality Improvement Program and 2010–2019 National Surgical Quality Improvement Program data. The performance of the RAI and several potential variants were compared between Black and White cases using various metrics to predict mortality (180-day for Veterans Affairs Surgical Quality Improvement Program, 30-day for National Surgical Quality Improvement Program).

Results:

Using the current, clinical threshold, the RAI performed as good or better among Black cases across various performance metrics versus White. When a higher threshold was used, Black cases had higher true positive rates but lower true negative rates, yielding 2.0% higher balanced accuracy. No RAI variant noticeably eliminated bias, improved parity across both true positives and true negatives, or improved overall model performance.

Conclusions:

The RAI tends to predict mortality among Black patients better than it predicts mortality among White patients. As existing bias-reducing techniques were not effective, further research into bias-reducing techniques is needed, especially for clinical risk predictions. We recommend using the RAI for both statistical analysis of surgical cohorts and quality improvement programs, such as the Surgical Pause.

INTRODUCTION

Patient risk measures, such as comorbidity indices, are commonly used to assess and improve healthcare quality, allocate additional resources to patients in need, and adjust medical service payments. While these indices can be used to improve outcomes or reduce cost,¹ they can also manifest unintended racial bias.² Racial bias can occur in any predictive algorithm, driven by several possible causes. Training models on a biased outcome can drive racial disparities; for example, healthcare cost models often underestimate the healthcare risks of Black patients because such patients often have reduced access to care that artificially lowers their healthcare costs.² In addition, social risk factors can drive race-based differences in relationships between predictor variables and outcomes. The result of this phenomenon is often underappreciated: when any statistical model is calibrated across varying social groups (eg, race), the model will estimate coefficients closer to those of the larger group and make less accurate predictions for subgroups with fewer observations.³ Worse, these two, distinct sources of bias can co-occur.

Frailty is a syndrome of decreased physiologic reserve characterized by physical weakness, exhaustion, and low or slow activity levels.⁴ Although frailty often includes measures of comorbidity, its success in predicting surgical outcomes is largely due to its focus on functional performance assessment.^5,6 One of the most thoroughly validated measures of surgical frailty is the Risk Analysis Index (RAI),^7–12 which outperforms the now obsolete modified Frailty Index in surgical cohorts,¹³ and has been shown to improve perioperative outcomes in programs for identifying and mitigating perioperative frailty.^14,15 The RAI comes in 3 versions: (1) a clinical questionnaire,⁷ (2) an administrative score using variables from surgical registry data (Veterans Affairs Surgical Quality Improvement Program [VASQIP] and the American College of Surgeons’ National Surgical Quality Improvement Program),⁷ and (3) an index scoring International Classification of Diseases-10 codes.¹⁶ Based on converging evidence from all 3 versions,^7–12 programs are increasingly using a clinical questionnaire version of the RAI to guide perioperative decision-making. For example, Epic makes the RAI available to any client worldwide as an official clinical program,¹⁷ and the Veterans Health Administration’s National Surgery Office adopted the RAI-based Surgical Pause program at the national level in 2023.¹⁸ In the Surgical Pause, all patients being evaluated for surgery are screened for frailty using a clinical questionnaire RAI, with those in the highest risk decile⁷ directed to preoperative goal clarification or prehabilitation.¹⁹

With more than 1 million patients screened across more than 50 Veterans Affairs (VA) and private sector medical centers, the Joint Commission and the National Quality Forum recognized the Surgical Pause with the 2023 John M. Eisenberg Award for Patient Safety and Quality at a National Level,²⁰ setting the stage for even wider adoption of the Surgical Pause and the RAI. However, potential algorithmic bias in frailty measures, including the RAI, has not been properly assessed. Given the evidence that Black patients, compared with their White counterparts, have higher illness burdens at younger ages,²¹ have increased risk of death^22,23 from disproportionate exposure to chronic stressors (eg, “weathering”), and that such weathering has driven bias in other comorbidity measures,^2,24 the possibility of racial bias within the RAI requires investigation.

Using high-quality VASQIP data,²⁵ we designed this study to quantify and compare model performance between the administrative RAI and 3 modified versions intended to reduce racial bias between Black and White patients. We sought to quantify any Black-White racial bias present within the existing RAI and assess whether any commonly used technique was useful to alleviate such bias.

METHODS

Study Design and Cohort

The study was determined to be exempt by the VA Pittsburgh Healthcare System’s Institutional Review Board and reported according to Strengthening the Reporting of Observational Studies in Epidemiology guidelines.²⁶ The cohort included VASQIP cases performed between April 2010 and March 2019 and was a subset of the cohort used to calibrate the RAI.⁷ VASQIP is a quality assessment database containing nurse-abstracted data on VA surgeries, using methods described elsewhere.²⁵ VASQIP provides superior capture of patient risk factors and outcomes²⁷ and tracks 180-day mortality. We excluded cases that had missing race/ethnicity (15.3%), were not Black or White (6.2%), or were missing variables needed to calculate the RAI (0.003%, 78.4% of cases remaining). Because surgical mortality is a rare outcome, the Asian/Pacific Islander and Native American groups lacked adequate cases with 180-day mortality to adequately assess the RAI’s performance (32 and 80 cases, respectively).

Outcome

The VASQIP variable DWIN6MO was used for 180-day mortality. DWIN6MO is derived from various sources, including VA sources, Medicare, and Social Security Death Master File, to capture patient mortality more completely.

Exposure

The administrative RAI⁷ was calculated with preoperative VASQIP variables. Prior work demonstrates that the RAI provides a composite measure of patient-related risk that avoids problems with model fit associated with models that include each of the underlying VASQIP variables.^8,10

Bias Mitigation Strategies

To assess possible solutions to any detected racial bias, we recalibrated the base RAI scoring system (RAI_Base) using 3 approaches used to minimize bias in other algorithms: (1) balancing the sample by reweighting racial strata such that Black and White patients were equally represented (ie, 50/50),²⁸ (2) matching the mean age of Black patients to the mean age of White patients (ie, adding 3.5 years to each Black patient’s age),²⁸ and (3) creating a new RAI scoring system using a penalized form of logistic regression designed to balance false positive and false negative rates between the 2 groups.²⁹ For each approach, we used the dataset from the original RAI recalibration analysis, re-solving the regression equation for new parameter estimates, and generating predicted probabilities of 180-day mortality for the 3 RAI variants: the (1) reweighted balanced-sample RAI (RAI_Balanced), (2) age-adjusted RAI (RAI_Age-adjusted), and (3) Fair RAI (RAI_Fair).

Bias Quantification and Comparison

Logistic regression models between each version of the RAI and 180-day mortality, with fixed effects adjustment for the VA site, were performed on the entire cohort. C-statistics were calculated for each model to evaluate statistical discrimination. To generate dichotomized predictions of mortality, several thresholds were assessed: (1) an RAI value of 30 with a 6.7% predicted probability of 180-day mortality (used by the Surgical Pause program), (2) an RAI value of 36 with a 15% predicted probability (phenotypically obvious frailty), and (3) an RAI value of 26 with a 3.5% predicted probability (cutoff for optimal sensitivity and specificity³⁰). After ensuring that the optimal threshold (RAI = 26) was similar in the models generated by each of the 3 bias mitigation strategies, the cohort was then split into Black and White patients, with the sensitivity, specificity, false positive and false negative rates, accuracy, balanced accuracy,³¹ Matthew’s correlation coefficient,³² and F1 score³³ calculated separately for each race group. We compared the performance metrics listed above, stratified by Black and White subgroups. In addition, and following methods described elsewhere,³⁴ we calculated metrics of equal opportunity (the ratio of true positives in Black and White samples) and predictive equality (the ratio of true negatives in Black and White samples). The fairness criterion of equalized odds is defined as having both equal opportunity and predictive equality. Finally, we examined separate receiver operating characteristic (ROC) curves for Black and White patients in the RAI_Base to characterize its predictive performance between the 2 groups. Analyses were performed using R 4.3.1, with fair regression implemented using a Python script²⁹ shared by O.M. and B.Z. in the acknowledgments and adapted by authors R.M.H. and M.A.J.

Sensitivity Analysis

Sensitivity analyses replicating the approach described above were conducted in a sample of National Surgical Quality Improvement Program (NSQIP) cases to examine if findings were similar in a national-level, private sector sample of surgical cases. We examined the same years (2010–2019), analyzed only the RAI_Base, and selected a threshold of 2.3% predicted probability of 30-day mortality (RAI = 30). We examined performance metrics between Black and White cases, and separate ROC curves, as described above.

RESULTS

Patient Demographics

Analyses included 377,107 surgical cases (Table 1), 92.3% male with mean (SD) age of 60.8 (12.9). Black patients represented 18.6% of the sample. Most White cases had normal frailty (RAI 21–29, 56.0%), followed by robust (RAI ≤20, 33.5%), frail (RAI 30–39, 8.2%), and very frail (RAI ≥40, 2.3%), while most Black cases had robust frailty (50.2%), followed by normal (38.7%), frail (7.8%), and very frail (3.2%). RAI components with noticeable racial differences were renal failure (1.2% in White cases vs. 3.7% in Black cases, P < 0.001), cognitive deterioration (3.4% in White cases vs. 4.6% in Black cases, P < 0.001), and functional status (7.6% partially or totally dependent White cases vs. 9.0% Black cases, P < 0.001).

TABLE 1.

Cohort Demographics

	Overall	Black	White	P value
Number (%)*	377,107	70,301 (18.6)	306,806 (81.4)
Age mean [SD]	60.8 [12.9]	58.0 [12.1]	61.5 [13.0]	<0.001
Male	347,912 (92.3)	61,745 (87.8)	286,167 (93.3)	<0.001
Cancer	7,755 (2.1)	1,535 (2.2)	6,220 (2.0)	0.009
Weight loss	10,141 (2.7)	2,270 (3.2)	7,871 (2.6)	<0.001
Renal failure	6,185 (1.6)	2,617 (3.7)	3,568 (1.2)	<0.001
Congestive heart failure	2,645 (0.7)	565 (0.8)	2,080 (0.7)	<0.001
Dyspnea				<0.001
None	330,746 (87.7)	63,106 (89.8)	267,640 (87.2)
At minimal exertion	42,194 (11.2)	6,445 (9.2)	35,749 (11.7)
At rest	4,167 (1.1)	750 (1.1)	3,417 (1.1)
Admission from				<0.001
Admitted from home	366,018 (97.1)	67,817 (96.5)	298,201 (97.2)
Acute care hospital	4,282 (1.1)	891 (1.3)	3,391 (1.1)
Nursing home	5,635 (1.5)	1,256 (1.8)	4,379 (1.4)
Other	1,172 (0.3)	337 (0.5)	835 (0.3)
Cognitive deterioration	13,657 (3.6)	3,258 (4.6)	10,399 (3.4)	<0.001
Functional status				<0.001
Independent	348,744 (92.5)	63,972 (91.0)	284,722 (92.8)
Partially dependent	22,100 (5.9)	4,713 (6.7)	17,387 (5.7)
Totally dependent	6,263 (1.7)	1,616 (2.3)	4,647 (1.5)
RAI mean [SD]	21.7 [7.4]	20.8 [8.0]	21.9 [7.3]	<0.001
RAI				<0.001
Robust (≤20)	138,068 (36.6)	35,312 (50.2)	102,756 (33.5)
Normal (21–29)	199,125 (52.8)	27,235 (38.7)	171,890 (56.0)
Frail (30–39)	30,669 (8.1)	5,484 (7.8)	25,185 (8.2)
Very frail (≥40)	9,245 (2.5)	2,270 (3.2)	6,975 (2.3)
30-day mortality	4,195 (1.1)	806 (1.1)	3,389 (1.1)	0.350
6-month mortality	13,870 (3.7)	2,617 (3.7)	11,253 (3.7)	0.493
1-year mortality	22,212 (5.9)	3,959 (5.6)	18,253 (5.9)	0.001

^*Percentages are by row, the rest are by column.

RAI indicates Risk Analysis Index; SD, standard deviation.

Bias Assessment of the RAI_Base

Race-stratified model performance metrics are tabulated for the RAI_Base and 3 RAI variants at the RAI 30 threshold (Fig. 1) and RAI 36 threshold (Fig. 3). Bias is quantified as the difference in each model performance metric between Black and White cases. Green and red shading represent bias in favor of Black or White cases, respectively, whereas gray shading represents the absence of bias. At the RAI 30 threshold, the RAI_Base performed as good or better among Black cases across all performance metrics versus White, with a 2.8% higher true positive rate, equivalent true negative rate, and 1.4% greater balanced accuracy (Fig. 1). At the RAI 36 threshold, the patterns of bias are more pronounced: Black cases had 4.5% higher true positive rates but 0.5% lower true negative rates, yielding 2.0% higher balanced accuracy. The differences between the 2 thresholds are further illustrated by the ROC curves, which favor Black cases across most RAI thresholds (Fig. 2A), but converge and cross at higher RAI values (Fig. 2B).

Figure 1.

Measures of predictive performance between Black and White cases for the threshold of detectable frailty (RAI 30, 6.7% predicted 180-day mortality).

Figure 3.

Measures of predictive performance between Black and White cases for the threshold of phenotypically obvious frailty (RAI 36, 15% predicted 180-day mortality).

Figure 2.

Receiver operating curves for RAI base among White and Black cases. RAI indicates Risk Analysis Index.

Comparison of Model Performance and Bias Across the RAI Variants

Differences in model performance between the RAI variants are quantified for both thresholds in Figure 1 and Figure 3, where green and orange shading represent increased and decreased performance, respectively, in each metric compared to the RAI_Base. At the RAI 30 threshold (Fig. 1), the RAI_Balanced demonstrated discrimination equivalent to the RAI_Base (c = 0.837) with a slightly higher balanced accuracy among both Black (Δ_Acc = 0.1%) and White (Δ_Acc = 0.2%) cases, producing a simultaneous increase in the true positive rate and decrease in true negative rate. However, these differences were small. Changes in model performance for the RAI_Age-adjusted were more complex: Overall discrimination increased (c = 0.878), but the balanced accuracy among White cases decreased 0.3%, representing increased disparity between Black and White cases. Finally, the RAI_Fair demonstrated the lowest overall discrimination (c = 0.765) with large reductions in balanced accuracy for both White (Δ_BalAcc = −8.1%) and Black (Δ_BalAcc = −10.0%) cases, effectively reversing the direction of bias—Black cases had worse detection of true positives and better detection of true negatives.

At the threshold of RAI 36 (Fig. 3), the magnitude of bias between Black and White cases was both larger and more heterogeneous. For example, Black cases now had better detection of true positives (Δ_Sens = 4.5%) and worse detection of true negatives (Δ_Spec = −0.5%). The bias between Black and White cases changed direction and magnitude across the 3 RAI variants, with RAI_Fair demonstrating the least biased equal opportunity ratio (0.90), but with better true positive detection in White cases than Black.

Results at the threshold of RAI 26 are provided in the Supplement (Supplemental Table 1, http://links.lww.com/AOSO/A401).

Sensitivity Analyses at the Threshold that Optimizes Sensitivity and Specificity

At the threshold jointly maximizing sensitivity and specificity, model metrics showed similar patterns to those at the RAI 30 threshold (Fig. 1) where the RAI_Base performed as well or better among Black cases versus White across all performance metrics—consistent with the ROC curves in Figure 2. At this threshold, the equal opportunity and predictive equality ratios were 1.01 and 1.05, respectively, and none of the RAI variants effectively reduced the bias favoring Black cases.

Sensitivity analysis of NSQIP data

Examining an RAI 30 threshold in NSQIP data, Black cases had better specificity (1.5%) but worse sensitivity (−1.3%), yielding similar balanced accuracies (73.3% in Black cases vs. 73.0% in White) and better accuracy in Black cases (1.5%, Supplemental Table 2, http://links.lww.com/AOSO/A401). At this threshold, the equal opportunity and predictive equality ratios were 0.96 and 1.02, respectively. The ROC curve showed better predictive performance in Black cases across most RAI thresholds (Supplemental Figure 1, http://links.lww.com/AOSO/A401). However, the ROC curves for Black and White cases crossed at the upper range of RAI values.

DISCUSSION

Given the wide and expanding application of the RAI in clinical practice, this study was designed to probe for any algorithmic bias according to race. We found that, at the clinical cutoff used by the Surgical Pause program (RAI 30), the RAI_Base demonstrated a slight algorithmic bias in favor of Black cases. None of the other RAI variants noticeably eliminated bias, improved parity across both true positives and true negatives, or improved overall model performance. The pattern favoring Black cases remained similar at various thresholds. We consider the minor differences found to be reassuring, and our data suggest that the RAI can continue to be used for preoperative decision-making. However, the ROC curves for Black and White cases crossed at the upper range of RAI values, suggesting that certain clinical thresholds could yield predictions that disadvantage Black patients. We recommend the existing threshold of 30 for the VASQIP-derived RAI, which corresponds to a threshold of 37 for the clinical questionnaire version of the RAI currently used in Surgical Pause screening.⁷ Although our data limited authoritative comparisons for non-Black minority groups, absent compelling evidence of poor RAI performance in these groups, we consider the benefits of the RAI to outweigh the potential risks (especially considering that RAI-based frailty screening can produce an almost threefold reduction in postoperative mortality).¹⁵

Ideally, bias mitigation strategies should reduce the advantage observed in 1 group, not degrade overall model performance for the disadvantaged group, and performance across groups should not be markedly worse. Our data demonstrate that this is not always possible. For example, the RAI_Fair had markedly worse true positive rates than the RAI_Base among both White and Black cases, making this model worse for both races. Although the RAI_Balanced had slightly improved true positive rates among Black and White cases, the true negative rates were lower in both racial groups, and a small bias emerged in favor of Black patients. Although an argument could be made to use the RAI_Balanced because of its slightly improved equal opportunity ratio, we contend that the change is too small to warrant revising the scoring systems currently in use. These findings suggest that RAI_Base remains acceptable for use in both statistical analysis of surgical cohorts and quality improvement programs, such as the Surgical Pause—at least for patients who are White or Black.

To the extent that the RAI_Base had better predictive performance, particularly in sensitivity, among Black patients versus White, the RAI increases the likelihood that Black patients receive appropriate prehabilitation and accurate counseling about surgical risk—which may help alleviate racial disparities in surgical outcomes. Algorithms should include race/ethnicity as a predictor when it increases the chance of appropriate care for an at-risk group but not when it reduces their chance of appropriate care.³⁵ In our case, while race or ethnicity are not included in the RAI, Black patients have an increased opportunity for appropriate treatment when the RAI is used.

Our findings highlight issues with “bias-reducing” statistical methods. Namely, in the pursuit of equity, these methods can worsen a model’s predictive ability.^36,37 In our case, the RAI_Fair achieved higher specificity in Black patients, but drastically reduced sensitivity for both Black and White patients. Such changes are in direct contrast with the express purpose of the RAI, to accurately capture frailty in patients considering surgery and enable either preoperative intervention or alternative, nonsurgical treatment.¹⁵ The impact of bias mitigation strategies also varies across model performance metrics, which highlights the importance of defining the most clinically relevant metric. For example, if priority is placed on identifying frailty and mitigating its associated risks, efforts should be focused on reducing sensitivity bias, even at the expense of specificity. Alternatively, if the priority is placed on reducing the inconvenience, expense, and potential harm of falsely categorizing patients as frail, then efforts should focus on maximizing specificity. Global measures of performance, such as balanced accuracy, F1, or MCC, may not always reflect what is of greatest clinical importance.

Concerns about racial bias in predictive algorithms are well-founded and healthcare algorithms need to be screened for potential racial disparity. To our knowledge, the RAI is the first frailty measure to be assessed for racial bias, and various nonfrailty comorbidity indices, such as the Charlson,³⁸ Elixhauser,³⁹ and Gagne⁴⁰ scores, have also not yet been tested for racial bias. Until such testing occurs, we advise caution when adjusting for comorbidity indices to estimate the causal effects of racial discrimination, social risk factors, or any nonclinical risk factor heavily related to race. Without definitive evidence of racial bias or its absence, researchers should perform sensitivity analyses with and without comorbidity adjustment when investigating such factors.

More broadly, meaningful differences in medical conditions persist across certain groups. The risk of breast cancer is not the same between men and women.⁴¹ The risk of sickle-cell anemia is not the same between Black and White patients.⁴² As such, it may not be possible to expect a model to both (1) make the same predictions of risk across both groups and (2) have equal false positive and false negative rates between both groups.^43,44 Decisions about trade-offs in such situations require input from physicians, statisticians, and patients. While using race in clinical algorithms can reinforce disparities or racial stereotypes, “excluding factors that have proven to be predictive, albeit highly imperfect… can sometimes carry great human costs”.³⁷ Patients expect predictions of their risk to be as accurate as possible, given the information we have. Reducing accuracy in the name of equity accomplishes neither.

Finally, predictive algorithms can have ROC curves that cross,⁴⁵ so the choice of threshold is critical for many algorithmic fairness measures. A model can be racially fair or more predictive in the minority group across a range of thresholds but be racially biased when other thresholds are chosen. Examining and comparing different thresholds is a critical component of developing fair and equitable predictive algorithms and, in some situations, might be more effective than “bias-reducing” statistical methods. Further research into bias-reducing techniques is needed, especially for clinical risk predictions.

Limitations

We limited our investigation to Black and White patients for 2 main reasons: (1) preexisting evidence of surgical disparities between Black and White patients and (2) insufficient cases of 180-day mortality for other race/ethnicity groups in our data. Because surgical registries are designed to be systematic samples of procedures,^25,46 they will necessarily reflect the small number of racial minorities undergoing surgery. Furthermore, for Asian patients, our data lacked the necessary granularity. Meaningful differences between subgroups of Asian Americans exist, and combining these subgroups into one “Asian” group can dramatically skew results and mask meaningful health disparities.⁴⁷ Future research should examine racial disparities in risk prediction for Hispanic, Asian, Pacific Islander, and Native American populations, and should use larger cohorts with more refined racial data to do so. Further explorations of racial disparities might include data on social risk factors or genetic ancestry.

CONCLUSIONS

We examined racial bias within the RAI, a commonly used measure of frailty. We found that, across various thresholds, the RAI tends to predict mortality among Black patients better than it predicts mortality among White patients. Attempts to reduce bias with the RAI failed to (1) noticeably reduce bias, (2) improve parity across true positives and true negatives, or (3) improve overall model performance. As existing bias-reducing techniques were not effective, further research into bias-reducing techniques is needed, especially for clinical risk predictions. We recommend using the RAI for both statistical analysis of surgical cohorts and quality improvement programs, such as the Surgical Pause.

ACKNOWLEDGMENTS

We thank Brian Ziebart, PhD and Omid Memarrast, PhD for both sharing Python code from their previous work and assisting the authors in implementing it.

Supplementary Material

as9-5-e490-s001.pdf ^{(561.1KB, pdf)}