Mortality Prediction Models: Another Barrier to Racial Equity in a Pandemic

Surges in patient volume during the course of the coronavirus disease (COVID-19) pandemic have raised the very real concern that hospitals may run out of critical resources such as mechanical ventilators and ICU beds. In response to these concerns, crisis standards of care (CSCs) were developed to provide a framework for the allocation of scarce resources. CSCs are designed to be objective, efficient, and ethical, frequently abiding to the principle of maximizing the number of lives or life-years saved. CSCs prioritize the allocation of resources to patients who are more likely to survive to hospital discharge, and they do this by incorporating a tool to predict in-hospital mortality (1). The most commonly used prognostic tool in CSCs is the Sequential Organ Failure Assessment (SOFA) score, although the Laboratory-based Acute Physiology Score version 2 (LAPS2) has been suggested as well (2). In addition, many CSCs include a system to account for a person’s likelihood of postdischarge survival based on their comorbidities. It is important to note that neither the full scoring systems used by CSCs nor individual components of CSCs such as the SOFA score had previously been validated for use in allocating scarce resources but were suggested at the onset of the pandemic to fulfill an urgent need where no validated tool existed.

There is now abundant evidence that Black persons are significantly more likely to contract, be hospitalized with, and die of COVID-19 than white persons (3). These differences are not due to biological features but are rather due to socioeconomic disparities associated with race and racism in the United States, such as decreased access to high-quality health care, exposures related to employment, and higher prevalence of chronic diseases such as obesity and diabetes (4). With recognition of these disparities, concerns have grown that CSCs may place nonwhite patients at an additional disadvantage because of disparities in the performance of mortality prediction models or increased prevalence of medical comorbidities in these communities (5). A recent study found that one CSC priority scoring system, which employed the SOFA score to estimate short-term mortality and comorbidities to estimate longer-term mortality, was not associated with race or ethnicity in a cohort of 1,127 adults admitted with COVID-19 at two urban U.S. hospitals (6). Although this is somewhat reassuring, the small study size and limited patient population raise the importance of further research on the performance of not only CSC systems as a whole but also the individual component scores among larger numbers of patients and in different populations.

In this issue of the Journal, Dr. Ashana and colleagues (pp. 178–186) report the results of their assessment of the prognostic accuracy of the SOFA and LAPS2 scores among 113,158 Black and white patients with sepsis or acute respiratory failure at 27 U.S. hospitals (7). The authors frame their analyses within the context of the inclusion of these two scores as components of CSCs developed for use during the COVID-19 pandemic, but, importantly, none of the patients included in this study had COVID-19, as data were collected before 2019. To assess the scores, the authors evaluated two main features of prognostic models, discrimination and calibration. Discrimination is the ability of a model to separate people within categories; a model with good discrimination should give a higher risk estimate, or score, for patients who experience the outcome (in this case, hospital mortality) than for those who do not. Calibration is the agreement between observed and predicted risk. The authors found that both the SOFA and, to a lesser extent, the LAPS2 score, as well as several modified versions of the scores, had poor to acceptable discrimination overall, and both underestimated hospital mortality for white patients and overestimated hospital mortality for Black patients. This has very important implications; when used in a CSC system that prioritizes allocation of resources to patients with the lowest risk of hospital mortality, Black patients would thus systematically be underallocated to receive scarce resources relative to white patients.

The population included in this study was mostly white, and compared with white patients, the Black patients were significantly younger (mean age 62 vs. 68 yr), were more likely to be female (52% vs. 46%), and had lower hospital mortality (7.5% vs. 8.6%). This is important, as both older age and male sex are associated with increased risk for hospital mortality among patients with sepsis or acute respiratory failure. However, even after adjusting models for age and sex, the authors still identified significant miscalibration. One potential reason for miscalibration in the SOFA score is the higher renal subscore among Black patients, possibly because of higher creatinine levels given the same glomerular filtrate rate found in some previous studies. To address this, the authors created and tested several modifications of the SOFA score, including versions that lessened, or eliminated entirely, the renal subscore. Again, although in some cases the adjusted scores were less poorly calibrated than the original score, differences in performance between Black and white patients remained. The results of these analyses suggest that other factors associated with Black versus white race were contributing to the calibration errors.

This study is timely, of critical importance in the midst of the COVID-19 pandemic, and reinforces the necessity of examining each component of our public health and acute healthcare delivery systems for race-based disparities and inequities. The large size and multicenter nature of the population as well as the study’s rigorous statistical approach and well-conceived sensitivity analyses are important strengths. However, the use of a non–COVID-19 patient population and the focus solely on mortality prediction scores rather than also evaluating full CSC scoring systems makes interpretation of these results in light of the pandemic or comparison to other studies, such as that by Gershengorn and colleagues (6), challenging.

Given concerns that scoring systems may be biased against nonwhite persons as well as the overwhelming evidence of higher prevalence and worse outcomes associated with COVID-19 among nonwhite communities, authors of some CSCs have proposed the use of correction factors that give credits to minoritized or disadvantaged groups. One example is the Area Deprivation Index, which uses a person’s address to rank their degree of socioeconomic disadvantage (5, 8, 9), although this does not explicitly address the racial differences in calibration identified in this study. The use of such correction factors in CSCs is just one example of several potential approaches to address the systematic racial disparities identified by Dr. Ashana and colleagues. This important study highlights the crucial need for more research, validation, and refinement of CSC scoring systems to ensure that they achieve their goals of equitable distribution of resources while maximizing lives or life-years saved.