Considerations for Using Predictive Models that Include Race as an Input Variable: The Case Study of Lung Cancer Screening

Abstract

Indiscriminate use of predictive models incorporating race can reinforce biases present in source data and lead to an exacerbation of health disparities. In some countries, such as the United States, there is therefore a push to remove race from prediction models; however, there are still many prediction models that use race as an input. Biomedical informaticists who are given the responsibility of using these predictive models in healthcare environments are likely to be faced with questions like how to deal with race covariates in these models. Thus, there is a need for a pragmatic framework to help model users think through how to include race in their chosen model so as to avoid inadvertently exacerbating disparities. In this paper, we use the case study of lung cancer screening to propose a simple framework to guide how model users can approach the use (or non-use) of race inputs in the predictive models they are tasked with leveraging in electronic health records and clinical workflows.

Structured abstract:

Objective

Indiscriminate use of predictive models incorporating race can reinforce biases present in source data and lead to an exacerbation of health disparities. The use of such prediction models is not always straightforward. Biomedical informaticists who are given the responsibility of using such models are likely to be faced with questions like how to deal with race covariates. Thus, there is a need for a pragmatic framework to help model users think through how to use predictive models incorporating race so as to avoid inadvertently exacerbating disparities.

Methods

In this paper, we use the case study of lung cancer screening to propose a simple framework to guide how model users can approach the use (or non-use) of race in the predictive models they are tasked with leveraging in electronic health records and clinical workflows.

Results

A five-step pragmatic framework to help informaticists think through how to use predictive models that include race inputs.

Conclusion

There is unlikely to be any single path to using or not using race that is appropriate for all models and conditions. Therefore, among those responsible for the use of prediction models, an explicit discussion of the role and use of race within each model is warranted. A framework to guide the process can assist model users and provide a structure to weigh these model usage decisions with a consideration for health disparities, fairness, and other institutional priorities.

INTRODUCTION

Indiscriminate use of predictive models incorporating race can reinforce biases present in source data and lead to an exacerbation of health disparities.¹ For example, by systematically lowering risk estimates for Black patients, the use of models with race covariates or race ‘correction factors’ as inputs in some clinical decision support tools (e.g. STONE score,^2,3 Breast Cancer Surveillance Consortium Risk Calculator,^4,5 Osteoporosis Risk SCORE^6,7) may inappropriately guide clinicians away from recommending further diagnostics or more aggressive surveillance.¹

In response, in some countries, such as the United States, there is a push to remove race from prediction models.⁸ Consequently, race has been deliberately removed from some models, such as in the National Kidney Foundation’s estimated glomerular filtration rate (eGFR) calculator. While this model reflects the correlation of race with glomerular filtration rate (GFR), the differences in GFR observed between races is mediated by lean muscle mass. Black people tend to have more lean muscle mass than White people on average, but there is significant overlap in body composition between races.⁹ Therefore, when race-was used as ‘correction factor’ in the calculation of eGFR, Black patients were systematically estimated to have a higher eGFR than White patients, regardless of actual body composition for the individual. Thus, this potentially hindered the early detection and management of kidney disease for many Black individuals whose elevated creatinine was not due to more lean muscle mass.

Indeed, while often observed as a predictor of risk, race is rarely a causative factor in condition development pathways. While frequently used, race is often a surrogate for actual causative factors (e.g., greater lean muscle mass causing elevated creatinine). There can be multiple factors that contribute to condition development that are either unknown or difficult to measure, many of which are correlated with race, including social and environmental factors such as access to healthcare, exposure to carcinogens, and socioeconomic status, as well as disease causing alleles. Thus, if the actual causal factors could be accurately measured, race would likely drop out of many prediction models.

The removal of race from prediction models, however, is not always the best approach, particularly in settings with limited availability of data on causal factors, limited ability to routinely measure causal factors in practice, or an incomplete understanding of causal pathways. In cases where race is associated with the outcome, but there are no suitable alternatives factors to replace it (e.g., association of Black race with a higher cancer risk due to childhood exposures that are hard or impossible to measure), the removal of race from prediction models carries the potential to worsen health disparities.¹ For example, Black men have a 1.5 to >3 times higher incidence rate of ischemic stroke than White men.¹⁰ Without the addition of an alternate factor to replace the race covariate to stroke incidence rates, removing race from a stroke prediction model is likely to underestimate the stroke risk for Black men. This could lead to an increased disparity in care if more intensive treatments are reserved for patients at higher stroke risk and that risk is systematically underestimated for Black men. While not a biological or genetic determinant of health outcomes,¹¹ the very real associations observed between race and many health outcomes challenges the notion that race should not be considered in clinical decision making.¹²

Biomedical informaticists are often given the responsibility of using existing predictive models in the electronic health record (EHR). During this process, these model users are likely to be faced with questions on how to deal with race inputs. Healthcare information technology is used in complex and fast-paced environments. Thus, there is a need for a pragmatic framework to help informaticists think through whether and how to use models incorporating race so as to avoid inadvertently exacerbating disparities. In this paper, we use the case study of lung cancer screening (LCS) to propose a simple framework to guide how informaticists can approach the use (or non-use) of race in the predictive models they are tasked with using in EHRs and clinical workflows.

While race is considered a more biological characteristic and ethnicity more a cultural characteristic, in practice, both are often intertwined. For example, in risk models for lung cancer screening used as an example in this manuscript,^13,14 an important model input combines race and ethnicity, with possible values of Hispanic, non-Hispanic Black, non-Hispanic White, and Other. Thus, for simplicity and the purposes of this manuscript, we consider race and race/ethnicity to be synonymous. However, a similar framework may be applied to race and ethnicity separately, or to other individual characteristics.

Race in lung cancer prediction

Low dose computed tomography (LDCT) LCS presents an opportunity to shift the stage at which lung cancer is diagnosed and can decrease the lung cancer death rate by 20–24% compared to other screening methods.^15,16 However, uptake has been limited and there are significant racial and ethnic disparities in LCS rates, where Black and Hispanic patients have a significantly lower odds of receiving LDCT as compared to White patients.^17–19 Consequently, when diagnosed with lung cancer, these groups face worse outcomes compared to White patients, including later diagnosis, lower treatment rates, and lower survival rates.^20,21

LCS represents a prime example of where a decision of whether and how to use a predictive model incorporating race poses a dilemma, as the inclusion of race can influence health disparities in divergent ways.²² Specifically, Black patients have a significantly increased risk of developing and dying from lung cancer, such that using race when estimating cancer risk can reduce health disparities by promoting screening for more members of this population.²³ On the other hand, Black patients also have reduced average life expectancy as a group, such that using race when estimating life expectancy can exacerbate health disparities by labeling Black patients as unlikely to benefit from screening due to limited predicted life expectancy.^14,24 There are many factors that impact lung cancer risk and life expectancy that are either unknown or difficult to measure, many of which are correlated with race, such as access to healthcare, exposure to carcinogens, and other social determinants of health.^23,25 In lung cancer and life expectancy prediction models, race can act as a surrogate, albeit imperfect, for these unmeasured factors. Race may also serve to account for race-associated differences in the completeness of EHR data.^26–28 If the causal factors contributing to the observed racial differences in lung cancer risk and life expectancy could be measured accurately, race would likely drop out of these models, but given feasibility and the current state of science, race remains an important, if not always straightforward, variable to consider in lung cancer and life expectancy prediction models.

Removing race covariates for lung cancer risk can increase disparities by limiting the number of potentially vulnerable high-risk people who are encouraged to receive LCS.²⁹ As it currently stands, neither the United States Preventive Services Taskforce (USPSTF) or CHEST LCS guidelines, nor the frequently used Bach et al., 2003 lung cancer risk model, take race directly into consideration; instead, they use factors including age, gender, and smoking history to designate high-risk patients.^29–31 The removal of race as a potential variable in these LCS guidelines disproportionately disadvantages high-risk Black patients from LCS eligibility as compared to White patients.³² Indeed, among those diagnosed with lung cancer, a significantly lower percentage of Black smokers were eligible for screening according to USPSTF guidelines as compared with White smokers (32% vs. 56%).³²

In an attempt to address potential disparities resulting from racial differences in smoking habits,³² the USPSTF LCS guidelines were updated to reduce the pack years required for screening eligibility and the CHEST guidelines provide a more nuanced approach based on patient smoking history and age.³¹ While this increases the number of Black patients who smoke who qualify for screening, the guideline changes have been estimated to actually widen disparities, as proportionately more White smokers qualify for screening, while Black smokers continue to be underrepresented among individuals eligible for LCS.³³ Lung cancer risk prediction models that include race repeatedly outperform the race-blind guidelines for identifying high-risk Black and Hispanic patients with LCS-avertable lung cancer deaths.^13,34–36

Conversely, using LCS models that incorporate race also has the potential to exacerbate health disparities. In particular, this can been seen in the use of life expectancy when estimating LCS benefit, which has been proposed as a potential strategy to identify high-benefit individuals.^14,37 In the US, Black Americans have an average life expectancy several years less than other populations.²⁴ Consequently, straightforward use of race in life-expectancy calculations could negatively impact screening recommendations for Black individuals due to a lower estimated life expectancy or lower estimated life-year gain. Thus, when estimating life expectancy, using a model that incorporates race can potentially worsen disparities.

LCS model example

To illustrate the use of the framework discussed below, we reference our experiences using the LCS models developed by Katki et al.¹³ and Cheung et al.,¹⁴ and freely available as an R package,³⁸ into the EHR.³⁹ The LCS models estimate the benefits of LCS by calculating individual risks of lung cancer incidence, lung cancer mortality, and all-cause mortality, in both the absence and presence of screening, based on covariates: age, education, sex, race/ethnicity, smoking intensity/duration/quit-years, body mass index, family history of lung-cancer, and multiple comorbidities.¹⁴ These models use the covariates to estimate the expected impact of lung cancer screening on increasing life expectancy and reducing lung cancer mortality. The model estimates both the risk of developing lung cancer and life expectancy.

A decision-making framework for considering race in prediction models

In our own implementation of an LCS decision tool into the EHR,³⁹ acknowledgement of the disparate impacts of race covariates on the propagation of LCS disparities required the development of a framework within which we evaluated the potential impact of race inputs in our prediction models. Working within the framework provided a structured means for addressing critical questions, such as 1) Does including race help close outcome disparities? 2) Do overall gains in health improvements outweigh harms? and 3) Is there an alternate model with a non-race factor within the causal pathway that is feasible to use? When faced with the dilemma of whether and how to use prediction models with race inputs, model users may benefit from employing a similar framework to guide their inquiry.

Here we describe a framework using examples from LCS to guide model users as they think about and address race in the models they are using. Figure 1 depicts the assessment framework developed. This pragmatic framework guides users through pertinent questions to assist in decision making related to the use of models that include race covariates. The five steps of the framework are as follows:

Figure 1:

Assessing the impact of race on inequity and its application to lung cancer screening

Step 1:

Determine places in which race appears in the model.

A first step in the framework is to review the prediction model variables to identify each use where there is the inclusion of race as an input. Step 1 identifies model input variables that include race itself, or are felt to be so intimately associated with race that they are virtually synonymous. Race synonymous variables may include country of origin or preferred language. This step is not intended to identify variables that are merely correlated with race in a way that a modeler would consider including a race-based covariate to that variable.

LCS example:

For the LCS models, the model race input variables identified occur in two places as covariates for lung cancer risk and life-expectancy. In this example, lung cancer rates are higher in Black patients, so the model developer included a race-based rate modifier to increase the base rate of lung cancer for someone who is Black.

Step 2:

Evaluate the suitability of model race variables through the lens of potential comparable models that use alternative variables that represent factors within the causal pathway of the condition of interest.

Race is infrequently involved directly within causal pathways, but it is often used as a surrogate for causal factors that are unknown or unmeasured.^23,25,40 While including race is potentially useful, much of the data used to generate conclusions about the associations between race and health outcomes are based on sources that represent samples with predominantly White populations, with racial differences found in large data sets most likely reflecting sampling bias or the effects of being Black in America rather than being Black itself.⁴¹ For this reason, when available, alternate but otherwise comparable models with better covariates, such as those with other more causative factors should be evaluated as replacements. If a better factor is available for a race covariate identified in step 1, seek an alternative model that employs those variables and evaluate that model beginning at step 1; otherwise, if no better alternate model exists, continue to step 3. For example, in the previously discussed case of the eGFR calculator,^12,42,43 lean muscle mass represents the variable that could replace race in the model.^9,44 If using a GFR calculator, an informatician should select an otherwise comparable model that includes a lean muscle mass variable as opposed to a model with a race variable.

LCS example:

No causative factors in the lung cancer development pathway were identified as potential replacements for the race covariate in the LCS prediction model and upon assessment of other existent LCS prediction models;³¹ thus, the decision was made to continue using the selected model. Socioeconomic status (SES) was considered as a potential alternative to race due to its significant association with lung cancer risk.⁴⁵ However, SES is similarly outside the causal pathway for lung cancer and limited SES data availability in the EHR hinders the feasibility of applying SES for risk prediction.

Step 3:

Determine the impact of race in the model by answering the questions: Does using the actual race reduce inequity? Does a counter-factual race input reduce inequity?

Once uses of race as covariates are identified and potential alternate measures examined, a systematic evaluation is needed to determine the direction and magnitude of the impact on disparities of including actual race or a counter-factual race in a model. The outcomes of these analyses may not always be straight forward and opposing effects may be observed. Removing race entirely from a model may be burdensome; likely requiring model redevelopment. Therefore, one method for evaluating the potential impact of race covariates in a prediction model is to examine the counter factual to determine the disparity impact if the patient were a different race. In a counter-factual analysis, the model outcomes for someone of one race are compared to the model outcomes for another race, while holding all other factors constant. This can allow for the examination of model fairness and potential impact on disparities.⁴⁶

To accomplish this step, an appropriate outcome measure to evaluate impact on inequity must be determined. These measures will be dependent on a model’s focus and purpose, as well as institutional/societal priorities. Measures may include health outcomes, life expectancy, receipt of care, etc. While our LCS model specifically assessed disparities and inequity as they relate to the differential receipt of LCS by race, the way in which outcome disparity is defined will be dependent on a model’s subject content and the priorities of the users. For example, when using a model designed to guide the use of Prostate-Specific Antigen (PSA) testing, a stakeholder with a priority to reduce over overdiagnosis⁴⁷ may choose to use a fairness outcome such as rate of positive tests leading to unnecessary follow-up procedures.

LCS example:

The two outcome variables with race covariates identified in the LCS prediction model produced divergent results when assessed in counter-factual scenarios. Altering race in lung cancer risk estimates exacerbated inequities leading to fewer recommendations for LCS among Black patients as compared to White patients.⁴⁸ On the other hand, altering race in life expectancy estimates reduced inequity, with more Black patients receiving LCS, but had an unintended consequence of reducing the number of Asian patients referred to LCS as a result of Asian-Americans on average having a longer life expectancy than White Americans.

Step 4:

Through the lens of fairness, and within the context of institutional (or societal etc.) priorities, weigh which race input variables should be kept or modified within the model being used.

Decisions about how to use race in the delivery of healthcare should never occur in isolation. The results of counter-factual analyses will likely need to be placed in the context of use goals and institutional priorities, along with a consideration of what would be considered “fair” for those impacted by these decisions. While there are multiple definitions within healthcare of what is “fair,” in this setting, fairness is defined as not disproportionately harming or benefiting those of a particular race.⁴⁹

LCS example:

In the LCS model, the decision was made to not modify race for estimating lung cancer risk, but to modify race when estimating life-expectancy using “counterfactual eligibility”, that recalculates life-expectancy for a minority member as if they were White-American, and then uses the larger estimate.⁵⁰ In fairness analyses, using an non-race specific average cancer risk variable to simulate removing race from lung cancer risk significantly increased LCS disparities and additionally could increase harms due to adverse events from over-screening among Hispanic- and Asian-Americans, who have a lower observed lung cancer risk than other groups. When estimating life expectancy, however, it was observed that modifying race (making life expectancy at least as high as a comparable White patient) had a substantial positive impact on LCS rates among Black patients, but could negatively impact Hispanic and Asian patients. Of note, a counter-factual use of race can be applied selectively, e.g., such that a counter-factual race is used only when it reduces disparities. For example, if an Asian patient would be estimated to live longer than a White patient, their actual race can be used rather than the counter-factual White race. This is in fact what we decided to do in this example for LCS. Therefore, the use of counter-factual eligibility increases African-American eligibility, without reducing eligibility for any other group. We feel that counterfactual eligibility is more fair than simply removing race from a model.⁵⁰ Within the institutional context, the priority of increasing LCS among high-risk populations further reinforced these decisions.

Step 5:

Create transparency by reporting the approach to how and where race is used in the model where appropriate.

Reporting the approach of how race was used in the model can be important for creating transparency that allows end users to make a more educated decision of how to apply the model results in their decision making. With the input of potential users an appropriate and informative description of the use of race in the model can be included in the user-facing tools. In addition, such transparency has the potential to improve patient and clinician trust in the model. This step may not be applicable if there are no race variables in the model used.

LCS example:

Focus groups were held with patients and providers to determine preferences for how the model inclusion decisions would be best presented. Using feedback from these stakeholder populations, information explaining how race may impact model results and the interpretation of the results is planned to be incorporated into relevant LCS decision tools.

DISCUSSION

An empirical approach that critically and systemically evaluates the impact of race exclusion and inclusion in prediction models can aid in decision making related to prediction model use. As with the example of LCS, a framework to guide the process can assist informaticists to tease apart sometimes opposing effects of using race within prediction models and provide a structure useful for weighing these decisions with regard to health disparities and fairness, along with other priorities. However, due to the thorny nature of including race in prediction models, further research is needed to answer questions such as 1) Is it appropriate to ignore race for some measures like life expectancy, but not others? 2) How can measures that include race account for multi-racial individuals? And 3) How do you explain these decisions to relevant stakeholders?

Weighing the fairness of a model (framework step 4) is the step that requires the most work and deliberation. This is particularly true when not all groups are impacted similarly by including or excluding race from the model. When determining model fairness with disparate population-specific effects, it may be pertinent to consider the thought experiment of the “original position” proposed by political philosopher John Rawls. Rawls built on the foundational premise of social contract theory to argue that rational people will develop fair and equal conceptions of justice in society when hypothetically isolated from knowledge of their own personal circumstances (social position, intelligence, appearance) – thereby circumventing the biases of thought that emerge when protecting self-interests.^51,52 Translating this mode of thinking to the use of race in prediction models would encourage developers not to think of the benefits or harms of including race as a variable from the perspective of just one population, but from the perspective of all groups. With this foundation, developers could better weigh what a “fair” model would be and strive to make decisions that would not be overwhelmingly harmful or advantageous for any one group. However, this approach remains imperfect and additional research is needed to guide developers in what to do in these cases in particular where conflict is present.

Given the current inescapability of potential bias whether including or excluding race, transparency has the potential to ensure clinicians and others using the prediction models can make educated decisions regarding clinical decisions that may be based on race. However, there is additional research needed to determine optimal strategies for conveying the choices made in model development and the implications of using race within prediction models. More generally, to improve transparency, prediction model users could be encouraged to summarize how race impacts results of a particular recommendation so clinicians can be better informed.⁵³ A sociotechnical systems methods may serve to help guide these decisions as they are designed to elucidate and solve important challenges related to communication, human-computer interaction, cognition, and motivation.^54–58

Furthermore, how race is defined deserves reflection, as definitions can differentially capture phenotypic, cultural, and genetic factors associated with race and thus impact model interpretation. Most frequently biomedical research relies on self-reported race, which may provide a representation of particular phenotypes or cultural affiliations.⁵⁹ Similarly, race may also be defined by a third-party observer, such as a healthcare provider, which may capture phenotypic characteristics, but miss cultural factors. Both of these categorizations, however, may not be reflective of genetic ancestry as there is significant genetic homogeneity within racial categories. Additionally, race can also be categorized by genetic ancestry, which may capture biological factors contributing to health, but not account for societal and cultural implications of that individual’s unique phenotypes.

Finally, there is a general need to develop better data collection strategies and analyses of causative pathways to more accurately account for the mediating factors that lead to the observed racial associations with health outcomes. When possible, factors within a causal pathway should be used to replace race. However, existing models are often functionally limited by the type of data routinely collected in operational clinical care, which, more often than not, does not include potentially impactful circumstantial factors. Therefore, alternate data collection strategies could be considered to obtain usable data on key known risk factors, such as using neighborhood-level measures by zip code as a stand-in for SES.⁶⁰ Along with addressing causal factor considerations, wise decisions about including race can lead to the development of less biased prediction models that improve racial disparities in care delivery.

CONCLUSION

Decision making surrounding the use of race in the use of prediction models is not always straight forward. There is unlikely to be any single path to including or excluding race that is appropriate for all models and conditions. Therefore, among those responsible for the use of prediction models, an explicit discussion of the role of race within each model is warranted. A framework to guide the process can assist model users and provide a structure to weigh these model usage decisions with a consideration for health disparities, fairness, and other institutional priorities. However, additional attention is needed to evaluate the broader implications of selectively using race in the use of these models and how to best communicate these decisions to clinicians and patients. Further efforts are also needed to identify and make data available for factors that may better replace race in prediction models.

Statement of Significance.

Problem or Issue

Indiscriminate use of race in clinical algorithms can reinforce biases present in source data and lead to an exacerbation of health disparities.

What is Already Known

Biomedical informaticists who are given the responsibility of implementing clinical algorithms are likely to be faced with questions of how to deal with race in these models. For those instances where the answer is not simple, there is a need for a pragmatic framework to help informaticists think through how to include race in their models.

What this Paper Adds

We propose a simple framework to guide the use (or non-use) of race clinical algorithms.