An Empirical Characterization of Fair Machine Learning For Clinical Risk Prediction

Abstract

The use of machine learning to guide clinical decision making has the potential to worsen existing health disparities. Several recent works frame the problem as that of algorithmic fairness, a framework that has attracted considerable attention and criticism. However, the appropriateness of this framework is unclear due to both ethical as well as technical considerations, the latter of which include trade-offs between measures of fairness and model performance that are not well-understood for predictive models of clinical outcomes. To inform the ongoing debate, we conduct an empirical study to characterize the impact of penalizing group fairness violations on an array of measures of model performance and group fairness. We repeat the analysis across multiple observational healthcare databases, clinical outcomes, and sensitive attributes. We find that procedures that penalize differences between the distributions of predictions across groups induce nearly-universal degradation of multiple performance metrics within groups. On examining the secondary impact of these procedures, we observe heterogeneity of the effect of these procedures on measures of fairness in calibration and ranking across experimental conditions. Beyond the reported trade-offs, we emphasize that analyses of algorithmic fairness in healthcare lack the contextual grounding and causal awareness necessary to reason about the mechanisms that lead to health disparities, as well as about the potential of algorithmic fairness methods to counteract those mechanisms. In light of these limitations, we encourage researchers building predictive models for clinical use to step outside the algorithmic fairness frame and engage critically with the broader sociotechnical context surrounding the use of machine learning in healthcare.

1. Introduction

The use of machine learning with observational health data to guide clinical decision making has the potential to introduce and exacerbate health disparities for disadvantaged and underrepresented populations [1–6]. This effect can derive from inequity in historical and current patterns of care access and delivery [1, 7–11], underrepresentation in clinical datasets [12], the use of biased or mis-specified proxy outcomes during model development [3, 13, 14], and differences in the accessibility, usability, and effectiveness of predictive models across groups [1, 15]. In response, considerable attention has been devoted to reasoning about the extent to which clinical predictive models may be designed to anticipate and proactively mitigate harms to advance health equity, while upholding ethical standards [1, 2, 4, 7, 16–19].

The role that algorithmic fairness techniques should have in the development of clinical predictive models is actively debated [1, 2, 16, 17, 20]. While these techniques have been extensively studied and scrutinized in domains such as criminal justice, hiring, and education [21–25], and have been made accessible by several open source software frameworks [26–28], the scope of their examination in the context of clinical predictive models has been relatively limited [29–34]. Algorithmic fairness methods specify a mathematical formalization of a fairness criterion representative of an ideal (such as equal error rates between male and female patients), and provide procedures for minimizing violations from the fairness criterion without unduly deteriorating model performance [22, 35–39]. Furthermore, these mathematical formalizations can be used as auditing mechanisms to identify problematic characteristics of a predictive model and to promote transparency in the model’s output [40–43]. In the context of this debate, it is essential to recognize that algorithmic fairness techniques enable monitoring and manipulating the output of predictive models, but are generally insufficient by themselves to mitigate the introduction or perpetuation of health disparities resulting from model-guided interventions [2, 3, 16, 17, 20, 24, 44–49].

Health disparities arise as a result of structural forms of racism and related inequities in areas such as housing, education, employment, and criminal justice that affect healthcare access, utilization, and quality [11, 50]. These effects are further compounded by under-representation of the elderly, women, and ethnic minorities in clinical trials and cohort studies [51, 52], as well as warped financial incentives in the healthcare system [53]. Typical formulations of algorithmic fairness, particularly group fairness criteria, are unaware of this context, because they are primarily defined in terms of a model’s predictions, observed outcomes, as well as membership in a pre-specified group of demographic attributes, and evaluated on cohorts derived retrospectively. As a result, algorithmic fairness criteria may be misleading in several situations, including when the observed outcome is a biased surrogate for the construct of interest [3, 49] and when predictive models are not appropriately contextualized in terms of the heterogeneous and dynamic impact of the complex interventions and policies that they enable, contributing to an erroneous conflation of model performance metrics with an accrual of benefit [44, 47, 54–56].

Some researchers argue that techniques from algorithmic fairness still have a role to play in promoting health equity [1], particularly if their use is informed by a deeply embedded understanding of the sociotechnical and clinical contexts that surround the use of a predictive model’s output. From this perspective, it is relevant to reason about the trade-offs between properties such as model performance and fairness criteria satisfaction, as long as those properties can be appropriately contextualized. However, it is generally impossible to satisfy conflicting notions of fairness simultaneously [44, 57–59] and methods that impose fairness constraints are known to do so at the cost of substantial trade-offs between various measures of model performance and fairness criteria satisfaction, in ways that are sensitive to the properties of the dataset and learning algorithm used [43, 44, 57, 58, 60–64]. As the evidence base that surrounds the use of algorithmic fairness methods in the context of clinical risk prediction remains limited, the extent to which these trade-offs manifest when adopting algorithmic fairness approaches for training clinical predictive models remains unclear.

To inform this discussion, we conduct a large-scale empirical study characterizing the trade-offs between various measures of model performance and fairness criteria satisfaction for clinical predictive models that are penalized to varying degrees against violations of group fairness criteria. We repeat the analysis across several databases, outcomes and sensitive attributes in an attempt to identify patterns that generalize. We make all code relevant for reproducing these experiments available at https://github.com/som-shahlab/fairness_benchmark.

The structure of the remainder of the paper is as follows. In Section 2, we formally define the prediction problem, group fairness criteria, metrics, and learning objectives that we evaluate. We then describe the experimental setup, including datasets used, definitions of cohorts, outcomes and sensitive attributes, as well as procedures for feature extraction, model training and evaluation. In Section 3, we report on the results. In Section 4, we provide recommendations for model developers and policy makers in light of both the trade-offs that we find as well as the limitations inherent to the algorithmic fairness framework. Furthermore, we discuss the appropriateness of typical abstractions used for evaluation of group fairness, including the use of discrete categories for sex, race, and ethnicity.

2. Methods

Across twenty five combinations of datasets, clinical outcomes, and sensitive attributes, we train a series of predictive models that are penalized to varying degrees against violations of fairness criteria, and report on model performance and group fairness metrics. Figure 1 outlines the experimental procedure.

Figure 1:

An overview of the experimental procedure. We extract cohorts from a collection of databases and label patients on the basis of observed clinical outcomes and membership in groups of multiple demographic attributes. For twenty five combinations of database, outcome, and sensitive attribute, we train a series of predictive models that are penalized to varying degrees against violations of conditional prediction parity. We report on the effect of such penalties on measures of model performance and fairness criteria satisfaction. Shown here is process for the cohort drawn from the STARR database.

We first define the mathematical formulation of group fairness from which fairness metrics and training objectives are derived (Section 2.1). We present three classes of group fairness criteria – conditional prediction parity (Section 2.1.2), calibration (Section 2.1.3), and cross-group ranking (Section 2.1.4) – each of which is operationalized by one or more fairness metrics that quantify the extent to which the associated criterion is violated. We evaluate each of these metrics for models developed with six regularization strategies that penalize violation of conditional prediction parity (Section 2.1.5). The computational experiments that examine the behavior of these constructs are described in Section 2.2, which includes a description of the datasets (Section 2.2.1), definitions of the cohort inclusion criteria, outcomes, and sensitive attributes (Section 2.2.2), as well as details of feature extraction (Section 2.2.3), model training (Section 2.2.4), and evaluation (Section 2.2.5).

2.1. Formulations of Group Fairness Criteria

2.1.1. Notation and Problem Formulation

Let $X\in \mathcal{X}={ℝ}^m$ be a variable designating a vector of covariates; $Y\in \mathcal{Y}=\{0,1\}$ be a binary indicator of an outcome; and $A\in \mathcal{A}$ be a discrete indicator for a protected or sensitive attribute, such as race, ethnicity, gender, sex, or age with K groups. The objective of supervised learning with binary outcomes is to use data $\mathcal{D}={\left\{\left(x_i,y_i,a_i\right)\right\}}_{i=1}^N~P(X,Y,A)$ to learn a function $f_{\theta }(X):{\mathbb{R}}^m\to [0,1]$, parameterized by θ, to estimate $\mathbb{E}[Y\mid X]=P(Y=1\mid X)$. The output of the predictor may be further thresholded at T to produce binarized predictions $\widehat{Y}=\mathbb{1}\left[f_{\theta }(X)\ge T\right]\in \{0,1\}$. For notational convenience, we refer to an empirical mean defined over a dataset as an expectation involving $\mathcal{D}$. For example, ${\mathbb{E}}_{x~\mathcal{D}\mid Y=1}{f}_{\theta }(x)$ refers to the empirical mean of the predicted probability of the outcome over the set of patients for whom the outcome is observed.

Many of the group fairness criteria that we evaluate can be stated as a form of conditional independence involving a function of some or all of the variables $\{X,Y,f_{\theta },\widehat{Y}\}$ with A. Given access to a dataset and model predictions, quantifying the extent to which a group fairness criterion is violated relies on a fairness metric that compares the empirical distribution of the relevant function across groups. The objective of supervised learning subject to group fairness is to learn a model f_θ such that the violation of one or more fairness criteria is small. This framing does not preclude the use of the group indicator A as a component of the vector of covariates X.

We consider fairness criteria belonging to three general categories: conditional prediction parity, calibration and cross-group ranking. The first - conditional prediction parity - will later be incorporated into our training objective. Below, we describe the motivation behind considering each class, as well as mathematically define particular metrics which will be used to evaluate our experiments.

2.1.2. Fairness Criteria Based on Conditional Prediction Parity

We refer to a class of group fairness criteria that assess conditional independence between model predictions $\widehat{Y}$ or f_θ and the categorical group indicator A as conditional prediction parity. When an instance of a fairness criterion of this class depends on a binarized prediction $\widehat{Y}$, we say it is a threshold-based criterion, and when it depends on a probabilistic prediction f_θ, we say it is a threshold-free criterion. This form captures demographic parity $(\widehat{Y}\perp A\text{or}{f}_{\theta }\perp A)$ [21, 35, 65], equalized odds $(\widehat{Y}\perp A\mid Y\text{or}{f}_{\theta }\perp A\mid Y)$ [22], and equal opportunity $(\widehat{Y}\perp A\mid Y=1\text{or}{f}_{\theta }\perp A\mid Y=1)$ [22], among others [66].

These criteria have natural interpretations that can motivate their use in some contexts. For instance, violations of demographic parity can imply allocation discrepancies across groups. Equal opportunity and equalized odds can be understood as either a notion of demographic parity within strata defined by the outcome Y or as a notion of error rate balance. For binarized predictions $\widehat{Y}$, the equal opportunity criterion implies equal true positive rates across groups, and the equalized odds criterion implies equal opportunity as well as equal false positive rates across groups. Satisfying the threshold-free variants of these criteria implies satisfaction of the threshold-criteria at any threshold. An implication of this is that the threshold-free variant of the equalized odds criteria corresponds to the condition of identical ROC curves across groups [22].

We now present metrics that assess the extent to which a predictive model violates notions of threshold-free conditional prediction parity, starting with demographic parity (M_DP) before extending it to equal opportunity (M_EqOpp) and equalized odds (M_EqOdds). Threshold-free demographic parity is satisfied if P(f_θ | A = A_k) matches $P\left(f_{\theta }\right)={\sum }_{A_k\in \mathcal{A}}P\left(f_{\theta }\mid A=A_k\right)$, for each group A_k. This motivates the use of a metric of the following form:

$$M_{\text{DP}}=\sum _{A_k\in \mathcal{A}}D\left(P\left(f_{\theta }\mid A=A_k\right)\Vert P\left(f_{\theta }\right)\right),$$ (1)

where D is a function that measures a notion of distance between probability distributions. As equalized odds and equal opportunity may be interpreted as demographic parity within strata defined by the outcome, the associated metrics can be constructed analogously, with

$$M_{\text{EqOpp}}=\sum _{A_k\in \mathcal{A}}D\left(P\left(f_{\theta }\mid A=A_k,Y=1\right)\Vert P\left(f_{\theta }\mid Y=1\right)\right)$$ (2)

for equal opportunity, and

$$M_{\text{EqOdds}}=\sum _{Y_j\in \mathcal{Y}}\sum _{A_k\in \mathcal{A}}D\left(P\left(f_{\theta }\mid A=A_k,Y=Y_j\right)\Vert P\left(f_{\theta }\mid Y=Y_j\right)\right)$$ (3)

for equalized odds.

With this construction, the form of function D determines the properties of the metric used to assess fairness criteria violation. If D is chosen to be a divergence or integral probability metric [67], then the fairness criteria is satisfied if the associated metric M is equal to zero, and is positive otherwise. In our experiments, we evaluate these metrics using the Earth Mover’s Distance (EMD) [68], an example of an integral probability metric.

These metrics may be further decomposed into group- and outcome- specific components. For instance, if

$$M_k=D\left(P\left(f_{\theta }\mid A=A_k\right)\Vert P\left(f_{\theta }\right)\right),$$ (4)

then $M_{\text{DP}}={\sum }_{k=1}^K{M}_k$. Similarly, if

$$M_k^1=D\left(P\left(f_{\theta }\mid A=A_k,Y=1\right)\Vert P\left(f_{\theta }\mid Y=1\right)\right),$$ (5)

and

$$M_k^0=D\left(P\left(f_{\theta }\mid A=A_k,Y=0\right)\Vert P\left(f_{\theta }\mid Y=0\right)\right),$$ (6)

then $M_{\text{EqOpp}}={\sum }_{k=1}^K{M}_k^1$, and $M_{\text{EqOdds}}=M_{\text{EqOpp}}+{\sum }_{k=1}^K{M}_k^0$.

As an alternative, we also consider metrics based on the difference in means between the distributions of interest. As before, these metrics decompose into group-specific components. For instance, if we define the difference in the mean predicted probability of the outcome for patients in group k versus the marginal distribution ($M_k^{\text{mean}}$) as

$$M_k^{\text{mean}}={\mathbb{E}}_{x~\mathcal{D}\mid A=A_k}\left[f_{\theta }(x)\right]-{\mathbb{E}}_{x~\mathcal{D}}\left[f_{\theta }(x)\right],$$ (7)

then an aggregate metric of demographic parity violation can be constructed as

$$M_{\text{DP}}^{\text{mean}}=\sum _{k=1}^K{\left(M_k^{\text{mean}}\right)}^2.$$ (8)

Expressions for equal opportunity and equalized odds can be constructed analogously by nesting these comparisons within strata defined by the observed outcomes. If

$$M_k^{1,\text{mean}}={\mathbb{E}}_{x~\mathcal{D}\mid A=A_k,Y=1}\left[f_{\theta }(x)\right]-{\mathbb{E}}_{x~\mathcal{D}\mid Y=1}\left[f_{\theta }(x)\right],$$ (9)

and

$$M_k^{0,\text{mean}}={\mathbb{E}}_{x~\mathcal{D}\mid A=A_k,Y=0}\left[f_{\theta }(x)\right]-{\mathbb{E}}_{x~\mathcal{D}\mid Y=0}\left[f_{\theta }(x)\right],$$ (10)

then

$$M_{\text{EqOpp}}^{\text{mean}}=\sum _{k=1}^K{\left(M_k^{1,\text{mean}}\right)}^2$$ (11)

and

$$M_{\text{EqOdds}}^{\text{mean}}=M_{\text{EqOpP}}^{\text{mean}}+\sum _{k=1}^K{\left(M_k^{0,\text{mean}}\right)}^2.$$ (12)

2.1.3. Fairness Criteria Based on Calibration

We construct a measure of fairness in calibration to estimate the extent to which observed event rates conditioned on the predicted risk differ across groups. We do so by first presenting a measure of absolute calibration, which can be understood as a performance metric that assesses overall model calibration, before extending that measure to account for relative differences in calibration across groups.

Calibration in the context of clinical risk prediction typically refers to the extent to which probabilistic predictions are faithful estimates of the observed event rates, such that a model is well-calibrated if P(Y = 1 | f_θ = p) = p for p ∈ [0, 1]. In other words, a model is well-calibrated if the outcome is observed p ∗ 100% of the time among patients whose prediction is p.

The absolute calibration error (ACE) assesses deviations from perfect calibration, relying on an auxiliary estimator g_ϕ : [0, 1] → [0, 1] that provides an approximation of P(Y = 1 | f_θ):

$$\text{ACE}={\mathbb{E}}_{x~\mathcal{D}}\left[{\left(g_{\varphi }\left(f_{\theta }(x)\right)-f_{\theta }(x)\right)}^2\right].$$ (13)

This measure may be interpreted as an instance of the weighted mean squared calibration error proposed in Yadlowsky et al. [69], used here in the context of uncensored binary outcomes, and is also similar to the integrated calibration index proposed in Austin et al. [70]. In addition, we consider a signed variant of the metric:

$${\text{ACE}}^{\text{signed}}={\mathbb{E}}_{x~\mathcal{D}}\left[g_{\varphi }\left(f_{\theta }(x)\right)-f_{\theta }(x)\right].$$ (14)

The signed measure can assess the directionality of mis-calibration, but may be misleading when positive deviations offset negative ones. For clarity, a positive value for the signed measure corresponds to under-prediction of risk, as it corresponds to an excess number of observed outcomes given the risk estimates.

In the context of group fairness, it is relevant to reason about model calibration in a relative sense. The matching conditional frequencies [22, 58] (MCF) or sufficiency [71] condition requires that the probability of the outcome not differ across groups conditioned on the value of the risk score i.e. Y ⊥ A | f_θ. An alternative (but not equivalent) formulation is to require the risk score to be well-calibrated within groups [57, 72]. We present a measure of relative calibration to estimate the extent to which observed event rates conditioned on the predicted risk differ across groups, thus assessing the violation of the MCF criterion. Given an auxiliary estimator g_ϕ that estimates the marginal density P(Y = 1 | f_θ) for the entire population and an estimator g_k that estimates P(Y = 1 | f_θ, A = A_k), the relative calibration error (RCE) for group A_k is defined as

$${\text{RCE}}_k={\mathbb{E}}_{x~\mathcal{D}\mid A=A_k}\left[{\left(g_k\left(f_{\theta }(x)\right)-g_{\varphi }\left(f_{\theta }(x)\right)\right)}^2\right].$$ (15)

with the corresponding signed metric defined as

$${\text{RCE}}_k^{\text{signed}}={\mathbb{E}}_{x~\mathcal{D}\mid A=A_k}\left[g_k\left(f_{\theta }(x)\right)-g_{\varphi }\left(f_{\theta }(x)\right)\right].$$ (16)

The measures that we present here depend on the form of the estimators g_ϕ and g_k. In our experiments, we use logistic regression as an estimator. Alternatives include loess regression [70], kernel density estimates over P(f_θ | Y ) combined with simple conditional probability rules, or binning estimators.

2.1.4. Cross-Group Ranking Measures

We now consider notions of cross-group ranking performance [73, 74]. These measures can be understood as variations on the area under the receiver operating characteristic curve (AUROC). The AUROC is a typical metric for assessing the performance of a clinical predictive model. This metric has an equivalent interpretation as the probability with which predictions for members of the positive class are ranked above those for members of the negative class:

$$\text{AUROC}={\mathbb{E}}_{x^1~\mathcal{D}\mid Y=1}{\mathbb{E}}_{x^0~\mathcal{D}\mid Y=0}\mathbb{1}\left(f_{\theta }\left(x^1\right)>f_{\theta }\left(x^0\right)\right].$$ (17)

A comparison of the AUROC across groups can be misleading as a comparison of relative model quality and as a fairness criterion since that comparison does not account for the degree with which positive examples of a group are ranked above negative examples of other groups. In order to gain insight into this phenomenon, we consider multi-group extensions to measures defined in the xAUC framework [73, 74] and consider deviations in these measures across groups as a violation of a measure of group fairness. We define ${\text{xAUC}}_k^1$ as the probability with which positive instances of group A_k are ranked above negative instances of all other groups:

$${\text{xAUC}}_k^1={\mathbb{E}}_{x^1~\mathcal{D}\mid Y=1,A=A_k}{\mathbb{E}}_{x^0~\mathcal{D}\mid Y=0,A\ne A_k}\mathbb{1}\left[f_{\theta }\left(x^1\right)>f_{\theta }\left(x^0\right)\right].$$ (18)

Similarly, we define ${\text{xAUC}}_k^0$ as the probability with which negative instances of group A_k are ranked below positive instances of all other groups:

$${\text{xAUC}}_k^0={\mathbb{E}}_{x^1~\mathcal{D}\mid Y=1,A\ne A_k}{\mathbb{E}}_{x^0~\mathcal{D}\mid Y=0,A=A_k}\mathbb{1}\left[f_{\theta }\left(x^1\right)>f_{\theta }\left(x^0\right)\right].$$ (19)

2.1.5. Regularized Objectives for Conditional Prediction Parity

Several techniques for constructing models that satisfy measures of group fairness have been proposed, including representation learning [35, 75–77], post-processing [22], and constrained or regularized learning objectives [36–38, 66, 78]. We focus our attention on regularized learning objectives that penalize violations of measures of threshold-free conditional prediction parity. Given a loss function $\mathcal{L}$, such as the negative log-likelihood, the general form of this objective is as follows:

$$\underset{\theta }{\min }{\mathbb{E}}_{(x,y)~\mathcal{D}}\mathcal{L}\left(y,f_{\theta }(x)\right)+\lambda R,$$ (20)

where R is a non-negative regularizer indicative of the extent to which the fairness criterion is violated, and λ is a non-negative scalar. In general, tuning the value of λ allows for exploring the trade-offs between measures of model performance and fairness.

In our experiments, we evaluate six regularization strategies to penalize the violation of threshold-free conditional prediction parity measures. These strategies correspond to regularizers that penalize violation of demographic parity, equal opportunity, and equalized odds, either with regularizers of the form of Equations (1), (2), or (3), respectively, using an empirical maximum mean discrepancy (MMD) [79] with a Gaussian kernel to estimate D, or with regularizers in the form of Equations (8), (11), or (12), respectively, that penalize the sum of the squared difference in the mean prediction in the relevant outcome strata. During training, the value of the regularizer R is computed on the basis of comparisons between the distributions of logf_θ, computed via a log-softmax transformation of the model logits, rather than on the basis on f_θ directly. In some instances, we refer to regularizers that penalize violation of demographic parity, equalized odds, and equal opportunity as unconditional, conditional, and positive conditional, respectively, corresponding to the outcome strata used to evaluate the regularizer, and include a modifier for either MMD or mean to indicate how the relevant distributions are compared.

2.2. Experiments

2.2.1. Datasets

STARR

The Stanford Medicine Research Data Repository (STARR) [80] is a clinical data warehouse containing records from approximately three million patients from Stanford Hospitals and Clinics and the Lucile Packard Children’s Hospital for inpatient and outpatient clinical encounters that occurred between 1990 and 2020. This database contains structured longitudinal data in the form of diagnoses, procedures, medications, and laboratory tests that have been mapped to standard concept identifiers in the Observational Medical Outcomes Partnership Common Data Model (OMOP CDM) version 5.3.1 [81–83]. De-identified clinical notes with clinical concepts extracted and annotated with negation and family-history detection are also made available [80].

Optum©’s de-identifed Clinformatics® Data Mart Database

Optum©’s de-identifed Clinformatics® Data Mart Database (Optum CDM) is a statistically de-identified large commercial and medicare advantage claims database. The database includes approximately 17–19 million annual covered lives, for a total of over 57 million unique lives over a 9 year period (1/2007 through 12/2017). We utilize a variant of the database that makes available the month and date of death, sourced from internal and external sources including the Death Master File maintained by the Social Security Office, as well as records from the Center for Medicare and Medicaid Services. However, this version of the data does not provide access to detailed socioeconomic, demographic, or geographic variables. We utilize version 7.1 of the data mapped to OMOP CDM v5.3.1.

MIMIC-III

The Medical Information Mart for Intensive Care-III (MIMIC-III) [84] is a widely studied database containing comprehensive physiologic information from approximately fifty thousand intensive care unit (ICU) patients admitted to the Beth Israel Deaconess Medical Center between 2001 and 2012. We utilize MIMIC-OMOP¹, a variant of the database that has been mapped to OMOP CDM v5.3.1.

2.2.2. Cohort, Outcome, and Attribute Definitions

We define a set of analogous cohort, outcome, attribute and group definitions for each database that differ on the basis of differential availability of data elements. For the STARR and Optum CDM databases, we consider the set of inpatient hospital admissions that span at least two distinct calendar dates for which patients were of age 18 years or older at the time of admission. For admissions in the STARR database, the start of the admission is variably defined as the date and time of admission to the emergency department, if available, and admission to the hospital otherwise. For patients with more than one admission meeting this criteria, we randomly sample one admission. In each case, we consider the start of an admission as the index date and time. For each admission, we derive binary outcome labels for prolonged length of stay (defined as a hospital length of stay greater than or equal to seven days) and 30-day readmission (defined as a subsequent admission within thirty days of discharge of the considered admission). For admissions derived from the STARR database, we also derive a binary outcome label for in-hospital mortality. This procedure returns admissions from 198,644 patients in STARR (Table 1) and 8,074,571 patients in Optum CDM (Supplementary Table A.1).

Table 1:

Cohort characteristics for patients drawn from STARR. Data are grouped on the basis of age, sex, and the race and ethnicity category. Shown, for each group, is the number of patients extracted and the incidence of hospital mortality, prolonged length of stay, and 30-day readmission

		Outcome Incidence
Group	Count	Hospital Mortality	Prolonged Length of Stay	30-Day Readmission
[18–30)	23,042	0.00681	0.175	0.0461
[30–45)	43,432	0.00596	0.130	0.0396
[45–55)	27,394	0.0178	0.205	0.0527
[55–65)	35,703	0.0251	0.227	0.0558
[65–75)	36,084	0.0284	0.234	0.0548
[75–90)	32,989	0.0400	0.238	0.0545
Female	112,713	0.0161	0.166	0.0452
Male	85,923	0.0271	0.244	0.0571
Asian	29,460	0.0209	0.171	0.0536
Black	7,813	0.0198	0.240	0.0581
Hispanic	33,742	0.0180	0.193	0.0544
Other	20,270	0.0327	0.226	0.0442
White	107,359	0.0196	0.202	0.0488

For MIMIC-III, we replicate in MIMIC-OMOP the cohort and outcome definitions defined as benchmarks in the MIMIC-Extract project [85]. In particular, we consider hospital admissions associated with each patient’s first ICU stay, further restricting the set of allowable ICU stays to those lasting between twelve hours and ten days. We set the index date and time to be twenty-four hours after hospital admission, and further restrict the set of admissions to those for which the index time is at least six hours prior to ICU discharge and for which patients are between 15 and 89 years old at the index time. We define four binary outcomes, corresponding to ICU length of stay greater than three and seven days and mortality over the course of the ICU stay and hospital admission, following the approach of Wang et al. [85]. This procedure returns 26,170 patients (Supplementary Table A.2).

For the purposes of evaluating the extent to which a model satisfies group fairness criteria, we define discrete groups of the population on the basis of demographic attributes. We consider (1) a combined race and ethnicity variable based on self-reported racial and ethnic categories, (2) sex², and (3) age at the index date, discretized into categories at 18–29, 30–44, 45–54, 55–64, 65–74, 75–89 years for STARR and Optum cohorts and 15–29, 30–44, 45–54, 55–64, 65–74, 75–89 years for MIMIC-III.

The race and ethnicity attribute is constructed by assigning Hispanic if the ethnicity is recorded as Hispanic, and the value of the recorded racial category otherwise. In line with the categories provided by the upper level of the OMOP CDM vocabulary, we consider groups corresponding to “Asian”, “American Indian or Alaska Native”, “Black or African American”, “Native Hawaiian or Other Pacific Islander”, “Other”, and “White”. We additionally aggregate groups of the race and ethnicity attribute with the “Other” category when few observed outcomes are available within a group in a database specific manner. This reduces the categorization to “Asian”, “Black or African American”, “Hispanic”, “Other”, and “White” for STARR, and to “Other” and “White” for MIMIC-III. Race and ethnicity data is not available for the version of the Optum CDM database that we use. We further contextualize this operationalization of race and ethnicity in Section 4.

2.2.3. Feature Extraction

We extract clinical features with a generic procedure that operates on OMOP CDM databases, similar to Reps et al. [83]. A graphical depiction of this procedure is provided in Supplementary Figure B.1. The process returns 539,823, 438,369, and 21,026 unique features in the STARR, Optum CDM, and MIMIC-III cohorts, respectively. In brief, we consider a set of binary features based on the occurrence of both unique OMOP CDM concepts and derived elements prior to the index date and time. We extract all diagnoses, medication orders, procedures, device exposures, encounters, and labs, assigning a unique feature identifier for the presence of each OMOP CDM concept, in time intervals defined relative to the index date and time. For lab test results, we construct additional features for lab results above and below the corresponding reference range, if available, and indicators for whether the result belongs to the empirical quintiles for the corresponding lab result observed in the time interval. For STARR data, we include additional features for the presence of clinical concepts derived from clinical notes, modified by an indicator for whether the concept corresponds to a present and positive mention [80]. For MIMIC-III, we do not use diagnoses or clinical notes to derive features. We include time-agnostic demographic features, including race, ethnicity, gender, and age group (discretized in five year intervals), using the OMOP CDM concept identifiers directly rather than the attribute definitions described previously.

We repeat the feature extraction procedure for a set of time-intervals defined relative to the index date and time in a database-specific manner. For the STARR and Optum CDM cohorts, we define intervals at 29 to 1 days prior to the index date, 89 to 30 days prior, 179 to 90 days prior, 364 to 180 days prior, and any time prior. For STARR and MIMIC-III data, we include additional time-intervals defined only over the subset of the data elements recorded with date and time resolution, repeating each extraction procedure described for which that set is not empty. For STARR, these intervals correspond to 4 hours prior to the index time, 12 hours to 4 hours prior to the index time, 24 to 12 hours prior, three days to 24 hours prior, and seven days to three days prior. For MIMIC-III, these intervals correspond to 4 hours prior to the index time, 12 hours to 4 hours prior to the index time, 24 to 12 hours prior, and any time prior.

2.2.4. Model Training

We conduct a modified cross-validation procedure designed to enable robust model selection and evaluation. For each cohort, a randomly sampled partition of 10% of the patients is set aside as a test set for final evaluation. For STARR and MIMIC-III, we further partition the remaining data into ten equally-sized folds, each of which can be considered as a validation set corresponding to a training set composed of the remainder of the folds. Due to the computational constraints imposed by the large size of the Optum CDM cohort, we do not perform cross-validation and instead randomly partition the data such that 81% of the patients are used for training, 9% of the patients are used for validation, and 10% used for testing.

In all cases, we leverage fully-connected feedforward neural networks for prediction, as they enable scalable and flexible learning for differentiable objectives in the form of equation (20). Tuning of the unpenalized models begins with a random sample of fifty hyperparameter configurations from a grid of architectural and training-dynamic hyperparameters (Supplementary Table C.1). For each combination of dataset, outcome, training-validation partition, and hyperparameter configuration, we train a model with the Adam [86] optimizer for up to 150 iterations of 100 batches, terminating early if the cross-entropy loss on the validation set does not improve for 10 iterations. For each combination of dataset and outcome, we select model hyperparameters on the basis of the mean validation log-loss across folds (the selected hyperparameters are provided in Supplementary Tables C.2, C.4, and C.3). For models trained on the Optum CDM cohort, we select model hyperparameters on the basis of the log-loss measured on the validation set. We then train models with regularization that penalizes fairness criteria violation using an objective in the form of Equation (20) for ten values of λ distributed log-uniformly on the interval 10⁻³ to 10, repeating the process separately for each dataset, task, attribute, and form of regularizer (of which there are six, as discussed in Section 2.1.5), holding the model hyperparameters fixed to those selected for the corresponding unpenalized model. As before, we train each model for up to 150 iterations of 100 batches, but perform early stopping and model selection on the basis of the penalized loss that incorporates the regularization term. Pytorch version 1.5.0 [87] is used to define all models and training procedures.

2.2.5. Model Evaluation

We report model performance and fairness metrics as the mean ± the standard deviation (SD) for metrics evaluated on the held-out test set for the set of ten models derived from the procedure described in Section 2.2.4. For models trained on the Optum CDM cohort, we report results on the test set for the selected model without an associated standard deviation. To assess within-group model performance, we report the AUROC, average precision, and cross entropy loss at baseline and each value of λ. To assess conditional prediction parity violation, we report on the decomposed group-specific components of the metrics presented in Section 2.1. In particular, we report the EMD and difference in means between the distribution of predictions for each group with the marginal distribution constructed via aggregation of predictions from all groups (Equations (4) and (7)), respectively, as metrics that assess violations of demographic parity. We repeat the process in the strata of the population for which the outcome is and is not observed, as metrics that assess violations of equalized odds and equal opportunity (Equations (6), (10), (5), and (9)). We report on cross-group ranking discrepancies in the form of Equations (18) and (19) for each group.

To assess absolute and relative model calibration, we estimate ACE and RCE post-hoc on the test set. To do so, for each predictive model f_θ, we train an auxiliary logistic regression model to estimate P(Y | f_θ) on the basis of log(f_θ) for the aggregate test set and for each group. The resulting group-level estimates of ACE and RCE are constructed by plugging-in the data to the relevant estimators using Equations (13), (14), (15), and (16). The logistic regression models are fit using the LBFGS [88] algorithm implemented in Scikit-Learn [89]. We report ACE alongside model performance measures and RCE alongside other fairness metrics.

3. Results

Given the breadth of experimentation conducted, and in the interest of brevity, we focus our reporting on general trends that replicate across experimental conditions and on notable exceptions to those trends. In the main text, we focus on the results for the models derived on STARR, and provide, as examples, figures for models that predict 30-day readmission in the STARR cohort using MMD-based penalties, and exclude results for decomposed fairness metrics computed on the strata of the population for which the relevant outcome is not observed. Analogous figures for the complete set of experimental conditions and evaluation metrics are provided in the Appendix.

We observe that, in the absence of fairness-promoting regularization, models exhibit substantial differences in group-level model performance measures (AUROC, average precision, and cross entropy loss), and show clear violation of measures of conditional prediction parity as well as differences in cross-group ranking performance (Figures 2, 3, 4, 5, 6, and 7). These baseline models tend to have low absolute calibration error for each group, with small differences in relative calibration error across groups. For example, the model that predicts 30-day readmission in the STARR cohort shows a large degree of variability in AUROC across groups of the race and ethnicity attribute (Mean AUROC: 0.78, 0.66, 0.77, 0.80, 0.71 for the Asian, Black, Hispanic, Other, and White groups, respectively; Figure 2) and further shows a difference in the mean predicted probability across groups that disproportionately affects the Black population (Figure 3). However, the signed absolute and relative calibration errors by group are small (Mean ${\text{ACE}}_k^{\text{signed}}:-0.0067,-\mathrm{0.0074,}-\mathrm{0.0021,}-\mathrm{0.0059,}-0.0021$; Mean ${\text{RCE}}_k^{\text{signed}}:-0.0027,-0.0022,0.0017,-0.0027,0.00090$ for the Asian, Black, Hispanic, Other, and White groups, respectively; Figures 2 and 3).

Figure 2:

Group-level model performance measures as a function of the extent λ that violation of the fairness criterion is penalized when the race and ethnicity category is considered as the sensitive attribute for prediction of 30-day readmission in the STARR database. Results shown are the mean ± SD for the area under the ROC curve (AUROC), average precision (Avg. Prec), the cross entropy loss (CE Loss), the absolute calibration error (ACE), the signed absolute calibration error (Sign. ACE), and cross group ranking performance (xAUC; ${\text{xAUC}}_k^1$ is indicated by (y=1) and ${\text{xAUC}}_k^0$ by (y=0)) for each group for objectives that penalize violation of threshold-free Demographic Parity, Equalized Odds, and Equal Opportunity with MMD-based penalties. Dashed lines correspond to the mean result for the unpenalized training procedure.

Figure 3:

Fairness metrics as a function of the extent λ that violation of the fairness criterion is penalized when the race and ethnicity category is considered as the sensitive attribute for prediction of 30-day readmission in the STARR database. Results shown are the mean ± SD for decomposed group-level metrics that assess conditional prediction parity (EMD and Mean Diff.) and relative calibration error (RCE and Sign. RCE) for objectives that penalize violation of threshold-free Demographic Parity, Equalized Odds, and Equal Opportunity on the basis of MMD-based penalties. Measures of conditional prediction parity are separately assessed in the whole population and in the strata for which the outcome is observed (suffixed with (y=1)). Dashed lines correspond to the mean result for the unpenalized training procedure.

Figure 4:

Group-level model performance measures as a function of the extent λ that violation of the fairness criterion is penalized when sex is considered as the sensitive attribute for prediction of 30-day readmission in the STARR database. Results shown are the mean ± SD for the area under the ROC curve (AUROC), average precision (Avg. Prec), the cross entropy loss (CE Loss), the absolute calibration error (ACE), the signed absolute calibration error (Sign. ACE), and cross group ranking performance (xAUC; ${\text{xAUC}}_k^1$ is indicated by (y=1) and ${\text{xAUC}}_k^0$ by (y=0)) for each group for objectives that penalize violation of threshold-free Demographic Parity, Equalized Odds, and Equal Opportunity with MMD-based penalties. Dashed lines correspond to the mean result for the unpenalized training procedure.

Figure 5:

Fairness metrics as a function of the extent λ that violation of the fairness criterion is penalized when sex is considered as the sensitive attribute for prediction of 30-day readmission in the STARR database. Results shown are the mean ± SD for decomposed group-level metrics that assess conditional prediction parity (EMD and Mean Diff.) and relative calibration error (RCE and Sign. RCE) for objectives that penalize violation of threshold-free Demographic Parity, Equalized Odds, and Equal Opportunity on the basis of MMD-based penalties. Measures of conditional prediction parity are separately assessed in the whole population and in the strata for which the outcome is observed (suffixed with (y=1)). Dashed lines correspond to the mean result for the unpenalized training procedure.

Figure 6:

Group-level model performance measures as a function of the extent λ that violation of the fairness criterion is penalized when the age group is considered as the sensitive attribute for prediction of 30-day readmission in the STARR database. Results shown are the mean ± SD for the area under the ROC curve (AUROC), average precision (Avg. Prec), the cross entropy loss (CE Loss), the absolute calibration error (ACE), the signed absolute calibration error (Sign. ACE), and cross group ranking performance (xAUC; ${\text{xAUC}}_k^1$ is indicated by (y=1) and ${\text{xAUC}}_k^0$ by (y=0)) for each group for objectives that penalize violation of threshold-free Demographic Parity, Equalized Odds, and Equal Opportunity with MMD-based penalties. Dashed lines correspond to the mean result for the unpenalized training procedure.

Figure 7:

Fairness metrics as a function of the extent λ that violation of the fairness criterion is penalized when the age group considered as the sensitive attribute for prediction of 30-day readmission in the STARR database. Results shown are the mean ± SD for decomposed group-level metrics that assess conditional prediction parity (EMD and Mean Diff.) and relative calibration error (RCE and Sign. RCE) for objectives that penalize violation of threshold-free Demographic Parity, Equalized Odds, and Equal Opportunity on the basis of MMD-based penalties. Measures of conditional prediction parity are separately assessed in the whole population and in the strata for which the outcome is observed (suffixed with (y=1)). Dashed lines correspond to the mean result for the unpenalized training procedure.

As expected, training with an objective that penalizes violation of a measure of conditional prediction parity typically leads to better satisfaction of the fairness criterion that corresponds to the form of the regularizer used in the objective. For instance, training with an unconditional penalty to encourage demographic parity typically minimizes the EMD and difference in means between the distribution of predictions for each group and the corresponding marginal distribution constructed via aggregation of the data from all groups. In some cases, the regularization strategy is less successful at minimizing violation of the targeted fairness criteria, such when using a conditional penalty in the outcome-positive strata to encourage equal opportunity on the basis of sex for the 30-day readmission model in the STARR cohort (Figure 5I). In this case, the relevant fairness criteria is actually violated to a greater extent when λ = 10 (M_EqOpp = 0.0069) than at baseline (M_EqOpp = 0.0056).

We observe a tension between equal opportunity and demographic parity, but these trade-offs are not consistent across experimental conditions. For instance, conditional penalties in the strata for which hospital mortality is observed in the STARR cohort lead to further violation of demographic parity, relative to baseline, across all three of the sensitive attributes that we test (Supplementary Figures D.2, D.4, and D.6). In other cases, this penalty actually leads to improved satisfaction of demographic parity, such as in the case of 30-day readmission prediction in the STARR cohort when age is the sensitive attribute (M_DP: 0.041 for λ = 0 vs. M_DP: 0.0010 for λ = 10; Figure 7C), but at the cost of a major reduction in AUROC and Average Precision for all groups (Figure 7C and 7F). A similar phenomenon is observed when considering the impact of unconditional penalties that target demographic parity on metrics that assess equal opportunity in that two criteria appear to coincide in some cases, such as for the model that predicts hospital mortality in the STARR cohort when race and ethnicity is considered the sensitive attribute (Supplementary Figures D.2) and conflict in others, such when sex is considered to be the sensitive attribute for models that predict prolonged length of stay (Supplementary Figure D.20) or 30-day readmission (Supplementary Figure D.24) in the Optum CDM cohort.

With few exceptions, the effect of increasing the weight on the conditional regularization penalties that target equalized odds or equal opportunity is a monotonic reduction in group-level model performance measures for all groups. In contrast, the effects of unconditional penalties that encourage demographic parity are more heterogeneous. For instance, while the effect of unconditional penalties on model performance measures that we observe over the trajectory of λ are often similar to those that we observe for conditional penalties, we note that unconditional penalties can result in little change in model performance measures (Figures 4 and 6), and in some cases, actually results in improved model performance for one or more groups relative to baseline (Figure 2 and Supplementary Figures D.1 and D.3). For example, penalizing the violation of demographic parity increases the performance of the model that predicts 30-day readmission for the Black population when λ = 3.6 compared to baseline (AUROC: 0.69 vs. 0.66; Average Precision: 0.17 vs. 0.15) (Figures 2A and 2D).

In general, models become less well-calibrated, in the absolute sense, at the group level, as the weight on either conditional penalty increases, as measured by changes in the ACE or signed ACE relative to baseline. In many cases, unconditional penalties seem to have little impact on group-level model calibration relative to that which is observed for conditional penalties (Figure 4), whereas in other cases the effect of the unconditional penalty is similar to that of the conditional penalties (Figure 6). However, both unconditional and conditional penalties can, but do not always, introduce relative calibration error across groups, and in a way that appears unrelated to the changes to absolute calibration. For example, for models that predict hospital mortality in the STARR cohort when age is the sensitive attribute, the magnitude of the effect on absolute calibration differs substantially on the basis of the type of regularization applied (Supplementary Figure D.5), while the effect on relative calibration is similar across all penalties (Supplementary Figure D.6). In cases where the effects on relative calibration are large, such as this set of models, the impact of the effect on relative calibration concentrates in relatively few groups (Supplementary Figure D.6).

We observe heterogeneity in the manner in which training with fairness-promoting objectives impacts measures of cross-group ranking across the combinations of regularizer, dataset, outcome, and sensitive attribute. In many respects, the trajectories of xAUC measures are similar to those that we generally observe for the AUROC, in that primary effect that we observe is a decline in cross-group ranking accuracy as a function of λ, regardless of the type of penalty selected (Figures 2Q, 2R, 2T, and 2U). However, in some cases, the trajectories of xAUC measures are convergent in a way that both improves fairness and allows for an improvement in the measure for at least one group at the expense of one or more other groups. In some cases, we observe this effect only for unconditional penalties (Supplementary Figures D.2, D.4, and D.6) and in others, we observe it for both unconditional and conditional penalties (Supplementary Figures D.10 and D.12).

4. Discussion

Our experiments aim to provide a comprehensive empirical evaluation of the effect of penalizing group fairness criteria violations on measures of model performance and group fairness for clinical predictive models. Our results reveal substantial heterogeneity in the effect of imposing measures of group fairness across datasets, outcomes, sensitive attribute and group definitions, and regularization strategies. These results quantify the extent to which the trade-offs among measures of model performance and group fairness described by “impossibility theorems” [57, 58, 72, 90] manifest when learning clinical predictive models from real-world databases.

We acknowledge technical limitations of our work that may limit the generalizability of our results. First, the regularizers and metrics used to quantify conditional prediction parity and relative calibration are the result of “one vs. marginal” comparisons where a measure computed for one group is compared to the measure computed for the aggregate population. This choice is one of several ways to construct fairness metrics, including “one vs. other” comparisons between one group and all other groups and pairwise comparisons across all pairs of groups. An effect of this choice is that metrics that assess violation of conditional prediction parity for an over-represented group are more likely to be small since the over-represented group comprises a larger fraction of the population than under-represented groups do. Furthermore, our use of penalized objectives could exaggerate the extent of the reported trade-offs, relative to the alternative of a Lagrangian formulation that directly encodes the fairness criteria as a constraint [36–38, 66, 78]. While the constrained approach typically only provides guarantees of constraint satisfaction in the case of a convex objective, recent work has demonstrated empirical success with a modified proxy-Lagrangian formulation that is effective for non-convex constrained optimization problems [36]. It remains to be seen whether reformulating the problem as constrained optimization allows for satisfaction of fairness constraints with less severe trade-offs than those reported here.

Our work inherits the fundamental limitations of the group fairness framework and of algorithmic fairness more broadly. The group fairness framework, which arose from legal notions of anti-discrimination, reinforces a perspective that groups based on categorical attributes are well-defined constructs that correspond to a set of homogeneous populations – a perspective that has several problematic implications. For example, the definitions of racial categories are entangled with historical and on-going patterns of structural racism, and their continued use reinforces the idea of race as an accurate way to describe human variability, rather than a socially constructed taxonomy [6, 11, 48, 50, 91–94]. This framework further marginalizes groups that are not well-represented by the attributes used to assess group fairness, including intersectional identities [48, 61, 95–97]. Furthermore, in addressing each attribute independently, the group fairness framework treats various sensitive attributes as abstract, interchangeable constructs, without awareness of meaningful contextual differences between them. For example, while observed differences on the basis of race should be primarily interpreted as deriving from systemic and structurally racist factors [6, 11, 48, 50, 91, 92], those observed for sex could potentially be attributed to clinically meaningful differences in human physiology as well as sociological factors [98].

Alternative forms of algorithmic fairness raise additional normative questions that require context-specific judgement and domain knowledge. Individual fairness measures require robustness over a metric space that encodes domain-specific norms, which concern how outputs of an algorithm may change over the space of observed covariates [21]. Counterfactual fairness provides a particular instantiation of individual fairness, defining closeness in the domain-specific metric in terms of counterfactuals with respect to a sensitive attribute [29, 99]. However, this requires specification of the causal pathways between the sensitive attribute, outcomes, and discrimination. It further assumes manipulability of sensitive attributes within the context of a well-defined structural equation model, which is particularly unrealistic for complex high-dimensional data and contestable whenever race is considered to be the sensitive attribute [48, 92].

4.1. Recommendations

Striving for health equity requires designing policies that directly counteract the systemic factors that contribute to health disparities, primarily structural forms of racism and economic inequality [11, 100]. By considering only changes to observable properties of a model, evaluations of models using the group fairness framework ignore the inequities in the data generating and measurement processes. They also miss the decision-theoretic and causal framing necessary to connect predictions to the interventions they trigger, as well as the resulting downstream effect on health disparities [2, 45, 101]. In the absence of this context, a requirement that a predictive model satisfy a notion of group fairness provides little more than a “veneer of neutrality” [17, 102]. Overall, constraining a model such that group fairness is achieved is insufficient for, and may actively work against, the goal of promoting health equity using machine learning guided interventions. To be sure, this lack of sufficiency does not mean that explicitly optimizing for fairness criteria satisfaction can not be useful. However, the value of achieving algorithmic fairness should be defined in terms of the impact of an algorithm-guided intervention on individuals, groups, and on status quo power structures that directly or indirectly perpetuate health disparities [103].

In light of these limitations, model developers in healthcare should engage in participatory design practices that explicitly incorporate perspectives from a diverse set of stakeholders, including patient advocacy groups and civil society organizations. Doing so is necessary to identify mechanisms through which measurement error, bias, and historical inequities affect data collection, measurement, and problem formulation, as well as to reason about the mechanisms by which the intervention triggered by the model’s prediction interacts with those factors [20, 104–106]. However, it is important to allow that the conclusion derived from this process may be to abstain from algorithm-aided decision making entirely if it is not practical to do so responsibly [107, 108].

5. Conclusion

The debate on the use of algorithmic fairness techniques in healthcare has largely proceeded without empirical characterization of the effects of these techniques on the properties of predictive models derived from large-scale clinical data. We explicitly measure and comprehensively report on the extent of the empirical trade-offs between measures of model performance and notions of group fairness such as conditional prediction parity, relative calibration, and cross-group ranking. These constructs are generally well-understood in theoretical contexts, but under-explored in the context of clinical predictive models. Given our results, and the known limitations of the algorithmic fairness framework, we recommend that the use of algorithmic fairness for proactive monitoring and auditing of a clinical predictive model proceed only if measures of model performance and fairness can be appropriately contextualized in terms of the impact of the complex intervention that the model enables.