Skip to main content
. 2022 Feb 7;161(6):1621–1627. doi: 10.1016/j.chest.2022.02.001

Table 1.

Recommendations at Each Step of the Framework to Integrate Equity Into Machine Learning Algorithmsa

Model design
  • Ensure that the primary goal of the machine learning algorithm is achieving optimal equitable patient outcomes.
    • Review the relevant literature and community-level data for your patient service area to learn about the existing health inequities related to the health condition(s) that will be addressed by the machine learning algorithm.
    • Discuss how and why this new machine learning algorithm could create new disparities or could exacerbate existing disparities.
  • Ensure that the goals of the model developers and users are aligned together and consistent with overall goal of achieving health equity (as outlined in Fig 1).

  • Discuss how potential nonclinical goals such as efficiency, saving money, and increasing revenue relate to the clinical goals and how these nonclinical goals may impact the overall goal to achieve health equity.

  • Check that the model development and review team have sufficiently diverse expertise and perspectives.
    • Required stakeholders include patients living with the health conditions that the machine learning algorithm will address and their family members.
    • Identify a stakeholder leader or facilitator with the training and expertise to lead group discussions on potentially emotionally charged and difficult topics, such as racism and other forms of oppression and discrimination.
Data collection
  • Check for bias in the historical data on which the model will be developed.

  • This step is vital to ensure that the protected group is represented adequately in your potential model’s outcome and predictors. The data scientist developing the model should evaluate for the following biases in the data:
    • Minority bias: insufficient numbers of the protected groupb
    • Cohort bias: groups defined so broadly that more granular protected groups are not identified
    • Missing data bias: data missing from protected groups because of nonrandom, biased reasons
    • Informativeness bias: features (predictor variables) are less informative in the protected group
    • Label bias: outcome (label) is an imperfect proxy, rather than the truth for protected group because of health disparities
    • Training-serving skew: Training data not representative of the groups on which the model will be used.
Model training and validation
  • Intentionally consider model performance metrics, patient outcomes, and resource allocation during model development.

  • Data scientist to use machine learning techniques that reduce overfitting and have been shown to be accurate in predicting the same clinical outcome or similar clinical outcome in the literature.

  • Model developers should share the patient demographics of the training dataset transparently to ensure that this information is available to future data scientists who may want to use the model.

  • Evaluate performance metrics such as sensitivity, specificity, positive predictive value, false-positive rate, negative predictive value, area under the receiver operating characteristic curve, and area under the precision-recall curve for the outcome of interest in the test dataset across potential demographic variables.
    • This includes race, ethnicity, age, sex, sexual orientation, gender identity, socioeconomic status, payer status, religion, citizenship status, preferred language, and disability.
Evaluation of model deployment
  • Discuss with diverse stakeholders whether to deploy the model, explicitly considering the ethical concepts of health equity and transparency, and the analysis of model performance metrics, patient outcomes, and resource allocation.
    • Based on the model’s validation data, key stakeholders can decide whether the model should be deployed.
    • If differences are found in the model’s performance across patient groups, the stakeholders will have to decide how much difference is too much to deploy the model.
    • How are stakeholders ensuring that interventions are tailored to meet the needs of different patient groups?
    • In larger health systems, the modeling team needs to ensure that equity exists in the deployment locations of predictive models across their clinics and hospitals.
Monitored deployment
  • Launch and evaluate the model, incorporating feedback of diverse stakeholders.

  • All models that are deployed in the clinical environment must be monitored actively by a dedicated team to check whether the model’s accuracy changes over time and to ensure that frontline clinicians are using the model as intended.
    • This requires both continued evaluation of model performance (quantitative feedback) and stakeholder focus groups or interviews (qualitative feedback).
    • A hospital or health system modeling team should monitor that the model is not deployed to a new patient population without evaluating the model’s accuracy in this new population.
  • Look for biases in interactions with clinicians:
    • Automation bias: clinicians automatically act on model that is less accurate for the protected group
    • Feedback loops: if clinicians accept the recommendation of a model even when it is incorrect to do so, the model’s recommended and actual prescribed treatment will overlap, and if the model is updated in the future with newer data, it will learn to continue these mistakes
    • Dismissal bias: clinicians compensate for model that is less accurate in protected groups by ignoring the model’s recommendations
    • Allocation discrepancy: fewer resources allocated to the protected group because that group has disproportionately fewer positive predictions.
  • Look for biases in interactions with patients:
    • Privilege bias: protected group has less access to the model or the benefits that could result from the model, which could be identified with qualitative data from patients and clinicians
    • Informed mistrust: protected group avoids care that uses the model because of prior or current exploitation, or both, or unethical practices
    • Agency bias: protected group does not have input into the development and deployment of the model, which is something we hope is mitigated by using our proposed framework.
a

Biases based on the definitions of Rajkomar et al.13

b

Populations that experience individual and structural biases are the protected group in this table.