Addressing health disparities in the Food and Drug Administration’s artificial intelligence and machine learning regulatory framework

Abstract

The exponential growth of health data from devices, health applications, and electronic health records coupled with the development of data analysis tools such as machine learning offer opportunities to leverage these data to mitigate health disparities. However, these tools have also been shown to exacerbate inequities faced by marginalized groups. Focusing on health disparities should be part of good machine learning practice and regulatory oversight of software as medical devices. Using the Food and Drug Administration (FDA)'s proposed framework for regulating machine learning tools in medicine, I show that addressing health disparities during the premarket and postmarket stages of review can help anticipate and mitigate group harms.

INTRODUCTION

The United States has stark health disparities. Racial and ethnic minority groups have lower overall life expectancy and worse outcomes for major diseases, including heart disease, and higher prevalence of others such as diabetes.¹ Though some population-specific biological differences explain some of these disparate outcomes, gaps between groups can largely be attributed to social factors, such as differences in exposure to environmental hazards, access to high-quality food, and effects of structural racism and discrimination in medical facilities and society at large.^2,3 The coronavirus disease 2019 (COVID-19) pandemic brings these disparities into sharp relief.⁴ Addressing these social determinants of health is key to mitigating health disparities and achieving health equity in the United States.

Though addressing social issues are important, technological developments offer promising avenues to pursue. The exponential growth of health data from devices, health applications, and electronic health records coupled with the development of data analysis tools such as machine learning (ML) offer opportunities to leverage these data to mitigate health disparities. However, these tools have also been shown to exacerbate inequities faced by marginalized groups, such as recidivism risk scoring algorithms that have higher error rates for Black defendants and higher error rates for women in mortality prediction algorithms.^5,6 Despite the growing evidence of these kinds of negative impacts of algorithmic technologies, the recently proposed regulatory framework governing ML applications in health care from the Food and Drug Administration (FDA) makes explicit mention of neither the potential for group harms nor how these tools could either exacerbate or mitigate health disparities. As ML gains ground in health, considerations of health disparities and the potential for group harms must be central to comprehensive regulatory policy. In this commentary, I argue that in order for the FDA to ensure patient safety with the use ML in medicine, regulation must (1) require attention to health disparities and the potential for group harms as part of the clinical evaluation process in premarket review and (2) continue this focus on disparities and group harms through a “health equity review” that is part of postmarket, real-world performance monitoring. This health equity review can be an inclusive process that includes multiple stakeholders, including device manufacturers, patients, clinicians, and informatics professionals.

THE FDA’S PROPOSED REGULATORY FRAMEWORK FOR ARTIFICIAL INTELLIGENCE AND ML IN MEDICINE

The FDA has issued guidance regarding the use of SaMD (software as a medical device), recognizing the growing importance of data analytics tools in health.⁷ In April of 2019, the agency proposed a regulatory framework for ML tools in medicine, noting that these tools demand distinctive regulatory consideration because they may change and adapt after their initial development and deployment, and use new data to improve their performance and effectiveness.⁸ The document features the agency’s effort “to reimagine an approach to premarket review for artificial intelligence (AI)/ML-driven software modifications” that is a “new, total product lifecycle (TPLC) regulatory approach.”⁸ This approach has several components: a demonstration of good ML practices by the device maker; an assessment of the risk of the ML tool as well as plans for updating it with new data or new algorithm architecture; an ongoing review by device makers to check whether any changes to the ML tool as it is deployed deviates significantly from the premarket proposed changes; and ongoing, postmarket transparency about the tool’s performance through updates to the FDA, collaborators, clinicians, and the public.

As part of premarket review, manufacturers must have good ML practices in place, which the document describes as demonstrating the ML tool’s valid clinical associations, analytical validity, and clinical validity. However, these having these “good practices” in place may not be sufficient to prevent group harms. A recent article by Obermeyer et al⁹ provides an instructive example of the missteps that can befall an algorithm that does not consider health disparities and the potential for group harm as part of good ML practices. In this case, the amount of money spent on patient care was selected as a proxy measurement of health status, as generally more money is spent on sick people than healthy people.⁹ The purpose of the model was to direct more healthcare resources to sicker people. However, this tool ended up favoring healthier white patients over sicker Black patients. If the model developers had been required to demonstrate knowledge of health disparities that exist in their domain of interest, they may have discovered that in the United States, less money is spent on Black patients, even when they are sicker.¹⁰ This discovery could have led to adjustments to their input data, and scrutiny of the validity of the association between healthcare expenditure and illness before the model was developed. This example shows that even though the model worked, in the sense that it used accurate data and “correctly processed” these data as the FDA outlines in the proposed framework, it still favored white patients over Black patients.

Rajkomar et al¹¹ have argued that even attention to how a model performs between groups may not be enough to ensure that health disparities are not exacerbated. For example, they note that it is impossible for a model to have both equalized odds and equal positive and negative predictive value across groups.¹¹ What this means is that model developers must make these kinds of decisions about performance, and these decisions could have significant impacts, depending on the groups in question and the existing health disparities. The FDA’s current regulatory framework does not draw attention to this in its description of good ML practices. Under the current framework, there is little substantive difference between a decision for the model to have equal performance across groups or to have equal positive and negative predictive values across groups, as either choice would fulfill the requirements for the model to be analytically valid. This is a blind spot in the proposed regulation that must be addressed.

In addition to demonstrating good ML practices, and submitting an algorithm change protocol detailing how the ML tool might change as it learns from ingesting data when it is deployed, the FDA also proposes that device manufacturers conduct “real-world performance monitoring” that documents how the tool performs under actual clinical conditions after the device has been approved. Currently, the proposed regulation asks device makers to update the FDA on the tool’s performance and suggests updating others, such as clinicians and patients. The document also lists potential changes that the agency should be made aware of like “change in inputs,” and updates to the “specifications or compatibility of any impacted supporting devices.”⁸ The real-world performance of the tool is defined in terms of purely technical aspects, and does not include any mention of how nontechnical factors might impact the tool’s performance, effectiveness, and intended purpose, especially on groups already experiencing health disparities.

Consideration of factors that are known to influence health disparities such as differential healthcare access, structural discrimination, and clinician bias should be part of a health equity review that would be part of the larger process of real-world performance monitoring of AI/ML devices. IDx-DR, the first AI-powered diagnostic SaMD approved by the FDA, can serve as a thought example to illustrate the potential benefits of including a health equity review as part of AI/ML performance monitoring. IDx-DR is an AI-powered tool that detects diabetic retinopathy (an eye disease caused by diabetes); this disease is marked by significant racial disparities. African Americans have a 4 times higher risk of developing this condition than do non-Hispanic whites, and this group is also less likely than whites to have eye examinations, which can detect diseases such as diabetic retinopathy (DR).¹²

The IDx-DR tool allows for the screening of DR in primary care settings, as the system analyzes the images and provides the diagnosis, rather than the clinician. Because nonspecialist clinicians can use the tool as part of primary care, the IDx-DR tool could lower some of the barriers to receiving eye exams. As an article on the tool notes, “[f]or people with diabetes, autonomous AI systems have the potential to improve earlier detection of DR, and thereby lessen the suffering caused by blindness and visual loss.”¹³

There is some evidence that cost of testing and lack of or limited health insurance can explain why African Americans are less likely to receive eye examinations than whites are. However, this gap may also be due to clinicians being less likely to offer eye examinations to Black patients.¹⁴ Though the data on this particular factor are emerging, it dovetails with other research that shows that clinicians are less likely to offer Black patients multiple forms of medical treatment and screening.^15–17

A health equity review that would be part of periodic real-world performance monitoring of the IDx-DR tool would include information on its use across different racial groups. Ostensibly, because this tool is being implemented in primary care, its use across groups should mirror population demographics. If the use rates deviate from what is expected, then over- or underuse of the tool in specific groups should be investigated. For example, if the tool is being used less often with Black patients, this could signal a form of clinician bias, in which the screening tool might not be used as often due to a belief that Black patients would not proceed with follow-up care. The tool might continue to improve its performance in disease detection overall, and even could demonstrate better disease detection within particular groups, but these improvements might be overshadowed if certain groups are less likely to be offered and receive the test over time. If this were the case, instead of lowering barriers, this AI tool might be exacerbating health disparities. Alternately, overuse of the tool among Black patients might suggest that clinicians see this group as at a higher risk of DR, and might be using the tool as a way to respond to higher vulnerability in this patient population. In this case, using the IDx-DR tool in the “real-world” clinical setting might be both accurately detecting this disease overall and helping to ameliorate health disparities.

RECOMMENDATIONS

The FDA regulates products to ensure that they are safe for consumers. With ML tools in health, health disparities are a safety issue, as inattention to potential negative impacts can increase the risk and danger to groups already marginalized and discriminated against in health care. Consideration of health disparities must not be out of scope, or an optional dimension to consider when developing ML tools for medicine. To that end, considerations of health disparities can be integrated into the FDA’s AI/ML regulation in 4 ways, in both the premarket and postmarket stages (Figure 1):

Figure 1.

This figure is adapted from the FDA’s Overlay of the TPLC approach on AI/ML workflow, in the Proposed Regulatory Framework for Modifications to Artificial Intelligence/Machine Learning (AI/ML)-Based Software as a Medical Device (SaMD). Author’s recommendations are in bold italics.

Premarket review and good ML practices

1. Knowledge of Health Disparities. Device manufacturers should include evidence of knowledge of health disparities in their clinical domain of interest. Having this background knowledge and context can aid in the design of the ML tool as well as in the monitoring of its performance. This could be included as part of their demonstration of culture of quality and organizational excellence. This kind of background knowledge will help manufacturers appropriately identify which groups in their clinical area of interest should be classified as protected or marginalized groups.

2. Data Bias Review. Device manufacturers should document decisions that were made regarding the data that were used to train the model, including the representativeness of protected groups in the sample(s). This should include whether there is a representative sample, or if these groups are under- or overrepresented. The data bias review should also document an examination of the potential for latent biases in data, such as data that reflect histories of unequal access to health care, data that may be present but not as informative for all groups, or data that reflect racial, gender, or other corrections that may be clinically questionable or disputed. The Algorithm Change Protocol would include checks to ensure that any new data added to the model would be examined for data bias.

3. Group Impacts and Performance Decisions: Device manufacturers should document decisions that were made regarding the performance of the model, including discussions of how different choices can impact groups already experiencing health disparities in the clinical condition of interest. The rationale for how these decisions were made would ideally include input from multiple stakeholders, including clinicians and patients from groups experiencing health disparities in the clinical domain of interest.

Postmarket real-world performance monitoring

4. Health Equity Review: Device manufacturers’ updates to the FDA should include specific information on how the ML tool is impacting protected groups, and whether any new group differences have emerged because of the deployment of the tool. This update would also include details about any activities undertaken to address unfairness or group harms, even if these activities do not involve changes to the model’s inputs or architecture. Device makers may also identify how the use of their tool has mitigated health disparities. Ideally, this update would reflect an inclusive process that includes engagement, involvement, and feedback from multiple stakeholders including patients, clinicians, informatics professionals, and others.

The FDA’s Office of Minority Health and Health Equity could be involved in implementing these recommendations. This office currently works on issues such as diversifying clinical trials and research on population differences in disease biomarkers, treatments, and health communication and messaging. The office could help guide device manufacturers in expanding their good ML practices to include attention to health disparities, as well as provide assistance in building diverse partnerships for postmarket evaluation and monitoring.

CONCLUSION

These recommendations could be part of a first step in a journey toward comprehensive ML policy for advancing health equity. The FDA has argued that regulatory processes for software used in medicine should be “agile” in order to “accommodate the faster rate of development and potential for innovation in software-based products.”¹⁸ This is a laudable goal. However, an emphasis on “agility” should not come at the expense of those already experiencing unfair health access, treatment, and health outcomes. ML offers promises of efficiency, cost savings, and better health outcomes, but regulation of this burgeoning field must explicitly address and call out health disparities, rather than leave unmentioned how these tools could have significant impacts on marginalized groups—both positive and negative. The current COVID-19 pandemic is yet another illustration of just how entrenched health disparities are in the United States. Health care and society at large are once again in the position of reacting to drastic differences in disease incidence, treatment, and outcomes in this pandemic. Though ML tools alone will not address the multiple historical and structural forces that influence health disparities, the FDA could take a more proactive stance in regulation so that these tools anticipate that there could be increased risks for harm for those already experiencing health disparities. The guidance for ML tools in health is still in development, and not yet final, so there is an opportunity to think carefully and deliberately about good governance of AI/ML in medicine.

AUTHOR CONTRIBUTIONS

KF is the sole author of this work.

CONFLICT OF INTEREST STATEMENT

The author has no competing interests to declare.