Promoting Ethical Deployment of Artificial Intelligence and Machine Learning in Healthcare

INTRODUCTION

The ethics of artificial intelligence (AI) and machine learning (ML) exemplify the conceptual struggle between applying familiar pathways of ethical analysis versus generating novel strategies. Melissa McCradden et al.’s “A Research Ethics Framework for the Clinical Translation of Healthcare Machine Learning” puts pressure on this tension while still attempting not to break it—trying to impute structure and epistemic consistency where it is currently lacking (McCradden et al. 2022). They highlight an “AI chasm” “generated by a clash between the…cultures of computer science and clinical science,” but argue that the “ethical norms of human subjects research” are still the right pathway to bridge this divide.

The Open Peer Commentaries included in this issue agree with this central premise while critiquing the insufficiency of current ethics and regulatory solutions to adequately protect communities at higher risk for ML bias. The current U.S. human subjects research regulations (HSRR) were developed from traditional conceptions of research ethics (Schupmann and Moreno 2020). Research ethicists have subsequently been asked to apply existing concepts to new areas over which the regulatory structure does not reach. AI/ML is an excellent example of the strengths and limitations of this current default.

STAGE 0: DATASET GENERATION AND PROCUREMENT

McCradden et al. propose an evaluation framework that considers ethical issues in clinical applications of ML. An initial critic of this framework is where, exactly, that analysis should begin. The proposed framework begins with hypothesis-generating data access, moves to a ‘silent’ period of evaluation, and then ends with the post-evaluation of ML models in clinical settings. But Kadija Ferryman questions whether evaluating justice might be a useful analysis earlier in its lifecycle (Ferryman 2022). Doing so would likewise address concerns, which Effy Vayena and Alessandro Blasimme warn against, where ML algorithms carry forward biases reflected in the data on which they are trained (Vayena and Blassime 2022). Nicole Martinez-Martin and Mildred Cho agree. While clinical evaluation can measure select impacts, it cannot correct non-representative data in the learning models (Martinez-Martin and Cho 2022).

Ethics and equity evaluation is needed earlier in ML development, including during data generation and procurement, to ascertain who is (and is not) represented and why. Biases include the fact that training datasets generated from “patients” represent people with access to healthcare and datasets generated from “research participants” reflect those recruited to enroll. There are also known differences in consent rates between historically included and excluded populations (Spector-Bagdady et al. 2021). As Margaret Levi et al. argue, these early stages of development are critical (Levi, Bernstein, and Waeiss 2022). Far from preliminary work, dataset generation and procurement can have significant downstream implications and disparate impacts for some patients and their communities (Benjamin 2019).

LIMITS OF THE LAW

Authors also debate the usefulness and applicability of the current U.S. approach to governing research with humans. These ethical and legal frameworks are cyclical insofar as principlism initially drove the conception of the regulations, and research ethicists then arecalled upon to fill remaining and new gaps in regulatory oversight due to advancing technologies, perspectives, and methodologies. Closing the McCradden et al. “AI chasm” is harder due to these regulatory gaps—which may require more flexibility on the part of ethics.

One legal challenge to controlling all human subjects research in the United States is that the law must have a regulatory hook—a legally and (sometimes) socially justifiable right to curtail the free choice of individuals. One prospective source of that right derives from the researcher’s use of government funds. The Office of Human Research Protections (OHRP)-governed HSRR (the first subpart of which is referred to as the “Common Rule”) apply to research supported by the federal offices and agencies which signed it (45 CFR § 46.101(a)). But most academic medical centers require all researchers to follow the HSRR across funding types. Thus, the regulations from a practical perspective cover much academic human subjects research.

Another potential regulatory hook is that if researchers want to submit their data in support of a Food and Drug Administration (FDA) application (e.g., for a new device) the researchers must have followed FDA’s research regulations (which are substantially like the OHRP ones) (21 CFR §§ 50, 56). Thus, while technically true that researchers prospectively must follow the regulations only if they have federal funds, practically many academic researchers must do so anyway, and retrospectively many industry researchers must have done so to market their ML device. But as Vayena and Blasimme point out, the “wicked problem” of AI/ML is that research building the models is often done with private funding and FDA does not consider many ML systems to be devices over which they have authority. Many ML technologies, therefore, fall squarely into this legal chasm (Price et al. 2019). FDA is currently attempting to fill this gap, and, as Ho and Malpani argue, there are other promising approaches internationally (Ho and Malpani 2022), but the gap persists in the United States.

An additional legal limitation is that research involves a “human subject” under the HSRR only if it is interventional or involves identified data or biospecimens. When ML research only uses de-identified data or specimens, it is neither covered by the OHRP nor the FDA regime and has no Institutional Review Board (IRB) review or informed consent obligations. Even if data or specimens are identified, researchers can also apply for an IRB waiver of consent if their work is low risk. This generates additional differences regarding the consent status of data that ends up in ML training sets: some are from interventional or clinical trials where participants fully consented to the primary research as well as many secondary uses (as required by the revised Common Rule) with attendant biases in who is recruited and enroll, but others are generated from pragmatic clinical trials or other types of low-risk research, receive an IRB waiver, and neither require consent from nor even notice to patients whose medical records are used (Morain and Largent 2021) but are sometimes more demographically diverse (Spector-Bagdady et al. 2021).

Under the current legal regime, if we were to increase protections for all ML research, that would involve: 1) the government regulating all human subjects research no matter funding or whether it involves an FDA-marketed product, 2) changing those regulations such that de-identified data protocols are required to go through IRB review, and 3) that individual informed consent (the sharpest tool in the IRB box) could alleviate community-level equity issues. The enormity of these tasks brings us back to the creativity of the proposals in this issue.

ETHICS-DRIVEN SOLUTIONS

The HSRR are intended to (however imperfectly) be responsive to the risks of research. Research involving physical interventions and manipulations tops this hierarchy, research involving identified data and specimens that is otherwise low risk is at the bottom, and research with de-identified data and specimens is not even covered. Given the limitations of a legal structure based on traditional notions of research ethics—which focus on avoiding physical or psychological harm to individuals due to their creation in the wake of research scandals like the Public Health Service’s syphilis experiments in Tuskegee—McCradden et al. call upon us to again rethink how to approach novel tensions. A new set of characteristics upon which AI/ML should be evaluated may include non-welfare interests (De Vries et al. 2016), pernicious bias (Obermeyer et al. 2019), or, as James Shaw articulates, the “adaptive nature of data science, opaque analytic methods, and non-hypothesis driven uses of secondary data” (Shaw 2022).

As aptly explored by Tijs Vandemeulebroucke and colleagues, these risks generally redound at the community level (Vandemeulebroucke et al. 2022). But, as Levi et al. point out, the IRB framework specifically “excludes review of risks to society…” (Levi, Bernstein, and Waeiss 2022) and gives IRBs no legal authority to require review for, or opine on the community impact of, AI/ML protocols (Bernstein et al. 2021). Also, as Danton Char argues, ethical evaluation run by a certain group can be just as biased as the outcomes they are purporting to assess (Char 2022)— particularly if the IRB is directly hired by commercial companies that do not have a standing academically oriented IRB. And, even if an IRB found a protocol to be riskier due to these factors, individual informed consent cannot resolve community-level risk (Lynch, Wolf, and Barnes 2019). From this perspective, IRBs might not be the proper pathway to mitigate the risks of AI/ML as they have limited legal authority over which protocols they review in the first place, their focus is inherently on the individual, and their remit for high risk is only rejection if the risks outweigh the benefits or individual consent.

This is a hard problem. McCradden et al., as well as others in this issue, highlight important facets of it and make novel proposals regarding its resolution. Whether it can be resolved with existing regulatory tools or requires new ethical approaches, we defer. But whatever commonalities AI/ML research has with previous autonomy-based case studies, its implications have important differences in terms of risk of impact—and this kind of thoughtful deliberation is a critical step toward its resolution.