Patient Consent and The Right to Notice and Explanation of AI Systems Used in Health Care

Abstract

Given the need for enforceable guardrails for artificial intelligence (AI) that protect the public and allow for innovation, the U.S. Government recently issued a Blueprint for an AI Bill of Rights which outlines five principles of safe AI design, use, and implementation. One in particular, the right to notice and explanation, requires accurately informing the public about the use of AI that impacts them in ways that are easy to understand. Yet, in the healthcare setting, it is unclear what goal the right to notice and explanation serves, and the moral importance of patient-level disclosure. We propose three normative functions of this right: (1) to notify patients about their care, (2) to educate patients and promote trust, and (3) to meet standards for informed consent. Additional clarity is needed to guide practices that respect the right to notice and explanation of AI in healthcare while providing meaningful benefits to patients.

INTRODUCTION

The widespread use of artificial intelligence (AI) raises some of the most pressing ethical issues of our time. As AI becomes increasingly used for decision-making purposes in high-impact sectors like healthcare that have a large impact on the public’s well-being, consensus is forming around the need for enforceable guardrails for its creation and implementation. To this end, the Biden administration recently issued an Executive Order to ensure responsible and safe AI, following the U.S. Office of Science and Technology Policy’s (OSTP) release of a Blueprint for an AI Bill of Rights (AIBoR)—a set of five principles and accompanying practices to help guide the design, use, and implementation of AI systems while simultaneously protecting the public from potential harms in our age of constantly evolving technology (Blumenthal-Barby 2023; The White House 2023; Office of Science and Technology Policy 2022). The Blueprint is intended to underpin the creation of new policies and practices. A disclaimer on the AIBoR site emphasizes that the Blueprint does not create any actual legal right or constitute policy. Rather, it offers guiding principles for later development and application. The language of the AIBoR is also sector-neutral, though it nods throughout to specific applications of AI systems, like healthcare diagnostics, and suggests the need for further sector-specific guidance in the future. While some of the principles focus on the responsible development of AI systems, such as ensuring accuracy and validity and representative data sets, one in particular, the right to notice and explanation (RNandE), highlights the importance of keeping individuals informed about the use of AI technologies that could meaningfully impact their life. In the context of healthcare, this means keeping patients informed about any use of AI systems in their care. The proposed right explicitly addresses patients saying that “you should know that an automated system is being used and understand how and why it contributes to outcomes that impact you.” The Blueprint specifies the need for notice and explanation in a format that is technically valid, simple (in plain language), and up to date.¹

The idea of a right to notice and explanation in healthcare underscores a commitment to patient engagement and education in delivering relevant, available information to patients and the public (Paterick et al. 2017). The relevance of such a right is expected to grow as clinicians rapidly integrate AI into clinical practice, with applications in precision medicine, surgery, diagnostic radiology, and risk stratification (Johnson et al. 2021; Mun et al. 2020; Bohr and Memarzadeh 2020; Hashimoto et al. 2018).

In its current form, the AIBoR posits that the RNandE can help protect patients and the public by facilitating timely identification of errors made by AI systems, increased safety and efficacy by allowing expert oversight to verify the reasonableness of AI-mediated recommendations, risk assessment and validation, potential for contestation or remedying the impacts of automated systems on human lives, and a greater public understanding of decisions made with AI that impact them. While these are all admirable and justifiable aims, there is a lack of clarity about what the right to notice and explanation is meant to achieve and how (and why) it is morally important at the individual patient level in healthcare. Applying the RNandE to the clinical setting demands that patients have a right to know when, where, how, and why an AI was utilized in their care. Yet, given the potential burden that this right may place on clinicians, clinical flow, and demand for new healthcare professionals (i.e. AI experts and communicators hired by hospitals), it should be clear what exactly the right is attempting to achieve in a patient care setting. Indeed, a 2017 article from International Data Privacy Law describes exactly why a right to explanation of automated decision-making does not exist in the General Data Protection Regulation (GDPR) which, despite applying only to Europe, deals with similar concerns tackled in the AIBoR (Selbst and Powles 2017). Articles such as this bring into question the ultimate value and function of a right to notice and explanation. For this reason, our paper focuses less on the goal of the Blueprint as a whole, and instead explores the normative purpose of patients having a RNandE about how and why an AI system was used in their healthcare. We propose three potential normative functions of notice and explanation with respect to the use of AI in patient care, each with incrementally increasing levels of demandingness: (1) mere notification to keep patients in-the-loop about tools and technologies used in their care, (2) increased patient understanding and trust with respect to AI, and (3) as a necessary component of valid informed consent to the use of these technologies and/or the medical decisions that make use of them. Greater clarity about the purpose of providing patients with information that AI was used in their healthcare can help guide the development of practices which provide meaningful notice and explanation of AI-mediated tools to patients in clinical settings.

THREE USE CASES

Before advancing our arguments, it will be helpful to provide an overview of what the right to notice and explanation might look like in a variety of clinical examples, in part to acknowledge the significant heterogeneity in both the forms AI tools take and their applications in healthcare. These use cases do not encompass the entire range of applications of AI in healthcare that could be discussed but help to illustrate some of the potential practical implications of the RNandE². The three use cases we will discuss include:

1. AI in mental health chatbots

2. AI in diagnostic tools

3. AI in prognostic tools

Mental Health Chatbots

Within the span of several months following the release of ChatGPT3 in December of 2022, there was a significant rise in the development and advertising of AI-mediated chatbots used for mental health. Most often advertised as a cheaper, quicker, and more accessible way to provide mental health care to the average individual, many companies began to deploy and advertise such AI-powered mental health chatbots with varying abilities. Some companies have chosen to brand their AI chatbots as AI “therapists,” capable of providing treatment information and preventative measures in addition to even facilitating psychotherapy with patients. Woebot Health, as an example, claims on their website to “digitize CBT” and deliver therapeutic techniques with the “insight and empathy inherent in human relationships” (Woebot Health 2024). Other companies promise less from their chatbots, focusing instead on teaching individuals new skills and techniques to improve mindfulness and recognize detrimental behavioral patterns (Sedlakova and Trachsel 2022). Recent developments in large language models (LLMs) such as ChatGPT may soon contribute significantly to the improvement of these chatbots. Trained on vast amounts of text from the Internet, some proponents suggest that these models may eventually (and soon) be capable of offering to patients more useful clinical feedback than some human therapists can provide (Eliot 2023; Reardon 2023; Sharma et al. 2023). One issue that needs sorting out, however, is that some individuals who are notified that the entity they are speaking with is a trained AI rather than a human are likely to respond with aversion (Lee and Lee 2023; McGuire et al. 2023; Mozafari, Weiger, and Hammerschmidt 2021; Schuetzler, Grimes, and Giboney 2019; Weizenbaum 1966). In this case, what is the desirable response? While certainly patient preferences (and aversions) should be at the forefront, other considerations may be important as well, including whether interactions with the chatbots were observed to have clinical benefits for certain patients, and (controversially) whether effective care is available from human healthcare professionals, given the resource constraints of that particular setting. In some cases, N&E may introduce aversions to receiving AI-assisted care in ways that lead to objectively worse outcomes, though over time, such reluctance to AI tools may wane as the integration of AI systems into healthcare continues and their presence becomes routine.

Diagnostic Tools

AI is increasingly used in healthcare to identify and diagnose medical conditions such as lung and breast cancer, skin lesions, and diabetic retinopathy using a wide variety of data inputs, including biometrics, medical images, patient medical history, lab results and digital health data from computer perception (sensor) devices (J. S. Ahn et al. 2023; Huang et al. 2023; Smak Gregoor, Sangers, Bakker, et al. 2023; Smak Gregoor, Sangers, Eekhof, et al. 2023; Lim et al. 2022; J. C. Ahn et al. 2021). Diagnostic tools also employ a broad array of algorithmic approaches (e.g. deep learning, convolutional neural nets, LLMs, etc.) which can impact their explainability. Additionally, these approaches may be supervised and unsupervised (or some combination of the two) introducing even more complexity to the use of AI-driven diagnostic tools. Recent studies have shown promising accuracy in AI diagnostics, with some algorithms for tumor detection and classification reaching performance levels on par with expert radiologists. Should these algorithms ever reliably surpass the diagnostic accuracy of today’s radiologists, questions about the ethical obligation of physicians to communicate to patients how big of a role AI has played in their diagnosis arise by the proposed RNandE (Lang, Nyholm, and Blumenthal-Barby 2023).

Prognostic Tools

AI models have also been introduced in several medical fields as a prognostic tool for predicting clinical outcomes, treatment response and disease progression, and mortality/survival estimates (Elyoseph, Levkovich, and Shinan-Altman 2024; Torrente et al. 2022; Lancellotti et al. 2021). Like diagnostic AI tools, prognostic tools also tend to utilize either machine learning or deep learning based models, although they focus more specifically on preventive measures, such as scanning mammogram images for precancerous formations (Arasu et al. 2023), or identifying biomarkers that may indicate a more complicated recovery journey for COVID-19 infection (Zhou et al. 2020). One ongoing project, for example, involves examining a data mining and machine learning-based model utilizing Bayesian Networks for mortality and risk prediction in patients with advanced heart failure who are considering left ventricular assist device (LVAD) implantation. The goal of such a model is to provide patients with survival/mortality estimates both with and without the LVAD intervention along with an understanding of potential implantation risks in order to make an informed, value-congruent decision about whether or not to receive the implantation (Kostick-Quenet et al. 2024; Kostick-Quenet et al. 2022; Mehra et al. 2022). Patients receiving this kind of information to inform their clinical decision-making under the RNandE would need to be made aware of the model’s role in determining survival estimates and risk prediction, but the question remains to what extent patients understand how such models produce their estimates and how much information patients need or desire about the model and its use in their high-stakes care.

What is the Function of Notice and Explanation?

These examples raise the broader question regarding what the goal, or ethical rationale, behind [the right to] notice and explanation is. Here are three potential contenders, each with varying implications for practice:

1. To notify patients about their care

2. To educate patients and promote trust

3. To meet standards for informed consent

For each of these potential goals, we examine: 1) a theoretical rationale articulating its normative purpose and potential benefits to patients and 2) a brief exploration of the practical implications of the goal, including what achieving it may look like in practice as well as implications for not achieving the goal with respect to use of AI in patient health care.

POTENTIAL GOAL 1: TO NOTIFY PATIENTS ABOUT THEIR CARE

Theoretical Ethical Rationale

The goal of keeping patients notified about their care at a basic level is perhaps the simplest and most straightforward potential normative function of the right to notice and explanation—it can be viewed as the bare minimum. It can be thought of as a sort of “for your information.” This understanding of the purpose of notice and explanation consists of making it a requirement in clinical settings, meaning that clinicians need to ensure that they cover two main points with their patients, 1) that an AI-mediated tool is being used in their care and 2) possibly how (at a very basic level) the AI-mediated tool is being used (e.g., “to help your doctor read your mammogram or x-ray”). The broadest and least demanding of proposed goals, this option focuses on solidifying notice and explanation as a standard routine practice in clinical settings rather than on the thoroughness, content, or language of the explanations themselves. This approach acknowledges the importance of keeping patients in-the-loop about what is happening in their medical care, namely when AI-mediated tools are being utilized, and may also encourage patients to seek out further education or spark conversations about these tools should patients themselves desire to inquire more about them.

The rationale behind this goal is underpinned by not only the simple value of showing others basic respect in communication but it also has a strong foundation in standard medical practice which keeps patients in-the-loop about their care regardless of where in the clinical process they may be. We could, for example, draw similarities to clinicians letting patients know that a medical student will be observing their clinical interaction at the beginning of a yearly checkup, or to clinicians verbally walking patients through their physical exam as it happens; this helps patients to know what comes next, to prepare for certain sensations or any discomfort, and if patients desire, to initiate more in-depth conversations about their exam or care, including whether or not they want it to proceed. It’s important to highlight that if anything, these “FYI”s are more akin to receiving patient assent, if anything, and are surely insufficient for receiving patient informed consent. Should patients desire to end a physical exam prematurely or inquire more about a medical student’s expected role in their care, then patients have the ability to speak up, but the purpose of notification at its core does not require asking. In this way, notifying patients automatically “opts them in” but does not give them the explicit opportunity to “opt out” unless they bring it up themselves. Based on this definition, utilizing the right to notice and explanation for notification alone is perhaps the most straightforward or feasible version of the right to implement, but whether or not this level is sufficient to protect patients against potential harms is worthy of further discussion.

Practical Implications

Executing the right to notice and explanation with a goal of patient notification of basic aspects of their medical care can be compared to the standard process of notifying patients when medical interventions such as tests are being run or medical imaging is required to ascertain more information about a patient’s health. In these scenarios, like when clinicians inform patients that a metabolic panel will be run on a given blood sample or that a computed tomography (CT) or positron emission tomography (PET) scan is recommended, clinicians typically already provide 1) notice that a test or tool is going to be used and 2) how or why they feel the tool will contribute to ascertaining information necessary for the patient’s care. For blood sample testing, this may include identifying what the test is measuring, and for medical imaging, this may include a description of what the patient can expect during the imaging session, what the machine may look or sound like, and what exactly the clinicians are hoping to get a better picture of. Although clinicians likely have at least a working understanding of how these technologies function mechanically, biologically, or computationally, it is rare for them to cover this information with patients unless prompted. For example, it is not common practice to explain to patients how magnetic fields work in functional magnetic resonance imaging (fMRI), but clinicians are presumably able to describe that fMRI detects changes in blood oxygenation and flow in response to brain activity. Thus, if the goal of the right to notice and explanation is to merely inform patients about basic elements of their care, then the explanations provided can be fairly straightforward given that they provide 1) notice that an AI-mediated tool may be used and 2) how the AI tool contributes to outcomes that are relevant to the patient.

For the use case of AI diagnostic tools that we presented earlier, this might look something like: “An AI algorithm trained to identify malignant tumors will be used to analyze your radiology images; in conjunction with my own clinical expertise, we can put together a treatment plan.” Here is some sample language for the use case of mental health chatbots (mediated by clinicians): “As you chat with the bot, an AI algorithm analyzes your speech patterns. If it picks up on certain harmful patterns such as negative self-talk, we can make sure to target those issues during therapy.” These explanations require fairly low patient engagement and put the weight of the responsibility on clinicians to notify their patients of possible uses of AI tools. For patients requesting additional information from their clinicians, it may be sufficient—for the purposes of this goal—to create and provide them with additional resources (perhaps in the form of a physical handout or pamphlet or on a website dedicated to helping patients “learn more” about how a particular AI tool works and how its outputs may contribute to the patient’s care). Finally, it is important to note that ensuring patient understanding of how these AI tools work is different (and a higher standard that we will discuss further) than making sure that patients are notified of their use and function. Thus, for this goal, only confirmation of notice and explanation of an AI-mediated tool would be required for authorization of its use in a patient’s clinical care.

POTENTIAL GOAL 2: TO EDUCATE PATIENTS AND PROMOTE TRUST

Theoretical Ethical Rationale

A second possible goal of the right to notice and explanation regarding the use of AI in patient care is to educate patients and promote patient trust. This rationale can be viewed as a sort of middle ground option that is more involved than mere notification and information sharing with patients and viewing disclosure as a necessary element of informed consent (discussed below). Although keeping patients in-the-loop about the tools used in their care is a step toward informedness, the goal of education and trust places a stronger emphasis on patient understanding and the quality and extent of explanation being shared with patients in order to promote knowledge-sharing about how these AI tools work and employ a patient’s data to inform clinical recommendations and patient outcomes. In alignment with this goal, some have posited that user understanding of AI is a crucial and necessary prerequisite for trusting AI, as it is the only way that users can truly evaluate the utility of outputs for themselves (Longoni, Cadario, and Morewedge 2021; Markus, Kors, and Rijnbeek 2021), and empirical data has shown this positive correlation between increased understanding and increased trust (Branley-Bell, Whitworth, and Coventry 2020). Moreover, a significant portion of research in the “explainable AI” literature has been devoted to examining the impact of explainability on public trust of AI, pointing to evidence that the content and quality of explanations (such as the inclusion of algorithm visualizations and level of transparency into processes in the AI system) can contribute to proper trust calibration and higher levels of satisfaction (Kim et al. 2024; Starke et al. 2022; Zerilli, Bhatt, and Weller 2022; Starke 2021; Branley-Bell, Whitworth, and Coventry 2020). We might attempt, then, to use the right to notice and explanation to build AI literacy and facilitate greater trust in AI technologies by providing more detailed explanations and leaving space for patients to ask questions and have an open dialogue with clinicians about the AI tools utilized in their care.

Practical Implications

Unlike the previously discussed goal, aiming for trust and patient understanding and education about the use of AI tools in their care will likely require more patient engagement and exchange between patients and clinicians. An in-depth educational discussion with patients may be best suited for this goal, allowing clinicians to share informational materials to offer transparency—to the extent possible—into the processes of the AI system and a comprehensive discussion of how the particular AI tool contributes to patient outcomes and to inquire with patients about their remaining informational or clarification needs. The extent of information provided in these explanations likely need to exceed what was sufficient for the previous goal, potentially by describing more about the algorithm’s mechanisms, training data, and performance (especially for patients like them), and for algorithms capable of transparency, perhaps insight into how their conclusions are reached. It may even be beneficial to include visualizations in these materials, images that will help patients to understand the decision-making progress that occurs in different types of algorithms. Which features matter most to patients in understanding the use of AI in their care are likely to vary by patient, however, and while transparency about mechanisms may be necessary for some patients, others may care more about factors that have more personal significance, such as how well an AI system performs for patients like them. Further exploration of the informational preferences of patients will be crucial for the development of explanations that adequately satisfy standards for trust.³

Executing this goal in practice then introduces a big challenge: generating explanations that are technically valid, context- and algorithm-specific, yet in plain language clear enough for a patient to understand and evaluate its trustworthiness. Some researchers have cast doubt on the ability to achieve this at all, uncertain that users of AI in healthcare or those affected by AI (healthcare workers and presumably patients, respectively) will be capable of judging the quality of an AI’s decision by receiving an explanation to it (Ghassemi, Oakden-Rayner, and Beam 2021). Others have noted similar concerns about the ability to provide clear and meaningful explanations of these tools to patients (Zerilli, Bhatt, and Weller 2022; Babic et al. 2021; Zawati and Lang 2020); Reddy (2022) adds that if clinicians “cannot reasonably explain the decision-making process [found in an AI system], the patient’s trust in them will erode,” an issue that may be particularly salient with AI models that already struggle with transparency—such as black box algorithms—and that offer no insight into how a model has weighed its variables or arrived at its outputs (Reddy 2022, e214). Similarly, patients who fail to understand the explanation given to them (whether they have difficulty with the technical aspects, are left with remaining questions or feelings of dissatisfaction with the education session), may distrust or experience frustration with the use of AI-mediated tools in their clinical care. More than just eroding trust, some patients who feel that they receive inadequate explanations for how an AI tool works may grow uncomfortable with the use of AI tools in their care altogether, having somewhat of an opposite effect to the intended goal of increasing AI familiarity and enhancing trust (Starke and Ienca 2022; Dietvorst, Simmons, and Massey 2015). One complex consideration for notification and trust involves notification of patients utilizing AI-based chatbots. As we noted previously, notification could incite distrust and patient aversion to receiving AI-generated care, but so might the opposite; a lack of notification that a patient is communicating with an AI-based chatbot may feel deceptive and similarly contribute to the erosion of patient trust in the intervention. With applications such as this, implementing the RNandE in a way that effectively facilitates trust (when appropriate) may be a complex balance. Regardless of feasibility, patient understanding of the AI-mediated tools used in their care is neither necessary nor sufficient for facilitating trust, as some users may already be inherently distrusting of machines or AI (Schepman & Rodway, 2023; Castelo and Ward 2021), while others place their trust in developers to create trustworthy models with accurate and reliable outputs (Aroyo et al. 2021). Patient trust in an AI system may also differ due to its particular application and context, regardless of the quality of the explanation given (Scharowski et al. 2023). It may be easier, for example, for patients to trust a tool that merely confirms their clinician’s opinion about a diagnosis, such as the radiology example we discussed earlier, than an AI-driven prognostic calculator that produces survival estimates, which feels more “personal” to patients. On the opposite side of this spectrum of trust, there is also evidence pointing to explanations increasing people’s confidence in an algorithmic decision to an unreasonable extent, exacerbating automation bias and over-trust of AI systems (Eiband et al. 2019; Lyell and Coiera 2017). Moreover, patients self-reported understanding and trust in an AI system’s outputs based on a given explanation may be biased by other factors, such as the nature of the output itself (Branley-Bell, Whitworth, and Coventry 2020). With these complexities in mind, further discussion is needed to discern whether notice and explanation to promote trust is truly a goal that we should be aiming for, given that trust needs to be properly calibrated based on the capacities and particular context of an algorithm’s use and given that under- or over- trust may impact patient health and safety outcomes (Kostick-Quenet et al. 2023). Further discussion with stakeholders about what level of granularity these explanations can achieve, in what plain language terms they can best be communicated to patients, and what patient informational needs are considered conditions for understanding, transparency, and trust would be crucial to implementation of the right to notice and explanation if education and properly calibrated trust is the goal.

POTENTIAL GOAL 3: TO MEET STANDARDS FOR INFORMED CONSENT

Theoretical Ethical Rationale

The third potential rationale for the right to notice and explanation with respect to AI in healthcare is the view that notice and explanation of how AI is used is required to meet the standards for informed consent. While it is not common practice to explain to patients how minor tests work biologically or mechanically, communicate every potential risk, or have patients prove their understanding of these minor interventions by having them articulate it to clinicians in their own words, in situations involving more preference-sensitive, risky, or invasive interventions such as major surgeries, procedures, treatment plans, or medications, physicians are expected to ascertain informed consent from patients. According to the American Medical Association (AMA), this includes “presenting relevant information accurately, in keeping with the patient’s preferences for receiving medical information, including… the nature and purpose of recommended interventions” and what will be crucial to our discussion shortly, “the burdens, risks, and expected benefits of all options, including forgoing treatment” (Informed Consent, n.d.). Additionally, the AMA recommends “assessing the patient’s ability to understand relevant medical information and the implications of treatment alternatives,” (Leo 1999). Using standards of informed consent as the goal for the right to notice and explanation in practice would mean that for any AI-mediated intervention to be implemented in a patient’s care, that patient would need to provide informed consent—prove understanding of the nature and purpose of the AI tool as well as the burdens, risks, and expected benefits of using it versus not using it—in order for use of the AI tool to be authorized.

Scholarship is well under way on the question of whether the standard of informed consent is applicable, and if so, in what ways, for the use of AI in clinical care. There are several distinct reasons posited for why a patient might be owed information pertaining to the use of AI in their care. Cohen, for instance, has considered how patients may assume exclusivity in their relationships with their care team (Cohen 2020). Insofar as the introduction of an AI decision-maker represents an undisclosed third party, the patient may be entitled to know and consent to the extent of the AI’s involvement in their care. As Cohen notes, some scenarios may have clearer or weightier expectations regarding exclusivity. For example, a patient may more readily assume that only humans will be performing the surgery they are to undergo than assume that the diagnosis or prognosis is entirely human-derived. Regardless, it is interesting to consider parallels to this “undisclosed third party” that already exist in the clinical setting. Take for example, the practice of seeking a second opinion, in which clinicians may call upon colleagues to weigh in about a patient’s diagnosis, prognosis, treatment plan, etc. Though this practice is quite common, patients rarely, if ever, are asked for explicit consent for their physician to seek alternative opinions. This is likely because there are certain assumptions and expectations of the process of going to the doctor: that an examination may be performed, that tests may be run, that your doctor will consult all necessary information (test results, data, other specialties or colleagues). Whether or not this kind of implicature should extend to clinicians seeking out the “opinion” of AI in the same way it extends to a physician colleague is up for debate, and likely dependent on how we define the decision-making capacity of AI systems and the weight attributed to them. Working backwards, Richardson et al., argue that if most patients would refuse AI involvement in their care if given the chance (and notice), then this is sufficient reason for why AI involvement falls under the purview of informed consent (Richardson et al. 2021). If a patient is not provided notice of AI involvement and the opportunity to opt out, then it appears like a violation of their autonomy to decide the nature of their care. Interestingly, both arguments rely on contingent empirical facts about peoples’ attitudes and beliefs. Assuming, for the sake of argument, that Cohen and Richardson et al., are correct in these facts does not commit us to thinking these attitudes or beliefs are immutable. It could just as easily come to be the case that, in an increasingly digitized and automated society, patients begin to assume the opposite—that the use of AI in their care is the default or standard of care. Likewise, whether most patients would want to exercise a right to refusal of AI-integrated care seems to depend entirely on how the general public feels and thinks about such technology. At the very least, adopting Cohen or Richardson et al.’s position would demand an ongoing sensitivity to shifts in public opinion or default assumptions held by patients, as well as developments in the standard of care, in order to determine which AI interventions incur a right to notice and demand consent for their use. This is not to discount the significance of these debates about informed consent for AI in the present, only to point out that contemporary discourse has not isolated any principled reasons why AI ought to be included in informed consent.

Binkley and Pilkington give reasons to doubt the informed consent approach to AI in healthcare when they note that the consent aspect may not be as feasible or appropriate as assumed in many cases of AI-integrated care. If AI-human collaborations come to yield greater quality care and are accepted as the standard of care, allowing patients to opt out of AI-integrated care may amount to simply downgrading the quality of their care (Binkley and Pilkington 2023). This is especially perilous if, as Binkley, Pilkington and others have suggested, some deskilling on the part of human clinicians may result from highly advanced AI automating certain tasks. Medical institutions may fear the liability of allowing their clinicians to provide care unaided by AI and require clinicians to utilize and rely on AI technology. Binkley and Pilkington conclude that the consent for AI technology may then be, in effect, illusory. Moreover, as noted elsewhere in this paper, even if one were to focus just on informedness sans consent, the opacity of many AI tools designed for use in clinical settings represents an obstacle. If an AI is a black box, then the patient can be told that an AI is involved in their care, in what ways it is involved (whether it is generating diagnostics, prognostics, or treatment recommendations, etc.), but appreciating why the AI outputs what it does will remain impossible.

Practical Implications

In addition to dealing with the challenge introduced in the discussion of other potential goals of the right to notice and explanation (generating technically valid, clear, and plain language explanations for how an AI system contributes to a patient’s outcomes) the idea that patients have to give informed consent to the use of AI in their care faces another critical obstacle: making certain that patients reach a meaningful understanding of AI tools sufficient for supplying valid informed consent. We can start by examining what information would need to be disclosed and explained to patients, if we thought that they needed to consent to the use of AI in their care. As we noted above, the AMA suggests that disclosure includes information regarding “the nature and purpose of recommended interventions.”⁴ While it may be reasonable to suggest that conforming to a standard as high as informed consent requires that patients should have a genuine understanding of AI algorithms, to say that every individual needs to understand the underpinnings and mechanisms of an AI system and its particular decisions related to their care in order to provide informed consent is likely too high of a bar to set.⁵ Indeed, these are not expectations we hold for patients utilizing other complex medical devices or even pharmaceuticals in their care, pointing to the fact that this may not be the most novel problem, but the newest iteration of complex technologies to deal with.⁶ The amount and kind of information needed to include in these explanations could very well be information overload for patients who struggle with AI, technology, or health literacy and who are generally, largely unfamiliar with these tools. Some clinicians may not be any better suited for understanding them either, and clinicians who cannot decipher how an AI tool came to its results may not be able to appropriately communicate and disclose information with patients. Reddy (2022) warns that this could affect “the patient’s autonomy and ability to engage in informed consent,” (e214), and again, is not something generally asked of physicians when recommending pharmaceuticals or other complex products. Between difficulties in interpretability and transparency for certain algorithms, the extent of patient heterogeneity and the compounding issues of AI and health literacy, and additional variation from clinicians’ communication styles and individual understanding of AI systems, there are several factors working simultaneously that may impede patients’ ability to understand these AI systems. It may be that for certain patients, the level of understanding and ability to manipulate information about these tools back to a clinician needed for ensuring true consent is not feasible to meet at all.

A slightly more achievable standard for what a meaningful understanding of these AI systems could look like as part of informed consent might be to educate patients on the well-documented potential risks and limitations associated with the use of AI-based outputs in clinical care and decision-making, and seek consent for their use. In fact, Andreotta, Kirkham, and Rizzi (2022) suggest something quite similar for the future of AI and informed consent, emphasizing the need to use “potential harm[s] that [an AI-based] decision could cause” in order to decide how much transparency is appropriate or sufficient for an explanation to patients. They provide a useful example, emphasizing that patients can consent to heart surgery without understanding the ins and outs of the operation, by instead having a full grasp of the consequences of not going through with the surgery on their health, as well as the potential risks and benefits associated with getting the surgery (Andreotta, Kirkham, and Rizzi 2022, 1719). Though this is a useful starting point for AI tools—making sure that patients are aware of potential harms such as poor algorithm training, incomplete or biased data sets, inaccuracies, and misleading or inappropriately used outputs—several questions still remain. How large of a scope of potential risks raised by AI tools should clinicians be required to inform patients about? How much technical knowledge is required of patients for them to recognize the magnitude of harm posed by each of these risks? How do we regularly account for or keep up with unanticipated consequences and risks of new forms of AI?

Regarding different forms and applications of AI in patient care, we have also, up to this point, only been discussing how AI tools contribute to outcomes that impact patients quite broadly. To get a true picture of what informed consent for the use of these tools might look like in practice, one final consideration worthy of acknowledgement is that different AI use cases and applications might also require different standards for consent. Refer back to two of our sample use cases to illustrate this point: 1) AI diagnostic tools such as the use of an AI algorithm to analyze radiology images and to confirm a clinical diagnosis vs. 2) AI prognostic tools such as the use of an AI-based calculator to generate survival estimates based on a patient’s decision to receive a specific intervention. In these two use cases, there is a salient difference in what each patient would need to be informed of and what they are consenting to. Consent in the first case is fairly straightforward; the patient needs to know that an AI algorithm is going to be used to analyze their images so that they can consent to the algorithm’s use. In this scenario, it is likely that the patient’s clinician is utilizing the AI tool as a confirmatory measure for their own clinical opinion, somewhat lowering the amount of weight being given to the AI tool’s outputs. For the second patient, however, an AI-driven tool is generating a new piece of information that will become crucial to the patient’s life-changing medical decision-making process. Because of this, it makes sense for a patient in this scenario to not only receive notification that an AI-driven tool is being used to generate such a key piece of information relevant to decision-making, but to also receive information about how the AI tool works so that they can make an informed and voluntary medical decision based on the tool’s output. Patients making medical decisions based on outputs that they do not truly understand may unintentionally be consenting to an assessment that they may otherwise disagree with had they known how it was derived. This demonstrates the point that it is important to identify applications of AI tools in healthcare that require patients to consider AI-generated information in decisions that may significantly impact their future health to determine whether standards for informed consent should reflect these differences.

It is unclear whether we will ever be able to ensure with confidence that patients are able to reach understanding needed for traditional consent regarding AI-mediated healthcare tools as long as the standard requires a technical understanding of AI algorithms. If we choose to view the right to notice and explanation as a key component of informed consent, then patients who cannot prove comprehension and understanding of how the AI-mediated tool contributes to their outcomes and who cannot properly weigh the risks of using such a tool in their decision-making cannot properly authorize the use of the AI-mediated intervention in their care. While this restriction would be in place to protect patients from potentially uninformed decision-making or from the negative impacts of making a clinical care decision influenced by automation bias, it may be too high of a standard to realistically advance given the complexity and ever-developing nature of AI tools and their use in healthcare. If we choose to implement the right to notice and explanation as a key component of informed consent, then, further discussion will be required to determine how patients can responsibly consent to the use of AI tools in their care with a reasonable amount of information and level of understanding of how the tools function.

RECOMMENDATIONS MOVING FORWARD

While the AI Bill of Rights Blueprint presents a right to notice and explanation that sounds good in theory, the devil is in the details. What exactly does the implementation of this right look like in patient care and what is its main function–to provide a basic “FYI” and open the door for questions, to promote education and build trust with patients, or is it part of a patient’s right to informed consent? Choosing the most effective and beneficial format for the right to notice and explanation depends heavily on the function that we want it to serve in the first place, which is yet to be firmly established in the AIBoR. We have proposed some possible functions of the right to notice and explanation as well as what such a right could feasibly look like in practice depending on its function. It will remain crucial for policy makers and practitioners to decide to what end and thoroughness we need to be explaining the use of AI in clinical care to patients before we can move on to more detailed questions of formatting, content, and further implementation of the right.

Deciding how in-depth explanations for patients must be will also depend significantly on contextual considerations, such as the specific use cases and applications of AI tools and actual patient desire for a certain standard of medical information and explanation. Research is underway to examine patient preferences and informational needs for AI in healthcare, with existing studies noting desires for transparency in algorithm credibility, information about how recommendations are derived, and insights into algorithm reliability, generalizability, and accuracy (Park 2024; Chew and Achananuparp 2022). Many of these studies focus more broadly on topics relating to patient trust and acceptance of these tools, as well as the impact they may have on the physician-patient relationship (Baldauf, Froeehlich, and Endl, 2020). More research is needed, however, to better understand patient preferences for aspects of implementation affected by the RNandE notice versus consent and the quality of explanation, how to communicate such complex concepts to patients in a meaningful way, and how informational needs change based on the particular AI application. Perhaps the best way forward is to employ each of these goals in the most appropriate contexts, tailoring requirements and conditions of the implementation of the right to notice and explanation to the specific application or use case. It may be that notice, or what Amann et al. (2020) calls “first level explanation,” (our Potential Goal 1) is actually sufficient for some forms of AI tools used in clinical care, while informed consent may be required and the best way to safe-guard patients with other AI tool applications, like patients who are making a high-risk medical decision about their health or survival (Amann et al. 2020). This proposal even shares similarities to the European Union’s current regulatory approach of risk-based stratification for AI (AI Act).⁷

Finally, we must remember that whichever goal we recommend or decide on in theory needs to be feasible for implementation in the clinical setting. It is easy to claim that every individual has a right to notice and explanation of AI tools in their health care that is technically valid, up-to-date, and in plain language, but it is an entirely different challenge to actually create such explanations that will reach a broad patient population and to execute or enforce them consistently in a way that maintains quality and is practically feasible from clinician to clinician and across hospitals and practices. This will only be made more difficult by consistently new advancements in AI in healthcare and the task of keeping patients up-to-date on new applications of AI in a way that benefits them and keeps them responsibly informed about their care. Keeping these feasibility concerns in mind as we continue to explore stakeholder perspectives on the best goal for the right to notice and explanation will be crucial for moving forward in a way that will successfully serve the AI Bill of Rights’ larger purpose of protecting the rights of the public.