An Ethically Supported Framework for Determining Patient Notification and Informed Consent Practices When Using Artificial Intelligence in Health Care

Abstract

Artificial intelligence (AI) is increasingly being used in health care. Without an ethically supportable, standard approach to knowing when patients should be informed about AI, hospital systems and clinicians run the risk of fostering mistrust among their patients and the public. Therefore, hospital leaders need guidance on when to tell patients about the use of AI in their care. In this article, we provide such guidance. To determine which AI technologies fall into each of the identified categories (no notification or no informed consent [IC], notification only, and formal IC), we propose that AI use-cases should be evaluated using the following criteria: (1) AI model autonomy, (2) departure from standards of practice, (3) whether the AI model is patient facing, (4) clinical risk introduced by the model, and (5) administrative burdens. We take each of these in turn, using a case example of AI in health care to illustrate our proposed framework. As AI becomes more commonplace in health care, our proposal may serve as a starting point for creating consensus on standards for notification and IC for the use of AI in patient care.

Artificial intelligence (AI) is increasingly being used in health care for a variety of purposes, yet, to our knowledge, no standard way to inform patients about the use of AI in their care exists. Without an ethically supportable, standard approach to knowing when patients should be informed about AI, hospital systems and clinicians run the risk of fostering mistrust with their patients and the public. Therefore, hospital leaders need guidance on how much, or how little, to tell patients about the use of AI in their care.

Informed consent (IC) has become essential for ethical medical practice in day-to-day patient care.¹ IC requires that physicians deal honestly and transparently when providing care. In practice, obtaining IC with patients requires that clinicians convey complex medical information in a way that respects the patient’s capacity to distill information and choose freely according to their own conception of self and values as part of shared decision-making.² Without the opportunity to understand the relevant elements for a proposed treatment or test, as well as that intervention’s risks and benefits, a patient cannot choose freely. Therefore, demonstrating respect for a patient requires that we offer the opportunity for that patient to choose—or decline—treatment based on pertinent information that can inform that decision.

To determine which AI technologies fall into each of the identified categories (no notification or no IC, notification only, and formal IC), we propose that AI use-cases can be evaluated using the following criteria: (1) level of AI model autonomy, (2) level of departure from standards of practice, (3) whether the AI model is patient facing, and (4) the degree of clinical risk. We also suggest that assessing the level of administrative burden of notification and IC is a critical factor to consider. We take each of these in turn, using a case example to illustrate our proposed framework.

Case Example

Ms Prya Anderson (this patient and all those mentioned are fictional; any similarity to existing people or clinical scenarios is coincidental) is a 70-year-old woman who was admitted to the hospital for complications related to a knee replacement. She sought treatment at the ED with fever, shortness of breath, as well as pain, redness, and swelling around the postoperative knee. Ms Anderson’s treatment was supported in several ways by tools that use various forms of AI. First, after the ED team decided that she needed to be admitted, an AI model called AdmitAI helped the administrative team to determine what hospital bed to assign to Ms Anderson, based on hundreds of factors, including her medical needs and wait time. Ms Anderson was not told about the use of the AdmitAI because her clinical team did not know that AI was being used in this way. Next, in accordance with standard processes in place at this hospital, an AI model (ImageAI) was used to assist the radiology team in the reviews of various radiology test results of Ms Anderson’s knee. When she received the reports in the patient portal, a notification at the end of the report stated that AI had been used in conjunction with human radiologists. Unfortunately, Ms Anderson’s condition quickly deteriorated, and she was transferred to the ICU, where an AI model was used for all patients to determine the risk of sepsis developing (SepsisAI). The SepsisAI model deemed her to be at low risk of sepsis, and she was never told about this model in her care. Ms Anderson’s condition did worsen, however, initiating use of another AI tool being used for all patients in the ICU to help clinicians to predict patients who might be eligible for a do not resuscitate (DNR) code status discussion (DNRAI), primarily based on the prognosis. DNRAI changed the default code status from full code to unilateral DNR. At this point, Ms Anderson was told about the use of the DNRAI, given that the results of this model were the reason the clinicians wanted to discuss goals of care and code status with her, but only after the default had been changed. Becoming concerned by this discussion, Ms Anderson requested her daughter’s assistance in investigating her prognosis on a popular internet website. The website stated that it used a patient education chatbot (EdAI) and was powered by AI, not a human. The website also provided a detailed description of the tool used, its use of data, and other key factors related to the limitations of this AI system. With her daughter’s help, Ms Anderson created an account, acknowledged that she understood the risks and benefits, and proceeded to use EdAI to help her investigate her situation.

Criterion 1: Level of AI Model Autonomy

Beginning with our first criterion, we consider the ethical implications of varying AI autonomy levels. The role that AI plays in medical diagnosis and treatment choices will create different transparency obligations. Bitterman et al³ provide insight into such distinctions, differentiating between assistive AI algorithms and autonomous AI algorithms. With assistive AI, the AI model is merely analyzing and presenting data. In such cases, so long as the decision remains entirely within the expertise of the human clinician and the clinician or hospital employee understands the evidence and method used within the model, then this tool is not meaningfully different than standard prediction modeling tools (such as complex calculators, for example) that provide such information and currently do not require notification or IC. AdmitAI falls into this category.

As the role of AI shifts away from humans, however, ethical concerns central to IC arise, particularly regarding trust, transparency, and disclosure. A patient’s freedom to choose a treatment is accepted widely as a fundamental element of ethical medical practice. To support such freedom, a patient must be given adequate information to make informed choices. Included in such information is knowledge of who or what is making medical decisions in their care. Without disclosure of such information, patients cannot trust that the hospital system has fulfilled their duty to be transparent in their actions.

If the AI model is a more autonomous AI algorithm, such as clinical decision-support tools that offer recommendations and go beyond mere data presentation, then notification should be used, with some case-based exceptions requiring IC. Examples of this category are SepsisAI and ImageAI. This is because although a clinician is responsible for any final clinical decisions and plan of action, the AI model is using its own algorithmic judgement, which is independent of the treating clinician’s process for weighing and discriminating relevant information. Just as patients would be informed if medical trainees were involved in providing their care under supervision, patients similarly should be informed when AI is offering input for supervising physicians to use in clinical decision-making.

Even more significant are purely autonomous AI decisions that are fully independent of and not reliant on clinician judgment, such as with DNRAI. Although purely autonomous AI models currently are not common in health care and are controversial, they may become more widely used in the future, despite concerns about liability.⁴ Autonomous AI models are those in which decisions may not always be approved or reviewed by humans.³ In fact, this may be the point of AI models: to make some decisions automated without requiring time-consuming, expensive human interactions.⁵

As a result, the ethical requirements for patient involvement in accepting the use of such AI tools increase because the ultimate decision or recommendation may not be made by the human clinician. Given that patients would not reasonably expect that autonomous decision-making AI would be involved in their medical care, involving autonomous AI in such decisions should always require IC; to preserve transparency and the patient’s ability to make free, autonomous, and informed decisions about their own health care and health care providers, patients must be informed of who (or what) is guiding patient care. Importantly, one implication of our proposal is that patients should be able to decline models, such as DNRAI, when IC is required.

Criterion 2: Departure from Standards of Care

Although most current AI models are aimed at merely ensuring that the existing standards of care are deployed consistently (akin to quality improvement models), we can imagine a near future when AI will suggest actions or treatments that meaningfully deviate from current standards of care, with the goal of improving outcomes. This may lead to AI models that—if well supported with clinical evidence—could result in the adoption of new standards of care. If the defaults of the decision support for AI systems are altered in ways that deviate from patients’ and clinicians’ currently established reasonable expectations, the IC process should be used. The rationale for this lies with a similar justification outlined above: if patients’ reasonable expectations for health care are deviating meaningfully from established defaults, then they should be informed.

For example, the DNRAI used in Ms Anderson’s care is an instance of such a model that alters the standards of medical care as a whole; the current standard defaults in the United States are to remain a full code until a DNR order is deemed medically appropriate and is discussed with the patient and family, and unilateral DNR orders are relatively rare in the United States.⁶ If the AI model is shifting the norm to more commonly enter DNR orders based on the criteria used in the AI models, then such a shift in practice requires IC before the model is used in a patient’s clinical care. In the case of Ms Anderson, this notification was offered only after the model was used; we believe that IC should have been used before the use of this model in her care.

Criterion 3: Patient-Facing AI

Chat features on websites of all kinds are common and familiar to most users. However, the distinction between human-staffed virtual support through a chat function on a website, on the one hand, and an AI-powered chat bot like the one Ms Anderson and her daughter used to research her prognosis, on the other, is not always easy for users to identify. This can be particularly confusing for users when chat bots are imbued with human characteristics—such as human names like Justin or Cheri—and are programmed to use idiomatic and informal language styles. Such a chat bot might offer a series of questions to the patient, using a decision tree to identify patient concerns and to offer information taken from the website’s pages. Although the chat bot might be intended only to offer a streamlined route for the patient to receive pertinent information, if that patient believes they are communicating with a live nurse, for example, they may place a different kind of value on and trust in the responses they receive. They may believe that a trained health care professional has asked the appropriate questions and is relaying advice that is personalized to their needs. Not knowing that they have spoken with a chat bot—rather than a human health care professional—could lead a patient to refrain from seeking follow-up care for a condition or situation that will require medical intervention to avoid a negative outcome or unnecessary harm.

Hospitals and health care providers must avoid the introduction of unnecessary patient deception and potential harm when using chat bots to support clinical practice. Therefore, when using a chat bot or other patient-facing tools, it will be obligatory at minimum to include notification alerting users when they are not engaging with a medical professional. Further, it may be prudent to reduce confusion by refraining from humanizing available chat bots. Unless an IC process is completed with a health care provider or company before a patient receives access to a website’s chat bot, it may not be possible to facilitate IC before chat bot use. Because obtaining IC through a website may not be achievable (even if desirable), careful consideration should be given to the use of chat bots in patient-facing websites and tools. In cases warranting notification, consider using text that can provide a so-called off-ramp from the chat bot, directing that patient to a medical professional. In the case of Ms Anderson, the hospital that created EdAI used IC.

Criterion 4: Clinical Risk

Health care almost always involves some risk of harm, and these risks can be broken down broadly into low, medium, and high risk.⁷ The introduction of AI often seeks to reduce these risks and to make health care safer.⁸ However, even when AI models are tested fully and implemented carefully, risks remain, as with any medical intervention or technology. Some of these risks remain unchanged as a result of the AI, whereas other risks may be introduced, increased, or reduced because of AI implementation. These risks, taken together, need to be analyzed when considering the degree of transparency needed with patients. For low-risk situations, notification may not be required. However, as the riskiness of the interventions being considered increases, notification, IC, or both may be necessary. Our rationale is similar to that provided in criterion 2: although patients’ general consent for medical care falls within their reasonable expectations for low-risk interventions, more risk deviates from these routine expectations. Therefore, patients should be notified, and as the risk level increases, IC is needed so that they can decide autonomously and prospectively whether this risk is something they are willing to accept. For example, if the implications of the decisions are critical for survival or may result in death, such as in the DNRAI model, this is deemed to be high risk, requiring IC, even if the AI is perfectly accurate in its predictions, introducing no additional risk to an already high-risk situation. In the case of ImageAI, if the risks of routine imaging are minimal to the patient and the introduction of AI does not alter the existing risks, then the current standards used for imaging (without AI) are sufficient. However, in the case of EdAI, although patient education usually is considered low risk, the introduction of AI might introduce medium risks, or even high risks, including misinformation or emotional harm.⁹

Consideration: Administrative Burdens

Finally, although not all administrative burden warrants ethical consideration, times may arise when such burdens are ethically significant. Administrative burdens can arise from many factors, including the practicality of obtaining IC or notifying patients, the complexity of information being provided, and patient-related factors such as health literacy and health care access. Hospital leaders need to be mindful not only of the burden of notification or IC for individual models, but also of the entire scope of models that are used in the care of individual patients. Notifying patients about all the AI tools that might be used in their care, such as with Ms Anderson’s admission, could be extremely time-consuming and confusing to patients and surrogate decision-makers, given that many models may be used at any given time for such admitted patients. In cases where no consent would usually be obtained—such as in emergency, lifesaving circumstances in which consent is presumed—consent for AI model use in treatment similarly would be presumed by necessity. When IC is needed based on the factors described above, however, then the formal IC process should be followed regardless of the administrative burden and should be added to the institution’s IC policy. Further, as AI models are disseminated in health care, patient transparency processes need to be considered carefully. Importantly, administrative burden of notification and IC should not absolve hospitals of their responsibility to follow the recommendations provided using the criteria discussed. It is important to note that the administrative burdens are not calculated formally into the scoring system proposed herein, but these burdens should be graded and analyzed qualitatively.

Putting It All Together

We propose that each of these five criteria should be considered, at minimum, when deciding whether either IC or notification are required. Therefore, we also have provided a method for decision-making that may help to combine the analysis for each of these five domains. In Table 1, we provide the points that each domain should receive based on the assessment scores. Then, in Figure 1, we illustrate how the scores can be used to guide decision on if notification or IC are needed. Importantly, we propose this only as a guide; exceptions may exist based on unique considerations that may diverge from our proposed framework.

Table 1.

Scoring System for Notification and IC Decisions for Use of AI Models in Health Care

Criteria Name	Level 1: No Notification or No IC Required	Level 2: Notification Required	Level 3: IC Required
Model autonomy	Model is focused on data presentation and clinician is making decisions.	Model is focused on clinical decision support, yet clinician makes final decisions.	Model incorporates some level of decision automation, and clinician does not make all final decisions.
	Score: 1	Score: 2	Score: 3
Departure from standards of care	No departure from standard of care is possible, regardless of AI output.	Information offered to patient or surrogate for use in decision-making will include AI output.	Deviations from standard of care are suggested by AI output and may differ from physician recommendation.
	Score: 1	Score: 2	Score: 3
Patient-facing AI	Purely text-based branching logic with no patient engagement features or personalized AI outputs.	Conversational or responsive text that selects relevant information to share based on user input, but does not humanize the AI.	Conversational, responsive text with AI humanized character assigned common proper human name and provides medical guidance.
	Score: 1	Score: 2	Score: 3
Clinical risk	Risks of intervention and the AI, combined, is low risk.	Risks of intervention and the AI, combined, is medium risk.	Risks of intervention and the AI, combined, is high risk.
	Score: 1	Score: 2	Score: 3
Administrative burdens	No burden because no action required by administration.	Minimal burden with notification routinized or IC readily incorporated into existing practices.	High burden: notification or IC will require new processes or education or are very complicated.
	Burden level: A	Burden level: B	Burden level: C

AI = artificial intelligence; IC = informed consent.

Figure 1.

Diagram showing the decision tree scoring system for notification and IC decisions of use of AI models in health care. AI = artificial intelligence; IC = informed consent.

Further, although we have offered a numeric system for scoring AI involvement in medical practice and the relative need for notification or inclusion in the IC process, this does not come without caveats. Just as with any clinical tool or technology, it is ultimately the treating clinician or administrator who is responsible for the care provided to the patient; using such tools does not diminish this responsibility, and having a system that can aid in determining appropriate IC standards when using AI does not remove a hospital’s or a physician’s responsibility to consider carefully the implications of the AI in use and the manner in which such AI is implemented. As with the application of ethical principles in practice, the application of the scoring system provided here must be used in conjunction with careful consideration and attention to the ethical concerns that motivated the tool’s development. Scoring systems are not a replacement for regular attention to ethical reasoning in medical practice.

Conclusions

The focus of this article is on providing broad guidance and on starting the process of the health care field developing finer-tuned, specific guidelines. Therefore, we expect our proposals to spark healthy controversy and debate. In particular, our proposal will not satisfy those who claim that notification or IC should be a standard part of all AI use in health care,¹⁰ given that we suggest that many models should not require patient awareness. However, we believe that our recommendations are well aligned with the current standards of IC and disclosure, are patient focused, and are practical. Furthermore, our recommendations align with many hospital systems’ efforts to create rigorous AI governance committees and can provide a basis for such committees to establish requirements for patient awareness of AI being used in medical care. Currently, little exists in the literature to guide such processes, with the exception of Iserson’s¹¹ article on the use of AI and consent in emergency care and Cohen’s⁴ article, which provides an excellent discussion of many key ethical and legal implications of IC and AI.

Remaining details not addressed herein include how notifications and IC of AI involvement in patient care should happen and what should be disclosed. Future research on these topics is needed and will require input from multidisciplinary teams. For notification and IC to be high quality, clinicians also will need better education on the models they use in their practice, and research should explore effective educational interventions for clinicians.¹² In addition to addressing clinician education and AI experience, research also is needed to integrate better patient perspectives on this evolving field into standards used in practice. Although studies have explored patient perspectives on the use of AI in health care,^{13, 14, 15, 16, 17, 18} most are exploratory and do not include sample sizes large enough to support generalized conclusions.

We do not address the many additional ethical concerns related to AI in health care in this article, including embedded bias, safety, accountability, data use, and others.^{19, 20, 21} Instead, we suppose that these additional ethical concerns can be addressed for AI used in health care and that the models in use have been proven sufficiently accurate and effective, that the data used to create the models was collected ethically, and that the models are aligned with an organizational governance processes and approvals before considerations of patient notification processes are considered. AI models that are being tested as part of research protocols fall outside our scope as well; AI used in research must follow different standards. Importantly, if the model is likely to fall under the US Food and Drug Administration’s jurisdiction as a medical device, the current standards for IC should be followed, noting that in certain US states, some of our examples (such as DNR orders) face varying degrees of legal permissibility.²²

Finally, if a patient asks about any aspect of their care, including AI involvement in clinician decision-making, they should be told the truth. In cases where a patient declines the use of AI or is notified and wants to opt out, further consideration of ethically appropriate responses should be undertaken. Although for many models it may be simple to avoid using the model in patients' care, for other models, such as those that are embedded into electronic medical records, a hospital system may find it difficult to remove one patient from AI tools embedded into system-wide applications. Even when patient information can be extracted or AI model suggestions can be dismissed, clinician attitudes and beliefs nonetheless may be influenced by information generated by AI models. On a related note, some patients may prefer to know, or not to know, about AI in their care. Although patient preferences are an important consideration, not all patient preferences can or should be met.

As AI becomes less novel and more integrated in routine medical care, and as patients and health care professionals become more familiar with such technologies, recommendations regarding patient awareness likely will change. At this time, however, stakeholders from industry, health care, patient advocacy groups, and professional associations must come together to create greater consensus on the standards for no notification, notification, and IC when AI is used in patient care, in addition to wider issues regarding the use of AI in clinical care.²³ We hope that what we have outlined here sparks fruitful discussion and creates consensus on standards for notification and IC for the use of AI in patient care.

Financial/Nonfinancial Disclosures

The authors have reported to CHEST the following: S. L. R. serves on Blue Cross Blue Shield Association’s Pharmacy and Medical Policy Committee and the Pharmacy and Therapeutics Committees as an ethicist. This role is not related to the content of this manuscript. None declared (D. S.).