Artificial Intelligence and Liability in Medicine: Balancing Safety and Innovation

Abstract

Policy Points.

• With increasing integration of artificial intelligence and machine learning in medicine, there are concerns that algorithm inaccuracy could lead to patient injury and medical liability.

• While prior work has focused on medical malpractice, the artificial intelligence ecosystem consists of multiple stakeholders beyond clinicians. Current liability frameworks are inadequate to encourage both safe clinical implementation and disruptive innovation of artificial intelligence.

• Several policy options could ensure a more balanced liability system, including altering the standard of care, insurance, indemnification, special/no‐fault adjudication systems, and regulation. Such liability frameworks could facilitate safe and expedient implementation of artificial intelligence and machine learning in clinical care.

After more than a decade of promise and hype, artificial intelligence (AI) and machine learning (ML) are finally making inroads into clinical practice. AI is defined as a “larger umbrella of computer intelligence; a program that can sense, reason, act, and adapt,” and ML is a “type of AI that uses algorithms whose performance improves as they are exposed to more data over time.”¹ From 2017 to 2019, the US Food and Drug Administration (FDA) approved or cleared over 40 devices based on AI/ML algorithms for clinical use.², ³, ⁴ Many of these devices improve detection of potential pathology from image‐based sources, such as radiographs, electrocardiograms, or biopsies.⁵, ⁶, ⁷, ⁸ Increasingly, the FDA has permitted marketing of AI/ML predictive algorithms for clinical use, some of which outperform physician assessment.⁵, ⁹ Despite their promise, AI/ML algorithms have come under scrutiny for inconsistent performance, particularly among minority communities.¹⁰, ¹¹, ¹², ¹³

Algorithm inaccuracy may lead to suboptimal clinical decision‐making and adverse patient outcomes. These errors raise concerns over liability for patient injury.¹⁴ Thus far, such concerns have focused on physician malpractice.¹⁴ However, physicians exist as part of an ecosystem that also includes health systems and AI/ML device manufacturers. Physician liability over use of AI/ML is inextricably linked to the liability of these other actors.¹⁵, ¹⁶ Furthermore, the allocation of liability determines not only whether and from whom patients obtain redress but also whether potentially useful algorithms will make their way into practice: Increasing liability for use or development of algorithms may disincentivize developers and health system leaders from introducing them into clinical practice. We examine the larger ecosystem of AI/ML liability and its role in ensuring both safe implementation and innovation in clinical care.

Overview of AI/ML Liability

Physicians, health systems, and algorithm designers are subject to different, overlapping theories of liability for AI/ML systems (see Table 1). Broadly, physicians and health systems may be liable under malpractice and other negligence theories but are unlikely to be subject to products liability. Conversely, algorithm designers may be subject to products liability but are not likely to be liable under negligence theories.

Table 1.

Current Landscape of AI/ML Liability

Type of Liability (Definition)	Implications for Physicians	Implications for Developers or Health Systems
Medical malpractice (Deviating from the standard of care set by the profession)	Physicians may be liable for failing to critically evaluate AI/ML recommendations. This may change as AI/ML systems integrate into clinical care and become the standard of care.	Health systems or practices that employ or credential physicians and other health care practitioners may be liable for practitioners’ errors (“vicarious liability”).
Other negligence (Deviating from the norms set by an industry and courts)	Physicians may be liable for (1) their decision to implement an improper AI/ML system in their practice, or (2) their employees’ negligent treatment decisions related to AI/ML systems (“vicarious liability”).	Hospital liability for negligent credentialing of physicians could extend to failure to properly assess a new AI/ML system. In general, health systems may be liable for failing to provide training, updates, support, maintenance, or equipment for an AI/ML algorithm.
Products liability (Designing a product that caused an injury)	Physicians might be involved in these cases if they work or consult for designers of AI/ML devices.	The law is unsettled in this area. As AI/ML software integrates into care or becomes more complex, algorithm developers may have to contend with liability.

Abbreviations: AI, artificial intelligence; ML, machine learning.

Negligence

Medical Malpractice

Medical malpractice requires an injury caused by a physician's deviation from the standard of care. This standard of care is set by the collective actions of the physician's professional peers, based on local or national standards.¹⁴, ¹⁷, ¹⁸

A physician who in good faith relies on an AI/ML system to provide recommendations may still face liability if the actions the physician takes fall below the standard of care and other elements of medical malpractice are met. Physicians have a duty to independently apply the standard of care for their field, regardless of an AI/ML algorithm output.¹⁹, ²⁰ While case law on physician use of AI/ML is not yet well developed, several lines of cases suggest that physicians bear the burden of errors resulting from AI/ML output. First, courts have allowed malpractice claims to proceed against health professionals in cases where there were mistakes in medical literature given to patients or when a practitioner relies on an intake form that does not ask for a complete history.²¹, ²², ²³, ²⁴ Second, courts have generally been reluctant to allow physicians to offset their liability when a pharmaceutical company does not adequately warn of a therapy's adverse effect.²⁵, ²⁶, ²⁷ Third, most courts have been unwilling to use clinical guidelines as definitive proof of the standard of care and require a more individualized determination in a particular case.¹⁹, ²⁸ Fourth, many courts are disinclined to excuse malpractice based on errors by system technicians or manufacturers.²⁹, ³⁰, ³¹

Other Negligence

Outside of the direct physician/practitioner‐patient interaction, health systems, hospitals, and practices are also responsible for the well‐being of patients.¹⁷, ³² Negligent credentialing theories may hold a health system or physician group liable for not properly vetting a physician who deviates from the standard of care.¹⁸ Some legal experts have suggested that a similar liability may extend to health systems that insufficiently vet an AI/ML system prior to clinical implementation.¹⁸ Indeed, health systems already have a well‐defined duty to provide safe equipment and facilities and to train their employees to use health system‐provided equipment.³³, ³⁴, ³⁵ Thus, evidence of poor implementation, training, or vetting of an AI/ML system could form a basis of a claim against a health system.

To illustrate how this negligence would apply to a case involving AI/ML, take an example of an algorithm trained on a predominantly young, healthy population that uses vital signs and laboratory readings to recommend treatment for hypertension. Current guidelines suggest that older adults may benefit from tailored management of hypertension with higher blood pressure thresholds than those used to indicate interventions for younger adults.³⁶, ³⁷, ³⁸, ³⁹, ⁴⁰ A health system serving a predominantly older population could be liable for injuries resulting from this off‐the‐shelf AI/ML system since it will likely provide recommendations that do not conform to the current standard of care for that age group.

Vicarious Liability

Health systems, physician groups, and physician‐employers can be liable for the actions of their employees or affiliates.¹⁷, ³² Vicarious liability differs from other negligence in that one is liable for someone else's actions. For example, whereas negligent credentialing means that the hospital itself has been negligent, vicarious liability means the physician is negligent but the hospital is held responsible. One reason for vicarious liability is to distribute costs for injuries among hospitals and groups to compensate a victim.⁴¹, ⁴²

Vicarious liability is easiest to establish for an employee. For example, a hospital could be sued over the actions of its physician employees for unsafe deployment of an AI/ML algorithm. However, some courts have become more flexible in inferring a relationship that leads to hospital liability in cases that do not involve direct employees.⁴³

To illustrate how vicarious liability might apply in a case involving AI/ML clinical tools, consider a hospital that purchases a sepsis prediction algorithm for the emergency department (ED) or intensive care unit (ICU).⁴⁴ A court could hold the hospital vicariously liable for the negligence of an emergency medicine physician or intensivist who incorrectly interprets an output from the AI/ML system. The rationale for this type of decision would be that patients at risk for sepsis who are seen in the ED or admitted to the ICU usually seek hospital care but do not typically choose their individual ED physician or intensivist; instead, these physicians are usually closely associated with the hospital.⁴³, ⁴⁵

Products Liability

Products liability law related to medical AI/ML products is more unsettled. On the one hand, medical algorithm developers, like the manufacturer of any product, could be liable for injuries that result from poor design, failure to warn about risks, or manufacturing defects.¹⁷, ⁴⁶ Indeed, if an AI/ML system used by health care practitioners results in a patient injury, liability for a design defect may implicate issues with the process for inputting data, software code, or output display.

On the other hand, injured patients will face difficulties bringing such claims for medical AI/ML. In many jurisdictions, a patient would need to show that a viable, potentially cost‐effective alternate design exists.⁴⁶ Additionally, software—as opposed to tangible items or combinations of software and hardware—does not fit neatly into the traditional liability landscape.⁴⁷, ⁴⁸, ⁴⁹ A desire to promote innovation has made courts and legislatures reluctant to extend liability to software developers.⁴⁷, ⁴⁸, ⁴⁹

As a result, the case law in health care software products liability is inconsistent and sometimes unclear.⁵⁰, ⁵¹, ⁵² For instance, one court dismissed a patient's claims against a surgical robot manufacturer because the patient could not show how the robot's error messages and failure caused his particular injuries.⁵⁰ However, another court approved with relatively little discussion a jury award against a developer whose software caused a catheter to persistently ablate heart tissue.⁴⁸, ⁵² This legal uncertainty can lead injured patients to pursue parties other than software developers—including clinicians and health systems—for redress.

Toward a Balanced Liability System

Some clinical software and systems have already failed and caused injury,⁴⁷, ⁵³ and future injuries attributable to AI/ML systems are possible and perhaps inevitable. Going forward, courts and policymakers should design liability systems that seek to achieve the following goals:

• balancing liability across the ecosystem;

• avoiding undue burdens on physicians and frontline clinicians; and

• promoting safe AI/ML development and integration.

The precise solution will depend on how much AI/ML innovation one considers optimal, balanced against one's views on compensation for injured patients. The level of liability directly influences the level of development and implementation of clinical algorithms. Increasing liability may discourage physicians, health systems, and designers from developing and implementing these algorithms. Increasing liability may also discourage experimentation because increasing the cost of errors encourages actors to pursue other ways of improving medical care. In fact, some commentators have observed that the traditional liability system encourages innovation because it accommodates medical advances to foster growth.⁵⁴, ⁵⁵, ⁵⁶, ⁵⁷

Nonetheless, in service of these goals, policymakers must answer several perennial questions of liability design.

Is the Traditional Liability System the Right Way to Process AI/ML Injury Claims?

The traditional liability system incentivizes practitioners and stakeholders to invest in care‐enhancing activities and safer products (Figure 1).⁴⁹, ⁵⁸, ⁵⁹ The precise distribution of liability among actors varies depending on specialty, practice type, locality, and time. Regardless of the baseline, in the traditional liability system, physicians encounter conflicting signals about how the use of clinical AI/ML systems could affect their liability. On the one hand, existing liability structures could encourage physicians to adopt AI/ML in order to improve diagnosis or prediction and reduce the potential for misdiagnoses in clinical care. On the other hand, physicians may be reluctant to adopt opaque AI/ML systems that expose them to liability if injuries occur.¹⁴

Figure 1.

Models of Liability in Artificial Intelligence and Machine Learning

The precise division of liability in the figure models is arbitrary and not meant to reflect empirical liability, which varies by specialty, practice type, and locality, among other factors. In some cases, the precise distribution of liability is not known for a particular group due to the fragmentary nature of data in this area (e.g., settlements under individual insurance companies with nondisclosure agreements). Model A shows the division of liability under the traditional liability model. In model B, stakeholders enter into contracts to transfer and shift liability among themselves. These agreements can take the form of indemnification (one party assuming some or all of the liability of another) or insurance (spreading risk over policyholders), among others. Further, indemnification agreements can be insured (not shown). In model C, a legislature exempts AI/ML partially or completely from the traditional liability system and the government takes over all or some of the risk. Often, these systems rely on taxes or fees on relevant stakeholders. These modifications do not exist in isolation. For instance, if the legislature enacts a program to shield stakeholders from most of the liability risk (model C), stakeholders could still purchase insurance on their residual risk (model B).

Physicians are not alone in facing conflicting liability signals around AI/ML. Health systems might be held liable for adopting AI/ML systems that they do not have the expertise to fully vet, but they could also be liable for failing to adopt AI/ML systems that improve care.

Developers face other challenges. Because software is traditionally shielded from products liability, developers might introduce AI/ML systems too quickly into clinical care, offloading liability onto physicians and health care systems. However, inconsistent or unclear legal decisions against developers could stifle innovation. Furthermore, liability provides just one set of incentives for developers: regulators, payers, and other stakeholders introduce others.⁶⁰, ⁶¹

One solution to the challenges of the traditional liability system that requires no legislative or regulatory action is changing the standard of care. In articulating the standard of care used in the traditional liability system (Figure 1A), courts often look to professional norms.⁵⁵, ⁵⁹ Thus, stakeholders—acting separately or in concert—could alter the standard of care by encouraging the rigorous evaluation of AI/ML algorithms in prospective trials and pressing hospitals and medical practices to appropriately vet AI/ML before implementation.⁶² All stakeholders—physicians, health systems, AI/ML developers, and others—could affect their desired reform by changing their behavior and convincing their peers. Some commentators have already begun to develop frameworks to assess models and algorithms.⁶³, ⁶⁴, ⁶⁵ This process will both smooth clinical integration of AI/ML systems and encourage the development of standardized safety approaches, reducing mismatched incentives and protecting patients.

Further, stakeholders can shift liability to other stakeholders or share liability with them. Because injured patients seek recovery from as many stakeholders as possible, liability can sometimes be a zero‐sum game: placing liability on one stakeholder can release others from liability (Figure 1B). Although courts and traditional liability principles provide some guidance on dividing liability, stakeholders can use contracts to provide more certainty before a lawsuit or to promote other objectives. Some people might argue that tort liability should be focused (at least in part) on imposing safety costs on those most able to bear them. However, we recognize that some actors may wish to promote AI/ML system adoption or innovation by using contracts to shift liability burdens. As such, physicians, health systems, and AI/ML designers might choose to reallocate liability by using contractual indemnification clauses, which allow one party to compensate another in litigation.⁶⁶ Insurance provides a similar function by spreading liability and imposing certain safety norms across policyholders, a practice that is familiar to physicians. The precise division of liability would depend on the negotiating and market power of the stakeholders; of course, some level of liability on all actors is desirable to promote safety. To facilitate this type of risk sharing in AI/ML, stakeholders and their trade/professional societies can provide standard indemnification clauses and can partner with insurance companies to offer insurance plans and risk pools.

Should Specialized Adjudication Systems Be Used?

Legislatures could exempt AI/ML product use from the traditional liability system through specialized adjudication systems. Although such changes are difficult because they require concerted political action, there are precedents for this strategy. For example, Florida and Virginia have neonatal injury compensation programs,⁶⁷ and Congress enacted a similar program for vaccine manufacturers.⁶⁸ These programs tend to collect revenue from a large group, provide relief to specific groups, streamline adjudication, and provide compensation to more individuals than traditional litigation.⁶⁷, ⁶⁹ For instance, the Florida Birth‐Related Neurological Injury Compensation Program collects fees from physicians and taxes each birth to provide a specialized adjudication system that may reduce practice costs for obstetricians while compensating patients for injury.⁶⁷ However, specialized adjudication systems may have unintended consequences.⁷⁰, ⁷¹ In Florida, jurisdictional technicalities have provoked duplicative litigation or placed liability on hospitals for physician errors.⁷², ⁷³

The need for specialized adjudication systems will likely depend on the type of AI/ML deployed. Two algorithm characteristics can assist in making this determination: “opacity” and “plasticity.”⁷⁴ Opacity refers to the user or designer's ability to understand how an algorithm weighs inputs in determining an answer, with fully opaque systems not being understandable by any observer. Plasticity refers to the algorithm's ability to self‐learn and change based on inputs, with highly plastic systems dynamically changing its weightings based on every new input.

On the one extreme, a non‐updating algorithm based on a fixed, limited number of variables with known weights—i.e., an algorithm that is neither plastic nor opaque—is akin to a clinical scoring tool and may not require special treatment.⁵⁷ This is not to say that such an algorithm is error proof, but it can be assessed and checked by physicians, health systems, and payers independently and transparently. Traditional liability systems are a better fit to address errors arising from closed, non‐updating algorithms because these algorithms are more easily assessed by actors.

At the other extreme, a “black‐box” algorithm—i.e., one that is both plastic and opaque—poses more difficult questions. A black‐box algorithm is self‐learning and generally cannot provide the precise reasons for a particular recommendation.¹⁸, ⁷⁵, ⁷⁶ The traditional liability system could respond by imposing a duty on designers, hospital systems, or other stakeholders to test or retrain such algorithms or perhaps to engage in some forms of adversarial stress testing, a well‐established form of assessment in cybersecurity.⁷⁴, ⁷⁵, ⁷⁶ These stress tests would “stress” the algorithm with diagnostic or therapeutic challenges to assess the algorithm's responses to difficult situations not necessarily envisioned by its designers.⁷⁴ That said, it is a formidable challenge to craft an administrable standard of negligence for such cases that would give algorithm makers fair warning about how often they should test or retest in such a case, especially given the heterogeneous nature of algorithm design and patient populations. Moreover, at the level of physician liability, black‐box algorithms may not be a good fit for traditional liability theories because the actors potentially paying for algorithm‐driven mistakes—physicians, health systems, or even algorithm designers—may not be able to avoid making these mistakes when using an opaque and plastic system for a particular patient.¹⁸, ⁷⁴, ⁷⁵ Some may argue that it is appropriate for liability risks to deter adoption of black‐box algorithms, as a way to encourage transparency of algorithms. However, the positive contribution of these black‐box algorithms arises from their ability to constantly update and improve prediction without human intervention across ever more diverse clinical scenarios.

Is there a way to design liability regimes that do not deter promising, beneficial clinical algorithms while also ensuring their appropriate design and deployment? For at least a subset of these algorithms, policymakers could consider more fundamental legal reforms: Courts (or, more plausibly, legislatures) could depart from the traditional liability system or change the standard of proof to distribute liability for a patient injury equitably between all stakeholders.⁷⁴, ⁷⁵ Legislatures may choose to go further and create special agencies to consider claims. These special adjudication systems could develop the expertise to adjudicate AI/ML liability among stakeholders. Alternatively, legislatures could create a compensation program that does not consider liability (no‐fault systems), instead assessing fees on stakeholders (e.g., a fee per patient affected by an algorithm or a fee on physicians/practitioners) (Figure 1C). In the black‐box context, these no‐fault systems would have the added advantage of avoiding the difficult question of what precisely caused an error. Indeed, policymakers could fine‐tune the program depending on the desire to promote AI/ML, ensure safety through manageable personal liability, allocate responsibility among stakeholders, and facilitate other public policy goals. Whether such alterations to liability are politically feasible in a world where tort reform is politically fraught remains an open question.

How Do Regulators Fit into Liability?

AI/ML systems hold lessons for regulators. The National Institutes of Health and several radiology professional societies have collaborated on an agenda to foster AI/ML innovation that includes identifying knowledge gaps and facilitating processes to validate new algorithms.⁷⁷ These efforts indirectly affect liability by providing potential benchmarks that courts and regulators can use to assess a particular system.

However, in another vein, the FDA has published several guidance and discussion documents that treat AI/ML systems as devices.⁷⁸, ⁷⁹ These regulatory efforts also provide information to the liability system about how to assess a particular AI/ML system.

Additionally, binding regulatory actions by the FDA and other agencies can directly affect the liability system. The FDA's decision to regulate clinical AI/ML tools as devices has important—and unsettled—implications for whether regulation displaces traditional liability systems.⁸⁰, ⁸¹, ⁸², ⁸³ Although FDA regulation will likely not directly affect physician or health system liability, developer liability could decrease with more stringent regulatory requirements. This could displace or alter state liability reforms (altering the standard of care or special adjudication systems). Decreased developer liability may also have indirect implications for physician and health system liability under a system of shared liability among stakeholders. Furthermore, FDA regulation might, at least in theory, also work to ensure that AI/ML training populations are representative of patient populations, akin to guidance requiring clinical trials to enroll participants from groups that have been underrepresented in biomedical research studies, including racial minorities and older adults.⁸⁴

So far, the FDA has acted cautiously in this area, instituting incremental reforms or pilot programs to learn more. For instance, the Software Precertification Pilot Program has identified particular organizations or divisions of companies that are committed to the safety and efficacy of their various software applications and is studying how a reformed regulatory process might apply to submissions from these types of organizations. Formally, the FDA's overarching principles for the program are patient safety, product quality, clinical responsibility, cybersecurity responsibility, and proactive culture.⁸⁵ If a precertification program is instituted, the FDA envisions that certain applications from precertified entities might undergo a streamlined approval process or require no FDA review at all. Although such a program may not formally displace state liability schemes, the practices of FDA precertified entities would likely inform courts, trade organizations, policymakers, and others in developing standards of conduct and safety.

Conclusion

AI/ML systems have the potential to radically transform clinical care. The legal system moves at a slower pace, but it cannot be static with regard to this innovation. The relatively unsettled state of AI/ML and its potential liability provide an opportunity to develop a new liability model that accommodates medical progress and instructs stakeholders on how best to respond to disruptive innovation. To fully realize the benefits of AI/ML, the legal system must balance liability to promote innovation, safety, and accelerated adoption of these powerful algorithms.

Funding/Support: IGC and SG were supported by a grant from the Collaborative Research Program for Biomedical Innovation Law, a scientifically independent collaborative research program supported by a Novo Nordisk Foundation grant (NNF17SA0027784).

Disclosure: This article is written for academic purposes only and should not be taken as legal advice.

Conflict of Interest Disclosure: All authors completed the ICMJE Form for Disclosure of Potential Conflicts of Interest. IGC reports that he serves as a bioethics consultant for Otsuka for its Abilify MyCite product, outside the submitted work. RP reports personal fees from GNS Healthcare, Nanology, and the Cancer Study Group; an honorarium from Medscape; and grants from the Veterans Health Administration, the National Palliative Care Research Center, the Prostate Cancer Foundation, the National Cancer Institute, the Conquer Cancer Foundation, and the Medical University of South Carolina, outside the submitted work.