Evaluation and Regulation of Artificial Intelligence Medical Devices for Clinical Decision Support

Abstract

Artificial intelligence (AI) methods were first developed nearly seven decades ago. Only in recent years have they demonstrated their potential to improve clinical care at the bedside. AI systems are now capable of interpreting, predicting, and even generating important medical information. AI medical devices share many similarities with traditional medical devices but also diverge from them in important ways. Despite widespread optimism and enthusiasm surrounding the use of such devices to improve care processes, patient outcomes, and the healthcare experience for patients, caregivers, and clinicians alike, little evidence exists so far for their effectiveness in practice. Even less is known about the safety or equity of AI medical devices. As with any new technology, this exciting time is accompanied by appropriate questions regarding if, how much, when, and who such AI systems really help. Different stakeholders, ranging from patients to clinicians to industry device developers, may have divergent preferences or assessments of risk and benefits, warranting an informed public discussion to guide emerging regulatory efforts. This review summarizes the rapidly evolving recent efforts and evidence related to the regulation and evaluation of AI medical devices and highlights opportunities for future work to ensure their effectiveness, safety, and equity.

1. INTRODUCTION

The concept of artificial intelligence (AI) was proposed during a summer conference at Dartmouth College in 1956 (1). Since then, many attempts have been made to use algorithmic or computerized methods to improve clinical decision-making at the bedside (2, 3). AI offers a promising tool kit for this purpose because clinical decisions are complex, rely on ever-growing and ever-changing data sources, and carry a high degree of uncertainty.

The confluence of several technical and policy developments has brought the promise of AI in medicine closer to reality during the past decade. First, the Health Information Technology for Economic Clinical Health Act has fostered nearly complete adoption of electronic health records (EHRs) among hospitals in the United States (4). This transition away from paper records has yielded a trove of easily accessible and shareable data sources amenable to the development of AI tools. Standardized EHR datasets, common data models, and robust deidentification techniques (5–9) have further enabled researchers to develop and test new approaches to medical AI models.

Second, the creation and widespread availability of open-source programming languages and software have lowered barriers to access for researchers, developers, and data scientists. The Office of Data Science Strategy of the National Institutes of Health encourages researchers to operate under FAIR (findability, accessibility, interoperability, and reusability) principles (10), to which open-source software is well-aligned. R and Python have consistently ranked among the most popular programming languages for statistical computing. Newer languages like Julia continue to gain popularity. All have active user communities with a wealth of online tutorials freely available to interested learners. Such tools have nurtured opportunities for research, innovation, and collaboration by minimizing barriers to accessing state-of-the-art computational tools.

Third, the growth of learning health systems has led to nascent institutional models in which hospitals use their clinical data to inform accurate, timely, and individualized treatment recommendations (11). These efforts have yielded new oversight models that bring together the diverse stakeholders needed to develop, deploy, and oversee the use of AI medical devices (12–14). As these models mature, they will create road maps for the needed infrastructure—including data, computing, governance, ethics, and diverse technical and clinical domains—for hospitals to safely and effectively use medical AI devices in practice.

Fifth, emerging financial incentives for the development and use of AI tools in medicine have created an attractive market for these devices. In 2020, the Centers for Medicare and Medicaid (CMS) approved the first New Technology Add-On Payment for an AI medical device (15). Under this ruling, hospitals could receive payments from CMS for up to $1,400 each time the device is used. Since that time, many other AI medical devices have received such a designation with growing financial incentives (16). However, the rate of investment has outpaced the rate of high-quality evidence production for the clinical effectiveness and safety of such systems, including those undergoing formal regulatory review (17–19). These trends underscore the importance of systems and practices that provide robust oversight and accountability for emerging AI medical devices. As the medical profession has seen the birth of new and revolutionary technologies—X-rays, antibiotics, and vaccines—and found ways to safely incorporate them into practice, so the challenge for AI medical devices presents itself today.

Therefore, here we review several important concepts and processes necessary to ensure that AI medical devices serve their ideal purpose of promoting clinical care that is safe, effective, and equitable. First, we review the definition of a medical device and its implications for oversight and design. Second, we review current frameworks for the regulation of AI medical devices. Third, we review the current state of knowledge regarding the evaluation of AI medical devices. Finally, through this review we identify several emerging themes and questions with associated opportunities for researchers and policymakers.

2. DEFINING AN ARTIFICIAL INTELLIGENCE MEDICAL DEVICE

AI tools typically rely on advanced statistical methods to learn patterns in large datasets and produce a set of parameters, weights, or other features that define a fitted model. Such models can be encoded using widely available computer software and easily shared, updated, and distributed. According to the International Medical Device Regulators Forum (20, p. 1), “software intended to be used for one or more medical purposes that perform these purposes without being part of a hardware medical device” can be classified as software as a medical device (SaMD). Designation as an SaMD is in contrast to other types of software related to medical devices (20). For example, there may be software in a medical device, such as software used to control the pressure and flow through a mechanical ventilator. Or software may be used for the manufacture or maintenance of a medical device, such as software used to evaluate the functioning of a medication pump. The software in these two latter categories, however, are not themselves devices.

2.1. US Food and Drug Administration Device Criteria

AI medical devices fall under the broader category of SaMD. AI decision support tools might reasonably be considered statistical models, bots, agents, sociotechnical systems, or nudges, among other designations, depending on their particular clinical context and use. But it is their designation as medical devices that undergirds the current policy and regulatory framework. The term device might sometimes be used informally to describe any clinical decision support system (CDSS), especially those relying on AI or machine learning (ML) technologies. But a precise definition of what constitutes an AI medical device has evolved over the past several years and concretized into specific criteria relevant for federal oversight. Most recently, the US Food and Drug Administration (FDA) released their final guidance document in September 2022 outlining the four key criteria (see the sidebar titled Defining Device Clinical Decision Support) for what distinguishes a clinical decision support device from a nondevice CDSS (21). A device designation hinges on how a CDSS acquires and displays information, the types of recommendations it provides, and how the end user—often a healthcare professional (HCP)—is meant to use those recommendations.

Criterion 1 excludes machines like electrocardiograms (ECGs), pulse oximeters, and imaging systems (e.g., magnetic resonance imaging or ultrasound), all of which acquire signals directly from patients and are therefore considered devices. Furthermore, software functions that analyze such signals from patients, even if they do not directly capture those signals, also fail to meet Criterion 1 and would still be considered devices. For example, a system that might enhance an image taken from an ultrasound machine or detect an arrhythmia from an ECG signal, even when the original signal was acquired through another device, would still be considered a device.

Criterion 2 requires that a software function is for the purpose of displaying patient medical information, in contrast to providing specific recommendations about that information. For example, a software function that displays—in contrast to a device that captures these data directly from a patient—the latest blood pressure measurement, serum creatinine value, or a radiologist’s report from a chest radiograph would fulfill this criterion. Displaying patient demographic information and clinical resources such as textbooks and peer-reviewed studies would also qualify as nondevice functions.

Criterion 3 outlines how a software function might provide decision support that does not qualify it as a device. Examples of nondevice decision support include standardized order sets, background information on a disease, drug formulary information, and reminders about preventive care. As long as the software function is used to “enhance, inform, and/or influence” a clinical decision, it meets nondevice criteria. However, if the software function is used to “substitute, replace, or direct” a clinician’s judgment, it will not fulfill this criterion and would be considered a device (21). Although the FDA guidance document does provide examples of both device-like and nondevice decision support, the specific criteria used to make this distinction are not well-established.

Criterion 4 highlights that for a software function to receive a nondevice designation, it must enable the HCP to independently review the basis for the recommendations in such a way that the HCP is not primarily relying on those recommendations for a clinical decision. Notably, the FDA guidance document specifically interprets “critical, time-sensitive” tasks to fail this criterion (21). Therefore, all software functions for such tasks, including those related to decision support for sepsis, stroke, or other time-sensitive conditions, would be considered devices.

This last criterion may be challenging to implement in practice because there are no validated instruments for assessing whether an HCP is able to independently review the basis for an AI/ML model’s recommendations. Thus, more empiric research is needed to fully operationalize this criterion for clinical contexts not mentioned in the guidance document. This is particularly important as AI/ML models, and HCPs’ ability to review the basis for their recommendations, may change over time.

Software functions must meet all four of these criteria in order to be considered a nondevice CDSS. If a CDSS fails any one of them, it is considered a medical device and likely subject to FDA review. Notably, this recent document outlines the criteria for a CDSS that intended for use by an HCP. The FDA has previously clarified that a CDSS intended for use by non-HCPs, such as patients or caregivers, qualifies as a device regardless of the severity of the clinical state or if the end user is able to independently review the basis of the recommendations (22).

2.2. Exclusions to Device Designation

In the United States, the 21st Century Cures Act laid out a more detailed framework for the FDA’s role in regulating AI medical devices (23). A section entitled “Clarifying Medical Software Regulation” [Section 3060(a)] amended section 520(o)(1)(A) of the Federal Food, Drug, and Cosmetic Act (FD&C Act) to make several exceptions for software that might otherwise be classified as a device and therefore subject to FDA review. Software functions that provide “administrative support of a health care facility” or those “intended for maintaining or encouraging a healthy lifestyle,” among several other categories, are considered exempt from designation as a device. Software functions intended for “population health management” are also exempted from classification as a device (23, p. 2). It is unclear if widely used commercial population health management algorithms, such as those shown to reinforce racial disparities (24), would meet this exemption criterion.

3. ARTIFICIAL INTELLIGENCE MEDICAL DEVICE REGULATION

There are several evolving layers of oversight that are intended to ensure the safety, effectiveness, and equity of AI medical devices. While AI medical devices bear some similarities to traditional medical devices, they also possess several unique characteristics that require new approaches to regulation and oversight. Thus, federal agencies, AI researchers, and hospitals have all so far proposed and, in some cases, implemented novel approaches of overseeing AI medical devices. More input from diverse stakeholders, creative solutions from policymakers, and rigorous evaluations of oversight approaches will be needed to achieve the optimal balance of safety and innovation for AI medical devices.

3.1. Federal Oversight

The US government oversees some AI medical devices through a multifaceted and evolving strategy.

3.1.1. US Food and Drug Administration.

How does the FDA have authority to determine which AI systems qualify as medical devices? The FD&C Act, passed in 1938, gave the FDA authority to regulate food, drugs, cosmetics, and some medical devices. The FDA only began to actively regulate medical devices in 1976 after growing public concern about the safety of the Dalkon Shield contraceptive device (25). At the time, there was no federal requirement for a premarket assessment of safety or efficacy of any medical device (26). Only after significant injury to patients was reported, including deaths, infections, and other complications, did Congress pass the Medical Device Amendments that gave the FDA authority to regulate all medical devices (27, 28). This legislation established a risk-based classification scheme for evaluating medical devices and created the premarket approval (PMA) and 510(k) pathways that are still used today (29).

Most medical devices are now regulated by the FDA through the Center for Devices and Radiological Health through one of three pathways (30, 31). First, new devices that are designated as high risk (Class III), such as ventilators and pacemakers, require a PMA with data indicating safety and effectiveness, often obtained through a clinical trial. Second, a manufacturer may petition the FDA to reevaluate a device’s risk category, potentially shifting the designation to moderate (Class II). In this case, a device may be classified as de novo and undergo review, and potentially clearance, through a less stringent premarket notification (PMN) pathway. Third, a moderate risk device (Class II) that is deemed “substantially equivalent” to a previously authorized and marketed medical device, known as a “predicate,” can also be evaluated through a PMN, or 510(k), pathway (30). Although the PMN pathway requires some assurance of safety and effectiveness, it does not require the same extensive clinical evidence needed for a PMA. While other pathways for medical devices to receive FDA authorization exist, they do not typically apply to modern AI medical devices (28–31).

However, in addition to requiring substantially less robust clinical evidence compared to a PMA, the PMN pathway is susceptible to the problem of “serial predicates” (30). This problem arises when a device receives clearance based on a predicate, which in turn received clearance based on another predicate, and so on, potentially spanning many generations of medical devices over several decades. For example, in a review of AI critical care medical devices by Lee et al. (18) published in 2023, among nine devices cleared through a PMN, only three had any AI-related predicates and the earliest predicate in the predicate tree stretched back to 1982. Another review of the genealogy of PMNs in the FDA AI database found one device with 15 generations of predicates and highlighted the key safety concern of “predicate creep,” or “iterative design changes [across generations] resulting in unproven devices dissimilar from original predicate” (32). Thus, the PMN pathway that requires little clinical evidence if substantial equivalence to a predicate device is established may undermine efforts to establish safety and effectiveness for modern AI medical devices.

To promote transparency in the regulation of AI medical devices, since 2020 the FDA has published a public database of all approvals and clearances for devices that rely on AI and ML tools.¹ The database is updated every few months and has served as a rich data resource for public awareness and for research into AI/ML device regulation (18, 19, 32, 34). Based on the information contained in this database, most FDA-authorized AI devices are cleared through the PMN pathway.

FDA leaders have recently outlined priorities for future regulatory approaches that include life cycle management, special mechanisms for generative AI systems, and a focus on patient health outcomes (35).

3.1.2. Changing devices over time.

A key advantage of AI medical devices is that they may continue to learn from new data even after they are deployed. Although this feature is important for AI systems, it is not unique to them, and it is relevant for the oversight of non-AI medical devices, too. Since 2017, the FDA began offering guidance about how device manufacturers should deal with changes, or updates, to already authorized systems. After releasing several guidance documents and incorporating public comments, the FDA released most recently a draft guidance document in August 2024 titled “Predetermined Change Control Plans for Medical Devices” (36). This updated document outlines requirements for reasonable assurances of safety and effectiveness, and with substantial equivalence for devices with predetermined change control plans (PCCPs). Manufacturers may submit PCCPs for devices receiving authorization through the PMA or PMN pathways, including those with de novo designation. Typically they are expected to be submitted at the time of the submission for the medical device and the PCCP is considered part of the PMN or PMA. The PCCP allows a device to be updated using additional data that might improve its predictive performance. However, changes such as using a new data source or predicting a new outcome are likely not appropriate for a PCCP and would instead warrant authorization as a new device.

3.1.3. Regulatory pilot program.

In 2017, the FDA developed a new regulatory framework appropriate for SaMDs because they are so different than traditional medical devices. The new Digital Health Software Precertification (Pre-Cert) Pilot Program allowed manufacturers to bring devices to market in a streamlined process after the manufacturers themselves demonstrated adherence to best practices in promoting safety, quality, responsibility, and other best practices for AI device development. Over 100 companies expressed interest in the program (37). Ultimately, the nine manufacturers selected for the pilot precertification program included Apple (Cupertino, California), Fitbit (San Francisco, California), Johnson & Johnson (New Brunswick, New Jersey), Pear Therapeutics (Boston, Massachusetts), Phosphorus (New York, New York), Roche (Basel, Switzerland), Samsung (Seoul, South Korea), Tidepool (Palo Alto, California), and Verily (Mountain View, California). The pilot program was ultimately closed in 2022, in part because the FDA reported that it did not have the statutory authority to fully implement necessary elements of its program (38). Although the program has remained closed, newer proposals to revive firm-based regulation of AI medical devices would likely require action from Congress to provide extended statutory authority in order for such an approach to have a chance of success (39).

3.1.4. Office of the National Coordinator for Health Information Technology.

The Office of the National Coordinator for Health Information Technology (ONC) is responsible for the certification of EHR systems in the United States. The ONC released a proposal for the Health Data, Technology, and Interoperability (HTI-1) rule in April 2023. After publication and incorporation of public feedback, the ONC released its final rule in December 2023 that lays out extensive reporting requirements for predictive decision support interventions (DSIs) (40). Importantly, these requirements apply only to those DSIs embedded within certified information technology (IT), the purview of the ONC, which is most commonly the EHR. Some of the mandatory reporting elements include measures of predictive performance, a description of the approaches to ensure fairness, and details of an external validation process. These quantitative assessments and process details far exceed any other federal reporting requirements of AI medical devices. However, the reporting requirements are focused on model performance and do not explicitly require any assessments of a model’s clinical effectiveness. At this time, HTI-1 requires disclosure of these elements by the manufacturer of an ONC-certified IT system. But HTI-1 does not require public disclosure of all of these elements, potentially leaving hospital leaders, clinicians, and patients without sufficient data to inform important investment or clinical decisions.

3.1.5. Biden’s artificial intelligence announcement.

In November 2023, President Biden said that, with respect to AI, “It must be governed” (41) and released the Executive Order on the Safe, Secure, and Trustworthy Development and Use of Artificial Intelligence (42). Although intended to address AI technologies across a broad array of industries and domains, the order had specific and sometimes uncertain implications for AI medical devices (43, 44). First, the order laid out how existing federal laws, even if not created specifically for AI, would still apply to AI systems. For example, laws related to privacy, discrimination, and unfair business practices are still relevant for protecting people against potential harms of AI medical devices. Second, the order required the establishment of standards for “trust and safety” of AI systems. This requirement would create a tracking system for medical errors committed through the use of AI systems and an “AI assurance policy” that would create an infrastructure to evaluate medical AI systems. Third, the order required the appointment of a new chief AI officer within the Department of Health and Human Services to align efforts across multiple federal agencies. Finally, the order recognized the importance of a skilled workforce to evaluate and implement these laws as they apply to AI systems in mandating the National Science Foundation to create at least four new institutes dedicated to training in AI-related skills.

There are still many open questions and uncertainties about how the federal regulatory landscape will evolve. For example, what regulatory requirements should be in place for device CDSSs that are already in widespread use but have not yet received authorization from the FDA? For those manufacturers or health systems that do not comply, how will the FDA approach enforcement? Will CMS include AI-related safety requirements in their Conditions of Participation (45)? Existing regulatory frameworks for laboratory testing provide several lessons and insights for how the federal government and local hospitals might collaborate in overseeing AI medical devices, but it would likely require expanded statutory authority to implement such a system (46, 47). Will Congress act to further empower CMS, and/or other federal agencies, to take responsibility for additional aspects of AI medical device regulation? Only through further congressional action, active public comments, and engaged discourse across stakeholders and sectors of society will the federal policy landscape realize the potential of AI medical devices.

3.2. Local Hospital Oversight

Regardless of where the federal oversight framework, currently led by the FDA and ONC, evolves, there will be a critical need for a local, hospital-level oversight infrastructure (22). There is substantial evidence that predictive AI models require ongoing validation and retraining as their performance changes over time and in new locations (48–51). Thus, if an AI medical device receives authorization from the FDA based on clinical data from one or a handful of centers and is then deployed around the country, there is no guarantee of how it will perform elsewhere or that its performance or effectiveness will remain consistent. This need for local validation and monitoring is further underscored by the limited geographic diversity in public datasets commonly used to train AI systems (52).

Thus, hospitals, clinics, and local healthcare delivery organizations will have an important role to play in the local oversight and governance of AI medical devices. Some health systems have already built sophisticated infrastructures to oversee their local AI systems (12, 14). But most of these efforts are concentrated in well-resourced, urban academic centers. Although well-suited to the task because of available infrastructure, personnel, and mission, such centers are unlikely to reflect the workflows, populations, and local clinical needs of many hospitals around the rest of the country (52). These efforts have also been accompanied by exciting organizational innovations, including creating clinical AI departments (53) and appointing chief AI officers (54).

3.3. Locally Developed Artificial Intelligence

An important area of uncertainty in the regulation of AI medical devices is how current oversight mechanisms will consider systems developed and deployed locally in a hospital. The FDA regulatory guidance is intended for industry manufacturers, while the ONC’s regulatory efforts are focused on certification of commercial EHR platforms. What about the many AI CDSSs that are developed by researchers, data scientists, informaticians, and other stakeholders for use in their own hospital? Such AI systems may not fall under the purview of any federal regulatory agency directly but still warrant review to ensure their effectiveness, safety, and equity. The local hospital oversight efforts described above are a promising first step but are likely limited to only well-resourced academic centers. Thus, new policy frameworks and oversight infrastructure will be needed to effectively balance the safety and innovation of these locally developed AI medical devices.

3.4. Generative Artificial Intelligence

Generative AI systems are those that create content—text, images, audio, and/or video—in contrast to traditional AI systems that typically provide a comparatively simpler prediction of a disease, treatment, or outcome. Such generative systems rely on stochastic sampling from probability distributions learned from massive datasets using combinations of unsupervised and supervised learning procedures. Consequently, the output of generative AI systems is often startlingly human-like, confidently composed, and seems promising to support a broad array of clinical activities and decisions (55–58). However, these systems have a tendency to confabulate (59), provide inappropriate medical advice based on a person’s race or insurance status (60, 61), and disregard prompts requiring compliance with the FDA’s medical device criteria (62). Thus, generative AI systems pose particular difficulties for regulators trying to balance innovation and safety while awaiting the growth of more mature environments, or ecosystems, with sufficient resources to operationalize mature oversight program (63, 64).

4. ARTIFICIAL INTELLIGENCE MEDICAL DEVICE EVALUATION

Just as traditional medical devices warrant evaluation before they are used at the bedside, AI medical devices require the same (65). Many proposals from clinical and methodologic experts have outlined criteria for the reporting, evaluation, and oversight of these devices (12,66–72). But these guidelines and frameworks remain variably adopted and adhered to. Most published evaluations of AI medical devices are in silico or are based on a model’s predicted performance in a retrospectively collected dataset. There are comparatively very few studies reporting evaluations in clinico, or how a model affects clinical decisions or patient care when deployed at the bedside (73). This gap in clinical effectiveness evaluations is due, in part, to the still nascent and evolving scientific, clinical, and regulatory knowledge about AI medical devices. Still, this gap poses current challenges for patients, clinicians, researchers, and hospital leaders who must make decisions about investment, deployment, and clinical care without substantial clinical evidence. Additionally, these stakeholders’ incentives many not align to support rigorous, randomized evaluations of AI medical devices (74). Many hospitals also lack the resources and infrastructure needed to conduct robust evaluations of AI medical devices. Thus, robust AI medical device evaluations are critically lacking. Increased understanding of the different types of AI medical device evaluations, each with a distinct role, may help inform clinical, scientific, and operational decisions.

4.1. Phases of Research

In silico evaluations of a model’s performance are essential for its comprehensive assessment. However, estimates of predictive performance should not be conflated with estimates of clinical effectiveness or safety. The framework outlined by Park et al. (75) (Figure 1) is helpful in interpreting different types of evaluations of AI medical devices aligned with more familiar evaluations of drugs and traditional medical devices.

Figure 1.

Phases of research and exemplar studies at each phase in the development of drugs, traditional medical devices, and artificial intelligence (AI) medical devices. Because AI medical devices are relatively new, the clinical relevance of the distinction between phases of research may be less familiar. Figure adapted from Reference 75 (CC BY 4.0).

Each type of evaluation is important and represents a key step in the development of AI medical devices that are safe, effective, and equitable. And each step should be interpreted according to the quality of the evidence, the inherent limitations of the study design, and its consequences for patient care.

4.2. Performance Evaluation

Retrospective, in silico evaluations are essential components of any AI medical device’s comprehensive assessment. Such studies typically focus on estimating the predictive performance of an AI system or how well that system is able to guess some outcome or state compared to what was actually observed. These evaluations typically correspond to Phase 0 or Phase 1 research.

Predictive performance evaluations require investigators to choose metrics by which to assess a model (76–78). Model discrimination, or the ability of a model to distinguish between two or more outcome categories, is commonly measured with the concordance statistic (or C-statistic), sensitivity, specificity, or positive or negative predictive value. Model calibration, or the ability of a model to produce a probability of an outcome that is aligned with the observed probability in the case of a binary classifier, is commonly measured with the calibration slope, inspection of calibration curves, or integrated calibration index. There are also composite measures of model performance that simultaneously capture discrimination and calibration, such as the scaled Brier score, R² value, and the logarithmic loss. Importantly, no single measure captures all aspects of a model’s performance. Therefore, a comprehensive AI model performance evaluation will report multiple measures.

None of these measures, however, directly addresses a model’s clinical impact. Several measures have been developed to translate a model’s predictive performance into more clinically relevant terms. These include the number needed to evaluate, also known as the workup to detection ratio (79), the net benefit (80), and formal decision analysis (when the effects of interventions deployed in response to an AI tool’s suggestions are known) (81). The use of those more clinical measures can extend the insight gained through an in silico evaluation. But they should be interpreted as tentative, or back-of-the-napkin, rather than definitive estimates of clinical impact (82).

Retrospective performance evaluations far outnumber prospective evaluations. For example, Fleuren et al. (83) reviewed 172 published manuscripts of ML models for use in critical care and found that 160 (93%) were still in the prototyping and development phase. There are several reasons why such retrospective performance evaluations are likely the most common type of AI medical device study. First, they are the least costly and time-consuming to conduct. These in silico studies typically rely on the same data on which a model is trained. These studies do not require human factors inquiries, EHR integration, clinician or patient engagement efforts, coordination with hospital operations leaders, or federal regulatory review. Second, although adherence remains variable (84), the reporting standards for studies of predictive performance are reasonably well-established with consensus on core elements of appropriate methods, outcomes, and study designs (71, 85–87).

The key threats to assessing an AI medical device’s predictive performance in a retrospective evaluation are overfitting and drift. Overfitting occurs when an AI model captures not just the signal but also the noise in a dataset during training (88). Overfitting leads to overly optimistic performance assessments if a model is evaluated only on that training data. Thus, other approaches to validation, such as random splitting, cross validation, or bootstrap correction, can provide appropriately optimistic measures of a model’s performance. Drift occurs when an AI model’s performance degrades over time or space due to changes in the input and/or output variables (49). Thus, temporal, geographic, or other types of external validation are essential for assessing how an AI medical device may perform in a population distinct from that used for its development. However, external validation is not always necessary. If an AI tool is intended for local use only, then a targeted validation in the population where it was trained may be sufficient (89). The development of national assurance networks focused on independent testing could fill some gaps in needed AI performance evaluations (90).

Strong predictive performance in a retrospective evaluation is a first step in assembling a record of evidence for how an AI medical device might perform when deployed in clinical practice. However, a predictive performance evaluation is necessary but not sufficient for a comprehensive assessment of an AI medical device.

4.3. Clinical Effectiveness Evaluation

Prospective, in clinico evaluations measure the effects of an AI medical device on care processes and patient outcomes when deployed in practice. While comparatively few such studies have been published, they are essential in understanding a device’s impact in the real world. These evaluations typically correspond to Phase 2 or Phase 3 research.

There are several ways in which a prospective clinical effectiveness evaluation of an AI device could differ significantly from a retrospective performance evaluation. First, the data sources may change dramatically. Pulling retrospective data and pulling prospective data may require different extraction pipelines, exhibit different missingness patterns, and be represented using different fields or ontologies. Second, the end user may respond to a model’s predictions, whether they are correct or not, in unpredictable ways. Thus, rigorous assessments of human factors and consideration of a device’s deployment as a sociotechnical system are likely to improve the chances of success (91–93). The actual effect of an AI medical device on clinical practice cannot be known until it is deployed and evaluated.

There are few randomized clinical trials of AI medical devices relative to the number of published retrospective performance evaluations (94, 95). For example, the clinical trial reported by Abràmoff et al. (96) of an AI system to detect diabetic retinopathy was the first preregistered clinical trial of a diagnostic AI medical device and led to the first FDA approval for such a system (97). Among the most rigorous clinical evaluations of an AI medical device is the randomized clinical trial by Wijnberge et al. (98). This study showed how an AI-based prediction model led to improvements in care processes and patient outcomes by detecting hypotension during noncardiac surgery and making specific recommendations for interventions. This study was described as akin to a “positive phase 2b drug trial” (99, p. 1044). Even among those AI medical devices that have already undergone review by the FDA and received an approval or clearance, there are very few reported peer-reviewed investigations or reports of clinical effectiveness (18, 19).

4.4. Safety

There is a concerning lack of data about how AI medical devices may cause harms. Even AI systems that are intended to allocate resources, rather than make care decisions directly, may harm patients. For example, a prediction model that identifies clinical deterioration among hospitalized patients is intended to prompt an emergent evaluation. In doing so, the alert focuses resources on some patients while reducing attention paid to other patients, who are then at increased risk of a deterioration event (100). Thus, the evaluation of safety of an AI medical device, regardless of its intended end user, is a critical but often missing component of AI medical device testing. Importantly, although retrospective evaluations could identify potential safety issues related to errors in discrimination or calibration of suggested treatments, they are insufficient to fully evaluate the safety of using an AI medical device.

A key concern related to the safety of AI systems is automation bias, sometimes called over-reliance (101, 102). Automation bias occurs when a clinician cedes cognitive authority to an AI system and heeds the advice of that system without adequately reviewing the appropriateness of that advice. Although having a human in the loop is a practice hypothesized to ensure the safety of AI medical devices (103, 104), the presence of automation bias undermines that as a sole strategy with growing evidence that humans may overrely on the suggestions of an AI system even when they are incorrect (105, 106). Further evidence suggests that simply explaining the predictions of an AI system, in an attempt to overcome the black box, is not sufficient to fully ameliorate automation bias (107, 108). This phenomenon is especially important in clinical settings like the intensive care unit because automation bias increases with the complexity of the decision (109) and is especially risky in time-critical environments (110).

On the other hand, algorithm aversion bias refers to the tendency to ignore an AI system even when its suggestions are correct (111). This phenomenon risks clinicians overlooking the potential benefit of AI systems when they do have insights to offer. Thus, there is a significant knowledge gap about how end users will interact with AI medical devices broadly and across varied clinical contexts. More research is needed to understand these dynamics and how their effects might be modified by clinician, context, and task-specific characteristics. More work is needed prior to AI device deployment to understand if and to what extent these interaction patterns might be present and to test optimal approaches to mitigation.

4.5. Equity

A critical component of any AI medical device evaluation is that of its effect on health equity. Many groups of people already experience significant and well-documented disparities in their medical care and outcomes in a broad range of health contexts. These disparities are apparent across sex, gender, race, ethnicity, sexual orientation, rurality, preferred language, and other characteristics (112–117). Many of these disparities are explained by large-scale, structural differences in resource allocation, historical deprivation, and policy, and/or by individual-level biases of decision-makers. AI medical devices are then trained on large datasets that are the result of social patterning in the underlying data generating process. Thus, AI medical devices simultaneously offer tremendous promise to ameliorate bias and inequity in healthcare while also carrying commensurate risk to reinforce and even exacerbate these inequities. Even clinical algorithms, which may or may not qualify as AI medical devices, may have a substantial impact on racial and ethnic disparities (24, 118–121).

Existing federal regulatory guidelines are not sufficient to ensure equitable AI medical device development (18, 21). More recent scientific reporting guidelines have emphasized the need for fairness evaluations in the reporting of AI prediction models (71). Local hospital oversight mechanisms will likely be necessary to verify fairness in AI medical device development and deployment (12). Additionally, the adoption of research practices that promote equity, inclusion, and access will foster innovative environments that are more likely to lead to the development of equitable AI medical devices (122).

DEFINING DEVICE CLINICAL DECISION SUPPORT.

In September 2022, the FDA released a final guidance document (21) outlining four criteria to distinguish device from nondevice CDSSs. That is, software may be considered not a device if it is:

Criterion 1

Not intended to acquire, process, or analyze a medical image or signal from an in vitro diagnostic device or a pattern or signal from a signal acquisition system.

Criterion 2

Intended for the purpose of displaying, analyzing, or printing medical information about a patient or other medical information.

Criterion 3

Intended for the purpose of supporting or providing recommendations to an HCP about prevention, diagnosis, or treatment of a disease or condition.

Criterion 4

Intended for the purpose of enabling an HCP to independently review the basis for the recommendations that such software presents so that it is not the intent that the HCP rely primarily on any such recommendations to make a clinical diagnosis or treatment decision regarding an individual patient.

SUMMARY POINTS.

1. Medical devices relying on artificial intelligence (AI) technologies share both similarities and differences with traditional medical devices.

2. AI medical devices show significant promise for improving care processes, clinical workflows, and patient outcomes.

3. AI medical devices could undermine patient safety by recapitulating biased care, producing unsound clinical suggestions, or not having their output adequately reviewed by bedside clinicians.

4. There is currently little high-quality evidence about the effectiveness, safety, or equity of using AI medical devices in clinical practice.

FUTURE ISSUES.

1. New regulatory and oversight approaches are needed that account for the unique features of AI medical devices.

2. Innovative federal, state, city, and hospital policies can all contribute to ensuring the safety, effectiveness, and equity of AI medical devices.

3. Increased funding, training, infrastructure, and incentives are all needed to develop a rigorous body of evidence for the use of AI medical devices in clinical practice.

4. More prospective research is needed to better identify safe, effective, and equitable approaches for integrating AI medical devices into clinical practice.