United States Food and Drug Administration Regulation of Clinical Software in the Era of Artificial Intelligence and Machine Learning

The U.S. Food and Drug Administration (FDA) has been regulating medical devices since 1976. The introduction of novel medical devices, including software as a medical device (SaMD) has led to more recent challenges given the rapid evolution of laws, technologies, and review pathways.¹ The FDA defines medical devices as an instrument, apparatus, implement, machine, appliance, implant, reagent for in vitro use, software, or material that can be used in the diagnosis, cure, mitigation, treatment, and prevention of diseases.² This definition excludes medical devices, which primarily achieve their intended purpose through pharmacological, immunological, or metabolic means, but devices may be assisted in their function by these means.² Software as medical device is a rapidly expanding area of development, with a global market size worth $18.5 billion.³ The FDA defines SaMD as software intended to be used for one or more medical purposes that perform these purposes without being part of a hardware medical device.⁴ This definition distinguishes SaMD from software in a medical device, both of which are regulated by the FDA. For example, SaMD might consist of X-ray imaging processing software, but software that improves visualization within digital radiography systems (the primary device) would be classified as software in a medical device. With ongoing changes in regulatory expectations and software technology, there is a need to understand the benefits and limitations of the arduous medical device review process to ensure safe technology reaches end-users. This commentary will focus on explaining the regulatory requirements for SaMD, with special attention to artificial intelligence/machine learning (AI/ML), and offer suggestions to improve this complex approval landscape while prioritizing patient safety.

Current SaMD Carveouts

The FDA has published guidelines to help manufacturers determine if their software technology is a medical device that requires SaMD classification and FDA oversight (Figure 1). The SaMD excludes tools with the intended use of supporting administrative tasks (eg scheduling tool), electronic health records (EHR) (eg medical data displays), general health and wellness (eg fitness applicatios), and Medical Device Data Systems.^5,6 On the basis of the 21st Century Cures Act, clinical decision support (CDS) software are also exempt from FDA regulations, but this ruling has been contentious due to overlapping features between CDS tools and SaMD.^6,7 The CDS software is not intended to acquire, process, or analyze medical data; it may offer providers diagnostic and treatment recommendations in non-urgent settings but cannot replace or direct provider judgment (Table).^4,8,9 For example, a sepsis calculator that predicts a patient’s risk of sepsis on urgent or emergent admission into a singular value to drive clinical judgment is SaMD. However, a tool that determines cardiovascular risk for a patient in non-emergent settings and offers multiple recommendations on how to minimize this risk is considered CDS.¹⁰ Applying this exclusion criterion ensures that health-related but non-medical software like a diet tracker or EHR organizational tool can bypass rigorous approval processes for easy accessibility for consumers.

Figure 1.

Framework to determine if a software is considered a medical device that requires FDA regulation.

Table.

FDA Definitions of SaMD, SiMD, and CDS

FDA Term	Definition	Example
Software in medical device (SiMD)	Software that cannot operate separately from a device and/or performs its primary function with a device	•Bluetooth or WiFi connectivity software in an ultrasound •Image enhancer software in an X-ray machine
Software as medical device (SaMD)	Software intended to be used for one or more medical purposes that perform these purposes without being part of a hardware medical device	•Electrocardiogram analysis software •X-ray imaging processing software
Clinical decision support (CDS)	A software that: •Does NOT acquire, process, or analyze medical image or signal •Displays medical information •Supports a healthcare professional but does not offer diagnosis or treatment •Enables providers to independently review recommendations	•Computerized alerts/reminders for upcoming appointments in a patient's medical chart •Vaccination reminder in electronic health record

AI/ML Algorithms as Medical Devices

When used for health care purposes, AI/ML algorithms may represent a type of SaMD. The FDA defines AI as “a machine-based system that can make predictions, recommendations, or decisions that influence real or virtual environments”,¹¹ with ML defined as “the set of techniques that can be used to train and optimize AI systems.”¹¹ The FDA approved its first AI/ML device, PAPNET—AI image processing tool to identify abnormal cells in Pap smears—in 1995 and has since approved more than 1000 AI/ML tools.^12,13 Many AI/ML devices focus on image acquisition and processing, early disease detection, pattern identification in human physiology, and development of personalized diagnostics in fields like radiology and cardiology.¹⁴ Artificial intelligence is also being used in drug development and clinical research to assist in participant recruitment, trial analysis, and post-market surveillance.¹³ Given oversight challenges, the FDA is yet to approve a generative AI—a type of AI that can create new content based on patterns learned from existing data—large language model that creates new content based on available data.¹³

Regulatory Pathway of SaMD

The FDA’s medical device regulation process is dependent on a device’s novelty and potential for patient harm, which leads to a low-risk, intermediate-risk, or high-risk classification.¹⁵ Most low-risk devices like a steps counter mobile application are exempt from review, especially if they are considered similar to existing devices.¹⁵ Intermediate-risk devices such as infusion pumps or AI-powered software that identifies heart failure in echocardiograms undergo a premarket notification (PMN), or 510(k) review, to determine if they are similar to existing devices, and this process typically does not require clinical trial data.^15,16 High clinical risk devices such as pacemakers and AI-powdered software that identifies occult lesions on mammography are required to submit an investigation device exemption (IDE) and premarket approval (PMA) to collect clinical trial data and report device safety and effectiveness, respectively.^15,17 Devices are automatically labeled high-risk if they are novel. However, if there is a predicate device that reports a high-risk device is substantially equivalent, manufacturers can follow the PMN process instead of a IDE/PMA.^15,18 For novel devices with low-to-moderate risk, manufacturers can submit a 513(g), requesting reclassification of the device as de novo and enter the PMN process.¹⁹ The FDA has aimed to optimize these processes and attempts to complete IDE, 513(g), PMN, and PMA approvals within 30, 60, 90, and 180 days, respectively.¹⁹

Given their novelty, many AI/ML SaMD are initially labeled as intermediate- to high-risk. A study by Muehlematter et al²⁰ reviewed AI/ML devices approved by the FDA from 2015-2020 and found that 204, 15, and 3 devices went through the PMN, de novo, and PMA approval process, respectively.²⁰ However, many PMN-approved AI/ML devices have intended uses that are dissimilar from their predicates, which raises safety concerns.²¹ To date, the FDA has only approved AI/ML tools that utilize “locked” algorithms, which the FDA defines as, “an algorithm that provides the same result each time the same input is applied to it and does not change with use.”²² The process of ML itself typically implies an iterative process that benefits from ongoing training data. However, when an FDA-approved AI/ML device was put on market, manufacturers were required to lock the algorithm to limit any future training of the AI on new data. Until recently, to update an AI/ML algorithm, manufacturers submitted a new PMN if the proposed change altered device safety and performance, significantly modified the algorithm, or affected clinical functionality.^23,24 If the proposed modification introduced a new, high-risk intended use, manufacturers would have to submit a de novo or PMA.^23,24 For example, if manufacturers wanted to update an AI/ML tool for physician-users that identified benign and malignant skin lesions with updated real-world data, the manufacturer would have to submit a new PMN.²² Although the goal of this framework was to ensure that marketed devices had consistent safety and performance, re-approval processes can prevent the use of the most up-to-date, secure, and accurate technologies.²⁵ Furthermore, many AI/ML devices utilize adaptive algorithms, meaning they have different outputs for a given set of inputs due to continuous analysis and integration of updated datasets.²² This is important when AI/ML algorithms are applied in new locations and need to be optimized and trained on local patient data to accurately identify diagnoses or treatments.

Updated Regulatory Framework for AI/ML Medical Devices

To address this unique set of regulatory challenges, the FDA proposed the AI/ML SaMD Action Plan in 2021 that was finalized in December 2024.¹¹ Because AI/ML technologies are constantly adapting to real-world performance evaluations and data, the FDA is using a total product life cycle (TPLC) regulatory approach.¹³ Part of this plan includes good machine learning practices, which are 10 principles to guide the development of high-quality and safe AI/ML devices.²⁶ Good machine learning practices encourages manufacturers to apply robust cybersecurity and coding practices, use unbiased and representative datasets, and maintain transparency with end-users about algorithm updates.²⁶ For the latter, the action plan introduces predeterminate change control plans (PCCP) that consist of SaMD pre-specifications and algorithm change protocol (ACP) (Figure 2).²² The SaMD pre-specifications sets the scope of permissible device modifications. For each modification, the ACP provides detailed information on data management practices, re-training protocols, performance evaluation, and update procedures.²² Before device implementation, the FDA must approve the proposed PCCP. If an AI/ML device is updated within the scope of the PCCP, manufacturers will need to maintain and submit documentation of that modification but will not require a re-approval processes.²² Using the previous example of an AI/ML tool that identifies skin lesions, if the software improved its performance by learning from real-world data, as was described in the approved ACP, then the modified ML algorithm would not need additional FDA review.²² However, if a proposed change is not addressed in the PCCP or leads to a new intended use, manufacturers have to contact the FDA for a focused review of the proposed changes and may even need to apply for a PMA.²² Continuing with the previous example of an AI/ML tool that identifies skin lesions, if manufacturers wanted to make the device patient-facing, introducing a new and unintended use, they may need to submit a PMA.²² Through these changes, post-market AI/ML devices can be safely monitored for their real-world applications and receive timely modifications to meet consumer needs.

Figure 2.

Updated AI/ML regulatory framework with consideration for adaptive algorithms.

The PCCPs are a critical and appropriate step in addressing this iterative nature of AI/ML development, but this system will result in high volume post-market regulatory needs that the FDA may be unprepared to meet. SaMD algorithms can undergo daily updates, and it can be difficult to describe future modifications and software instabilities without evaluating real-world data outcomes.²⁵ Consequently, many devices may require changes that are beyond the scope of approved PCCPs, and the FDA will have to review modified PCCPs.²² Currently, it takes the FDA 90 days to review 510(k)s, but it is unclear if PCCP reviews led by technical experts will be conducted in a shorter timeline. Furthermore, if the FDA wants to make it easier for AI/ML manufacturers to independently modify their devices, they may want to obtain consistent and clear documentation of real-world outcomes. In their 2021 Action Plan, the FDA suggests adapting existing programs like Pre-Cert, which provide real-world performance analytics.²² Warraich et al¹³ identify clinical and nonclinical performance testing, software verification, validation and hazard analysis, and human use assessment as critical controls to monitor AI/ML safety. Given the frequency of these changes, the FDA may want to incorporate unbiased external stakeholders to assist them in timely evaluations during the post-market surveillance stage.

Another central issue regarding AI/ML regulatory practices is improving transparency between manufacturers and end-users. A study by Muralidharan et al²⁷ reviewed 692 FDA-approved AI/ML devices from 1995-2023 for transparency and safety reporting and found that only 46.1% provided detailed performance studies, 1.9% linked a scientific publication with safety and efficacy data, and 9% conducted a prospective study for post-market surveillance. Muehlematter et al²¹ reviewed FDA’s de novo and PMN database and identified that 9.4% of approved AI/ML devices have been recalled, but a third were re-approved. This lack of transparency raises concerns regarding legal liability in the event of incorrect outputs by AI/ML SaMD. In general, reviews of medical error-related lawsuits against AI software companies, healthcare systems, and individual physicians suggest that lawsuits are rarely successful against AI software companies, with some case law suggesting that software companies have no duty to provide an accurate diagnosis, as they are not health care providers and licensing agreements stipulate that final decision-making responsibility lies with the clinician.²⁸ However, the inability to understand the precise mechanisms by which an output is generated may make it challenging for physicians relying on the algorithm to know if its application was appropriate for the specific patient. Thus, improvements in transparency are critical. The FDA also underscores the importance of increased transparency in their 2025 AI/ML action plan and implemented public submission summaries, where manufacturers are required to describe the characteristics, applications, and expectations of their AI/ML tools.²⁹ A key takeaway from the FDA’s digital health advisory committee meeting in November 2024 was that patients want to know when and how AI/ML is used in their care.³⁰ All implemented modifications and reported adverse events should be regularly updated in an accessible and organized database for the FDA, clinical partners, and patients. Currently, the FDA is accomplishing this through the TPLC Database, but there are also external organizations like The Medical Futurist that catalog FDA-approved AI/ML devices.^23,31

Future Directions for FDA Regulation of AI/ML Devices

As the FDA rolls out their updated regulatory framework for AI/ML devices, they may want to consider how they will collect data from other entities engaging with these tools. Gerke et al²⁴ argue for a systems-based regulatory approach, which includes additional data collection from insurers about reimbursement, legal bodies for liability concerns, and health systems about clinician reactions to AI/ML tools. In the preapproval state, the FDA should also underscore the importance of training and testing algorithms with diverse patient data sets to ensure AI/ML tools do not amplify health inequities.²⁷ There have also been studies that report discrepancies between FDA-approved indications for use and device marketing, with companies claiming AI/ML use in their technology despite not seeking such clearance.^23,32 With the adaptation of new approval methods, the FDA may want to track company announcements to ensure manufacturers are honestly communicating with medical partners.

The push toward incorporating generative AI tools like ChatGPT into health care settings will introduce a new layer of regulatory challenges that the FDA will have to address.^13,30 Technological companies like Oracle and Microsoft are partnering with EHRs to build generative AI tools like note summarizers and speech-to-text software that can be integrated into existing medical record systems like Cerner and Epic.^33,34 Although such tools may improve efficiency, a big concern with generative AI models is hallucinating, or generating outputs that are not based on training data.³⁵ Additional challenges include poor model generalizability between health systems, unethical patient data sourcing, and propagation of pre-existing bias and stereotypes.³⁶ To address these limitations, the FDA may want to build a process for spotting and reporting errors, publishing summaries of the data that was used in the model and disclosing when content was generated by AI for the end-users.^37,38

Since January 2025, AI/ML device developers can find the FDA’s guidance for the development and marketing of such devices using TPLC. While this new regulatory framework will help introduce exciting AI/ML technologies, health systems will require training to integrate these tools into their clinical workflow, and manufacturers will need to establish trust that AI/ML technologies will not affect the quality of care. AI/ML tools are inevitable to the ongoing practice of medicine, and it is critical to ensure that manufacturers, regulators, and health care partners are equipped with the tools and communication systems to ensure the safe and appropriate use of these technologies.

Potential Competing Interests

The authors report no competing interests.