Using Existing Regulatory Frameworks to Apply Effective Design Controls to AI/ML in Medical Devices

Current Regulatory Landscape

The emergence of artificial intelligence (AI)/machine learning (ML)-enabled medical devices has the potential to revolutionize diagnostic and treatment technology across a broad spectrum of intended uses. A conundrum faced by all medical device manufacturers, however, is the significant lag in regulatory legislation or guidance for medical devices and systems that leverage AI/ML.

According to the International Medical Device Regulators Forum (IMDRF), AI is “a branch of computer science, statistics, and engineering that uses algorithms or models to perform tasks and exhibit behaviors such as learning, making decisions and making predictions”¹ Machine learning (ML), states IMDRF, is a “subset of AI [that] allows computer algorithms to learn through data, without being explicitly programmed, to perform a task.”

Hundreds of papers have been written by scholars and industry experts proposing modified regulatory frameworks to accommodate AI/ML-enabled medical devices, and regulatory authorities such as the Food and Drug Administration (FDA), UK Medicines & Healthcare products Regulatory Agency, and Health Canada have proposed regulatory considerations for public review. At the time this article was written, however, only the Saudi Food & Drug Authority had published an enforceable guidance establishing regulatory requirements for AI/ML in medical devices, while the Chinese National Medical Products Administration had also published an AI software classification guideline.

With this regulatory desert as backdrop, both regulators and manufacturers are forced to utilize existing regulations, standards, and guidance in the design, development, testing, release, distribution, and maintenance of AI/ML-enabled medical devices. Two of the most noticeable impacts of this reliance on older and more generalized regulatory guidance include (1) the heavy use of serialized “waterfall” thinking, which results in overly extensive mapping of iterative algorithm development methods to regulatory requirements, and (2) strict controls on change management, which limits deployed AI/ML-enabled medical devices to locked algorithms that cannot dynamically and autonomously adapt to new data in the field.

Two Considerations

To effectively leverage existing medical device regulations, standards, and guidance, we recommend manufacturers pay special attention to the categorization of an AI/ML-enabled system to ensure compliance activities are appropriately targeted. In the U.S., medical device data systems and clinical decision support software, for example, are subject to specific guidance that may either remove the medical device classification from a system or provide for the exercise of “enforcement discretion” by the FDA, without holding the manufacturer accountable for compliance to certain regulatory requirements. That said, manufacturers operating internationally must consider the regulatory oversight and legal requirements of all relevant regions, which likely are dissimilar.

A second consideration, and the focus of the remainder of this article, is the effective use of current design controls, software in a medical device (SiMD)/software as a medical device (SaMD), and risk management guidance when adapting the software development life cycle and general quality system to AI/ML-enabled medical devices. The model and case study described make specific use of the latest revisions of IEC 62304:2006/AMD 1:2015 (Medical device software—Software life cycle processes —Amendment 1),² IEC 82304-1:2016 (Health software—Part 1: General requirements for product safety),³ ISO 14971:2019 (Medical devices—Application of risk management to medical devices),⁴ IEC/TR 80002-1:2009 (Medical device software—Part 1: Guidance on the application of ISO 14971 to medical device software),⁵ and AAMI technical information report TIR45:2012 (Guidance on the use of AGILE practices in the development of medical device software),⁶ as well as SaMD guidance documents from various regulatory authorities (e.g., Australia's Therapeutic Goods Administration, Health Canada, FDA) and IMDRF.

Design Controls in AI/ML-enabled SiMD/SaMD Development

AI/ML-enabled medical devices are realized through a software representation of an AI/ML life cycle model. Typically, software development is accomplished using agile software development methodologies. That said, the design control requirements identified in 21 CFR 820.30 still must be met.7 Existing guidance (e.g., AAMI TIR45) can help here but does not answer all the relevant compliance questions. Further, AAMI TIR45 principally is focused on FDA design control regulations (21 CFR 820.30) and therefore may be of more limited utility in foreign regulatory contexts.

Even with the use of agile development approaches, elements of the development process still must use a serialized waterfall approach to meet regulatory requirements. Initially, user needs, clinical application needs, and business needs translated into system requirements are established and placed under change control prior to the development of user stories and the execution of “sprints” (assuming scrum is among the utilized agile methods). These initial records still may change but likely are modified less often than the lower-level requirements, design, and associated code.

Agile techniques are used to iteratively generate software requirements, architecture, and detailed design, as part of the “definition of done” within user stories. The sprints (and associated reviews) can feedback to the high-level software requirements to update them through the system backlog; a testable (viable) set of code is developed concurrent with a software design specification used as the basis for testing (unit verification). Ongoing development of user stories and associated definition-of-done artifacts eventually will represent the entire software system in the device (or as the device), resulting in a complete set of software requirements, architecture, and design and associated unit-, integration-, and system-level verification testing. The results of verification testing are also traceable back through the design specifications to the system-level requirements, as depicted in Figure 1.

Figure 1.

Example of iterative software development for AI/ML-enabled medical device software.

Once a given iteration results in a minimally viable product, training of the model can begin. Alternatively, the model can be trained to an extent outside of the device software, though training still will need to be performed within the confines of the market-intended software. A data set(s) adequately representing the intended use population and containing all information (variables) necessary for the model to operate as intended must be available for training and testing purposes. The data set(s) must be controlled to ensure there are no purposeful or inadvertent changes or mixing of training and testing data. Through the training process, there will be changes to how the variables influence the results, while changes to the software code representing the model may also be necessary.

Change control to the software requirements, architecture, and design must be followed to ensure compliance to design controls requirements. Additional training or testing data may be needed, so plan for this—and ensure that records of the training are retained as evidence of design verification within the device's design history file (DHF).

After the model has been optimized, the software code representing the algorithm should be locked (at least until there is guidance on what is acceptable for self-adapting models in the AI/ML space). A “locked algorithm” is one that provides the same result each time the same input is applied to it and does not change with use.

At this point, the process exits the use of agile methods and once again enters a set of serialized waterfall activities for system-level validation, including confirmation testing of the model with an unused and representative data set. The risk management file also must be reviewed to determine if additional critical clinical decision points or scenarios exist that introduce new hazards and/or hazardous situations requiring risk mitigations and subsequent verification. Because design validation is intended to ensure the device operates safely and effectively in its intended use environment, a measure of “truth” will be required during the validation testing. In many cases, this is testing against the standard of care, which often is a clinician assessment (e.g., radiologist looking for evidence of lesions within images).

Managing change control of product design records (i.e., DHF, technical file) is important for all medical devices but critical for design processes incorporating agile development methodologies. The most efficient and effective tools to support agile approaches are electronic object-driven configuration and document management systems. These systems allow the manufacturer to generate regulatory-compliant records at any point in the development process and meet the needs of highly iterative changes occurring during sprints. Consolidation of configuration items/objects into approved DHF documents are finalized, at a minimum, prior to the execution of design validation. Confirmation that the DHF (including design validation records) is complete occurs just prior to design transfer, which can initiate following successful design validation.

Fictitious Case Study

A medical device manufacturer with an existing FDA-compliant quality management system decides to develop an AI/ML-based algorithm using information gathered from peer-reviewed research. The research indicates that cardiovascular health can be assessed through use of the following variables: age, weight, heart rate, and pulse wave velocity. The initial high-level system requirements and risk analysis are established for the algorithm to be effective among males and females and developed in a manner that meets the requirements of the FDA's quality system regulation (21 CFR 820.30), ISO 13485:2016 (Medical devices—Quality management systems—Requirements for regulatory purposes),⁸ and ISO 14971.

An agile scrum software development process is used to iteratively convert the algorithm described in research to executable software (i.e., using sprints to iteratively develop software requirements, software architectural design, software detailed design, and software source and object code) and is established compliant with IEC 82304-1, IEC 62304, and AAMI TIR45. The company uses AI/ML with an appropriate data set to optimize the variable weighting in the algorithm. Initial software system-level design verification testing indicates that the area under the receiver operating characteristic is too low to meet effectiveness objectives. Additional research indicates that incorporating adiposity, estimated by hip circumference and height, can improve the algorithm.

The development team updates the software system requirements and the software's detailed design specifications, performs an impact analysis against the remaining DHF documentation and software code, which leads to software source code updates, retraining, and design verification testing at the unit, integration, and software system levels. The results indicate that adequate effectiveness (as measured through predetermined acceptance criteria) can only be achieved by reducing claims to apply specifically to men and women between the ages of 20 and 70 years. This results in a change to the system requirements that requires an update to the risk analysis, downstream requirements, and design documents. The algorithm is updated, retrained, and verified at the unit, integration, and software system levels.

Design verification testing indicates that effectiveness levels are possible (i.e., meet acceptance criteria). The development team then incorporates the algorithm software into the final intended system: a SaMD for use on a mobile platform. Human factors testing of the full SaMD system during final system-level design verification ensures usability risk mitigations are identified and updated in the product safety risk analysis. The team again updates the software system requirements and downstream documents. During design validation, the team performs human factors testing to ensure the risk mitigations are successful. Final design validation also includes a mixture of existing real-world data and prospectively collected data from a company-sponsored validation study.

The results of analysis of the combined data sets indicate that the desired claims can be supported. The company uses its electronic document and configuration management systems to finalize the DHF (including the device master record and risk management file) and compile the documentation required for a submission to the FDA. After FDA grants marketing authorization, the company transfers the software to the go-live site, where end users can access and download the software application for use.

Conclusion

Manufacturers of AI/ML-enabled medical devices are faced with the dilemma of current regulatory legislation and guidance lagging innovation in this space. The guidance described in this article can aid manufacturers in one approach to effectively leveraging existing medical device regulations, standards, and guidance. Other approaches may be possible, provided that the intent of the design control regulations, including paying close attention to the categorization of an AI/ML-enabled system to ensure compliance activities, are appropriately met and can be demonstrated.