Regulatory responses to medical machine learning

Abstract

Companies and healthcare providers are developing and implementing new applications of medical artificial intelligence, including the artificial intelligence sub-type of medical machine learning (MML). MML is based on the application of machine learning (ML) algorithms to automatically identify patterns and act on medical data to guide clinical decisions. MML poses challenges and raises important questions, including (1) How will regulators evaluate MML-based medical devices to ensure their safety and effectiveness? and (2) What additional MML considerations should be taken into account in the international context? To address these questions, we analyze the current regulatory approaches to MML in the USA and Europe. We then examine international perspectives and broader implications, discussing considerations such as data privacy, exportation, explanation, training set bias, contextual bias, and trade secrecy.

1. INTRODUCTION

The term ‘artificial intelligence’ (AI) has been broadly defined as ‘the science and engineering of making intelligent machines, especially intelligent computer programs’.¹ Within the field of AI, machine learning (ML) can be used to design and train algorithms to learn, identify patterns, and act on data. Depending on their design, ML algorithms can be ‘locked’ or allowed to continuously learn from data to adapt and optimize their performance in real-time.²

AI/ML has found applications in basic biomedical research, translational research, and clinical practice.³ Companies and healthcare providers are currently investing heavily in developing medical grade AI/ML applications (hereinafter medical artificial intelligence—‘MAI’ and medical machine learning—‘MML’). Examples of applications include AI-driven X-ray image analysis,⁴ smartphone apps that detect skin cancer,⁵ and the use of AI-driven monitoring systems to identify elderly patients at risk of falling.⁶ AI is also being applied in biomarker and drug discovery⁷ and recently, the US Food and Drug Administration (FDA) approved the first digital pill, consisting of a drug combined with an ingestible electronic sensor that helps to improve patients’ medication adherence.⁸

These kinds of technologies may increase patient safety and help to control, for example, the use of antibiotics to reduce antimicrobial resistance. Yet, they also raise concerns and challenges with regard to safety, effectiveness, transparency, data sharing, property rights, antitrust, cybersecurity, privacy, and algorithmic bias and discrimination. Hence, regulators, such as the US FDA, the European Medicines Agency (EMA), and national competent authorities for medical devices are working to figure out how to interpret, apply, or modify their existing regulatory frameworks in the MML context. Several changes to the current frameworks are being proposed and discussed around the globe, including regulatory frameworks for AI/ML-based medical device software, privacy, and cybersecurity. In addition, ethical frameworks have increasingly been demanded and developed by different stakeholders.⁹

In the following, we focus on two foundational questions in this narrative: (1) How will regulators evaluate MML-based medical devices to ensure their safety and effectiveness? and (2) What additional MML considerations should be taken into account in the international context?

2. US REGULATORY APPROACH

According to a recent forecast, AI has the potential to create $150 billion in yearly savings for the US healthcare economy in the next 7 years; the AI health market is expected to grow to $6.6 billion by 2021.¹⁰ MML regulation and the path to market are key concerns for AI developers. The FDA has recognized the importance of digital health products generally and AI specifically, releasing a Digital Health Innovation Action Plan in July 2017 that proposed new guidances (some since issued), increased in-house expertise through hiring, and a more flexible approach to approving software products (implemented through the pilot Pre-Cert program).¹¹ Many of the recent developments discussed in this section arise from the Action Plan.

The first step for manufacturers or developers intending to market an MML product is to determine whether the product is intended as a ‘general wellness product’ (consumer-grade) or as a ‘medical device’ (clinical-grade). The recently enacted 21st Century Cures Act contains an important exclusion for software intended ‘for maintaining or encouraging a healthy lifestyle’ that is ‘unrelated to the diagnosis, cure, mitigation, prevention, or treatment of a disease or condition’.¹² Such software does not fall under the medical device definition and FDA does not intend to examine low risk wellness products according to its Guidance ‘General Wellness: Policy for Low Risk Devices’.¹³ Examples include apps that use ML but (1) the intended uses involve claims about sustaining or offering general improvement to functions associated with a general state of health that ‘do not make any reference to diseases or conditions (eg claims to promote weight management, sleep management, physical fitness) or (2) claims that make a reference to diseases or conditions (eg intended uses to promote, track, and/or encourage choices which as part of a healthy lifestyle that “may help to reduce the risk” or “may help living well”’ with certain chronic diseases or conditions, where the claim is generally accepted because the associations are described in peer-reviewed scientific publications or official statements made by healthcare professional organizations). However, if the ML product is labeled, promoted, or used in a manner that meets the definition of ‘device’ in section 201(h) of the Federal Food Drug & Cosmetic (FD&C) Act, it will be regulated by the FDA as a ‘medical device’ and require pre-marketing and post-marketing regulatory controls, including: (1) FDA establishment registration, device listing, and pre-market notification requirements (21 CFR Part 807); (2) labeling requirements (21 CFR Part 801 and 21 CFR 809.10); (3) current good manufacturing practice (CGMP) and quality system (QS) requirements (21 CFR Part 820); and (4) medical device reporting requirements (21 CFR Part 803).

Accordingly, the manufacturer/developer should carefully formulate the intended use, claims, labeling, and associated marketing of the MAI/MML product, since these decisions have significant regulatory consequences. A medical device is defined in 201(h) of the FD&C Act to include ‘an instrument, apparatus, implement, machine […] which is intended for use in the diagnosis of disease or other conditions, or in the cure, mitigation, treatment, or prevention of disease’ that does not achieve its ‘primary intended purposes through chemical action within or on the body of man’ and which is ‘not dependent upon being metabolized for the achievement of’ its primary intended purposes. For example, Apple’s electrocardiogram (ECG) app, intended for use with the Apple’s watch to ‘create, record, store, transfer, and display a single channel electrocardiogram (ECG) similar to a Lead I ECG’,¹⁴ and Apple’s irregular rhythm notification feature, intended to identify and notify the user of episodes of irregular heart rhythms,¹⁵ are both medical devices and subject to FDA regulatory controls even though the apps are ‘not intended to interpret or take clinical action’ and the resulting medical device ‘is not intended to provide a diagnosis’.¹⁶

The FDA classifies medical devices in three classes based on their risk profile, intended use, indications for use, technological characteristics, and the regulatory controls necessary to provide a reasonable assurance of safety and effectiveness.¹⁷ The class of the device determines the type of pre-marketing submission or application required for marketing. The FDA reviews medical devices through one of three pre-market pathways: (1) the 510(k) pre-market notification/clearance, (2) De Novo classification, and (3) pre-market approval (PMA) pathways. Low-risk devices (Class I) are subject to general controls, but most are exempted from 510(k) pre-market notification. Moderate-risk devices (Class II) are subject to general controls, special controls, and typically require a 510(k) pre-market notification (unless the specific product code is 510(k) Exempt) or De Novo classification process. High-risk devices (Class III) require general controls and a PMA application.¹⁸ As an illustrative example, the FDA recently reviewed the Apple’s apps mentioned above through the De Novo classification process and classified them as Class II medical devices.¹⁹ General controls apply to all medical devices (unless exempted by regulations) and include FDA establishment registration and device listing, pre-market notification (eg FDA 510 k submission or De Novo classification), and records and reports on devices (eg quality management system, device tracking, unique device identification system, adverse event reporting, and reports of removals and corrections). Special controls are regulatory requirements for class II devices (ie devices for which general controls alone are insufficient to provide reasonable assurance of the safety and effectiveness of the device). These include (1) performance standards, (2) pre-market clinical data requirements, (3) post-market surveillance, (4) special labeling requirements, (5) patient registries, and (6) guidelines.

Depending on the specific application, MML algorithms can be an integral part of the hardware of a medical device and necessary to achieve its intended use (Software in a Medical Device) or standalone software intended to be used for medical purposes without being part of a hardware medical device (software as a medical device or ‘SaMD’).²⁰ Software apps designed to be used on consumer-grade devices such as an Apple’s watch are examples of SaMDs since the underlying hardware is not itself a medical device. Recognizing the regulatory challenges of software as standalone medical devices, the International Medical Device Regulators Forum (IMDRF)‘s SaMD working group has authored a framework, principles, and guidance for SaMD. The IMDRF framework for risk categorization of SaMD has four categories based on (1) the significance of the information provided by the SaMD to a healthcare decision (inform clinical management, drive clinical management, treat, or diagnose) and (2) the state of healthcare situation or condition (non-serious, serious, and critical).²¹ For example, as shown in Table 1, a SaMD that merely informs clinical management of a non-serious condition will be categorized as Level I (lowest risk), while a SaMD intended to treat or diagnose a critical condition as a Level IV (highest risk).

Table 1.

SaMD risk characterization matrix developed by the IMDRF–SaMD Working Group (framework for risk categorization). It is expected that MML SaMDs will be allowed to continually change by ‘adaptive learning’ for Level I categories (eg informing clinical management of non-serious conditions). This is considered a change in performance due to learning from data and would not require a further review by FDA. However, a change in the SaMD intended use (eg from ‘informing clinical management’ of a non-serious condition to ‘diagnosing’ of a serious or critical condition) would likely require a submission to the FDA for appropriate review (eg 510 k, De Novo, PMA).

State of healthcare situation or condition	Significance of information provided by SaMD to healthcare decision
	Diagnose or treat	Drive clinical management	Inform clinical management
Critical	IV	III	II
Serious	III	II	I
Non-serious	II	I	I

The MAI/MML devices the FDA has reviewed to date have ‘locked’ algorithms that do not automatically change over time as new data are collected. In April 2019, the FDA published a discussion paper that proposes a new regulatory approach that would allow AI/ML-SaMD to continually improve (ie to learn, adapt, and optimize performance in real-time) while providing effective safeguards.²² Consistent with the IMDRF definition, in the proposed FDA, AI/ML-SaMD regulatory approach ‘SaMD’ is defined as ‘software intended to be used for one or more medical purposes that perform these purposes without being part of a hardware medical device’.²³ Medical purposes are ‘purposes that are intended to treat, diagnose, cure, mitigate, or prevent disease or other conditions’.²⁴ The proposed framework analyzes the types of modifications (eg changes to performance, inputs, or intended uses) in the context of a total product lifecycle regulatory approach to MAI/MML-SaMD based on QS and good ML practices. This framework does not apply to ‘non-device’ software functions.²⁵ The FDA plans to issue draft guidance based on the input received from the discussion paper but admits that additional statutory authority may be needed to fully implement the proposed framework.²⁶

3. EUROPEAN UNION REGULATORY APPROACH

Similar to the FDA, the EMA has also acknowledged the significance of digital health products and their interface with Big Data, MML, and MAI. Although many activities were put on halt due to the withdrawal of the United Kingdom from the European Union (EU) and the resulting move of the EMA headquarters, the EMA and the Heads of Medicines Agencies (HMA) have been working intensively on understanding the acceptability of big data-derived evidence in support of the evaluation and supervision of medicines. Their first Joint Big Data Taskforce report, published in February 2019, set out a number of steps towards regulatory acceptance of big data, ranging from data standardization and the need to link genomics data with clinical outcomes to the pressing demand for timely access to data that is representative of the European population.²⁷ Most recently, in January 2020, the HMA–EMA Joint Big Data Taskforce published their second report, which proposes 10 priority actions for the European medicines regulatory network to evolve its approach to data use and evidence generation to best use big data to facilitate innovation and public health.²⁸ Moreover, on 19 February 2020, the European Commission published a White Paper to promote a European ecosystem of excellence and trust in AI (1), a data strategy communication (2) and a report on the safety and liability aspects of AI.²⁹ However, the focus has been laid so far on the development of enabling infrastructures, rather than specific guidelines on the use of MML and MAI in drug development, medical procedures, and devices.

As a general requirement, all medical devices and in vitro diagnostic medical devices, including MML devices, must comply with the CE marking requirements under the relevant EU regulatory frameworks to be lawfully marketed within Europe.³⁰ On 25 May 2017, two major regulatory changes entered simultaneously into force, which are highly relevant for medical device manufacturers: EU Regulation 2017/745 on medical devices (MDR)³¹ and EU Regulation 2017/746 on in vitro diagnostic medical devices (IVDR).³² The MDR will repeal the Directive 93/42/EEC concerning medical devices (MDD)³³ and the Directive 90/385/EEC on active implantable medical devices (AIMD),³⁴ whereas the IVDR will replace the Directive 98/79/EC on in vitro diagnostic medical devices (IVDD)³⁵ and Commission Decision 2010/227/EU.^36,37

Since neither the MDR nor the IVDR includes a ‘grandfathering’ provision, all currently marketed medical devices and in vitro diagnostic medical devices must be recertified in conformity with the new requirements.³⁸ The new requirements must generally be fulfilled by 26 May 2020 for the MDR and 26 May 2022 for the IVDR.³⁹

For medical AI developers, the new MDR is particularly relevant. The MDR continues to divide devices into classes, taking into account risks associated with their manufacture and technical design.⁴⁰ There are four classes of medical devices of increasing risk: Class I, Class IIa, Class IIb, and Class III.⁴¹

The MDR changes the EU regulatory landscape for MML devices by introducing, inter alia, heightened requirements for post-market surveillance⁴² and medical device traceability.⁴³ Most significantly, ‘medical device’ is defined more broadly and now includes software for the ‘prediction’ and ‘prognosis’ of disease.⁴⁴ Under the MDR, some currently marketed devices will shift to higher classes. In particular, the MDR introduces a new classification rule for software. For example, Section 6.3 of Chapter III of Annex VIII (Rule 11) of the MDR provides that ‘Software intended to provide information which is used to take decisions with diagnosis or therapeutic purposes is classified as class IIa, except if such decisions have an impact that may cause: —death or an irreversible deterioration of a person’s state of health, in which case it is in class III; or, —a serious deterioration of a person’s state of health or a surgical intervention, in which case it is classified as class IIb. Software intended to monitor physiological processes is classified as class IIa, except if it is intended for monitoring of vital physiological parameters, where the nature of variations of those parameters is such that it could result in immediate danger to the patient, in which case it is classified as class IIb. All other software is classified as class I’. Notably, SaMD intended to merely ‘monitor physiological processes’ will be classified Class IIa or higher. Accordingly, it is expected that most MML-based medical devices will be classified as Class IIa (even if the monitoring information is not intended for diagnosis or therapeutic purposes) or Class IIb if the MML SaMD monitors vital physiological parameters (eg heart rate, blood pressure, and respiration). This implies that most MML SaMD will be subjected to a conformity assessment, including the approval of the technical documentation by a notified body. The new Rule 11 surprisingly does not mention or refer to the new terminology ‘prediction’ and ‘prognosis’. This raises the question if this omission is a mere oversight, or if the concept of prediction and prognosis of a disease is embedded into the following formulation: ‘Software intended to provide information […] used to take decisions with diagnosis or therapeutic purposes’.⁴⁵

In contrast to FDA in the USA, EMA is not the primary regulator for medical devices. Instead, medical devices are regulated by the Member States who can designate independent accredited ‘notified bodies’ to conduct the required conformity assessments and by national competent authorities appointed by the Member States that are, inter alia, responsible for monitoring notified bodies.⁴⁶ In the case of low-risk devices (ie Class I), the manufacturers are able to ‘self-certify’ the conformity assessment without or with limited involvement of a notified body (Table 2).⁴⁷

Table 2.

A comparison of basic aspects between the USA and EU regulatory approach to MML in selected key domains.

Regulatory approaches to MAI/MML devices	USA	Europe
Medical device definition	‘An instrument, apparatus, implement, machine […] which is intended for use in the diagnosis of disease or other conditions, or in the cure, mitigation, treatment, or prevention of disease’ that does not achieve its ‘primary intended purposes through chemical action within or on the body of man’ and which is ‘not dependent upon being metabolized for the achievement of’ its primary intended purposes (201(h) of the FD&C Act)	‘Any instrument, apparatus, appliance, software, implant, reagent, material, or other article’ intended by the manufacturer to be used, alone or in combination, for human beings for one or more of the following specific medical purposes: Diagnosis, prevention, monitoring, prediction, prognosis, treatment, or alleviation of disease Diagnosis, monitoring, treatment, alleviation of, or compensation for, an injury or disability Investigation, replacement, or modification of the anatomy or of a physiological or pathological process or state Providing information by means of in vitro examination of specimens derived from the human body, including organ, blood, and tissue donations,
		and which does not achieve its principal intended action by pharmacological, immunological, or metabolic means, in or on the human body, but which may be assisted in its function by such means. […]” (MDR, Art. 2(1))
Classification of medical devices	Three classes based on their risk profile, intended use, indications for use, technological characteristics, and the regulatory controls necessary to provide a reasonable assurance of safety and effectiveness:	Four classes of medical devices, taking into account risks associated with their manufacture and technical design:
	Class I, Class II, and Class III	Class I, Class IIa, Class IIb, and Class III
Distinctions between diagnostic, monitoring, and predictive diagnostic software	• Software in a medical device	‘Software’, which drives a device or influences the use of a device, shall fall within the same class as the device
	➔ Software used to ‘drive or control’ the medical device hardware	If the software is independent of any other device, it shall be classified in its own right” (MDR, Section 3.3. of Chapter II of Annex VIII)
	➔ Software required by a hardware medical device to perform the hardware’s medical device intended use, even if sold separately from the hardware medical device	‘Software’ intended to provide information which is used to take decisions with diagnosis or therapeutic purposes is classified as Class IIa, except if such decisions have an impact that may cause:
	• Software as a medical device	—death or an irreversible deterioration of a person’s state of health, in which case it is in Class III; or,
	➔ software intended to be used for one or more medical purposes that perform these purposes without being part of a hardware medical device	—a serious deterioration of a person’s state of health or a surgical intervention, in which case it is classified as Class IIb
	• Not all software is classified as a ‘medical device’ under the FD&C Act. Non-device software functions include:	Software intended to monitor physiological processes is classified as Class IIa, except if it is intended for monitoring of vital physiological parameters, where the nature of variations of those parameters is such that it could result in immediate danger to the patient, in which case it is classified as Class IIb
	a) Software functions that are intended ‘for administrative support of a healthcare facility’	All other software is classified as Class I” (MDR, Section 6.3 of Chapter III of Annex VIII (Rule 11))
	b) Software function that are intended ‘for maintaining or encouraging a healthy lifestyle’	• Rule 11 MDR does not explicitly mention or refer to the new terminology ‘prediction’ and ‘prognosis’ of disease in Art. 2
	c) Software functions that are intended ‘to serve as electronic patient records’	• However, the concept of ‘prediction’ and ‘prognosis’ of a disease might be embedded into the formulation: ‘Software intended to provide information […] used to take decisions with diagnosis or therapeutic purposes’
	d) Software function that are intended ‘for transferring, storing, converting formats, or displaying clinical laboratory test or other device data and results’, and	• Most MML-based medical devices will be classified as Class IIa (even if the monitoring is not intended for diagnosis or therapeutic purposes) or Class IIb if the MML SaMD monitors vital physiological parameters (eg heart rate, blood pressure respiration)
	e) Certain clinical decision support software functions (FD&C Act, Sec. 520(o))
Post-marketing surveillance (PMS)	• CGMP and QS requirements (21 CFR Part 820)	Post-market surveillance:
	• Medical device reporting requirements (21 CFR Part 803)	‘All activities carried out by manufacturers in cooperation with other economic operators to institute and keep up to date a systematic procedure to proactively collect and review experience gained from devices they place on the market, make available on the market or put into service for the purpose of identifying any need to immediately apply any necessary corrective or preventive actions’ (Art. 2 (60) MDR)
	• Recalls, corrections, and removal (21 CFR 7, 21 CFR 806, 21 CFR 810)	• Post-market surveillance system (MDR, Art. 83)
	• Medical device tracking	➔ For all devices
	➔ For certain Class II and Class III devices (21 CFR Part 821)	• Post-market surveillance plan (MDR, Art. 84)
	• Post-market surveillance studies	• Post-market surveillance report (MDR, Art. 85)
	➔ Certain Class II and Class III devices (FD&C Act, Sec. 522)	➔ For Class I devices
	• Post-approval studies	• Periodic safety update reports (PSUR) (MDR, Art. 86 MDR)
	➔ For certain devices	➔ For Class IIa, Class IIb, and Class III devices
	• Third-party inspections
Implementation of the relevant regulations and assessment of medical devices	FDA	• EMA is not the primary regulator for medical devices
		• Medical devices are regulated by the Member States and by national competent authorities appointed by the Member States
		• Member States can designate independent accredited ‘notified bodies’ to conduct the required conformity assessments
		The national competent authorities are responsible for monitoring notified bodies
MAI/MML specific initiatives and regulatory guidelines	FDA’s discussion paper on AI/ML-based SaMD48	Lack of specific guidelines on the use of MAI/MML devices. The focus is laid on the development of enabling infrastructures. This might change in the future when the MDR becomes effective on 26 May 2020

4. OTHER REGULATORY PERSPECTIVES AND BROADER IMPLICATIONS

While there are country-specific differences with regards to the particular pre-market and post-market review procedures, most developed jurisdictions regulate medical devices similarly to the US or European models.

In China, the National Medical Products Administration (NMPA), formerly the China Food and Drug Administration, is responsible for pharmaceuticals and medical device regulation.⁴⁹ NMPA also classifies medical devices in three categories and conducts pre-market approval and PMA. The increased medical device regulatory convergence across these jurisdictions is due, in part, to the work of the IMDRF and the adoption of ISO standards for medical devices (eg ISO 13485:2016-Medical Devices-Quality Management Systems-Requirements for Regulatory Purposes). The IMDRF is a voluntary group of medical device regulators who is continuing the work of the Global Harmonization Task Force (GHTF) on medical devices with the overall objective of accelerating international medical device regulatory harmonization and convergence.⁵⁰ It includes representatives from the medical device regulatory authorities of Australia, Brazil, Canada, China, EU, Japan, USA, and the World Health Organization.⁵¹ As an example, the IMDRF has been instrumental in creating the risk categorization and overall regulatory framework for SaMDs.⁵²

Low- and middle-income countries (LMICs) face similar challenges in regulatory design to the USA and EU, though often with fewer regulatory resources.⁵³ Some LMICs address such resource limitations by recognizing approval from the FDA or CE marking in Europe as a substitute for or supplement to their own regulatory review, and such a pattern might be expected to continue with MML. AI governance frameworks are also under development in multiple LMICs. For instance, India is developing a governance framework, but both focus on AI generally rather than on medical AI specifically.⁵⁴ International efforts are also spreading; in 2019, 42 countries, including several LMICs, signed the OECD Principles on AI, which endorsed inclusivity, diversity, transparency, and security, among other aims,⁵⁵ though again not focused on medical applications specifically.

LMICs also face some unique issues. While recognizing that each regulatory setting is distinct, we highlight two clusters of issues that play out slightly differently in this setting.

4.1. Data privacy, exportation, exploitation, and equitable access

Countries differ substantially when it comes to data privacy regulation. A 2018 survey found that ‘101 countries have enacted general federal personal data protection laws, 12 countries have pending legislation, and 82 countries have neither enacted nor pending legislation’.⁵⁶ Of course, the picture becomes more complicated when it comes to health, because some countries may sectorally protect health care data without a general data protection regime (such as the US HIPAA Privacy Rule),⁵⁷ while others may both offer general data protection and also special additional rules for health privacy (such as the EU’s GDPR).⁵⁸ Furthermore, inferences about health can increasingly be made from non-healthcare data, and used for many different purposes, such that many forms of personal data may arguably become ‘health data’.⁵⁹

Because privacy protections differ,⁶⁰ companies seeking to develop MAI/MML may face incentives to use data from LMICs with less stringent data protection, and national governments may have strong incentives to sell or license patient data to big players in the AI space—a phenomenon some refer to as ‘data colonization’. Some countries, such as China, place stringent limitations on export of population data, but many do not. When such deals result in benefits to the patients whose data are used—MML usable in the country where the data originate, or proceed used to finance public health or healthcare improvements—the deals can be appropriate. But as we have seen with other forms of transnational medical practices, such as drug development or medical migration, value created often does not flow back to its LMIC sources. As noted in the OECD principles on AI, all players in the development of medical AI should respect principles of equity, especially when using data from LMICs.

Data exportation could also result in data insecurity. Release of data from LMICs should include verification both of strong cybersecurity practices on the recipient’s part and of governance by well-established principles of sharing (such as the ‘Fair Information Practice Principles’).⁶¹ These could include limitations on the amount of data sharing, use only for pre-specified purposes, provisions for data destruction after use, and transparency about the contours of such data-sharing. Finally, because breaches of health and other data have, unfortunately, become a fact of life, before any sharing takes places LMICs should institute robust antidiscrimination and other protections for their patients should their data be released.

4.2. Training set bias, contextual bias, and trade secrecy

An additional problem, in some tension with the first, is bias. As multiple real-world examples have shown, MAI can reach disturbingly inaccurate results depending on the subject’s age or disabilities, ethnic origins,⁶² skin color⁶³, or gender.⁶⁴ False diagnoses or improper treatment recommendations present a real threat to patient health. Such biases can result from non-representative data, data-reflecting biases in underlying care, biases in the teams and development of MML, or many other interrelated sources. To take an example of the first, cardiomyopathy genetic tests have been found to be better at identifying pathogenic variants in white patients than patients of other ethnicities because of non-representative training data.⁶⁵ MML trained on such data might provide poor results for non-white patients unless retrained on more representative data.

When it comes to the use of MML developed in high-income settings for LMICs, there is an additional risk of ‘contextual’ bias: algorithms trained to function in elite hospitals with experts applying the technology may not necessarily recommend appropriate, safe, and cost-effective treatments in low-resource settings. With contextual bias, even an algorithm trained on representatives’ populations (itself a feat) might make improper recommendations. For example, in a leading cancer center it might be best to recommend to a patient a powerful chemotherapeutic cocktail that requires careful monitoring by an experienced oncology team to manage potentially deadly side effects, but that recommendation could be more likely to kill than to save the patient in an LMIC (or rural, or non-specialist) hospital.⁶⁶ In many cases, contextual bias will be far more subtle. Contextual bias will be harder to detect in opaque MML since the care-provider may not know the contextual variations making the algorithm’s recommendation inappropriate for her setting. Such ignorance may be compounded by secrecy among makers of MML.⁶⁷ Software companies in healthcare may often have good reasons to conceal their in-house testing procedures and other data for competitive advantages or to refer to privacy protection.

Solutions are not easy here, especially if one expects that much MML will at least initially be developed for high-income countries. More transparency about training context and algorithmic limits will help, though competition concerns may hinder that effort. However, while knowing that a valuable MML may not work well in one’s country is better than not knowing, that knowledge does not help one’s countries patients reap the MML’s benefits. Longer-term solutions must incentivize MML makers to design for LMIC needs. This may involve retraining on local population data but also with local context in mind, or helping to build AI-development expertise for healthcare in LMICs to begin with. But, it is unclear whether the LMIC markets are lucrative enough to naturally push developers towards these options, or whether more direct carrots (or sticks) are needed, not just from the LMICs themselves but also from high-income countries.

5. CONCLUDING REMARKS

Despite recent regulatory developments in the USA and Europe, there are currently no harmonized standards or laws that specifically regulate the use of MML-enabled medical devices. In general, MML devices are currently reviewed to ensure compliance with the regulatory requirements for all medical devices. The current regulatory frameworks lag behind the use of MML, leading to regulatory uncertainty with risks for manufacturers. MML manufacturers must spend substantial effort and resources to understand how regulations apply in the particular context of MML devices. This includes working closely with the responsible regulatory authorities to achieve consistency in interpretation. Positive developments in this specific area include FDA’s recent discussion paper on continuously learning MML devices and IMDRF development of foundational guidance intended to establish common SaMD definitions and develop harmonized approaches for appropriate SaMD regulatory controls.

Medical device authorities must manage and administer an increasingly complicated regulatory system. Accordingly, they are in need of experts with technical competency to review the safety and effectiveness of a new generation of MML-enabled devices while taking into account complex considerations such as training set bias, contextual bias, and how these technologies will evolve across complex value chains and life cycles. These considerations are important to be taken into account by the IMDRF as they continue the work of the GHTF to accelerate medical device regulatory harmonization and convergence in order to facilitate the global deployment of MML innovations while ensuring their safety and effectiveness.

From a global policy perspective, a sustainable regulatory system of AI applications in the health and life sciences should ensure that such new uses benefit the global population by promoting inclusive growth, sustainable development, and well-being, as noted in the recently adopted OECD Principles on AI.⁶⁸ In accordance with these principles, it will be important to embed respect for the rule of law, human rights, diversity, and fairness, as well as other societal and democratic values in the design of AI systems and devices.⁶⁹ To implement and protect these values, AI systems should also include appropriate safeguards, such as enabling human intervention where necessary.⁷⁰ This also means that MML systems should incorporate a reasonable amount of transparency and responsible disclosure mechanism that allow medical practitioners, patients, and their relatives to understand AI-based outcomes.⁷¹

Finally, to promote responsible stewardship of trustworthy and secure⁷² MML, as well as a fair transition from traditional healthcare, it will be vital that governments work together across border and sectors. Such international collaboration should also extend to the education of medical practitioners, healthcare providers, patients, and other stakeholders to increase their understanding of AI/ML⁷³ and the associated advantages, risks, and limitations of MML.