Clinical Evaluation of Artificial Intelligence-Enabled Interventions

Abstract

Artificial intelligence (AI) health technologies are increasingly available for use in real-world care. This emerging opportunity is accompanied by a need for decision makers and practitioners across healthcare systems to evaluate the safety and effectiveness of these interventions against the needs of their own setting. To meet this need, high-quality evidence regarding AI-enabled interventions must be made available, and decision makers in varying roles and settings must be empowered to evaluate that evidence within the context in which they work. This article summarizes good practices across four stages of evidence generation for AI health technologies: study design, study conduct, study reporting, and study appraisal.

Clinical studies are how researchers provide evidence for healthcare decision makers to evaluate and prioritize various interventions. These decision makers include regulators, payers, and other healthcare leaders; all have different scopes of influence and interest, but each has impactful decisions to make.¹ The quality of these decisions depends on four critical stages of evidence generation and evaluation: study design, study conduct, study reporting, and study appraisal. A limitation of just one of these four stages compromises the evidence base it offers for intervention evaluation (Fig. 1). End users of an intervention also benefit from understanding this process. Their understanding of the evidence underpinning the scope and limitations of an intervention will help appropriately develop users’ trust and support its safe and effective use.

Figure 1.

A schematic of the process of intervention evaluation through four stages of research design, conduct, reporting, and appraisal. The design of clinical research establishes the maximum potential value of evidence that can be offered, with each subsequent step acting as sequential and modifiable filters on the value ultimately delivered to healthcare practice. In the scenario illustrated in the figure, it is the reporting of the research that limits the evidence it generates to inform practice, but it could be any combination of the four stages.

The surge of interest in artificial intelligence (AI) and a desire to accelerate its perceived benefits to struggling healthcare systems has put pressure on our traditional models of evidence generation and evaluation. Although the next decade is likely to see a range of different approaches to this, the principles of evidence generation and evaluation remain. The information we have on the performance and safety of an intervention depends on the quality of the studies undertaken (their design, conduct, and reporting). The quality of our decision relating to that intervention then depends on our ability to appraise those studies and our wider understanding of the sociotechnical context into which the intervention is going to deploy.

In this article, we focus on AI-enabled interventions using machine learning (ML). This focus was chosen, as ML technologies (particularly the deep-learning subgroup of ML technologies) are responsible for the sustained interest and investment in AI-enabled interventions and the additional considerations recommended for their evaluation.^2–8 These considerations stem from characteristics of AI that may challenge regulators and other evaluators. They include (1) the capacity of AI to “learn” associations in new data (which may be spurious), (2) low explainability of how an AI model arrived at its outputs, (3) high dependency of AI performance on training context (such as population and setting), and (4) assignment of AI to use cases with a relatively high degree of autonomy and clinical risk (such as diagnosis).⁹ Ideally, researchers, decision makers, and end users would feel confident in accommodating these considerations in an efficient and effective evidence generation and appraisal process for AI-enabled interventions. The reality is that contemporary oversight of best practices in the evolving landscape of AI-enabled intervention evaluation can be challenging to maintain. This limits key stakeholders’ understanding and ultimately the rate and scale of responsible AI innovation in health care.

This commentary seeks to address this challenge by summarizing contemporary guidance and best practices on evaluating AI health technologies. It draws together resources for researchers, decision makers, and potential end users to support their different roles in the generation and clinical evaluation of evidence for AI-enabled interventions.

Designing and Conducting Clinical Studies

Studies of AI health technologies should aim to generate evidence that justifies and directs investment in the next step of an intervention along the translational pathway. The selection of an appropriate study design therefore depends upon how far a specific AI technology is on that translational pathway. This is a large part of the reason why the exponential growth of clinical studies of AI is largely composed of preclinical studies and small-scale prospective clinical studies (equivalent to early-phase clinical trials).¹⁰ However, whether considering a pharmacotherapy, a physical medical device, or an AI technology, larger scale interventional clinical trials should ultimately inform decisions to implement healthcare interventions into real-world care. Estimating real-world effectiveness from preclinical performance is often challenging but is particularly difficult with AI-enabled interventions. They are complex interventions that can have unpredictable impacts on a healthcare pathway, such as underperformance for a specific subpopulation.^11,12 Interventional studies provide evidence beyond simple technology performance (e.g., diagnostic accuracy) and can provide vital information about the actual effect on the patient and other downstream consequences. The randomized controlled trial (RCT) design is held up as the benchmark for evidence generation due to its use of random allocation to tackle the major bias of unequal allocation between interventional and control groups. To maximize their value, RCTs should be designed to reflect the intended real-world application as closely as possible.

In contrast to the exponential growth of “early-phase” clinical studies of AI health interventions, the number of larger scale clinical trials of AI health technologies (phase 3 equivalents) is still small.^10,13 This scarcity of published late-stage study designs is not purely a rational reflection of the translational stage of AI technologies, as they number far less than the number of AI-enabled medical devices that have been granted regulatory approval for clinical use.^13,14 Two factors may account for this. First, there may be evidence available to regulators that is not in the public domain. Such publication is not an obligation for AI manufacturers making submissions to regulators. Publishing also requires significant resource allocation from manufacturers and may even risk their intellectual property. Such failures to publicly report studies are unhelpful, though, and manufacturers should be encouraged to share results openly to support better evaluation decisions across healthcare systems. Second, it appears that many regulatory approvals for AI-enabled interventions are based on non-interventional studies alone.¹⁵ This may benefit AI technology manufacturers who avoid the costs associated with large-scale interventional trials but does not support decision makers evaluating AI-enabled interventions for patient and service benefit. It is striking that, despite the huge interest in AI health technologies, a systematic review looking for prospective RCTs of AI health technologies in any clinical setting identified just 65 eligible publications since September 2020.¹³ Most of these studies (n = 24) took place in China, with Europe (n = 14), the United States (n = 12), and Japan (n = 5) being the major contributors. Categorizing eligible studies clinically, the largest contributors were gastroenterology (n = 15) and radiology (n = 5), with primary care, emergency medicine, diabetology, and cardiology each contributing four eligible RCTs. Despite their scarcity, this systematic review indicated a good overall quality of study design across these RCTs (Table).

Table.

Key Considerations in Clinical Study Design for AI-Enabled Interventions and Potential Design Considerations

Design Element	Description	Example of Good Practice
Study design	The study design for the intervention should provide sufficient evidence across key domains (including efficacy, safety, and cost effectiveness).	Failure mode and effect analysis could be included to illustrate the AI safety impact and inform mitigation strategies.
Population	The study population should reflect the population in which it is intended to be used and should be sufficiently diverse to detect underperformance in population subgroups, such as minority populations.	A recruitment strategy could seek to enrich the study population for minority groups within the target population through site selection or the participant eligibility screening process.
Setting	The study setting should reflect the setting (or range of settings) where it is intended to be used to provide sufficient confidence in the performance outside of “ideal” settings. AI-specific considerations here include the heterogeneity of devices used for data acquisition, data handling and processing procedures, and differences in clinical pathways.	Survey data describing digital infrastructure in healthcare providers within the target jurisdiction could inform the recruitment of fully representative study sites (e.g., prevalence of cloud computing capabilities).
Intervention	The AI component of any intervention should be described sufficiently to guarantee that results are ascribed to a specific AI system (including version identifiers) and would enable replication of the study.	Standardized training materials and processes could be produced to ensure consistent and replicable user competencies.
Comparator	The comparator should be a clinically relevant and accepted reference. This reference is commonly “standard practice” or “best practice,” with a view to informing decision makers as to whether and how much the intervention can bring improvement to the outcome, compared to a reference.	Contemporary clinical guidelines or care pathways published by government or professional bodies in the target jurisdiction could guide the selection of a comparator that reflects standard care.
Outcome	The outcomes measured should include those that are the most important to patients and other key stakeholder groups and be specified at the outset. Use of core outcome sets is recommended where they exist for the condition of interest. Pre-specification avoids bias through retrospective selection of the most favorable outcome or of a positive result arising through chance and multiple testing.	Prior primary qualitative research could inform outcome measure selection by fully characterizing stakeholder priorities and ensuring that proposed outcome measures are viewed as valid representations of those priorities.
Process measures	Relevant impacts on the overall health pathway should be considered, such as changes in time to diagnosis or treatment, efficiency of service provision, accessibility to care, and cost of healthcare delivery.	The volume of clinical activity associated with each case (e.g., frequency and length of clinical encounters) could be measured to enable the evaluation of economic and workforce impact.
Balancing measures	Relevant consequences of implementing the AI system to the other parts of the healthcare system should be considered including changes in behavior, changes in resource requirements, and potential ethical implications (such as loss of patient or clinician autonomy).	Think-aloud usability methods could be embedded within the study to better understand user experience and the mechanisms by which observed outcomes are generated.
Analysis	Analysis should be prespecified. It should include consideration of subgroups to ensure that risks of harm that only affect specific groups are detected. Errors should be analyzed at the individual error level to identify the reasons for failure where possible.	The rationale for patient characteristics used to shape subgroup analyses could be built upon patient or charity perspectives of current disease-specific care inequities or inequities anticipated to be introduced by AI.
Study protocol	The study should be registered (e.g., on the World Health Organization International Clinical Trials Registry Platform) in advance.	A detailed protocol with analysis plan could be submitted for peer-review publication prior to commencement of the study in addition to registration on a prominent trial registry.

Although well-designed, large-scale RCTs are a valuable source of evidence in the evaluation of an AI health technology, it is important to understand their limitations. To complement the quantitative evidence available from RCTs, researchers should also design qualitative research studies that use stakeholder perspectives and experiences to generate evidence regarding the sociotechnical mechanisms by which an AI health technology influences outcomes in a specific healthcare context.¹⁶ Many such studies are also preclinical in nature; just 20 studies of stakeholder perspectives of AI-enabled interventions in prospective clinical use were identified by a recent bibliometric study.¹⁷ Ideally, qualitative studies would complement quantitative research methods to help support improvements in intervention design and implementation within various healthcare contexts.¹⁸ These facets concerning the mechanism by which complex healthcare interventions exert their impact on the wider health system are addressed elsewhere.¹⁹

Reporting Clinical Studies

The design and conduct of a clinical study establish the potential evidence that it can offer to decision makers and potential end users of an AI technology (Fig. 1). This potential can only be fully realized through researchers’ commitment to transparent and complete reporting by (1) highlighting potential limitations of design and delivery that could introduce bias and (2) allowing other researchers and practitioners to test the reproducibility and replicability of the study.²⁰ This is a key step in validating research findings in science generally, but it is a particular issue in the context of AI-enabled interventions, where performance of a model is vulnerable to shifts in clinical, technical, and operational contexts.¹¹ Complete and transparent reporting by researchers also maximizes the amount of information that is available to decision makers and end users regarding the likely strengths and limitations of the intervention.

Academic journal editors and peer reviewers act as gatekeepers to the reporting and dissemination of clinical evidence. They exercise that duty based on their quality assessments of clinical studies and their reporting. To support those assessments, reporting guidelines provide explicit and accessible standards of best practice regarding what should be reported and how it should be reported. The accessibility of these reporting guidelines also benefits researchers, who can use them as an explicit guide to what the wider research community will expect them to report. The standard bearer for reporting guidelines is the Enhancing the QUAlity and Transparency Of health Research (EQUATOR) Network, which has supported the development of guidelines for a wide range of study types and contexts, most notably the Consolidated Standards of Reporting Trials (CONSORT) guidelines and, more recently, a series of extensions for application to AI health technologies (Fig. 2).^2–9,21,22 In addition, there are several independent guidelines that are specifically focused on AI technology. There is, reassuringly, a great deal of overlap in the requirements listed by these different AI-specific reporting guidelines.²³

Figure 2.

Schematic of evidence generation across the translational pathway and relevant research reporting guidelines for AI-enabled interventions. Reporting guidelines specific to AI are shaded black, with other relevant guidelines shaded gray. The figure is only indicative of the primary application of the guidelines; a number of them may contain elements that are also of value in other contexts. *Guidelines with AI-specific expansions in development.

EQUATOR reporting guidelines are organized according to study type. Additional guidelines (or extensions to guidelines) address issues that may not be adequately addressed by the core family of guidelines, notably for specific interventions (e.g., psychological interventions), outcomes (e.g., patient-reported outcomes), and newer study designs (e.g., n = 1 trial).

Reporting Guidelines for the Early Developmental Stages of AI Health Technologies

The majority of AI health technology research concerns a preclinical stage, which produces evidence that cannot account for the various factors that influence real-world performance. Nevertheless, these studies may represent the only available evidence and provide important insights into the provenance of an AI health technology. These insights into the architecture of models and their training data are important and may not be as clear in later studies. Minimum Information about Clinical Artificial Intelligence Modelling (MI-CLAIM) is a reporting guideline that supports high-quality evidence production at this earlier translational stage.² Of particular note is part 6 of the MI-CLAIM checklist, which aims to support the reproducibility of research through promoting the reporting of sufficient end-to-end technical details (including relevant data, code, and dependencies).² In the development of AI health technologies, this is often challenged by researchers’ concerns over the potential intellectual property associated with their work and the ethical and governance safeguards regarding data sharing.

Reporting Guidelines for Diagnostic and Prognostic Accuracy Studies of AI Health Technologies

At present, the majority of regulated AI health technologies are diagnostic systems that classify disease presence/absence, disease subtype, and/or disease severity.^10,24 Indeed, many areas where AI is expected to have the largest impact are in automated diagnosis of disease for population screening programs, such as breast cancer screening and diabetic eye disease screening.²⁵ These studies should adhere to the 2015 Standards for Reporting Diagnostic Accuracy Studies (STARD) guidelines in the absence of AI-specific reporting guidelines.²⁶ At the time of writing, an AI-specific extension, STARD-AI, is under development.³

Clinical prediction models estimate the likelihood of an individual having or developing disease using predictor variables (risk factors such as age, sex, and biomarkers). The ability of AI to analyze large and complex datasets of predictor variables has led to the development of several potential AI prediction models, such as AI health technologies for predicting sepsis.²⁷ The widely accepted EQUATOR reporting guideline for prediction and prognostic model studies is the Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis (TRIPOD 2015) guideline, also with a recently published AI-specific extension.^8,28

Reporting Guidelines for Early Interventional Studies of AI Health Technologies

The way in which AI technologies are designed to interact with humans and the learning curves associated with this are critical for successful implementation. Complex decision-support tools including those based on AI may have unpredictable human interaction properties warranting specific investigation.²⁹ There is an argument that early, small-scale feasibility studies are necessary after initial validation of algorithmic performance, prior to launching into large, expensive prospective trials. To support this stage of evaluation, Developmental and Exploratory Clinical Investigation of DEcision-support systems driven by Artificial Intelligence (DECIDE-AI) was published in 2022 and makes new recommendations for how such studies should be reported.⁴

Reporting Guidelines for RCTs of AI Health Technologies

RCTs are considered the gold-standard experimental design in the hierarchy of evidence, providing the most rigorous assessment of preventative, diagnostic, and therapeutic interventions. Standard Protocol Items: Recommendations for Interventional Trials (SPIRIT) and CONSORT are the accepted reporting standards for randomized trials and their protocols, endorsed by the International Committee of Medical Journal Editors.^22,30 Extensions to these two guidelines, SPIRIT-AI and CONSORT-AI, were published in 2020.^5,6 These extensions include 15 and 14 new items, respectively, considered as minimum standards of reporting for AI health technologies, in addition to reporting core items as outlined by SPIRIT and CONSORT. AI-specific recommendations include, but are not limited to

• •Description of the type and versions of the AI model and its intended use
• •Access and restrictions to access or reuse of the AI model and/or its code
• •How the AI system was integrated in trial sites
• •Inclusion and exclusion criteria at the level of participants and input data
• •Assessment and handling of poor-quality or unavailable input data
• •Any human–AI interaction elements
• •The output of the AI intervention and its impact on decision making or other elements of clinical practice
• •Analysis of performance errors of the system

Other Relevant Reporting Guidelines

An area of increasing importance to the evaluation of AI health technologies is health economics. The relevant EQUATOR guideline extension is the Consolidated Health Economic Evaluation Reporting Standards (CHEERS-AI).⁷ The mechanisms by which AI health technologies should be reimbursed and how their economic value should be defined and measured are evolving, but evidence of economic value remains a key consideration for decision makers.^31,32

As AI health technologies become more widely adopted, observational studies based on real-world evidence will emerge as valuable forms of evidence for decision makers. These studies should apply the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) research reporting guidelines.³³ Another more practice-oriented guideline that can be used alongside others is the 2014 Template for Intervention Description and Replication (TIDieR).¹⁶ It demands more practical disclosures from healthcare studies, including the assumed social and organizational basis for the effect of an intervention, a characterization of the actors involved in the implementation, and adaptions to the intervention or the context to which it was being applied. These details can help decision makers to assess the feasibility of an AI intervention in their own healthcare context.

Appraising Clinical Studies

The design, conduct, and reporting of clinical studies represents evidence production, whereas their appraisal represents the evaluation of that evidence regarding quality, relevance, and other factors. Decision makers can then synthesize the available evidence alongside other sources of information relating to need and context (including input from patients, professionals, and other stakeholders) to make better, evidence-informed choices about AI-enabled healthcare interventions. The ability to appraise clinical evidence is also valuable for potential end users of AI health technologies, who need to have appropriate trust in the performance of any technology they may adopt in the context of their own practice. By gaining this authentic and independent sense of the strengths and limitations of a technology, end users can support better procurement decisions within their organizations, they can be more confident about the limitations of use (e.g., when to challenge a “decision”), and be better informed to communicate with patients about the device and the interpretation of its outputs.

Reporting guidelines can be used as frameworks to guide appraisal, but their primary goal is to define what should be reported (focused on transparency) rather than defining how it should be done (focused on quality and minimizing bias). For potential end users of AI health technologies, there is a complementary range of appraisal tools with the primary goal of supporting systematic quality appraisal of evidence.³⁴ These tools were mainly developed for evidence synthesis research but are highly relevant to the evaluation of evidence in isolation and are accessible to potential end users of AI health technologies without deep experience of research or commissioning. The Quality Assessment of Diagnostic Accuracy Studies (QUADAS) tool identifies key considerations for diagnostic accuracy studies, and an AI specific extension (QUADAS-AI) is under development.³⁵ Similarly, the Prediction model Risk Of Bias ASsesment Tool (PROBAST) identifies key considerations for prognostic models and also has an AI extension (PROBAST-AI) under development.³⁶ There are also resources that propose heuristics to aid the evaluation of studies of AI-enabled healthcare interventions of all study designs.^34,37,38

When appraising clinical evidence of an AI health technology, potential end users should first consider the intervention being described. Understanding the operating principle of the tool itself alongside its inputs and outputs allows a basic check of the viability of an AI-enabled intervention in a specific clinical context. The method used for training the AI technology may also offer insight into this; for example, training a model using a federated learning approach can address information governance barriers to multi-institutional datasets, helping to produce more robust models with generalizable performance.³⁹ Understanding the data on which an AI technology was trained will also permit an appraisal of its likely performance in a specific population and any subgroups for whom it may underperform.⁴⁰ The capacity and expertise of real-world end users of the AI technology should be compared to the individuals who used it in the clinical study. This becomes particularly important when appraising AI-enabled interventions where the AI has an assistive rather than automating function. For many use cases, diagnostic performance of AI-assisted clinicians is greater than either clinician or AI performance in isolation.⁴¹ This performance differential is sensitive to the timing of clinician–AI interaction in the decision-making process and the degree of explainability that is presented to the clinician. However, the benefits of an assistive use case over an automated one are offset by the instability it introduces to intervention performance (requiring long-term monitoring by adopters) and risk (between users and over time within the same user) and the missed opportunity to reallocate clinician time to other tasks.^11,13,29 To assess the balance of these costs and benefits for a particular AI-enabled intervention, choices made by researchers about how the AI interacts with clinicians in a study should be justified. This would ideally be based on usability methods that test different AI–clinician interaction protocols but should at least consider the practical demands of the disease and care pathway that the intervention targets. The practical demands of the digital infrastructure and data flows required for a given AI-enabled intervention are also relevant for decision makers. Flexibility from AI manufacturers can help here. For example, a product that can be used with software installed locally at the healthcare provider institution or remotely on a secure cloud-platform can be expected to align with many different institutions’ needs and priorities.¹²

Conclusions

If the emergence of AI into health care is to improve health care, decision makers must see through the surrounding hype to invest in the minority of AI-enabled interventions that promise evidence-based benefits. To do this, effective evaluation is critical. The evaluation of AI health technologies is unfamiliar territory for most decision makers in health care, but they already hold most of the relevant expertise. It is important to recognize that the evaluation of these technologies builds on existing, well-established methodologies and that evaluators need not be overawed by the technological sophistication of the interventions under consideration. The distinct features (and risks) of AI health technologies are outlined in the emerging AI-specific guidelines, with the EQUATOR network providing reporting guidelines tailored for specific trial designs. Given the sensitivity of AI health technologies to their implementation context, it is also crucial that decision makers ensure that they are provided with evidence that addresses these issues. There is a challenge here for the community to invest in later phase studies such as RCTs and other interventional studies that provide such evidence, so that we can ensure that the AI health technologies that come to patients are effective, safe, and equitable.