AI as an intervention: improving clinical outcomes relies on a causal approach to AI development and validation

Abstract

The primary practice of healthcare artificial intelligence (AI) starts with model development, often using state-of-the-art AI, retrospectively evaluated using metrics lifted from the AI literature like AUROC and DICE score. However, good performance on these metrics may not translate to improved clinical outcomes. Instead, we argue for a better development pipeline constructed by working backward from the end goal of positively impacting clinically relevant outcomes using AI, leading to considerations of causality in model development and validation, and subsequently a better development pipeline. Healthcare AI should be “actionable,” and the change in actions induced by AI should improve outcomes. Quantifying the effect of changes in actions on outcomes is causal inference. The development, evaluation, and validation of healthcare AI should therefore account for the causal effect of intervening with the AI on clinically relevant outcomes. Using a causal lens, we make recommendations for key stakeholders at various stages of the healthcare AI pipeline. Our recommendations aim to increase the positive impact of AI on clinical outcomes.

Artificial intelligence (AI) holds great promise to improve healthcare. Modern AI excels at modeling complex associations in data to automate clinical workflows, predict patient outcomes, diagnose conditions, recommend interventions, or broaden clinical knowledge.^1–3 The democratization of AI with open-source software, datasets, and large-scale computing has led to AI innovation in healthcare.⁴ This success, in concert with the availability of healthcare data for research,⁵ has generated significant interest in harnessing AI to improve patient outcomes.^6–8 So far, the broader success of AI has remained unrealized in healthcare.^9–11 Here, we highlight that this is partly because improving patient outcomes is not prioritized in the AI development, evaluation, and validation pipeline.

Currently, AI development for healthcare often begins with training state-of-the-art AI methods on a healthcare task, where success is measured using broadly used model performance metrics, like AUROC (Area Under the Receiver Operating Characteristic Curve) and DICE score (harmonic mean of precision and recall, also called F1-Score), on whatever held-out data are at hand. However, clinical validation of AI evaluates the utility of intervening with these models on clinically relevant outcomes. The difference between model evaluation during development and the goal of improving clinically relevant outcomes implies a disconnect. As one example, metrics such as AUROC, unlike decision-analytic measures,¹² only evaluate performance in a predictive task, disconnected from the effect of intervening with the AI model.¹³ As another example, the target population to which the model will be applied may differ from the held-out population on which the model was retrospectively validated. For instance, predictions may only be available for a subset of clinical units that have been educated about an AI’s use, a form of selection bias.

Consider identifying structural heart disease (SHD) using electrocardiograms (ECGs).¹⁴ SHD is diagnosed using an echocardiogram (echo), an expensive data modality not available for all patients. ECG is 10 times more prevalent and 20 times less expensive than echo, but human experts cannot identify SHD using ECG alone. There is emerging evidence that AI models can use ECGs to diagnose SHD, turning cheap and common ECGs into a screening tool.¹⁴ Here, ground truth labels exist only in patients who also received an echo. Thus, the subpopulation of patients with both ECG and an echo constitutes the training and validation data used for model selection. Clinically, the model could be deployed to match the current workflow. That is, the model can run only on patients for whom a clinician has ordered an echo. Used in this way, the model could help avoid unnecessary testing but would not address SHD underdiagnosis. To identify cases that would otherwise be missed, ideally the model would apply to all patients with an ECG, including those who may not have an echo. However, model performance measures like positive predictive value (PPV), on patients with both an ECG and an echo, do not necessarily translate to a high PPV on patients who received an ECG, since prevalence differs between these cohorts.

Understanding how the AI will be integrated into clinical workflows can help identify differences in evaluation versus deployment cohorts and inform model development, such as adjusting for the difference using re-weighting.¹⁵ Accounting for such population differences during development will reduce the chance that these challenges are only uncovered during prospective validation. An example of prospective validation is a silent trial, where the model output is not available or visible during clinical decision-making but validated on a relevant outcome.^16–18 For example, a silent validation identified that a population shift due to age distribution differences, proportion of left and right kidneys, and format differences caused an AI predicting obstructive hydronephrosis in infants to deteriorate.¹⁷ Reviewing differences in populations prior to silent testing could have saved time and resources.

Integrated AI solutions should be actionable, ie, improve downstream decision-making and subsequently, patient outcomes.¹⁹ The effect of AI on such changed decisions is an intervention on the patient. Quantifying the effect of such an intervention on patient outcomes is causal inference.

Viewing AI solutions as interventions that should improve patient outcomes allows us to reason backward from this goal to enhance the AI development pipeline. We highlight critical considerations to build the “best feasible” AI, ie, interventions most likely to deliver good outcomes considering the effort required to improve the quality of estimates of the AI intervention on patient outcomes.

Task conception

Prioritizing patient outcomes can organically inform the correct learning task.

Prospective integration and validation

Ideal evaluation of AI solutions should measure relevant primary endpoints and safety. Other secondary outcomes must be justified in the context of the primary endpoint.

Model selection and evaluation

Prospective validation of AI, especially Randomized Controlled Trials (RCTs), is expensive. AI development leading up to experimental validation should make the best use of the data at hand for model selection and evaluation. We emphasize 2 forms of evaluation crucial for healthcare AI.

1. Silent trials: Silent trials enable running a model in the background without influencing clinical decision-making. A silent trial that measures a proximal outcome correlated to the target outcome of interest can help quantify model behavior under natural differences from the training conditions without affecting clinical decisions, mitigating safety concerns.

2. Causal off-policy evaluation (OPE)^20,21: Retrospective evaluation and model selection should go beyond evaluation on held-out data and simulate the effect of the AI intervention, for example, using causal OPE.

Model design

AI training and design choices should proactively reduce the burden on evaluations to surface potential obstacles to trial success, such as differences in the training and deployment population, or inadequate handling of statistical biases. AI solutions that seek to improve outcomes through optimizing decisions (eg, recommend treatments) target a causal task and should use methods such as observational causal inference.^22–24

Continual monitoring, validation, and improvement

Evolving patient populations and the effect of deploying the AI intervention itself suggest that static models cannot be guaranteed to consistently improve patient outcomes. However, compared to pharmaceutical interventions, AI interventions can provide shorter feedback loops on patient outcomes, enabling improvements in subpopulations for whom the model may currently be underperforming.

Figure 1 summarizes the proposed healthcare AI development pipeline. We identify actionable recommendations for key stakeholders in various phases of the healthcare AI pipeline.

Figure 1.

Deploying AI in healthcare is an intervention that can impact clinical outcomes. The figure shows the proposed healthcare AI development pipeline which centers on a causal approach to AI development with the goal of improving clinically relevant outcomes. Knowledge of outcomes of interest should inform relevant endpoints, and hence the task conception, data used for training, model design, the design of silent validation and model selection protocols, and prospective validation. Continual monitoring, validation, and improvement should involve measuring the effect of the AI intervention on clinically relevant outcomes.

Recommendations to the AI research community

The AI community has and will likely play a defining role in determining which clinical questions are the focus of potential AI solutions, such as the choice of methods usually employed, early-stage model evaluation prior to silent and prospective validation, and methodological innovation for post-deployment model improvement.

• Task conception: The AI community should consider: What task is important and how to train the AI model to succeed in this task? While the precise task will likely be defined in collaboration with clinicians and patients (and based on the availability of data), it is important to consider that AI solutions are used by humans. Therefore, the AI community should account for the AI solution’s impact on decision-making when defining the task.

• Model selection and evaluation: Moving beyond standard evaluation approaches, healthcare AI developers should consider advanced techniques that account for potential differences in population once deployed. Advanced evaluations such as OPE are currently largely unknown, under-utilized, or not broadly acceptable, despite their utility in quantifying the benefit of AI interventions. We argue for a broader dissemination and acceptance of such methods among clinical model developers and peer reviewers. Nonetheless, these methods require checking sensitivity to assumptions that may not hold in practice.²⁰

• Model design: Healthcare AI developers should consider whether the candidate training data are close to the deployment population and of reasonable quality, ie, with acceptable levels of statistical bias. This may be measured according to multiple objectives, eg, equal subgroup performance if desirable. Model designs that capture generalizable associations (as opposed to spurious correlations)²⁵ are more likely to perform well when the training and deployment populations differ. AI models should be trained to proactively account for biases such as missingness and censoring, for example, using survival analysis,^26–29 including accounting for any equity goals.^30–33

• Continual monitoring, validation, and improvement: New research for continual model monitoring, validation, and improvement based on improvement in patient outcomes is necessary.

Recommendations to the causal inference, implementation science, and trial design community

There is a prolific community focused on methods for drawing causal inferences from observational and experimental data^23,24,34 that could inform advances in prospective integration and validation of AI models, and experiment design for continual monitoring and model improvement, primarily to bridge the current methodological chasm.

• Prospective integration and validation: We surface key lessons learned by this community to enable rigorous evaluation of AI interventions: appropriate experimental design and evaluating patient-focused endpoints are crucial. The effect of an AI intervention on patient outcomes may be heterogeneous and should be quantified using robust analysis. When randomization is not possible, observational causal inference methods may be useful but must be used carefully, allowing for unmeasured confounding, such as using sensitivity analysis.^35–38 We also encourage the causal inference community to engage closely with AI development and consider clinical AI interventions as worthy of evaluation.

• Continual monitoring, validation, and improvement: To enable continual validation, dynamic trials, such as adaptive trial designs,³⁹ are valuable, but may not keep up with AI technological improvements. In addition, new validation and AI improvement strategies such as safe micro-randomization may be necessary.⁴⁰

Recommendations for publishing guidelines of healthcare AI

Publication guidelines indicate minimal reporting requirements acceptable to the scientific community. The current reporting guidelines for AI development and evaluation may be underspecified to improve the impact of AI solutions and hence benefit from further informing reporting guidelines for model design, and the development of new guidelines for post-deployment monitoring, validation, and improvement.

• Model design: For example, TRIPOD+AI,⁴¹ a minimal reporting guideline for initial model development does require a discussion on how the AI intervention will be incorporated into the clinical pathway and inform decision-making. While the guidance also asks for modeling rationale, they do not explicitly ask for a report of how the clinical use justifies model design and evaluation choices.⁴² DECIDE-AI,⁴³ CONSORT-AI,⁴⁴ and SPIRIT-AI⁴⁵ guidelines require reporting design and evaluation choices in the context of the clinical use case. We recommend that these discussions, specifically tying the clinical use case to model design choices be incorporated in the TRIPOD+AI checklist.

• Continual monitoring, validation, and improvement: New publishing guidelines and post-market regulatory policies are necessary for reporting and emphasizing improved infrastructure for continual monitoring, validation, and improvement of AI interventions.⁴⁶ Such regulations should reconcile with ethical tensions in consent-based publishing and the ethics of improving care for all.

• Recommendations to healthcare institutions: Healthcare institutions can benefit from improved healthcare AI pipelines if there is adequate infrastructure in place to enable safe AI integration, deployment, monitoring, and improvement.

• Continual monitoring, validation, and improvement: Healthcare institutions should prioritize compute and data infrastructure, and personnel training for continual monitoring for safety and efficacy,^47–49 beyond monitoring performance metrics and enabling feedback of AI-supported actions on patient outcomes.

These recommendations surface the need for tight-knit collaborations between several stakeholders to address multiple aspects of the healthcare AI pipeline, with a progressively higher number of involved stakeholders over the stages of the healthcare AI pipeline.

The United States Food and Drug Administration (FDA) has consolidated a comprehensive set of “Good Machine Learning Practices” (GMLP) for the healthcare AI pipeline.⁵⁰ We discuss a few key guiding principles from this practice in the context of the causal lens. The GMLP recommends that “Data Sets Are Representative of the Intended Patient Population” and “Testing Demonstrates Device Performance During Clinically Relevant Conditions.”

Since AI solutions are causal interventions, not only is representativeness a necessity, but there is also a need to proactively design AI to prevent any anticipated mismatch in training and deployment populations and enhance model design if such mismatch is not preventable. Even after establishing the desirable efficacy of an AI intervention, several post-deployment steps are necessary to monitor and improve the effect on patient outcomes. For example, GMLP guidelines recommend that “Deployed Models Are Monitored for Performance and Re-training Risks Are Managed.” A focus on model performance alone is insufficient and the operationalization of the GMLP principles in the “Pre-determined Change Protocol Plans” should be grounded in how the AI intervention affects clinically relevant outcomes, in addition to “device performance” throughout its life cycle, including assessing efficacy-risk tradeoffs.

Furthermore, there may be scenarios where an existing AI model is applied to a new setting, ie, “off-label use” of AI due to infeasibility of model development.⁵¹ Here, causal thinking is equally important in evaluating the appropriateness. Careful retrospective evaluation with attention to patient population differences and primary outcomes can save time and effort if the model does not inform new uses.

To deliver on the promise of healthcare AI, considering the desired causal impact of an AI solution on downstream decision-making and clinically relevant outcomes is imperative. There are additional cognitive, research, and resource costs to achieving the desirable causal impact and may entail longer and iterative model development time, reliance on more sophisticated model selection methods, and pre-deployment user studies for evaluating impact on decision-making. However, such considerations will ultimately increase the chances of success of models once deployed. This work highlights several desirable changes in how the healthcare AI research community should approach model development, evaluation, and validation. Implementation science, causal inference, and trial design communities can lend practical insights for validating AI interventions, and the need for new healthcare infrastructure to support such developments. Additionally, several advances in regulatory, ethical, and publication guidelines for healthcare AI while upholding bioethical principles are necessary.

Author contributions

Shalmali Joshi and Rajesh Ranganath: Conceptualization, Writing—original draft, Writing—reviewing and editing; Iñigo Urteaga, Wouter A. C. van Amsterdam, George Hripcsak, Pierre Elias, Benjamin Recht, Noémie Elhadad, James Fackler, Mark P. Sendak, Jenna Wiens, Kaivalya Deshpande, Yoav Wald, Madalina Fiterau, Zachary Lipton, Daniel Malinsky, Madhur Nayan, Hongseok Namkoong, Soojin Park, Julia E. Vogt, and Rajesh Ranganath: Writing—reviewing and editing.

Funding

The authors acknowledge support from Machine Learning for Healthcare.

Conflicts of interest

JF, MPS, JW, and RR serve on the Board of Machine Learning for Healthcare.

Data availability

There are no new data associated with this article.