Transparency of medical artificial intelligence systems

Abstract

Artificial intelligence (AI) systems are now prevalent in our daily lives and hold promise for transforming high-stakes fields such as healthcare. Medical AI systems are showing significant potential to support diagnostics and treatment recommendations. As these systems play an increasingly significant role in clinical decision-making, ensuring transparency in their design, operation, and outcomes is essential for building trust among key stakeholders, including patients, providers, developers, and regulators. However, many systems still function as “black boxes,” making it challenging for users–such as clinicians, patients, and other stakeholders–to interpret and verify their inner workings. Here, we examine the current state of transparency in medical AIs, identifying key challenges and risks these opaque systems pose. After motivating the need for transparency in all aspects of the machine learning pipeline, from training data to model development to model deployment, we explore a range of techniques that promote explainability throughout the pipeline while highlighting the importance of continual monitoring and system updates to ensure that AI systems remain reliable over time. Finally, we address the need to overcome barriers that inhibit the integration of transparency tools into clinical settings and review regulatory frameworks that prioritize transparency in emerging AI systems. Through this survey, we aim to increase awareness of current challenges and offer actionable insights for stakeholders, such as researchers, clinicians, and regulators, on how to build trustworthy and ethically responsible AI healthcare solutions.

Introduction

Recent rapid advances in medical AI, accelerated by breakthroughs in deep learning techniques, are transforming healthcare. Medical AI models are trained on clinical data to learn underlying patterns; once trained, they can analyze new clinical data to support diagnostic decision-making and treatment planning, helping to identify diseases earlier, personalize treatments, and improve patient outcomes (Box 1). Applications of these technologies span varied domains, including radiology for image interpretation¹, pathology for cancer detection², dermatology for lesion classification^3,4, and cardiology for cardiovascular disease diagnosis^5,6.

Box 1 ∣. Artificial intelligence basics.

General Artificial Intelligence Concepts

Deep learning.

A subfield of machine learning that uses artificial neural networks, consisting of many hidden layers containing nonlinear activation functions, to model complex patterns in data. Compared to traditional machine learning approaches, it automatically learns features from raw input (e.g., image or text) in an end-to-end manner, without relying on manually crafted features.

Supervised learning.

A machine learning paradigm where a model is trained using labeled datasets, with each input paired with a corresponding output (e.g., skin lesion images labeled with their diagnoses).

Self-supervised learning.

A machine learning paradigm where a model is trained on unlabeled datasets via generating supervisory signals from the data itself (i.e., leveraging the data’s structure to create pseudo-labels). For example, we can hide part of a sentence or image and train a model to predict the missing part. This approach enables training on large-scale unlabeled datasets.

Contrastive learning.

A training strategy where a model learns representations by contrasting similar and dissimilar data pairs.

Foundation models.

Large-scale neural networks that are pre-trained on vast amounts of diverse data, often using self-supervised learning. Once trained, these models can be adapted for a wide variety of downstream tasks without needing to be retrained from scratch. For example, ChatGPT, which can perform various tasks, such as text summarization, sentiment analysis, and question answering, is a foundation model.

Generative models.

Models designed to generate new data resembling the training data, such as synthetic images (e.g., GANs or diffusion models) or natural language (e.g., ChatGPT).

Fine-tuning.

A process of taking a pre-trained model and further training it on a new (typically smaller) dataset by updating model weights to adapt to the specific characteristics of the new task. For example, a general language model can be fine-tuned on a medical dataset to equip it with clinical knowledge and enable it to perform tasks such as clinical text classification or diagnosis prediction.

Model Reliability and Generalization Concepts

Fairness.

Ensuring that medical AI provides equitable performance across diverse population groups. It is quantified through criteria like demographic parity, predictive parity, and equalized odds, which assess performance differences across protected subgroups.

Robustness.

The ability of models to maintain their performance under varying conditions, such as noise, outliers, or adversarial attacks.

Distribution shifts.

Changes to the distribution of data between the training set and the test set or the deployment environment, which can degrade model performance. For example, a classification model trained primarily on images of light skin tone may show reduced accuracy when applied to images of darker skin tones due to a shift in data distribution. Spurious correlations. These occur when variables are associated but lack a causal relationship. In medical images, artifacts such as ruler marks or stray hairs may be correlated with the label but hold no clinical relevance to the task at hand. AI models can inadvertently exploit these associations, leading to deceptively high performance on training data without capturing meaningful clinical relationships.

Confounders.

Variables that are associated with both the input features (e.g., patient characteristics, imaging data) and the target (e.g., diagnosis). Confounders can result in the development of spurious correlations.

Overfitting.

A phenomenon that occurs when a model learns patterns specific to the training data, including noise and irrelevant details, rather than generalizable features. Overfitted models perform very well on the training dataset but poorly on new, unseen datasets.

Beyond diagnostic and treatment support, medical AI models are increasingly being applied to drug discovery and patient monitoring, offering new pathways to improve patient outcomes^7,8. This growing promise is spurring active research and development, with more than 1,000 such models receiving regulatory approval to date from the U.S. Food and Drug Administration (FDA) as medical devices^9-12. Notably, some models are now directly accessible to patients through smartphone applications for self-assessment^13,14.

Despite these advances, integrating medical AI systems into clinical practice poses significant challenges. First, these systems, deployed in high-stakes clinical settings, remain subject to failure; although they can perform well on curated datasets in controlled environments, they often struggle to generalize to real-world scenarios due to distribution shifts, such as variations in imaging techniques, patient demographics, or rare diseases^15-17. Second, they are susceptible to learning shortcuts, i.e., spurious associations between features and target labels in training data, that limit their generalizability^18-21. For example, although AI models developed to detect COVID-19 from chest radiographs initially showed impressive performance in research studies, subsequent evaluations revealed that they relied heavily on spurious associations between labels and dataset-specific artifacts, such as text markers and differences in radiographic projections, rather than genuine markers of medical pathology^22,23. Such failure modes lead to delays in diagnoses and treatments and undermine the vital element of trust in medical AI systems.

Further compounding these issues is the “black-box” nature of many medical AI models^24-26. Frequently developed in siloed environments using private data, these systems lack the transparency required for necessary external scrutiny and validation. Furthermore, they are inherently complex, often involving neural networks with millions to billions of parameters, exacerbating user efforts to comprehend and interpret their mechanism^27-30. Without clear insights into how these systems operate, when and why they may fail, or even why they succeed, integrating them into clinical workflows remains problematic and potentially dangerous.

Transparency has therefore emerged as a critical pillar for developing and deploying safe, reliable medical AIs^31,32. By ‘transparency,’ we mean the extent to which essential information about AI systems, such as characteristics of the data used to train and test the model, their intended use, development process, underlying logic, performance, and known limitations, is clearly communicated to stakeholders. Insight into how models function improves model interpretability, helping developers debug and refine AI systems to ensure they function as intended; it thereby fosters trust among clinicians and patients³³. Even when an AI system proves effective in randomized controlled trials^34,35, transparency into their inner workings remains critical. Patients may be skeptical of AI-assisted diagnoses without understandable explanations^36-38. Likewise, clinicians, who are accountable for medical decisions, must be able to explain them to patients and respond to their concerns. Furthermore, transparency helps meet evolving regulatory standards that increasingly prioritize explainability and accountability³¹.

Achieving transparency requires a holistic approach that addresses every stage of medical AI development: data collection, model development, and clinical deployment. In this review, we analyze transparency at each stage (Fig. 1). We scope this review to focus on complex deep learning models that are widely applied to different clinical data domains, such as imaging, electronic health records, and waveforms. First, we examine challenges to achieving data transparency, such as limited access to datasets used to train and test AI models, and explore potential solutions. Next, we review methods and trends in explainable AI (XAI), a subfield of AI that aims to explain the inner workings of AI models. We then explore strategies to maintain transparency and safety during clinical deployment. Finally, we identify emerging trends in medical AI systems and propose opportunities to improve the utility of transparency tools. We note that many of the principles and strategies we discuss are applicable to other biomedical AI domains, such as drug discovery and genomics, but those are beyond the scope of this review.

Fig. 1 ∣.

a, The development pipeline for medical AI systems, consisting of three key stages: data collection, model training, and deployment. For each stage, we review key issues of transparency and techniques to address them. The feedback loops highlight the iterative process, where issues identified during deployment monitoring can inform adjustments in model training and data collection procedures. b,–d, Examples illustrating transparency challenges at each stage. b, In the data collection stage, identifying and documenting the properties of the training data is essential for assessing biases and generalizability. For example, training datasets for melanoma detection AI often overrepresent Fitzpatrick skin types I–II (light skin tones) and underrepresent Fitzpatrick skin types V–VI (dark skin tones). A model trained on such disproportionate data can show biased performance across populations. c, During model training, it is important to go beyond evaluating accuracy and interrogate how models make predictions. For instance, COVID-19 classifiers trained on chest X-rays that initially reported high performance were later found to rely on spurious cues such as text markers rather than medically meaningful signals. d, Transparency during deployment includes continuous monitoring of model performance. For example, an AI system triaging diabetic retinopathy via fundus photography can show a performance drop after a change in image acquisition hardware, underscoring the need for continuous monitoring and adaptive model updates. The chest X-ray illustration was adapted from Wikimedia Commons by Jmarchn under CC BY-SA 3.0 license.

Data transparency

As the lifeblood of medical AI models, data fuels their development and evaluation. Therefore, data transparency is vital to ensuring the safety and reliability of these systems (Fig. 2).

Fig. 2 ∣.

Key components of data transparency in medical AI systems. a, Data documentation involves recording key properties of the data (e.g., patient demographics, labeling procedures, collection methods) to ensure transparency and track biases. Several initiatives aim to standardize these practices, including the MI-CLAIM Checklist⁵⁹, the MINIMAR Reporting Standards⁶⁰, and the STANDING Together initiative⁶¹, and the TRIPOD+AI statement⁶². b, The scarcity of diverse, publicly available datasets hinders both the development of medical AI models that are generalizable and their rigorous external validation. The Medical AI Data for All (MAIDA) initiative seeks to address this challenge by establishing a global framework for medical imaging data sharing⁴¹. MAIDA collaborates with partner institutions worldwide, providing standardized templates for institutional review boards (IRBs) and data-transfer agreements (DTUAs). It also supports partners with the data standardization and de-identification processes by providing custom DICOM identification tools and data-recording worksheets. c, Privacy-enhancing technologies are also being developed to facilitate data sharing. For example, federated learning allows institutions to collaboratively train AI models, without transferring raw patient data, by sharing local weight updates only with a central server. Data sharing facilitates collaboration across institutions to build medical AIs that generalize across varied clinical settings. d, Synthetic data generation uses generative models to create datasets that replicate the statistical properties of real data. Users can control the generation process through queries to produce images with desired properties (e.g., rare conditions or artifacts). The chest X-ray image was adapted from Wikimedia Commons, originally by Hellerhoff, under CC BY-SA 3.0 license.

Data documentation

Transparency in medical AI models starts with identifying and documenting the properties of the data used to train them. Clear insights into the characteristics of training datasets are essential to identify whether biases exist and assess if AI models are generalizable, i.e., they perform equally well across diverse clinical settings and patient populations. However, transparency into data provenance can be limited by a lack of detailed information on data acquisition and pre-processing^39,40. Specifically, key properties of data used to train and test medical AIs include patient demographics, data labeling procedures, and data collection methods, which are often incomplete or entirely unknown^39,41. This deficiency is particularly problematic in the medical domain, where most medical datasets are siloed due to their sensitive nature and access to raw data is not possible^39,42,43.

Patient demographic information, such as patient age, sex, and race/ethnicity, is crucial for evaluating whether a model can generalize across different groups. If training data is biased towards certain demographics, models will often exhibit poor performance on underrepresented populations, perpetuating healthcare inequities^16,40,44,45. Nonetheless, many medical AI models, whether developed in academia or industry, fail to reveal such demographic details^39,46,47. While documenting demographic information is essential, this need for transparency must also be balanced with the recognition that such data is inherently sensitive and must be handled carefully. Demographic attributes may increase re-identification risks, especially when combined with other quasi-identifiers^48,49. As such, disclosure risk must be rigorously assessed before releasing individual-level demographic data. Moreover, how demographic variables are defined and derived can influence which populations are made visible in analyses^50-54. Therefore, documenting how they were recorded and derived is as important as reporting the data itself.

Equally important is a thorough description of data labeling procedures, including (1) who performed the labeling (e.g., level of expertise and number), (2) how the labels were verified (e.g., biopsy-confirmed diagnoses for skin lesions), and (3) what diagnostic criteria were used. In most cases, this information is not available, or labeling is performed inconsistently, giving rise to issues of label noise and lack of trustworthiness^39,44. Furthermore, diagnostic criteria often evolve over time or vary by region, leading to labeling inconsistencies that can affect model predictions when deployed across different healthcare systems¹⁷. For example, Behcet’s disease, with 17 different diagnostic criteria proposed globally, illustrates how temporal and regional variations in labeling criteria can undermine model consistency^17,55,56. Models trained on older classifications may perform poorly or inconsistently under updated standards.

Finally, descriptions of data collection methods, such as (1) sources of data (e.g., hospitals or online databases), (2) the equipment used for image acquisition, and (3) any pre-processing steps applied (e.g., sample filtering or image normalization), are vital for understanding the context of the data and how it might influence the AI model. For example, differences in imaging devices across hospitals or variations in lighting conditions in dermatology images can introduce biases that affect model predictions. Additionally, (4) artifacts in datasets should be documented. For example, in dermatology images, common artifacts, such as marker ink, gel bubbles, color charts, ruler marks, and skin hair, can correlate with target labels, causing a model to exploit these spurious correlations instead of clinically meaningful signals^57,58.

To address these issues, several initiatives have aimed to standardize reporting guidelines for documenting medical AI datasets. These include the MI-CLAIM Checklist⁵⁹, the MINIMAR Reporting Standards⁶⁰, the STANDING Together initiative⁶¹, and the TRIPOD+AI statement⁶², paralleling initiatives in general AI domains^63-65 that requires labels essential to understanding potential biases during model training. Having detailed documentation lets developers mitigate potential biases during model training using computational techniques even if the biases are discovered after data collection^17,66,67.

Data sharing

Access to large-scale datasets collected from diverse sources is essential for training robust models capable of generalizing across varied clinical settings. Further, open access to data enables broader scrutiny and validation by the scientific community, fostering transparency and promoting collaborative research. However, data sharing in medicine is limited due to patient privacy concerns^42,68 as well as legal and institutional barriers^41,69-71. The sharing of medical data containing sensitive patient information increases the risk of data breaches, which can result in severe legal liabilities, harm to patients, and a loss of patient trust. Also, institutions, including hospitals and private companies, are often reluctant to share data, and data sharing often gives rise to lengthy data use agreement negotiations. As a result, public datasets are currently scarce and lack diversity due to limited data sources¹⁶, posing a significant obstacle to training models that generalize well and to replicating medical AI models and fairly evaluating their performance^40,58,72-74.

To address this challenge, efforts are being made to facilitate responsible data sharing while maintaining patient privacy. Institutional frameworks, such as open-source data initiatives, aim to promote transparency, interoperability, and responsible data use by developing standardized procedures for data collection, de-identification, and sharing^42,75-81. For example, the MAIDA initiative has established a global framework for sharing medical imaging data that involves collaborating with diverse institutions, standardizing data collection and de-identification protocols, and addressing local challenges with tailored solutions^41,82. Specifically, for de-identification, MAIDA provided detailed data-recording worksheets that were designed to omit variables classified as protected health information (PHI). In addition, partners were instructed to carefully review and modify any potentially identifying data before sharing. Such initiatives aim to improve the accessibility and diversity of medical data while complying with ethical and legal requirements.

On the technical front, privacy-enhancing technologies are being developed to facilitate medical data sharing. De-identification tools systematically remove or obfuscate patient identifiers in datasets, facilitating broader data sharing while maintaining privacy standards⁴¹. Further, federated learning enables the collaborative training of medical AI models across multiple institutions without the need to centralize sensitive patient data^17,78,83-88; this approach lets multiple institutions contribute to the development of a shared model without directly exposing their patient data^17,71.

Synthetic data generation

Synthetic data generation is emerging as a promising solution to address the challenge of limited access to large-scale and diverse medical datasets^43,89. Synthetic data refers to artificially created data that replicates the statistical properties and patterns of real data without revealing actual patient information⁹⁰. Generative models, such as generative adversarial networks (GANs) and diffusion models, are trained on real datasets to learn their underlying data distribution and generate new, artificial data points that statistically resemble the original data^43,91-93.

One significant advantage of synthetic data is its ability to facilitate data sharing. Since synthetic datasets contain no identifiable patient information, they can be safely shared for research and training purposes. Additionally, synthetic data can augment existing medical training datasets^90,94,95. This is particularly useful for addressing bias and fairness concerns in imbalanced training data^96,97 since generative models can produce data that reflect underrepresented populations or rare conditions. By doing so, synthetic data can contribute to more representative and equitable models.

However, synthetic data generation also imposes limitations. First, generative model data is not completely free of privacy issues: if the generative model overfits and memorizes training samples, it could generate synthetic data that reveals sensitive personal information, posing the risk of patient re-identification^43,89. Second, evaluating the quality of synthetic data remains challenging^89,90 since it must consider multiple dimensions: fidelity (realism of generated samples)^22,98-100, diversity (capturing of real data variability)⁹⁹, privacy or identifiability (distinguishability of synthetic from real samples)¹⁰¹, and utility (effectiveness of data for downstream tasks)¹⁰². However, the lack of a standardized framework for selecting these metrics complicates evaluation. Moreover, there are inherent trade-offs between these dimensions. For example, stricter privacy constraints to limit information leakage could reduce synthetic data’s utility by making it less representative of real data and omitting outliers⁸⁹. Optimizing multiple quality metrics simultaneously involves prioritizing different dimensions depending on specific synthetic data use cases. Lastly, the generative models themselves pose challenges to transparency and explainability. They are complex deep-learning models, such as GANs and diffusion models, making it difficult to understand how they generate specific outputs. This is particularly concerning given that synthetic data could be used for training downstream medical AI systems. Recent studies have begun to address important explainability questions, such as 1) what concepts are encoded in the internal representations of generative models^103,104, 2) which training samples significantly impact the quality of the generated data^105,106, and 3) how biases in generative models can be detected and mitigated¹⁰⁷.

Model transparency

We next examine model transparency in the development pipeline by describing how AI models trained on these datasets operate. Explainable AI (XAI), a subfield of AI, provides tools to investigate and improve AI model interpretability by explaining how models (1) process data, (2) make predictions, and (3) generate outcomes in clinical settings (Fig. 3a). These insights are crucial for identifying potential biases, errors, or unintended behaviors, enabling safer, more reliable integration of AI into medical workflows as well as more trustworthy models (Table 1; Box 2). Here, we provide an overview of XAI methods, grouped into key categories, and highlight representative examples of their applications in clinical contexts. This structure is intended to help readers identify techniques that are suitable for their particular needs. We also acknowledge that some methods may not be exclusive to a single category, and in such cases, we classify them into a category for which they were originally developed.

Fig. 3 ∣.

Key components of different model transparency frameworks. a, AI models are increasingly applied to diverse types of medical data for tasks such as classification, risk prediction, and segmentation. However, these models are often opaque, limiting our understanding of their underlying mechanisms. Explainable AI techniques aim to elucidate how these models work and improve their transparency. b, Feature attribution methods assign importance scores directly to the input features. c, Concept-based explanation methods aim to explain model behavior in terms of high-level, human-understandable concepts. d, Counterfactual models use generative models to produce slightly modified examples that flip the model’s prediction. By examining what changed, users can identify which concepts are important to the model’s decision. e, Inherently interpretable models are designed with interpretability built into their architecture or training process. f, Large complex models like LLMs pose new transparency challenges. One promising approach to improving their transparency is retrieval-augmented generation (RAG), which grounds model outputs in external sources. b-e, We highlight the output of each transparency framework in red. The chest X-ray image and retinal image were adapted from Wikimedia Commons, originally by Spideog and Tmhlee, respectively, under CC BY-SA 3.0 license.

Table 1 ∣.

Overview of model transparency methods. For each method, the proposed approach, applications, advantages, limitations, and a set of key references are provided.

Method	Approach	Medical applications	Advantages	Limitations	Refs
Feature attribution methods
GradCAM	Uses the spatial gradients of CNN layers	Highlighting regions in MRI images that are influential for brain tumor detection models234 Visualizing important features for COVID-19 prediction from CT-scans and chest X-rays235	Low computational overhead	Restricted to CNNs Requires access to model weights Gradient can be unintuitive	[109, 236]
Integrated Gradients	Integrates gradients from a baseline input to the actual input and computes the cumulative contribution	Discovering that retinal lesions are important for diabetic Retinopathy grade prediction from retinal fundus images111 Identifying spurious confounders that COVID-19 chest X-ray classifiers rely on21	Model-agnostic Grounded in axiomatic attribution theory	Sensitive to the baseline input selected Requires multiple gradient calculations	[111, 237]
LIME	Explains individual predictions of a black-box model by fitting an interpretable surrogate model (e.g., linear model or decision tree) to locally perturbed samples around the input instance	Identifying important patient characteristics for the prediction of heart disease and diabetes by six deep-learning algorithms120 Generating explanations for a deep learning model trained for retinoblastoma diagnosis122	Model-agnostic Flexibility in choosing specific surrogate model types, distance metrics, explanation units	Assumes that the complex model can be approximated locally using a linear surrogate, which may not always hold	[115]
SHAP	Averages the marginal contribution of a feature to the model prediction across all possible feature subsets	Identifying influential radiomic features that distinguish between mediastinal cysts and tumors from CT images123 Explaining model predictions for intraoperative hypoxaemia risk by attributing predicted risk scores to specific clinical features124	Strong theoretical grounding Satisfies useful properties Model-agnostic	High computational cost Sensitive to feature removal method	[118, 129-132]
Concept-based explanations
TCAV	Learns concept vectors by training linear classifiers to separate positive and negative samples and then computes the directional derivative of classifier predictions along that vector	Identifying influential concepts (e.g., microaneurysms, pan-retinal laser scars) for an AI model predicting diabetic retinopathy139 Interpreting important clinical concepts used by AI models trained for cardiac structure segmentation140 and prediction of acute kidney injury from EHR data141	Concepts can be expressed in terms of visual examples Quantifies concept influence	Requires manual selection and annotation of concepts, which is time-consuming	[139]
Automatic concept explanation	Utilize large-scale pretrained models (e.g., CLIP or image-captioning models) to obtain the concept annotations automatically from images and then use them for model explanation	Explaining dermatology AI models’ behavior in terms of medical concepts they use (e.g., papule, bullae, ulcer) without relying on manually concept annotation144	Human-interpretable explanations Enables zero-shot annotation	May inherit biases from pretrained models	[143, 149]
Counterfactual explanations
StylEx	Uses the latent space of a StyleGAN to manipulate concept-specific latent attributes and generates the corresponding counterfactuals	Identifying clinically-relevant and confounding features by analysing counterfactual images on different sides of the decision boundary for eight prediction tasks across three medical imaging modalities156	No need for predefined concepts or labels Explanations are based on visual differences, making them easier to interpret	Latent attributes can correspond to uninterpretable signals Multiple concepts can get encoded in a single latent attribute Generated counterfactuals may vary across training runs	[152]
EBPE	Uses a GAN-based generative model to progressively update a reference image which gradually changes the posterior probability to the negation class	Auditing five melanoma classifiers to reveal the features that they depend on154 Identifying signals that cause dermatology classifiers to predict protected attributes like sex155	Can directly control a predicted class probability Preserves features unrelated to classification decision	Generated Counterfactuals may vary across training runs	[153]
Inherently interpretable models
ProtoPNet	Learns a set of prototypical image patches and compares input regions to these prototypes for decision making.	Localizing important areas for classification of mass lesions from mammogram images170 Using a set of prototype images to identify important brain regions for Alzheimer’s disease classification from brain MRI scans163	Prototypes are actual image patches, making the model reason like humans Only image-level labels are needed for training	Learned prototypes may not align with concepts used by humans and are thus not directly interpretable Matching input features to many prototypes can be compute and memory intensive	[238]
Concept bottleneck model (CBM)	First predicts the concepts from the input using a deep learning model and then uses them to predict the target using a linear model	Training a linear model for diagnosing pediatric appendicitis from ultrasound images using concepts like surrounding tissue reaction, pathological lymph nodes extracted from a CNN168 Using a CBM with automatically annotated concepts for Melanoma detection144	Highly interpretable since the linear model weights directly quantify concept importance Easy to edit the model behavior (e.g., reduce reliance on spurious signal) by changing the corresponding weight in linear model	Needs a curated list of concepts along with concept labels to train the first stage	[164, 171]
Large language model transparency
Self explanation	Uses the inherent ability of LLMs to generate explanations for their outputs	Generating detailed rationale for medical question answering27,239	No additional training required Explanations are in natural language, making them accessible and user-friendly	Explanations may not be faithful to the model’s internal reasoning process	[177, 178]
Mechanistic interpretability	Aims to reverse-engineer LLMs to understand how they operate internally, such as what features they learned and what model components, such as neurons and attention heads, are important	Identifying how internal feature activations in LLMs reflect differential diagnostic reasoning in clinical prompts, such as representing preeclampsia or subarachnoid hemorrhage240	Systematic and fine-grained; can explain model behavior at the component level Enables hypothesis testing via internal interventions (e.g., suppressing specific features)	High computational and analytic overhead due to the significant number of model components to explain	[184-186, 241]
Retrieval augmented generation (RAG)	Augments LLMs with specific external information (like web documents) during inference to reduce hallucinations and improve explanations	Assisting clinical decision-making by retrieving up-to-date treatment guidelines and medical evidence during response generation242 Streamlining pharmaceutical regulatory compliance by aligning responses to specific guideline documents243	Improves factual consistency and reduces hallucinations Allows output trace-ability by linking to retrieved documents	Dependent on quality and coverage of the retrieval corpus, which may be noisy Real-time retrieval can increase inference time	[197, 198]

Box 2 ∣. Explainability and interpretability concepts.

Explainable AI concepts

Post hoc explanations.

Explanation methods applied to already-trained models to make their complex decision-making processes interpretable and understandable.

Feature attribution.

Methods that quantify the contribution of input features (e.g., pixels in an image or words in text) to a model’s prediction (e.g., disease probability).

Attribution map.

Visual representation of feature attributions shown as heatmaps. These maps use colors to highlight the features or regions of the input that strongly influence a model’s prediction, helping users easily localize important features. Concept-based explanations. Methods that explain the AI models’ behavior using human-understandable semantic concepts.

Counterfactual explanations.

Methods that explain the AI models’ behavior by showing how a model’s inputs would need to change to achieve a desired outcome from a machine learning model, providing insights into decision boundaries. Mechanistic interpretability. A reverse-engineering approach to understanding how neural networks operate internally, focusing on learned features (e.g., patterns or properties recognized by the model) and circuits (functional units performing computations). This is analogous to how brain functions are studied in neuroscience.

Inherently interpretable models.

Models designed to provide explanations alongside their predictions without requiring post hoc methods.

Chain-of-thought prompting.

A prompting method that guides AI models to solve complex tasks by breaking them into intermediate reasoning steps, mimicking human problem-solving processes. This can be achieved by providing the model with several examples of step-by-step reasoning during usage or simply instructing the model with prompts like, “Let’s think step by step.”

Retrieval-augmented generation (RAG).

A technique that enables language models to access external knowledge during text generation. Given a user query, the technique retrieves relevant documents from external sources and appends them to the query before passing it to the language models. This allows the model to leverage external information beyond what is encoded in its parameters, leading to more accurate responses. Also, it enhances the explainability of LLM-generated text by providing users with references used during the generation process.

Black-box model.

A machine learning system whose internal workings are not interpretable or accessible to the user. Model agnostic. Methods that can be applied to any machine learning model, regardless of its architecture or training regime.

Zero-shot annotation.

Process of labeling or classifying data without seeing any examples for that specific task. Hallucinations. Instances where the machine learning model (typically LLMs) generates incorrect or fabricated outputs that are not grounded in factual knowledge.

Feature attribution methods

Feature attribution methods address a fundamental question in model transparency: which input features contribute to a model’s prediction? These methods quantify the importance of input features, offering insights into the model’s decision-making process (Fig. 3b). Examples of such features include specific regions for medical images like X-rays and MRI scans, segments of waveform data like ECG, features from operating room signals, or words in electronic health record (EHR) text data.

One approach for quantifying the importance of input features is gradient-based saliency maps, i.e., a visual representation that highlights influential features by leveraging the gradients (partial derivatives) of the model’s output with respect to the input features. The magnitude of these gradients indicates how sensitive the model’s prediction is to changes in each input feature. Common methods for creating these maps include (1) taking the partial derivatives of the output with respect to the input and multiplying them by the input itself¹⁰⁸, (2) generating attribution maps based on the spatial feature gradients in convolutional layers associated with a particular class^109,110, and (3) computing the cumulative contribution of input features by integrating gradients along a path from a baseline input to the actual input¹¹¹.

Another approach, removal-based attribution, assesses feature importance by explicitly withholding one or more input features from the model and observing the impact on model predictions¹¹². For images, removing inputs corresponds to masking pixels or patches in the image by setting them to zero¹¹³ or applying blurring to them¹¹⁴. One such method that only requires black-box access to a classifier is called Local Interpretable Model-agnostic Explanations (LIME)¹¹⁵, which explains the individual predictions of a black-box model by approximating it with a simple, interpretable model, such as sparse linear model, that is locally faithful to the original model’s behavior. It does this by generating perturbed samples around an input instance, observing the model outputs, and using them to fit a surrogate model to estimate the importance of each feature. LIME is designed as a general framework that allows users to flexibly select the surrogate model class, similarity kernel, and fidelity loss function. It is model-agnostic—meaning it can be used with any classifier—and provides per-instance explanations rather than global interpretability. Another method, SHapley Additive exPlanations (SHAP), is a widely used removal-based method which uses concepts from cooperative game theory to fairly assign credit among input features¹¹⁶. Each feature is treated as a player in a cooperative game, and its contribution is calculated by averaging its marginal contribution to the model prediction across all possible subsets of the features. SHAP is renowned for its strong theoretical foundation, broad applicability, and well-documented implementations, making it a popular choice in healthcare¹¹⁷. Its theoretical grounding ensures useful properties like efficiency (attributions sum to the total prediction) and symmetry (identical features receive equal attributions). SHAP can be seen as a special case of LIME. If LIME uses the Shapley kernel as its kernel function and fits a linear model, it produces SHAP values^112,118. SHAP retains LIME’s model-agnostic local approximation strengths while also offering theoretical guarantees from game theory.

These approaches have all been applied to explain medical AI models. One study identified that deep learning systems to detect COVID-19 from chest radiographs rely on confounding factors rather than clinical signals²¹. Another study visualized the specific ECG leads and portions of ECG waves that were most influential for detecting myocardial infarction¹¹⁹. LIME has been used for revealing key features which are important for chronic diseases in older adults like heart disease and diabetes¹²⁰, for Alzheimer’s disease prediction¹²¹, and for retinoblastoma diagnosis¹²². Removal-based methods have been used to audit medical AI models trained for a variety of tasks, like distinguishing between mediastinal cysts and tumors¹²³, preventing hypoxemia during surgery¹²⁴, joint modeling of multiple neuropathological measures of Alzheimer’s disease¹²⁵, and estimating biological age from tabular hospital features¹²⁶.

However, feature attribution methods have inherent limitations. Gradient-based methods require access to the model weights, making them inapplicable to black-box models, which is the case for most of the deployed AI models. Gradients are also susceptible to adversarial manipulation¹²⁷ and the magnitude of the gradient can be unintuitive, since it relates to model sensitivity rather than quantifying feature importance. Additionally, some gradient-based methods have been shown to fail simple sanity checks and produce outputs similar to those of edge detectors¹²⁸. LIME explanations are based on an assumption that the model behaves linearly in a small neighborhood around the instance, which might not always hold for complex models. SHAP requires enumeration over many possible feature subsets, which can prove prohibitively expensive for larger datasets. Mitigating this cost has prompted the development of techniques focusing on practical approximations^129,130 and extensions to specific models like decision-trees¹³¹ and vision-transformers¹³².

Beyond these technical challenges, a key conceptual limitation of feature attribution methods is their inability to explain why a feature is influential. Although these methods can identify important features, the responsibility of interpreting the highlighted regions in an input image, e.g., whether it is important due to size, color, or another characteristic, ultimately falls on the users. Since users such as clinicians must independently derive meaningful insights from the highlighted features, these methods increase subjectivity and often place a cognitive burden on users^133,134. Furthermore, for some medical modalities, such as dermoscopic images of skin lesions, saliency maps often degrade to highlight the whole lesion, offering scant insight into model behavior.

Concept-based explanations

In medical practice, clinicians typically rely on high-level, semantically meaningful concepts to describe images and make predictions. For example, dermatologists describe lesions using concepts such as erythema, scale, pigmentation, and dome-shaped features^135,136. Explaining AI models directly in terms of these concepts aligns more closely with the clinical reasoning process than feature attribution methods.

To bridge the gap between feature importance and domain-specific knowledge that clinicians use for decision making, a different paradigm, i.e., concept-based explanations, has gained popularity. By attributing model predictions to high-level, human-accessible concepts, these methods enable more interpretable AI systems that are easier to integrate into clinical workflows (Fig. 3c). Several techniques facilitate concept-based explanations^137-139. One well-known approach is called Testing with Concept Activation Vectors (TCAV)¹³⁹; this approach first obtains concept activation vectors (CAVs), which are vector representations of a high-level concept in a neural network’s intermediate layers. To obtain a CAV of a specific concept, a linear classifier is trained to distinguish the activation of an intermediate layer for positive examples (e.g., 100 images containing the concept) and negative examples (e.g., 100 images without the concept). The weights of the trained classifier represent the CAV. Next, the model’s sensitivity to that concept is quantified by computing the directional derivative of its predictions along the CAV. TCAV has been previously applied to medical AI models. For example, the original study identified concepts like microaneurysms and pan-retinal laser scars as being important predictors of diabetic retinopathy levels¹³⁹. Subsequent studies used TCAV to interpret an AI model for segmenting cardiac structures¹⁴⁰ and time-series models for EHR data¹⁴¹.

One key limitation of TCAV is its reliance on manually selecting and annotating concepts, a time-consuming process that introduces subjectivity. To address this challenge, automatic annotation procedures have been developed. One method segments images at multiple resolutions and applies a clustering algorithm to intermediate representations of a classifier, grouping images with similar activations into concepts¹⁴²; however, this approach depends on segmentation accuracy and is limited to localizable concepts. Other techniques leverage vision-language foundation models, such as Contrastive Language-Image Pretraining (CLIP) models¹⁴³. These models are trained on a large set of image-caption pairs using a contrastive learning objective to learn a joint representation space; during training, the paired image and text are forced to be close in the joint representation space, whereas those from different pairs are forced to be far apart. The models have zero-shot capability, where concepts in a given image can be identified in terms of natural language without requiring additional training.

Extensions of CLIP methods to medical AI include the Medical Concept Retriever (MONET), a framework for leveraging medical CLIP models to enable large-scale concept annotation and explanations across the medical AI development pipeline¹⁴⁴. Medical CLIP models have been trained on PubMed articles^145,146 social media posts¹⁴⁷ and YouTube videos¹⁴⁸. These models can then be used to identify clinically relevant and spurious concepts of significance to disease classifiers; for example, MONET was used to audit a medical AI model trained to predict malignancy from dermoscopic lesions; it found that concepts related to lesion ‘redness’ led to the highest error rate, where malignant images were predominantly misclassified as benign. More recently, VisDiff¹⁴⁹ was proposed to identify concept differences between image sets without requiring predefined concepts using captioning models¹⁵⁰ and vision LLMs like GPT-4V. While originally developed in general domains, the method’s potential for medical application will grow as vision-language models improve their medical capabilities.

Counterfactual explanations

Since concept-based explanation methods rely on a pre-defined set of concepts, they might prove less robust in clinical settings where medical concepts are difficult to curate and require domain expertise. An alternative is counterfactual explanations¹⁵¹, which answer the question, “What changes to the input features would result in a different output of the medical AI model?” Answering this question is accomplished by generating and investigating counterfactual images— synthetic images that slightly alter attributes of a reference image in order to cross the model’s decision boundary and flip the model’s prediction (Fig. 3d). Examining the difference between the reference image and the counterfactual image, we can infer that the classifier uses the differing visual signals as part of its reasoning process. Counterfactual explanations provide specific and actionable insights into the causal interpretation of important concepts: we learn which concepts must be altered in the images to elicit a different prediction from the classifier.

One approach to generating counterfactuals, called StylEx, identifies classifier-specific attributes in the latent space of a StyleGAN and manipulates them to analyze how the generated image affects the classifier’s output probability¹⁵². Another approach uses a GAN-based generative model to produce a progressive set of plausible variations of a reference image, each of which gradually changes the posterior probability from its original class to its negation¹⁵³. Extensions of these approaches have been used to identify important visual concepts in medical AI classifiers. One study used counterfactual analysis along with insights from board-certified dermatologists to rigorously audit five melanoma classifiers. The framework revealed that the classifiers rely both on features used by clinicians, such as lesional pigmentation patterns, and on undesirable features, such as background skin texture and color balance¹⁵⁴. Using a similar framework, a different study identified signals which caused classifiers to predict protected attributes like sex with unexpectedly high performance, which is surprising since the task is difficult even for trained clinicians¹⁵⁵. Another study demonstrated the broad applicability of counterfactual visualizations on eight different prediction tasks across three medical imaging modalities, including retinal fundus images, eye photographs, and chest X-rays; the study revealed clinically known features (e.g., types of cataract and enlarged heart) as being significant while also identifying confounders, like the correlation between chest X-ray underexposure with abnormality prediction as well as eye makeup’s correlation with the prediction of low hemoglobin levels¹⁵⁶.

One limitation of counterfactual explanations is the potential for the Rashomon effect, i.e., given the same input image, different training runs of the generative model can produce differing counterfactuals that can be valid in terms of crossing the decision boundary to get a different model prediction^151,157. This can cause different concepts to be altered across different runs since multiple concepts can lead to similar changes in model output, introducing ambiguity about which concepts are truly important for classifier predictions.

Inherently interpretable models

The explanation methods discussed so far operate in a post hoc manner, attempting to interpret black-box models after training. However, these approaches might be challenging to use effectively without deep technical expertise. Clinicians, in particular, would find it easier and more practically useful to work with models that are inherently interpretable since they are easier to analyze and understand. A classic example is the linear classifier, which makes predictions as a weighted sum of input features. The learned weights indicate whether each feature affects positively or negatively toward the model output, making the model’s internal workings transparent. Other simple models with similar benefits include decision trees^158,159 and generalized additive models^160,161. When such simple models achieve performance comparable to that of complex models¹⁶², they are generally preferred due to their interpretability. However, deep learning models capable of learning complex patterns from data and delivering high performance are becoming more widely available in clinical settings. To bridge the gap between performance and interpretability, recent research has explored deep learning architectures that are inherently interpretable.

One such example is the Prototypical Part Network (ProtoPNet), which is designed for image classification¹⁶³. Instead of relying on dense feature representations, ProtoPNet learns a set of visual prototypes, which are small image patches that represent key parts of each class. During inference, the model compares parts of the input image to these learned prototypes and makes a prediction based on weighted similarity scores with the prototypes. This allows the predictions to be visually and semantically explainable, as the model shows the prototype that influenced its decision. However, a limitation of this approach is that the prototypes might not align with human-defined concepts, making them difficult to interpret.

Concept bottleneck models (CBMs) are another popular approach to develop inherently interpretable models¹⁶⁴. Unlike complex black-box models, CBMs make predictions in two stages: (1) they predict concepts from input using models like Convolutional Neural Networks (CNNs) and then (2) use these predicted concepts to predict the target output via a linear model (Fig. 3e). Since each node in the bottleneck layer corresponds to a human-defined concept, the linear model’s associated weights show how each concept affects the model output. CBMs have been applied in varied domains, ranging from general vision and natural language processing tasks^165-167 to medical applications^135,168,169. An extension of this technique was used in a study to predict malignancy from mass lesions in mammograms and identified clinically relevant features leveraged by the model, e.g., mass margin characteristics¹⁷⁰.

However, CBMs are limited in that they require a well-curated concept list and dense concept annotations in the training data to learn the bottleneck layer. These limitations have been addressed by prior work that automates concept annotation using CLIP-based models^144,171. MONET¹⁴⁴, for example, trained a CBM for melanoma prediction using concept scores that were automatically calculated; it found that CBM performance matches that of black-box models and that concept weights align well with prior medical knowledge in the dermatology domain (e.g., asymmetry, erosion, multiple colors).

Large language model transparency

The AI field is seeing rapid advances in the development of large language models (LLMs), like ChatGPT, Claude, Gemini, and LLaMA, that can perform a wide range of natural language tasks. Medical LLMs are developed by fine-tuning these base general-domain LLMs on medical datasets, such as MedQA (USMLE-style questions)¹⁷² and PubMed articles^27,173. Such LLMs can support various stages of clinical workflows, including intake tasks (e.g., chatbots that help patients prepare for medical visits), core clinical tasks (e.g., decision-support systems or talk therapy), and discharge tasks (e.g., clinical report generation or insurance claim letter drafting)^174,175.

Though the unprecedented scale and capabilities of LLMs open exciting new opportunities for medical AI, they also introduce new challenges in achieving transparency. Since they are highly complex transformer-based models with billions of parameters, applying XAI methods is inherently computationally burdensome. In addition, LLMs are generative models that produce text stochastically–meaning their outputs can vary across runs even when given the same input–further complicating the application of traditional XAI methods. Finally, LLMs demonstrate reasoning abilities beyond the simple recognition of patterns, requiring explanation techniques that offer deeper insights into their behavior¹⁷⁶.

Self-explanation.

A widely discussed approach to enhancing transparency in LLMs is self-explanation. This technique leverages the inherent ability of LLMs to generate explanations for their outputs by eliciting intermediate reasoning steps¹⁷⁷. Techniques like chain-of-thought prompting can guide LLMs to break their decision-making process into discrete steps¹⁷⁸. For example, users can simply ask the model “Why do you think so?” to obtain a self-generated explanation. However, though these explanations may seem plausible and user friendly, they are not always faithful to the model’s actual internal workings: LLMs are often trained on human-generated text and optimized to align with human preferences, which means their explanations are designed to seem reasonable rather than accurately reflect the models’ underlying mechanism^179-181.

Future directions could explore incorporating faithfulness to model’s internal reasoning in addition to human preference in the model training process. Doing so would ensure that explanations align more closely with the model’s true reasoning^182,183.

Mechanistic interpretability.

Another line of research focuses on mechanistic interpretability, which aims to reverse-engineer LLMs to understand how specific components (e.g., neurons, layers, attention heads) affect model behavior^184-187. While some progress has been made in applying mechanistic interpretability to biomedical models^188,189, current efforts focus more on advancing scientific understanding than on providing actionable explanations for real-world use, and their applicability to clinical settings has been less explored. Challenges include computational burden due to the significant number of large LLM components and the need for explanations to address the myriad individual components. These factors currently limit the practicality of this approach for clinical use—where clear, concise, and actionable explanations are essential—but future work may help bridge this gap by making the techniques more scalable and user-facing.

Retrieval-augmented generation.

Another promising direction for increasing LLM transparency involves grounding LLM outputs in specified external evidence, although this method does not focus on explaining the model itself^190-196. Though LLMs demonstrate deeper knowledge as their size and training data increase, they continue to suffer from hallucinations–phenomenon where the model generates plausible-sounding but factually incorrect or fabricated information—and lack fine-grained, domain-specific knowledge. A well-known technique to address these limitations is retrieval-augmented generation (RAG)^197,198: instead of relying solely on the LLM’s internal knowledge, RAG augments LLMs with specific external information during inference (Fig. 3f). When a user task is received, the AI system first looks up relevant documents and sends retrieved ones to the LLM, which can then generate responses that include references to a specified set of documents. In medicine, this approach is particularly valuable since it grounds model outputs in clinical guidelines and facilitates fact-checking by helping users verify the referenced material. By anchoring responses in trusted external sources, RAG enhances medical AI transparency and reliability.

Evaluation of explanation methods

Despite the growing number of explanation methods, systematically evaluating and comparing them remains a key challenge. Evaluation is essential both for researchers developing new techniques and for practitioners selecting appropriate methods for their specific applications. In this section, we outline the commonly used evaluation criteria—faithfulness and plausibility—along with representative metrics used for each.

The first evaluation criterion is faithfulness, which refers to how accurately an explanation reflects the decision-making process of models. In other words, a faithful explanation should highlight features that the model genuinely relies on for prediction. Several quantitative metrics have been developed to evaluate faithfulness. The insertion/deletion metric¹¹³ measures how the model’s output changes as important features are added or removed from the input. The sensitivity-n metric¹⁹⁹ measures how well the sum of attribution aligns with the variation in the model output when subsets of features are withheld. Another approach is ROAR²⁰⁰, which retrains models on data where features identified as important are removed. A significant drop in performance after retraining suggests that the removed features were indeed relevant.

The second evaluation criterion is plausibility, which assesses how well the explanation aligns with human intuition or domain expertise. One evaluation technique is the “pointing game”²⁰¹, in which the explanation is compared to ground-truth annotations, such as bounding boxes in images. Alternatively, plausibility can be evaluated qualitatively through user studies, such as surveys or interviews, where domain experts assess whether the explanation looks reasonable¹⁰⁹. These evaluation methods, however, rely on a key assumption that the features a model uses are similar to those used by humans. This assumption may not always hold, which means that low plausibility scores do not necessarily indicate that an explanation is incorrect—only that it may be different from human expectations.

In summary, a variety of criteria and metrics are available for evaluating explanation methods. The relative importance of factors, such as faithfulness, plausibility, or computational cost, may depend on the specific use case— whether for model debugging or building end-user trust. As a result, no single explanation method is universally optimal; the choice of evaluation metrics should be guided by the goals and constraints of the particular application context.

Deployment transparency

Beyond the challenges of transparency during AI development, ensuring transparency in deployment is equally critical for medical AI. Transparency in deployment extends beyond understanding how AI models operate—it requires clear and accessible reporting on their real-world performance, safety, and clinical impact. Even models that perform well in controlled environments can face unforeseen challenges once integrated into complex healthcare systems, where multiple stakeholders interact with AI in dynamic and unpredictable ways. One major challenge arises from the interaction between users and AI systems. Clinicians may under-rely on AI, dismissing valuable insights, or over-rely on its predictions without sufficient oversight. Furthermore, real-world clinical environments are not static. Shifts in patient demographics, disease prevalence, and data acquisition methods can introduce biases, degrade performance, and create new safety risks. Without ongoing transparency and monitoring, such issues can go undetected, leading to unintended clinical consequences. This section explores three critical aspects of transparent medical AI deployment. We first examine the need to rigorously evaluate AI models in real-world settings to ensure they provide meaningful improvements in clinical outcomes. Next, we discuss the importance of continuous monitoring and model updates to detect failures and maintain system reliability. Finally, we review regulatory frameworks designed to ensure transparency in deployed medical AI systems by enforcing safety and accountability standards.

Evaluating the impact of medical AI in clinical practice

Though usually developed in siloed and controlled settings, medical AI models are deployed as part of multi-faceted systems with varied stakeholders in uncontrolled environments²⁰². Many studies show that problems with medical AI models are more often related to how the device is used or with issues concerning data rather than with flaws in the algorithms ^203-205. For example, in diagnostic imaging, an AI model may flag potential abnormalities, but the final interpretation and decision-making are the responsibility of a radiologist to minimize errors such as false positives or negatives. Even with a “perfect” model, clinicians unfamiliar with its use can misdiagnose or impose other serious consequences. These findings highlight the need to assess how effectively the AI interacts with clinicians and integrates into clinical workflows^206,207. Thus, real-world evaluation is essential to maintain transparency regarding the AI systems’ effectiveness and clinical impact.

To assess whether AI models translate into meaningful improvements in patient care, several studies have evaluated their real-world impact across different specialties. One work assessed the ability of dermatologists to detect melanoma risk from skin lesion images before and after exposure to an AI algorithm²⁰⁸; authors observed that the dermatologists’ ability improved significantly after AI exposure and that such exposure had an equivalent or even higher net benefit compared to performing biopsies of all lesions. Other studies (1) conducted a population-based trial to assess the non-inferiority of cancer detection within 3 months of mammography by one radiologist plus an AI model compared with double reading by two radiologists^209,210; replacing one radiologist with an AI yielded a 4% higher non-inferior cancer detection rate, and (2) analyzed the real-world applicability and generalizability of an AI-based triaging system for chest X-rays across a diverse demographic cohort, observing a 77% faster turnaround time compared to radiologists²¹¹. Such studies highlight the necessity of evaluating medical AI at a systemic level to gain deeper insights into how models integrate into clinical workflows and the extent to which they translate into measurable improvements in real-world clinical outcomes.

Continuous monitoring and model updating

Medical AI models deployed in clinical settings must adapt to ever-changing environments²¹². Medical environments are dynamic, with evolving patient cohort demographics, newer diseases, shifts in input acquisition techniques, and changes in treatment protocols that can cause a distribution shift in the data, in turn resulting in fluctuations in AI system performance. Without proactive monitoring, these changes can introduce biases, degrade performance, and compromise patient safety, ultimately leading to reduced transparency.

Continuous monitoring of input data quality and model behavior enables early detection of such distribution shifts¹³¹, facilitating timely model updates via retraining, fine-tuning, or other adaptive techniques^213,214. Despite its importance, continuous monitoring remains underutilized in practice. A prior study reviewing 43 predictive medical AI tools for primary care—including 25 from regulatory databases and 18 from peer-reviewed literature—found that only 2% provided evidence of deployment monitoring. This mismatch between expectations and reality has driven discussions within the medical AI community about the urgency of incorporating continuous monitoring to maintain required levels of performance and transparency²¹⁵.

Beyond tracking system health, continuous monitoring serves as a cornerstone for developing robust and safe medical AI via continual model updating^214,216. For example, continual learning, also known as lifelong learning, enables medical AI models to integrate new clinical data while retaining previously learned knowledge²¹⁷. The insights gained from continuous monitoring can be fed back into the model training process, allowing models to dynamically adapt and improve^17,66,67,155. Thus, continuous monitoring is not just a diagnostic tool but also serves as a pathway for improving long-term AI safety and clinical effectiveness.

Regulatory frameworks for medical AI systems

As AI-driven medical tools become increasingly integrated into clinical practice, ensuring transparency via regulatory frameworks is essential for maintaining trust, safety, and accountability. Similar to traditional medical devices, medical AI systems undergo rigorous regulatory assessment, with a growing emphasis on transparency^31,218. In the U.S., the FDA regulates AI-based medical tools under the Software as a Medical Device (SaMD) framework, which outlines key requirements for transparency, performance evaluation, test data quality, and continuous monitoring^10,219,220.

Over time, the FDA has refined its regulatory approach to medical AI by introducing several key frameworks to promote ongoing evaluation and transparency. The agency has long promoted a Total Product Lifecycle (TPLC) approach, which emphasizes continuous evaluation from development to post-market surveillance. Under TPLC, AI systems must undergo premarket evaluation, real-world performance monitoring, and adaptive updates to ensure safety and reliability over time. This framework acknowledges that AI models are dynamic and must evolve with changing clinical conditions. In 2021, the FDA collaborated with Health Canada and the United Kingdom’s Medicines and Healthcare products Regulatory Agency (MHRA) to introduce Good Machine Learning Practice (GMLP) for medical devices. GMLP emphasizes transparency by requiring that users—including healthcare providers and patients— have access to clear, essential information, such as a device’s intended use, performance metrics, data characteristics, acceptable inputs, limitations, and user interface interpretation. Also, GMLP highlights the role of the human-AI team, shifting the focus from isolated model performance to real-world team effectiveness in clinical workflows. Furthermore, GMLP mentions that deployed models must be continuously monitored in real-world use, with mechanisms to track and explain performance changes over time.²²¹. Most recently, in 2024, the FDA, Health Canada, and MHRA have established guiding principles focused on transparency—Transparency in Machine Learning-Enabled Medical Devices (MLMDs). These principles require that critical information, such as intended use, performance, risks, and decision logic, is clearly communicated to relevant stakeholders, emphasizing explainability and proactive risk management²⁰⁷.

Regulatory oversight of medical AI is a global effort, with agencies worldwide introducing frameworks to enhance transparency and accountability. To provide a broader perspective on these developments, Fig. 4 shows a chronological timeline of all the major guidelines for medical AI established by governmental institutions around the world. Collaborative efforts across stakeholders and countries are essential to establishing comprehensive guidelines that ensure medical AI models remain transparent, adaptable, and aligned with patient-centered care.

Fig. 4 ∣.

Over the years, regulatory bodies across the world, including the EU, FDA, WHO, and US Congress, have established institutional guidelines for appropriate deployment of medical AI systems. These frameworks address different aspects such as data privacy, ethical principles, transparency, safety, and reliability. As medical AI systems continue to evolve rapidly, these guidelines also need to be regularly updated to reflect the emerging capabilities and challenges.

Outlook

We next explore further opportunities and challenges in advancing transparency in medical AI systems. We focus on addressing transparency issues posed by emerging model types, overcoming barriers to adopting transparency tools in clinical practice, and evaluating the real-world clinical utility of explainability methods.

Emerging model types

Democratization of large models

The immense scale of foundation models makes their training prohibitively expensive, and training is thus affordable only for a select few well-funded organizations. Further, deploying these models on local machines requires high-performance GPU clusters, resources which are often unavailable in many medical institutions^175,222,223. This issue is especially prominent in low-resource healthcare settings, which lack computing resources and IT professionals, further exacerbating existing healthcare disparities. Consequently, most medical institutions may need to rely on proprietary LLMs accessed via application programming interfaces (APIs).

The reliance on these external systems poses significant transparency and privacy issues. Many high-performing LLMs are closed-source and do not disclose details about their training methods or datasets, limiting their transparency and trustworthiness^224,225. Moreover, with API access only, it would be extremely difficult to ascertain how the model is processing patient data on the backend. For example, entering a patient’s health information into the AI chatbot systems hosted by a third party raises the risk of protected health information (PHI) breaches and potential HIPAA violations²²⁶. Additionally, closed-source models are prone to unpredictable changes. The lack of archived versions for regulatory review and dependency on proprietary vendors make closed models pose challenges for long-term medical deployment.

To address these challenges, efforts are being made to democratize the use of large models through techniques such as model distillation and efficient inference. Model distillation aims to create smaller, lightweight versions of large language models that retain much of the original model’s performance while being more resource-efficient²²⁷. Efficient inference techniques, such as model quantization and pruning as well as on-device inference engines, further enhance accessibility by enabling models to run locally on less powerful hardware, reducing dependency on external APIs^228,229. By enabling local deployment, these approaches allow healthcare institutions to maintain control over their data and processing workflows.

Regulating emerging model types

Regulating emerging AI models such as LLMs presents unprecedented challenges¹⁷⁴. Unlike traditional AI models with fixed functionality, LLMs are general-purpose models that can be adapted to various tasks, making it difficult to define their intended use—a critical component of current regulatory frameworks. Also, their performance varies depending on which prompts are used, complicating standardized evaluation. Additionally, the vast, complex, and often proprietary nature of their training data makes auditing challenging. Recent proposals advocate for creating a distinct regulatory category for LLM to ensure their transparent and accountable deployment²²². Furthermore, novel regulatory strategies are being explored, including treating LLMs similarly to human clinicians by applying comparable methods, such as periodic evaluations, supervised clinical use, and public reporting of performance²³⁰.

Integrating transparency tools in clinical practice

Addressing the computational cost of XAI methods in resource-constrained environments

The computational overhead of XAI methods poses a hurdle to their integration into clinical practice. Many widely used XAI techniques, such as SHAP and LIME, require post hoc analysis that involves additional computational steps. This increases latency and resource requirements, making these methods challenging to deploy in time-sensitive clinical settings where real-time explanations are essential. In critical scenarios, such as emergency diagnostics, clinicians lack time to wait until the output of XAI methods is ready. To overcome these constraints, researchers and engineers must prioritize the development of lightweight, real-time explainability techniques and optimize models for efficient deployment on resource-limited systems^{129,132,231-233}.

Clinician-centered integration

Seamlessly embedding transparency tools into clinical workflows is essential for ensuring their practical utility in real-world healthcare settings. A key strategy is integrating these tools with electronic health record (EHR) systems, enabling clinicians to access explanations directly within their existing workflows. To maximize usability, the design of explainability interfaces should be intuitive and tailored to clinicians’ needs. Features like interactive visualization dashboards, voice-assisted tools, and concise textual summaries can make explanations more accessible and actionable. For example, a visualization dashboard might highlight critical regions in medical images, while a voice-assisted tool could provide real-time explanations during patient consultations. These designs should minimize clinicians’ cognitive load, ensuring they enhance rather than complicate clinical workflows.

Education and training

Educating clinicians about the strengths and limitations of explainability methods is critical to their effective use in clinical practice. Explainability methods are not perfect, and it is essential to ensure that clinicians do not become overly reliant on or overly confident in medical AI systems based solely on the outputs of these methods. For example, a saliency map might highlight a skin lesion as the most influential region for a model’s prediction while ignoring surrounding artifacts, like ruler marks or color charts. Although this may appear reassuring, it does not guarantee that the model is making decisions based on clinically relevant features in a way that aligns with clinicians’ expectations. Training programs should emphasize the importance of understanding explainability tools and recognizing their limitations. Clinicians should be equipped with the knowledge to critically evaluate AI-generated explanations and incorporate them as supplementary insights rather than definitive answers.

Evaluating the clinical utility of explainability

Transparency in medical AI is widely regarded as a means to enhance trust, reduce bias, and improve accountability. However, quantifying its real-world impact is not straightforward. While explainability is often cited as critical for trust and adoption, studying its direct impact on clinical utility through prospective trials remains underexplored. For example, does a more interpretable model lead to improved decision-making by healthcare providers or a reduction in diagnostic errors? Most existing studies focus on theoretical benefits or rely on retrospective data, offering limited insights into real-world outcomes.

Key questions remain unanswered. Can clinicians effectively utilize explainability tools in time-constrained settings? Do these tools genuinely improve trust, or do they inadvertently add to the cognitive burden of already overwhelmed healthcare providers? What types of explanations are the most beneficial for specific clinical outcomes? Addressing these questions requires robust evaluation frameworks and benchmarks that directly measure the impact of explainability in clinical practice. These frameworks could incorporate metrics such as improvements in diagnostic accuracy, reductions in time-to-decision, and enhanced collaboration between clinicians and AI systems and between clinicians and their patients. Additional metrics, including clinician trust scores, patient satisfaction, regulatory adherence, and AI adoption rates, could serve as proxies for the practical benefits of transparency. Gathering real-world evidence will be critical to ensuring that explainability methods achieve their intended goals in clinical practice.

Key points.

• As AI systems play an increasingly significant role in clinical decision-making, ensuring transparency in their design, operation, and outcomes is essential for their safe and effective deployment and for building trust among stakeholders.

• Achieving transparency requires a holistic approach that spans the entire development pipeline: from data collection and model development to clinical deployment.

• Explainable AI techniques, including feature attributions, concept-based explanations, and counterfactual explanations, elucidate how medical AI models process data and make clinical predictions.

• Transparent deployment of medical AI systems demands rigorous real-world evaluation, continuous performance monitoring, and evolving regulatory frameworks to maintain safety, reliability, and clinical impact over time.

• Advancing transparency further requires democratizing access to large models, integrating transparency tools into clinical workflows, and systematically evaluating their clinical utility.

Citation diversity statement

We acknowledge that papers authored by scholars from historically excluded groups are systematically under-cited. Here, we have made every attempt to reference relevant papers in a manner that is equitable in terms of racial, ethnic, gender and geographical representation.