Explainable artificial intelligence models using real-world electronic health record data: a systematic scoping review

Abstract

Objective

To conduct a systematic scoping review of explainable artificial intelligence (XAI) models that use real-world electronic health record data, categorize these techniques according to different biomedical applications, identify gaps of current studies, and suggest future research directions.

Materials and Methods

We searched MEDLINE, IEEE Xplore, and the Association for Computing Machinery (ACM) Digital Library to identify relevant papers published between January 1, 2009 and May 1, 2019. We summarized these studies based on the year of publication, prediction tasks, machine learning algorithm, dataset(s) used to build the models, the scope, category, and evaluation of the XAI methods. We further assessed the reproducibility of the studies in terms of the availability of data and code and discussed open issues and challenges.

Results

Forty-two articles were included in this review. We reported the research trend and most-studied diseases. We grouped XAI methods into 5 categories: knowledge distillation and rule extraction (N = 13), intrinsically interpretable models (N = 9), data dimensionality reduction (N = 8), attention mechanism (N = 7), and feature interaction and importance (N = 5).

Discussion

XAI evaluation is an open issue that requires a deeper focus in the case of medical applications. We also discuss the importance of reproducibility of research work in this field, as well as the challenges and opportunities of XAI from 2 medical professionals’ point of view.

Conclusion

Based on our review, we found that XAI evaluation in medicine has not been adequately and formally practiced. Reproducibility remains a critical concern. Ample opportunities exist to advance XAI research in medicine.

INTRODUCTION

The emergence of modern data-rich technologies will require physicians to interpret high-dimensional heterogeneous medical data while also making efficient and accurate decisions for diagnosis and treatment.¹ Artificial intelligence (AI) techniques are critical tools that can assist physicians with such analyses and decision-making.² Referring to Norvig and Russel’s classic AI textbook,³ in this review article we define AI as acting humanly through machine learning (ML) and, more specifically, ML-based predictive analytics.

Szolovits⁴ defines AI in medicine (AIM) as “AI specialized to medical application.” In recent years, AIM has contributed to healthcare in the light of digitized health data.⁵ The wide adoption of electronic health record (EHR) systems by healthcare organizations and subsequent availability of large collections of EHR data have made the application of AIM more feasible.^6,7 EHR data contain rich, longitudinal, and patient-specific information including both structured data (eg, patient demographics, diagnoses, procedures) as well as unstructured data, such as physician notes, among other clinical narratives.⁸

Despite their promising performance, the production of AIM systems for actual clinical use is challenging.⁹ A survey of medical professionals in 2018 showed a lack of trust in AIM.¹⁰ Limited access to large data, lack of integration to clinical workflows, and, especially, the ambiguity of requirements for regulatory compliance are among the development and deployment challenges of AIM systems.¹¹ In 2017, the Defense Advanced Research Projects Agency (DARPA) released a public update report of their research program on explainable AI (XAI).¹² They reported that the new generation of AI systems have limited effectiveness due to the inability of humans to understand why an AI system makes particular decisions.

General Data Protection Regulation (GDPR) in Europe is an example of the increasing needs for XAI from a regulatory perspective. This regulation is a data protection and privacy law for all citizens of the European Union, which regulates any organization that uses personal data including EHRs of European Union residents for automated decision-making. Among other regulations, it requires organizations to provide meaningful explanations about how the algorithm reaches its final decisions.¹³ However, since there is no concrete formulation and quantification of what an adequate explanation should be, regulatory enforcement seems challenging in this context.

Some researchers argue that if physicians could rely on drugs like aspirin despite the fact that their underlying mechanism was unknown, should they expect AI to give explanations if its performance is promising?¹⁴ On the other hand, drugs have to go through rigorously designed and conducted randomized clinical trials for regulatory approval before production. Post-marketing surveillance allows regulatory agencies, such as the Food and Drug Administration in the US, to withdraw them from the market in cases of serious adverse events. AI, built inside labs using potentially biased and limited data with challenges like generalization to new samples, does not have comparable mechanisms to ensure efficacy and safety in the real world. XAI helps to understand whether AIM decisions are valid and come to a consensus with medical professionals and as a result, promote their trust in AIM.¹⁵ Thus, XAI for medicine is of vital importance to support the implementation of AI in clinical decision support systems.^16,17

The increasing capabilities of AIM married to the necessity of XAI demand a review of the state-of-the-art research in the field. Our review summarizes a decade of research on the enhancement of interpretability in EHR-based AIM. We aim to provide insights into the current research trend by categorizing ML methods, XAI approaches, and targeted ML prediction tasks to identify potential gaps and suggest future research direction in the field. We also assess the reproducibility of the included studies. Finally, we review and evaluate the studies from the medical professional’s perspective on their interpretability enhancement deliverables.

METHOD

Literature selection strategy

We conducted a systematic scoping review of XAI on EHR data using MEDLINE, Web of Science, IEEE Xplore, and Association for Computing Machinery (ACM) databases based on the Preferred Reporting Items for Systematic Review and Meta-Analyses (PRISMA) framework¹⁸ to search studies published between January 1, 2009 and May 1, 2019. In this article, we refer to explainable ML methods that are used for predictive analytics as XAI. We used Covidence—a systematic review management system—to conduct this systematic scoping review. We considered different combinations of relevant search keywords in Table 1.

Table 1.

The search queries

Database	Query	# initial results
MEDLINE and Web of Science (via Covidence software)	(explainable OR explainability OR interpretable OR interpretability OR understandable OR understandability OR comprehensible OR comprehensibility OR intelligible) AND (machine learning OR artificial intelligence OR prediction model OR predictive model OR deep learning OR AI OR neural network)	1487
IEEE Xplore		2208
ACM digital library		2734

Abbreviations: AI, artificial intelligence.

We found 6429 articles from MEDLINE, Web of Science, IEEE Xplore digital library, and ACM digital library. After removing 651 duplicates, 2 authors (SNP and ZC) screened the titles and abstracts of the remaining 5778 studies based on a set of inclusion and exclusion criteria (see Supplementary Appendix Table 1). They retained 157 articles for full-text review and deemed 42 articles relevant to include in the final full-text extraction. The PRISMA flow chart is depicted in Figure 1.

Figure 1.

The PRISMA diagram depicts the number of records identified, included and excluded, and the reasons for exclusions.

The rest of the study team supervised the screening process, resolved conflicts, and provided clarifications based on their expertise. Since this review focuses on EHR data, studies based on any data other than EHR are not included. Building predictive models using medical images and electroencephalogram data, for instance, do not share similar characteristics with those based on patient EHRs. We refer interested readers to other existing survey literature such as those about AI in medical imaging^19,20 and electroencephalogram signal processing.²¹

Data extraction

We considered the following aspects when evaluating the full text of the 42 articles (referred to as “the articles” throughout the paper) included in this paper: 1) year of publication; 2) ML prediction tasks (eg, incident of a disease, mortality, re-admission, risk assessment); 3) ML algorithm; 4) XAI method; 5) the dataset used to build the model; 6) scope of XAI method (ie, intrinsic/posthoc, local/global, model-specific/model-agnostic); 7) category of XAI method (ie, feature interaction and importance, attention mechanism, data dimensionality reduction, knowledge distillation and rule extraction, and intrinsically interpretable models); and 8) evaluation of XAI method. We also assessed the articles in terms of reproducibility based on 2 objective criteria: 1) if the datasets are accessible to the public (ie, proprietary or not), and 2) if the availability of the source codes/implementations is explicitly mentioned in the manuscript or in the supplementary material.

We reviewed XAI methods in the articles and 2 medical professionals (PRM and JC) evaluated the clinical utility of these methods. This can help identify the potential perception gaps between model builders and the end users of the models. We also identified open issues and challenges in XAI that can serve as suggestions for future work in the field. To the best of our knowledge, this paper is the first attempt to review XAI in AIM with real-world EHR data.

RESULTS

Research trends

We have seen a surge of XAI studies in AIM applications using EHR data since 2015, with only a small number of studies from 2009 to 2011, as shown in Figure 2. The limited number of publications indicates there is a demand for more research focused on XAI in biomedical applications using EHR data.

Figure 2.

Publication trend of XAI studies using EHR data between January 1st, 2009 and May 1st, 2019.

Prediction tasks, methods, and datasets

As of 2017, cardiovascular diseases, cancer, diabetes, and Alzheimer’s disease are the leading causes of death in the United States.²² The majority of the articles (∼60%) in our review focused on 1 or more of these diseases. Table 2 lists all the articles, the ML methods, the ML prediction tasks, and the datasets. Researchers used different ML methods in the articles including: 1) logistic regression (LR), 2) support vector machines (SVM), 3) decision trees (DT), 4) ensemble, 5) Bayesian networks, 6) fuzzy logic, 7) deep learning (DL), and 8) other approaches.

Table 2.

All the articles grouped based on the ML method along with the associated medical prediction task(s)

ML Method	Prediction task(s)	Dataset(s)a	Article(s)
Logistic regression	Incidence of Medium-chain acyl-coA dehydrogenase deficiency	A systematic newborn screening by the PCMA screening center (Belgium)	Van den Bulcke et al23
	In-hospital mortality (all-cause)/ hospital-acquired infections/ICU admissions/development of pressure ulcers during the patient’s stay	Premier healthcare EHR data	Fejza et al24
Support vector machines	Incidence of diabetes mellitus	A diabetes dataset in Oman25	Barakat et al26
	Incidence of leukemia/prostate cancer/colon cancer	Unnamed datasets27–29	Hajiloo et al30
	Gut and skin microbiota/ inflammatory bowel diseases	Unnamed dataset31	Eck et al32
	Incidence of type 2 diabetes	Federazione Italiana Medici di Medicina Generale	Bernardini et al33
	Hospitalization due to heart diseases or diabetes	Boston Medical Center	Brisimi et al34
Decision trees	Protein solubility and gene expressions	40 datasets in the University of California Irvine (UCI) repository, Solubility database of all E. coli proteins, and 9 Gene Expression Machine Learning Repository datasets	Stiglic et al35
Ensemble	Risk of developing Type 2 diabetes	Practice Fusion Diabetes Classification Dataset	Luo36
	Stage of acute myeloid leukemia/breast invasive carcinoma	The Cancer Genome Atlas	Jalali and Pfeifer37
	Incidence of central precocious puberty in girls	Pediatric Day Ward of the Endocrinology Department at Guangzhou Women and Children’s Medical Center	Pan et al38
	Incidence of multiple diseases	13 data sets of life sciences in the UCI Repository	Valdes et al39
	Adverse drug events	Stockholm electronic patient record corpus (HealthBank)	Crielaard and Papapetrou40
	Drug side effect	Side Effect Resource 4	Zhang et al41
Bayesian networks	Incidence of heart disease, fetal pathologies	3 heart disease datasets in the UCI repository	Bouktif et al42
Fuzzy logic	In-hospital mortality (all-cause)	MIMIC-III, Diabetes, Heart Disease and Liver datasets in the UCI repository	Davoodi and Moradi43
	Incidence of type 2 diabetes	Pima Indian Dataset in UCI repository	Settouti et al44
Deep learning	Splice site detection	Unnamed datasets45–47	Zuallaert et al48
	Hospital readmission due to heart failure	Congestive Heart Failure	Xiao et al49
	Illness severity/ in-hospital mortality	University of Florida Health, MIMIC-III	Shickel et al50
	Heart failure/cataract	Health Insurance Review and Assessment National Patient Samples (Republic of Korea)	Kwon et al51
	Cell-type specific enhancer	National Institutes of Health Epigenome Roadmap data; National Human Genome Research Institute ENCODE database; the Encyclopedia of DNA Elements	Kim et al52
	Mortality/ventilator-free days due to acute lung injury.	Pediatric ICU dataset from the Children's Hospital Los Angeles	Che et al53
	Mortality (all-cause)	Asan Medical Center	Ge et al54
	Incidence of autism	N/A	Ghafouri-Fard et al55
	Long-term survival from glioblastoma multiforme	The Cancer Genome Atlas	Hao et al56
	Stage of several cancer	Cancer microarray data sets obtained from Gene Expression Model Selector	Hartono57
	HIV genome integration site	Retrovirus Integration Database	Hu et al58
	Daily sepsis/myocardial infarction/ vancomycin antibiotic administration	MIMIC-III	Kaji et al59
	Heart failure	Sutter Health	Choi et al60
	Incidence of vascular diseases	Seoul National University Bundang Hospital	Park et al61
	Future hospitalization	De-identified EHR data from the University of Virginia Health System	Zhang et al62
Multifactor affiliation analysis	Dementia stage of Alzheimer’s disease	Open Access Series of Imaging Studies	Aditya and Pande63
Measuring similarity to exemplars of clusters	Stratify patients with stage 1 lung cancer	Xena	Zhao and Bolouri64
	Stratify the risk of 30-days mortality in patients with cardiovascular disease	Portuguese real Acute Coronary Syndrome patients’ dataset	Paredes et al65
Different predictive models on specific clusters of patient population.	Survival from cardiac transplantation	United Network of Organ Sharing	Yoon et al66
Logic optimization for binary input to continuous output	Drug response of cancer cell lines	Genomics of Drug Sensitivity in Cancer, Cancer Therapeutics Response Portal	Knijnenburg et al67
Rule-based	Diabetes and breast cancer stage classification	Wisconsin Breast Cancer Dataset and Pima Indian Diabetes Dataset in the UCI repository	Ming et al.68
	Incidence of asthma/diabetes/depression/lung cancer/leukemia/myelofibrosis	Medical diagnosis records of about 150K patients collected by a web-based EHR company and 2 other nonmedical datasets	Lakkaraju et al69
Diagonal quadratic discriminant analysis	Incidence of leukemia	Unnamed datasets26,70	Huang71
Artificial hydrocarbon networks	Incidence of breast cancer	Wisconsin Breast Cancer Dataset in the UCI repository	Ponce and Martinez-Villaseñor72
Sparse high-order interaction model with rejection option	Incidence of Alzheimer’s disease	Alzheimer's Disease Neuroimaging Initiative	Das et al73

Abbreviations: EHR, electronic health record; ICU, intensive care unit; ML, machine learning; UCI, University of California, Irvine.

^aWe have either mentioned the name of the dataset/reference, the referenced paper that is indicated by the authors as the source of dataset, or the institute that the dataset is generated by. If there was no information regarding the dataset in a study, we indicated it as N/A in the table.

DL is the most popular ML method as approximately 38% of the articles used DL with different architectures including feed-forward neural networks, convolutional neural networks, recurrent neural networks with long short-term memory or gated recurrent unit. SVM, ensemble techniques, and logistic regression (LR) are the second (∼14%) and third (∼12%, ∼12%) most popular methods, respectively. Other popular techniques in AIM are fuzzy logic and DT.

XAI methods

We grouped the XAI methods that were employed in the articles into 5 categories: 1) feature interaction and importance, 2) attention mechanism, 3) data dimensionality reduction, 4) knowledge distillation and rule extraction, and 5) intrinsically interpretable models. We synthesized these categories from extant XAI review papers.^74,75Figure 3 depicts the type of XAI methods employed along with different ML methods and Table 3 summarizes different approaches in each category of XAI methods for each of the articles.

Figure 3.

Explainable artificial intelligence (XAI) methods vs machine learning (ML) methods used for interpretability enhancement. The links between ML and XAI methods illustrate that ML method was used with that specific XAI method in a paper. The thicker the links are, the more frequent that combination of ML and XAI method has been practiced. A thicker box around the XAI method shows that it has been applied by more of the articles.

Table 3.

Different explainable artificial intelligence methods, their category, and scope

XAI Category	Articles	Approach	Intrinsic/Posthoc	Local/Global	Model-specific/-agnostic
Feature interaction and importance	Ge et al54	Feature weights in the model	Posthoc	Global	Model-agnostic
	Zuallaert et al48	Contribution score of each neuron activation (DeepLIFTundefined18)
	Eck et al32	Approximately marginalizing features out		Global, Local
	Pan et al38	LIME77		Local
	Ghafouri-Fard et al55
Attention mechanism	Kwon et al51	The model’s capability to find a set of positions in a sequence with the most relevant information to the prediction task78	Intrinsic	Global, Local	Model-specific
	Kaji et al59
	Shickel et al50
	Hu et al58			Local
	Choi et al60
	Zhang et al62
	Park et al61
Data dimensionality reduction	Zhao and Bolouri64	Identifying most informative exemplars through cluster analysis and LASSO	Intrinsic	Global	Model-agnostic
	Kim et al52	Building a model based on the most important features
	Hao et al56	Finding the gene pathways and their interactions using sparse DL
	Bernardini et al33	Sparse-balanced SVM
	Zhang et al41	Selecting optimal feature subset from the most critical dimensions of features
	Huang71	Diagonal quadratic discriminant analysis on chi2 selected features		Local	Model-specific
	Aditya and Pande63	Affiliation analysis based on capturing inter-feature relationships (knowledge base)
	Hartono57	Providing clearer mathematical description
Knowledge distillation and rule extraction	Xiao et al49	Distilling complex relationships between hospital readmission and potential risk factors	Posthoc	Global	Model-specific
	Settouti et al44	Fuzzy rules	Intrinsic	Global, Local
	Davoodi and Moradi43			Global
	Hajiloo et al30
	Che et al53	Mimic learning	Posthoc	Global, Local
	Barakat et al26	Intelligible representation of the SVM’s classification decision	Intrinsic	Global
	Das et al73	sparse high-order interaction model with rejection option		Global, Local
	Crielaard and Papapetrou40	Rule induction from transparent oracle-coached predictive models		Global
	Paredes et al65	Rule extraction from regions of belonging associate with each class
	Lakkaraju et al69	Decision-sets (nonhierarchical if-then rules)		Local
	Ming et al68	Visualizing rules	Posthoc	Global
	Ponce and Martinez-Villanseñor72	Rule extraction based on predictive features (polynomial weights)	Intrinsic
	Luo36	Automatically pruning and manually refining association rules	Posthoc	Global, Local	Model-agnostic
Intrinsically interpretable models	Fejza et al24	Distributed logistic regression framework	Intrinsic	Global	Model-specific
	Van den Bulcke et al23	Parameter and threshold optimization of decision tree, LR, and ridge LR
	Bouktif et al42	Ant colony optimization of combining Bayesian classifiers
	Brisimi et al34	Alternating clustering and classification
	Yoon et al66	A tree of clusters with base learners associated with each cluster
	Valdes et al39	Accurate decision trees based on boosting		Global, Local
	Knijnenburg et al67	Logic optimization for binary input to continuous output		Global
	Jalali and Pfeifer37	Ensemble of regularized linear SVM		Global, Local
	Stiglic et al35	Automated pruning for decision tree.		Global

Abbreviations: DL, deep learning; LIME, local interpretable model-agnostic explanation; LR, logistic regression; SVM, support vector machine; XAI, explainable artificial intelligence.

We analyzed the XAI methods’ scope and categorized them into 1) intrinsic/posthoc (ie, interpretation as a result of inherited characteristics of the ML method/interpretation as an additional step on top of the ML model), 2) global/local (ie, interpretation of the whole logic of the model/interpretation of a specific decision for an instance), and 3) model-specific/-agnostic (ie, interpretation method limited to a specific model/interpretation method not tied to a specific model). We referred to the definition of these categories in previously published XAI review papers.^74,76 Visualization techniques are often used as a complementary tool to facilitate the interpretation of the results in most of the articles. Thus, we did not consider visualization as a separate interpretability enhancement method.

According to the articles, the majority of researchers chose “if-then” rules (∼28%) to enhance the interpretability of complex ML methods. Another major trend is to preserve the interpretability of less complex ML methods while boosting their performance and applying optimization (∼21%). These 2 major trends are followed by dimensionality reduction techniques (∼19%).

Feature interaction and importance

Researchers have used feature importance and pairwise feature interaction strengths to provide interpretability to ML models.⁷⁵ The level of contribution of the input features to the output prediction has been extensively used for XAI in AIM.⁷⁶ Ge et al⁵⁴ used feature weights to report top-10 contributing features for intensive care unit (ICU) mortality prediction. Researchers have also used sensitivity analysis for deriving the feature importance.⁸⁰ Based on sensitivity analysis, the most important features are those to which the output is most sensitive. Eck et al³² determined the most important features for microbiota-based diagnosis by approximately marginalizing features out and evaluating the effect on the model’s output.

Ribeiro et al⁷⁷ introduced the local interpretable model-agnostic explanation (LIME) method. LIME produces explanations for any classifier by approximating the reference model with a “locally faithful” interpretable representation. To produce explanations, LIME perturbs an instance, generates neighborhood data, and learns linear models in that neighborhood. Pan et al³⁸ used LIME to investigate the contribution level of features of new instances for predicting central precocious puberty in girls. Ghafouri-Fard et al⁵⁵ took the same approach for diagnosing autism spectrum disorder.

Shrikumar et al⁷⁹ introduced DeepLIFT, which is a backpropagation-based interpretability approach. Backpropagation approaches calculate the gradient of an output with respect to the input via the backpropagation algorithm to report the feature importance. Zuallaert et al⁴⁸ used DeepLIFT to build interpretable deep models for splice site prediction by calculating the contribution score of each nucleotide.

Attention mechanism

The main idea behind the attention mechanism⁷⁸ is the model’s capability to find a set of positions in a sequence with the most relevant information to the prediction task. This idea is proved to apply to interpretability enhancement as well.⁸¹

Attention mechanism has been used to 1) highlight the specific times when the input features have mostly influenced the predictions of clinical events in ICU patients,⁵⁹ 2) present an interpretable acuity score framework based on DL (DeepSOFA) that can evaluate a patient’s severity of illness during an ICU stay⁵⁰; 3) provide “mechanistic explanations” on accurate prediction of HIV genome integration sites (DeepHINT)⁵⁸; 4) feed gradient-weighted class activation mapping (Grad-CAM) with feature representations that include embedded time intervals information to recurrent neural networks to predict vascular diseases⁶¹; and 5) to learn a representation of EHR data which captures the relationships between clinical events for each patient (Patient2Vec).⁶²

Choi et al⁶⁰ introduced a reverse time attention model (RETAIN), which uses 2 sets of attention weights, 1 for visit-level (to capture each visit’s influence) and the other for variable-level. RETAIN is a reverse attention mechanism to preserve interpretability, mimic medical professional’s behavior, and incorporate sequential information. Kwon et al⁵¹ developed a visually interpretable DL model for heart failure and cataract risk prediction based on RETAIN (RetainVis). The commonality of these articles is in their aim to enhance the interpretability of DL models by highlighting specific position(s) within a sequence (eg, time, visits, DNA) in which certain input features influence the prediction outcome.

Data dimensionality reduction

Researchers used data dimensionality reduction to build models only by including the most important features. Bernardini et al,³³ for instance, used the least absolute shrinkage and selection operator (LASSO)⁸² to induce sparsity for SVMs for type 2 diabetes early diagnosis. Hao et al⁵⁶ developed pathway-associated sparse DL to find the gene pathways and their interactions in patients with glioblastoma multiforme. Kim et al⁵² selected the most important input features from the datasets based on domain knowledge and built a DL model. Then, they ranked the features based on their weights in the model and visualized the result for predicting cell-type-specific enhancers.

In another work, Zhao and Bolouri⁶⁴ stratified patients with stage-1 lung cancer by identifying the most informative exemplars through supervised learning. They proposed a hybrid approach for dimensionality reduction by integrating pattern discovery and regression analytics to identify a group of “exemplars” and create a “dense data matrix.” Then, they included those exemplars that are the most predictive of the outcome in the final model.

Zhang et al⁴¹ built a model for drug side effects prediction based on the optimal dimensions of input features by combining multi-label k-nearest neighbor and genetic algorithm techniques. To provide more transparency to the model, Hartono⁵⁷ visualized cancer classification using a clearer mathematical description. This was achieved by introducing Softmax restricted radial basis function networks. Huang⁷¹ developed an integrated method for cancer classification. He reduced the feature dimension by selecting important genes using the Chi2 algorithm. Then, he applied diagonal quadratic discriminant analysis for classification. Finally, he used general rule induction to extract association rules.

Knowledge distillation and rule extraction

In their influential work, Hinton et al⁸³ proposed a knowledge distillation technique for neural networks. This technique transfers knowledge from a complex and accurate model to a smaller and less complex one which is faster but still accurate. Che et al⁵³ used knowledge distillation to build an interpretable prediction model for ICU outcome (ie, mortality, ventilator-free days) by feeding learned features from the base model into the helper classifier (mimic model) and reporting feature importance of the mimic model to deliver interpretability to the basic complex model. A similar approach was taken by Ming et al⁶⁸ to extract rules by approximating a complex model using model induction on several tasks, such as breast cancer diagnosis and diabetes classification.

Xiao et al⁴⁹ developed a DL model (CONTENT) that distills complex relationships between hospital readmission and potential risk factors for patients by transforming patients' EHR events into clinical concept embeddings. As a result, they produced a context vector that characterizes the overall condition of the patient. Also, classification rules were derived as a means to provide human interpretable representations of the black-box predictive models. Other researchers used rule extraction techniques to 1) provide decision-sets (nonhierarchical if-then rules) with use case on several diseases diagnosis⁶⁹; 2) automatically and manually prune association rules for explanations of type 2 diabetes risk prediction³⁶; 3) classify micro-arrays,³⁰ predict mortality in ICUs,⁴³ and classify diabetes⁴⁴ by fuzzy rule extraction; 4) diagnose Alzheimer’s disease⁷³ by adding a rejection option (on hard-to-classify samples); 5) diagnose diabetes mellitus²⁶; and 6) stratify patients with cardiovascular disease risk.⁶⁵

Intrinsically interpretable models

Besides common interpretability enhancement techniques described earlier, many researchers have taken a different strategy to provide interpretability to their predictive models. These approaches mainly rely on preserving the interpretability of less complex ML methods while enhancing their performance by boosting and optimization techniques. Researchers implemented 1) distributed logistic regression framework to enhance the accuracy of logistic regression dealing with large data with application to daily in-hospital mortality prediction during the patient stay,²⁴ 2) automated pruning of decision trees for multiple disease classification,³⁵ 3) ensembles of regularized linear SVMs for gene expressions,³⁷ 4) logic optimization for binary input to continuous output to infer logic models for drug response in cell lines,⁶⁷ 5) accurate decision trees based on boosting for stratification of patients into subpopulations,³⁹ 6) clusters of base learners (LR, linear perceptron, Cox regression) to create a tree of classifiers with application to mortality prediction after cardiac transplant,⁶⁶ 7) alternating clustering and classification optimization using sparse linear SVM framework for hospitalization prediction due to heart disease and diabetes,³⁴ 8) ant colony optimization of combining Bayesian classifiers with application to heart diseases and cardiotocography-based fatal pathologies prediction,⁴² and 9) parameter and threshold optimization of DT, LR, and ridge LR for medium-chain acyl-CoA dehydrogenase deficiency classification.²³

Reproducibility assessment

Many research fields, including AI, have been struggling with a reproducibility crisis over the past decade.⁸⁴ A survey of 400 algorithms presented in the 2 top AI conferences shows that only 6% of the presenters have shared their implementation code, around 30% shared data, and only 50% shared “pseudocode” with the public.⁸⁵ Another recent study emphasizes the importance of reproducibility in AIM research to ensure safety and effectiveness.⁸⁶

We believe the reproducibility in this field deserves more attention. In general, ∼43% (18/42) of the articles did not explicitly (ie, mentioned in the manuscript or in the supplementary material) make their datasets accessible to the public; in ∼57% (24/42) of the studies, source codes were not made publicly available; and ∼31% (13/42) did not meet both criteria (see Supplementary Appendix Table 2). Interested audiences should refer to the original work for more details about their reproducibility. Nevertheless, more in-depth analyses are required to rigorously evaluate the reproducibility of the articles, which is out of the scope of this review paper.

Interpretability evaluation

We observed that more than ∼26% (11/42) of the articles did not explicitly report any evaluation of the XAI method they used, ∼28% (12/42) have either referred to the common medical knowledge and medical literature or compared the results with hypotheses, and only ∼7% (3/42) reported human expert confirmation of the results. The rest of the papers took different strategies for the XAI method evaluation, especially regarding effectiveness measurement. Barakat et al,²⁶ for instance, measured the fidelity of the rules they derived from the model against the original model as a way of measuring the effectiveness of the proposed interpretability method. They also measured “comprehensibility” which they define as the number of rules. Ponce and Martinez-Villaseñor⁷² compared different ML methods for breast cancer classification, reporting accuracy percentage and interpretability level (low, medium, high). However, the logic behind this categorization is not clear.

Lakkaraju et al⁶⁹ defined several metrics for evaluation including 1) fraction overlap (the extent of overlap between every pair of rules of a decision set); 2) fraction uncovered (the fraction of data points not covered by any rule); 3) average rule length (average number of predicates a human needs to parse in a decision set); 4) number of rules in a decision set; and 5) fraction of classes (the fraction of class labels predicted by at least 1 rule). To qualitatively analyze their XAI method, Kwon et al⁵¹ verified whether the medical codes that were highly predictive of heart failure in their model are supported by general medical knowledge. Both of these studies^51,69 performed a user study to evaluate different aspects of interpretability enhancement in their proposed approach. Kwon et al⁵¹ concluded that AIM applications should incorporate more human interactions with the system.

DISCUSSION AND CONCLUSIONS

Medical professionals’ perspectives of XAI for medicine

We studied the articles included in this systematic scoping review from the lenses of 2 medical professionals (co-authors PRM and JC), aiming to highlight the 1) general opportunities and challenges in XAI for medicine and 2) examples of specific pros and cons regarding each XAI category from the end-users' perspective rather than from the XAI researchers’ perspective. We summarized our findings in Table 4. Nevertheless, more extensive studies are required to systematically collect feedback from medical professionals (eg, Diprose et al⁸⁸) and analyze potential pros and cons, opportunities, and challenges from their perspective. Such a study can assist to identify the gaps between XAI researchers’ and end-users' needs in real-world scenarios.

Table 4.

Potential pros and cons of explainable artificial intelligence (XAI) categories from the medical professional’s point of view

XAI Category	Pros	Cons
Feature interaction and importance	Illustrates not only important features, but also their relative importance toward clinical interpretation	Numerical weights are often not easily interpretable, or might be misinterpreted
Attention mechanism	Does not directly inform the clinical end user of the answer but does highlight the areas of most interest to support easier decision-making. Thus, user might be more tolerant of imperfect accuracy	Simply providing this information to a clinical end user might not be useful. Major issues are information overload, alert fatigue, etc. Providing areas of attention without clarity on what to do with the results can potentially be even more confusing if the end user is unsure of what to make of a highlighted section (and also likely to miss nonhighlighted areas that are sometimes crucial)
Data dimensionality reduction	Simplifying the data down to a small subset can make the model’s underlying behavior comprehensible. It also can be generally advantageous with potentially more robust regularized models that are less likely to overfit training data	Risk of missing other features that can still be important in individual cases, but the reduced models inadvertently do not include them
Knowledge distillation and rule extraction	Potentially more robust models with summarized representations of complex data that allows clinical end users to naturally infer meaning from87	If clinical end users cannot intuitively interpret the meaning of these representations, then the representations are likely to make it even harder for the end users to interpret and explain
Intrinsically interpretable models	Simple models that are more familiar and intuitive to clinical end users. Even if they don't understand how these types of models are constructed, many medical professionals will at least have some familiarity with how to apply them	If ensemble of simple models is used to enhance the accuracy, then a clinical end user is not able to interpret the results

Challenges: 1) Not all visualizations are interpretable by medical professionals. In other words, visualization does not necessarily provide better interpretability. 2) There is a need to incorporate more longitudinal features in XAI (as opposed to just using aggregated values of a lab feature in a period) to improve the robustness of the models. 3) The absence of a definition for sufficient explainability, and how it can vary substantially in different use cases, is an ongoing issue. 4) Including more features may help improve model accuracy. However, this may also result in overfitting that is not robust to variations and, thus, less usable and trustworthy for medical professionals. 5) Predictive analytics on uncommon diseases might reveal some causations that are not known now and can be used to prevent extensive and expensive workups.

Opportunities identified by these medical professionals are: 1) more transparent predictive models for major diseases (eg, diabetes, cancer) that are the reason for extensive pathologies in the field of preventive medicine, 2) more emphasis on studying uncommon diseases for possible etiologies in predictive analytics to prevent extensive and expensive workups, 3) rerouting healthcare funds to outpatient care and implementing preventive strategies using explainable hospital readmission prediction due to chronic diseases, 4) educating new generations of medical professionals with basic AI knowledge to overcome the gap between AI systems and medical professionals, as an ultimate goal of XAI, 5) using XAI to assist medical professionals overcome their medical knowledge biases and become more objective, 6) enforcing more regulations to ensure AI methods are evaluated rigorously, reproducible, and accompanied by clear circumstances under which the methods are applicable, and 7) more focus on integrating causal inference with AI to provide explanations.

Potential gap in the perspectives of designers and medical professionals in XAI for medicine

We observed that knowledge distillation and rule extraction is the most popular approach, followed by intrinsically interpretable models. XAI can assist designers to debug model development and do sanity checks for spurious associations. Medical professionals, on the other hand, may not require specific explanations of predictions and recommendations if they have been empirically validated through other mechanisms (eg, randomized clinical trials). However, being informed of the features/elements of a prediction model that are important for risk assessment can itself be instructive to medical professionals.

XAI can provide transparency to the prediction models that are built on a cohort that excluded certain types of patients (eg, pregnant patients), thus, help medical professionals understand when it may be unfair to directly apply the XAI methods to individual patients. A closer look at the scope of the approaches proposed in different XAI method categories reveals that the majority of current approaches focus on the global scope. While valuable, more methods for local explanation need to be explored. Medical professionals work with individual patients more often and need specific explanations tailored to each patient’s situation to assess how the XAI results do, or do not, apply to individual patient contexts.

If we look further at each XAI method category, we can see that the attention mechanism (100%) and feature interaction and importance (60%) are the top approaches supporting local explanations. This trend is followed by knowledge distillation and rule extraction (∼38%), intrinsically interpretable models (∼22% local), and data dimensionality reduction (12.5%) focus on the local explanations, while they appear to be quite popular for XAI in medicine. This can represent a potential gap between theory and practice.

Potential limitations of XAI methods

Despite the valuable effort in providing interpretability for black-box models such as neural networks, researchers call for more caution and evaluation while applying these methods. Here we provide some examples of these concerns. Ghorbani et al⁸⁹ were able to compute small adversarial perturbations (in a similar way to computing adversarial examples to neural networks⁹⁰) that cause substantial changes to feature importance maps of several interpretation methods. However, such adversarial perturbations in the case of real-world datasets such as EHR and their impact on XAI methods for EHR-based models need to be studied further.

Sokol and Flach⁹¹ emphasize the importance of deriving more meaningful concepts when using an XAI method by personalizing the explanations through user’s input. Miller et al⁹² also note that people maintain mental models of each other to tailor explanations to individuals. This has not been the focus of much XAI research. From another point of view,⁹³ using attention mechanisms does not necessarily provide more transparency to the black-box model. Based on their experiments, the relationship between the attention weights and model output is unclear. Another similar study⁹⁴ argued that whether or not attention is explanation depends on the definition of explanation. However, both studies confirmed that researchers should be cautious when using attention distributions for explanations.

Rudin⁹⁵ refers to post hoc XAI methods as “problematic” in the case of high-risk decision-making. From her point of view, the way forward is to create inherently interpretable models rather than creating methods that explain black-box models. She also calls the accuracy and interpretability trade-off a myth. However, from our perspective, we argue that such a statement might not always be valid in medical predictive modeling using EHR data. Further, an inherently interpretable model does not give an explainable decision. A large decision tree may still require explanation to a nontechnical user; and concepts such as contrastive and counterfactual explanations are independent of the interpretability of the model. Nevertheless, XAI and, more specifically, XAI for medicine, is a relatively new topic that is still in its initial stage of formation. Thus, diverse points of view and approaches to addressing existing limitations and shortcomings of different XAI methods should be welcomed.

XAI evaluation issue

Few researchers have considered XAI evaluation in their work. There is no consensus on the definition of interpretability yet; thus, there is no agreed-upon approach to evaluate the results of the XAI methods. We argue that making effective XAI methods for medicine requires more interdisciplinary collaboration between different professionals such as AI researchers and medical professionals. Other researchers also have emphasized the importance of including human expertise in the explanation process.⁹² The most recent report on XAI program of DARPA⁹⁶ explicitly mentioned that “an XAI system’s explanation effectiveness must be assessed according to how its explanations aid human users. This requires human-in-the-loop psychologic experiments to measure the user’s satisfaction, mental model, task performance, and appropriate trust.”

Reproducibility issue

Based on the results of this systematic review, there is not enough emphasis on the reproducibility of the research work published in this field. Considering the issue of XAI evaluation, in addition to the critical nature of AI applications for medical practices, research reproducibility is crucial.⁸⁶ To evaluate their new ideas, researchers in the field need to compare their work to the previous work done by other researchers. It would be easier and faster to examine and compare ideas if researchers use publicly available datasets, describe how they select specific features, clearly mention the dimensionality of the dataset as well as the infrastructure they use, and provide the source code. Thus, we suggest that publication venues should require authors to meet certain reproducibility criteria before publishing their research work.

FUNDING

This project was partially supported by the National Institute on Aging of the National Institutes of Health (NIH) under award number R21AG061431, the National Cancer Institute under award number R01CA246418, and the University of Florida Clinical and Translational Science Institute, which is supported in part by the NIH National Center for Advancing Translational Sciences under award number UL1TR001427. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.

AUTHOR CONTRIBUTIONS

SNP and ZH conceived the main conceptual idea. SNP, ZH, ZC, TM, and JB contributed to research design. SNP and ZC performed the literature search, record screening, and data analysis. PRM and JHC acted as medical professionals and provided medical insights. XL, TM, and JB acted as experts on artificial intelligence, machine learning, and deep learning throughout. SNP wrote the initial draft of the manuscript. All the authors reviewed and edited the paper iteratively for important intellectual content. All the authors approved the version to be published. ZH supervised this research and takes primary responsibility for the research reported here.

CONFLICT OF INTEREST STATEMENT

None declared.