Recent Advances in AI for Automated ICD Coding: A Systematic Literature Review

Abstract

International Classification of Diseases (ICD) codes enable correct billing, insurance reimbursement, and healthcare analytics. However, manual coding is time-consuming, expensive, and error-prone, creating bottlenecks in clinical workflow and limiting scalability. Artificial intelligence (AI) has emerged as a promising solution for automated ICD code assignment from unstructured clinical text. This systematic review explores the current state of automated ICD coding research, examining models applied to diverse clinical documents including discharge summaries, electronic health records, nursing notes, and pathology reports. Following PRISMA guidelines, we searched six databases for studies published between 2019 and 2024, selecting 54 relevant studies from 4,280 initial citations. Our analysis reveals the use of diverse datasets, preprocessing techniques, and feature extraction methods, alongside a clear evolution from traditional machine learning to deep learning approaches, with substantial architectural diversity across convolutional, recurrent, transformer, and hybrid models. Performance varies considerably across dataset configurations, with models achieving higher accuracy on frequent code subsets compared to full label spaces. However, critical gaps persist: overreliance on single-language, single-institution datasets limits generalizability; difficulties in predicting rare codes remain unresolved; lack of model interpretability undermines clinical trust; and inconsistent evaluation protocols hinder meaningful comparison. To address these challenges, we propose a 5P evidence-grounded research agenda: Population Diversity, Performance Robustness, Prediction of Rare Codes, Provenance Transparency, and Practical Integration. These findings underscore AI’s potential to transform ICD coding while highlighting the need for standardized benchmarks, rigorous external validation, multilingual datasets, and explainable architectures to enable equitable and effective deployment in real-world healthcare systems.

Introduction

The International Classification of Diseases (ICD), developed by the World Health Organization (WHO), is a foundational system for encoding medical diagnoses and procedures across global healthcare systems. With approximately 14,000 codes in the WHO ICD-10, over 70,000 codes in ICD-10-CM, and around 17,000 codes in ICD-11, accurate ICD coding is essential for clinical documentation, reimbursement, public health surveillance, and health policy formulation [1, 2]. Traditional ICD coding involves highly skilled medical coders meticulously reviewing extensive clinical documentation such as physician note, lab results and discharge summaries. Coders must extract relevant clinical information, interpret it according to complex coding guidelines, and then assign the most accurate and specific ICD codes. This process, requiring significant cognitive effort, domain expertise and continuous professional development to keep pace with evolving medical knowledge and coding updates has long been a manual endeavor. However, manual ICD coding remains labor-intensive and error-prone, with error rates reported as high as 20–30%, particularly in complex or multimorbid cases [3]. These inefficiencies are especially acute in low-resource settings, where shortages of trained coders exacerbate disparities in healthcare delivery and data quality. To address these challenges, automated ICD coding systems based on natural language processing (NLP) and machine learning (ML) have emerged as promising alternatives. These systems aim to assign ICD codes directly from unstructured clinical texts such as discharge summaries, progress notes, and radiology reports, reducing manual workload while improving accuracy and consistency [4]. In recent years, transformer-based models such as BERT [5], ClinicalBERT [6], and BioBERT [7] have demonstrated significant improvements in predictive performance for multi-label ICD classification tasks, achieving state of the art micro-F1 scores in several benchmark datasets [8].

Despite these advancements, several limitations persist. Rare ICD codes, those appearing in fewer than 1% of records are often underrepresented in training data and poorly predicted by most models [9]. Interpretability remains a key challenge: while attention mechanisms are frequently used, they are not inherently explanatory or clinically transparent [10]. Moreover, most studies rely on a narrow set of English-language datasets, limiting the global applicability of these methods [11].

Several systematic reviews have examined automated ICD coding [12, 21], but their scope and analytical focus vary substantially. Early reviews were conducted before the widespread adoption of deep learning and were largely descriptive or restricted to specific document types such as discharge summaries. More recently, Ji et al. [36] presented a unified review of deep learning approaches for automated medical coding, with a primary emphasis on architectural design patterns and model components rather than systematic study selection, dataset characteristics, or reproducibility assessment. Similarly, Yan et al. [37] provided a broad survey of automated ICD coding methods and challenges but did not adopt a PRISMA-guided methodology or perform structured quality assessment across included studies.

In 2025, two additional systematic reviews further examined AI-based automated ICD coding. A recent review focused specifically on Large Language Model based approaches, synthesizing studies that evaluated transformer and generative models in supervised, few-shot, and zero-shot settings [38]. While this review reported promising performance, particularly for frequent codes, its scope was intentionally restricted to LLM-based methods and did not examine traditional deep learning architectures, preprocessing strategies, dataset composition, or evaluation consistency across the broader literature. In parallel, a systematic review emphasized high-level benefits, challenges, and ethical considerations of AI-based ICD coding but included a limited number of primary studies and did not provide a detailed technical synthesis of datasets, modeling strategies, or performance metrics [39].

As a result, important gaps remain in the literature. Existing reviews often focus on specific model families, emphasize conceptual or architectural perspectives, or include a limited number of studies, which restricts comparative technical analysis.

To address these limitations, this review systematically synthesizes studies published between 2019 and 2024 that apply AI methods for automated ICD coding across diverse clinical text types, modeling strategies, and healthcare contexts. The review follows PRISMA 2020 guidelines and structures its analysis using a modified PICO framework (Population, Intervention, Comparison, and Outcome), enabling a systematic examination of data sources, modeling approaches, reported performance, and key findings.

This review makes three main contributions. It is a comprehensive PRISMA-2020–compliant overview of automated ICD coding research from 2019 to 2024. It also provides a comparative synthesis of the major model families, including CNNs, RNNs, Transformers, and knowledge-enhanced approaches, summarizing their performance ranges and the available evidence on cross-dataset robustness. Finally, it proposes a structured research agenda, the 5P evidence grounded research agenda, which focuses on population diversity, performance robustness, prediction of rare codes, provenance transparency, and practical integration. Taken together, these contributions aim to guide the field toward ICD coding systems that are not only accurate but also equitable, reliable, and ready for real-world clinical practice.

The paper is structured as follows: “Methodology” section outlines the PRISMA-guided methodology, “Detailed Review of Selected Literature” section presents the detailed review, “Discussion” section is the discussion section, and “Conclusion” section concludes with recommendations for advancing the field.

Objective and Research Questions

This systematic review synthesizes ML, DL, and NLP methods for predicting International Classification of Diseases (ICD) codes from unstructured clinical texts, including discharge summaries, electronic health records (EHRs), nursing notes, and pathology reports. This review includes 54 studies in total, 53 published between 2019 and 2024 and one published in January 2025 retrieved during the final database search update, addresses the trends in methodology, comparative performance, and major issues in the development of strong, transparent, and clinically applicable ICD coding systems.

Main Research Question

Which AI techniques demonstrate the most reliable and clinically relevant performance for automated ICD code prediction from unstructured clinical text?

Sub-Questions

To ensure analytical clarity and methodological rigor, the research questions are organized using an adapted PICO framework (Population, Intervention, Comparison, Outcome) [34], tailored to AI-driven ICD coding research.

Population (P)

• What types of clinical text and patient datasets have been used as the primary data sources for automated ICD coding, and how do their characteristics influence model performance and generalizability?

Intervention (I)

• Which AI modeling strategies such as CNNs, RNNs, Transformers, or hybrid/knowledge-enhanced architectures have been applied to ICD coding, and what innovations distinguish the most recent approaches?

• How have NLP preprocessing strategies (such as tokenization, section filtering, and domain-specific embedding) improved the effectiveness and robustness of these models?

Comparison (C)

• How do different model families (CNN, RNN, Transformer, etc.) compare in terms of predictive performance, ability to manage class imbalance, computational efficiency, and cross-dataset robustness?

Outcome (O)

• Which evaluation metrics are most commonly used to assess model accuracy, reliability, and fairness?

• What are the main findings, limitations, and methodological gaps identified in the literature, and how do they inform future research directions?

Background

The ICD system has undergone substantial evolution since its origin in the mid-20th century. While ICD-1 comprised fewer than 2,000 codes, the WHO ICD-11, released in 2022, defines approximately 17,000 unique diagnostic codes and more than 120,000 codable terms, with post-coordination enabling the representation of a much larger range of clinical scenarios [1, 13].

Early attempts at automating ICD code assignment began in the 1970s and 1980s with rule-based and dictionary-driven systems [14]. These methods worked for structured inputs but performed poorly on narrative, unstructured data such as clinical notes and discharge summaries. The 2000s saw the emergence of statistical and classical ML approaches such as support vector machines (SVMs) and logistic regression that offered improved flexibility and data-driven learning [4]. The 2010s ushered in the deep learning era, enabling better representation learning through convolutional neural networks (CNNs) and recurrent neural networks (RNNs). These models, including CAML and DR-CAML [8], improved multi-label classification by incorporating attention mechanisms. MultiResCNN [9] introduced a hierarchical representation to better handle rare codes.

Before transformer-based architectures, attention mechanisms were already used in CNN and RNN-based models for automated ICD coding to highlight clinically relevant text segments [43]. A major advance was the introduction of label-wise attention, where each ICD code attends to different parts of the clinical text, improving interpretability in extreme multi-label settings, as demonstrated in CAML and related models [8, 44]. Subsequent work explored ontology- and hierarchy-aware attention mechanisms that explicitly incorporated ICD structure to better align predictions with the coding taxonomy [45].

Transformer-based models marked a paradigm shift. Introduced by Devlin et al. in 2019, BERT [5] uses bidirectional self-attention to model complex language dependencies. This approach was adapted to biomedical and clinical contexts through BioBERT [7], ClinicalBERT [6], and BioClinicalBERT [15], all of which significantly improved ICD prediction tasks, particularly for longer and more complex clinical narratives.

The progress achieved by these architectures has been facilitated by the availability of large-scale clinical datasets. Among these, MIMIC-III has emerged as the benchmark dataset for ICD coding research. It includes 59,652 discharge summaries annotated with 8,929 ICD-9 codes [16].

Methodology

In conducting our systematic review, we adhered to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) 2020 guidelines to ensure comprehensive and transparent reporting of our methods and findings [17]. The review process followed main phases: identification, screening, and inclusion as shown in Fig. 1.

Fig. 1.

The PRISMA flow diagram used in the study

Identification

A comprehensive literature search was conducted across major scientific databases, and to ensure comprehensive and unbiased coverage of the literature, we selected six databases including PubMed, Google Scholar, IEEE Xplore, ScienceDirect, ACM Digital Library, and Mendeley, based on their relevance to the multidisciplinary nature of automated ICD coding research. This approach aligns with systematic review best practices, which recommend searching multiple databases to minimize selection bias and maximize recall [18, 19]. PubMed and ScienceDirect were chosen for their extensive biomedical content, while IEEE Xplore and ACM Digital Library provide access to technical and computer science literature essential for this topic. In the same way, Google Scholar and Mendeley were included to capture grey literature and broader scholarly outputs [20, 32, 33]. By combining these sources, we balanced subject-specific depth with broad coverage, ensuring our review is both thorough and methodologically robust [19].

Search Query and Keywords

To retrieve relevant studies for this systematic review, a comprehensive Boolean search strategy was employed, combining terms related to ICD code prediction, machine learning techniques, clinical text types, and performance evaluation metrics. This approach is consistent with recommended practices for systematic literature reviews that emphasize the effective use of Boolean operators to enhance search precision and recall [22]. Two primary search queries, as shown in Table 1, were developed and applied across selected databases. Additionally, alternative versions of these queries were created using synonymous keywords and adapted for specific database requirements. Both queries were executed independently for each database, and the retrieved records were combined prior to duplicate removal.

Table 1.

Summary of search queries used for literature retrieval

Query	Search query	Purpose / Notes
Query 1	(“ICD code prediction” OR “ICD-9 codes” OR “ICD-10 codes” OR “Automated coding”) AND (“Machine learning” OR “Deep learning” OR “NLP” OR “Natural language processing”) AND (“Clinical text” OR “Electronic Health Records” OR “Clinical notes” OR “Discharge summaries”) AND (“Accuracy” OR “Precision” OR “Recall” OR “F1 score”)	Focused on retrieving studies related to ICD coding using ML/NLP methods with specific emphasis on performance metrics and types of clinical data.
Query 2	(“ICD coding” OR “ICD code assignment” OR “ICD prediction” OR “ICD classification”) AND (“Artificial Intelligence” OR “Machine Learning” OR “Deep Neural Networks” OR “Transformer models” OR “BERT” OR “NLP”) AND (“Electronic Medical Records” OR “EHR” OR “Clinical documentation” OR “Discharge summaries” OR “Medical notes”) AND (“Evaluation metrics” OR “Performance” OR “F1-score” OR “Precision” OR “Recall” OR “Accuracy”)	Designed to increase retrieval of recent and broader research, especially those using state-of-the-art deep learning techniques.

Database Search Results

A total of 4,280 articles were retrieved across six major scientific databases, with the number of results from each source as follows: PubMed (1,438), Google Scholar (1,131), IEEE Xplore (156), ScienceDirect (676), ACM Digital Library (787), and Mendeley (92). The summarized database search results are shown in Table 2.

Table 2.

Number of articles retrieved from each database

Database	Number of articles retrieved
PubMed	1,438
Google Scholar	1,131
IEEE	156
ScienceDirect	676
ACM Digital Library	787
Mendeley	92
Total	4,280

Screening

Duplicate records were identified and removed using EndNote Clarivate, following the de-duplication protocol proposed by Bramer et al. [23], a widely accepted method in systematic literature reviews. A total of 4,280 references were initially imported into the EndNote library. A total of 1087 duplicate records were removed. Of those, automated duplicate detection resulted in the removal of 902 records, while an additional 185 duplicates were identified and eliminated through manual inspection. The remaining 3,193 unique records were subjected to a title and abstract screening based on predefined inclusion and exclusion criteria as shown in Table 3. This process led to the exclusion of 3,061 records. The remaining 132 full-text articles were assessed in detail. Furthermore, 78 articles were excluded for various reasons; for example, 38 studies were removed due to a lack of open access. In the same manner, reviewing articles in the academic field was 9 in number, and 31 studies didn’t involve computational methods. Following title and abstract screening, full-text articles were retrieved for eligibility assessment. Despite comprehensive efforts 38 studies remained inaccessible due to subscription restrictions. These studies were documented as “Not open access” in the PRISMA flow diagram, consistent with PRISMA 2020 guidelines, which specify that this category should include reports that cannot be located regardless of reason [19]. The 38 studies excluded due to access restrictions are listed in Supplementary Table 15.

Table 3.

Inclusion and exclusion criteria for study selection

Inclusion criteria
1	Published between 2019 and 2024 to ensure recent developments are included.
2	Written in English and based on clinical textual data (e.g., discharge summaries, progress notes, or other medical documents).
3	Involves automated ICD code assignment using clinical narratives or medical documentation.
4	Utilizes computational methods such as NLP, machine learning, or deep learning for ICD code prediction.
5	Reports quantitative performance metrics (e.g., accuracy, precision, recall, F1-score) to evaluate system effectiveness.
Exclusion criteria
1	Studies do not involve any clinical coding or classification system.
2	Focused solely on assigning ICD codes for a single disease or procedure without broader applicability.
3	Based on medical image classification instead of text-based data.
4	Did not report any evaluation metrics for the ICD coding system.
5	Focused on time-series medical data (e.g., EEG signals) unrelated to text-based ICD prediction.

Table 15.

Studies excluded due to access restrictions

No.	Title	Year	Venue
1	Enhancing ICD 9 code prediction with BERT: a multi label classification approach using MIMIC III clinical data	2024	IEEE
2	Multi label classification of ICD 10 coding and clinical notes using MIMIC and CodiEsp	2021	IEEE
3	Automatic medical code assignment via deep learning approach for intelligent healthcare	2020	IEEE
4	Hybrid method for ICD prediction using word embedding and natural language processing	2023	IEEE
5	Cascading approach for automatic ICD 10 codes association to diseases in Bulgarian	2022	Contemporary methods in bioinformatics and biomedicine and their applications
6	A deep learning driven disease classification method using MIMIC III and NLP	2023	IEEE
7	Using a sequence to sequence model for large scale automated ICD coding	2019	IEEE
8	Hierarchical fine grained deep learning model for automated ICD coding	2024	IEEE
9	Automatic ICD 10 code association: a challenging task on French clinical texts	2023	IEEE
10	Leveraging TFR BERT for ICD diagnoses ranking	2023	Progress in Artificial Intelligence
11	Medical code prediction from discharge summary: document to sequence BERT using sequence attention	2021	IEEE
12	Autoencoder based prediction of ICU clinical codes	2023	Artificial intelligence in medicine
13	Transfer learning improves unsupervised assignment of ICD codes with clinical notes	2023	IEEE
14	Hybrid attention knowledge fusion network for automated medical code assignment	2024	Bioinformatics research and applications
15	Automated ICD coding via contrastive learning with back reference and synonym knowledge for smart self diagnosis applications	2024	IEEE
16	MKFN: multimodal knowledge fusion network for automatic ICD coding	2023	IEEE
17	ComplicaCode: enhancing disease complication detection in electronic health records through ICD path generation	2024	Artificial neural networks and machine learning
18	TransICD: transformer based code wise attention model for explainable ICD coding	2021	AIME
19	Automatic ICD coding based on segmented ClinicalBERT with hierarchical tree structure learning	2023	DASFAA
20	A deep learning method for ICD 10 coding of free text death certificates	2017	Progress in artificial intelligence
21	UMLS mapping and word embeddings for ICD code assignment using the MIMIC III intensive care database	2019	IEEE
22	GrabQC: graph based query contextualization for automated ICD coding	2021	PAKDD
23	ChatICD: prompt learning for few shot ICD coding through ChatGPT	2023	IEEE
24	Automatic international classification of diseases coding via note code interaction network with denoising mechanism	2023	Journal of computational biology
25	FarSight: long term disease prediction using unstructured clinical nursing notes	2020	IEEE
26	Automated ICD 9 medical code assignment from given free text using deep learning approach	2022	Advances in data and information science
27	Embed wisely: An ensemble approach to predict ICD coding	2021	ECML PKDD
28	Disease correlation enhanced attention network for ICD coding	2021	IEEE
29	Predicting multiple ICD 10 codes from Brazilian Portuguese clinical notes	2020	Intelligent systems
30	A large scale hierarchical structure knowledge enhanced pre training framework for automatic ICD coding	2021	Neural information processing
31	Hienet: Bidirectional hierarchy framework for automated ICD coding	2022	Database systems for advanced applications
32	Co occurrence graph enhanced hierarchical prediction of ICD codes	2024	IEEE
33	A hybrid pooling based deep learning framework for automated ICD coding	2021	IEEE
34	Cascadenet: An LSTM based deep learning model for automated ICD 10 coding	2020	Advances in information and communication
35	Automated ICD coding using deep learning	2022	Applied intelligence for health and clinical informatics
36	Predicting ICD 9 codes from medical notes does the magic of BERT applies here	2020	Thesis
37	Automatic ICD coding based on multi granularity feature fusion	2022	Bioinformatics research and applications
38	Multigranularity label prediction model for automatic international classification of diseases coding in clinical text	2023	Journal of computational biology

Inclusion

A final set of 54 studies were included for qualitative synthesis. From each selected study, key information was extracted and analyzed, including the type of dataset used, preprocessing techniques applied, the machine learning models and algorithms employed, the evaluation metrics reported, and any applied methods for explainability or model interpretability.

Quality Assessment

The quality of 54 studies on automated ICD code prediction was assessed by two independent reviewers, with disagreements resolved via consensus, using 12 criteria (Q1–Q12) evaluating study design, methodology, reproducibility, and ethics, following established systematic review and quality assessment guidelines [17, 35]:

1. Clear objective relevant to ICD code prediction

2. Detailed data source description

3. Adequate clinical text preprocessing description

4. Class imbalance methods discussed

5. Clear explanation of machine/deep learning models

6. Justification of model/algorithm choice

7. Appropriate evaluation metrics

8. Comparison with baseline/existing approaches

9. Publicly available code

10. Sufficient methodological detail for replication

11. Model generalizability assessment

12. De-identified, ethically compliant dataset

Inter-rater reliability was quantified using Cohen’s kappa (), computed across all 648 individual quality judgments (54 studies 12 criteria) [46]. The overall agreement between reviewers was high (), indicating almost perfect agreement according to established interpretation guidelines [47]. All disagreements were resolved through discussion and consensus prior to final scoring.

Studies meeting each criterion received a (). For each study, overall methodological coverage was summarized by the total number of satisfied criteria (Count , range 7 to 12), as reported in Table 4.

Table 4.

Quality assessment of included studies

Study	Q1	Q2	Q3	Q4	Q5	Q6	Q7	Q8	Q9	Q10	Q11	Q12	Count ✓
Li et al. [48]	✓	✓	✓	✗	✓	✓	✓	✓	✗	✓	✗	✓	9
Zeng et al. [49]	✓	✓	✓	✓	✓	✓	✓	✓	✗	✓	✗	✓	10
Chen et al. [50]	✓	✓	✓	✗	✓	✓	✓	✓	✗	✓	✗	✓	9
Huang et al. [51]	✓	✓	✓	✗	✓	✓	✓	✓	✗	✓	✗	✓	9
Yu et al. [52]	✓	✓	✓	✗	✓	✓	✓	✓	✗	✓	✗	✓	9
Chen and Ren [53]	✓	✓	✓	✓	✓	✓	✓	✓	✗	✗	✗	✓	9
Atutxa et al. [54]	✓	✓	✓	✗	✓	✓	✓	✓	✗	✓	✗	✓	9
Biseda et al. [55]	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✗	✓	11
Gangavarapu et al. [56]	✓	✓	✓	✓	✓	✓	✓	✓	✗	✓	✗	✓	10
Li and Yu [57]	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✗	✓	11
Zhang et al. [58]	✓	✓	✓	✗	✓	✓	✓	✓	✗	✓	✗	✓	9
Vu et al. [59]	✓	✓	✓	✓	✓	✓	✓	✓	✗	✓	✗	✓	10
Raja and Tuwani [60]	✓	✓	✓	✗	✓	✓	✓	✓	✗	✓	✗	✓	9
Hsu et al. [61]	✓	✓	✓	✗	✓	✓	✓	✓	✗	✓	✗	✓	9
Sonabend et al. [62]	✓	✓	✓	✗	✓	✓	✓	✓	✗	✓	✓	✓	10
Velichkov et al. [63]	✓	✓	✓	✓	✓	✓	✓	✓	✓	✗	✗	✓	10
Cao et al. [64]	✓	✓	✓	✗	✓	✓	✓	✓	✗	✓	✗	✓	9
Schäfer and Friedrich [65]	✓	✓	✓	✗	✓	✓	✓	✓	✗	✓	✗	✓	9
Singaravelan et al. [66]	✓	✓	✓	✓	✓	✓	✓	✓	✗	✓	✗	✓	10
Sen et al. [67]	✓	✓	✓	✓	✓	✓	✓	✓	✗	✓	✗	✓	10
Chraibi et al. [68]	✓	✓	✓	✗	✓	✗	✓	✓	✗	✗	✗	✓	7
Hu et al. [69]	✓	✓	✓	✗	✓	✓	✓	✓	✗	✓	✗	✓	9
Zhou et al. [70]	✓	✓	✓	✓	✓	✓	✓	✓	✓	✗	✗	✓	10
Popescu et al. [71]	✓	✓	✓	✗	✓	✓	✓	✓	✗	✗	✗	✓	8
Wang et al. [72]	✓	✓	✓	✓	✓	✓	✓	✓	✗	✓	✗	✓	10
Rajendran et al. [73]	✓	✓	✓	✗	✓	✓	✓	✓	✗	✓	✗	✓	9
Huang et al. [74]	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✗	✓	11
Wu et al. [75]	✓	✓	✓	✓	✓	✓	✓	✓	✗	✓	✗	✓	10
Liu et al. [76]	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✗	✓	11
Chomutare et al. [77]	✓	✓	✓	✗	✓	✓	✓	✓	✓	✓	✗	✓	10
Yuan et al. [78]	✓	✓	✓	✗	✓	✓	✓	✓	✓	✓	✗	✓	10
Wu et al. [79]	✓	✓	✓	✗	✓	✓	✓	✓	✓	✓	✗	✓	10
Zhang et al. [80]	✓	✓	✓	✓	✓	✓	✓	✓	✓	✗	✗	✓	10
Ren et al. [81]	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✗	✓	11
Wu et al. [82]	✓	✓	✓	✓	✓	✓	✓	✓	✗	✓	✗	✓	10
Qiu et al. [83]	✓	✓	✓	✓	✓	✓	✓	✓	✗	✓	✗	✓	10
Boukhers et al. [84]	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✗	✓	11
Nath et al. [85]	✓	✓	✓	✗	✓	✓	✓	✓	✗	✓	✗	✓	9
Masud et al. [86]	✓	✓	✓	✗	✓	✓	✓	✓	✗	✓	✗	✓	9
Zhao et al. [87]	✓	✓	✓	✗	✓	✓	✓	✓	✗	✓	✗	✓	9
Liu et al. [88]	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✗	✓	11
Mou et al. [89]	✓	✓	✓	✗	✓	✓	✓	✓	✗	✓	✗	✓	9
Chen et al. [90]	✓	✓	✓	✗	✓	✓	✓	✓	✗	✗	✗	✓	8
Carberry and Xu [91]	✓	✓	✓	✗	✓	✓	✓	✓	✗	✓	✗	✓	9
Aden et al. [92]	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✗	✓	11
Wu et al. [93]	✓	✓	✓	✓	✓	✓	✓	✓	✗	✓	✗	✓	10
Merchant et al. [94]	✓	✓	✓	✓	✓	✓	✓	✓	✗	✓	✗	✓	10
Sun et al. [95]	✓	✓	✓	✗	✓	✓	✓	✓	✓	✓	✗	✓	10
Caralt et al. [96]	✓	✓	✓	✗	✓	✓	✓	✓	✓	✓	✗	✓	10
Ayden et al. [97]	✓	✓	✓	✗	✓	✓	✓	✓	✗	✓	✗	✓	9
Bhutto et al. [98]	✓	✓	✓	✗	✓	✓	✓	✓	✗	✗	✗	✓	8
Peng et al. [99]	✓	✓	✓	✓	✓	✓	✓	✓	✗	✗	✗	✓	9
Wang et al. [100]	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✗	✓	11
Zhuang et al. [101]	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	12

High levels of compliance were observed for key methodological criteria, including clearly stated objectives (Q1), dataset description (Q2), preprocessing procedures (Q3), evaluation metrics (Q7), baseline comparisons (Q8), and ethical considerations (Q12), all of which were satisfied by all included studies (54/54).

In contrast, several important weaknesses were identified in reproducibility and robustness. Only 45 studies reported sufficient details to enable replication (Q10), 26 explicitly addressed class imbalance (Q4), and 18 made their code publicly available (Q9). Most notably, only two studies explicitly evaluated model generalizability across datasets or settings (Q11).

Operational definitions and examples used during the quality assessment are provided in Supplementary Table 16 to support transparency and consistency.

Table 16.

Quality assessment criteria used in the review

ID	Quality assessment question	How the criterion was judged ()	Example of ()
Q1	Is the study objective clearly defined and aligned with ICD code prediction from clinical text?	The paper clearly states that predicting ICD codes from clinical text is a main objective.	The study explicitly focuses on automated ICD code prediction.
Q2	Does the study provide a detailed description of the data source?	The data source is named and described clearly enough to understand what was used.	The dataset and type of clinical notes are clearly reported.
Q3	Is the preprocessing of clinical text adequately described?	The paper explains how the clinical text was processed before modeling.	Text cleaning or preprocessing steps are described.
Q4	Are methods for addressing class imbalance discussed?	The authors acknowledge label imbalance and explain how it was handled.	The study discusses imbalance in ICD labels and mitigation methods.
Q5	Are machine or deep learning models clearly explained?	The model used is described beyond just naming it.	The model architecture and main components are explained.
Q6	Does the study justify the choice of models and algorithms?	The paper explains why a particular model was chosen.	The authors provide a reason for using a specific model.
Q7	Are the evaluation metrics appropriate and clinically meaningful?	Metrics appropriate for multi label ICD coding are reported.	Standard multi label evaluation metrics are used.
Q8	Does the study compare results with baseline or existing approaches?	The proposed method is evaluated against at least one baseline or prior method.	Results are compared with baseline models.
Q9	Is the code or implementation publicly available?	The authors provide a public link to the code or implementation.	Source code is shared in a public repository.
Q10	Are methods described in sufficient detail to allow replication?	Enough methodological detail is provided to reasonably reproduce the study.	Training and evaluation procedures are clearly described.
Q11	Does the study assess model generalizability?	The model is evaluated beyond a single dataset or experimental setting.	Performance is tested on an external or independent dataset.
Q12	Is the dataset de identified and ethically compliant?	The paper states that the data are de identified and ethically approved or publicly released.	Ethical approval or use of a de identified public dataset is reported.

Detailed Review of Selected Literature

This section presents a detailed review of 54 selected studies on automated ICD coding, examining each study in terms of the dataset used, the methodology employed, and the performance evaluation reported. To ensure a coherent and structured analysis, the review is organised around the PICO sub-questions defined in “Objective and Research Questions” section. Specifically, “Dataset Used in the Studies (Population – P)” section addresses the Population (P) sub-question by investigating the clinical text types, dataset configurations, and patient populations used across the reviewed studies. “Preprocessing Techniques (Intervention – I) through “Loss Functions and Optimization Strategies (Intervention – I)” sections address the Intervention (I) sub-questions by documenting how NLP preprocessing strategies, feature extraction methods, external knowledge integration, prediction architectures, and optimisation strategies have been applied across the literature. “Comparative Performance Analysis (Comparison – C)” section addresses the Comparison (C) sub-question by evaluating how different model families compare in predictive performance, external robustness, and temporal evolution. “Evaluation Metrics Used in Automated ICD Coding Studies (Outcome – O)” and “Main Findings and Limitations Across Reviewed Studies (Outcome – O)” sections address the Outcome (O) sub-questions by documenting the evaluation metrics utilised across the field and summarising the main findings and methodological limitations identified throughout the literature.

Dataset Used in the Studies (Population – P)

The 54 studies in this systematic review used a variety of different datasets to develop and evaluate automated ICD coding models based on clinical texts, discharge summaries, nursing notes, and electronic health records (EHR). The datasets annotated with ICD-9 or ICD-10 codes are of varying sizes, language, accessibility and coding scope (full vs. top frequent codes). Table 5 provides a summary of the three dataset groups identified across the reviewed studies, with full details for all 54 studies provided in Supplementary Table 17. In the following, we describe the main datasets and patterns identified for the studies.

Table 5.

Summary of datasets used in included studies (n=54). Full dataset details including note types, exact data sizes, ICD variants, and train/validation/test splits for all 54 studies are provided in Supplementary Tables 17

Group	Datasets	ICD system	Typical split	n
MIMIC Only	MIMIC-III; 9 studies also include MIMIC-II	ICD-9-CM predominant; 1 study uses mixed ICD-9 and ICD-10	Standard 47,719/1,631/3,372; some studies use 80/20	31
MIMIC + Other	MIMIC-III paired with Xiangya, CodiEsp, BioASQ, PHS Biobank, Fuwai-CHD, CDTD-TP, CPLT, or TPD	Mixed ICD-9-CM and ICD-10 (most); ICD-10 only (2 studies)	Standard MIMIC split for MIMIC portion; custom splits for external datasets	9
Non-MIMIC	French CépiDc, Bulgarian Medical, Stockholm EPR, Shandong Hospital, Hainan Hospital, Taiwan OPD, PAP Pathology, French EHR, Chinese hospital records	ICD-10 predominant; ICD-9-CM (1 study); mixed (1 study)	Variable: 70/15/15, 80/20, 90/10, 10-fold CV	14

CV: cross-validation

Table 17.

Datasets used in studies for automated ICD coding

Study	Dataset(s)	Note type and data size (if reported)	ICD system	Split (Tr/Val/Test)
a) MIMIC Only (31 entries; pure MIMIC III or MIMIC III + MIMIC II usage)
MIMIC III Std Full: 47,719/1,631/3,372; Top50: 8,067/1,574/1,730; MIMIC II Std: 20,533/0/2,282
Li et al. [48]	MIMIC III; MIMIC II	Summaries; 52,962 (MIMIC III), 22,815 (MIMIC II); ICD codes reported: 6,984 (MIMIC III)	ICD 9 CM	47,665 / 0 / 5,297; MIMIC II: Std
Chen et al. [50]	MIMIC III	Summaries; 46,600; Top50 subset	ICD 9 CM	43,000 / 1,800 / 1,800
Huang et al. [51]	MIMIC III	Admissions; 52,726; ICD codes reported: 6,918	ICD 9 CM	50% / 25% / 25%
Biseda et al. [55]	MIMIC III	Top50 and Top100 subsets; size not specified	ICD 9 CM	N/S
Gangavarapu et al. [56]	MIMIC III	Notes; 223,556 notes	ICD 9 CM	70% / 15% / 15%
Li and Yu [57]	MIMIC III; MIMIC II	Summaries; ICD codes reported: 8,921 (MIMIC III) and 5,031 (MIMIC II); Top50 used	ICD 9 CM	Full: Std; Top50: Std; MIMIC II: Std
Vu et al. [59]	MIMIC III; MIMIC II	Summaries; 52,722; ICD codes reported: 8,929; Top50 used; MIMIC II notes mentioned	ICD 9 CM	Top50: 11,317 / 0 / 0; MIMIC II: Std
Raja and Tuwani [60]	MIMIC III	Summaries; 47,427; ICD codes reported: 8,921; Top50 used	ICD 9 CM	Top50: Std
Cao et al. [64]	MIMIC III; MIMIC II	Summaries and notes; ICD codes reported: 8,921 (MIMIC III) and 5,031 (MIMIC II)	ICD 9 CM	Full/Top50: Std; MIMIC II: Std
Hu et al. [69]	MIMIC III	Summaries; 52,722; ICD codes reported: 8,929	ICD 9 CM	Full/Top50: Std (Var: 47,724 / 1,632 / 3,372)
Zhou et al. [70]	MIMIC III; MIMIC II	Summaries and notes; ICD codes reported: 8,921 (MIMIC III) and 5,031 (MIMIC II)	ICD 9 CM	Full/Top50: Std; MIMIC II: Std
Rajendran et al. [73]	MIMIC III	Admissions; 44,765; ICD codes reported: 32 ICD 10	Mixed (ICD 9 CM + ICD 10 CM)	28,201 / 3,134 / 12,430; Top50: 8,044 / 804 / 1,725
Huang et al. [74]	MIMIC III; MIMIC II	Summaries and notes; size not specified (row reports split variant only)	ICD 9 CM	Std (Var: 47,724 / 1,632 / 3,372); MIMIC II: Std
Wu et al. [75]	MIMIC III	Summaries; size not specified	ICD 9 CM	80% / 0% / 20%
Liu et al. [76]	MIMIC III	Subsets (CN, CN DS, Top50); Top50 size reflected in split	ICD 9 CM	Top50: 8,066 / 0 / 1,729
Yuan et al. [78]	MIMIC III	Summaries; size not specified (row reports split variant only)	ICD 9 CM	Full/Top50: Std (Var: 47,723 / 1,632 / 3,372)
Zhang et al. [80]	MIMIC III	Summaries; ICD codes reported: 8,929; test partitioning described	ICD 9 CM	Std (test divided: head 1,446; body 1,779; tail 860 codes)
Ren et al. [81]	MIMIC III	Summaries; 52,722; ICD codes reported: 8,929	ICD 9 CM	Full/Top50: Std
Wu et al. [82]	MIMIC III; MIMIC II	Summaries; 52,728 (MIMIC III) and 22,815 (MIMIC II); ICD codes reported: 8,921 and 5,031	ICD 9 CM	Full/Top50: Std (Var: 47,724 / 1,632 / 3,372); MIMIC II: Std
Qiu et al. [83]	MIMIC III	Summaries; 52,722; ICD codes reported: 8,929	ICD 9 CM	Full/Top50: Std
Boukhers et al. [84]	MIMIC III	Summaries; ICD codes reported: 4,216	ICD 9 CM	Full/Top50: Std
Nath et al. [85]	MIMIC III	Admissions and Top50 subset sizes reported in row: 2,976 and 11,371	ICD 9 CM	2,976 / 0 / 11,371
Liu et al. [88]	MIMIC III; MIMIC II	Patients and stays reported: 46,520 patients (MIMIC III), 25,328 stays (MIMIC II); ICD codes reported: 8,929 and 5,031	ICD 9 CM	Std (Var: 47,719 / 1,632 / 3,372); MIMIC II: 19,392 / 1,141 / 2,282
Carberry and Xu [91]	MIMIC III	Doctor notes plus diagnoses; size not specified	ICD 9 CM	0.80 / 0 / 0.20
Aden et al. [92]	MIMIC III	Top10 and Top50 rows reported: 677,738 and 1,058,988	ICD 9 CM	0.80 / 0 / 0.20
Wu et al. [93]	MIMIC III; MIMIC II	Summaries; MIMIC II summaries mentioned; size not specified (row reports split variant only)	ICD 9 CM	Std (Var: 47,724 / 1,632 / 3,372); MIMIC II: Std
Merchant et al. [94]	MIMIC III	Summaries plus lab reports; sizes reflected in split	ICD 9 CM	8,066 / 1,473 / 1,729
Caralt et al. [96]	MIMIC III	Summaries; ICD codes reported: 8,921 with diagnosis and procedure breakdown stated	ICD 9 CM	Full/Top50: Std (Var: 47,724 / 1,632 / 3,372)
Ayden et al. [97]	MIMIC III	Records reported: 58,576; structured plus text; time span 2001 to 2012	ICD 9 CM	70% / 15% / 15%
Peng et al. [99]	MIMIC III	Summaries; Top50 subset; size reported: 11,317	ICD 9 CM	8,067 / 1,574 / 1,730
Wang et al. [100]	MIMIC III	Summaries; size not specified (row reports split variant only)	ICD 9 CM	Full: Std (Var: 47,724 / 1,632 / 3,372); Top50: Std
b) MIMIC + Other (9 entries; MIMIC combined with non MIMIC datasets)
Zeng et al. [49]	MIMIC III; BioASQ3	Summaries (MIMIC III) and BioASQ3 samples; sizes reported: 52,962 and 12,208,342	ICD 9 CM	47,665 / 0 / 5,297
Yu et al. [52]	Xiangya (private); MIMIC III	Records reported: 7,732 (Xiangya); admissions reported: 52,962 (MIMIC III)	Mixed (ICD 9 CM + ICD 10)	Xiangya: 0.90/0/0.10; MIMIC III: N/S
Chen and Ren [53]	MIMIC III; TPD	Notes (MIMIC III) and descriptions (TPD); sizes reported: 49,583 and 9,257	Mixed (ICD 9 CM + ICD 10)	N/S
Sonabend et al. [62]	MIMIC III; PHS Biobank	Summaries (MIMIC III) and notes (PHS Biobank); sizes reported: 52,691 and 193,677	Mixed (ICD 9 CM + ICD 10)	0.60/0.20/0.20 (CV)
Schäfer and Friedrich [65]	CodiEsp corpus; MIMIC III	Summaries; size reported for MIMIC III: 59,652	ICD 10 (mapped from ICD 9)	750 / 0 / 250
Wu et al. [79]	MIMIC III; Xiangya (private)	Summaries and records; sizes reported: 52,726 and 7,732	Mixed (ICD 9 CM + ICD 10)	80%/10%/10%
Zhao et al. [87]	Fuwai CHD; MIMIC III CHD	CHD records; sizes reported: 75K and 4,361	Mixed (ICD 9 CM + ICD 10)	70%/10%/20%
Mou et al. [89]	CDTD TP; MIMIC III	Descriptions; sizes reported: 22,038 and 22,528	Mixed (ICD 9 CM + ICD 10)	70%/10%/20% (both)
Bhutto et al. [98]	CPLT; MIMIC III	Notes; CPLT size reported: 1,380; MIMIC III Top50 summaries referenced	ICD 10 CM	CPLT: 1,104 / 138 / 138; MIMIC III Top50: Std
c) Non MIMIC (14 entries; no MIMIC usage, often multilingual or private)
Atutxa et al. [54]	CépiDc; ISTAT4; KSH5	135K certs, French; 18K certs, Italian; 100K certs, Hungarian	ICD 10	French: 125,384/0/11,931; Italian: 14,502/0/3,618; Hungarian: 84,703/0/21,176
Zhang et al. [58]	EHR notes (private)	7.5M notes, about 1M patients	ICD 10	70%/10%/20%
Hsu et al. [61]	Medical center	168,186 Records	Mixed (ICD 9 CM + ICD 10 CM)	10 fold CV
Velichkov et al. [63]	Bulgarian Medical	345,591 Diagnoses	ICD 10	80%/10%/10%
Singaravelan et al. [66]	Medical center	146,343 Records; 140K used	ICD 9 CM	N/S
Sen et al. [67]	PAP	Pathology reports; sizes reported: 39K reports, 66K annotations	ICD 10 CM	70%/15%/15%
Chraibi et al. [68]	French EHR	Summaries; size reported: 134K	ICD 10	N/S
Popescu et al. [71]	CépiDC	Certificates and texts; sizes reported: 19,995 and 33,953	ICD 10	0.9055/0.0945/2,500
Wang et al. [72]	Shandong Hospital	EMRs; size reported: 138,503	ICD 10	N/S
Chomutare et al. [77]	Stockholm EPR Gastro	Summaries; size reported: 6,002	ICD 10	80%/0%/20%
Masud et al. [86]	Taiwan OPD	Notes; size reported: 21,953	ICD 10 CM	90%/0%/10%
Chen et al. [90]	Hainan Hospital	Diagnoses; size reported: 1,003,558; 162,085 used	ICD 10	143,680 / 9,214 / 9,191
Sun et al. [95]	Chinese hospital	Summaries; size reported: 12,788	ICD 10	70%/20%/10%
Zhuang et al. [101]	Chinese centers	Notes; size reported: 584,969; 9,337 used; 13 codes	ICD 10	5,734 / 1,913 / 2,007

N/S: Not specified; certs: certificates; ICD: ICD 9 or ICD 10 codes; CV: cross validation

MIMIC Datasets

The Medical Information Mart for Intensive Care (MIMIC) datasets, particularly MIMIC-III, were the most frequently used datasets, appearing in 40 studies as the primary benchmark due to their scale, accessibility, and rich clinical information. MIMIC-III is a publicly available dataset consisting of 52,722–59,652 discharge summaries from over 40,000 patients, containing approximately 8,929 unique ICD-9 codes for diagnosis and procedure codes, as detailed in Supplementary Table 17. To address class imbalance and enhance model training by limiting labels to the most common codes, several studies focused on MIMIC-III Top50 or Top100 subsets. Notably, nine studies combined MIMIC-II and MIMIC-III to validate model generalizability across versions and varying data distributions. These datasets serve as benchmarks owing to their scale and availability, though class imbalance remains a key challenge, often mitigated through subsetting.

More recent resources, such as MIMIC-IV [40], extend this benchmark with updated electronic health records and ICD-10 coding, while annotation-focused datasets such as MDACE [41] provide explicit code evidence spans that support explainable and auditable automated coding.

MIMIC-III Combined with Other Datasets

The combination of MIMIC-III with other datasets was employed in nine studies (Supplementary Table 17b) to enable cross-lingual ICD coding, ICD-9 to ICD-10 conversion, transfer and unsupervised learning, and domain-specific tasks. For cross-lingual analysis, MIMIC-III was paired with the Xiangya dataset containing 7,732 Chinese clinical records with 1,177 ICD-10 codes [52, 79]. To integrate domain knowledge, it was combined with the TPD dataset consisting of 9,257 Chinese code descriptions and 345 ICD-10 codes [53] and with BioASQ’s 12,208,342 samples for transfer learning [49]. Unsupervised learning was supported by pairing with the PHS Biobank containing 193,677 notes [62]. Additionally, ICD-9 codes from MIMIC-III were mapped to ICD-10 for use with the CodiEsp corpus consisting of Spanish clinical documents [65]. In disease-specific applications, MIMIC-III was used alongside the Fuwai-CHD dataset with 75,000 Chinese records [87], the CDTD-TP dataset with 22,038 Chinese textual descriptions and 3,579 ICD-10 codes [89], and the CPLT dataset with 1,380 Pakistani liver transplant clinical notes, targeting the Top50 ICD codes [98].

Other Public and Proprietary Datasets

As detailed in Supplementary Table 17c, fourteen studies utilized non-MIMIC datasets, often multilingual or proprietary, to test models on ICD-10 systems, multilingual text, and specialized clinical domains, supporting real-world evaluation in non-English and non-public settings. Public datasets included the French CépiDc with 135,000 death certificates [54], the CLEF eHealth 2017 dataset with 19,995 certificates and 33,953 diagnostic texts [71], the Bulgarian Medical Dataset with 345,591 diagnoses and 5,854 ICD-10 codes [63], and the Stockholm EPR Gastro corpus with 6,002 de-identified Swedish summaries [77]. Large-scale EHR datasets featured 7.5 million notes from approximately 1 million patients [58] and medical center records from 2003–2017 (168,186 records with 2,017 ICD-9 codes, [61]) or 2012–2017 (146,343 records with 1,871 ICD-9 codes, [66]). Proprietary datasets encompassed Chinese clinical data from Hainan Hospital (1,003,558 diagnoses, 162,085 used, [90]), Shandong Hospital (138,503 EMRs, [72]), a Chinese hospital (12,788 summaries, [95]), and Chinese medical centers (584,969 notes, 9,337 used for 13 ICD-10 codes, [101]), as well as Taiwanese outpatient data (21,953 notes, [86]), French EHR records (134,000 summaries, [68]), and pathology reports (39,000 annotations, [67]). While these datasets enhance diversity for ICD-10 and multilingual coding, proprietary ones limit public accessibility and reproducibility.

Train-Test Split Variability

The train-test splits observed across the datasets, as detailed in Supplementary Table 17, vary significantly, ranging from standard fixed splits (e.g., 80% train / 20% test in MIMIC-Only [75, 91]) to more complex configurations like 10-fold cross-validation (Non-MIMIC [61]) or stratified sampling (Non-MIMIC [63]). In the MIMIC-Only section (31 entries), the MIMIC-III dataset frequently employs a consistent fixed split (e.g., 47,719 train / 1,631 validation / 3,372 test, with slight variations in [69, 74, 78, 82, 88, 93, 96, 100]), which promotes benchmarking but may overlook data drift or evolving clinical practices. In contrast, datasets in the Non-MIMIC section, such as the Chinese Clinical Dataset [90] with 143,680 training diagnoses and smaller validation/test sets (9,214 and 9,191, respectively), emphasize maximizing training data for large-scale models. The MIMIC + Other section (9 entries) often blends these approaches, such as in [98] (CPLT: 1,104/138/138; MIMIC-III Top50: standard), to address class imbalance across diverse sources. The lack of explicit split details in some datasets (for example, MIMIC-only [55]; MIMIC + Other [52, 53]; Non-MIMIC [66, 68, 72]) poses challenges to reproducibility and underscores the need for standardized reporting in medical coding research. Cross-validation, as seen in Non-MIMIC [61] and MIMIC + Other [62], provides a robust evaluation method but remains less common, possibly due to computational demands or dataset scale limitations.

Clinical Text Type Distribution

Looking at the clinical text types used across the 54 included studies (Table 5), there is a clear concentration on discharge summaries. Around 34 studies rely exclusively on this document type, 12 use a combination of note types or full EHR records including summaries, nursing notes, and laboratory reports, and only about 8 focus primarily on other document types such as death certificates or pathology reports. This dominance of discharge summaries has been noted in earlier reviews of the field [21] and continues to characterise the literature through the most recent period covered here [36, 37]. The concern with this pattern is that discharge summaries are inherently more structured and clinically focused than documents such as progress notes or nursing notes [21], meaning that strong performance on discharge summaries does not necessarily reflect how well a model would perform on the broader range of documents encountered in clinical practice. This is further supported by the quality assessment results in Table 4, where only two studies explicitly evaluated model generalisability across different datasets or settings.

Preprocessing Techniques (Intervention – I)

Preprocessing is one of the critical components in studies involving large-scale text or time series data, as it improves data quality, enhances model performance, and ensures feature consistency across datasets [22, 24, 25]. Common preprocessing strategies can be broadly categorized into text-based techniques (e.g., tokenization, stopword removal, sentence splitting), data cleaning approaches (e.g., filtering rare words, removing short or irrelevant texts), and domain-specific adjustments (e.g., de-identification, timestamp extraction) [22, 25].

As shown in Fig. 2 and Table 6 The most widely used technique was tokenization, which was reported in 38 studies and used to segment text into manageable units. Stopword removal was applied in 17 studies to eliminate common, low-information words, while lowercasing or text normalization appeared in 17 studies to standardize textual input. Punctuation and symbol removal were reported in 15 studies, typically to reduce noise in clinical narratives.

Fig. 2.

Preprocessing techniques used across ICD coding studies. (See also Table 6)

Table 6.

List of preprocessing techniques used in automated ICD coding studies

Preprocessing technique	Studies	Count
Tokenization	[48–52, 54–60, 63, 64, 68–70, 72–74, 77, 79–83, 85–87, 91–94, 96–98, 100, 101]	38
Stopword removal	[48, 50–52, 56, 63, 66, 68, 73, 75, 79, 85, 86, 92, 97–99]	17
Lowercasing / Text normalization	[53, 54, 57, 59–61, 71, 76, 81, 83, 87, 91, 94, 97–100]	17
Truncation / Length limiting	[55, 57, 59, 69, 74, 78, 79, 83, 87, 94, 100, 101]	12
Punctuation / Symbol removal	[57, 59, 61, 65, 66, 73, 75, 81, 86, 87, 90, 94, 97, 98, 100]	15
Sentence splitting / Chunking	[55, 67, 76]	3
Padding	[55, 58, 75, 79]	4
TF-IDF / Word selection	[66, 72, 79, 84]	4
Filtering rare words	[60, 69]	2
Named Entity Recognition (NER)	[55, 62, 68, 84]	4
Short text filtering / Thresholds	[49, 51, 58, 89]	4
Custom word segmentation (Jieba, etc.)	[52, 61, 66, 87]	4
Abbreviation expansion	[55]	1
Discourse extraction / Segmentation	[50, 80]	2
String matching / Synonym handling	[56, 68, 70]	3
De-identification / Anonymization	[48, 55, 68, 88, 95]	5
Regular expression cleaning	[68, 89]	2
Section prioritization / Extraction	[76, 85]	2
UNK token replacement	[60]	1
ID removal	[91, 100]	2
Keyword extraction (KeyBERT)	[101]	1

To manage sequence input for models, 12 studies used truncation or length limiting, and 4 studies [55, 58, 75, 79] used it with padding, to ensure uniform input lengths. Additional preprocessing strategies included TF-IDF-based word selection [66, 72, 79, 84], sentence splitting or chunking [55, 67, 76], and rare word filtering [60, 69], each found in 2 to 4 studies.

Meanwhile, more targeted preprocessing techniques were also reported. These included short text filtering or thresholding [49, 51, 58, 89], and de-identification or anonymization [48, 55, 68, 88], each used in 4 to 5 studies. Some of the other methods included regular expression cleaning [68, 89], discourse extraction or segmentation [50, 80], and string matching or synonym handling [56, 68, 70]. A few studies adopted specialized preprocessing, such as custom segmentation using Jieba [52, 61, 66], abbreviation expansion [55], section or timestamp extraction [76, 85], UNK token replacement [60], ID removal [91, 100], and keyword extraction using KeyBERT [101].

These preprocessing strategies contribute to model effectiveness and robustness in complementary ways. Techniques such as tokenisation and text normalisation improve input consistency and reduce vocabulary sparsity, enabling more reliable feature learning across diverse clinical documents [22]. Domain-specific methods including de-identification, clinical segmentation, and section prioritisation reduce noise and focus model attention on diagnostically relevant content, which is particularly important for generalisation across datasets and document types [24]. Length management strategies such as truncation address the computational constraints imposed by long clinical narratives, balancing coverage with efficiency. Together, these strategies form a critical foundation for downstream model performance, and their increasing domain-specificity across the reviewed period reflects a maturing understanding of the unique challenges posed by clinical text.

Feature Extraction Techniques (Intervention – I)

A wide range of feature extraction techniques have been employed across the 54 reviewed studies, reflecting the evolving landscape of natural language processing for automated ICD coding. Different types of feature extraction techniques were used in different studies, are shown in Fig. 3 and Table 7 includes:

Fig. 3.

Feature extraction techniques used across ICD coding studies. (See also Table 7)

Table 7.

Representation and feature construction techniques used in automated ICD coding studies

Representation category	Specific methods	Studies	Count
Sparse lexical representations	TF–IDF, term weighting, bag-of-words	[51, 68, 69, 71–73]	6
Static word embeddings	Word2Vec (CBOW/Skip-gram), custom-trained embeddings	[48, 50–53, 60–62, 64, 66, 68, 69, 72, 73, 75, 77–81, 83, 85–87, 96–99]	28
Document-level embeddings	Doc2Vec	[48, 56]	2
Contextual embeddings from pretrained language models	BERT, ClinicalBERT, BioBERT, XLNet, RoBERTa (used as representations)	[55, 58, 63, 65, 71, 74, 76, 81, 88–90, 92, 97, 101]	14
Hierarchy-aware label representations	Tree-LSTM, hyperbolic embeddings, hierarchical curriculum learning	[53, 62, 64, 76, 77, 81, 93]	7
Knowledge-grounded representations	Knowledge graphs, UMLS synonyms, code descriptions	[62, 72, 78, 84, 100]	5
Meta-embeddings	Ensemble, mean, locally linear combinations	[73, 94]	2
Specialized text representations	Character-enhanced embeddings, section-aware embeddings, reconciled code embeddings	[52, 76, 80, 97]	4
Traditional NLP feature engineering	LDA, topic modeling, rule-based term features	[56, 85]	2
Other representation techniques	Segment pooling, sentence extraction, hybrid representation pipelines	[74, 83, 91, 101]	4

Sparse Lexical Representations

such as TF–IDF or term weighting were used in 6 studies, mainly as baselines or in lightweight systems. While efficient, they do not capture semantic context and perform poorly on long and complex notes.

Static Word Embeddings

most commonly Word2Vec or custom-trained embeddings, were the most widely used representation, appearing in 28 studies. These dense vectors provide a compact semantic representation and are typically used as inputs to neural predictors. Document-level embeddings such as Doc2Vec were used in only 2 studies, reflecting their limited ability to localize fine-grained evidence.

Contextual Representations from Pretrained Language Models

(e.g., BERT variants) were adopted in 14 studies, reflecting a shift toward context-aware modeling. These representations are particularly well suited to clinical text, where meaning is highly context dependent.

Hierarchy-Aware Label Representations

were used in 7 studies to encode the ICD taxonomy structure directly in the representation space, mainly to support rare and related codes.

Knowledge-Grounded Representations

which integrate code descriptions, UMLS resources, or knowledge graphs, appeared in 5 studies.

Specialized Representations

such as character-level or section-aware embeddings were used in 4 studies, while meta-embedding strategies combining multiple representation sources appeared in 2 studies. Finally, traditional NLP feature engineering approaches such as topic models were used in only 2 studies, indicating a clear move away from handcrafted features.

Overall, these trends show a shift from sparse and static representations toward contextual, structure-aware, and knowledge-enhanced representations, which form the input foundation for the prediction models discussed in the next section.

External Knowledge Integration in Automated ICD Coding (Intervention – I)

Several studies enhance ICD code prediction by incorporating external knowledge beyond raw clinical text. Across all reviewed studies, 13 explicitly incorporated the ICD hierarchy as a structural prior (Table 8), making this the most widely adopted strategy for rare-code learning [59, 64, 76, 81].

Table 8.

Comparison of external knowledge usage in automated ICD coding studies

External knowledge category	Studies	Representation enhanced	Why it helps	Count
ICD hierarchy / Label tree structure	[53, 59, 63, 64, 76, 77, 79–81, 83, 88, 93, 99]	Label representations and training dynamics	Enables knowledge transfer from frequent to rare codes and preserves the medical taxonomy	13
Code descriptions / Titles (Label Text)	[48, 53, 54, 62, 74–76, 78–80, 82, 83, 87, 90, 94, 95]	Label representations and label-aware document vectors	Grounds predictions in medically meaningful semantics and supports rare or zero-shot codes	16
Graph-based knowledge (Hierarchy / Co-occurrence / Knowledge Graphs)	[64, 65, 70, 78, 82, 84, 93, 100]	Label representations and document–label interactions	Captures label dependencies beyond flat multi-label classification	8
Ontology / Synonyms / Terminology resources	[52, 54, 62, 63, 68, 77, 78, 85]	Document–code alignment	Bridges informal clinical language and standardized ICD terminology	8
Data augmentation / Distribution knowledge	[55, 60, 63, 65, 72, 91, 98, 99]	Input signals and training robustness	Improves recall, robustness, and long-tail code performance	8
Discourse / Temporal / Structural note knowledge	[52, 80, 88, 96]	Document representations across time or sections	Reduces noise and improves evidence aggregation	4

Another widely adopted approach integrates ICD code descriptions or titles, enabling semantic grounding of labels and improved alignment between clinical text and target codes, particularly for infrequent or unseen labels [74, 79, 82, 95].

Graph-based representations, including hierarchy graphs and label co-occurrence graphs, are used to capture dependencies among ICD codes that are not addressed by flat multi-label models [64, 84, 93]. Ontology-driven and synonym-based resources further support document–code alignment by resolving lexical variation in clinical narratives [54, 78, 85].

A smaller subset of studies applies data augmentation or distribution-aware strategies to mitigate class imbalance and improve robustness [55, 60, 91]. Finally, discourse-level and temporal knowledge is occasionally incorporated to exploit document structure or longitudinal information in clinical records [80, 96].

Overall, the comparison highlights substantial diversity in how external knowledge is defined and injected, ranging from representation-level enhancements to training-time constraints. While these strategies are motivated by different objectives, such as rare-code prediction, interpretability, or robustness, their evaluation settings vary considerably, underscoring the need for cautious cross-study comparison.

Techniques Used for Prediction (Intervention – I)

This section reviews the main families of prediction models used for automated ICD coding, as summarized in Table 9 and Fig. 4. We organize the literature according to four dominant model families: CNN-based, RNN-based, Transformer-based, and Hybrid or ontology-based predictors. These techniques differ significantly in terms of their capacity to model semantic content, manage high-dimensional inputs, and generalize across imbalanced and hierarchically structured code sets [8].

Table 9.

Study counts by model type (2019–2024, with one additional 2025 study). Totals exceed 54 due to multi-architecture and hybrid methods

Year	CNN-based	RNN-based	Transformer-based	Hybrid / Ontology-based	Total
2019	[48–51]	[52–54]	–	–	7
2020	[55, 57, 61, 64]	[59]	[58, 63, 65]	[56, 60, 62, 64]	12
2021	[69, 72]	[66–68]	[70–72]	[73]	9
2022	[75]	[79, 80]	[74, 76, 78, 81, 82]	[77, 78, 80–82]	13
2023	[84, 86]	[83, 89]	[85, 87, 88, 90]	[84]	9
2024	[92, 98]	[94, 97]	[91, 93, 95, 96, 99, 100]	[93–95, 100]	14
2025	–	–	[101]	–	1
Total (non-exclusive)	15	13	22	15	65*

* Study [101] was published in January 2025 and was identified during the database search conducted in January 2025, as documented in Table 14. It is retained given its direct relevance to emerging prompt-based approaches and its retrieval within the active search window

Fig. 4.

Trends in model architecture for automated ICD coding. (See also Table 9)

CNN-Based Predictors

CNN-based models are among the earliest and most widely adopted neural architectures for ICD coding. They capture local n-gram and phrase-level patterns and have evolved from basic multi-channel designs [48–50] to more sophisticated architectures such as MultiResCNN [57]. Several extensions further improve their effectiveness, including knowledge-enhanced variants such as KG-MultiResCNN [84] and multi-level dilated CNNs designed to better handle long documents and long-tail code distributions [100].

RNN-Based Predictors

RNN-based models, including LSTM, BiLSTM, and GRU variants, explicitly model sequential dependencies in clinical narratives. These models have been widely used in early and mid-period studies [52, 53, 59, 78, 82]. Key extensions include label-aware attention mechanisms (LAAT) [59], and tree-LSTM structures for modeling the ICD hierarchy [53].

Transformer-Based Predictors

Transformer-based models currently represent the dominant paradigm in automated ICD coding, accounting for the largest number of studies in recent years. This family includes domain-adapted ClinicalBERT models [55, 92, 97] and specialized architectures such as BERT-XML with EHR-specific vocabularies [58]. Other extensions include ensemble systems combining multiple pretrained models [65, 71] and recent work exploring GPT-4-based or prompt-based approaches [91, 101]. Several studies also propose hierarchical encoding or segment-level pooling strategies to better handle long clinical documents [76]. In addition, off-the-shelf LLMs have been investigated for automated ICD coding in zero-shot and few-shot settings by framing the task as information extraction guided by ICD code descriptions rather than supervised classification [42].

Hybrid and Ontology-Based Predictors

Hybrid approaches combine multiple modeling strategies or explicitly integrate domain knowledge and label structure. Several studies incorporate hierarchical information using hyperbolic or structure-aware representations [64, 81, 93]. Other methods integrate UMLS synonyms [78] or medical knowledge graphs [84], or apply curriculum learning from general to specific codes [81]. Multi-modal approaches that combine clinical text with numerical or structured data are also included in this category [94].

Overall, the distribution in Table 9 shows a clear shift from earlier CNN and RNN architectures toward transformer-based and hybrid systems, reflecting the increasing emphasis on context-rich representations and explicit modeling of label structure in automated ICD coding.

Attention Mechanisms in Automated ICD Coding

Before the introduction of transformer architectures, attention mechanisms were widely adopted to improve automated ICD coding by identifying clinically relevant text segments in long medical documents. Early studies applied global or sequence level attention on CNN or RNN representations to emphasize important words and phrases, improving both predictive performance and interpretability while treating all ICD codes uniformly [52, 54]. To better address the extreme multi label nature of ICD coding, later work introduced label wise attention mechanisms, where each ICD code is associated with its own attention vector, enabling code specific alignment between text spans and labels and improving scalability and accuracy [59]. Subsequent studies extended label wise attention by incorporating hierarchical structure and external medical knowledge. Hierarchy aware and knowledge guided attention mechanisms aligned attention with the ICD taxonomy or code descriptions, improving semantic consistency, interpretability, and rare code prediction [67, 75]. Despite these advances, pre transformer attention models relied on static embeddings and shallow contextualization, limiting their ability to capture long range dependencies in clinical text. These limitations motivated the transition to transformer based architectures, which generalize attention into a unified self attention framework with deeper contextual representations [58, 74].

Temporal Evolution of Model Families

Early work from 2019 to 2020 was dominated by CNN and RNN-based models, which established the first strong neural baselines for large-scale ICD coding, as shown in Fig. 4 and Table 9. From 2022 onward, transformer-based approaches accounted for the largest share of studies each year. Hybrid and knowledge-integrated systems were present throughout the reviewed period, appearing in each year from 2020 onward, though their numbers varied across years rather than following a consistent upward trend. Alongside these dominant families, a smaller number of studies explored alternative formulations, including neural machine translation as a sequence-to-sequence approach [89] and prompt-based methods built on GPT-based and pretrained language model pipelines [42, 91, 101].

Loss Functions and Optimization Strategies (Intervention – I)

Due to the multi-label and imbalanced nature of ICD datasets, specialised optimization strategies were employed across the reviewed studies. Advanced loss functions including focal loss [53] and asymmetric loss [81] were specifically designed to address class imbalance and improve prediction of rare codes. Innovative techniques such as contrastive learning with hierarchy-guided samples [93], self-distillation for iterative refinement [70], and bias removal methods [99] further improved performance on challenging cases. The diversity of optimisation strategies documented here reflects the field’s ongoing efforts to address class imbalance across model families, though the consistently poor macro F1 scores reported across studies suggest that existing approaches including specialised loss functions and hierarchical sampling have not yet fully resolved this challenge.

Evaluation Metrics Used in Automated ICD Coding Studies (Outcome – O)

Evaluating automated ICD coding models is inherently challenging due to the multi-label nature of the task, the presence of severe class imbalance, and the hierarchical structure of ICD taxonomies. Consequently, the reviewed studies employ a wide range of evaluation metrics, each capturing different aspects of model performance and practical relevance.

Classification-based metrics, including precision, recall, and F1-score, are the most commonly reported (n=41). These metrics are typically presented using micro- or macro-averaging schemes. Micro-averaged metrics emphasize overall performance and are influenced by frequent codes, whereas macro-averaged metrics weight all labels equally and are more sensitive to rare-code behavior. As a result, both perspectives are often reported to provide a more complete view of model performance (e.g., [48, 50, 61]).

To better align evaluation with clinical workflows, where only a limited number of suggested codes can be reviewed, many studies additionally report ranking-based metrics such as Precision@k or Recall@k (n=22). These measures assess whether relevant ICD codes appear among the top-ranked predictions and are particularly relevant for decision-support scenarios (e.g., [59, 69]).

Several studies also adopt threshold-free metrics, including ROC-AUC and PR-AUC, to evaluate discriminative performance without fixing a decision threshold (n=23). Such metrics are especially useful in settings with highly imbalanced label distributions and complement threshold-dependent measures such as F1-score (e.g., [60, 74]).

A smaller subset of studies reports accuracy-based metrics (e.g., overall or subset accuracy, top-1 accuracy; n=13), multi-label–specific metrics such as Hamming Loss (n=2), and additional specialized measures including loss-based metrics (n=2), hierarchical evaluation (n=2), and interpretability-related measures (n=3) (e.g., [52, 71, 97]).

Table 10 provides a consolidated overview of the evaluation metrics used across the reviewed studies and links each metric category to the corresponding publications. Together, these metrics reflect differing evaluation priorities, including overall accuracy, rare-code sensitivity, ranking quality, and alignment with clinical use cases.

Table 10.

Evaluation metrics used in automated ICD coding studies

Metric category	Studies	Count (%)
Classification metrics (Micro / Macro / Weighted Precision, Recall, F1-score)	[48–55, 61–66, 68–70, 72–80, 82–84, 86–89, 91–93, 97–101]	41 (75.9%)
Threshold-free metrics (AUC, AUROC, AUPRC)	[50, 56, 58–60, 64, 69, 73, 74, 76, 78, 79, 81–84, 87, 88, 93, 94, 96, 100, 101]	23 (42.6%)
Ranking-based metrics (P@k, Recall@k, MRR@k)	[50, 57, 59, 60, 69, 70, 73, 74, 76, 78, 79, 81–84, 88, 93–96, 98, 100]	22 (40.7%)
Accuracy-based metrics (Accuracy, Subset Accuracy, Top-1 Accuracy)	[51, 61, 63, 67, 68, 71, 72, 90–92, 97, 99, 101]	13 (24.1%)
Multilabel-specific metrics (Hamming Loss)	[52, 53]	2 (3.7%)
Loss-based metrics (Cross-Entropy, Logit Loss, LRL)	[56, 97]	2 (3.7%)
Hierarchical evaluation metrics	[61, 71]	2 (3.7%)
Interpretability and other metrics	[85, 87, 90]	3 (5.6%)

Comparative Performance Analysis (Comparison – C)

Table 11 presents a normalized comparative synthesis of ICD code prediction performance from 2019 to 2024, focusing exclusively on studies that employed the standard MIMIC-III train-validation-test splits for either the full ICD label set or the Top-50 code configuration. This restriction ensures valid cross-study comparison and mitigates confounding effects arising from heterogeneous data partitions, label spaces, and evaluation protocols.

Table 11.

Performance trends by model family and dataset configuration (2019–2024)

Year	Model family	Micro F1	Macro F1	Studies (Micro F1)	n
MIMIC-III Full Dataset (8,921 codes; Standard split: 47,719/1,631/3,372)
2019–20	CNN-based	0.552	0.085	MultiResCNN [57] (0.552)	1
2019–20	Hybrid	0.551	0.090	HyperCore [64] (0.551)	1
2019–20	RNN-based	0.575	0.107	JointLAAT [59] (0.575)	1
2021–22	Hybrid	0.553–0.588	0.086–0.140	Discourse-aware [80] (0.588); HiCu [81] (0.571); JAN [82] (0.553)	3
2021–22	RNN-based	0.584	0.103	MSMN [78] (0.584)	1
2021–22	Transformer	0.559–0.598	0.104–0.119	PLM-ICD [74] (0.598); Self-distillation [70] (0.559)	2
2023–24	Hybrid	0.561–0.605	0.102–0.112	Multi-KG [100] (0.605); HyperGCN [93] (0.578); KG-MultiResCNN [84] (0.561)	3
2023–24	RNN-based	0.595	0.108	DILM-ICD [83] (0.595)	1
2023–24	Transformer	0.460–0.583	0.108	XR-Transformer [88] (0.583); LAHST [96] (0.460)	2
MIMIC-III Top-50 Dataset (50 codes; Standard split: 8,067/1,574/1,730)
2019–20	CNN-based	0.670	0.606	MultiResCNN [57] (0.670)	1
2019–20	Hybrid	0.663	0.609	HyperCore [64] (0.663)	1
2019–20	RNN-based	0.716	0.661	JointLAAT [59] (0.716)	1
2021–22	CNN-based	0.652	0.603	SWAM [69] (0.652)	1
2021–22	Hybrid	0.671	0.612	JAN [82] (0.671)	1
2021–22	RNN-based	0.725	0.683	MSMN [78] (0.725)	1
2021–22	Transformer	0.717–0.735	0.679–0.690	HiLAT [76] (0.735); Self-distillation [70] (0.717)	2
2023–24	Hybrid	0.695–0.734	0.645–0.692	Multi-KG [100] (0.734); HyperGCN [93] (0.729); KG-MultiResCNN [84] (0.695)	3
2023–24	RNN-based	0.729	0.683	DILM-ICD [83] (0.729)	1
2023–24	Transformer	0.633	0.586	Debiasing [99] (0.633)	1

Note: This table includes only studies using the standard MIMIC-III splits shown in the dataset headers (train/dev/test) to ensure valid cross-study comparison. For configurations with multiple studies (), all contributing papers and their individual Micro F1 scores are listed in the Studies column. Micro F1 and Macro F1 ranges denote minimum and maximum values within each configuration. n denotes the number of studies contributing to each row

Trends on the MIMIC-III Full Label Set

For the full ICD label space, early studies (2019–2020) showed similar performance across architectural families. RNN-based architectures (JointLAAT [59]: 0.575), CNN-based (MultiResCNN [57]: 0.552), and hybrid (HyperCore [64]: 0.551) models clustered around 0.55 Micro F1, reflecting the difficulty of modeling extreme label imbalance in the full label setting. Between 2021 and 2022, transformer-based models emerged as competitive alternatives. PLM-ICD [74] achieved 0.598 Micro F1, representing a +4% improvement over the 2019–20 baseline. Hybrid models showed wider variability (0.553–0.588), with Discourse-aware [80] reaching 0.588. Macro F1 scores remained below 0.14 across all families, suggesting gains on infrequent codes remained constrained by dataset sparsity.

In 2023–2024, hybrid models incorporating auxiliary knowledge achieved the highest performance. Multi-KG [100] reached 0.605 Micro F1, demonstrating consistent improvements across knowledge-enhanced architectures (0.561–0.605). RNN-based models maintained steady progression (DILM-ICD [83]: 0.595), while transformers exhibited substantial variability (0.460–0.583), highlighting sensitivity to design choices in highly imbalanced settings. Figure 5A illustrates these temporal trends on the Full dataset, showing transformers’ consistent improvement trajectory alongside the steady progression of hybrid models.

Fig. 5.

Temporal performance trends by model architecture (2019–2024). Scatter plots show peak Micro F1 scores for studies using standard MIMIC-III splits. Point sizes are proportional to the number of studies. Dashed lines indicate trend directions. Panel A shows MIMIC-III Full dataset; Panel B shows MIMIC-III Top-50 subset. Transformers demonstrate consistent improvement on Full dataset, while Top-50 performance plateaus after 2021-22

Trends on the MIMIC-III Top-50 Configuration

Restricting to the Top-50 most frequent codes yielded higher performance. In 2019–2020, RNN-based models (JointLAAT [59]: 0.716) substantially outperformed CNN-based (0.670) and hybrid (0.663) approaches, demonstrating the effectiveness of recurrent architectures when label sparsity is mitigated.

From 2021 to 2022, transformers became dominant. HiLAT [76] achieved peak performance of 0.735 Micro F1, representing the highest reported score across all configurations and time periods. Performance differences between RNN (MSMN [78]: 0.725), transformer (0.717–0.735), and hybrid (JAN [82]: 0.671) models narrowed at the high end.

In 2023–2024, hybrid models incorporating external knowledge achieved the highest performance. Multi-KG [100] reached 0.734, closely approaching the 2021–22 transformer peak. RNN-based models maintained stable high performance (DILM-ICD [83]: 0.729). The convergence near 0.73 suggests potential saturation effects in the Top-50 setting, as visualized in Fig. 5B, where performance plateaus after 2021–22.

Interpretation and Limitations

No single model family consistently dominates across all time periods and label spaces. Hybrid architectures incorporating external knowledge achieved the highest recent performance on both configurations (Multi-KG [100]: 0.605 Full, 0.734 Top-50), suggesting that auxiliary supervision from medical ontologies provides consistent benefits. Transformer-based models show promise, particularly HiLAT [76] (0.735), but exhibit substantial variability. Figure 6 provides a comprehensive view of performance distributions across model families (Panels A & B) and year-over-year peak performance evolution (Panels C & D), demonstrating both improvement patterns and saturation effects. These comparisons are restricted to studies using identical MIMIC-III splits and should be interpreted as relative performance trends rather than definitive rankings.

Fig. 6.

Performance distribution and year-over-year trends by model family. Panels A & B show Micro F1 score distributions across model families for MIMIC-III Full and Top-50 datasets respectively. Box plots display median (center line), interquartile range (box), and outliers; black points show individual studies. Panels C & D display year-over-year peak performance evolution, demonstrating improvement patterns and saturation effects

Head-to-Head Performance and External Validation

Table 12A presents the highest-performing model from each architectural family. Multi-KG [100] achieved the strongest Full dataset performance (Micro F1: 0.605), while HiLAT [76] demonstrated peak Top-50 performance (0.735). Notably, only 50% (4 of 8) top performers conducted external validation, highlighting a critical gap in generalizability assessment.

Table 12.

Top-performing models and external validation performance

(A) Top-Performing Models by Family Under Standard MIMIC-III Evaluation
Model (Year)	Family	Micro F1	Macro F1	Micro AUC	Macro AUC	P@5	P@8	Ext. Val.
MIMIC-III Full Dataset (47,719/1,631/3,372)
Multi-KG (2024)	Hybrid	0.605	0.112	0.994	0.948	–	0.784	No
PLM-ICD (2022)	Transformer	0.598	0.104	0.989	0.926	0.844	0.771	Yes
JointLAAT (2020)	RNN	0.575	0.107	0.988	0.921	–	–	Yes
MultiResCNN (2020)	CNN	0.552	0.085	0.986	0.910	–	0.734	Yes
MIMIC-III Top-50 Dataset (8,067/1,574/1,730)
HiLAT (2022)	Transformer	0.735	0.690	0.950	0.927	0.681	–	No
Multi-KG (2024)	Hybrid	0.734	0.692	0.950	0.928	0.683	–	No
DILM-ICD (2023)	RNN	0.729	0.683	0.950	0.928	0.683	–	No
MultiResCNN (2020)	CNN	0.670	0.606	0.928	0.899	0.641	–	Yes
(B) External Validation Performance: MIMIC-III to MIMIC-II Generalization
Model (Year)	Family	MIMIC-III	MIMIC-II	Degradation	Citation
PLM-ICD (2022)	Transformer	0.598	0.504	−15.7%	[74]
MultiResCNN (2020)	CNN	0.552	0.464	−15.9%	[57]
JointLAAT (2020)	RNN	0.575	0.491	−14.6%	[59]
HyperCore (2020)	Hybrid	0.551	0.477	−13.4%	[64]
HyperGCN (2024)	Hybrid	0.578	0.513	−11.2%	[93]
Self-distillation (2021)	Transformer	0.559	0.498	−10.9%	[70]
JAN (2022)	Hybrid	0.553	0.504	−8.9%	[82]
Mean (n=7)	–	0.567	0.493		–
Std. Dev.	–	0.018	0.017	2.7%	–

Note (A): Represents the single highest-performing model from each family under standard MIMIC-III splits to enable direct head-to-head comparison. Bold values indicate peak performance within each metric column for the respective dataset configuration. Ext. Val. indicates whether external validation (MIMIC-II or other datasets) was performed. P@5/P@8 values shown when reported; – indicates metric not reported. Multi-KG (Full, 0.605) and HiLAT (Top-50, 0.735) represent the highest benchmarks through 2024. Only 4 of 8 top performers (50%) conducted external validation

Note (B): Only 7 of 29 standard-split studies (24.1%) performed external validation on MIMIC-II, revealing limited generalization assessment. Mean performance degradation of 13.0% (SD=2.7%) indicates moderate overfitting to MIMIC-III benchmark characteristics. Models sorted by degradation magnitude (worst to best). PLM-ICD and MultiResCNN exhibited largest degradation ( 16%), while JAN demonstrated most robust generalization (-8.9%). Higher MIMIC-III performance did not correlate with better generalization. Degradation calculated as: (MIMIC-II F1 - MIMIC-III F1) / MIMIC-III F1 100%

Table 12B reveals that only 24.1% of standard-split studies (7 of 29) performed external validation on MIMIC-II. Among validated models, mean performance degradation was 13.0% (SD=2.7%), with PLM-ICD [74] and MultiResCNN [57] exhibiting the largest drops (16%), while JAN [82] demonstrated the most robust generalization (-8.9%). Importantly, higher MIMIC-III performance did not correlate with better external validation, as PLM-ICD (0.598) showed similar degradation to MultiResCNN (0.552), suggesting that benchmark performance alone may be insufficient for assessing real-world deployment potential.

Incomparability of Non-MIMIC Dataset Results

Non-MIMIC datasets reported higher F1 scores (mean 0.87±0.09) but are not comparable to MIMIC benchmarks due to: (1) smaller label spaces (mean 847 vs. 8,929 codes), (2) single-institution reduced variability, (3) ICD version differences, and (4) domain-specific subsets with simplified coding contexts [71, 86, 90]. Tables 11 and 12 therefore restrict analysis to MIMIC-III standard splits to ensure valid quantitative comparison.

Detailed outcomes from mixed MIMIC and non-MIMIC studies are reported descriptively in Supplementary Tables 18 and 19.

Table 18.

Results for mixed settings using MIMIC and external datasets. These studies are presented descriptively because datasets, label spaces, and evaluation protocols differ across papers

Year	Study	Datasets	Main reported results
2019	[49]	MIMIC III and BioASQ3	MIMIC III: Micro F1 0.420 (Precision 0.483, Recall 0.371)
2019	[52]	Xiangya and MIMIC III	Xiangya: Micro F1 0.639 (Precision 0.704, Recall 0.586). MIMIC III: Micro F1 0.420 (Precision 0.432, Recall 0.408)
2019	[53]	MIMIC III and TPD (Chinese)	MIMIC III: Micro F1 0.613, Macro F1 0.456 (Micro Precision 0.598, Micro Recall 0.630). Chinese: Micro F1 0.972, Macro F1 0.566 (Micro Precision 0.785, Micro Recall 0.966)
2020	[62]	MIMIC III and PHS Biobank	AUC 0.920
2020	[65]	CodiEsp and MIMIC III	F1 0.337, MAP 0.259
2022	[79]	MIMIC III and Xiangya	MIMIC III Full: Micro F1 0.583, Macro F1 0.098 (AUC Micro 0.986, AUC Macro 0.941). MIMIC III Top 50: Micro F1 0.699, Macro F1 0.629. Xiangya Full: Micro F1 0.806, Macro F1 0.161. Xiangya Top 50: Micro F1 0.856, Macro F1 0.818
2023	[87]	Fuwai CHD and MIMIC III CHD	Fuwai CHD Top 100: Macro F1 0.962 (AUC Macro 0.989). MIMIC III CHD Top 50: Macro F1 0.405 (AUC Macro 0.661)
2023	[89]	CDTD TP and MIMIC III	Chinese CDTD TP: Precision 0.837, Recall 0.795, F1 0.816. MIMIC III: Precision 0.429, Recall 0.320, F1 0.366
2024	[98]	CPLT and MIMIC III Top 50	CPLT: Micro F1 0.862, Macro F1 0.250. MIMIC III Top 50: Micro F1 0.705, Macro F1 0.655 (P at 5 equals 0.664)

Table 19.

Results reported on non-MIMIC datasets. These studies are not directly comparable due to differences in datasets, ICD versions, languages, and task formulations, and are therefore presented descriptively

Year	Study	Dataset	Main reported results
2019	[54]	CépiDC, ISTAT, KSH	French: F1 0.838 (Precision 0.841, Recall 0.835). Hungarian: F1 0.963 (Precision 0.970, Recall 0.955). Italian: F1 0.952 (Precision 0.960, Recall 0.945).
2020	[58]	Private EHR	AUC Micro 0.970, AUC Macro 0.927.
2020	[61]	Medical Center	Full code: F1 0.579, Accuracy 0.409. Chapter: F1 0.492, Accuracy 0.580. Block: F1 0.431, Accuracy 0.494. Three digit: F1 0.404, Accuracy 0.433.
2020	[63]	Bulgarian Medical Dataset	Macro F1 0.800, Accuracy 0.810, MRR at 5 0.880.
2021	[66]	Medical Center	F1 0.393, Recall 0.579, Top 10 Recall 0.674.
2021	[67]	PAP Dataset	Micro F1 0.700, Accuracy 0.640.
2021	[68]	French EHR	Precision 0.845, Recall 0.826, F1 0.828, Accuracy 0.842.
2021	[71]	CépiDC	F1 0.943, Accuracy 0.952.
2021	[72]	Shandong Hospital	F1 0.921, Accuracy 0.917.
2022	[77]	Stockholm EPR	Top 10 Accuracy 0.190 to 0.340 (Full codes), 0.480 to 0.670 (3 character codes).
2023	[86]	Taiwan OPD	Cardiology: F1 0.980, Precision 0.960, Recall 0.990, Accuracy 0.940. Metabolism: F1 0.860. Psychiatry: F1 0.750. Neurology: F1 0.710.
2023	[90]	Hainan Hospital	Precision 0.979, Recall 0.983, F1 0.981, Accuracy 0.986.
2024	[95]	Chinese Hospital	Full set: AUC Micro 0.912, R at 10 0.839, P at 10 0.084, MRR at 10 0.657. Top 50: AUC Micro 0.955, R at 10 0.944, P at 10 0.094, MRR at 10 0.798.
2025	[101]	Chinese Multi center	MDR BERT: Micro F1 0.838, AUC Macro 0.958. mT5 XXL: Micro F1 0.846, AUC Macro 0.945. Qwen2.5: Micro F1 0.822, AUC Macro 0.848.

Main Findings and Limitations Across Reviewed Studies (Outcome – O)

Table 13 presents a representative selection of 12 studies drawn from across the reviewed period, documenting the main objectives, key findings, and reported limitations across each major model family from 2019 to 2024. Full study-by-study details for all 54 included studies are provided in Supplementary Tables 20. The implications of these findings and limitations for future research directions are examined in “Discussion” section.

Table 13.

Representative studies on automated ICD coding: key findings and limitations. Full study-by-study details for all 54 included studies are provided in Supplementary Tables 20

Study	Summary of objective & Key finding	Limitation
Huang et al. [51]	Compared CNN and RNN with traditional ML on MIMIC-III; deep models showed superior performance for ICD-9 assignment.	Focused on top codes; limited rare-code analysis.
Li and Yu [57]	Developed MultiResCNN combining multi-filter and residual CNNs; achieved top ICD prediction results on MIMIC-III.	High computational cost; English-only data.
Vu et al. [59]	Introduced label attention with hierarchical joint learning; improved rare-code accuracy on MIMIC-III.	Computationally heavy; evaluated only on English data.
Cao et al. [64]	Proposed HyperCore using hyperbolic space and GCN for code hierarchy modelling; improved rare-code accuracy.	Complex model; interpretability issues.
Hu et al. [69]	Developed SWAM linking convolutional filters to code snippets; improved interpretability and clinical transparency.	Tested on top-50 ICD codes only.
Huang et al. [74]	Proposed PLM-ICD addressing label and length limitations; achieved state of the art performance on MIMIC-III.	Needs validation beyond MIMIC-III.
Liu et al. [76]	Developed HiLAT transformer with hierarchical label-wise attention; achieved peak Top-50 performance.	Top-50 focus; high computational requirements.
Yuan et al. [78]	Used ICD code synonyms via UMLS for code matching; improved rare-code representation.	Memory intensive; tested only on English MIMIC-III.
Qiu et al. [83]	Designed iterative learning model refining predictions; improved recall for rare codes.	Evaluated only on MIMIC-III ICD-9.
Boukhers et al. [84]	Integrated Wikidata knowledge graph with residual CNN; enhanced prediction for unseen clinical text.	Knowledge graph coverage dependent.
Wu et al. [93]	Built hyperbolic GCN with contrastive learning; improved label relation modelling and class imbalance handling.	Scalability and interpretability limitations.
Wang et al. [100]	Integrated auxiliary knowledge via GCN; achieved highest Full dataset Micro F1 of 0.605 across all reviewed studies.	Limited auxiliary sources; evaluated on MIMIC-III only.

Table 20.

Summary of selected studies on automated ICD coding

Study	Summary of objective & Key finding	Limitation
Li et al. [48]	Proposed DeepLabeler (CNN + Doc2Vec) for ICD-9 coding; improved multi-label accuracy via local and global feature fusion.	Dataset imbalance and small samples for rare codes.
Zeng et al. [49]	Applied MeSH-based transfer learning with CNN for ICD-9 prediction; enhanced performance through domain adaptation.	Dependent on large external datasets and imbalanced codes.
Chen et al. [50]	Used multi-channel CNN with attention for ICD coding; captured discourse and medical semantics effectively.	Evaluated only on Chinese data; limited generalizability.
Huang et al. [51]	Compared CNN/RNN with traditional ML on MIMIC-III; deep models showed superior performance for ICD-9 assignment.	Focused on top codes; limited rare-code analysis.
Yu et al. [52]	Introduced BiRNN with multi-layer attention and char embeddings for Chinese summaries; improved semantic capture.	Small dataset and weak rare-code handling.
Chen and Ren [53]	Used hierarchical tree-LSTM leveraging ICD descriptions; improved focus and disambiguation.	Did not use structured EHR data; low-frequency codes remain weak.
Atutxa et al. [54]	Proposed multilingual seq-to-seq model with attention for ICD-10; achieved interpretable mappings.	Errors with abbreviations and informal expressions.
Biseda et al. [55]	Used ClinicalBERT with augmentation and label balancing for MIMIC-III ICD coding; improved efficiency.	Limited to top codes; generalizability uncertain.
Gangavarapu et al. [56]	Employed fuzzy similarity-based cleansing and supervised classification for nursing notes; improved code group prediction.	Focused on groups not individual codes; time-aware prediction absent.
Li and Yu [57]	Developed MultiResCNN combining multi-filter and residual CNNs; achieved top ICD prediction results on MIMIC.	High computational cost; English-only data.
Zhang et al. [58]	Built EHR-specific BERT with multi-label attention for ICD-10; outperformed baselines, esp. for rare codes.	Did not model ICD hierarchy; resource-intensive.
Vu et al. [59]	Introduced LAAT (Label Attention) with hierarchical joint learning; improved rare-code accuracy on MIMIC.	Computationally heavy; evaluated only on English data.
Raja and Tuwani [60]	Analyzed adversarial robustness of ICD models; minor input typos significantly reduced accuracy.	Focused solely on typo attacks; no defense methods.
Hsu et al. [61]	Predicted ICD-9 codes using subjective text via CNN; showed potential for remote diagnosis.	Lower accuracy due to limited data and modality.
Sonabend et al. [62]	Developed UNITE unsupervised ICD assignment using semantic relevance; achieved high portability.	Single-disease prediction and CUI dependency.
Velichkov et al. [63]	Compared BERT models for Bulgarian ICD-10 texts; multilingual fine-tuning improved accuracy.	Class imbalance and limited medical vocabulary.
Cao et al. [64]	Proposed HyperCore (hyperbolic + GCN) for code hierarchy modeling; improved rare-code accuracy.	Complex model; interpretability issues.
Schäfer and Friedrich [65]	Used BioBERT/XLNet ensemble for ICD-10; achieved best MAP with multilingual data.	Low performance on unseen codes.
Singaravelan et al. [66]	Predicted ICD-9 from subjective reports using LSTM/GRU; demonstrated recall up to 57.9%.	Single-center data; only subjective input.
Sen et al. [67]	Reformulated ICD-10 as hierarchical tagging with supervised attention; improved interpretability and F1.	Pathology-only dataset; limited labels.
Chraibi et al. [68]	Built deep learning system for French ICD-10 coding; achieved strong accuracy on free text.	Imbalanced and mislabeled data; CPU-limited.
Hu et al. [69]	Developed SWAM (explainable CNN) linking filters to code snippets; improved interpretability.	Tested on top-50 ICD codes only.
Zhou et al. [70]	Introduced shared-representation network with self-distillation; improved rare-code prediction.	Limited to English MIMIC ICD-9 data.
Popescu et al. [71]	Compared ML vs. BERT for death certificates; BERT achieved 0.95 accuracy and 0.94 F1.	Single-label focus; limited complexity.
Wang et al. [72]	Built MA-GRN (CNN+attention) for Chinese ICD-10 EMRs; achieved more than 91% accuracy.	Single-label focus; Chinese-only data.
Rajendran et al. [73]	Combined structured/unstructured meta-embeddings for ICD coding; improved ensemble accuracy.	Focused on top codes; no BERT models tested.
Huang et al. [74]	Proposed PLM-ICD addressing label/length limits; achieved SOTA on MIMIC.	Needs validation beyond MIMIC.
Wu et al. [75]	Built KAICD (Bi-GRU + CNN + code knowledge); improved text-title integration.	Rare codes remain weak; limited feature diversity.
Liu et al. [76]	Developed HiLAT transformer with label-wise attention; provided explainability via attention maps.	Top-50 focus; high computation.
Chomutare et al. [77]	Hybrid deep + fuzzy matching for rare ICD-10; improved top-10 accuracy for rare codes.	Limited vocabulary and deep model accuracy.
Yuan et al. [78]	Used ICD synonyms (MSMN) for code matching; improved UMLS-based representation.	Memory-intensive; tested only on English MIMIC.
Wu et al. [79]	Introduced pseudo label-wise attention for BERT; reduced redundancy and improved zero-shot coding.	Dataset-specific tuning required.
Zhang et al. [80]	Incorporated discourse structure for balanced ICD coding; improved macro-F1 on MIMIC.	English-only; adaptation needed for other languages.
Ren et al. [81]	Proposed hierarchical curriculum (HiCu) leveraging ICD hierarchy; improved rare-code learning.	Dependent on label hierarchy quality.
Wu et al. [82]	Joint attention network (JAN) combining doc & label attention; improved interpretability and F1.	High complexity; limited rare-code gains.
Qiu et al. [83]	Designed iterative learning (DILM-ICD) refining predictions; improved recall for rare codes.	Evaluated only on MIMIC-III ICD-9.
Boukhers et al. [84]	Integrated Wikidata with residual CNNs (KG-MultiResCNN); enhanced unseen-text prediction.	Knowledge graph coverage-dependent.
Nath et al. [85]	Developed NEAR (unsupervised NER + embeddings); achieved explainable ICD-9 coding without labels.	Limited by NER and embedding coverage.
Masud et al. [86]	Built CNN for ICD-10 using SOAP/drug data; achieved F1 up to 0.98 in cardiology.	Single-center; moderate dataset size.
Zhao et al. [87]	Developed BW_att (BERT+word2vec+attention) for CHD ICD coding; achieved high macro-F1.	Focused on CHD codes only.
Liu et al. [88]	Compared PLM-ICD, XR-Transformer, XR-LAT; improved rare-code macro-AUC.	ICD-9 only; rare-code bias persists.
Mou et al. [89]	Applied NMT (RAANMT) for multi-label ICD mapping; outperformed prior baselines.	Short text focus; single-institution data.
Chen et al. [90]	Used semantic matching + analogical reasoning for ICD-10; achieved F1=0.981 on Chinese data.	Slow; diagnostic text only.
Carberry and Xu [91]	Used GPT-4 for semantic extraction + hierarchical classification; improved explainability and accuracy.	Relies on GPT-4; subset evaluation.
Aden et al. [92]	Benchmarked RNN, LSTM, BiLSTM, BERT; BioClinicalBERT achieved F1>0.8 on MIMIC.	Focus on top codes; limited scale.
Wu et al. [93]	Built hyperbolic GCN with contrastive learning; captured label relations and improved imbalance handling.	Scalability and interpretability limits.
Merchant et al. [94]	Developed CD-LAAT + ensemble using structured/unstructured data; achieved strong F1/P@k.	English-only; rare codes remain challenging.
Sun et al. [95]	Proposed SuM (summarization + matching) for ICD-10; improved long-text and tail-code performance.	Single-center oncology data.
Caralt et al. [96]	Built LAHST for early ICD prediction; accurately predicts before discharge.	ICU-only data; high compute.
Ayden et al. [97]	Two-stream model combining text + numerical data; boosted micro-F1 to 0.75.	Limited to ICU; missing-data sensitivity.
Bhutto et al. [98]	Developed DRCNN-ATT for ICD-10-CM; achieved F1=0.86 on private dataset.	Not validated across domains.
Peng et al. [99]	Introduced debiasing method improving rare-code prediction; outperformed LLM baselines.	Evaluated on top-50 codes only.
Wang et al. [100]	Integrated DRG/CPT knowledge + GCN; improved long-text ICD prediction accuracy.	Limited auxiliary sources; MIMIC-only.
Zhuang et al. [101]	Used prompt learning + keyword-filtered BERT for real-time ICD-10; achieved micro-F1 0.84.	Cardiovascular codes only; limited generalizability.

Discussion

This systematic review of 54 studies from 2019 to 2024 offers a comprehensive and critical synthesis of ICD coding, a key task in healthcare informatics that leverages NLP and ML to assign standardized diagnostic and procedural codes from clinical texts, covering a range of clinical text types, including discharge summaries, nursing notes, EHRs, and progress notes. This review addresses limitations in prior work and illuminates the field’s technological evolution, persistent challenges, and areas requiring further development.

Synthesis of Key Findings

Population: Dataset Scope and Generalisability

The analysis of the data set in “Dataset Used in the Studies (Population – P)” section confirms that the field remains heavily concentrated on a single benchmark. Forty of the 54 reviewed studies are based primarily on MIMIC-III, and as documented in Table 12B, external validation was conducted in only 24% of standard-split studies. Its ICU-centric and English-language scope limits the applicability of reported findings to outpatient settings, non-English clinical environments, and diverse patient populations [21]. The variability in train-test splits documented in “Train-Test Split Variability” section further complicates cross-study comparison, and the near-exclusive reliance on discharge summaries across approximately 34 studies raises additional concerns about document-type generalisability [21].

Intervention: Methodological Progress

The preprocessing, feature extraction, and modelling trends documented in “Preprocessing Techniques (Intervention – I)” through “Techniques Used for Prediction (Intervention – I)” sections reflect meaningful advancement over the review period. Preprocessing strategies have matured toward clinically adapted approaches, external knowledge integration has become a distinguishing feature of leading architectures as summarised in Table 8, and the progression from CNN and RNN baselines toward transformer-based and hybrid systems, documented in Table 9, has produced consistent improvements in overall predictive performance [74, 76]. However, as Table 11 shows, Macro F1 scores remained below 0.14 across all model families throughout the review period, indicating that gains on frequent codes have not translated to meaningful progress on rare ones.

Comparison: Model Family Performance and Robustness

The comparative analysis in “Comparative Performance Analysis (Comparison – C)” section and Table 11 shows that no single model family consistently outperformed the others across time periods, label configurations, and evaluation settings. Hybrid architectures incorporating structured medical knowledge showed relatively consistent performance across the reviewed studies, as evidenced by Table 12. However, the external validation results in Table 12B reveal a mean performance degradation of 13.0% across the seven models tested on MIMIC-II, with higher benchmark scores showing no correlation with better generalization [57, 74, 82].

Outcome: Evaluation Metrics and Clinical Alignment

The evaluation landscape documented in “Evaluation Metrics Used in Automated ICD Coding Studies (Outcome – O)” section and Table 10 highlights a persistent gap between commonly reported metrics and those most relevant to clinical practice. Micro-averaged F1, reported in 41 of 54 studies, rewards performance on frequent codes and can mask complete failure on rare ones. Macro-averaged F1 and ranking-based metrics such as Precision@k, which better reflect real coding workflow requirements, remain inconsistently adopted in the literature [28]. The near-complete absence of hierarchical evaluation metrics that account for the structure of the ICD taxonomy further limits the reliability of cross-study comparisons.

Critical Gaps and Challenges

Rare Code Prediction

Class imbalance remains a persistent problem. Rare ICD codes that appear in fewer than one percent of records are consistently underpredicted, regardless of which model architecture is used. As established in “External Knowledge Integration in Automated ICD Coding (Intervention – I)” section, 13 of the 54 studies incorporated the ICD hierarchical structure to help with rare code learning [59, 76, 81], making this one of the most popular approaches for dealing with infrequent codes. Despite this fairly widespread adoption, performance on long tail codes remains poor [77]. This suggests that using hierarchical embeddings by itself is not enough to solve extreme class imbalance. Most studies still use standard cross-entropy loss and balanced sampling techniques. Very few have explored newer methods such as few-shot learning, meta-learning, contrastive learning, or distribution-aware training strategies that might actually help when label distributions are severely skewed.

Dataset Accessibility and Standardization

Proprietary datasets (e.g., Xiangya, Hainan Hospital) enrich real-world validation but restrict access, disproportionately impacting researchers in low-resource settings. Even open-access datasets like MIMIC-III exhibit inconsistent train-test splits, undermining comparability as documented in “Train-Test Split Variability” section. The absence of standardized datasets, preprocessing pipelines, and benchmarking protocols limits the development of generalizable models, particularly for multilingual and low-resource contexts.

Metric Inconsistency and Clinical Relevance

Despite advances in automated systems for clinical tasks, there remains a notable disconnect between commonly used performance metrics (e.g., micro and macro-averaged F1 scores) and their actual clinical relevance. As highlighted by [28] and [29], while statistical metrics can provide precise performance estimates and confidence intervals, they often fail to capture the complexity and priorities of real-world clinical decision-making. Furthermore, as discussed by [30] in the context of multi-label learning, traditional evaluation methods may not adequately reflect the nuanced requirements of healthcare applications. This inconsistency emphasizes the need for evaluation frameworks that prioritize both technical accuracy and meaningful clinical outcomes.

Explainability Deficit

The limited focus on explainability undermines clinical trust and adoption. Clinicians require granular, interpretable outputs to validate code assignments, particularly for rare or ambiguous diagnoses. Most existing approaches rely on attention mechanisms or evidence extraction strategies to highlight relevant text spans [59, 69, 79], with only a handful of studies applying post-hoc analysis tools to inspect model behaviour. While such techniques provide useful cues about which parts of the clinical note influence predictions, they do not constitute true clinical explanations, as they do not explicitly justify why a specific ICD code is assigned according to coding guidelines or medical reasoning. This gap is particularly acute for transformer-based models, whose complexity exacerbates their black-box nature [26, 27]. More recent work has explored LLMs to generate natural language rationales for code assignments [91], which improves readability but raises concerns about factual grounding and faithfulness. Overall, current approaches tend to offer model-centric explanations rather than clinician-centric justifications, and closing this gap will require future systems to integrate explicit coding rules and medical knowledge rather than relying solely on attention mechanisms.

Multilingual and Global Applicability

Non-English datasets such as Xiangya for Chinese [52] and CodiEsp for Spanish [11] have advanced cross-lingual ICD coding. However, while approximately 17 studies addressed non-English clinical texts [52, 54] covering several languages, low-resource languages remain underrepresented. Many proprietary datasets limit public accessibility and reproducibility [21], restricting progress in scaling multilingual models through transfer learning, domain adaptation, and collaborative dataset curation.

Limitations of the Review

This review’s methodological rigor, guided by PRISMA 2020, ensures transparency, yet several limitations warrant acknowledgment. The exclusion of 38 studies due to access limitations represents a recognized challenge in systematic review methodology. This limitation is transparently reported in accordance with PRISMA 2020 requirements for documenting retrieval failures [19]. While this may introduce some selection bias, our approach follows established best practices for handling inaccessible literature in systematic reviews. Similarly, the focus on English-language studies excludes potentially relevant non-English research, limiting global perspectives. The decision to prioritize peer-reviewed studies over gray literature (e.g., preprints, conference abstracts) ensures quality but may miss emerging trends, a trade-off for methodological robustness. Variability in dataset accessibility, train-test splits, and evaluation metrics across studies precluded quantitative synthesis (e.g., meta-analysis), limiting the ability to derive pooled performance estimates. The reliance on six databases, while comprehensive, may overlook niche repositories, though pilot searches mitigated this risk. These limitations highlight the need for broader access to research outputs and standardized reporting in future reviews.

Future Directions

Large Language Models (LLMs)

Recent work has begun to explore LLMs (e.g., GPT-4, LLaMA) for automated ICD coding primarily in assistive or prompt-based roles, including sentence-level extraction and few-shot or zero-shot coding, rather than as fully end-to-end classifiers. Early studies suggest the potential of prompt-based approaches using pretrained language models for ICD coding assistance; however, widespread adoption remains constrained by data privacy requirements, reliance on proprietary systems, reproducibility challenges, and the absence of standardized evaluation protocols for clinical coding tasks [101].

Multimodal Data Integration

An emerging direction in automated ICD coding is the integration of clinical text with structured EHR data, such as laboratory results, vital signs, and diagnosis histories. Combining unstructured notes with structured variables can improve coding for complex admissions and comorbid conditions, particularly when key diagnostic evidence is sparsely documented in text alone. However, reported performance gains are often modest and highly dataset-dependent, and multimodal systems introduce additional challenges related to data heterogeneity, feature alignment, and cross-institution portability, which have limited their widespread adoption in current benchmark-driven research [31, 74].

Explainable AI for Clinical Trust

Future research should prioritise clinician-friendly interfaces that provide granular, natural-language explanations of code assignments, quantify model confidence, and highlight critical text segments, going beyond attention visualisation toward methods such as SHAP and LIME that offer more faithful explanations. Regulatory frameworks such as the EU’s AI Act underscore the urgency of this direction, making explainability a pressing design requirement rather than an optional addition.

Real-World Deployment and Validation

Translating research into practice requires addressing computational constraints, integrating models into EHR systems, and validating performance across diverse healthcare settings (e.g., urban vs. rural, high- vs. low-resource). Pilot studies in real-world environments, coupled with clinician feedback, could bridge the gap between research and application.

Ethical and Regulatory Considerations

As automated ICD coding systems scale, ethical challenges, data privacy, algorithmic bias, and equitable access must be addressed. Robust governance frameworks, informed by interdisciplinary collaboration (e.g., clinicians, ethicists, policymakers), will ensure responsible deployment.

The 5P Evidence-Grounded Research Agenda for Next-Generation ICD Coding

Although automated ICD coding has achieved substantial technical progress, our systematic review reveals persistent and structural limitations that hinder clinical generalization, reproducibility, and real-world adoption. These include the heavy reliance on narrow and homogeneous datasets, limited external validation, weak performance on rare codes, insufficient transparency, and poor integration into clinical workflows.

Rather than proposing a prescriptive evaluation checklist, we introduce the 5P agenda as an evidence-grounded research synthesis derived directly from the gaps observed across the 54 reviewed studies. This agenda identifies five strategic priorities for next-generation ICD coding research: Population Diversity, Performance Robustness, Prediction of Rare Codes, Provenance Transparency, and Practical Integration (Fig. 7). The agenda is grounded in the empirical findings of this review and is intended to complement reporting standards such as PRISMA and TRIPOD-AI. While those standards define how studies should be reported, the 5P agenda highlights what dimensions ICD coding systems should explicitly optimize.

Fig. 7.

The 5P evidence-grounded research agenda

Population Diversity

Our review demonstrates a strong over-reliance on narrow and homogeneous data sources. In particular, 40 out of 54 studies primarily depend on the English-only MIMIC-III dataset, with relatively few works using multilingual, multi-institutional, or geographically diverse cohorts (Table 5). This concentration limits the generalizability of current models across healthcare systems, documentation styles, and patient populations. Future ICD coding research should therefore explicitly broaden population coverage by incorporating multi-center, multilingual, and demographically diverse datasets. Concretely, studies should report the number of institutions, countries or regions, languages, and the presence or absence of demographic stratification in their evaluation.

Performance Robustness

Most current studies evaluate models only on in-distribution test splits drawn from the same dataset used for training. Our quality assessment shows that only two studies explicitly evaluated cross-dataset or cross-setting generalization (Q11 in Table 4), revealing a critical weakness in robustness assessment. Without external validation, reported performance may substantially overestimate real-world reliability. Robust ICD coding systems should therefore demonstrate stable performance across institutions, documentation styles, and data quality variations. At a minimum, future studies should include external test sets, cross-dataset validation, or stress tests using noisy or truncated clinical notes, and report performance stability rather than a single headline score.

Prediction of Rare Codes

Long-tail ICD codes represent clinically important but underrepresented conditions. However, most studies still prioritize frequent codes and micro-averaged metrics, which obscure performance on rare labels. Only a limited subset of works in this review employs hierarchical modeling, tail-aware loss functions, or head/body/tail evaluation splits. This creates a systematic risk of clinically incomplete coding systems. Future research should explicitly evaluate rare-code performance using macro-averaged metrics, tail-specific F1 scores, or stratified label frequency bands, and should increasingly adopt few-shot, zero-shot, and ontology-guided learning strategies to improve coverage of underrepresented conditions.

Provenance Transparency

Although most studies clearly report objectives, datasets, and evaluation metrics, reproducibility remains incomplete. Only 18 studies (Table 4) make their code publicly available (Q9), and only 45 provide sufficient methodological detail to enable full replication (Q10). Incomplete disclosure of preprocessing pipelines, data splits, and training protocols undermines scientific reliability and slows clinical translation. Future ICD coding studies should treat transparency as a first-class requirement by releasing code when possible, documenting full preprocessing and training procedures, and clearly specifying dataset partitions and hyperparameters.

Practical Integration

Nearly all reviewed studies remain at the prototype stage and evaluate performance only in offline experimental settings. Very few consider EHR integration, clinician verification workflows, inference latency, or regulatory and privacy constraints. For ICD coding systems to become clinically useful, they must be designed as decision support tools that support, rather than replace, professional coders and clinicians. Future work should therefore report at least basic integration considerations, such as how predictions would be presented to users, how human-in-the-loop verification would be implemented, and whether the system meets practical deployment constraints.

Implementation Perspective

The 5P agenda can be operationalized in future research through a simple progression: selecting multi-site or multilingual data during dataset construction (Population), including external or cross-dataset testing during evaluation (Performance), reporting explicit rare-code metrics (Prediction), releasing code and full preprocessing pipelines (Provenance), and addressing workflow integration and human-in-the-loop usage during system design (Practice).

Taken together, the 5P agenda reframes progress in automated ICD coding from narrow benchmark optimization toward clinically trustworthy, equitable, and deployable systems. By grounding these priorities in empirical gaps observed across the literature, it provides a concrete and actionable direction for the next phase of research in medical coding AI.

Implications for Research and Practice

Our findings reveal both the promise and limitations of automated ICD coding for improving healthcare efficiency and reducing administrative burden. While transformer-based architectures have demonstrated strong performance on frequent codes, significant challenges remain in translating these advances into clinical practice. Models must be computationally efficient for resource-constrained settings, robust across diverse healthcare environments, and capable of integrating into existing clinical workflows without disrupting established coding processes.

Moving forward, progress will require sustained collaboration across disciplines. Researchers need input from practicing clinicians to understand what makes a coding system truly useful. Administrators must work with both groups to navigate implementation challenges. Policymakers have a role in establishing standards and incentives that support adoption while ensuring quality and equity. Only through such partnerships can automated ICD coding fulfill its potential to support data-driven care delivery and move healthcare systems closer to universal health coverage goals.

Conclusion

This systematic review synthesizes 54 studies on automated ICD coding from clinical text published between 2019 and 2024, systematically analyzing dataset selection, preprocessing methods, feature representations, model architectures, and evaluation protocols. The field has evolved from traditional machine learning to deep learning approaches, with preprocessing techniques advancing from basic tokenization to sophisticated section segmentation and domain-specific filtering, while feature representations have shifted from hand-crafted methods to pretrained language models such as BERT and clinical variants like ClinicalBERT. The predominance of MIMIC-II and MIMIC-III datasets reflects both the value of publicly available corpora and the field’s overreliance on single-institution, English-language resources.

Comparative analysis of studies using standard MIMIC-III benchmarks reveals substantial architectural diversity across CNN, RNN, transformer, and hybrid models, with no single family consistently dominating across time periods and label spaces. While transformer-based models achieved peak performance on frequent code subsets (0.735 Micro F1), hybrid architectures incorporating medical knowledge graphs showed relatively consistent performance across both full and reduced label configurations, suggesting that auxiliary supervision from structured medical knowledge may offer advantages worth further systematic investigation. However, critical gaps persist: only 24% of studies performed external validation, exhibiting mean 13% performance degradation on external datasets; rare and emerging codes remain difficult to predict across all architectural families; black-box models undermine clinical trust and acceptance; and inconsistent evaluation protocols hinder meaningful cross-study comparison and reliable progress assessment.

To address these challenges, we propose a 5P evidence-grounded research agenda: Population Diversity (developing multilingual, multi-institutional datasets), Performance Robustness (ensuring rigorous external validation and generalization), Prediction of Rare Codes (advancing few-shot and zero-shot learning approaches), Provenance Transparency (enabling explainable architectures and clinical knowledge integration), and Practical Integration (establishing standardized evaluation metrics and real-world deployment pathways). By systematically addressing these five pillars, the field may move toward more robust and clinically relevant automated ICD coding systems. However, achieving this will require continued improvements in data availability, evaluation practices, and collaboration across different disciplines.

Appendix A: Formal Definitions of Evaluation Metrics

This appendix provides the formal mathematical definitions of evaluation metrics commonly used in automated ICD coding studies. These definitions are included for completeness and reference, while the main text focuses on conceptual interpretation and comparative usage across studies.

A.1 Classification-Based Metrics

Let TP, FP, TN, and FN denote the numbers of true positives, false positives, true negatives, and false negatives, respectively.

Accuracy

is defined as:

A1

Precision

measures the proportion of correct positive predictions:

A2

Recall

quantifies the proportion of true positives correctly identified:

A3

F1-score

is the harmonic mean of precision and recall:

A4

In multi-label settings, these metrics are commonly reported using micro- or macro-averaging across labels.

A.2 Ranking-Based Metrics

For a given instance, let k denote the number of top-ranked predicted labels.

Precision@k

is defined as:

A5

Recall@k

measures the fraction of true labels retrieved among the top-k predictions:

A6

These metrics are particularly relevant for clinical decision-support scenarios where only a limited number of code suggestions are reviewed.

A.3 Threshold-Free Metrics

Receiver Operating Characteristic Area Under the Curve (ROC-AUC)

evaluates the trade-off between the true positive rate (TPR) and false positive rate (FPR):

A7

Precision–Recall Area Under the Curve (PR-AUC)

summarizes performance across recall values:

A8

These metrics are commonly used in highly imbalanced classification settings.

A.4 Multi-Label Specific Metrics

Hamming Loss

measures the fraction of incorrectly predicted labels:

A9

where N is the number of instances, L is the number of labels, is the ground truth label, and is the predicted label.

Lower Hamming Loss values indicate better multi-label prediction performance.

Appendix B: Search Strategy and Reproducibility Audit Trail

This appendix documents the complete and reproducible search strategy used in this systematic review, in accordance with PRISMA 2020 recommendations and audit ready reporting standards.

The search strategy was developed iteratively. Multiple keyword combinations and term variants were tested to balance sensitivity and specificity. The final database specific search strings, exactly as executed, are reported in Supplementary Table 14 to ensure transparency and reproducibility.

Table 14.

Database-specific search queries and execution details

Database (Search date)	Query ID	Full search string (Executed as shown)
PubMed (Dec 18, 2024)	A	(“ICD coding”[Title/Abstract] OR “ICD code prediction”[Title/Abstract] OR “ICD classification”[Title/Abstract] OR “ICD code assignment”[Title/Abstract] OR “automatic coding”[Title/Abstract] OR “ICD9 coding”[Title/Abstract] OR “ICD10 coding”[Title/Abstract]) AND (“machine learning”[Title/Abstract] OR “deep learning”[Title/Abstract] OR “artificial intelligence”[Title/Abstract] OR “natural language processing”[Title/Abstract] OR NLP[Title/Abstract] OR BERT[Title/Abstract] OR transformer[Title/Abstract] OR transformers[Title/Abstract]) AND (“clinical text”[Title/Abstract] OR “clinical notes”[Title/Abstract] OR “discharge summaries”[Title/Abstract] OR “electronic health records”[Title/Abstract] OR “electronic medical records”[Title/Abstract] OR EHR[Title/Abstract] OR “clinical documentation”[Title/Abstract] OR “medical notes”[Title/Abstract])
PubMed (Dec 18, 2024)	B	(“ICD coding”[Title/Abstract] OR “ICD code prediction”[Title/Abstract] OR “ICD classification”[Title/Abstract]) AND (“machine learning”[Title/Abstract] OR “deep learning”[Title/Abstract] OR “artificial intelligence”[Title/Abstract] OR “natural language processing”[Title/Abstract] OR NLP[Title/Abstract] OR BERT[Title/Abstract] OR transformer[Title/Abstract] OR transformers[Title/Abstract]) AND (accuracy[Title/Abstract] OR precision[Title/Abstract] OR recall[Title/Abstract] OR “F1 score”[Title/Abstract] OR performance[Title/Abstract] OR evaluation[Title/Abstract])
IEEE (Dec 30, 2024)	A	(“ICD coding” OR “ICD code prediction” OR “ICD classification” OR “ICD code assignment” OR “automatic coding”) AND (“machine learning” OR “deep learning” OR “artificial intelligence” OR “natural language processing” OR NLP OR BERT OR transformer OR transformers) AND (“clinical text” OR “clinical notes” OR “discharge summaries” OR “electronic health records” OR “electronic medical records” OR EHR OR “clinical documentation” OR “medical notes”)
IEEE (Dec 30, 2024)	B	(“ICD coding” OR “ICD code prediction” OR “ICD classification”) AND (“machine learning” OR “deep learning” OR “artificial intelligence” OR “natural language processing” OR NLP OR BERT OR transformer OR transformers) AND (accuracy OR precision OR recall OR “F1 score” OR performance OR evaluation)
ScienceDirect (Dec 25, 2024)	A	(“ICD coding” OR “ICD code prediction” OR “ICD classification” OR “ICD code assignment” OR “automatic coding” OR “ICD9 coding” OR “ICD10 coding”) AND (“machine learning” OR “deep learning” OR “artificial intelligence” OR “natural language processing” OR NLP OR BERT OR transformer OR transformers) AND (“clinical text” OR “clinical notes” OR “discharge summaries” OR “electronic health records” OR “electronic medical records” OR “clinical documentation” OR “medical notes”)
ScienceDirect (Dec 25, 2024)	B	(“ICD coding” OR “ICD code prediction” OR “ICD classification”) AND (“machine learning” OR “deep learning” OR “artificial intelligence” OR “natural language processing” OR NLP OR BERT OR transformer OR transformers) AND (accuracy OR precision OR recall OR “F1 score” OR performance OR evaluation)
ACM Digital Library (Dec 8, 2024)	A	(“ICD coding” OR “ICD code prediction” OR “ICD classification” OR “ICD code assignment” OR “automatic coding”) AND (“machine learning” OR “deep learning” OR “artificial intelligence” OR “natural language processing” OR NLP OR BERT OR transformer OR transformers) AND (“clinical text” OR “clinical notes” OR “electronic health records” OR “electronic medical records” OR “clinical documentation” OR “medical notes”)
ACM Digital Library (Dec 8, 2024)	B	(“ICD coding” OR “ICD code prediction” OR “ICD classification”) AND (“machine learning” OR “deep learning” OR “artificial intelligence” OR “natural language processing” OR NLP OR BERT OR transformer OR transformers) AND (accuracy OR precision OR recall OR “F1 score” OR performance OR evaluation)
Google Scholar (Publish or Perish) (Jan 10, 2025)	–	“ICD coding” OR “ICD code prediction” OR “ICD classification” OR “automatic coding” “machine learning” OR “deep learning” OR “natural language processing” OR BERT OR transformer “clinical notes” OR “clinical text” OR “electronic health records” accuracy OR precision OR recall OR “F1 score” OR performance
Mendeley Search (Jan 5, 2025)	–	“ICD coding” OR “ICD code prediction” OR “ICD classification” OR “automatic coding” “machine learning” OR “deep learning” OR “natural language processing” OR BERT OR transformer “clinical notes” OR “clinical text” OR “electronic health records” accuracy OR precision OR recall OR “F1 score” OR performance

B.1 Filters and Eligibility Constraints

Across all databases, the following filters were applied where supported:

• Publication years: 2019–2024

• Language: English

• Document types: journal articles and conference papers

Appendix C: Studies Excluded Due to Access Restrictions

During the systematic literature search, 38 studies could not be accessed. These studies were excluded due to institutional access limitations, with full texts unavailable through our university library, interlibrary loan services, or open access repositories.

Table 15 lists all 38 studies excluded due to access restrictions, including their titles, publication years, and venues.

Appendix D: Supplementary Results for Mixed and Non-MIMIC Datasets

A subset of the included studies reported results on heterogeneous evaluation settings that could not be integrated into the normalized comparison. These studies differed in datasets, ICD versions, label spaces, languages, and evaluation protocols, which prevented direct quantitative synthesis.

Table 18 summarizes studies that used both MIMIC and external datasets within the same work. Because the reported metrics were generated under non-identical experimental conditions, these results are presented descriptively rather than as head to head comparisons.

Table 19 lists studies conducted entirely on non-MIMIC datasets. These works involve diverse clinical sources and task formulations, and therefore their performance outcomes are documented as originally reported by the authors.

Author Contributions

Abdul Rehman Khalid conceptualized the study, performed data extraction, developed the methodology and software, and drafted and formatted the manuscript. Haider Ali contributed to writing, review, editing, manuscript formatting, and validation. Kounen Fathima participated in manuscript review and validation. Kouayep Sonia Carole provided supervision, review, and validation. Hee-Cheol Kim supervised the study and contributed to investigation, review, and validation. All authors reviewed and approved the final manuscript.

Funding

This work was supported by the Institute of Information & Communications Technology Planning & Evaluation (IITP)-Innovative Human Resource Development for Local Intellectualization program grant funded by the Korea government (MSIT) (IITP-2026-RS-2024-00436773).

Data Availability

No datasets were generated or analysed during the current study.

Declarations

Ethics approval and consent to participate

Not applicable.

Consent for publication

Not applicable.

Clinical Trial Number

Not applicable.

Competing interests

The authors declare no competing interests.