Gold-standard-guided direct preference optimisation for ICD-10-AM and ACHI code assignment from discharge summaries

Abstract

Clinical codes assigned from discharge summaries support clinical research, reimbursement, health resource allocation, and service planning. In Australia, ICD-10-AM and ACHI are used to code diagnoses and interventions for admitted episodes of care, but manual coding is time-consuming, labour-intensive, and subject to variability. This study investigates large language models for automated ICD-10-AM and ACHI code assignment from discharge summaries. We first normalised a public ICD-9-CM coded dataset into the Australian coding system using General Equivalence Mappings and semantic similarity over code descriptions, with validation by a domain expert clinical coder. We then fine-tuned Meta-Llama-3-8B and compared full-parameter fine-tuning with a parameter-efficient QLoRA baseline. Finally, we applied Direct Preference Optimisation (DPO) using gold-standard-guided preference pairs, where the gold-standard ICD-10-AM/ACHI code list was treated as the preferred response and the supervised fine-tuned model output as the non-preferred response. Experiments were conducted on a mapped MIMIC-III subset and a de-identified Australian hospital dataset. Among the evaluated model variants, the DPO-aligned full-parameter model achieved the highest score, with micro-F1 scores of 0.7397 on the mapped dataset and 0.7834 on the Australian hospital dataset. These findings suggest that gold-standard-guided DPO can improve agreement with reference code sets compared with supervised fine-tuning alone. However, the results are limited to high-frequency diagnosis and procedure codes, short-document evaluation settings, and DPO preference pairs reviewed by one annotator.

Supplementary Information

The online version contains supplementary material available at https://doi.org/10.1186/s12911-026-03631-w.

Introduction

Accurate clinical coding is foundational to hospital reimbursement and activity-based funding, epidemiological surveillance, quality reporting, and clinical research. In practice, trained clinical coders review free-text clinical documentation - often the discharge summary written at the end of a hospital stay - and assign diagnosis and procedure codes. This process is labour-intensive and time-consuming, and it can be vulnerable to variability and error, which can propagate downstream to funding and reporting outcomes [1].

In Australia, inpatient episodes are coded using the Australian Health Classification System: the International Statistical Classification of Diseases and Related Health Problems, Australian Modification (ICD-10-AM) for diagnoses and health conditions; the Australian Classification of Health Interventions (ACHI) for procedures and interventions; and the Australian Coding Standards (ACS) for national coding rules. Although ICD is an international standard, country specific adaptations differ in code granularity, code availability, and coding rules. These differences can be clinically meaningful and can limit the direct transfer of automated coding models developed for other jurisdictions [2]. For example, in the U.S. classification system, ‘contact with venomous spider’ is coded as X21, whereas in Australia, there is more specificity added in the level of identifying the type of spider including X21.0 (Contact with funnel-web spider), X21.1 (Contact with red back spider), and X21.2 (Contact with white-tailed and other necrotising spider). The ICD-10-AM and ACHI code structure is given in Figure 1 and Figure 2 respectively.

Fig. 1.

Hierarchical structure of ICD-10-AM code

Fig. 2.

Hierarchical structure of ACHI code

Automating code assignment from discharge summaries is challenging for several reasons. Discharge summaries are long, heterogeneous, and contain abbreviations, and institution-specific phrasing. The clinical reasoning behind a diagnosis is sometimes implied rather than clearly stated, making it harder for a system to interpret. The target label space is large and highly imbalanced, and each record typically requires multiple diagnosis and procedure codes. Models must also respect hierarchical code structures and adhere to coding standards, while remaining robust to documentation variation across hospitals and time.

Given that manual assignment of ICD code relies on manual inspections and experience-based judgement, studies utilised Artificial Intelligence (AI) techniques to automate the ICD coding process. This includes both conventional machine learning [2–6] and deep learning approaches [7–12]. Most prior work on automated clinical coding has concentrated on United States coding systems(e.g., ICD-9-CM, ICD-10-CM/PCS), largely because large public datasets are available such as MIMIC-III [13]. In contrast, reproducible benchmarks and comparative evidence for Australian ICD-10-AM/ACHI coding remain limited, slowing progress and making it difficult to evaluate newer methods fairly in the Australian context. A second, related barrier is reliability: even when modern neural models achieve strong aggregate metrics, they can under-perform on rare codes and may produce clinically implausible or non-existent codes - behaviour that is especially problematic for billing and reporting workflows.

Large language models (LLMs) offer a promising direction because they can represent long-range context and can be adapted to generate structured outputs directly from free text. However, using these models straight out of the box has its drawbacks. They may not align with local coding expectations and may generate unsupported, invalid, or clinically implausible outputs. One way to address this is to align model outputs more closely with the gold-standard coding references after supervised fine-tuning. Direct Preference Optimisation (DPO) provides a way to optimise a model using preferred and non-preferred responses without training a separate reward model or using an explicit reinforcement-learning loop. In this study, we use DPO as a gold-standard-guided alignment step after supervised fine-tuning.

In this work, we investigate an LLM-based approach for assigning ICD-10-AM and ACHI codes from discharge summaries, focusing on both performance and practical training efficiency. First, to enable controlled experiments using public data, we propose an NLP-assisted mapping pipeline that maps codes from US-coded datasets (via official crosswalks where available) to their closest ICD-10-AM and ACHI counterparts using semantic similarity between code descriptions and expert review. Second, we fine-tune the Llama 3 model for clinical code generation and compare full fine-tuning against parameter-efficient adaptation (LoRA/QLoRA), quantifying the trade-offs between compute requirements and coding accuracy. Third, we apply DPO using gold-standard-guided preference pairs derived from reference code sets and supervised fine-tuned model outputs, aiming to improve agreement with gold-standard ICD-10-AM/ACHI assignments. For tractability and to reflect high-volume coding scenarios, our experiments focus on the top 50 ICD-10-AM diagnosis codes and top 50 ACHI procedure codes (100 codes total). We evaluate on a mapped subset of MIMIC-III discharge summaries and a de-identified Australian hospital dataset, reporting standard multi-label metrics (micro/macro precision, recall, and F1).

Our contribution:

This paper makes the following contributions:

1. Australian normalisation of public ICD-coded data. We propose a multi-step mapping pipeline to normalise MIMIC-III ICD-9-CM diagnosis and procedure labels into the Australian Health Classification System (ICD-10-AM for diagnoses and ACHI for procedures) via GEMs and semantic similarity over code descriptions, with validation by a domain expert clinical coder. This enables public-data benchmarking in an Australian coding setting.

2. Full fine-tuning vs QLoRA (bitsandbytes 4-bit) for ICD-10-AM/ACHI assignment. We fine-tune meta-llama/Meta-Llama-3-8B for ICD-10-AM/ACHI code assignment from discharge summaries and provide a systematic comparison between full-parameter fine-tuning (non-quantised) and a parameter-efficient QLoRA baseline (bitsandbytes 4-bit quantised backbone), highlighting the compute–performance trade-off on both the mapped dataset and a real-world Australian hospital dataset.

3. Gold-standard-guided DPO alignment. We curate DPO training pairs from gold-standard code lists and supervised fine-tuned model outputs using a Streamlit-based review workflow. The resulting preference data are used to align the full-parameter fine-tuned model and improve agreement with reference ICD-10-AM/ACHI code sets.

The remainder of the paper is organised as follows: Sect. Related Work reviews related work, Sect. Methodology describes the proposed methods, Sect. Experimental Setup details the datasets and experimental setup, Sect. Results and discussion presents the results and discussion, Sect. Limitations discusses limitations, Sect. Future Research Directions outlines future research directions, and Sect. 8 concludes the paper.

Related work

Automated ICD coding has been a prominent area of research in the healthcare domain for the last two decades. Many conventional machine learning [2–6] and deep learning approaches [7–12] have been explored to automate the assignment of ICD codes. Huang et al. [14] compared several methods, including recurrent neural networks (RNNs) and convolutional neural networks (CNNs), and reported that deep learning-based methods achieved higher performance than conventional machine-learning methods for ICD code assignment. Nevertheless, RNNs were found to have several shortcomings when it comes to capture long-range dependencies due to issues like vanishing gradients. CNNs, on the other hand are effective in capturing local features through convolutional filters but lack the ability to model sequential or temporal dependencies effectively. To overcome these limitations, Long Short-Term Memory networks (LSTMs) [14] were introduced, a variant of RNNs designed with gating mechanisms that allow them to retain information over longer sequences. Models like Convolutional Attention for Multi-Label (CAML) [8] utilised label-wise attention mechanisms to better capture the relationships between clinical text and ICD codes. Subsequently, attention-based models such DR-CAML [7] and HyperCore [15] further leveraged label dependencies to enhance coding accuracy. Despite these advancements, challenges persisted, particularly in handling the vast number of ICD codes, the hierarchical structure of ICD codes, unstructured nature of clinical documents, and struggled with rare or ambiguous codes.

The introduction of Transformers including Bidirectional Encoder Representations from Transformers (BERT) [16], and Generative Pretrained Transformer (GPT) [17], have largely replaced LSTMs by leveraging self-attention mechanisms that allow them to model global dependencies and enhance parallelisation. Though, transformers-based models have proven invaluable for many biomedical applications, but these models required significant memory for training, due to which the usage of these models become challenging. Large Language Models(LLMs) posses a great potential to analyse vast amount of unstructured data captured in healthcare, such as patient narratives, electronic health records (EHRs), clinical trial results, and research publications. The extensive datasets used to train these LLMs has enabled them for various transferable language tasks. However, the medical terminologies, domain-specific jargon, and unique reporting practices in healthcare contrast sharply with the general-purpose language used to train these models. To address these issues and improve the usability of pre-trained language models in the biomedical field, researchers have focused on domain-specific adaptations during the pre-training phase and have implemented fine-tuning strategies on the models. During pre-training, efforts are made to use the existing models in the biomedical domain by extending the pre-training process or stating a new with vast amount of raw biomedical data, making these models more adaptable through fine-tuning. Efficient models such as BioBERT [18], ClinicalBERT [19], PubMedBERT [20] have been specifically designed for the biomedical domain and are ideal for downstream task during the fine-tuning stage. ClinicalBERT, pre-trained on clinical notes, and SciBERT [21], pre-trained on scientific texts, are also tailored for medical text summarisation. BioBERT has been reported to improve over the original BERT model on biomedical text-mining tasks such as relation extraction, named entity recognition, and question answering.

Large language models (LLMs) such as GPT-3, GPT-4, and LLaMA have opened up new possibilities for automating ICD coding. These models, with their vast parameter sizes and pretrained on extensive corpora, exhibits remarkable capabilities in understanding and generating human-like text. Nawab et al. [22] fine-tuned GPT-3.5 Turbo on discharge summaries from MIMIC-III, improving top-10 ICD code prediction accuracy by over 30% compared to base models. However, showed reduced performance when tested on external datasets due to differences in documentation styles, coding guidelines and clinical terminology. To address these challenges, there is a need to incorporate human-in-the-loop strategies and reinforcement learning from human feedback (RLHF) to refine model outputs or guide training in real-world preferences.

These prior models provide important context for automated clinical coding; however, they were not re-implemented or evaluated on the same ICD-10-AM/ACHI label space and data splits in this study. Therefore, they are discussed as related work rather than direct experimental baselines.

Methodology

We formulate automated clinical coding as an instruction-following generation task: given a discharge summary, the model generates the corresponding set of ICD-10-AM diagnosis codes and ACHI procedure codes. We use meta-llama/Meta-Llama-3-8B [23] as the base model. Our pipeline proceeds in three stages. First, we perform supervised instruction fine-tuning (SFT) to adapt the base model to the coding task. Second, we benchmark a compute-efficient alternative using parameter-efficient fine-tuning (PEFT) with a 4-bit quantised backbone. Third, we apply gold-standard-guided Direct Preference Optimisation (DPO) to the full-parameter fine-tuned model to improve agreement with reference code sets.

Fine-tune large language model

Fine-tuning is an effective strategy that allows large language models to leverage their pre-trained knowledge and adapt to new tasks, enhancing their performance and generalisation abilities. Through fine-tuning, language models can apply their prior understanding to new domains and tasks, increasing their usefulness and effectiveness. This process involves taking an existing pre-trained model, such as BERT or GPT and training it further on domain-specific knowledge to improve its ability to generate accurate and contextually relevant responses. By leveraging the existing knowledge encoded in pre-trained models, fine-tuning saves considerable time and computational resources compared to training models from scratch.

There are several fine-tuning methods, notably domain adaptation, transfer learning, and task-specific fine-tuning. Domain adaptation involves training models on data what is similar yet distinct, allowing the model to adapt related contexts effectively. Transfer learning leverages a pre-trained model trained on related tasks and further trains it on a new but related task, making it particularly beneficial when training data is limited. On the other hand, task-specific fine-tuning involves training a model extensively from scratch on new data related to a particular task, making it ideal when the new task is very different from the original or requires significant customisation.

When fine-tuning is done properly, it can dramatically improve a model’s performance, making it effective for tasks like translation, summarisation, question-answering, and text generation. However, improper fine-tuning can lead to problems like overfitting, where models perform well on training data but poorly on unseen data; underfitting, which results from insufficient training, reduced generalisation ability, lack of transferability to other tasks, and increased computational complexity. To resolve these challenges, researchers applied techniques like regularisation, early stopping, dropout, cross-validation, and batch normalisation to prevent overfitting or underfitting. Fine-tuning very large models is often extremely computationally demanding and expansive. For example, fine-tuning a large -scale language model such as LLaMA with 65 billion parameters demands more than 780 GB of GPU memory [24].

Task formulation and supervised fine-tuning

Let x denote a discharge summary (free text describing the admission episode) and let denote the target multi-label code set, where is the set of ICD-10-AM diagnosis codes and is the set of ACHI procedure codes. We model code assignment as instruction-following generation: the input prompt consists of an instruction and the discharge summary, and the model outputs a structured response listing the predicted codes.

Training instances are represented as JSON records with fields . The instruction specifies the task (e.g., “Assign appropriate ICD-10-AM diagnosis and ACHI procedure codes for the given discharge summary.”), the input contains the discharge summary text, and the response contains the gold-standard code list formatted in a consistent template. Across experiments, we use a fixed output structure that separates diagnosis and procedure codes to support robust downstream parsing and evaluation.

Supervised fine-tuning adapts the base model to the clinical coding task by maximising the conditional likelihood of the reference response y given the prompt x. With an autoregressive policy , the SFT objective minimises the negative log-likelihood:

1

At inference time, the model is prompted with the same instruction template and generates a response that is post-processed to extract ICD-10-AM and ACHI codes. Post-processing includes normalising formatting, removing duplicates, and retaining only codes within the evaluation scope. The extracted code sets are then used for evaluation.

Parameter-efficient fine-tuning (LoRa/QLoRA baseline)

Full-parameter fine-tuning of large language models can be computationally expensive. To evaluate a compute-efficient alternative, we compare full fine-tuning against a parameter-efficient baseline based on Low-Rank Adaptation (LoRA) [25]. In LoRA, a frozen weight matrix W is augmented by a low-rank update Δ W = BA, where and with rank . Only the adapter parameters are trained, substantially reducing the number of trainable parameters.

We implement a quantised PEFT [26] variant using a bitsandbytes 4-bit quantised backbone (QLoRA-style), in which the base model weights are stored in 4-bit precision for memory efficiency while training LoRA adapters. This model is used as the “quantised” baseline in our full fine-tuning vs. quantised comparison.

Scope of DPO alignment

Although preference alignment can be combined with PEFT/QLoRA in principle, in this work we apply DPO only to the full-parameter, non-quantised fine-tuned model. The QLoRA model is used as a parameter-efficient baseline to compare resource-efficient adaptation with full-parameter fine-tuning. We then evaluate gold-standard-guided DPO as an additional alignment step on the full-parameter model. This design avoids introducing a separate quantised-DPO training condition with different adapter and hyperparameter choices. Therefore, the quantised PEFT baseline is reported as a separate comparison and is not DPO-aligned.

Direct Preference Optimisation (DPO)

Supervised fine-tuning can achieve strong average performance but may still produce extra codes, omit relevant codes, or generate malformed outputs. To better align model behaviour with reference coding, we apply Direct Preference Optimisation (DPO) [27], which learns from pairwise preferences relative to a reference policy (the full-parameter SFT model) without training a separate reward model.

Construction of gold-standard-guided preference pairs

We construct a gold-standard-guided preference dataset for DPO. For each discharge summary x, we form a preference pair where the preferred response is the gold-standard ICD-10-AM/ACHI code assignment (formatted in the same response template used at inference time) and the non-preferred response is the model-generated prediction for the same input. This provides a preference signal anchored to reference labels rather than independent human rankings of alternative model outputs.

We implemented a lightweight annotation interface using Streamlit to support review and curation. For each record, the interface displayed the model-generated codes and indicated whether each predicted code was present in the gold-standard set (e.g., “present” vs. “not present” in ground truth). The interface also surfaced missing gold-standard codes to highlight underprediction and enabled the annotator to record correctness judgements and optionally add codes. All annotations and metadata were stored in a SQLite database (record identifier, dataset source, query/prompt, predicted output, gold-standard output, and timestamp) to ensure traceability.

To focus the preference signal on informative failures, we retained records where the model output exhibited (i) extra-code errors, where predicted codes were not present in the gold standard, (ii) completely incorrect predictions (no overlap with the gold standard), or (iii) partially correct predictions (mixed correct and incorrect codes). We excluded instances where the model prediction exactly matched the gold-standard code set because these examples provide no informative preference signal for DPO. Using this procedure, we curated approximately 850 preference pairs from the mapped ICD dataset and 1,150 preference pairs from the hospital dataset (approximately 2,000 pairs total). Although the preference set is modest in size, it is intentionally enriched for difficult cases, including extra-code errors, partially correct predictions, and incorrect predictions, providing high-signal alignment examples. Preference pairs were reviewed and curated by a single annotator, who was also an author. Therefore, the resulting preference dataset should be interpreted as single annotator, gold standard guided curation of gold-versus-prediction pairs, not as expert RLHF or human preference learning based on independent ranking of candidate outputs.

Preference JSON format

We export the preference dataset as JSON records with fields query, chosen, and rejected, where chosen corresponds to the gold-standard formatted response and rejected corresponds to the model prediction. Listing 1 shows an example.

Listing 1

Example preference record (gold vs. prediction) used for DPO. Discharge summary text is omitted/redacted.

DPO objective

dpo2023 introduced Direct Preference Optimization (DPO) as an alternative approach to traditional RLHF, aims to address the computational challenges in fine-tuning large scale language models. RLHF [28] includes three primary phases: supervised fine-tuning (SFT), human preference-based reward modelling, and reinforcement learning optimisation. In RLHF, the reward function is trained using a human preference dataset,

2

where and represent preferred and non-preferred responses selected by annotators for input x. The reward model is trained by minimising the negative log-likelihood loss [29]:

3

penalises predictions that assign low probabilities to actual human preferences.

The reward model is then used to fine-tune the policy using reinforcement learning (RL) to maximise the reward. The optimisation objective is given in equation 4.

4

where, encourages the policy to generate responses with higher rewards, is the Kullback-Leibler (KL) divergence, and β is a weighting factor.

However, in DPO instead of training a reward model, DPO leverages a particular choice of reward model parameterisation that enables extraction of its optimal policy , without an reinforcement learning loop. To train the parameterised policy ,maximum likelihood estimation (MLE) based on pairwise preference data is used. The Direct Preference Optimization (DPO) loss is given in equation 5 as defined in [27]:

5

In this study, D is constructed from gold-standard-guided preference pairs rather than independent human preference rankings; therefore, denotes the formatted gold-standard ICD-10-AM/ACHI response, and denotes the supervised fine-tuned model output for the same discharge summary. DPO reframes the learning task using pairwise preferences and re-weights the reference policy by removing the need for complex RL algorithms.

Experimental setup

The experimental setup involves two distinct datasets to evaluate model performance. Below, we describe the datasets, code mapping methodology, and model training details.

1. MIMIC-III (Mapped to ICD-10-AM and ACHI): The discharge summaries acquired from the MIMIC-III [13], contains deidentified health-related data associated with over 40,000 patients who stayed in the ICU of the Beth Israel Deaconess Medical Center between 2001 and 2012. Each admission was tagged manually by coders using ICD-9 codes describing diagnoses and procedures carried out during the patient stay. This dataset contains 52,726 discharge summaries and 8,929 unique codes (6,918 diagnoses, 2,011 procedures). Since ICD-9-CM is not directly compatible with ICD-10-AM/ACHI, we performed a mapping process to align the codes to the Australian classification system. Therefore, we mapped top 50 most frequent ICD-9-CM diagnosis codes to ICD-10-AM codes and top 50 most frequent ICD-9-CM procedure codes to ACHI codes.

The mapped dataset contains 50 diagnosis labels but only 45 unique procedure labels (95 total labels) because ICD-9-CM procedure codes do not map one-to-one to ACHI. In several cases, multiple ICD-9-CM procedure codes collapsed to the same ACHI label. The mapping process and examples of procedure-code collisions are described in Section 4.1.

In this study, we used a subset of MIMIC-III database, specifically targeting discharge summaries that contained at least one of the mapped 95 diagnosis and procedure codes. The subset contains a total of 49,414 discharge summaries out of which 42,002 discharge summaries were used for training, 2,471 for validation and 4,941 for testing. In order to assess the performance of our model across varying length of discharge summaries, we further stratified the test data based on the word count. Table 1 shows the number of discharge summaries within test data according to the word count range, highlighting variations in document length. This stratification helps to evaluate model performance across varying document complexities. The frequency distributions of the top 50 ICD-10-AM diagnosis codes and the top 45 ACHI procedure codes in the mapped dataset are provided in Supplementary file 1 (Figures S1 and S2), respectively.

• 2.Hospital Discharge Summaries: The second dataset consists of hospital discharge summaries acquired from South Western Sydney Local Health District (SWSLHD) (HREC Reference: LNR/17/LPOOL/432). The source dataset was a complete extraction of available de-identified hospital discharge summaries for admitted episodes recorded between 31 October 2021 and 1 November 2023. It was not a random sample. No additional selection criteria were applied at source extraction, and records were not excluded on the basis of patient age, length of stay, admission type, clinical condition, or speciality. The extracted records contained routinely assigned ICD-10-AM diagnosis codes and ACHI procedure codes.

Table 1.

MIMIC-III mapped test data distribution by word count

Word Count Range	Number of Discharge Summaries
Up to 1,000	1,254
More than 1,000 but less than equal to 2,000	2475
More than 2,000 but less than equal to 3,000	964
More than 3,000	248

The discharge summaries were anonymised using Amazon Comprehend Medical, applying masking and de-identification techniques to protect patient information. The extracted dataset contained 113,785 discharge summaries and 14,018 unique codes (11,451 diagnoses and 2,567 procedures). For the modelling experiments, the target label vocabulary was restricted to the top 50 ICD-10-AM diagnosis codes and the top 50 ACHI procedure codes. Records were included in the modelling dataset if they contained at least one selected diagnosis or procedure code, resulting in 104,440 discharge summaries as shown in Table 2. Of these, 88,774 records were used for training, 5,222 records for validation, and 10,444 records for testing. The split was performed at the discharge-summary record level. Each discharge summary record was treated as an independent unit for splitting; no patient-level grouping was applied. Similar to the mapped MIMIC-III data, we further stratified the test data based on word count.

Table 2.

SWSLHD data distribution containing top 50 ICD-10-AM and/or top 50 ACHI codes

Data size	Record Count
Records with both top 50 ICD-10-AM codes and top 50 ACHI Code	79,683
Records with only top 50 ICD-10-AM codes	23,644
Records with only top 50 ACHI Code	1,113
Records with at least one of the top 50 ICD-10-AM or ACHI Code (total)	104,440

Table 3 shows the test data distribution according to word count. The frequency distributions of the top 50 ICD-10-AM diagnosis codes and the top 50 ACHI procedure codes for the SWSLHD dataset are provided in Supplementary file 1 (Figures S3 and S4), respectively.

Table 3.

SWSLHD test data distribution by word count

Word Count Range	Number of Discharge Summaries
Up to 1,000	5,001
More than 1,000 but less than equal to 2,000	4,156
More than 2,000 but less than equal to 3,000	1106
More than 3,000	181

To further clarify the effect of restricting the target vocabulary to high-frequency codes, we performed a label-scope coverage analysis on the SWSLHD experimental dataset. This analysis counts code instances rather than unique code labels. A code instance refers to one assigned code in one discharge summary. For example, if the same ICD-10-AM code appears in 100 discharge summaries, it contributes 100 code instances. This measure reflects how much of the original coding activity is covered by the selected top-50 diagnosis and top-50 procedure vocabulary.

As shown in Table 4, the selected high-frequency label set covered a larger proportion of procedure-code assignments than diagnosis-code assignments. The top-50 ACHI procedure codes accounted for 77.06% of procedure code instances, while the top-50 ICD-10-AM diagnosis codes accounted for 41.64% of diagnosis code instances. Codes outside the selected vocabulary were not used as prediction targets or evaluation labels.

Table 4.

Label-scope coverage in the SWSLHD experimental dataset

Measure	Value
Diagnosis code instances within selected vocabulary	253,245 of 608,191 (41.64%)
Diagnosis code instances outside selected vocabulary	354,946 of 608,191 (58.36%)
Procedure code instances within selected vocabulary	177,391 of 230,190 (77.06%)
Procedure code instances outside selected vocabulary	52,799 of 230,190 (22.94%)

Note: The selected vocabulary includes the top 50 ICD-10-AM diagnosis codes and top 50 ACHI procedure codes. A code instance refers to one assigned code in one discharge summary

Among diagnosis codes outside the selected vocabulary, frequent examples included codes related to infectious disease, respiratory conditions, symptom-related presentations, electrolyte or metabolic abnormalities, sepsis, kidney disease, and skin infection. Examples included A09.9, B97.2, R11, E87.1, A41.9, J45.9, J12.8, N18.5, and L03.13. Therefore, admissions dominated by codes outside the selected vocabulary may be under-represented in the evaluation, while admissions containing both selected and non-selected codes may be simplified because only selected high-frequency labels are retained. Accordingly, the reported metrics should be interpreted as performance within a common-code evaluation setting rather than full-vocabulary clinical coding.

Normalise clinical datasets pre-coded in different extensions of ICD classification system

Mapping ICD-9-CM diagnosis and procedure codes from MIMIC-III to the Australian Health Classification System (ICD-10-AM for diagnoses and ACHI for procedures) requires a multi-step approach because there is no official one-to-one crosswalk from ICD-9-CM to ICD-10-AM/ACHI. We therefore perform forward mapping via the U.S. ICD-10 extensions and then align to the Australian target vocabularies (Figure 3).

Fig. 3.

Mapping pipeline from ICD-9-CM to ICD-10-AM/ACHI via ICD-10-CM and ICD-10-PCS

In this study, we used the 2018 CMS/CDC General Equivalence Mappings (GEMs) for the ICD-9-CM to ICD-10-CM/PCS mapping step. For the final Australian target vocabulary, we used ICD-10-AM/ACHI/ACS Eleventh Edition code descriptions. Historical IHPA releases from the 1st to 12th editions were checked only to identify code availability, retired codes, and terminology changes; the final mapped target vocabulary was based on the Eleventh Edition.

Step 1: ICD-9-CM → ICD-10-CM/PCS using GEMs

We used the 2018 CMS/CDC General Equivalence Mappings (GEMs) to map ICD-9-CM diagnosis codes to ICD-10-CM and ICD-9-CM procedure codes to ICD-10-PCS. These intermediate codes are accompanied by code titles/descriptions that can be used for subsequent semantic alignment.

Step 2: ICD-10-CM/PCS → ICD-10-AM/ACHI using semantic similarity

Because no direct mapping exists between ICD-10-CM/PCS and ICD-10-AM/ACHI, we select the closest Australian codes using natural language processing (NLP). We embed the textual code descriptions from ICD-10-CM/PCS and ICD-10-AM/ACHI/ACS Eleventh Edition using a sentence-transformer model and compute cosine similarity between embeddings. For each source code, we retrieve the top-k=3 most similar target codes as candidates.

Domain-expert review and validation

The automatically retrieved top-k candidates for each code were reviewed by a domain expert clinical coder (with experience in Australian clinical classification systems). The reviewer selected the most appropriate ICD-10-AM/ACHI code for each source code based on the code definitions and clinical meaning. Ambiguous cases were resolved through discussion with the research team, and the final validated mapping was stored as a structured mapping dictionary used to convert the MIMIC-III labels into the Australian target label space for subsequent experiments.

Mapping audit

To quantify the clinical coding expert review, we audited the mappings that contributed to the final mapped MIMIC-III label set used in the experiments. This included 50 ICD-9-CM diagnosis mappings and 50 ICD-9-CM procedure mappings. The diagnosis mappings produced 50 ICD-10-AM labels, while the procedure mappings produced 45 unique ACHI labels after mapping collisions. Each mapping was classified as either accepted/retained after review or revised/clarified after review. For diagnosis mappings, 39 of 50 mappings were accepted or retained, while 11 were revised or clarified. For procedure mappings, 23 of 50 mappings were accepted or retained, while 27 were revised or manually corrected. The expert-review outcomes are summarised in Table 5.

Table 5.

Clinical coding expert review outcomes for final mapped source-code mappings

Mapping type	Reviewed mappings	Accepted/retained	Revised/clarified
ICD-9-CM to ICD-10-AM	50	39 (78.0%)	11 (22.0%)
ICD-9-CM to ACHI	50	23 (46.0%)	27 (54.0%)

Note: The audit includes only mappings contributing to the final mapped label set used in the experimentsThe 50 diagnosis source mappings produced 50 ICD-10-AM labels, while the 50 procedure source mappingsproduced 45 ACHI labels after mapping collisions. These values summarise expert-review outcomes andshould not be interpreted as independent mapping accuracy estimates

Procedure mappings required more revision than diagnosis mappings because ICD-9-CM procedure codes and ACHI differ substantially in structure, granularity, and coding rules. In several cases, the sentence-transformer candidate was clinically related but not the most appropriate ACHI code under Australian coding definitions. For example, some coronary artery bypass, bronchoscopy, ventilation, and pharmacological administration procedures required coder correction or coder rationale before inclusion in the final mapping dictionary.

Diagnosis mapping considerations (ICD-10-CM vs ICD-10-AM)

ICD-10-CM often encodes additional specificity using extra digits (e.g., laterality such as left/right/unspecified), whereas ICD-10-AM may represent the same concept at the stem/parent level within our label scope. Accordingly, when an ICD-10-CM code has no direct one-to-one equivalent at the same level of specificity in ICD-10-AM, we map it to the closest ICD-10-AM stem/parent code that preserves the clinical concept (a controlled many-to-one mapping). For example, sciatica may appear as M54.30 (unspecified side) in ICD-10-CM, whereas it is represented as M54.3 (Sciatica) in ICD-10-AM (Table 6). In contrast, codes that are shared across the modifications (e.g., I12.0, J18.9) map directly without truncation. Differences in granularity can cause multiple ICD-10-CM codes to map to a single ICD-10-AM code (mapping collisions), reducing the number of distinct labels after mapping.

Table 6.

Example mappings from ICD-9-CM to ICD-10-AM via ICD-10-CM

ICD-9-CM	ICD-10-CM	ICD-10-AM	Description of code
40301	I12.0	I12.0	Hypertensive chronic kidney disease with stage 5
7243	M54.30	M54.3	Sciatica
28521	D63.1	D63	Anaemia in chronic kidney disease
27541	E83.51	E83.5	Hypocalcaemia (disorders of calcium metabolism)
486	J18.9	J18.9	Pneumonia, unspecified organism

Procedure mapping considerations (ICD-10-PCS vs ACHI)

Procedure code systems differ more substantially than diagnosis code systems. ICD-10-PCS uses a seven-character alphanumeric structure, while ACHI uses numeric codes organised into blocks. As a result, one-to-one correspondences are less common and mapping collisions can occur more frequently. In the mapped MIMIC-III experiments, the initial top-50 ICD-9-CM procedure source codes produced 45 unique ACHI procedure labels because several ICD-9-CM procedure variants mapped to the same ACHI target label. For example, ICD-9-CM procedure codes 88.56 (coronary arteriography using two catheters) and 88.55 (coronary arteriography using a single catheter) both mapped to ACHI 38,215–00 (Coronary angiography). Similarly, ICD-9-CM procedure codes 36.06 (insertion of non-drug-eluting coronary artery stent) and 36.07 (insertion of drug-eluting coronary artery stent) both mapped to ACHI 38,306–00 (Percutaneous insertion of 1 transluminal stent into single coronary artery). These examples illustrate how procedure-code variants in ICD-9-CM can collapse into a smaller number of ACHI labels in the final mapped vocabulary.

Checking code validity across editions

During mapping, we verified that candidate Australian codes existed in the ICD-10-AM/ACHI/ACS Eleventh Edition target vocabulary used in this study. We also checked historical IHPA releases from the 1st to 12th editions to identify codes that had been introduced, revised, or retired over time. For example, ICD-10-AM code A08.1 underwent revision from the Tenth Edition to the Eleventh Edition. The Tenth Edition listed the condition as “Acute gastroenteropathy due to Norwalk agent”, whereas the Eleventh Edition updated the title to “Acute gastroenteropathy due to Norovirus”, reflecting a change in terminology. Additionally, the associated inclusion terms were also revised in the Eleventh Edition, highlighting updates in clinical language and accuracy.

Model training

All experiments were conducted on dedicated GPU setups to match their computational requirements:

• Quantised Model (QLoRA): Fine-tuning of the quantised Llama3 model using the QLoRA technique was carried out on a single GPU (NVIDIA GeForce RTX 3090 with 24 GB VRAM).

• Full-Precision Finetuned Llama3 with gold-standard-guided DPO: The full-precision version of Llama3 and preference alignment using Direct Preference Optimisation (DPO) required significantly greater computational resources. For these more demanding experiments, training was performed on an AWS instance equipped with four GPUs, each offering 24 GB of VRAM.

Evaluation metrics

We evaluate automated ICD coding as a multi-label prediction problem. For each discharge summary i, let denote the set of gold-standard codes and denote the set of predicted codes. We report results separately for the mapped dataset and the hospital dataset. For each dataset D, all metrics are computed within its effective label vocabulary (e.g., and ). Codes outside the selected target vocabulary were not included in the prediction targets or evaluation labels; therefore, the reported metrics measure performance within the restricted high-frequency label scope rather than full-vocabulary coding. Diagnosis codes were evaluated as an unordered multi-label set; principal diagnosis, additional diagnoses, and condition onset flags were not modelled or evaluated separately in this study.

Multi-label counts

For a code label , we treat prediction as a binary decision (predicted vs not predicted) and compute:

We do not emphasise true negatives (TN) because the label space is large and sparse, making TN-dominated measures (e.g., accuracy) less informative for clinical coding.

Micro-averaged precision, recall and F1

Micro-averaging aggregates counts across all labels, weighting each individual label decision equally and reflecting overall system performance on frequent codes:

Macro-averaged precision, recall and F1

Macro-averaging computes metrics per label and then averages them, treating rare and frequent codes equally:

If a denominator is zero for a label (e.g., no predicted positives), we define the corresponding term as zero.

Jaccard similarity

We report global Jaccard similarity using the aggregated true positives, false positives, and false negatives across all labels:

Higher JS indicates greater global overlap between predicted and reference code sets.

Confidence intervals

We computed 95% confidence intervals using non-parametric bootstrap resampling over test records. For each dataset and model, we sampled test records with replacement for 1,000 bootstrap iterations and recomputed micro-precision, micro-recall, micro-F1, macro-precision, macro-recall, macro-F1, and Jaccard similarity. We report percentile-based 95% confidence intervals using the 2.5th and 97.5th percentiles.

Results and discussion

Model performance varied by discharge summary length. During preliminary testing, longer discharge summaries, especially those exceeding 2,000 words, often led to repeated text, invalid outputs, or failure to generate a usable code list. Therefore, the quantitative results reported in the following performance tables are limited to discharge summaries within the 1,000-word evaluation setting. Longer summaries were excluded from the reported evaluation rather than truncated. As a result, the reported metrics should be interpreted as performance on a short-document subset rather than the full test set.

The comparisons in this section are limited to the evaluated Llama 3 variants and should not be interpreted as direct comparisons with external automated coding architectures. Tables 7 and 8 compare three approaches: QLoRA (bitsandbytes 4-bit) fine-tuning, full-parameter fine-tuning (non-quantised), and full-parameter fine-tuning with DPO alignment. DPO is applied only to the full-parameter model (not to the QLoRA variant). Results are reported separately for the mapped dataset ( effective labels due to procedure mapping collisions) and the hospital dataset ().

Table 7.

Performance comparison on the mapped ICD-10-AM/ACHI dataset for discharge summaries within the 1,000-word evaluation setting (n = 1,254;|L| = 95)

Metric	QLoRA (4-bit)	Full FT	Full FT + DPO
Micro precision	0.5277	0.6598	0.7262
95% bootstrap CI	[0.5108, 0.5433]	[0.6402, 0.6812]	[0.7074, 0.7444]
Micro recall	0.4951	0.6541	0.7538
95% bootstrap CI	[0.4766, 0.5136]	[0.6328, 0.6764]	[0.7347, 0.7715]
Micro F1	0.5109	0.6569	0.7397
95% bootstrap CI	[0.4956, 0.5247]	[0.6399, 0.6765]	[0.7236, 0.7562]
Macro precision	0.4042	0.6450	0.7259
95% bootstrap CI	[0.3887, 0.4190]	[0.6114, 0.6728]	[0.6968, 0.7500]
Macro recall	0.3389	0.5583	0.6717
95% bootstrap CI	[0.3240, 0.3537]	[0.5247, 0.5879]	[0.6423, 0.7000]
Macro F1	0.3687	0.5985	0.6978
95% bootstrap CI	[0.3559, 0.3808]	[0.5685, 0.6249]	[0.6711, 0.7215]
Jaccard similarity	0.3431	0.4891	0.5870
95% bootstrap CI	[0.3294, 0.3557]	[0.4705, 0.5112]	[0.5669, 0.6079]

Table 8.

Performance comparison on the Australian hospital dataset for discharge summaries within the 1,000-word evaluation setting (n = 5,001;|L| = 100)

Metric	QLoRA (4-bit)	Full FT	Full FT + DPO
Micro precision	0.4842	0.7267	0.7959
95% bootstrap CI	[0.4755, 0.4934]	[0.7143, 0.7379]	[0.7848, 0.8059]
Micro recall	0.4042	0.6899	0.7713
95% bootstrap CI	[0.3957, 0.4131]	[0.6778, 0.7014]	[0.7599, 0.7826]
Micro F1	0.4406	0.7078	0.7834
95% bootstrap CI	[0.4329, 0.4484]	[0.6967, 0.7182]	[0.7725, 0.7927]
Macro precision	0.4941	0.7714	0.8275
95% bootstrap CI	[0.4788, 0.5076]	[0.7587, 0.7828]	[0.8153, 0.8383]
Macro recall	0.3738	0.6726	0.7575
95% bootstrap CI	[0.3620, 0.3852]	[0.6584, 0.6866]	[0.7428, 0.7715]
Macro F1	0.4256	0.7186	0.7910
95% bootstrap CI	[0.4140, 0.4358]	[0.7061, 0.7303]	[0.7791, 0.8022]
Jaccard similarity	0.2825	0.5477	0.6439
95% bootstrap CI	[0.2763, 0.2890]	[0.5345, 0.5604]	[0.6294, 0.6567]

Note: FT denotes full-parameter fine-tuning. Extra-code errors are predicted codes not present in the gold-standard code set. Missed-code errors are gold-standard codes that were not predicted by the model. Error categories are not mutually exclusive; records with both extra and missed codes are counted in both relevant rows. Instance counts are code-level totals and may exceed the number of records because one discharge summary can contain multiple extra or missed codes

QLoRA (4-bit) vs full-parameter fine-tuning

QLoRA gave lower scores than full-parameter fine-tuning across both datasets. On the mapped dataset, QLoRA obtained a micro-F1 of 0.5109 (macro F1 0.3687) compared with 0.6569 (macro-F1 0.5985) for full fine-tuning. On the hospital dataset, QLoRA achieved a micro-F1 of 0.4406 (macro-F1 0.4256), whereas full fine-tuning achieved 0.7078 (macro-F1 0.7186). This reflects the expected efficiency–accuracy trade-off when training with a 4-bit quantised backbone and a limited set of adapter parameters.

Effect of DPO alignment

Applying gold-standard-guided DPO to the full-parameter model achieved the highest scores among the evaluated model variants in both datasets. On the mapped dataset, micro-F1 increased from 0.6569 to 0.7397, and Jaccard similarity increased from 0.4891 to 0.5870. On the hospital dataset, micro-F1 increased from 0.7078 to 0.7834, and Jaccard similarity increased from 0.5477 to 0.6439. These results suggest improved agreement with the reference code sets relative to supervised fine-tuning alone.

Error-type analysis

To better understand the effect of DPO alignment, we performed a code-level error-type analysis comparing the full-parameter fine-tuned model with the DPO-aligned model on the 1,000-word evaluation subsets. Extra-code errors were defined as predicted codes that were not present in the gold-standard code set, while missed-code errors were defined as gold-standard codes that were not predicted by the model. We also counted exact-match records and records with no overlap between the predicted and gold-standard code sets. This analysis was limited to code-level set errors. Long-document generation failures, such as repeated text, invalid outputs, or failure to produce a usable code list, were considered separately and were not included in the error-type counts.

As shown in Table 9, DPO alignment reduced both extra-code and missed-code errors compared with full-parameter fine-tuning. On the mapped dataset, exact-match records increased from 341 to 526, while no-overlap records decreased from 141 to 66. Extra-code instances decreased from 2,356 to 1,985, and missed-code instances decreased from 2,416 to 1,720. On the SWSLHD dataset, exact-match records increased from 2,644 to 3,211, while no-overlap records decreased from 608 to 423. Extra-code instances decreased from 4,269 to 3,254, and missed-code instances decreased from 5,102 to 3,762. These findings indicate that DPO improved code-set agreement by reducing both false-positive and false-negative code assignments in the evaluated short-document setting.

Table 9.

Code-level error-type analysis comparing full-parameter fine-tuning and DPO alignment

Error type	Mapped FT	Mapped DPO	SWSLHD FT	SWSLHD DPO
Records analysed	1,254	1,254	5,001	5,001
Exact-match records	341	526	2,644	3,211
Extra-code error records	718	617	1,959	1,517
Missed-code error records	777	604	2,125	1,606
Records with both error types	582	493	1,727	1,333
No-overlap records	141	66	608	423
Extra-code instances	2,356	1,985	4,269	3,254
Missed-code instances	2,416	1,720	5,102	3,762

Note: Full FT denotes full-parameter fine-tuning. For each metric, the first row reports the point estimate and the following row reports the 95% bootstrap confidence interval. Confidence intervals were computed using non-parametric bootstrap resampling over test records with 1,000 bootstrap iterations. Bold values indicate the highest score for each metric among the evaluated model variants

Mapped vs hospital dataset performance

For the full-parameter and DPO-aligned models, the scores were higher on the hospital dataset than on the mapped dataset. One possible explanation is that the hospital dataset uses native ICD-10-AM/ACHI labels, whereas the mapped dataset may introduce mapping-related uncertainty and label collisions due to cross-system mapping. For example, some ICD-9-CM procedure variants collapse to the same ACHI target label, reducing the final mapped procedure vocabulary from 50 source procedure codes to 45 ACHI labels. The two datasets also differ in source context: MIMIC-III contains United States ICU discharge summaries originally coded with ICD-9-CM, whereas the SWSLHD dataset contains contemporary Australian hospital discharge summaries coded directly using ICD-10-AM/ACHI. Differences in documentation style, clinical terminology, case mix, coding practice, and mapping may therefore all contribute to the observed performance difference. Because we did not perform a controlled matched-subset ablation to isolate the effect of mapping or dataset source, this explanation should be interpreted as a hypothesis rather than a confirmed cause.

Results are reported for discharge summaries up to 1000 words; longer notes led to generation failures (repetition or invalid output) in preliminary testing. Importantly, we did not truncate discharge summaries; longer notes were excluded from the reported evaluation. In future, we aim to explore transformer architectures with extended context windows and structured decoding frameworks to reduce long-document generation failures and improve coding performance on long-form discharge summaries.

Limitations

This study has several limitations. First, the evaluation was restricted to a limited high-frequency code set. The hospital dataset used the top 50 ICD-10-AM diagnosis codes and the top 50 ACHI procedure codes, giving 100 labels in total, while the mapped MIMIC-III dataset contained 95 labels after procedure-code mapping collisions. Codes outside the selected vocabulary were not used as prediction targets or evaluation labels. Therefore, the reported results should be interpreted as performance within a common-code evaluation setting rather than full-vocabulary clinical coding. This restricted label scope may under-represent admissions dominated by rare diagnoses, uncommon procedures, speciality-specific codes, complication codes, or complex multi-morbidity. Records containing both selected and non-selected codes may also be simplified because only selected high-frequency labels are retained for evaluation. Within this restricted label setting, we also treated diagnosis codes as an unordered multi-label set and did not distinguish the principal diagnosis from additional diagnoses or predict condition onset flags. Separating principal and additional diagnoses within this limited label scope may not be reliable because many clinically important diagnosis codes may fall outside the evaluation vocabulary. Future work should evaluate broader code sets and model principal diagnosis, additional diagnoses, and condition onset flags as separate prediction tasks.

Second, the quantitative evaluation was limited to shorter discharge summaries. The reported metrics were calculated only for discharge summaries within the 1,000-word evaluation setting. Longer summaries were excluded rather than truncated because preliminary testing showed repeated text, invalid outputs, or failure to generate a usable code list. This excluded 3,687 of 4,941 mapped MIMIC-III test records and 5,443 of 10,444 SWSLHD test records. Therefore, the reported metrics may describe a shorter-document subset rather than full real-world clinical coding, where longer admissions may contain more diagnoses, procedures, complications, and coding-relevant details. These long-document generation failures have practical implications for clinical coding workflows. If repeated, invalid, missing, or extra code outputs were used without careful human review, they could affect reimbursement, activity-based funding, health-service reporting, and quality indicators. This is particularly relevant for long and complex admissions, where coding decisions may carry greater administrative and clinical importance. Therefore, the current model should be viewed as decision support within the evaluated short-document setting, not as an autonomous coding system for long or complex discharge summaries. Future work should explore long-context models, section-aware chunking, retrieval-augmented generation, and structured decoding to improve performance on longer discharge summaries.

Third, the DPO preference data were gold-standard-guided rather than based on independent human preference rankings. The preferred response was the gold-standard code list, and the non-preferred response was the supervised fine-tuned model output. The preference pairs were reviewed and curated by one annotator, who was also an author, and no second reviewer, inter-annotator agreement analysis, or held-out audit set was available. This limits the reliability assessment of the preference-pair curation process. To reduce subjectivity, the pairs were anchored to gold-standard code sets rather than subjective preferences. In addition, DPO was applied only to the full-parameter fine-tuned model and not to the QLoRA baseline. Therefore, the current results show the effect of gold-standard-guided DPO within the full fine-tuning setting, but do not determine whether similar gains would be achieved when DPO is combined with PEFT/QLoRA. Although we added a code-level error-type analysis, we did not perform a clinician-reviewed qualitative error analysis by ICD chapter, ACHI block, clinical severity, or coding impact. Long-document generation failures were also not included in this code-level error-type analysis. Future work should include multiple independent clinical coding reviewers, formal agreement analysis, evaluation of DPO with parameter-efficient fine-tuning, and clinician-reviewed error analysis by code chapter, error type, and clinical importance.

Fourth, the mapped MIMIC-III dataset depends on indirect cross-system mapping from ICD-9-CM to ICD-10-AM and ACHI. Because no direct one-to-one mapping is available, we used GEMs, semantic similarity over code descriptions, and review by a clinical coding expert. We added a mapping audit summarising which mappings were accepted, retained with coder rationale, or revised after expert review. However, this audit should not be interpreted as an independent mapping accuracy estimate because the review was performed by one clinical coding expert only. No second independent mapping review, inter-coder agreement analysis, or formal mapping error analysis was performed. Differences in code granularity and many-to-one mappings may introduce label noise, especially for procedure codes. In addition, the hospital dataset was drawn from one Australian local health district, so documentation style, coding practice, case mix, and data quality may differ across other hospitals and jurisdictions. Future work should include independent mapping review, formal agreement analysis, and external evaluation across multiple Australian hospitals.

In addition, the reported performance metrics are based on one training run per model. Although we report 95% bootstrap confidence intervals over test records, these intervals quantify test-set sampling uncertainty and do not capture training-run variability due to random seed, optimisation path, or hardware nondeterminism. Because full-parameter fine-tuning and DPO alignment were computationally expensive, the current study was limited to one training run per model condition. Therefore, the observed differences should be interpreted as descriptive comparisons among the evaluated Llama 3 variants rather than statistically confirmed superiority. Future work should include repeated runs with different random seeds and direct comparison with established automated coding baselines under the same ICD-10-AM/ACHI label space and data splits.

Future research directions

In this work, we used LLaMA 3 (8B) and restricted quantitative evaluation to discharge summaries with a maximum length of 1,000 words because generation quality degraded for longer notes (e.g., repetition or failure to output a valid code list). Future work will focus on improving robustness to long clinical documents. Promising directions include long-context models, section-aware chunking and aggregation (e.g., encoding key sections such as diagnoses, procedures, and hospital course), structured decoding with stricter output constraints, and retrieval-augmented generation (RAG) to ground predictions in coding standards and code definitions. Larger-capacity models (e.g., LLaMA 3 70B) may also be explored, although they require substantially greater computational resources and do not by themselves guarantee robustness on long inputs.

We also observed that many high-frequency codes correspond to chronic comorbidities or additional diagnoses rather than the principal diagnosis driving the admission. Future work will explicitly model this distinction by predicting (or ranking) the principal diagnosis separately from additional diagnoses, consistent with Australian coding conventions, and by evaluating principal-diagnosis performance as a dedicated outcome.

Finally, we plan to extend experiments beyond the top-50 diagnosis and top-50 procedure scope by evaluating larger label sets (e.g., top-100 and ultimately broader code coverage), which will require strategies for class imbalance, rare-code learning and hierarchical structure. We also plan to explore additional language models and privacy-preserving approaches (e.g., federated learning), and to investigate retrieval-based methods that incorporate local clinical documentation patterns and coding standards.

Conclusion

Assigning ICD-10-AM diagnosis codes and ACHI procedure codes from discharge summaries is a challenging multi-label task because clinical narratives are heterogeneous and each episode may require multiple diagnosis and procedure codes. In this study, we investigated large language models for automated Australian clinical coding using both a mapped public dataset and a de-identified Australian hospital dataset.

We developed a multi-step normalisation pipeline to map ICD-9-CM labels from MIMIC-III into ICD-10-AM/ACHI using GEMs, semantic similarity over code descriptions, and clinical coding expert review. We then fine-tuned meta-llama/Meta-Llama-3-8B and compared full-parameter fine-tuning with a resource-efficient QLoRA baseline. Full-parameter fine-tuning gave higher scores than QLoRA on both datasets, suggesting a trade-off between computational efficiency and coding accuracy.

We also applied gold-standard-guided Direct Preference Optimisation (DPO) to the full-parameter fine-tuned model. Among the evaluated model variants, the DPO-aligned model achieved the highest score, with micro-F1 scores of 0.7397 on the mapped dataset and 0.7834 on the Australian hospital dataset. These results suggest that gold-standard-guided DPO can improve agreement with reference code sets beyond supervised fine-tuning alone. This should not be interpreted as full human preference learning, because the DPO pairs were based on gold-standard code sets and model outputs rather than independent human rankings of alternative outputs.

Overall, this study demonstrates the feasibility of LLM-assisted ICD-10-AM/ACHI code assignment and provides an initial benchmark for automated Australian clinical coding. Future work should evaluate broader code vocabularies, longer discharge summaries, principal diagnosis and condition-onset information, and multi-annotator preference curation to support stronger evidence for real-world clinical coding workflows.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Abbreviations

AI

Artificial Intelligence

ACS

Australian Coding Standards

ACHI

Australian Classification of Health Interventions

CNN

Convolutional Neural Network

DPO

Direct Preference Optimisation

EHR

Electronic Health Record

GEMs

General Equivalence Mappings

GPT

Generative Pretrained Transformer

HREC

Human Research Ethics Committee

CD-9-CM

International Classification of Diseases, Ninth Revision, Clinical Modification

ICD-10-AM

International Statistical Classification of Diseases and Related Health Problems, Tenth Revision, Australian Modification

ICD-10-CM

International Classification of Diseases, Tenth Revision, Clinical Modification

ICD-10-PCS

International Classification of Diseases, Tenth Revision, Procedure Coding System

LLMs

Large Language Models

LoRA

Low-Rank Adaptation

LSTM

Long Short-Term Memory networks

MIMIC-III

Medical Information Mart for Intensive Care III

PEFT

Parameter-Efficient Fine-Tuning

QLoRA

Quantized Low-Rank Adaptation

SFT

Supervised Fine-Tuning

SWSLHD

South Western Sydney Local Health District

RLHF

Reinforcement learning from Human Feedback

RNN

Recurrent Neural Network

Author contributions

RK conceived the study, performed the experiments, analysed the results, and drafted the manuscript. AG and OO contributed to study supervision, interpretation of results, and critical revision of the manuscript. All authors read and approved the final manuscript.

Funding

This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors. The work was conducted as part of the author’s doctoral studies at Western Sydney University.

Data availability

The public MIMIC-III dataset used in this study is available to credentialed users through the relevant public access process. The SWSLHD hospital dataset is not publicly available because it contains sensitive clinical data and is subject to ethics, privacy, and institutional data-governance restrictions, but may be available from the corresponding author on reasonable request and subject to approval from the relevant data custodian and ethics/governance processes.

Declarations

Ethics approval and consent to participate

This study analysed two datasets: (i) the publicly available, de-identified MIMIC-III clinical database, and (ii) a de-identified hospital dataset accessed via the CEDRIC database within South Western Sydney Local Health District (SWSLHD). The MIMIC-III data was accessed in accordance with the PhysioNet data-use requirements. For the SWSLHD dataset, this study was performed under ethics approval for the Multicultural Emergency Medicine Epidemiology (MEME) study granted by the South Western Sydney Local Health District Human Research Ethics Committee (HREC Reference: LNR/17/LPOOL/432; SSA Reference: LNRSSA/17/LPOOL/433; Project number: HE17/234). All data in CEDRIC are collected under a waiver of consent; therefore, informed consent to participate was waived under this approval. The study did not involve direct recruitment of, or contact with, participants. The study was conducted in accordance with the Declaration of Helsinki.

Consent for publication

Not applicable. No individual person’s data or identifiable information is included in this manuscript.

Competing interests

The authors declare no competing interests.