Can LLMs effectively assist medical coding? Evaluating GPT performance on DRG and targeted clinical tasks

Abstract

Background

Large language Models (LLM) have demonstrated near human-level performance in medical domain, from summarizing clinical notes and passing medical licensing examinations, to predictive tasks such as disease diagnoses and treatment recommendations. However, currently there is little research on their efficacy for medical coding, a pivotal component in health informatics, clinical trials, and reimbursement management. This study proposes a prompt framework and investigates its effectiveness for medical coding.

Methods

First, a prompt-based learning framework is proposed. This framework aims to improve the performance of GPT for complex medical coding tasks by augmenting it with specific prior knowledge, utilizing techniques such as meta prompt, many-shot learning, and dynamic in-context learning. Then its effectiveness is evaluated with three different medical coding tasks. Finally, ablation studies are presented to validate and analyze the contribution of each module in the proposed prompt framework.

Results

For the MIMIC-IV dataset, the prediction accuracies of top-1 and top-5 for the 30 most frequent MS-DRG codes are 68.1% and 90.0%, respectively. The clinical trial criteria coding task results in a macro-F1 score of 68.4 on the CHIP-CTC test dataset in Chinese. Both results are comparable to the best methods in comparison that training deep leaning models or fine-tuning LLMs.

Conclusions

This study demonstrates that for targeted coding tasks, DRG and other simplified scenarios, off-the-shelf GPT models guided by carefully designed prompts can achieve performance comparable to state-of-the-art methods. While current GPT models serve as helpful assistants to human coding specialists,they are not yet equipped to fully replace expert judgment. Importantly, the recent and drastic decline of the cost of deploying large language models makes their integration into medical coding workflows increasingly feasible. As these models continue to evolve, their potential to provide reliable support for DRG coding task is likely to grow rapidly over time.

Trial registration

Not applicable.

Introduction

Since OpenAI released ChatGPT two years ago [1], the world’s leading artificial intelligence (AI) research powerhouses have been relentlessly pushing the state of the art (SOTA) in large language models (LLM). For example, Google introduced Gemini [2], Meta released open source LLaMA [3], and many more. Taking advantage of this mega wave, applied AI research based on commercial, proprietary or open source LLM has flourished in many application domains, from biology, medicine, education, and software engineering to content creation and customer services.

In the medical and healthcare domain, the research literature has shown ChatGPT to be very effective in comprehending English as well as non-English medical documents. GPT-4 achieved scores above the passing level in Korean Pharmacist Licensing Examinations [4]. Subsequently, the enhanced version GPT-4o significantly outperformed average medical students in the United States Medical Licensing Examination [5]. Luo et al. proposed BrainGPT [6] and showed that enhanced LLM exceeded human neuroscience experts in behavioral prediction tasks that require neuroscience knowledge. In a systematic review of ChatGPT in health care applications [7], Wang et al. concluded that conversational LLMs perform well in summarizing health-related texts and answering general medical knowledge questions.

The role of AI in medical coding

The potential of LLM in improving accuracy or explainability in the automation of medical coding is also being investigated. Medical coding is a process of translating electronic health records (EHRs) containing free text descriptions about disease diagnoses, procedures, etc. to alphanumeric codes. They are crucial for effective health informatics management [8], financial reimbursement [9], and clinical trials [10], as well as for retrospective analysis of trends in patient population, allocation of resources, and clinical research. Manual coding is labor intensive, time consuming and error prone, which leads to a great deal of research exploring the potential of natural language processing and deep learning techniques to automate this process [11].

Using clinical text notes in the MIMIC-III dataset [12] to predict international classification of diseases (ICD) codes has been one of the most studied coding tasks. In [13], Hu et al. proposed SWAM, an ICD code prediction model based on a convolutional neural network (CNN). JLAN is a bidirectional long-short-term memory (Bi-LSTM) model proposed by Li et al. [14]. Using a Transformer-based pre-trained language model BioBERT [15] as the backbone, Huang et al. proposed PLM-ICD to tackle the challenges of the task at hand, including long input text and a large prediction label space [16]. In macro-F1 score, the prediction performance of the most frequent 50 ICD codes in the MIMIC-III dataset reached 0.603 and 0.615 by SWAM and PLM-ICD respectively, and further enhanced to 0.665 by JLAN.

Another frequently studied task is to predict Diagnostic Related Groups (DRG), a coding system used to classify hospital cases into groups based on the diagnosis, treatment, and risk of mortality of the patient. Liu et al. trained a long LSTM model over discharge summaries in the MIMIC-III dataset which resulted in a macro-F1 score of 0.041 in predicting all DRG codes [17]. The DRGCoder proposed by [18] leveraged the domain specific pre-trained Transformer ClinicalBERT [19] for the same task and reported a macro-F1 score of 0.101 over all DRG codes.

The vanilla BERT model and many of its variants have also been adopted as a backbone network for classifying clinical trial eligibility criteria, where deep expertise is often required for coding. In [20], RoBERTa-large [21] achieved a macro-F1 score of 0.709 to predict an institutional coding schema of 44 categories, where real clinical trial registration data collected from the Chinese Clinical Trial Registry were used. In [22], Feng et al. exploited the sentence-T5 [23] as a semantic feature generator to encode clinical text.

Development in the LLM era

Since the release of GPT, its potential for medical coding has been also explored. In [24], the free version ChatGPT 2023 was used to encode ICD-10 in more than 150 clinical cases selected from technical books. The authors concluded that ChatGPT can assist but does not replace medical coders. Mustafa et al. conducted a study to predict ICD-10 codes from discharge summaries [25] and found that later version ChatGPT 4 could enhance clinical coding accuracy, though humans still performed better overall. Using leading LLMs including GPT, Claude, Gemini, and Llama, Simmons et al. [26] conducted a large scale investigation with patient notes from the American Health Information Management Association. GPT-4 achieved the highest 15.2% agreement with human coders. The authors concluded that the LLM under evaluation performed poorly in extracting ICD-10 codes from hospital notes. Using Azure OpenAI GPT-3.5, Falis et al. [27] conducted a similar investigation and reported a macro-F1 score of 14.76 on the MIMIC-IV [28] test dataset. The authors reached the conclusion along the same line that GPT-3.5 with basic prompting alone is insufficient for ICD-10 coding.

Instead of relying on basic prompting barely, Wang et al. fine-tuned the Llama model [3] with hospital discharge summaries from the MIMIC-IV dataset for DRG prediction. The authors reported that their DRG-LLaMA [29] achieved a macro-F1 score of 0.327 on all DRGs, surpassing previous leading models in DRG prediction, including ClinicalBERT. In addition to fine-tuning LLM to improve performance on specific tasks, prompt-based learning is another paradigm from which many new techniques are emerging, from few-shot learning, in-context learning, and prompt ensemble to prompt template learning, and many [30].

Techniques improving LLM performance in general domain

When the context window size of LLMs increases rapidly, for example, from 2048 tokens in GPT-3.5 to 8192 tokens in GPT-4, complex tasks can benefit from scaling up the number of in-context examples in a prompt. With Gemini 1.5 Pro supporting up to 1 million token lengths, Agarwal et al. investigated the impact of example numbers on 11 different types of text generation and prediction tasks [31]. The authors introduced a many-shot in-context regime and reported that when prompting with example numbers several hundreds to thousands of shots, large performance jumps occurred, especially on complex reasoning tasks.

In prompt-based learning, the design of prompts can greatly affect task performance. Many prompt engineering methods have been proposed to overcome the suboptimal nature of hand-crafted task prompts. In [32], Gao et al. proposed a pipeline that automates prompt template generation and optimization. The authors reported that their methods outperformed the manual prompt by 11% on average in text classification tasks. Promptbreeder [33] introduced an optimization workflow that iteratively mutates task prompts and evaluates their fitness on a training dataset. Hou et al. argued that the starting point of searching for optimal task prompts matters. The authors proposed metaprompting [34], a task-agnostic framework to learn general meta-knowledge from specific task domains for a better initial task prompt.

Aim of the study

In most medical coding prediction works, GPT-3.5 or GPT-4 was utilized for performance evaluation. Whereas GPT-4o [35] released in mid-2024 has a context window of 128,000 tokens, 14 times larger than that of GPT-4 and 62 times larger than that of GPT-3.5. This drastic increase in token numbers supports the design of much larger and more sophisticated prompts to improve performance on complex tasks. This study proposes a prompt framework for medical coding tasks and uses GPT-4o to evaluate its performance for medical coding tasks over English and Chinese text.

Methods

Every medical code schema, be it an international or institutional standard, is a taxonomy system that comprises definitions of each category in the system and guidelines on how to assign which category to a given text note. In the paradigm of training a model for a specific task [14–22], the quality and amount of annotated text notes used to train the prediction model greatly impact task performance. Task specific knowledge such as taxonomy is not exploited. Inspired by the latest developments in prompt techniques [33–35], this paper proposes a prompt framework specifically designed for medical coding tasks. The rest of this section first presents the proposal and then discusses its application to three coding tasks and the corresponding prompt design.

Prompt framework

As shown in Fig. 1, in addition to an input text and task instruction, a medical coding prompt consists of system knowledge and many examples of learning. The system knowledge has two components: task context that provides an overview of the task and related medical terminologies; and guidelines that aim to improve text classification accuracy. For each task, guidelines are learned one time using a meta prompt to extract common patterns from text and code pairs of the corresponding train datasets. As the size of a training dataset is too large to fit in one context window, the guidelines are learned batch by batch and then summarized into a single set. The guideline learning components are shown in green color boxes in Fig. 1. GPT-4o understands more than 50 different languages. For non-English input text, the meta prompt first translates them into English and then learns guideline from translated text.

Fig. 1.

Framework of medical coding task prompt

In [31], Agarwal et al. experimentally showed that prompting a task with several hundreds to thousands of input and output examples for LLM to learn can significantly improve performance. The proposed prompt framework adopts this many-shot regime. But instead of randomly drawing hundreds of examples from a training dataset, a dynamic in-context retriever is designed. For each task, the texts in their training dataset are vectorized at one time using a small sentence-embedding LLM. At testing time, an input text is vectorized and then a batch of semantically most similar training examples and corresponding answers are retrieved to form the many-shot in-context learning examples as shown in the task prompt box in Fig. 1.

We hypothesize that instead of only increasing the absolute number of learning examples, the more relevant the learning examples to a test input, the more helpful to LLM performing the task. In the subsequent Sects. 3 and 4, it is experimentally validated. The next sections discuss the application of the above proposed prompt framework to three medical coding tasks.

Meta prompt design

Task 1: Semantic coding

This task is to categorize a pair of disease-related questions as similar or different. Similar means both questions asked about the same disease-related issues. The question pairs are in Chinese, see Table 1 for examples of original questions in Chinese and their English translation, where CHIP-STS is a dataset from the Chinese Biomedical Language Understanding Evaluation (CBLUE) benchmark [36].

Table 1.

Examples of CHIP-STS data and task guidelines learned

Applying a meta prompt that includes an instruction to translate all original questions from Chinese to English, a total of 20 classification guidelines are learned from 16,000 training samples in the CHIP-STS dataset. See a few guideline examples Table 1.

Task 2: Clinical trial criteria coding

This task consists of categorizing a short Chinese sentence into 1 of 44 categories of clinical trial criteria. The dataset used here is CHIP-CTC from the CBLUE benchmark, where all text data is from the Chinese Clinical Trial Registry. One challenge of the dataset is that the number of samples in each category is drastically different. As shown in Fig. 2, the categories Disease and Therapy or Surgery constitute more than 40% of the total samples, and about half of the 44 categories have samples of less than 1% of the total samples.

Fig. 2.

CHIP-CTC dataset sample frequencies by code categories

For this task, there are a total of 22,962 training samples in the CHIP-CTC dataset. For each category, the corresponding training examples are selected from the training set for a meta prompt to learn a classification guideline. A total of 44 guidelines. See a couple of examples in Table 2.

Table 2.

Examples of guidelines learned by meta prompt

Category	Classification Guideline Learned
Smoking Status	To determine if a text belongs to the ‘Smoking Status’ category, look for keywords and phrases related to smoking habits, history, or cessation. Common indicators include references to smoking frequency (e.g., ‘daily smoking’, ‘more than 5 cigarettes’), time frames (e.g., ‘past 3 months’, ‘within 6 months’), smoking cessation (e.g., ‘quit smoking’, ‘on-smoker’), and smoking history (e.g., ‘smoking history’, ‘pack years’). The context often involves clinical trials, medical assessments, or health evaluations. Texts may also mention nicotine products or smoking restrictions during study periods. If a text includes these elements, it likely pertains to smoking status.
Neoplasm Status	To determine if a text belongs to the ‘Neoplasm Status’ category, look for keywords and phrases related to the presence, size, stage, or spread of tumors. Common indicators include terms like ‘tumor’, ‘eoplasm’, ‘metastasis’, ‘stage’, ‘diameter’, ‘invasion’, and specific staging notations such as ‘T’, ‘N’, ‘M’ (e.g., T3N1M0). The context often involves descriptions of tumor characteristics, such as size measurements (e.g., ‘3 cm’), stages (e.g., ‘stage IIIB’), or spread to other organs (e.g., ‘liver metastasis’). Additionally, references to clinical assessments, imaging results, or pathological findings that describe the status or progression of a neoplasm are strong indicators of this category.

Task 3: MS-DRG coding

This task consists of predicting MS-DRG codes from hospital discharge summaries. There are more than 700 unique MS-DRG codes [37]. Each DRG code describes a set of patient attributes, including the primary diagnosis, specific secondary diagnoses, procedures, sex, and discharge status. The discharge status tells whether the patient was discharged alive or not, with or without major complications and comorbid conditions. We follow the dataset preparation schema in DRG-LLaMA [29] to randomly divide the MIMIC-IV dataset into 90% for training and 10% for evaluation.

In this study, we evaluate the most frequent 30 DRGs. DRG is an international standard where each DRG code has a definition serving as a manual for code mapping. These definitions are short phrases given in [37]. See a couple of definitions in Table 3. Instead of learning the classification guidelines from the task training set, the meta prompt of this task translates each DRG definition into more comprehensible cohesive short paragraphs, as shown in Table 3.

Table 3.

Examples of MS-DRG definition and corresponding guidelines learned

MS-DRG	Definition
392	Esophagitis, gastroenteritis and miscellaneous digestive disorders without MCC
871	Septicemia or severe sepsis without MV 96 hours with MCC
853	Infectious and parasitic diseases with O.R. procedure with MCC
394	Other digestive system diagnoses with CC
MS-DRG	Classification Guideline Learned
392	Assign to DRG 392 for cases of esophagitis, gastroenteritis, or other digestive disorders without major complications or comorbidities (MCC). The patient should have milder symptoms and require standard treatment without intensive care.
871	This DRG code applies to patients with septicemia or severe sepsis who do not require mechanical ventilation for more than 96 hours but have major complications or comorbidities (MCC).
853	Assign DRG 853 to cases involving infectious or parasitic diseases that required an operating room (O.R.) procedure and are accompanied by major complications or comorbidities (MCC). Look for severe infections necessitating surgical intervention and significant additional health issues.
394	Select DRG 394 for other digestive system diagnoses with a complication or comorbidity (CC). The patient should have moderate symptoms that require additional care but not as intensive as MCC.

Many-shot dynamic in-context retriever

In natural language processing (NLP), vector embedding is a technique that transforms words, phrases, or texts into numerical vectors so that quantitative metrics can be used for analysis. Since sentence-BERT [38] set a new SOTA on a range of NLP tasks, many small LLMs have been made available on Hugging Face. In this study, we curated two small LLMs for text embedding functionality in the retriever module.

The text data in tasks 1 and 2 are Chinese, so CoSENT [39] trained with Chinese corpora is used to generate embedding vectors from Chinese sentences and question pairs from respective datasets. At testing time, each input in its original language is first transformed into an embedding vector, and then the cosine similarity is used to find the most similar examples from the corresponding training set. The discharge summaries in task 3 are in English and on average much longer, so sentence-T5-large [23] model is chosen to embed text.

refers to the number of many-shot learning examples as shown in the task prompt box in Fig. 1. In Section 3, optimal value of for each task and other aspects of the evaluation setup will be discussed.

Results

Technical setup

For evaluating the performance of the prompt framework applied to these coding tasks, the GPT-4o 2024–08–01 preview version of the Azure OpenAI service was used. Azure OpenAI service is one of the three online GPT and similar services that are recommended for the responsible use of the MIMIC dataset [40].

Following related work, the macro-F1 and accuracy metrics are used for performance assessment in this study. A macro-F1 score is an average over F1 scores of each unique code, and gives a sense of effectiveness on minority classes in tasks 2 and 3, where data are extremely imbalanced. On the other hand, on average over an entire testing dataset, a micro F1 score reports a performance potentially skewed towards majority classes. In multiclass classification tasks, micro F1 is equivalent to accuracy.

For each task, samples of many-shot prompt are available at https://tinyurl.com/mfc5w5p9 for reference.

Performance of tasks 1 and 2

The CHIP datasets have training, development, and testing datasets, where true class labels are available only in training and validation sets. The training sets are used to learn classification guidelines for respective tasks. GPT-4o has a context window size of 128,000 tokens, it can support hundreds if not a thousand learning examples in the task prompt.

To find the optimal many-shot size for each task, a part of development sets, sample id 1–1000 in task 1 and sample id 1–500 in task 2, were used for grid searching. Considering the high latency and high cost of the Azure OpenAI service, we use a subset to determine the optimal hyperparameters in all tasks.

As shown in Fig. 3, when the number of shots increases, the performance increases significantly in both tasks. In task 1, the best macro-F1 score reaches 0.7909 when the number of shots is equal to 40, and the best result in task 2 is 0.8274 when the number of shots is equal to 20. When the size of many-shot continues to grow, task 2 performance quickly degrades and task 1 performance also gradually drops.

Fig. 3.

Comparison of prediction performance by many-shot size

The STS and CTC test set has 10,000 and 10,193 samples, respectively. Performance of tasks 1 and 2 are evaluated on the CBLUE benchmark platform with complete test sets. In Table 4, the results of our method are compared with the results of the leading methods reported in [36], where BERT networks with various improvements were trained with Chinese corpora. Task 1 predicts whether a given question pair is semantic similar or not. Existing supervised model training approaches outperform our prompt-based learning method. Compared to task 1, task 2 is more complex, where a given sentence is classified into 1 of 44 categories. Our prompt-based learning method performs on par with existing supervised model training approaches and is even better than ALBERT-xxlarge.

Table 4.

Performance macro-F1 score on CBLUE benchmark

Task	MacBERT-large	RoBERTa-large	ZEN	ALBERT-xxlarge	Ours
Task 1 (STS test)	0.856	0.847	0.835	0.848	0.797
Task 2 (CTC test)	0.686	0.709	0.686	0.669	0.684

To find out whether the classification guidelines learned by the meta prompts, as shown in Table 1 and 2 are helpful in improving the prediction performance, we remove the guidelines from the task prompts and run the experiments again, but with smaller evaluation sets as labeled in Fig. 3, namely ablation study sets. Compared to the best results shown in Fig. 3, without guidelines, the macro-F1 score on the CTC ablation set is 0.771, decreases by 0.056, and the macro-F1 score on the STS ablation set is 0.761, decreases by 0.03. Therefore, the guidelines are helpful for the performance of tasks 2 and 1.

The dynamic in-context in Sect. “Many-shot Dynamic In-Context Retriever” means that the many-shot learning examples in the task prompt are semantically most similar to a test input. To measure its effect, in ablation studies, the learning examples are randomly drawn from a task’s training dataset, that is, without correlation wit a test input. The results of the experiment reveal that this proposal is extremely helpful for task 2. Without being able to learn from examples semantically similar to test inputs, the macro-F1 score over ablation set drops from 0.827 to 0.675, as high as 0.152. However, both ways give the same performance in Task 1. One possible reason could be task 1 being relatively simple.

Performance of task 3

The most frequent 30 DRGs in the MIMIC-IV test set has 7665 samples, which form the test set of Task 3. The sample frequency is shown in the top plot of Fig. 4. The length of a discharge summary is much longer than the input data in the previous two tasks. The maximum many-shot size used for evaluation is 80, the maximum number that is supported by GPT-4o.

Fig. 4.

MIMIC-IV sample frequencies by MS-DRGs and prediction performance

To quantify the impact of many-shot size on DRG coding performance, 100 random samples were drawn from the test set for evaluation, that is, the ablation set. At 30 shots, the macro-F1 score for the 30 most frequent DRG codes is 0.735. Increasing the shot size to 80 improves the performance to 0.770. Note that, due to the limited sample size, these results cover only a portion of the 30 target codes.

In Table 5, the result of our method over the complete test set is compared with related works on MIMIC datasets, where IV is an updated version of III. Both CAML and DRG-LLaMA are supervised model training methods. Their difference is that in the CAML method, CNN features are pooled using the attention mechanism for each class. In the latter method, an LLaMA model is fine-tuned for the task.

Table 5.

Prediction performance of most frequent 30 MS-DRGs

Method	Dataset	Macro-F1	Accuracy	Top-5 Accuracy
CAML [17]	MIMIC-III	0.395	0.5021
DRG-LLaMA-7B [29]	MIMIC-IV	0.737	0.694	0.941
Ours	MIMIC-IV	0.621	0.681	0.900

In CAML, result was reported in micro-F1 score, equivalent to accuracy

We can see that the SOTA LLM exhibits a clear superiority over CNN. DRG-LLaMA outperforms CAML by 0.192 in the accuracy term. When considering the performance of minority classes as shown in the upper part of Fig. 4, DRG-LLaMA outperforms with a macro-F1 score of 0.342. Our prompt-based learning method also outperforms CAML at a similar level and is on par with DRG-LLaMA, when measured in terms of accuracy. Top-5 accuracy means that for each test input, five DRG codes are predicted, respectively.

Measured in macro-F1, DRG-LLaMA surpasses our method. In Fig. 4, the performance of each DRG code is plotted in the order of their sample frequency. We can see that there is a weak positive correlation between code prediction performance and the corresponding sample frequency. The bottom performing code predictions clutter at the below 1% frequency region.

Often, prediction performance of codes with more samples will be better than codes with fewer samples. However, the prediction scores sorted by sample frequency in Fig. 4 do not exhibit such a pattern. For example, the macro-F1 score of DRG code 27, one of the 3 least frequent DRG codes, is higher than 0.9. DRG code 27 is of surgical type. Another type is medical. Of the 30 most frequent DRG codes, 6 belong to surgical type and the rest are medical type. When average macro-F1 scores are derived by DRG type, it reveals that performance of most surgical codes is much better with an average macro-F1 score of 0.805. The average of the medical type of DRG is 0.575, while the overall macro-F1 is 0.621 as shown in Table 5.

In terms of the efficacy of task guidelines, task 3 is highly benefited compared to the efficacy analysis on tasks 1 and 2. Without the meta-prompt-learning classification guidelines as shown in Table 3, macro-F1 score on the ablation set is 0.427, a collapse of as large as 0.343.

Task 3 also benefits tremendously from the dynamic in-context method. when in-context examples are replaced by random examples, the macro-F1 score over the ablation set decreases to 0.584, a degradation of 0.186.

The results of these ablation studies indicate that the more complex the task, the more helpful both the task guidelines and dynamic in-context learning are.

Discussion

The above section provides an overview of the overall performance of the proposed prompt framework applied to three different medical coding tasks, from relatively simple semantic coding to more complex clinical trial criteria and MS-DRG coding tasks, as well as ablation studies on the efficacy of the proposed prompt meta-prompt design and dynamic in-context learning. We also investigated whether the same version GPT-4o mini performs on par or better and found that GPT-4o gives better results than GPT-4o mini. For example, on the CTC ablation set, GPT-4o mini produces a result of 0.033 lower than GPT-4o in macro-F1 term.

How will earlier version GPT perform on these tasks? Older models came with a much smaller context window, as discussed in Sect. “Techniques Improving LLM Performance in General Domain”. With GPT-3.5 and GPT-4 supporting a maximum of 2048 and 8192 tokens, both cannot accommodate the number of prompt tokens required for the DRG task. However, it can meet the token requirements of the other 2 tasks, as shown in Table 6. Performance with GPT-4 in the macro-F1 score is 0.822 and 0.742 in the ablation set of tasks 1 and 2, respectively. As shown in Fig. 3, when using GPT-4o, the macro-F1 score of task 1 is 0.791, a 0.031 decrease in performance; and 0.827 in task 2, a performance increase of 0.085 compared to older GPT-4. One takeaway from this comparison can be that for simple tasks, the older version GPT model will suffice, whereas a more complex task will benefit from the new GPT model.

Table 6.

GPT-4o cost estimation per 100 codes prediction

	Task1 (STS)	Task2 (CTC)	Task3 (DRG)
Input/Output Tokens	53,400/2,000	117,900/3,000	6,605,400/2,100
Realtime API	$0.15	$0.32	$16.53
Batch API	$0.08	$0.16	$8.27

The remainder of this section discusses the cost and limitation of the proposed method from a practitioner’s lens.

Computational cost of prompt scaling

Many-shots prompting substantially increase the number of tokens submitted to GPT models, directly increasing computational costs. For practitioners, the trade-off between performance and cost is one of the important aspects that influence the choice of methodology, specifically between use GPT or to train or fine-tune a much smaller model such as Sentence-T5 in a supervised manner. LLMs avoid annotation expenses, but scaling up learning shots erodes this advantage. In contrast, supervised approaches require high-quality clinical annotations, a substantial and recurrent operational burden.

Although the costs of the latter approach vary between different organizations, we estimate the cost of the proposed method for the tasks evaluated in the previous section. This enables a systematic comparison of inference expenses across. According to OpenAI’s published pricing1 (as of [30 June 2025]), GPT-4o realtime API charges USD $2.5 for per million input tokens and USD$10.0 for per million output tokens. Table 6 lists the tokens consumed by each task evaluated in Sect. 3 to predict per 100 codes and the corresponding API cost.

OpenAI also offer an asynchronous batch API option, which reduces the cost by 50%. It costs USD $8.27 to code 100 DRG diagnoses. By providing a mechanism to filter out low-confidence (and likely incorrect) predictions, the proposed method can significantly reduce annotation costs and project time, particularly given the high expense of human annotation labor.

Limitation

In tasks 1 and 2 when the size of the many-shot reaches a point, as shown in Fig. 3, the performance of the task is penalized when the many-shot size continues to increase. In task 3, as the input data are much larger, GPT-4o supports up to 80 shots in the task prompt used. However, an ablation study with a many-shot size of 30 results in a 0.035 decrease in macro-F1 score over its ablation set. This suggests that there is a potential that the performance of DRG code prediction could benefit from using an LLM that supports a larger context window size.

Conclusions

This study proposes a language-agnostic prompt framework to predict medical codes with LLM. This framework exploits the latest techniques developed in the prompt-based learning field to improve the performance of prompt-based learning in complex tasks, including meta prompt, many-shot, and dynamic in-context learning. The framework implementation combines the commercial Azure OpenAI GPT-4o service with small open-source LLMs. Subsequently, the effectiveness of this framework is evaluated using different tasks in the context of institutional and standard coding schema. Ablation studies show the key proposals, extracting task specific knowledge into classification guidelines and multi-many dynamic in-context learning are effective, and drastically lift the performance of complex tasks. Compared to related works that take a supervised model training approach, our prompt-based learning framework gives comparative performance on two complex coding tasks but underperforms on relatively simple semantic coding tasks. With the rapid advancement of LLMs, the size of their context window will increase rapidly. The proposed prompt framework has the potential to further enhance DRG code prediction performance.

Author contributions

All authors contributed to study conceptualization, data collection and analysis, and manuscript writing. All authors read and approved the final manuscript.

Funding

This research is supported by Amplify Health Asia.

Data availability

Publicly available datasets were analyzed in this study. These data can be found here: https://tianchi.aliyun.com/dataset/145741, and https://physionet.org/content/mimic-iv-note/2.2/.

Declarations

Ethics approval and consent to participate

Not applicable.

Consent for publication

Not applicable.

Competing of interests

The authors declare that they have no competing interests.