Using large language models for clinical staging of colorectal cancer from imaging reports: a pilot study

Abstract

Purpose

Accurate data collection and analysis are crucial in clinical research, particularly for extracting information from unstructured medical records in cancer research. Traditional methods often struggle with this task. Large language models (LLMs) specializing in natural language processing (NLP), like ChatGPT (OpenAI), show potential for automating this process. This study evaluated whether GPT-4 could accurately extract clinical stages of colorectal cancer (CRC) from imaging reports.

Methods

Using specific prompts based on the American Joint Committee on Cancer TNM staging system, GPT-4 was tested on the unstructured abdominal imaging reports of 100 CRC patients. The results were evaluated by a colorectal surgical oncologist and compared with data manually extracted by a nonspecialist data manager.

Results

GPT-4 demonstrated high accuracy in extracting lesion locations (96.0%) and T (89.0%), N (90.0%), and M (85.0%) stages, with an overall TNM stage extraction accuracy of 69.0%. The combined accuracy for TNM stage and lesion location was 67.0%. Human data managers had similar TNM stage accuracy but lower lesion-location accuracy (76.0%). Higher accuracy was observed when reports directly mentioned stages and were in English only.

Conclusion

This study confirms that LLM-based NLP, with proper prompt engineering, can accurately extract clinical stages from CRC imaging reports, particularly in English-only contexts.

INTRODUCTION

In clinical research, accurately collecting and analyzing patient data are crucial. Traditionally, data collection has relied on manual processes, and even with clinical data warehouses (CDWs), extracting information from unstructured medical records requires significant human effort. The process is laborious, time-consuming, and costly, posing obstacles to effective clinical research [1,2]. Data managers need to be well-versed in medical terminology and disease, necessitating extensive training. Consistency in data collection is also challenging, requiring strict criteria and definitions to ensure high-quality databases. In cancer research, including colorectal cancer (CRC), data extraction errors are reduced when semi-structured records, such as pathology reports or functional test reports for cardiac or pulmonary assessments, are used. On the other hand, collecting data from freeform medical records, such as imaging reports, surgical notes, medical charts, nursing records, and unstructured questionnaires, is difficult and the results are often inaccurate, which reduces the reliability of clinical research outcomes [3,4,5,6].

With the increasing use of big data in clinical research, there is a growing need to develop methods for automatically extracting data from unstructured medical records written in natural language. Numerous studies have reported the application of various artificial intelligence (AI) models for natural language processing (NLP) [3,4,6,7,8,9]. However, traditional NLP methods face significant challenges in clinical settings. These include the burden of extensive preprocessing, such as pretraining requirements, a lack of flexibility in understanding language context or accurately representing data, and scalability issues arising from difficulties in management, expansion, and performance degradation.

In comparison, large-scale language models (LLMs) are specifically designed for human conversation and language generation, making them well-suited for NLP tasks AI models. Since the introduction of ChatGPT by OpenAI in November 2022, efforts have been made to apply LLMs in various fields, including medicine. Research using LLMs encompasses disease occurrence and prognosis prediction, medical imaging analysis, clinical decision support system development, virtual simulations for medical education and training, and automated content creation [10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27]. Among these applications, NLP using LLMs holds particular promise for clinical settings due to their superior contextual understanding and usability. Unlike traditional NLP methods, LLMs require minimal preprocessing and can be effectively employed through prompt engineering or fine-tuning with small amounts of labeled data, eliminating the need for extensive labeling of medical records. Despite this potential, there are relatively few studies on the use of LLMs in clinical settings. In particular, extracting clinical stages from the medical records of cancer patients remains an underexplored task due to the complexity of clinical expertise required during the prompt engineering phase [28]. This study aims to determine whether LLMs can accurately extract CRC clinical stages from imaging reports written in natural language.

METHODS

This retrospective single-arm study was conducted in April 2024 and was approved by the Institutional Review Board of Korea University Anam Hospital (No. 2024AN0254). We utilized GPT-4 (Legacy model, Turbo version), implemented in ChatGPT (OpenAI, Inc.), as a representative LLM to extract tumor location and clinical TNM (cTNM) stages from preoperative abdominal CT reports of patients with CRC at the time of their initial diagnosis. The reports were unstructured free-text documents prepared by more than 5 different radiologists. While the imaging reports were primarily written in English, some included mixed Korean, and the lengths of the reports varied significantly. In this context, clinical TNM stage refers to radiologic TNM rather than pathologic TNM. The TNM staging system is an internationally recognized standard system for assessing cancer progression. TNM represents tumor (T), nodes (N), and metastasis (M), systematically categorizing the primary tumor, the involvement of regional lymph nodes (LNs), and the presence of distant metastases.

This study used preoperative imaging reports from CRC patients who underwent resection at the Division of Colon and Rectal Surgery, Korea University Anam Hospital. A colorectal surgical oncologist manually extracted the lesion location and TNM stage from the reports and selected those suitable for the study. Reports with tumor locations in the cecum, ascending colon, descending colon, sigmoid colon, and rectum were included. For simplicity in classification, less common lesion locations such as the appendix, hepatic flexure, transverse colon, splenic flexure, rectosigmoid junction, and anal canal were excluded. Additionally, reports containing multiple lesions, presenting more than 1 stage for a single lesion, or reporting no mention of the primary colonic lesion were excluded. However, reports without mentions of LNs or metastasis were included, and the absence of such mentions was evaluated as N0 and M0, respectively. A total of 100 imaging reports meeting these criteria were selected and used for the study.

Prompt engineering was performed to assign GPT-4 the role of data manager in the colorectal division, instructing it to extract the lesion location and TNM stage from reports according to specific rules (Appendix). To minimize potential errors commonly associated with LLMs (e.g., hallucination, misinterpretation, omission), we provided highly detailed extraction rules and a predefined output format. The extraction rules were as follows: (1) Use the TNM stage if it is directly recorded in the report. (2) If it is not directly recorded but the definition of TNM is included, use the corresponding stage. (3) If neither a direct TNM stage nor a definition is included but imaging findings or descriptions correspond to a TNM stage, use the corresponding stage. The TNM staging definitions and regional LNs (based on the CRC lesion location) from the 18th edition of the American Joint Committee on Cancer (AJCC) guideline were provided as references for rule 2, along with radiological expressions for each T and N stage for rule 3. The extraction format specified was: (1) lesion location (1 word), (2) TNM stage, (3) reason for extracting the TNM stage. Each report was processed in a new conversation to prevent previous extraction influences, repeating the task each time by including role assignment, extraction rules, extraction format, and report in 1 message window.

GPT-4’s extracted results were evaluated to determine lesion location, T stage, N stage, M stage, TNM stage, and overall accuracy (Fig. 1). The manually extracted data, provided by a colorectal surgical oncologist, served as the reference standard for the assessment. If GPT-4’s answers matched the reference exactly or were broadly similar (e.g., M1b vs. M1), they were considered correct. This comparison determined GPT-4’s extraction accuracy, which was also compared with results previously extracted by a nonspecialist human data manager. The extraction accuracy for each item was also analyzed based on lesion location (colon or rectum), the inclusion of direct mentions of the T and N stages, and the use of mixed language (English and Korean) in the reports.

Fig. 1. Response of ChatGPT (OpenAI, Inc.) powered by GPT-4.

RESULTS

Table 1 shows the features of the CRC lesions: 58 colon cancer cases and 42 rectal cancer cases. Stage T3 was the most common, with 57 cases of N1 or N2 and 43 cases of N0. Most cases had no intra-abdominal metastasis. The median length of the imaging reports was 94 words (Q1, 73 words; Q3, 119 words) as shown in Table 2. Fifty-five cases had direct mentions of the T or N stages in the reports, and 35 reports mixed Korean and English.

Table 1. Colorectal cancer characteristics.

Values are presented as number.

Table 2. Report characteristics.

Values are presented as median (interquartile range) or number.

Table 3 presents the accuracy of the lesion locations and TNM stages extracted by GPT-4 and the human data manager. GPT-4 showed 96.0% accuracy in lesion-location extraction and 89.0%, 90.0%, and 85.0% for the T, N, and M stages, respectively. Accuracy was higher for advanced T stages (T1, 50.0%; T2, 87.5%; T3, 91.4%; and T4, 90.0%) and positive LN cases (96.5%). GPT-4 had lower accuracy for intra-abdominal metastasis extraction (70.6%). Overall, it correctly extracted the TNM stage in 69.0% of cases and both the TNM stage and lesion location in 67.0%.

Table 3. Accuracy of GPT-4 and human data manager.

Values are presented as number only or number (%).

The human data manager had TNM stage extraction accuracy similar to that of GPT-4, but the lesion-location accuracy was significantly lower (76.0%, P < 0.001), resulting in only 54.0% correct both TNM stage and lesion-location extractions. The human data manager’s substage accuracy was higher for stages T2 and T3 but lower for T1 and T4, respectively (T1, 75%; T2, 81.3%; T3, 91.4%; and T4, 60.0%). Despite these differences, there was no statistically significant difference in the overall T stage extraction accuracy between the human data manager and GPT-4. The human data manager also showed no significant differences in N stage accuracy but had larger discrepancies for M stages compared to GPT-4 (M0, 97.6%; M1, 64.7%).

The subgroup analyses revealed that GPT-4’s extraction accuracy did not differ based on the word count of the imaging reports or the cancer location (colon vs. rectum) (Table 4). However, the T and TNM stage accuracy was significantly higher when direct mentions of these stages were included in the reports (T stage, 96.4% vs. 80.0%, P = 0.008; TNM stage, 80.0% vs. 55.6%, P = 0.012). Although the difference was not statistically significant, the N stage accuracy trended higher when direct mentions were present (94.5% vs. 84.4%, P = 0.088). M stage accuracy did not differ between the groups.

Table 4. Accuracy of GPT-4 by characteristics of the cancer and reports.

Values are presented as number (%).

Reports written exclusively in English showed significantly higher accuracy for T stage, TNM stage, and overall accuracy than mixed-language reports (T stage, 77.1% vs. 95.4%, P = 0.005; TNM stage, 54.3% vs. 76.9%, P = 0.019; overall accuracy, 51.4% vs. 75.4%, P = 0.015). The accuracy for lesion-location extraction did not differ statistically, but it trended higher in English-only reports (91.4% vs. 98.5%, P = 0.089). N and M stage accuracy did not differ between the groups.

DISCUSSION

Our study demonstrates that, with appropriate prompt engineering, LLMs can accurately extract clinical stages from imaging reports written in natural medical language. The extraction accuracy of the LLM was comparable to that of a human data manager, and it was notably higher when the imaging reports included direct mentions of the stage or were written solely in English.

Since the introduction of ChatGPT, various medical studies have explored LLM applications [10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27]. However, many of them have focused on single, straightforward tasks. For example, a previous study from our institution involved explaining the condition of CRC patients to GPT-3.5 and asking it to suggest treatment plans and then comparing those suggestions with previous multidisciplinary team results [13]. Although that might seem complex, the structure of the task was relatively simple because the state of the CRC patient was pre-assessed by human doctors, and GPT-3.5 did not influence that judgment. In contrast, our study required GPT-4 to complete a 2-step process: first, it had to process natural language inputs from imaging reports and output them in a structured format for lesion location and TNM staging; second, it had to use predefined rules to stage the patients from report descriptions. This study confirms that LLMs can complete such a complex, 2-stage task. Additionally, the fact that such tasks can be accomplished solely through prompt engineering is a significant advantage of LLMs.

We assigned GPT-4 the role of a data manager in the colorectal division, instructing it to extract the TNM stage and providing it with rules for that task. Notably, we found that most imaging reports do not use the exact definitions from the AJCC guidelines. For example, T3 is defined as “tumor invades through the muscularis propria into pericolorectal tissues,” but reports seldom use that phrasing. Instead, they describe it morphologically, such as “smooth or nodular extension of a discrete mass of tumor tissue beyond the contour of the bowel wall with extension into pericolic fat.” By including such radiological descriptions in the rules, we could improve the accuracy of the LLM’s judgment. This prompt engineering approach can be widely applied to other cancer types as well. We also specified the format for extracting answers. Without this, LLMs tend to respond in sentence form, which can cause readability issues. The formatted outputs can be converted into CSV, JSON, or Excel files for statistical analysis, indicating that such LLM capabilities could significantly enhance CDWs by extracting valuable information from unstructured data. However, when processing large volumes of medical data, practical considerations arise, such as the need to implement application programming interface (API)-based methods instead of relying solely on dialog-box-driven prompt engineering. This remains a separate topic for further discussion.

In our study, GPT-4 demonstrated high accuracy in extracting TNM stage (85%–90%) and lesion location (96%), comparable to an experienced human data manager. The higher accuracy in lesion-location extraction is likely because GPT-4 referenced only the report content, whereas human data managers incorporate additional records, such as colonoscopy and digital rectal examinations, that introduce variability. Similarly, the accuracy difference in M staging could stem from the inclusion of other imaging examinations. The key point is that GPT-4 accurately extracted clinical stages based solely on the provided examination results.

In this study, the extraction accuracy for lower T and N stages was lower, likely due to frequent false positives in descriptions of tumor depth or node metastasis. Negative prediction is more challenging than positive prediction for T and N stages [28]. For example, reports of early CRC (Tis or T1) mention only the presence of the primary tumor without specifying its depth, often leading to over-staging. For LNs, nonspecific mentions such as a “visible” LN, equivocal or benign-sounding descriptions, or the absence of any nodal mention often caused false positives. However, GPT-4 sometimes correctly identified negatives, suggesting that it considers the overall context of the report rather than just specific keywords.

Unlike the T and N stages, GPT-4 showed better negative M stage prediction than positive. Most of the errors in identifying metastatic cases occurred because the system did not classify non-regional LN metastasis, such as paraaortic LN, as M1, instead evaluating it as N positive, or because it made mistakes in the substaging of M1. These issues likely arise from insufficient prompt engineering details rather than inherent ambiguity in the reports or problems with the LLM’s comprehension. This issue could likely be improved by offering more specific and detailed explanations at the prompting stage.

The imaging reports used in this study were descriptive texts freely written by multiple radiologists without a standardized format or restrictions, resulting in heterogeneity in length, language usage, and structure. Therefore, we also examined factors that influenced GPT-4’s extraction accuracy for TNM staging: advances in the T, N, and M stages, lesion location (colon vs. rectum), direct stage mentions in reports, and mixed-language usage [6]. Direct mentions of T and N stages and English-only reports significantly improved extraction accuracy. As expected, reports with direct T and N stage mentions had high accuracy, with the rare errors caused by discrepancies between the descriptions and the stated stages. Because reports rarely present the M stage directly, those 2 groups did not appear to differ. The mixed-language reports showed lower accuracy for lesion location and T stage, with GPT-4 often suggesting higher stages than the reference.

Our study has limitations. First, discrepancies between the described content and the T and N stages stated in the reports could affect accuracy. According to the working rules, if the TNM stage was presented in the report, the LLM was instructed to use that stage. Therefore, any deviations were considered incorrect. However, if the content of the descriptions had been more broadly acknowledged, the accuracy might have been higher. Second, colorectal specialists evaluated the reference TNM stages, which could be erroneous. Reviewing instances in which several LLMs produced incorrect results revealed reference errors. Third, when evaluating the M stage, it is necessary to comprehensively assess metastasis to other organs, such as the lungs, in addition to abdominal organs, using various imaging results. However, this study evaluated extraction accuracy based only on metastasis confirmed in abdominal imaging. The human data manager might have reviewed multiple examinations comprehensively to extract the M stage, which could have led to some cases being considered incorrect. Future studies should assess multimodal evaluations.

Despite those limitations, our study is unique in using LLMs to complete complex, 2-step tasks involving judgment and showing that they achieved high accuracy comparable to that of experienced human data managers. The outputs can be directly applied to database construction and other practical uses. Moreover, this approach offers high potential for scalability and generalizability, as it can be extended beyond CRC to other cancer types, imaging modalities, report formats, and languages.

In conclusion, our study confirms that using LLM-based NLP to extract clinical stages for cancers such as CRC is feasible and highly accurate. Proper prompt engineering, including stage definitions and radiological descriptions, and the use of English-only reports enhances accuracy. Future efforts should simplify prompt engineering and integrate multimodal evaluations to facilitate the easy and direct application of LLM-based NLP in clinical settings.

Appendix

Prompt engineering to extract clinical stage of colorectal cancer from imaging reports

[Authors’ message]

You are a data manager at the department of colorectal surgery. Your mission is to extract the TNM stage of colorectal cancer by reading the abdominal CT or rectal MRI report of the patient. TNM stages are expressed as, for example, T3N1M0, T2N0M0, T4bN2M1a, etc. Let me give you some rules to follow when extracting TNM stage.

First, if the TNM stage is recorded directly on the abdominal CT or rectal MRI report, you can use it as is.

The second rule is that if the abdominal CT or rectal MRI report does not directly mention TNM stage, but instead contains information that meets the definitions below, use the corresponding stage.

1. Definition of Primary Tumor (T)

• - TX: Primary tumor cannot be assessed

  - T0: No evidence of primary tumor

  - Tis: Carcinoma in situ, intramucosal carcinoma (involvement of lamina propria with no extension through muscularis mucosae)

  - T1: Tumor invades the submucosa (through the muscularis mucosa but not into the muscularis propria)

  - T2: Tumor invades the muscularis propria

  - T3: Tumor invades through the muscularis propria into pericolorectal tissues

  - T4: Tumor invades^* the visceral peritoneum or invades or adheres^** to adjacent organ or structure

  - T4a: Tumor invades^* through the visceral peritoneum (including gross perforation of the bowel through tumor and continuous invasion of tumor through areas of inflammation to the surface of the visceral peritoneum)

  - T4b: Tumor directly invades^* or adheres^** to adjacent organs or structures

^*Direct invasion in T4 includes invasion of other organs or other segments of the colorectum as a result of direct extension through the serosa, as confirmed on microscopic examination (for example, invasion of the sigmoid colon by a carcinoma of the cecum) or, for cancers in a retroperitoneal or sub-peritoneal location, direct invasion of other organs or structures by virtue of extension beyond the muscularis propria (i.e., respectively, a tumor on the posterior wall of the descending colon invading the left kidney or lateral abdominal wall; or a mid or distal rectal cancer with invasion of prostate, seminal vesicles, cervix, or vagina).

2. Definition of Regional Lymph Node (N)

• - NX: Regional lymph nodes cannot be assessed

  - N0: No regional lymph node metastasis

  - N1: One to three regional lymph nodes are positive (tumor in lymph nodes measuring ≥0.2 mm), or any number of tumor deposits are present and all identifiable lymph nodes are negative

  - N1a: One regional lymph node is positive

  - N1b: Two or three regional lymph nodes are positive

  - N1c: No regional lymph nodes are positive, but there are tumor deposits in the subserosa, mesentery or nonperitonealized pericolic, or perirectal/mesorectal tissues.

  - N2: Four or more regional nodes are positive

  - N2a: Four to six regional lymph nodes are positive

  - N2b: Seven or more regional lymph nodes are positive

  - If the non-regional lymph node is positive, it is judged as M1 rather than N1 or N2.

3. Definition of Regional Lymph Nodes

Regional nodes are located 1) along the course of the major vessels supplying the colon and rectum, 2) along the vascular arcades of the marginal artery, and 3) adjacent to the colon—that is, along the mesocolic borders of the colon. Specifically, the regional lymph nodes are termed pericolic and perirectal/mesorectal and also are found along the ileocolic, right colic, middle colic, left colic, inferior mesenteric, superior rectal (hemorrhoidal), and internal iliac arteries.

The regional lymph nodes for each segment of the large bowel are designated as follows:

• - Cecum: Pericolic, ileocolic, right colic

  - Ascending colon: Pericolic, ileocolic, right colic, right branch of the middle colic

  - Hepatic flexure: Pericolic, ileocolic, right colic, middle colic

  - Transverse colon: Pericolic, middle colic

  - Splenic flexure: Pericolic, middle colic, left colic

  - Descending colon: Pericolic, left colic, sigmoid, inferior mesenteric

  - Sigmoid colon: Pericolic, sigmoid, superior rectal (hemorrhoidal), inferior mesenteric

  - Rectosigmoid: Pericolic, sigmoid, superior rectal (hemorrhoidal), inferior mesenteric

  - Rectum: Mesorectal, superior rectal (hemorrhoidal), inferior mesenteric, internal iliac, inferior rectal (hemorrhoidal)

4. Definition of Distant Metastasis (M)

• - M0: No distant metastasis by imaging, etc.; no evidence of tumor in distant sites or organs (This category is not assigned by pathologists.)

  - M1: Metastasis to one or more distant sites or organs or peritoneal metastasis is identified

  - M1a: Metastasis to one site or organ is identified without peritoneal metastasis

  - M1b: Metastasis to two or more sites or organs is identified without peritoneal metastasis

  - M1c: Metastasis to the peritoneal surface is identified alone or with other site or organ metastases

  - If a non-regional lymph node is positive, it is also judged as M1.

As a third rule, if the abdominal CT or rectal MRI report does not directly mention TNM stage or contain information that meets the definition given above, but does fit the expression below, use the corresponding stage.

5. Interpretation of multidetector CT (MDCT) for colorectal cancer

• - T1 lesions are depicted with intraluminal projection of the tumor without any distortion of the bowel wall layers.

  - T2 lesions present with asymmetric thickening projecting intraluminally but with preservation of the smooth muscle coat and clear adjacent pericolonic fat.

  - T3 lesions infiltrate beyond the muscularis propria, and features identifiable on MDCT include smooth or nodular extension of a discrete mass of tumor tissue beyond the contour of the bowel wall with extension into pericolic fat.

  - T4 lesions include tumors that penetrate the surface of the visceral peritoneum, directly invade or are adherent to other organs or structures, or tumors where there is evidence of perforation. The identification of peritoneal infiltration has been used to classify patients as high-risk by recent groups.

  - MDCT has limited accuracy for the detection of nodal stage, largely due to its inability to detect micrometastases. The criterion for a nodal mass is 1 cm or a group of three or more nodes is used for positivity. Accuracy rates are still poor and range from 60% to 80%. Some advocate the use of enhancement characteristics and suggest an enhancement of .100 HU to represent malignancy.

  - Extramural venous invasion as an independent prognostic factor has been evaluated by MDCT and classified. The appearances of nodular spread into small vessels or definite enhancing tumor spread along a large vein are considered to define positivity.

6. Interpretation of rectal MRI for rectal cancer

The important distinction of a T3 lesion is based on the presence of the tumor signal extending into the perirectal fat. This broad-based or pushing nodular configuration of an advancing T3 tumor margin is in continuity with the intramural portion of the tumor. This needs to be distinguished from spiculation associated with the lower signal intensity of fibrosis in order to avoid the over-staging of T2 tumors.

T4 lesions involve invasion into an adjacent organ or structure and those that have perforated the peritoneum. Anterior invasion into the bladder or uterus, lateral extension to the pelvic sidewall and posterior invasion to the sacrum should be evaluated. If the tumor is in the lower third of rectum, infiltration of the pelvic floor structures needs to be determined, although invasion of the levator alone does not constitute T4 disease.

Extramural vascular invasion is more easily identified on MRI for rectal tumors than MDCT for colonic tumors. As the tumor grows into or along a recognizable vessel, this vessel should be identified (e.g. superior rectal vein) and whether it threatens the mesorectal fascia (within 1 mm of the fascia).

Lymph node staging with MRI is not solely reliant on the size of the nodes. Criteria based on the outline of the node and features of signal intensity are more reliable. The use of high-resolution imaging can help classification. Lymph nodes with a smooth outline and uniform signal intensity are considered to be benign.

More suspicious nodes have irregular outlines and mixed signal intensities. The distinction between N1 and N2 is numerical: N1 includes one to three nodes, N2 includes four or more nodes. Any lymph node within 1 mm of the circumferential resection margin (CRM) should also be documented and also evaluated for malignant potential. Similarly, the location and size of pelvic sidewall lymph nodes and whether they have any suspicious features should be noted.

The CRM is formed by the mesorectal fascia, which envelops the mesorectum and constitutes the dissection plane in TME. Measurements are taken of the distance of the tumor to the mesorectal fascia, the potential CRM. If the tumor lies within 1 mm of the mesorectal fascia, then this is a potentially positive margin For these low rectal tumors at or below the puborectalis sling, comment should be made whether there is full thickness invasion of muscularis propria, invasion into the intersphincteric plane or external sphincter.

Task: Take the report below and extract the lesion location and its TNM stage using the format follow as:

• 1. Location: (nouns of a few words, e.g. mid ascending colon, upper rectum)

  2. T (number or letter) N (number or letter) M (number or letter)

  3. Reason for extracting TNM stage as above

(Imaging reports)