A Comparison of a Large Language Model vs Manual Chart Review for the Extraction of Data Elements From the Electronic Health Record

Large language models (LLMs) hold tremendous potential for accelerating clinical research and augmenting clinical care. One of the most promising LLM use cases is natural language processing (NLP) and the extraction of elements from unstructured text, for which LLMs may be superior to existing NLP software packages.^1,2 Due to its standardized Liver Imaging Reporting and Data System (LI-RADS),³ hepatocellular carcinoma (HCC) imaging provides an ideal test case for LLM-enabled NLP data extraction from unstructured text. We sought to assess the performance of a general-purpose LLM, permitted to be used with protected health information (PHI), vs human manual chart review in extracting 8 distinct data elements from HCC imaging reports.

The detailed methods are provided in the Supplementary Materials. In brief, we used the Versa generative pretrained transformer (GPT) 4 application programming interface (API), the PHI-compliant version of Microsoft Azure OpenAI GPT-4 LLM implemented at the University of California, San Francisco (UCSF), for this study.⁴ Versa GPT-4 has a token limit of 8192 tokens, defined as the unit used to compute text length.⁴ We manually reviewed 1101 longitudinal pre– and post–locoregional treatment (LRT) computed tomography or magnetic resonance imaging abdomen imaging reports for 753 patients with HCC enrolled in the Functional Assessment in Liver Transplantation (FrAILT) study at UCSF.⁵ The imaging may have been obtained at any time during the patients’ clinical course (before or after HCC diagnosis) and may not necessarily include HCC lesions. We manually tagged reports for 8 distinct data elements: (1) maximum LI-RADS for HCC lesions, (2) number of HCC lesions, (3) diameter (in centimeters) of the largest lesion, (4) sum of diameters (in centimeters) of all HCC lesions, (5) presence/absence of macrovascular invasion, (6) presence/absence of extrahepatic metastases, (7) presence/absence of previous LRT, and (8) study adequacy. The manual tagging process took about 1 to 2 minutes per record for an aggregate time of ~28 hours.

All 1101 imaging reports were trimmed to include only findings and impressions. We iteratively developed a fewshot prompt with testing on the first 5 records by alphabetical order of coded identifiers (Supplementary Figure 1).⁶ Because snowballing of data passed per prompt often led to execution failure from exceeding GPT4’s token limit, we ran 56 loops of the final prompt with 20 records per loop to process all 1101 records. The execution of the loop extraction code took ~2 hours.

We evaluated the performance of Versa GPT-4 vs manual chart review by calculating the accuracy, precision, recall/sensitivity, specificity, and F1 (harmonic mean of precision and recall) for each of the 8 data elements. For multilevel classifications (maximum LI-RADS, number of HCC lesions, diameter of the largest lesion, and sum of tumor diameters), we calculated weighted-average precision, recall/sensitivity, and F1.⁷ For binary classifications (macrovascular invasion, extrahepatic metastases, previous LRT, and study adequacy), we defined the presence of these features as positive cases for precision, recall/sensitivity, and F1. We treated a manual chart review value of 0 as the only negative class and calculated binary specificities. We estimated 95% confidence intervals through bootstrapping with 2000 iterations. All analyses were conducted in R, version 4.3.1 Beagle Scouts (R Core Team).⁸ This study was approved by the UCSF institutional review board in study no. 11–07513.

Performance metrics for the 8 data elements extracted by Versa GPT-4 compared to manual chart review are shown in Table 1. Overall accuracy was 0.934 (95% confidence interval, 0.928–0.939), with accuracies for specific data elements varying between 0.886 for sum of tumor diameters to 0.989 for extrahepatic metastases. In general, accuracy was higher for classification tasks (maximum LIRADS, macrovascular invasion, extrahepatic metastases, previous LRT, and study adequacy) vs comparison (maximum tumor diameter) or summation (number of tumors and sum of tumor diameters). Precision, recall, and F1 statistics for elements with multilevel classifications (maximum LI-RADS, number of tumors, maximum tumor diameter, and sum of tumor diameters) were calculated as weighted-average values: these values may have been biased because accurate predictions of the absence of imaging features (eg, Versa GPT-4 noting 0 tumors when there were no tumors by manual chart review) were included in the statistics. In general, Versa GPT-4 was highly specific, and overall accuracy was higher in normal reports, eg, 0.963 for those with no LR4 or LR5 findings.

Table 1.

Performance Evaluation Statistics of Versa GPT-4 vs Manual Chart Review

Data element	Accuracy	Binary event ratea	Precision	Recall/sensitivity	Specificityb	F1
Maximum LI-RADS	0.944 (0.928–0.955)	—	0.946 (0.931–0.958)c	0.944 (0.928–0.955)c	0.948 (0.928–0.965)	0.944 (0.930–0.956)c
Number of tumors	0.922 (0.903–0.936)	—	0.924 (0.906–0.938)c	0.922 (0.905–0.937)c	0.938 (0.917–0.956)	0.922 (0.904–0.938)c
Maximum tumor diameter	0.920 (0.902–0.944)	—	0.934 (0.914–0.944)c	0.920 (0.902–0.935)c	0.924 (0.902–0.945)	0.924 (0.907–0.937)c
Sum of tumor diameters	0.886 (0.866–0.903)	—	0.907 (0.881–0.918)c	0.886 (0.866–0.904)c	0.916 (0.894–0.938)	0.892 (0.871–0.908)c
Macrovascular invasion (binary)	0.936 (0.919–0.948)	0.005	0.054 (0.013–0.109)	0.800 (0.333–1.000)	0.936 (0.921–0.950)	0.101 (0.026–0.195)
Extrahepatic metastases (binary)	0.989 (0.980–0.994)	0.006	0.308 (0.071–0.600)	0.571 (0.143–1.000)	0.992 (0.986–0.996)	0.400 (0.125–0.667)
Previous LRT (binary)	0.910 (0.891–0.926)	0.761	0.924 (0.907–0.942)	0.961 (0.947–0.973)	0.750 (0.695–0.802)	0.942 (0.931–0.953)
Inadequate study (binary)	0.965 (0.952–0.974)	0.052	0.655 (0.528–0.780)	0.667 (0.546–0.788)	0.981 (0.973–0.989)	0.661 (0.553–0.755)
Overall accuracy	0.934 (0.928–0.939)	—	—	—	—	—

NOTE. Values in parentheses are 95% confidence intervals.

^aIf applicable.

^bSpecificity is calculated as a binary statistic with the assumption that values of 0 represent negative cases.

^cWeighted statistics given because these were multilevel classifications.

This is one of the first studies that compared the performance of the general purpose GPT-4 LLM, implemented in a PHI-compliant fashion, vs manual chart review for the extraction of clinical data. We showed high accuracy for simple extraction tasks, which degraded with more complex use cases but still showed reasonable accuracy (88.6%). For this specific demonstration use case of data extraction from HCC imaging reports, one immediate potential clinical application is automated determination of tumor burden for transplant eligibility. Potential applications resulting from the broader technology of LLM-enabled NLP via API integration include (1) text-based prediction modeling, (2) information retrieval from clinical records via a question-andanswer format, (3) augmented clinical decision support through active surveillance of clinical documentation, and (4) patient- and provider-facing chatbots. The future integration of LLMs for clinical uses will require the development of governance structures for the safe and ethical use of generative artificial intelligence tools. Moreover, further clarity on liability and whether these tools would be considered regulated medical devices is also needed.⁹

This specific study has multiple limitations. First, we used manual chart review as the gold standard comparison, but there could be human data extraction errors biasing the results. Second, general concerns remain around accuracy and algorithmic bias regarding using the pretrained GPT-4 model for clinical tasks despite prompt engineering. Finally, our institution is one of the few that has PHI-compliant LLMs implemented at this time. This study could not and should not be conducted on publicly available consumer-facing versions of GPT-4 due to patient privacy/security concerns regarding LLM use.¹⁰ As such, the potential clinical applications that we envision can only be possible with the availability of PHI-compliant LLMs. Despite these limitations, our study demonstrated 2 concepts: (1) the feasibility of using general purpose LLMs for data extraction and (2) the use of an LLM deployed in an isolated protected environment that accommodates PHI (as opposed to ChatGPT) for clinical use cases.

Abbreviations used in this paper:

API

application programming interface

GPT

generative pretrained transformer

HCC

hepatocellular carcinoma

LI-RADS

Liver Imaging Reporting and Data System

LLM

large language model

LRT

locoregional treatment

NLP

natural language processing

PHI

protected health information

UCSF

University of California, San Francisco

Data Availability

Aggregate data produced in the present study may be available upon reasonable request to and with approval by the authors.