Generative artificial intelligence for automated data extraction from unstructured medical text

Abstract

Objectives

Unstructured data, such as procedure notes, contain valuable medical information that is frequently underutilized due to the labor-intensive nature of data extraction. This study aims to develop a generative artificial intelligence (GenAI) pipeline using an open-source Large Language Model (LLM) with built-in guardrails and a retry mechanism to extract data from unstructured right heart catheterization (RHC) notes while minimizing errors, including hallucinations.

Materials and Methods

A total of 220 RHC notes were randomly selected for pipeline development and 200 for validation from the Pulmonary Vascular Disease Registry. The pipeline comprised three main components: the Engineered Preload Framework (EPF), which integrated schemas and instructions; the LLM module, enhanced by reasoning capabilities; and the validation and retry mechanism, which ensured data accuracy through iterative self-correction. A clinical expert manually extracted data from the validation cohort to establish the ground truth. Pipeline performance was evaluated using precision, recall, and F1 score. Additionally, the dataset was stratified into quartiles to assess the pipeline’s ability to handle varying levels of data availability.

Results

The pipeline achieved 99.0% precision, 85.0% recall, and a 91.5% F1 score, with an overall accuracy of 90% when evaluated at the note level. The most common error was missed values (5.2%), while hallucinations were the least frequent (<0.01%).

Discussion and Conclusion

This study demonstrates the feasibility of a robust GenAI pipeline for automating structured data extraction from unstructured RHC procedure notes. The approach highlights the potential of LLMs in medical data mining, improving research efficiency and clinical applications.

Background and significant

In medicine, up to 80% of information is embedded in unstructured, free-text formats such as clinical notes, procedure reports, and radiology records.¹ Traditionally, this data has been underutilized due to the labor-intensive nature of extraction.^2,3 This presents a critical barrier to harnessing existing medical records for research and clinical care.

Previous efforts in automating unstructured data extraction primarily focused on using Natural Language Processing (NLP) to mine clinical text.^4,5 However, NLP faces several challenges—from time-consuming annotation of large datasets by multiple clinicians,⁶ to variations in linguistic expressions in data.⁵

In June 2017, the Transformer architecture was introduced through “Attention is All You Need,”⁷ paving the way for the development of Large Language Models (LLM), notably ChatGPT in 2022.⁸ These LLMs have excelled in a number of medical domains, from achieving impressive scores on the United States Medical Licensing Examination (USMLE),⁹ to demonstrating proficiency in diagnostic reasoning.¹⁰ However, their limitations in medical knowledge and the risk of hallucinations—where the model generates fabricated yet seemingly plausible information—raise concerns regarding their application in healthcare.^2,11

Recently, several studies have shown promising results using LLM, primarily ChatGPT, to extract unstructured data from electronic medical records (EMR).^2,12,13 Prior Artificial Intelligence (AI) research in respiratory medicine has focused on NLP and machine learning to categorize disease severity, risk factors, and disease processes or predict exacerbation of underlining diseases.^14–18 However, no study to date has explored LLM-based extraction of right heart catheterization (RHC) data from free-text procedure notes.

Objective

This investigation aims to develop a Generative Artificial Intelligence (GenAI) pipeline using an open-source LLM, Llama 70b,¹⁹ with built-in guardrails for unstructured data extraction. Our primary objective is to automate the extraction of hemodynamic data from unstructured RHC procedure notes while ensuring high accuracy, efficiency, and error minimization, including hallucination reduction.

Materials and methods

Data source

The RHC notes were obtained from the Research Patient Data Registry (RPDR), a centralized clinical data warehouse for the Mass General Brigham (MGB) system.^20,21 Patients with at least one RHC performed in the MGB system were identified using the Pulmonary Vascular Disease Registry.²² Institutional Review Board (IRB) approval was obtained for our study.

A total of 420 notes were randomly selected: 220 for pipeline development and 200 for the validation cohort. The sample size was chosen based on a previous publication by Tam et al., in which they proposed a 100-130 sample size for LLM applications after reviewing 142 studies.²³ We chose a sample size of 200 notes to exceed the minimum recommendation and ensure robustness of the pipeline.

The development dataset was leveraged to build and refine the pipeline, while the validation dataset was used to assess the application’s performance. There was no overlap in the two cohorts to prevent data leak.

The selected notes varied in structure, were authored by multiple providers over a 15-year period, and contained diverse linguistic styles. Some notes also included additional cardiac procedures (eg, coronary angiogram, transcatheter aortic valve replacement), which were retained to reflect real-world data complexity.

Preprocessing protocol

To reduce computational load and enhance pipeline performance, we implemented an automated preprocessing protocol using rule-based methods.²⁴ A PVD expert guided the process by reviewing randomly selected RHC notes to identify common layouts, abbreviations, and documentation conventions throughout iterative development. While the notes lacked a uniform structure, they shared some similar labelling and phrasing, which the preprocessing step was designed to target. This protocol removed irrelevant sections (eg, such as medication lists and nursing notes) and standardized varied expressions (eg, “PA 40/25/30” was converted to “PASP 40, PADP 25, mPAP 30”) while preserving all essential information.

Although it was not feasible to standardize all the notes into one common format given their diversity, the protocol provided partial data harmonization to improve pipeline performance while allowing the model to handle the edge cases. Examples of the procedure notes are provided in Figure 1 and in the supplement (Section 2) to illustrate content complexity and diversity.

Figure 1.

An abbreviated example of RHC note.

JSON schema

To structure the desired output, a JSON schema was developed. This schema incorporated two key identifiers: the Medical Record Number (MRN) and the Report Number. It also accounted for three possible conditions under which the right heart catheterization (RHC) could be performed: room air, oxygen supplementation, and nitric oxide supplementation. For each of these conditions, the schema included 13 unique conditional data fields. This comprehensive structure allowed for the extraction of up to 39 numerical values from a single note, ensuring a thorough capture of relevant information from the RHC procedure. The complete schemas are shown in the supplement (Section 4).

Prompt template

A fundamental aspect of developing LLM applications is the construction of an effective prompt template. The initial prompt was crafted using past data extraction experience and domain expertise in PVD, combined with common prompting techniques including meta prompting, personification, and self-reflection, as these methods have been shown to enhance performance by directing the LLM’s attention and reasoning capabilities.^25,26

Iterative experiments were conducted using approximately 30 RHC notes from the development dataset to better understand the model’s behavior and knowledge, and the outputs were reviewed for accuracy and deficiencies. Due to suboptimal initial results, the “Working Space” concept was introduced to explicitly incorporate the Chain-of-Thought technique, which significantly improved context detection in Step 1.²⁷ After each prompt adjustment, the omission, misattribution, and hallucination rates were obtained. Once the performance reached a satisfactory level and the improvement curve plateaued, the experiments were further expanded to the remainder of the development dataset. The focus shifted from achieving the highest accuracy to identifying major or recurring issues. The “Working Space” was analyzed in these cases to identify gaps in knowledge or process, then the prompt was adjusted to steer the model behavior and reasoning. Overall, the prompt for each step was iterated approximately 50 times to reach the final version. Examples of the prompts are provided in the supplement (Section 3).

“Working space”—reasoning scratchpad

During development, we noticed that free-text hints indicating RHC testing conditions were frequently embedded within other text and often referenced preceding or succeeding contextual information. This complexity made it challenging for the human reviewers to discern relevant details.

To improve the performance of Step 1, a novel approach called the “Working Space” was introduced, inspired by physician workflows and OpenAI’s reasoning model, o1.²⁸ Through prompt engineering and additional computation, the LLM was guided to perform tasks in a structured order: think, plan, and then execute. This logical framework enabled the LLM to systematically address each problem, effectively linking the data to the appropriate testing conditions despite their obscurity. The Working Space output was studied throughout development to detect reasoning errors and limitations. This was instrumental in refining the prompts to enhance the LLM’s reasoning process. An example of the “Working Space” output is included in Section 5 of the Supplement.

GenAI pipeline

The extraction pipeline consisted of two main steps. Step 1 performed a multiclass classification task to categorize text from the original procedure notes into the corresponding testing conditions (Room Air, Oxygen Supplementation, and Nitric Oxide Supplementation). Step 2 extracted numerical values into appropriate data fields. Each step comprised three key components: Engineered Preload Framework (EPF), the LLM, and the validation procedure with the retry loop mechanism. The configurations of these components, however, were tailored to the specific requirements of each step. Figure 2 provides a visual overview of the entire pipeline structure, illustrating the seamless integration of these components and steps.

Figure 2.

Overview of the GenAI pipeline for RHC data extraction. The pipeline consists of three main components: (1) the EPF, (2) the LLM, and (3) the validation and retry mechanism.

In both steps, schemas, preprocessed notes, and prompts were integrated within the EPFs and sent to the LLM server through API calls. To enhance efficiency and accuracy, a validation framework with built-in guardrails was implemented to automatically assess LLM output. The first layer, JSON validation, parsed the raw LLM output into the designated JSON format, ensuring that the dataset was structured and ready for downstream tasks. Pydantic was then employed to enforce compliance with predefined variables and data types.²⁹ The Step 1 Pydantic module verified that the categorized text aligned with the correct testing condition, while the Step 2 module focused on identifying physiologically invalid values, often resulting from typographical errors. Thus, Pydantic validation safeguarded against misformatted results and prevented invalid outputs. At the final stage of Step 2, the Source-Text Validation module ensured an alignment between the LLM outputs and the original notes through cross-referencing all extracted values with the original text.

To simulate human workflow, a retry loop mechanism was implemented, identifying validation errors at any stage and providing tailored feedback to improve LLM performance. This approach enabled the LLM to self-improve by reprocessing notes with an understanding of its prior errors, mimicking human learning. Providing feedback during inference—whether from an external source or the LLM itself—has shown promising results.^30–32 Based on observations during development, the maximum number of retry attempts was set to two, as most errors were typically resolved within two iterations.

The outputs were accessed over multiple runs to ensure consistent generation. While identical outputs were not observed or expected, the overall performance was similar. The final pipeline output was evaluated by a Pulmonary Vascular Disease (PVD) expert through direct comparison with the original notes. Figure 3 provides a visual representation of the data flow through the development and validation of the pipeline.

Figure 3.

Dataflow during development and validation.

During the initial development phase, the pipeline only consisted of one step in which the model attempted to identify the testing condition and extract numerical data simultaneously. When the pipeline was challenged with complex reports in which the procedural conditions were not obvious or intertwined with each other, it often made mistakes and classified the wrong testing condition despite prompt engineering; therefore, a decision was made to serialize the tasks. This has significantly improved the accuracy of the pipeline due to the reduction of abstracted layers.

LLM server

The Llama 3.1 Nemotron 70B model and Llama 3 70B were selected for Step 1 and Step 2, respectively, after conducting initial tests to assess their ability to follow instructions and understand basic medical knowledge.^19,33 Llama 3.1 Nemotron 70B provided more verbose output, which was more suitable for reasoning, while Llama 3 70B was more conservative in its token usage, which helped reduce computational cost.

The model was configured with a temperature setting of 0.1 to prioritize explicitly documented information over “creative” generation and to decrease output randomness. A top-P value of .85 was also applied to support deterministic responses while balancing a degree of token diversity to enhance reasoning output.

The context window was set to 8,196 tokens, based on the average number of input and output tokens with an additional 30% buffer for headroom. A fixed seed value of 1234 was used to ensure repeatability. The batch size was set to one note, and parallel processing was employed to handle multiple batches concurrently.

The LLM was deployed using vLLM³⁴ on a local Graphics Processing Unit (GPU) server. To ensure HIPAA compliance, the data were processed within this local environment. The pipeline was developed using Python 3.10³⁵ and statistical analysis was performed using RStudio 2024.09.1.³⁶

Statistical analyses

Precision, recall, and the F1 score were used to evaluate the pipeline performance. The unit of calculation was a note-level datapoint, defined as a note in which all extractable variables were correctly identified. For example, if a note contains 10 extractable numerical values but the pipeline correctly extracted only 9, the note-level performance would be classified as inaccurate.

To establish ground truth for comparison, a PVD expert manually extracted numerical values from all the validation cohort notes. This process took approximately 7 to 8 hours to review all 200 notes. In addition to note-level evaluation, value-level performance analyses were conducted, where achieving 9 out of 10 correct extractions corresponded to 90% accuracy. Detailed results of these analyses are provided in Section 1 of the Supplement.

To assess how the pipeline handled varying levels of extractable data, the notes were categorized into quartiles based on the distribution of numerical values. The first quartile, containing notes with only descriptive values (eg, reported as ‘normal’) and no numerical data, was intentionally retained to evaluate the pipeline’s ability to deal with hallucinations in the absence of relevant data.

To evaluate reproducibility, the pipeline was executed 10 times on the validation dataset,³⁷ and Krippendorff’s alpha was calculated on extractable fields with at least 10 notes containing non-null and variable numerical values in order to meaningfully assess inter-run agreement.^38,39 Raw consistency rates were also computed to demonstrate the reproducibility regardless of null data or numerical values availability.

Results

From the original validation cohort of 200 notes, 191 were included in the final analysis, while 9 were excluded due to errors in the LLM output detected by the pipeline (Figure 3). In Step 1 (Categorization), errors were detected in four notes, all due to misplacement. The retry loop mechanism successfully corrected two of these four errors, allowing the affected notes to proceed to Step 2.

In Step 2 (Extraction), errors were detected in 11 notes. The retry loop mechanism successfully corrected four of these errors upon reprocessing. Notably, the LLM self-corrected only one out of four hallucinations, as it continued to generate the same incorrect values despite two retry attempts. For validation errors due to out-of-range values, the LLM successfully adjusted its outputs within the acceptable range in 50% of cases after receiving feedback.

Overall, as shown in Figure 3, the validation layers detected 15 notes with errors across both steps in the initial LLM output, successfully correcting 6 of them. The detailed results of the retry loop mechanism are provided in Table 1.

Table 1.

Assessment of retry loop mechanism.

Step 1 categorization
Types of errors	Number of notes	Details of errors	Status upon retry
Misplacement	4	LLM misattributed the testing conditions.	2 out of 4 were fixed.

Step 2 extraction
Types of errors	Number of notes	Details of errors	Status upon retry
Hallucination	4	LLM fabricated values that weren’t in the original text.	1 out of 4 was fixed.
Incorrect data type	1	LLM produced the wrong data type.	Resolved.
Pipeline restriction	4	LLM correctly extracted values, but they were out of acceptable range.	2 out of 4 were fixed.
Pipeline error	1	LLM correctly extracted values but rejected by validation mechanism.	Note was excluded from the final output.
Source-text error	1	Typos in the source text led to an incorrect extraction.	LLM hallucinated a new value upon retry, which was rejected.

Among the 191 analyzed notes, 172 were completely accurate, while 19 contained errors (Figure 4), demonstrating an overall accuracy of 90% at the note level. The most common error was missed values (10 notes or 5.2%), while the least common error was hallucination (1 note or <0.01%). On average, affected notes had two missed fields, with a maximum of three per note. Additionally, seven notes (3.66%) contained misplaced values in the final output, with a maximum of one misplaced field per affected note.

Figure 4.

Details of pipeline performance by category.

Pipeline performance metrics

The pipeline achieved a precision of 99% for correctly identifying values and a recall of 85% for collecting available values. The F1 score of 91.5% reflected a balance between obtaining relevant data and minimizing false positives (Table 2). These metrics demonstrate the pipeline’s strong performance in accurately categorizing and extracting data from RHC notes.

Table 2.

Precision, recall and F1 score of the validation cohort.

Type of statistics	Percentage
Precision	99.0 %
Recall	85.0%
F1 Score	91.5%

Additionally, the pipeline demonstrated strong reproducibility, with Krippendorff’s alpha values for all fields falling within the reliable range (0.80-0.98), except for one field, Room Air TPG, which yielded a tentative reliability value (α = 0.76). Detailed field-level alpha values and raw consistency rates are provided in Section 1 of the Supplement.

Performance analysis by quartiles

The pipeline’s performance was analyzed by dividing the validation cohort into quartiles based on the availability of numerical values. The first quartile contained notes with the fewest quantitative data, while the fourth quartile included those with the highest numerical data (Figure 5). In cases where no relevant numerical data were present, the pipeline did not generate any fabricated results, achieving an accuracy of 100% in the first quartile. The most common error occurred in the fourth quartile, where 17% of notes contained missed values. The most frequently missed field was Pulmonary Vascular Resistance (measured in Wood units), while the most commonly misplaced field was Right Ventricular Diastolic Pressure. These findings highlight the pipeline’s ability to maintain high accuracy even when processing notes containing limited quantitative data.

Figure 5.

Performance comparison across quartiles.

Discussion

This study explored the implementation of a GenAI pipeline with built-in checks and balances to automate data extraction from unstructured RHC procedure notes while minimizing errors. The pipeline correctly extracted unstructured data from 172 of 191 notes, achieving 90% accuracy, 99% precision, an F1 score of 91.5%, and 85% recall. This bias towards prioritizing true positive data over false negative and false positive was intentional, designed to maximize clinician confidence in the pipeline. This also explains the higher missing rate in the fourth quartile of Figure 5, as more numerical values became available. Notably, these results were achieved utilizing zero-shot learning without the need for few-shot learning or fine-tuning the LLM, which would have introduced significant computational expense, technical complexity, and time constraints.⁴⁰

The pipeline demonstrated the ability to handle diverse textual content in RHC procedure notes, which exhibited significant structural and linguistic differences between providers and clinical indications. The LLM’s performance can be attributed to its ability to grasp semantic meaning, enhanced by well-curated prompts, rather than merely identifying patterns or associations. The “Working Space” was a key innovation in enhancing the LLM's capacity for thinking and reasoning, allowing for auditing the LLM’s “thinking” process and identifying gaps in knowledge and reasoning. By making the LLM’s inner workings less opaque, this method reduced user uncertainty about how answers were derived, addressing doubts stemming from its decision-making process. Notably, while prompting techniques were a crucial scaffolding to achieve our results, the domain-specific knowledge required to construct them was derived from clinical expertise in pulmonary vascular disease (PVD). This highlighted the intersection of machine learning and medical knowledge in designing AI applications for healthcare.

The pipeline design was inspired by the physician workflow: the EPF serves as the instruction and template, the LLM functions as the reasoning engine, and the validation layers with a retry mechanism ensure quality assurance. Both the EPF and post-processing validation were designed to be customizable, enabling the adaptation of other note types without a complete pipeline redesign. Instead, leveraging clinical expertise, these components can be tailored to meet different objectives within a reasonable timeframe.

This approach offers an advancement over previous work. Sushil et al. have experimented with three different LLMs (GPT-4, GPT-3.5-turbo, and FLAN-UL2) using zero-shot learning to extract oncological data from 40 real-world clinical progress notes. GPT-4 demonstrated the best overall performance with an average accuracy of 68% compared to expert evaluation.¹² In contrast, our approach—utilizing an open-source model on a local server—achieved a higher accuracy of 90%.

Another study assessed the performance of GPT-4 compared to text string search in identifying the helmet status of patients visiting the ED for injuries related to bicycles, hoverboards, and powered scooters. GPT-4 was prompted through chat sessions and provided with a file containing unstructured text. Although the study highlighted LLMs’ potential to expedite unstructured data extraction, the authors acknowledged concerns regarding reliability and hallucinations, which remain barriers to real-world implementation.²

Recognizing the skepticism surrounding the use of LLMs in healthcare, our approach incorporated multiple mechanisms to minimize hallucinations. This effort proved effective, with only one hallucination occurring in the final output. The study further evaluated the inter-run reliability of the pipeline across multiple executions on the same validation dataset and demonstrated good reproducibility in all fields except one. Some minor variability was expected due to the non-deterministic nature of LLMs and the inherent ambiguity of multiple correct interpretations for the same field in the original notes.

This study utilized Llama 70B models that have demonstrated performance comparable to other closed-source frontier models across multiple benchmarks.⁴¹ Deploying a local GenAI pipeline ensured reliable and consistent performance, mitigating concerns about unexpected changes in LLM behaviors over time.^2,42,43 It also allowed us to contain all data processing within a local environment to protect privacy. This is crucial, as even de-identified data is not synonymous with anonymized data and could potentially be used to trace back to individuals.⁴⁴

Limitations

Our approach has several limitations. The pipeline has only been tested on a subset of RHC data from MGB, which may differ from other centers, potentially limiting its generalizability. However, the dataset exhibited considerable diversity due to multiple hospitals within MGB, where clinicians employ varied documentation styles, evolving structures, and diverse linguistic expressions. Only one PVD expert was employed to establish the ground truth, which introduces the possibility of single-rater bias and limits assessment of annotation reliability. Future studies should consider including two expert evaluators and a designated adjudicator to resolve any discrepancies in data interpretation.

Despite incorporating feedback, the pipeline did not always resolve the errors, leading to the elimination of some notes from the final output. While this only represented a small fraction of the data, it highlights the need for future improvement. Lastly, although the preprocessing protocol played an important role in enhancing pipeline performance, its dependence on institution-specific customization may pose challenges for broader implementation across diverse healthcare systems.

Conclusion

We have developed a novel and reliable GenAI pipeline that automates the data extraction process from unstructured RHC procedure notes, transforming them into a structured and computable dataset. The pipeline’s modular design allows for adaptation to extract data from other types of free-text notes. This study underscores the potential of LLM applications in healthcare, demonstrating their capacity to accelerate discoveries, reduce repetitive tasks, and enhance care delivery. By streamlining the extraction of valuable information from clinical notes, this technology has the potential to improve research efficiency and clinical decision-making, ultimately contributing to better patient outcomes.

Author contributions

Nam Dao (Conceptualization, Data curation, Formal analysis, Investigation, Methodology, Resources, Software, Visualization, Writing—original draft, Writing—review & editing), Luisa Quesada Arias (Investigation, Writing—review & editing), Syed Moin Hassan (Writing—review & editing), Monica Iturrioz Campo (Writing—review & editing), Shelsey Johnson (Writing—review & editing), Suchandra Ghose (Data curation, Resources), Raúl San José Estépar (Methodology, Writing—review & editing), Aaron Waxman (Writing—review & editing), George R Washko (Funding acquisition, Investigation, Resources, Supervision, Writing—review & editing), and Farbod N. Rahaghi (Conceptualization, Funding acquisition, Investigation, Methodology, Resources, Supervision, Validation, Writing—review & editing)

Supplementary material

Supplementary material is available at JAMIA Open online.

Funding

Nam Dao was supported by the National Institute of Health T32 program, grant number GR0125656.

Syed Moin Hassan was supported by a grant from the National Heart, Lung, and Blood Institute (NHLBI), grant number R01HL164717.

Raúl San José Estépar was supported by grants from the National Heart, Lung, and Blood Institute (NHLBI), grant numbers 1R01HL149877 and R01HL116473.

George Washko was supported by grants from the National Heart, Lung, and Blood Institute (NHLBI), grant numbers R01HL116473 and R01HL122464.

Farbod Rahaghi was supported by a grant from the National Heart, Lung, and Blood Institute (NHLBI), grant number R01HL164717.

Authors: Luisa Quesada, Monica Iturrioz Campo, Shelsey Johnson, Suchandra Ghose, Aaron Waxman reported no specific grant from any funding agency in the public, commercial, or not-for-profit sectors.

Conflicts of interest

All authors report no conflict of interest.

Outside the submitted work: Raúl San José Estépar is a founder and stockholder of Quantitative Imaging Solutions, has received investigator-initiated grants from Gossamer Bio and Boehringer Ingelheim, and serves on the Scientific Advisory Board of the CRIS Cancer Foundation. George Washko is a founder and stockholder of Quantitative Imaging Solutions Founder and stockholder.

Data availability

The data underlying this article cannot be shared publicly due to privacy concerns related to the individuals included in the study. The data may be shared upon reasonable request to the corresponding author.