Integrating Rule-Based NLP and Large Language Models for Statin Information Extraction from Clinical Notes

Abstract

Background:

Identifying patient-specific barriers to statin therapy, such as intolerance or deferral, from clinical notes is a major challenge for improving cardiovascular care. Automating this process could enable targeted interventions and improve clinical decision support (CDS).

Objective:

To develop and evaluate a novel hybrid artificial intelligence (AI) framework for accurately and efficiently extracting information on statin therapy barriers from large volumes of clinical notes.

Methods:

The hybrid AI framework consisted of a rule-based natural language processing (NLP) filter, an LLM-based refinement filter, and an LLM-based multi-category classifier. The framework was developed on 2,000 clinical notes and then retrospectively applied to a dataset of 197,761 notes from 47,192 patients at an academic medical center. Performance was evaluated against manual chart review for classifying statin intolerance, contraindications, and patient deferral.

Results:

The framework was efficient, with the initial filter removing over 77% of irrelevant notes while achieving a recall of 1.0 to ensure no relevant information was lost. The final classifier accurately categorized patient-level barriers with high F1 scores for intolerance (0.99), contraindications (0.81), and patient deferral (0.86). On the large dataset, the framework identified that 6.4% of patients (n=3,027) had documented intolerance, 0.7% (n=310) had contraindications, and 2.9% (n=1,391) had deferred therapy.

Conclusion:

The hybrid AI framework provides an efficient, scalable, and trustworthy solution for processing clinical notes. It has the potential to enhance clinical decision support (CDS) systems by integrating detailed patient-level insights, improving adherence to clinical guidelines, and reducing provider burden. Future research should focus on implementing CDS tools that leverage extracted information to address barriers to statin therapy and optimize patient outcomes.

INTRODUCTION

Atherosclerotic cardiovascular disease (ASCVD)-related conditions remains a leading cause of morbidity and mortality in United States,[1] with an estimated 24 million adults expected to have coronary heart disease by 2035, at a projected cost of $64 billion a year.[2] As an important part of prevention, lipid-lowering therapy with statins reduces relative risk of cardiovascular events by approximately 30%.[3] Despite their effectiveness, significant challenges in clinical practice persist.[4,5] Adherence to statin therapy remains suboptimal; approximately 40% of patients do not maintain optimal levels of adherence.[6] This adherence gap is often driven by factors such as patient concerns about side effects, particularly statin-associated muscle symptoms (SAMS), which are a leading cause of discontinuation.[7,8] These issues contribute to suboptimal management of ASCVD risk and increased adverse outcomes.

Clinical notes in electronic health records (EHRs) serve as a valuable source of data that contain detailed information about statin therapy, such as patient deferral, intolerance of statins medications, and the shared decision-making process. Free-text documents offer richer information and sometimes include essential details that are absent from structured data. However, manually extracting this information from free-text notes is time-consuming. Reliance on structured data might miss critical information documented in clinical notes, and manual chart reviews are time-consuming. Automating the extraction and analysis of this information offers a scalable approach to improving clinical decision support systems and enhancing patient outcomes.

At Vanderbilt University Medical Center (VUMC), a clinical decision support (CDS) alert is used to notify providers about prescribing statins for eligible patients. However, its acceptance rate is suboptimal,[9] with common override reasons including patient deferral and contraindications. These override reasons are often embedded within textual clinical notes, making it challenging to incorporate them into the alert design. Identifying patients with documented deferral or contraindications to statins could allow for tailored alert strategies, such as setting the default response to “patient deferral” for these individuals or transforming the alert into a passive reminder that minimizes workflow interruptions for healthcare providers. These modifications could improve the usability and clinical relevance of the alerts while reducing unnecessary cognitive burden on healthcare providers.[9] In addition, instead of a disruptive alert at the point of care, a provider could receive a passive inbox message summarizing the patient’s documented history of statin intolerance or deferral. This approach respects clinical autonomy and reduces alert fatigue while still providing valuable context for future decision-making.[10]

Recent advancements in large language models (LLMs) provide opportunities for more sophisticated analysis of textual data.[11–13] However, significant gaps remain. While LLMs excel at text analysis, their high computational cost makes them impractical for screening vast numbers of clinical notes. Furthermore, there is a lack of established, scalable methods to automatically identify and classify nuanced patient-specific barriers to statin therapy—such as intolerance versus deferral—from unstructured text. This gap limits our ability to design and implement targeted CDS interventions that can address the root causes of statin underutilization. To address these challenges, we developed a novel artificial intelligence (AI) framework combining rule-based natural language processing (NLP) and LLM-based methods to extract and classify information about statin use documented in clinical notes. The objective of this study was to develop and evaluate a hybrid AI framework that extracts patient information related to statin therapy, including intolerance, contraindications, and patient deferral. By leveraging this framework, we aim to provide actionable insights to improve alert design and optimize statin therapy, enhancing preventive care for patients at risk of ASCVD.

METHODS

This study was conducted at VUMC. It was reviewed by the Vanderbilt University Institutional Review Board and found to be exempt.

Hybrid AI Framework

We provided a hybrid AI framework specifically designed to extract and analyze patient information about statin medication from clinical notes. The framework comprises three interconnected components. First, we implemented a rule-based natural language processing (NLP) filter as a preliminary step to eliminate irrelevant documents. Second, we created an LLM-based refinement filter to further identify documents likely to contain meaningful information. Finally, we developed an LLM-based multi-category classifier to subdivide the relevant information into categories: intolerance, contraindications, and patient deferral. The overall process of developing and evaluating the AI framework is illustrated in Figure 1.

Figure 1.

Overview of the development and evaluation process of the hybrid AI framework, including the rule-based NLP filter, the LLM-based refinement filter, and the multi-category classifier for analyzing clinical notes.

Data Collection

Two datasets were assembled for this study. Dataset A comprised 2,000 randomly selected clinical notes dated between July 1, 2024, and August 1, 2024, for adult patients who received primary care at VUMC. These notes were manually labeled as relevant or irrelevant based on whether they contained information about statins. Dataset A was used to develop and evaluate the performance of the rule-based NLP filter and the LLM-based refinement filter. Dataset B included all clinical notes dated between August 1, 2024, and September 1, 2024, for adult primary care patients at VUMC. After filtering using the first two components of the framework, the remaining documents in Dataset B were used to train and evaluate the multi-category classifier. These documents were also manually labeled to evaluate the classifier’s accuracy. All notes were in their original format (i.e., plain text and broken down into 1950 characters or less) as stored in the Epic Data Warehouse. All notes contained at least one of the following terms: “%statin%”, “%cholesterol%”, “%ASCVD%”, “%lipid%”, or “%atherosclerotic cardiovascular disease%.”

Development of AI Framework

Dataset A was split into a training set of 1,200 documents and a testing set of 800 documents. The rule-based NLP filter was developed iteratively, with every 100 documents analyzed as a unit to identify patterns in irrelevant content. Rules were refined to maximize the exclusion of irrelevant documents while ensuring no relevant documents were mistakenly filtered out. This iterative process involved manually reviewing any incorrectly excluded documents and adjusting the rules until the filter achieved a recall of 1.0 on the training and testing sets, ensuring its suitability as a safe, conservative first-pass screen.

In addition, an LLM-based filter was developed using the same training set to identify documents with relevant information regarding statins. The performance of each of the two filters was compared to a manually labeled dataset on the testing set, and results such as precision, recall, F1 score, accuracy, and filter-out rate were reported. The classifier was developed using documents that manually labeled as relevant documents in the testing dataset from Dataset A. For each category, we selected typical examples as samples for few-shot learning.

We implemented the LLM filter and classification stages using the GPT-4o model, which features a 128,000-token context window, accessed via the Microsoft Azure OpenAI Service. This service was deployed within our institution’s secure, HIPAA-compliant cloud environment to ensure data privacy. For reproducible and deterministic results crucial to clinical research, the model’s temperature parameter was set to 0. The detailed prompts are available in the Supplementary File.

Retrospective Study

Dataset B was first processed through the rule-based NLP filter and then refined using the LLM-based filter to identify documents with relevant information. Then the identified relevant documents were processed through the LLM-based multi-category classifier. All of these documents were manually labeled to evaluate the LLM-based classifier’s performance, and metrics such as precision, recall, F1 score, and accuracy were reported for each category.

RESULTS

Rule-Based NLP Filter

Two rounds were undertaken to develop the rule-based NLP filter using Dataset A. In the first round, the filter excluded 1,004 irrelevant documents (84%) from the training dataset (n=1200) while retaining all relevant documents, achieving a recall of 1. On the testing dataset (n=800), the filter excluded 637 irrelevant documents (80%) but incorrectly filtered out one relevant document, resulting in a recall of 0.974. After analyzing these results, the filter criteria were adjusted to improve performance. In the second round, the updated filter achieved filter-out rates of 81% in the training dataset and 76% in the testing dataset, with recall values of 1 for both datasets. Detailed metrics for each round, including precision, recall, F1 score, accuracy, and filter-out rates, are presented in Table 1. The rule-based filter comprised 99 rules (see supplementary file), such as differentiating “statin” from “nystatin” or the verb “stating”, filtering sentences like “continue atorvastatin”, and irrelevant Lipid/Cholesterol references.

Table 1.

Performance of the rule-based NLP filter across two rounds of evaluation.

Round	Dataset	Precision	Recall	Accuracy	F1	Filter-Out Rate
1st Round	Training (n=1200)	0.321	1.000	0.889	0.486	0.837
	Testing (n=800)	0.235	0.974	0.844	0.378	0.798
2nd Round	Training (n=1200)	0.270	1.000	0.858	0.426	0.806
	Testing (n=800)	0.206	1.000	0.813	0.342	0.764

LLM-Based Filter

The LLM-based refinement filter was evaluated using the same training and testing datasets from Dataset A. On the testing dataset, the LLM filter misclassified three documents in total, achieving an accuracy of 0.996. The recall was 0.947, precision was 0.973, and the F1 score was 0.96. The filter-out rate was 0.954. This filter significantly reduced the number of irrelevant documents while maintaining a high level of precision for identifying relevant ones.

LLM-Based Multi-Category Classifier

Among the 2,000 documents in Dataset A, 101 (5%) were manually identified as relevant. We also identified examples from each category to serve as examples used in the few-shot learning. The examples are presented in Table 2.

Table 2.

Examples of categories: intolerance, patient deferral, and contradictions.

Category	Example
Intolerance	Allergy: Crestor [Rosuvastatin] Other (see comments) Myalgia Statins-Hmg-Coa Reductase Inhibitors muscle pain off statin due to myalgias Pravastatin Other reaction(s): nausea I will ask him to stop the rosuvastatin. He will call me in 2 weeks time. There is an outside possibility that rosuvastatin may be contributing to fatigue. Patient allergic to atorvastatin and rosuvastatin.
Patient Deferral	She is still reluctant to take statin. She is very hesitant for a statin. High Cholesterol: [x] Yes- declines cholesterol-lowering medication; continued focus on diet and regular exercise She was reluctant to consider statin therapy, she stated. Noted to have high cholesterol, but has opted to try lifestyle adjustment measures first rather than statin therapy.
Contraindications	Patient cannot take any statins because of liver disease.

The Overall Framework Performance

The AI framework was applied to Dataset B, which included 197,761 documents for 47,192 patients. The rule-based NLP filter excluded 152,508 documents (77%), leaving 45,253 documents for further analysis. After applying the LLM-based refinement filter, 13,704 documents were identified as potentially relevant, representing 5,982 patients and 7,265 encounters. Top five frequent note types were progress notes 7,486 (55%), letter 2,388 (17%), telephone encounter 475 (3%), assessment & plan note 416 (3%), and H&P 311 (2%). The LLM-based multi-category classifier was used to categorize these snippets into the predefined categories. The overall accuracy was 0.89. Performance metrics for the classifier are summarized in Table 3. Among the allergy documents, 4,382 (68%) originated from semi-structured sources, where clinicians directly imported the allergy information from a structured table.

Table 3.

Performance of the LLM-based multi-category classifier on Dataset B.

	Precision	Recall	F1 Score
Intolerance	0.99	0.98	0.99
Contraindications	0.81	0.82	0.81
Patient Deferral	0.95	0.78	0.86
Weighted Average	0.97	0.92	0.89

At the patient level, the framework identified 3,027 patients (6.4%) with documented intolerance to statins, 310 patients (0.7%) with contraindications, and 1,391 patients (2.9%) who deferred statin therapy. In terms of cost analysis, the LLM-based filter processed 20,606,392 input tokens and generated 1,274,221 output tokens, with an overall cost of approximately $64. The LLM-based classifier processed 786,467 input tokens and generated 776,633 output tokens, incurring an overall cost of about $10.

DISCUSSION

In this study, we developed and evaluated an AI framework to extract and analyze information about statins from a large number of clinical notes. The framework integrates a rule-based NLP filter, an LLM-based refinement filter, and a multi-category classifier. This approach demonstrated high efficiency in processing clinical notes, filtering out over 90% of irrelevant notes while achieving high recall in retaining relevant ones. The LLM-based classifier further achieved good performance in categorizing the extracted information into key categories, including intolerance, contraindications, and patient deferral.

Our hybrid framework was designed to maximize efficiency and minimize cost. Applying LLMs directly to large-scale EHR datasets is computationally expensive, especially when the information to be extracted is present in only a small subset of clinical notes—akin to finding a needle in a haystack. Extracting and analyzing information from clinical notes in a repeatable and scalable manner is critical for improving patient care.[14,15] These notes contain textual but invaluable information, such as smoking history, family history, and reasons for medication deferral—details often ignored by methods that rely on structured data.[16–18] It includes a broad range of document types beyond formal progress notes; for example, telephone encounters might record patient communications and decisions that occur outside of a clinical visit. While LLMs perform well with textual data,[19] applying them to large-scale EHR datasets can be computationally expensive and unnecessary, especially when the information to be extracted is present in only a small subset of clinical notes. In such cases, efficiently identifying the notes likely to contain this information becomes crucial—akin to finding a needle in a haystack. The rule-based NLP filter in our framework serves as an important preliminary step, excluding irrelevant notes while retaining all relevant ones. This filter is interpretable (“open-box”), allowing researchers to validate and adapt criteria based on specific needs, making it more conservative or inclusive as required. The subsequent LLM-based refinement filter enhances this process by extracting the most relevant content and summarizing it when necessary. Finally, the LLM-based multi-category classifier organizes the extracted data to address specific research questions, providing a scalable, accurate, and reliable solution for processing clinical notes.

Clinical notes not only document patient deferrals of statins but also often provide the reasons behind these deferrals.[20,21] This context offers significant opportunities for designing targeted interventions at both patient and institutional levels. At the patient level, personalized strategies, such as chatbots offering tailored education and support, could enhance understanding and acceptance of statin therapy. At the institutional level, identifying patients who defer statins can contribute to hospital quality measures, providing a more accurate reflection of adherence to clinical guidelines and enabling the development of interventions to address these barriers effectively.

The integration of structured data with clinical notes represents another critical area to solve in using LLM to analyze EHR data.[22–24] Clinical notes often incorporate structured data elements, such as lab results and medication orders, using Epic SmartPhrases (e.g., .lastv for labs and .medsord for medications). This integration likely contributed to the high classifier performance in recognizing statin intolerance. While such integration enriches the notes, it also raises questions about handling duplicated information—whether to retain it for context or filter it out and using structured data directly to avoid redundancy. Future research should explore effective strategies to integrate structured and unstructured data to maximize the utility of extracted information.

While this project was developed within a well-resourced academic medical center, the core principles of the hybrid framework are adaptable to other settings. Institutions with limited resources could begin by implementing only the rule-based filter to identify patient cohorts for manual review, which itself provides significant value. For the AI component, using smaller open-source LLMs (e.g., gpt-oss-20b) on a PHI-protected local server could offer a cost-effective alternative to large proprietary models like GPT-4o, without requiring a specialized Azure platform. The key to successful implementation in any setting is a strong partnership between clinical stakeholders, who can define the problem and validate outputs, and technical teams who can build and maintain the system. This approach allows for a phased implementation that can provide incremental value while managing technical and financial constraints.

Future Directions

Future research could focus on personalized management of statin therapy, including identifying patients eligible for lipid-lowering treatment and addressing barriers to adherence. Outreach programs could be designed to address specific reasons for patient deferral, improving acceptance rates. Statin intolerance, a common issue, often necessitates trials of multiple formulations or alternative therapies, such as ezetimibe or PCSK9 inhibitors. A CDS management tool could assist healthcare providers and patients in tracking trials, monitoring side effects, and optimizing therapy to improve outcomes. The Statin-Associated Muscle Symptom Clinical Index (SAMS-CI), a method for assessing whether muscle symptoms are caused or worsened by statin use, could also be integrated into the CDS management tool to help guide treatment decisions.[25] To build more intelligent CDS tools, researchers could extract comprehensive statin information from sources beyond clinical notes, including alert override comments and patient messages.[19] Integrating extracted data on patient deferral or contraindications into alert systems could result in more tailored notifications, reducing unnecessary interruptions and improving workflow efficiency for providers.

Limitations

This study has several limitations. First, the AI framework processed note passages with a 1,925-character limit rather than entire notes. This partitioning may result in missed information at the boundaries of segments. However, given that patients typically have multiple notes documenting relevant information, the overall impact of this limitation is likely minimal. When applied to larger datasets, such as those spanning a year, the effect would be further mitigated. Second, the study relied on retrospective data from a single institution, potentially limiting the generalizability of findings to other healthcare systems with different documentation practices or EHR configurations. Future research should validate the framework across diverse institutions and datasets to confirm its robustness and adaptability. Third, while our framework’s performance was evaluated in a retrospective study, its application in a live, continuous production environment would require periodic monitoring and potential recalibration to account for changes in documentation practices or patient populations, ensuring that performance does not degrade over time.

CONCLUSION

This study presented an effective AI framework for extracting and analyzing statin-related information from clinical notes. By integrating a rule-based NLP filter, an LLM-based refinement filter, and an LLM-based multi-category classifier, the framework demonstrated high efficiency in processing clinical notes, filtering out irrelevant data while retaining critical information. This approach provides a scalable and trustworthy solution for handling unstructured data in electronic health records. The findings highlight the potential for leveraging this framework to improve CDS tools, such as alerts and order sets, by incorporating detailed patient-level insights. Future research should focus on developing systematic CDS solutions that utilize the extracted statin information to address barriers to therapy, such as intolerance and deferral, and to optimize adherence to clinical guidelines. These advancements could further enhance patient care, reduce provider burden, and promote more effective management of cardiovascular disease risk.

DATA AVAILABILITY STATEMENT

The prompts used in the LLM-based filter and LLM-based multi-category classifier were reported in the Supplementary File.