Diagnostic accuracy of electronic medical record retrieval methods and a large language model for identifying cardiovascular events: a multisite retrospective validation study in a medical system in the United States

Abstract

Objective

To compare the diagnostic accuracy of four available automated electronic medical record (EMR) retrieval methods, including a large language model (LLM)-assisted workflow, against manual chart adjudication for identifying cardiovascular events.

Design

Retrospective diagnostic accuracy study.

Setting

Three sites within a single US tertiary health system.

Participants

Two adult cohorts with previously adjudicated cardiovascular outcomes were included. Cohort 1 included 2258 patients treated with immune checkpoint inhibitors, and Cohort 2 included 1426 patients who underwent transcatheter aortic valve replacement.

Primary and secondary outcome measures

The reference standard was clinician manual chart adjudication. Outcomes included ischaemic stroke or transient ischaemic attack, myocardial infarction (MI), heart failure (HF) exacerbation or hospitalisation and a composite major adverse cardiovascular events (MACE) outcome. Automated retrieval methods included International Classification of Diseases (ICD) codes, primary diagnosis, problem list and a zero-shot LLM workflow. Area under the (receiver operating characteristic) curve (AUC), sensitivity, specificity and net reclassification improvement were assessed.

Results

In Cohort 1, the LLM achieved the highest AUC for stroke (0.920; 95% CI 0.881 to 0.958), MI (0.938; 95% CI 0.905 to 0.971) and composite MACE (0.880; 95% CI 0.854 to 0.907), whereas ICD-based retrieval had a higher AUC for HF (0.882; 95% CI 0.845 to 0.918 vs 0.873; 95% CI 0.831 to 0.914). In Cohort 2, the LLM achieved the highest AUC for all evaluated outcomes: stroke (0.915; 95% CI 0.862 to 0.968), MI (0.928; 95% CI 0.839 to 1.000), HF (0.844; 95% CI 0.803 to 0.884) and composite MACE (0.862; 95% CI 0.829 to 0.895). In Cohort 1, differences in AUC between the LLM and ICD methods were not statistically significant across outcomes, whereas in Cohort 2 the LLM showed significantly higher AUC for stroke and composite MACE.

Conclusion

In this multisite retrospective validation study, the LLM-assisted workflow showed strong but context-dependent performance for identifying cardiovascular events from the EMR. Performance varied by outcome and cohort, and ICD-based retrieval remained competitive for some use cases. These findings support a complementary role for LLM-assisted extraction in retrospective cardiovascular outcomes research.

Strengths and limitations of this study.

• This retrospective diagnostic accuracy study compared four automated electronic medical record (EMR) retrieval methods against clinician manual chart adjudication as the reference standard.

• The study included two distinct adult cohorts across three sites within a single US health system, allowing evaluation across different clinical populations.

• Performance was assessed using multiple complementary measures, including sensitivity, specificity, receiver operating characteristic curves, area under the curve and net reclassification improvement.

• The large language model workflow used National Library of Medicine Medical Subject Headings-informed rule-based document filtering and zero-shot prompting without semantic retrieval, few-shot prompting or fine-tuning, which may limit comparison with more highly optimised pipelines.

• Manual chart adjudication was performed by single trained reviewers without planned duplicate review, and the EMR-integrated workflow was evaluated within one health system, which may limit external generalisability.

Introduction

Retrospective studies are valuable in evidence-based medicine, particularly with the widespread adoption of electronic medical records (EMRs).¹ These studies allow for rapid, cost-effective analysis of clinical data and often serve as foundational and hypothesis-generating work for subsequent prospective trials.² However, a major limitation of retrospective research lies in the labour-intensive process of manual data extraction from patient charts, which is time-consuming.³ To address this challenge, many healthcare institutions have implemented large-scale structured and unstructured electronic data retrieval platforms to expedite data extraction.

Available electronic extraction tools often use International Classification of Diseases (ICD) coding, whereby ICD codes may be documented in patient history, active problem list or specifically as the primary diagnosis in the EMR. Recently, a large language model (LLM)-based artificial intelligence tool was implemented within Mayo Data Explorer, enabling natural language processing of clinical note text sourced from the Longitudinal Patient Record, which is populated by Epic and other institutional systems such as those for imaging interpretation. LLMs have demonstrated high precision and recall; however, their diagnostic accuracy for identifying complex clinical outcomes in real-world, retrospective settings remains incompletely characterised.^4–7 Studies have shown that similar LLMs may achieve sensitivity levels approaching that of manual adjudication with high efficiency; however, further validation against gold-standard methods is still needed.⁶

This retrospective diagnostic accuracy study aimed to evaluate and compare the performance of four available automated EMR retrieval methods, including an EMR-integrated zero-shot LLM workflow, against clinician manual chart adjudication for identifying cardiovascular events.

Methods

Study design

This retrospective diagnostic accuracy study was conducted with approval from the Mayo Clinic Institutional Review Board (IRB 21–0 11 503). All included patients had previously provided authorisation for research, and additional consent was waived for this specific retrospective data analysis. Patients and/or the public were not involved in the design, conduct, reporting, or dissemination plans of this retrospective study using electronic health record data. The study reporting adheres to the Standards for Reporting of Diagnostic Accuracy Studies (STARD) 2015 guidelines.⁸

Patient and public involvement

Patients and/or the public were not involved in the design, conduct, reporting, or dissemination plans of this research. This is in light of the retrospective nature of this work and approval from the institutional review board.

Participants

The study analysed two distinct patient cohorts from three Mayo Clinic sites (Rochester, Minnesota; Phoenix, Arizona; and Jacksonville, Florida). Cohort 1 included adult (age >18 years) patients with cancer who received immune checkpoint inhibitor (ICI) therapy between January 2011 and June 2022.⁹ Cohort 2 consisted of adult (age >18 years) patients who underwent transcatheter aortic valve replacement (TAVR) between January 2012 and December 2018, identified from the Mayo Clinic institutional National Cardiovascular Data Registry - Transcatheter Aortic Valve Replacement (NCDR-TAVR) database.¹⁰ These two cohorts were selected because detailed, manually adjudicated major adverse cardiovascular events (MACE) outcomes were available from previously published cohort studies in these populations,^{9 10} which allowed efficient use of validated outcome data for the present analysis.

In Cohorts 1 and 2, MACE outcomes were manually adjudicated through chart review of EMR notes by trained clinicians (physicians or nurse practitioners) from specialised cardiology teams, under the supervision of two expert cardiologists. Discrepancies in the outcome adjudication were resolved by consensus with the supervising cardiologists. Each chart was reviewed by a single trained clinician. There was no planned overlap in chart review, so inter-rater agreement statistics were not calculated.

Test methods

The reference standard for both cohorts included the outcomes of ischaemic stroke or transient ischaemic attack (TIA), non-fatal myocardial infarction (MI) and heart failure (HF) hospitalisation or exacerbation. A MACE outcome was also evaluated, including all these outcomes. Events were adjudicated according to established clinical guidelines and definitions.^11–14 Outcomes included in Cohort 1 occurred after treatment with ICI therapy,⁹ while outcomes in Cohort 2 occurred after the TAVR procedure.¹⁰

Four automated data extraction methods available through our institution’s EMR were compared against the manual adjudication reference standard. The first method was based on the presence of the specific ICD-9 and ICD-10 codes anywhere in the EMR, where outcomes were identified by selecting earliest outcomes after the index event date (ICI administration or TAVR procedure) and removing duplicates (online supplemental figure 1). The ICD and miscellaneous codes used to define the four outcomes were derived from published definitions of MACE outcomes and then reviewed and confirmed by two cardiology experts. These codes were applied across all relevant EMR diagnosis fields associated with these encounters (eg, emergency department, inpatient and outpatient encounters, including admission and discharge diagnoses and billing/encounter diagnoses).

The second data retrieval method, primary diagnosis, was a subset of the ICD method that included only diagnostic codes listed as the primary reason for hospital admission (online supplemental figure 2). ICD and miscellaneous codes used for the first and second methods are detailed in online supplemental table 1. The third method (online supplemental figure 3) involved identifying outcomes if diagnoses appeared in the EMR problem list in Epic, identified as a structured list of active diagnoses selected by clinicians from standardised entries (online supplemental table 2) that are directly mapped to ICD codes; free-text entries are not used as problem list items. An outcome was considered present when an ICD-coded diagnosis corresponding to stroke/TIA, MI or HF appeared on the active problem list after the index event.

The fourth method used an EMR-integrated LLM tool, which at the time of analysis (June 2025) relied on a commercially available LLM, Gemini V.1.5 (Google, Mountain View, California, USA), deployed on the Mayo Clinic Cloud within Mayo Clinic’s secure environment. The model was used in a zero-shot setting with no additional fine-tuning or few-shot examples, and only fixed, standardised prompts were applied. The tool was used in three main steps (online supplemental figure 4). First, we entered the list of patient medical record numbers to be analysed. Second, we chose the search terms and the types of EMR documents to include. Third, we specified the questions that the LLM should answer. For the second step, we created lists of search terms for stroke/TIA, MI and HF based on concepts and synonyms from the National Library of Medicine Medical Subject Headings (MeSH). Two cardiologists with expertise in cardiovascular outcomes reviewed and refined these terms.

The LLM tool used these MeSH-based terms to retrieve all potentially relevant notes for each patient. This MeSH-based filtering step was intended to improve sensitivity by retrieving all potentially relevant documents across available EMR note types before model inference. In this study, we included all available document types, including emergency department notes, inpatient and outpatient progress notes, cardiology consultation notes, procedure reports, discharge summaries, imaging reports and nursing documentation. These search terms acted as simple rule-based filters inside the EMR to decide which documents were sent to the model. A separate vector database or embedding-based retrieval system was not used. In the third step, the LLM performed an information extraction task. For each patient and each outcome, it was given two standard questions: (1) ‘Did the patient present for [stroke/myocardial infarction/heart failure] as the primary reason for medical evaluation?’ with the answer limited to ‘yes’, ‘no’ or ‘not mentioned’, and (2) ‘What is the documented date of the event, if any?’ (online supplemental table 3). For the diagnostic accuracy analysis, we used the yes/no output as a binary label and used the date only to confirm that the event occurred after the index exposure (ICI or TAVR).

We then applied predefined rules to clean and interpret the LLM outputs. Patients were classified as having the outcome only if the model answered ‘yes’ and provided a valid, chronologically plausible event date after the index event. Any other response pattern, including ‘no’ or ‘not mentioned’ with or without a date, ‘yes’ without a date or internally inconsistent combinations such as ‘no’ with a date, was classified as not having the outcome, consistent with our manual adjudication approach, which excludes events without a clearly documented date. The main generation parameters for Gemini V.1.5 (temperature 0.0, topP 1.0, maxOutputTokens 10 480) and the default safety settings are summarised in online supplemental table 8.

Analysis

Classification performance for each outcome, cohort and extraction method was summarised using sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), precision and F1-score. For each comparison, we constructed 2×2 confusion matrices of true positives, false positives, true negatives and false negatives, and all metrics were calculated from these counts using standard definitions. Receiver operating characteristic (ROC) curves and area under the curve (AUC) values with 95% CIs were computed using the pROC package in R (R Foundation for Statistical Computing, Vienna, Austria), which uses DeLong’s non-parametric method to estimate AUC variance and compare correlated ROC curves. Confusion matrices and derived performance metrics were obtained using the confusion matrix function in the caret package. Net reclassification improvement (NRI) comparing ICD and LLM methods for each outcome and cohort was calculated and summarised in online supplemental table 6.

Manual outcome adjudication and electronic data extraction were performed independently by different researchers, each blinded to the results of the other. The final analysis was conducted by a separate investigator who had access to both index test and reference standard outcomes in binary format (0 or 1) only, without access to raw clinical records. Statistical analyses were conducted using IBM SPSS Statistics, V.28.0 (IBM Corp, Armonk, New York, USA) and RStudio (RStudio: Integrated Development for R, Posit Software, PBC, Boston, MA, USA).

Results

A total of 3684 patients were included in the study: 2258 patients in Cohort 1 and 1426 in Cohort 2. In Cohort 1, the LLM approach achieved the highest AUC for stroke (0.920; 95% CI 0.881 to 0.958), MI (0.938; 95% CI 0.905 to 0.971) and MACE (0.880; 95% CI 0.854 to 0.907) among the evaluated automated extraction methods. For HF, ICD-based retrieval had a slightly higher AUC than the LLM (0.882; 95% CI 0.845 to 0.918 vs 0.873; 95% CI 0.831 to 0.914). These results are shown in figure 1.

Figure 1. ROC curve for each diagnostic method in Cohort 1. ROC curves comparing the diagnostic accuracy of four data extraction methods; LLM, ICD codes, PD and PL, against manual chart review for identifying clinical outcomes in Cohort 1 (ICI). (A) shows the ROC curves for stroke, (B) for myocardial infarction, (C) for heart failure and (D) for MACE. ICD, International Classification of Diseases; ICI, immune checkpoint inhibitors; LLM, large language model; MACE, major adverse cardiovascular events; PD, primary diagnosis; PL, problem list; ROC, receiver operating characteristic.

In Cohort 2, the LLM achieved the highest AUC among the evaluated electronic extraction methods for all recorded clinical outcomes: stroke (0.915; 95% CI 0.862 to 0.968), MI (0.928; 95% CI 0.839 to 1.0), HF (0.844; 95% CI 0.803 to 0.884) and MACE (0.862; 95% CI 0.829 to 0.895), as shown in figure 2. Tables 1 and 2 summarise the AUCs, sensitivities and specificities across all index tests and outcomes in both cohorts. Online supplemental tables 4 and 5 provide detailed cross-tabulations (true positives, false positives, true negatives and false negatives) for each method, outcome and cohort.

Figure 2. ROC curve for each diagnostic method in Cohort 2. ROC curves comparing the diagnostic accuracy of four data extraction methods; LLM, ICD codes, PD and PL, against manual chart review for identifying clinical outcomes in Cohort 2 (TAVR). (A) shows the ROC curves for stroke, (B) for myocardial infarction, (C) for heart failure and (D) for MACE. ICD, International Classification of Diseases; LLM, large language model; MACE, major adverse cardiovascular events; PD, primary diagnosis; PL, problem list; ROC, receiver operating characteristic; TAVR, transcatheter aortic valve replacement.

Table 1. Diagnostic accuracy metrics in Cohort 1 (patients with ICI therapy).

Outcome	Method	Sensitivity	Specificity	AUC (95% CI)
Stroke	ICD	89%	94%	0.913 (0.878 to 0.948)
	PD	83%	96%	0.898 (0.857 to 0.940)
	PL	50%	72%	0.743 (0.682 to 0.804)
	LLM	86%	98%	0.920 (0.881 to 0.958)
MI	ICD	86%	94%	0.899 (0.863 to 0.935)
	PD	57%	98%	0.773 (0.718 to 0.828)
	PL	32%	72%	0.656 (0.597 to 0.716)
	LLM	89%	99%	0.938 (0.905 to 0.971)
Heart failure	ICD	82%	94%	0.882 (0.845 to 0.918)
	PD	49%	98%	0.738 (0.687 to 0.790)
	PL	39%	72%	0.649 (0.595 to 0.702)
	LLM	75%	99%	0.873 (0.831 to 0.914)
Combined MACE	ICD	87%	85%	0.855 (0.832 to 0.878)
	PD	66%	93%	0.796 (0.765 to 0.827)
	PL	40%	72%	0.686 (0.649 to 0.722)
	LLM	78%	98%	0.880 (0.854 to 0.907)

Sensitivity, specificity and AUC (with 95% CI) for each automated method. ICD codes, PD, PL and LLM across four clinical outcomes: stroke, MI, heart failure and MACE in Cohort 1 (ICI).

AUC, area under the curve; ICD, International Classification of Diseases; ICI, immune checkpoint inhibitors; LLM, large language model; MACE, major adverse cardiovascular events; MI, myocardial infarction; PD, primary diagnosis; PL, problem list.

Table 2. Diagnostic accuracy metrics in Cohort 2 (patients who underwent TAVR).

Outcome	Method	Sensitivity	Specificity	AUC (95% CI)
Stroke	ICD	70%	88%	0.787 (0.721 to 0.853)
	PD	32%	92%	0.650 (0.569 to 0.730)
	PL	39%	99%	0.691 (0.611 to 0.772)
	LLM	83%	100%	0.915 (0.862 to 0.968)
MI	ICD	71%	96%	0.836 (0.720 to 0.952)
	PD	38%	99%	0.687 (0.545 to 0.829)
	PL	10%	100%	0.547 (0.413 to 0.680)
	LLM	86%	100%	0.928 (0.839 to 1.000)
Heart failure	ICD	76%	91%	0.839 (0.801 to 0.877)
	PD	45%	98%	0.714 (0.665 to 0.762)
	PL	12%	99%	0.557 (0.509 to 0.605)
	LLM	72%	97%	0.844 (0.803 to 0.884)
Combined MACE	ICD	79%	81%	0.801 (0.769 to 0.833)
	PD	46%	96%	0.708 (0.666 to 0.750)
	PL	22%	98%	0.599 (0.556 to 0.642)
	LLM	76%	97%	0.862 (0.829 to 0.895)

Sensitivity, specificity and AUC (with 95% CI) for each automated method. ICD codes, PD, PL and LLM, across four clinical outcomes in Cohort 2 (TAVR): stroke, MI, heart failure and combined MACE.

AUC, area under the curve; ICD, International Classification of Diseases; LLM, large language model; MACE, major adverse cardiovascular events; MI, myocardial infarction; PD, primary diagnosis; PL, problem list; TAVR, transcatheter aortic valve replacement.

Detailed operating characteristics, including sensitivity, specificity, PPV, NPV and F1-score for each outcome, cohort and method, are reported in online supplemental table 7 and are consistent with the overall performance patterns described above. Across all outcomes and cohorts, the LLM produced internally inconsistent outputs (eg, ‘no’ with a date) in <1% of queries, and these were classified as negative according to the predefined post-processing rules.

Individual outcome assessment

Stroke

The LLM method achieved the highest AUC for stroke in both cohorts. In Cohort 1, the LLM yielded an AUC of 0.920 (95% CI 0.881 to 0.958), sensitivity of 86% and specificity of 98%. In Cohort 2, the LLM showed the highest AUC at 0.915 (95% CI 0.862 to 0.968), with sensitivity and specificity of 83% and 100%, respectively.

Myocardial infarction

For the detection of MI in Cohort 1, the LLM method achieved the highest AUC among the evaluated approaches at 0.938 (95% CI 0.905 to 0.971), with sensitivity of 89% and specificity of 99%. In Cohort 2, the LLM again showed the highest AUC at 0.928 (95% CI 0.839 to 1.000), with sensitivity of 86% and specificity of 100%.

Heart failure

In contrast to the other outcomes, the ICD-based extraction method yielded the highest AUC for HF in Cohort 1, at 0.882 (95% CI 0.845 to 0.918), with sensitivity and specificity of 82% and 94%, respectively. The LLM followed with an AUC of 0.873 (95% CI 0.831 to 0.914), sensitivity of 75% and specificity of 99%. In Cohort 2, the LLM had a slightly higher AUC than ICD-based retrieval, at 0.844 (95% CI 0.803 to 0.884) versus 0.839 (95% CI 0.801 to 0.877), with sensitivities of 72% and 76% and specificities of 97% and 91%, respectively.

Combined major adverse cardiovascular events

For the composite MACE outcome in Cohort 1, the LLM achieved the highest AUC at 0.880 (95% CI 0.854 to 0.907), with sensitivity of 78% and specificity of 98%. ICD-based retrieval followed with an AUC of 0.855 (95% CI 0.832 to 0.878), sensitivity of 87% and specificity of 85%. In Cohort 2, the LLM again achieved the highest AUC at 0.862 (95% CI 0.829 to 0.895), with sensitivity of 76% and specificity of 97%. ICD-based retrieval followed with an AUC of 0.801 (95% CI 0.769 to 0.833), sensitivity of 79% and specificity of 81%.

Comparative performance

DeLong’s test was used to compare the AUCs of the LLM and ICD methods for each outcome in both cohorts (online supplemental table 6). In Cohort 1, there were no statistically significant differences in AUC between the LLM and ICD methods across all outcomes: stroke (p=0.730), MI (p=0.058), HF (p=0.62) and combined MACE (p=0.071). In Cohort 2, the LLM showed significantly higher AUC than ICD for stroke (p<0.001) and combined MACE (p<0.001). No statistically significant differences were found for MI (p=0.140) or HF (p=0.740).

NRI metrics showed variable differences in classification performance between the LLM and ICD-based retrieval methods (online supplemental table 6). In Cohort 1, NRI values were lower, showing modest positive reclassification for stroke, MI and MACE, and slightly negative reclassification for HF: stroke 1.4%, MI 7.8%, combined MACE 5% and HF −1.8%. In Cohort 2, stroke had an NRI of 25.6%, MI 18.3%, combined MACE 12.3% and HF 1.4%.

Discussion

In this retrospective diagnostic accuracy study, the LLM-assisted EMR workflow showed strong performance for several cardiovascular outcomes, with performance varying depending on context (outcome and cohort). The LLM achieved the highest AUC for most outcomes, although differences compared with ICD-based retrieval methods were not statistically significant in Cohort 1 and ICD-based retrieval performed slightly better for HF in Cohort 1.

Consistent with prior validation studies, we found that ICD codes had high sensitivity and specificity for outcomes detection in both cohorts.^{15 16} The use of ICD codes has allowed researchers for decades to systematically identify patients with specific diseases, clinical outcomes or procedures, enabling the creation of large cohorts from EMR or multicentre registries. However, in our cohorts the performance of ICD codes was not consistent across outcomes and cohorts, underscoring known limitations of this method when applied across different populations.^17–23 In particular, restricting analysis to ‘primary diagnosis’ ICD codes, as we did with one method, yielded increased specificity but often at the cost of lower sensitivity and overall accuracy. This supports prior findings that suggest such trade-offs can lead to underestimation of event rates in clinical research.^{20 23}

Our study suggests that the use of structured data alone may be insufficient for accurate case ascertainment in retrospective studies. In contrast, the LLM method, which extracted information from all sections of clinical notes using natural language processing, showed strong diagnostic performance across several outcomes, although performance varied by cohort and outcome. Notably, in Cohort 2, which was a primary cardiology patient population who underwent TAVR, LLM-based data retrieval showed statistically significant improvements over ICD code-based data extraction in identifying stroke and combined MACE, with NRIs of 25.6% and 12.3%, respectively.

Interestingly, ICD coding had a higher AUC than the LLM model in one analysis: Cohort 1 for the outcome of HF. A potential explanation for this finding may be that this population comprised patients with cancer who were unlikely to have a previous history of HF. In populations with a pre-existing history of chronic conditions like HF, ICD-based retrieval often suffers from reduced specificity because historical codes frequently reappear in the patient’s chart for subsequent encounters, chronic management or billing. Even when filtering for codes recorded after a designated ‘day 0’, these recurrent historical codes can be mistakenly captured as new, incident outcomes. In contrast, the LLM reads through unstructured clinical notes to actively contextualise the text, allowing it to differentiate historical mentions from true acute onset. Despite these advances, expert manual chart review remains the gold standard for the collection of clinical outcomes in retrospective studies. In our study, it served as the reference standard for validating all four index tests. Manual adjudication enabled us to detect misclassifications and validate the precision of automated algorithms, a step highlighted as critical in multiple studies.^{21 22 24 25}

During interpretation of these findings, it is important to consider how the LLM is used in our health system. During the study period, our Epic EMR included an LLM-based tool (Gemini V.1.5) that clinicians and researchers could run directly within the patient chart. Other approaches, such as smaller language models or traditional rule-based text systems, were not integrated into the EMR and were therefore not readily available for routine data extraction in our setting. Our aim in this study was therefore not to compare all possible machine-learning methods, but to evaluate how well the LLM tool already available in clinical practice performs for identifying cardiovascular outcomes.

From a practical perspective, each diagnostic method evaluated in this study has advantages and disadvantages, and the choice of method may depend on the indication for data collection, the cohort under evaluation and the ultimate goal of the analysis, including the relative importance of sensitivity and specificity. For example, the current LLM model achieved higher AUC than ICD codes alone for most evaluated outcomes. However, particularly with the learning curve with initial use, our experience was that this method required additional processing time and cleaning of the initial data output, although formal timing analyses were not performed. Future studies should therefore evaluate operational efficiency more formally, including direct comparisons of processing time across methods. In contrast, retrieval of ICD-coded outcomes alone was faster but was associated with lower diagnostic specificity than the LLM model. At present, the optimal method for data extraction should therefore be determined on a study-by-study basis according to the required balance between efficiency and accuracy.

While this study offers valuable insights into EMR-integrated LLM workflows, several methodological limitations must be considered. First, the EMR-integrated workflow was evaluated within a single US healthcare system using a specific institutional configuration, which may limit the external generalisability; however many medical institutions in the USA and abroad use the same EMR or similar systems. Second, our reference standard relied on manual chart adjudication by single trained reviewers without routine duplicate review, meaning inter-rater agreement was not formally assessed.

Third, our evaluation was restricted to the specific version of the model active during the study period (Gemini V.1.5) in a zero-shot configuration without semantic retrieval or fine-tuning; performance may vary with newer iterations (such as Gemini V.2.5 Flash) or alternative pipeline architectures. Fourth, we evaluated structured code-based methods and unstructured text-based methods independently rather than exploring hybrid, combined strategies. Finally, while we noted qualitative operational trade-offs regarding data cleaning and processing time for the LLM workflow, a formal timing and efficiency analysis was not possible in this study.

In this multisite retrospective validation study, an EMR-integrated zero-shot LLM workflow showed strong but context-dependent performance for identifying cardiovascular outcomes. Although it achieved favourable discrimination for several outcomes, ICD-based retrieval remained competitive and, in some settings, performed slightly better. These findings support a complementary role for LLM-assisted extraction in retrospective research workflows, while highlighting the need for continued validation, transparency and assessment of operational trade-offs. An understanding of the strengths and limitations of LLM and ICD based electronic retrieval may help expedite cardiovascular outcome data retrieval for large retrospective research studies and identify instances where manual chart review is still required for accuracy of findings.

Data availability statement

Data are available upon reasonable request.