Challenges in Identifying Heart Failure Diagnoses from the Electronic Health Record

Abstract

Background:

To leverage the EHR effectively for clinical operations, quality improvement, and research, patients with a specific clinical condition such as heart failure are often identified through ICD diagnosis codes and structured data. These diagnoses are validated by clinician chart review, which is considered the gold standard for confirming true cases from the EHR. The purpose of the study was to explore obstacles that clinicians encounter when using chart review to validate heart failure diagnoses from the EHR.

Methods:

In prior work, a team of eight clinicians performed chart review to validate heart failure diagnosis based on the “Universal Definition of Heart Failure.” A portion of the charts were reviewed by two independent reviewers, and we assessed inter-rater reliability using Cohen’s kappa (κ). We conducted a content analysis of documents from the clinician chart review to understand challenges in validating EHR-based definitions of heart failure.

Results:

Clinicians reviewed 528 charts, with a κ of 0.62 among the 35% of charts with double review. In qualitative analysis, we identified two overarching themes related to challenges in validation of heart failure: 1) challenges in applying the Universal Definition of Heart Failure to real-world patient data, and 2) EHR interoperability and usability barriers impacting validation of heart failure diagnoses.

Conclusions:

Our study highlights clinical and technological limitations of clinician chart review for identifying heart failure patients. This has implications for patient care, quality improvement, and population health. Policies and digital solutions aimed at improving EHR data collection, organization, and data extraction are needed.

Introduction:

Electronic health records (EHRs) have evolved into an essential infrastructure that supports care delivery across multiple domains – from real-time patient assessment and clinical documentation to retrospective reviews that inform quality improvement initiatives, clinical research, and population health management.^1–4 These EHR functions are particularly important for complex conditions such as heart failure, where the integration of real-time clinical data with longitudinal documentation is crucial for navigating the syndrome's diagnostic complexity and optimizing adherence to evidence-based therapies. To leverage the EHR for many of these purposes, healthcare teams must develop EHR-based cohorts that identify specific patient populations. While exact methodologies vary, cohorts are typically defined by some combination of International Classification of Diseases (ICD) diagnosis codes found in places such as the problem lists, structured data elements including lab values, or unstructured data such as clinician notes. The accuracy of such cohort definitions can be validated by a clinician chart review of a subset of the patients; such validation is considered the gold standard for confirming cohort accuracy.^5–7 While abundant research has been published on methods for identifying cohorts from the EHR, far less attention has been devoted to the clinician chart validation process. Although regarded as the true outcome or gold standard, this process is itself susceptible to inaccuracies that risk introducing biases.^8,9

These validation challenges affect not only research methodology but also the delivery of clinical care. Clinicians routinely review individual patient charts in the EHR to assess the medical history and determine whether an individual truly has a condition such as heart failure. Clinicians routinely review individual patient charts in the EHR to assess the medical history and determine whether an individual truly has a condition such as heart failure. This, in turn, can impact downstream clinical decision making, such as a clinician’s decision to order further diagnostic workup, initiate heart failure-specific medication therapy, or refer to a specialist.¹⁰ However, clinicians face may face multiple barriers when assessing a patienťs history within the EHR, including incomplete and superfluous documentation and lack of compatibility across health systems that leaves key data inaccessible.^11–13

In a prior study, we developed and validated eight distinct EHR-based definitions of heart failure created from combinations of ICD codes and ejection fraction (EF).¹⁴ While validating these definitions against the gold standard of clinician-guided chart review, we observed an unexpected degree of discrepancies and questions amongst our clinician reviewers when confirming heart failure diagnosis. The purpose of the current study was to explore specific obstacles clinicians encounter when attempting to validate heart failure diagnoses using EHR data.

Methods:

Study Design and Data Source

This was a retrospective qualitative content analysis of documents from a previously performed clinician chart review to validate eight EHR-based definitions of heart failure.¹⁴ These definitions, which included combinations of ICD diagnosis codes and EF measures, were as follows: 1) EF ≤40%, 2) heart failure diagnosis based on clinical encounter, 3) heart failure diagnosis based on problem list, 4) heart failure diagnosis based on past medical history, 5) heart failure diagnoses in at least two EHR fields, 6) heart failure diagnosis in all three EHR fields, 7) EF ≤40% and a diagnosis in at least one EHR field, 8) EF≤40 or a diagnosis in at least one EHR field.¹⁵ The primary data source used for chart review was the EHR from patients within the NYU Langone Health System, a large, academic health system comprising over 300 outpatient sites and 4 tertiary hospitals that relies on Epic (Epic Systems, Madison, Wisconsin) as its EHR platform. This analysis follows the Standards for Reporting Qualitative Research guidelines.¹⁶ This study was approved by the Institutional Review Board at NYU Grossman School of Medicine.

Chart Selection

As previously described, we used stratified sampling to identify charts for clinician chart review.¹⁷ We included patients ≥18 years old with at least one outpatient or inpatient clinical encounter between the encounter period from 4/1/2021–10/31/2022. We randomly sampled both from patients who both met at least one of our definition criteria, i.e. had an EF ≤40% or a diagnostic code for heart failure, and from patients who did not meet these criteria. Sampled charts were then distributed for review.

Clinician Review Process

Our clinician reviewers consisted of a team of eight individuals – one heart failure cardiologist, one general cardiologist, one primary care physician, one internal medicine resident, three clinical cardiology fellows, and one heart failure cardiology physician assistant. Clinicians reviewed all available chart information up to and including a specified visit date, occurring within the encounter period, for each patient, while ignoring any data entered after that date. To ensure consistency, all reviewers initially reviewed the same 10 patient charts and then met to discuss findings, resolve inconstancies, and identify ambiguousness in data interpretation. Based on these discussions, we modified guideline tips for chart reviewers. Through the review process, reviewers were encouraged to email questions or concerns to the group. Two additional meetings were held as a group to refine the reviewer guidelines by consensus. At least two reviewers verified heart failure diagnosis in 35% of the patient charts, and any disagreements among reviewers were reviewed by a third reviewer. For the double reviewed charts, an agreement rate was calculated, and inter-rater reliability was measured via Cohen’s kappa (κ).

Defining Heart Failure

Heart Failure was defined in accordance with the “Universal Definition of Heart Failure”¹⁸ and operationalized as a clinical syndrome needing to meet each of the following three components: (1) current or prior symptoms or signs of heart failure, (2) a structural or functional cardiac abnormality, and (3) corroborating evidence from elevated natriuretic peptide levels or objective evidence of cardiogenic or systemic congestion by diagnostic modalities.¹⁸ From this, our team developed a guideline document for reviewers on how to diagnose heart failure from the EHR (Table 1). To help guide review, we developed a structured chart abstraction tool in Excel (Microsoft, Redmond, Washington) for reviewers to input their answer to the following structured questions: signs/symptoms of heart failure (yes, no, or unknown), EF (reduced EF, moderately reduced EF, normal EF, improved EF, unknown), other structural/functional abnormality (yes, no, or unknown), elevated B-type natriuretic peptide (BNP) (yes, no, or unknown), and heart failure diagnosis (yes or no). There was a final field labeled “comments” where clinicians could provide free text for any questions, comments, or relevant information used to make the determination of a diagnosis (Table 1).

Table 1–

Chart Abstraction Guidelines Created for Chart Reviewers Diagnosing Heart Failure from the EHR

Variable of interest	Input Options for chart reviewers	Description of Input Options
Visit date	(Pre-Filled)	Using to determine whether on this day, the patient has heart failure (HF) as a condition. Chart reviewers cannot look forward from that date, but can look back earlier in chart
Signs and/or Symptoms of HF	Yes, No, or Unknown	Yes: Refers to having signs or symptoms of HF, listed at bottom of table, either at listed visit date or prior No: No signs or symptoms of HF ever documented at listed visit date or prior Unknown: Unsure if patient had signs or symptoms of HF
Ejection Fraction (EF)	Reduced EF, Moderately Reduced EF, Normal EF, Improved EF, or Unknown	Reduced EF: EF≤40% Moderately reduced EF: 41≤EF≥49 and no prior EF≤40 Normal EF: EF≥50 and no prior EF≤40 Improved EF: Prior EF≤40 with at least 10% increase and now>40% OR Prior moderately reduced EF (41≤EF≥49 and no prior EF≤40) with at least 10% increase and now EF ≥50% Unknown: no EF of any modality found in EHR on or before visit date ***EF Hierarchy: Trust EF from cardiac MRI, resting transthoracic echocardiogram (not stress transthoracic echocardiogram or transesophageal echocardiogram), and documented provider notes commenting on prior cardiac MRI or resting transthoracic echocardiogram. If these are unavailable, then can rely on things like transesophageal echocardiogram, nuclear imaging, and catheterization
Other structural and/or functional abnormality	Yes, No, or Unknown	Yes: presence of abnormal cardiac chamber enlargement, E/E' >15, moderate/severe ventricular hypertrophy or moderate/severe valvular obstructive or regurgitant lesion No: no structural abnormalities present Unknown: missing from EHR on or before visit date, so unable to make a determination
Elevated BNP	Yes, No, or Unknown	Yes: Elevated BNP, either current or prior. Defined as: ambulatory: BNP≥35 or NT-ProBNP≥125; Hospitalization: BNP≥100 or NT-ProBNP≥300 No: Only normal BNP documented Unknown: no BNP results documented in EHR
HF Diagnosis	Yes or No	Decision primarily based on clinical judgement
Comments	Free text	Input text as needed if questions or comments arise. Can include the relevant information used to make determination
Notes about Defining HF		We are using the recently published "Universal Definition of Heart Failure" as a guide. The definition relies on having the following 3 criteria met: 1) symptoms and/or signs of heart failure; 2) structural changes; 3) elevated BNP or another objective measure as of the date listed on the visit date.
Signs and Symptoms of HF from the Universal Definition of HF		Typical symptoms – Breathlessness, orthopnea, paroxysmal nocturnal dyspnea, reduced exercised tolerance, fatigue/tolerance, ankle swelling, inability to exercise, swelling of parts of the body other than ankles, bendopnea Less typical symptoms – nocturnal cough, wheezing, bloated feeling, postprandial satiety, loss of appetite, decline in cognitive function/confusion, depression, dizziness, syncope Specific signs of HF – elevated jugular venous pressure, third heart sound, summation gallop with third and fourth heart sounds, cardiomegaly, laterally displaced apical impulse, hepatojugular reflex, Cheyne stokes respiration in advanced heart failure Less specific signs – peripheral edema (ankle, sacral, scrotal), pulmonary rales, unintentional weight gain, weight loss with muscle wasting and cachexia, cardiac murmur, reduced air entry and dullness to percussion at lung bases, tachycardia, irregular pulse, tachypnea, hepatomegaly/ascites, cold extremities, oliguria, narrow pulse pressure
Abbreviations		HF= heart failure, EF= ejection fraction, BNP= B-type natriuretic peptide, NT-proBNP = N-terminal pro-B-type natriuretic peptide, EHR = electronic health record

Analysis

To understand the challenges identified during the chart review process, we conducted a qualitative content analysis using a framework of thematic analysis.¹⁹ The data sources for this work consisted of the following documents: transcribed meeting notes, email correspondences among the team of reviewers, and chart abstraction tools completed by the reviewers, with a particular focus on the free-text “comment” field. Following an inductive approach, we developed a preliminary codebook through open coding of the documents and then our group met to discuss and refine those initial codes. Subsequently, two clinicians, who had also participated in the chart review, independently coded all materials. The two clinicians then conducted a reconciliation meeting to compare coding applications, resolve discrepancies through consensus discussions, and ensure intercoder reliability. In the final analytic phase, we used the refined codes to cluster them into higher order themes. We also separately analyzed the reviewer comments for double reviewed cases in which reviewers disagreed about the final diagnosis, and organized the reasons for disagreements into the pre-established themes from the thematic analysis. A single chart could include multiple reasons for disagreement.

Results

Among the 528 clinician reviewed charts, 35% were double reviewed, yielding an agreement rate of 82.5% and a kappa of 0.62 (95% CI: 0.5–0.74) for the presence of heart failure. Documents related to these reviewed charts formed the basis for the qualitative analysis. The data source for this analysis included transcribed notes from 3 meetings (word count 1975), email correspondences from 12 email exchanges including the entire team of clinician reviewers (word count 1695), and the reviewers’ free-text comments in the completed chart abstraction document (word count 7714). From qualitative analysis of the study, we identified two overarching themes: 1) challenges applying the Universal Definition of Heart Failure amidst incomplete or conflicting clinical data, and 2) EHR interoperability and usability barriers impacting validation of heart failure diagnoses (Table 2).

Table 2–

Themes, Subthemes, and Examples from Thematic Analysis

Theme 1: Challenges in applying the Universal Definition of Heart Failure to real-world patient data
Sub theme #	Code	Definition	Example
1	Ejection Fraction across different imaging modalities	Different modalities may be used to measure EF across multiple time points with variation in their reliability.	“Someone had stress test with low EF and then a year later had echo where it was normal – so reviewer could see both arguments as being valid. Becomes particularly difficult when there are different modalities at different times – we could create a rule that for someone to qualify as improved EF, needs to document a similar modality (vs comparing a catheterization with low EF to an echocardiogram with normal EF for example)”
2	Insufficient data collection by routine clinical care	Although diagnosis of heart failure is assessed using measurements including BNP and EF, many of these measures were never collected by routine clinical care	“Patient only has podiatry visits, no prior cardiac imaging, endorsement of heart failure symptoms, or BNPs. Cannot make heart failure diagnosis with the data available.”
3	Reversible Cardiac Conditions	Heart failure symptoms due to conditions such as valvular disease or stress cardiomyopathy that may be reversible	“I'd lean yes [to heart failure diagnosis]. Had heart failure symptoms due to valvular disease in 2014. But then resolved completely with surgery."
4	Differences in Clinical Interpretation of the universal definition	Reviewers disagreed in certain cases or needed to have lengthy discussions to finalize agreement on whether individuals had heart failure	“Left Atrial Dilation – technically not cardiac dysfunction per universal definition, but chamber dilation does count”
5	Non-Specific Signs/Symptoms	Diagnosing heart failure is complicated by non-specific symptoms	“Shortness of breath was from COVID, and thought to have lower extremity edema secondary to amlodipine”
Theme 2: EHR interoperability and usability barriers impacting validation of heart failure diagnoses
Sub theme #	Code	Definition	Example
6	Fragmented care delivered at multiple sites	Fragmented care delivery leads to clinical data often being inaccessible across institutions, especially across geographic areas or different EHR systems	“Not seeing any documented history of congestive heart failure, signs/symptoms of it, prior echocardiograms or BNPs. There are some care everywhere records I can’t see”
7	Lack of Corroborating Data	Diagnosis is coded in the chart or diagnosis of heart failure is written as free text in provider notes with no corroboration based on the universal definition of heart failure	“Got nothing here. Just listed as past medical history."
8	Data elements may be difficult to find within the EHR	The same clinical information may be presented in different formats or filed in varying locations within the EHR, complicating information retrieval.	Reviewer located right heart catheterization results buried in scanned documents instead of being properly categorized under cardiology reports, which allowed her to clinch the diagnosis of heart failure
9	Differences in Navigating the EHR	Providers conducting chart review had varying success in finding scattered information within patient charts	Example case where one reviewer documented a normal EF and commented that there was an echocardiogram report as a scanned document in a pre-operative note while the other reviewer listed unknown EF and commented that there was no echocardiogram in the chart

Theme 1: Challenges in applying the Universal Definition of Heart Failure to real-world patient data

The first theme reflected lack of clarity in both patient presentation as well as in the Universal Definition which made it hard to clinch the diagnosis of heart failure. This theme included five subthemes (Table 2). The first subtheme of “variations in ejection fractions across different modalities” highlighted the need for additional guidance to clarify which imaging modalities, such as echocardiography, magnetic resonance imaging (MRI), or nuclear scan, should be prioritized for EF interpretation. This was needed as there were instances where EF assessment differed between two imaging modalities. The second subtheme of “insufficient data collection by routine clinical care” highlighted the ambiguity of diagnosing heart failure when the patient’s care team had obtained an elevated BNP, but no cardiac imaging had been performed. As a result, reviewers were unable to determine if such a patient met the criterion for heart failure of having a structural or functional cardiac abnormality. Another sub-theme of “reversible cardiac conditions” included a discussion as to whether a patient with prior heart failure symptoms due to valvular disease that resolved with surgical treatment should still be considered as having a diagnosis of heart failure. The sub-theme, “differences in clinical interpretation of the Universal Definition”, included disagreements that arose when all reviewers met to discuss the initial ten charts. Reviewers needed to have lengthy discussions to reach agreement in some cases stemming from differences in interpreting situations such as whether left atrial dilatation was sufficient to meet the ‘structural abnormality’ part of the Universal Definition. Lastly, the subtheme, “non-specific signs and symptoms”, emerged from clinician reviewers noting that many patients presented with vague symptoms such as shortness of breath, which were not always clearly attributable to heart failure versus another conditions, such as respiratory disease.

Theme 2: EHR interoperability and usability barriers impacting validation of heart failure diagnoses

The second theme consisted of technical obstacles encountered when performing chart review within the EHR. This theme included four subthemes. The first subtheme, “fragmented care delivered at multiple sites,” highlighted the challenges of patients who had incomplete records available due to having received care at outside facilities. Clinician reviewers often encountered an ICD code for heart failure in the chart but could not locate the supporting diagnostic data to validate the diagnosis because testing and documentation resided in another health system to which reviewers did not have access. The sub-theme “lack of corroborating data” emphasized the multiple instances that the term “heart failure” was documented in the patient summaries in clinician notes without any quantitative data, such as BNP values or echocardiogram reports. This was often attributed to the clinicians commonly using copy-paste or the copy-forward functionality, which enables existing information to be reused without retyping any information. “Data elements may be difficult to find within the EHR” was a sub-theme which captured instances where data elements were stored in unusual places in the EHR. One such example was a scanned right heart catheterization which was stored in a tab for media imaging rather than in the usual location with other cardiology results. Meanwhile, the sub-theme “differences in navigating the EHR” highlighted reviewer differences in their workflow patterns and preferences for utilizing tools in the EHR to identify key information. These differences were identified during our discussion of the initial 10 charts reviewed by all clinicians when we learned that some were unfamiliar with the “Search” function in the EHR to search for information. Other clinician reviewers did not routinely utilize “Care Everywhere” which allows clinical providers to see patient records at specific outside institutions. These differences led to multiple instances in which reviewers found different pieces of information for the same patient chart, such as one reviewer finding low EF versus another not being able to find the EF.

In 32 out of 185 double reviewed cases, reviewers disagreed about the diagnosis, resulting in adjudication by a 3^rd reviewer. We summarized the reasons for disagreements by performing an additional analysis of the chart reviewer comments (Table 3). The most common reasons for disagreement were non-specific signs and symptoms resulting in different interpretations for the etiology of the patient presentation, “differences in navigating the EHR" resulting in reviewer’s reaching different diagnoses based on the different pieces of data they had found, and incomplete workup making reaching the diagnosis challenging for reviewers.

Table 3–

Analysis of Discordant Double-Reviewed Cases

Reason for disagreement	Count
Non-Specific Signs/Symptoms	15
Differences in Navigating the EHR	10
Insufficient data collection by routine clinical care	7
Fragmented care delivered at multiple sites	5
Differences in Clinical Interpretation of the universal definition	5
Reversible Cardiac Conditions	4
Ejection Fraction across different imaging modalities	3
Lack of Corroborating Data	2

Discussion

We found that there are challenges in identifying the diagnosis of heart failure when relying on a retrospective review of the EHR. Despite serving as a “gold standard” for validation of heart failure cohort definitions,^5,7 our clinician chart review resulted in a Cohen’s κ of 0.62. While this κ indicates substantial clinician agreement, it is likely insufficient for reliable heart failure identification.²⁰ We found that challenges and discrepancies in clinician chart review arose from two main issues – (1) clinicians had difficulty in applying the formal definition of heart failure to patients’ health records, especially when symptoms were vague, workups were incomplete, or clinicians interpreted findings differently and (2) the EHR itself posed obstacles, particularly when important information was buried in hard-to-find places or inaccessible across different hospitals. These issues undermined clinicians’ ability to confidently determine whether a patient truly had a history of heart failure. Our findings highlight important limitations in relying on the EHR for accurate case identification in both research and clinical settings.

While use of the Universal Definition of Heart Failure empowered our reviewers to assess for heart failure in a manner consistent with expert opinion, we encountered several challenges in practical application. The definition requires multiple components to establish the diagnosis, including evidence of structural or functional cardiac abnormalities, elevated natriuretic peptides, and objective measures of pulmonary or systemic congestion. However, we frequently encountered incomplete heart failure work-up by the clinical care team that prevented fulfillment of all diagnostic criteria. For example, some patients had elevated natriuretic peptides without echocardiograms to assess for the presence of structural cardiac disease. This pattern is consistent with findings in the literature, which show that patients with elevated BNP frequently lack follow-up left ventricular ejection fraction studies as is recommended by the heart failure guidelines.^21–23 Furthermore, we encountered variability in the objective measurements, specifically of left ventricular ejection fraction measurements across different imaging modalities including resting transthoracic echocardiography and MRI. Ejection fraction measurements are highly variable across various cardiac imaging modalities, highlighting the inherent challenges in applying standardized definitions to real-world clinical data.²⁴ Although we used resting transthoracic echocardiograms and cardiac MRIs as our gold standard for ejection fraction measurement, we also considered other modalities for patients without resting transthoracic echocardiograms or cardiac MRIs such as transesophageal echocardiogram, nuclear imaging, and cardiac catheterizations (Table 1).

Our reviewers’ challenges in applying the Universal Definition reflect broader limitations in the prevailing approach to heart failure classification, which continues to treat HF as a single disease entity primarily differentiated by ejection fraction.^25,26 This framework oversimplifies a biologically heterogeneous syndrome which represents a common pathway for multiple cardiac and non-cardiac pathophysiologic processes, including metabolic syndrome, chronic kidney disease, and valvular heart disease, ultimately manifesting as a multisystem syndrome rather than a uniform disease. Its multisystem nature results in overlapping clinical presentations with other conditions, a feature that contributed substantially to reviewer discordance in our double-reviewed cases (Table 2). These conceptual limitations are compounded by our reliance on imprecise diagnostic metrics. Left ventricular ejection fraction is subject to significant measurement variability and may remain preserved despite impaired contractility, particularly in the setting of concentric remodeling.²⁷ Emerging classification frameworks incorporating machine-learning clinical phenotyping and measures such as myocardial strain have been proposed, but have not yet been translated into standardized diagnostic constructs applicable to large-scale clinical research.^26–29

While the EHR has revolutionized the practice of medicine and storage of patient information, our findings highlight significant limitations in data quality, interoperability, and usability. Existing literature has demonstrated that these EHR quality and usability issues can lead to downstream consequences including patient safety risks.^30,31 For instance, we observed multiple charts in which a heart failure diagnosis was copied forward in provider notes without supporting clinical evidence. These examples correspond to evidence that 46% of the content in EHR notes is copied from other notes and only 18% of the text is original and entered manually.³² The use of copy and paste can impair clinical care by propagating errors such as incorrect diagnoses, introducing inaccuracies when text is copied into the wrong chart or in misleading context, creating internal inconsistencies, and contributing to note bloat which compromises clinical efficiency.³³ This reliance on the copy-paste functionality has serious patient safety ramifications as it is estimated to contribute to more than one-third of diagnostic errors in primary care settings.³⁴ Meanwhile, our issues with buried, hard-to-find, data elements correspond to well-documented EHR usability challenges. One such issue is the problem of availability of information where obtainment of clinically relevant data is hindered because it is entered or stored in the wrong location or is otherwise inaccessible within the EHR system.³⁵ These usability issues contribute to 9–33% of EHR related patient safety events.^30,35 Further complicating data accessibility is the interoperability challenge related to fragmented care.³⁵ Most health systems do not share comprehensive EHR-level data, creating information silos that prevent complete clinical picture assembly.³⁶ Both the usability and interoperability challenges translate directly into clinical consequences, including diagnostic delays, unnecessary duplicate testing, and fragmented care coordination.^37–42 This can be particularly detrimental for complex conditions like heart failure where timely diagnosis and treatment optimization are critical for patient outcomes. Lastly, we found significant variability in our reviewers’ ability to locate critical information within the same EHR system because of differences in familiarity with search functions and external data access tools such as “Care Everywhere”, the health information exchange functionality embedded in Epic. We addressed this by hosting several abstractor meetings and maintaining active email correspondences throughout the process. Numerous studies have found that clinical providers vary significantly in their EHR navigation patterns and documentation times.^43–46 This has important implications for clinical practice, as increased time spent in the EHR is associated with higher rates of physician frustration and burnout.^47–49 Despite the substantial variations in EHR usage patterns amongst clinicians, data on chart abstraction for research finds that only 20% of studies describe abstractor training,⁵⁰ and only 9% of studies describe processes to monitor abstractors to resolve disputes or review coding rules.⁵¹

Solutions to address these EHR-based shortfalls should be multifaceted. On the federal level, policies have been enacted to begin to address the issues related to care fragmentation and resultant incomplete information in the EHR. This effort began with incentivizing EHR adoption through the Health Information Technology for Economic and Clinical Health Act in 2009⁵², progressed to enforcing standardized, API-driven data exchange under the 21st Century Cures Act⁵³, and has more recently shifted toward building a nationwide interoperability infrastructure through frameworks such as the Trusted Exchange Framework and Common Agreement⁵⁴. While interoperability policies have enabled technical data sharing, they have failed to address fundamental issues of data quality, information accuracy, and the persistent barriers created by incompatible systems and inconsistent data management practices across healthcare networks.⁵⁵ Our findings suggest that despite these policy efforts, significant gaps remain in ensuring clinicians can access reliable and current diagnostic information when caring for patients from other institutions.

Technological innovations may also help to address some of the gaps we found. For instance, natural language processing (NLP) can be incorporated to leverage unstructured data, including provider notes, radiologic reports, and echocardiograms. This can be used to extract key clinical concepts, summarize important findings or diagnoses, or support downstream decision making.⁵⁶ As an example, Penn Medicine used NLP and optical character recognition tools to build the coordn8 platform which has automized the processing of healthcare information included in electronic faxes used to transmit patient records.⁵⁶ The platform is used to classify documents, separate compiled documents into distinct sub-documents (such as labs and images), filter duplicate or spam documents, and identify relevant patient information. This technology can address challenges we encountered with our human chart reviewers including “hidden data elements” within the EHR and provider differences in navigating the EHR.⁵⁶ Meanwhile, Stanford Medicine recently piloted a ChatEHR function which embeds a large language model with a chat into its EHR which clinicians can use to summarize a patient’s individual health records.⁵⁷ Nonetheless, further work is needed to determine whether these technologies can mitigate the data availability and usability issues that we observed. Such evaluations must be mindful of the data limitations and potential for bias that were seen even in manual chart review.

We employed a rigorous and iterative approach to ensure that chart abstractors applied consistent criteria when diagnosing heart failure. We believe this strategy aligns with current best practices for chart abstraction and may provide practical guidance for investigators undertaking similar studies. First, we developed a detailed reviewer guideline and chart abstraction tool that incorporated both structured and unstructured data fields with standardized definitions (Table 1). Second, all abstractors independently reviewed the same initial sample of 10 charts, followed by multiple meetings and email exchanges to resolve ambiguities and ensure consistent interpretation of the abstraction protocol. Third, although some procedures were specific to the Epic electronic health record, we ensured that all reviewers were familiar with key search features, including the search bar and Care Everywhere, to facilitate record review across institutions. Fourth, we applied the recently released Universal Definition of Heart Failure, which provides one of the most specific and objective diagnostic frameworks currently available. Finally, we incorporated heart failure specific clarifications within the guideline document to further standardize abstraction, including an ejection fraction hierarchy, standardized lists of signs and symptoms, outpatient and inpatient BNP and NT-proBNP thresholds, and a list of cardiac abnormalities supporting the diagnosis.

Our study has several limitations. First, we relied on one EHR vendor from one health system, thus findings may not be generalizable to other settings. Nonetheless, Epic is the most commonly used EHR vendor in the United States.^58,59 While we were unable to determine whether the challenges we faced are specific to our single institution and EHR, we believe many of the challenges, including limitations in data availability and user-centered design, are common to most health systems and EHR vendors.⁶⁰ Second, only 35% of our charts were double reviewed. Given the low agreement rate (82.5%), ideally all cases would have been double reviewed, however we were unable to do so because of resource limitations. Additionally, it is possible that our findings of a Cohen’s κ of 0.62 may have been related to the kappa paradox. However, given that the prevalence of heart failure among double-reviewed charts was moderate (69%), the kappa paradox is unlikely to substantially underestimate agreement.⁶¹ Unfortunately, direct benchmarking against existing HF phenotyping literature is limited by a widespread paucity of formal inter-rater reliability reporting, as most validation studies rely on single-abstractor or consensus-driven frameworks. Third, our reviewers had persistent differences navigating the EHR even after our multiple check-ins, which likely contributed to the low agreement rate. This highlights a problem with our methodology, and our reviewers could have benefited from more explicit EHR navigation instructions that could have been included in the reviewer guide in Table 1. Nonetheless, this finding highlights how the differences amongst clinician EHR navigation patterns could impact a determination of a clinical diagnosis. Fourth, our qualitative analysis relied on notes taken from meetings and chart reviews rather than actual transcripts of interviews and meetings or standardized post-review surveys, so we may have missed unrecorded content and reviewer insights. Furthermore, because both coders participated in the original chart review, their analyses may have been subject to interpretive bias. Fifth, we were unable to determine reviewer agreement across important subgroups such as inpatient versus outpatient encounter or presence versus absence of EF due to reviewer discordance within these strata.

In conclusion, the creation of the EHR has profoundly impacted all aspects of medicine including clinical care delivery, quality improvement initiatives, medical research, and health policy creation. While patient cohorts are often defined by ICD codes and structured data (such as lab values), these cohorts are then validated by clinician chart review, which is considered the gold standard for diagnosing from the EHR. Our study highlights the inherent limitations of this gold standard, particularly when studying a clinical syndrome as complex as heart failure. Given the current move toward artificial intelligence and machine learning, which rely on gold standard annotated datasets, these findings have wide-ranging implications.^57,62 Beyond the research implications, the diagnostic disagreement among clinicians during retrospective review highlights the barriers in using the EHR to confirm heart failure diagnoses for real-world clinical decision-making, which hinders optimal patient care. Policies and technological solutions aimed at improving EHR data collection, organization, and data extraction are needed.

Abbreviations List:

EHR

Electronic Health Record

ICD

International Classification of Diseases

NLP

Natural Language Processing

EF

Ejection Fraction

MRI

magnetic resonance imaging