Natural language processing of implantable cardioverter-defibrillator reports in hypertrophic cardiomyopathy: A paradigm for longitudinal device follow-up

Abstract

Background

The follow-up of implantable cardioverter-defibrillators (ICDs) generates large amounts of valuable structured and unstructured data embedded in device interrogation reports.

Objective

We aimed to build a natural language processing (NLP) model for automated capture of ICD-recorded events from device interrogation reports using a single-center cohort of patients with hypertrophic cardiomyopathy (HCM).

Methods

A total of 687 ICD interrogation reports from 247 HCM patients were included. Using a derivation set of 480 reports, we developed a rule-based NLP algorithm based on unstructured (free-text) data from the interpretation field of the ICD reports to identify sustained atrial and ventricular arrhythmias, and ICD therapies. A separate model based on structured numerical tabulated data was also developed. Both models were tested in a separate set of the 207 remaining ICD reports. Diagnostic performance was determined in reference to arrhythmia and ICD therapy annotations generated by expert manual review of the same reports.

Results

The NLP system achieved sensitivity 0.98 and 0.99, and F1-scores 0.98 and 0.92 for arrhythmia and ICD therapy events, respectively. In contrast, the performance of the structured data model was significantly lower with sensitivity 0.33 and 0.76, and F1-scores 0.45 and 0.78, for arrhythmia and ICD therapy events, respectively.

Conclusion

An automated NLP system can capture arrhythmia events and ICD therapies from unstructured device interrogation reports with high accuracy in HCM. These findings demonstrate the feasibility of an NLP paradigm for the extraction of data for clinical care and research from ICD reports embedded in the electronic health record.

Key Findings.

• •An automated natural language processing (NLP) process applied on interrogation reports can extract information regarding arrhythmias and device therapies (antitachycardia pacing, shocks) with high accuracy in patients with hypertrophic cardiomyopathy and implantable cardioverter-defibrillators (ICD).
• •The NLP process based on the narrative, free-text interpretation of the device interrogations had superior diagnostic performance compared to a system based on structured tabulated data of arrhythmia and ICD therapy events.
• •This approach provides an example of an NLP tool that can enable the precise and efficient longitudinal tracking of clinically important events in the care of patients with ICDs, but also for research applications.

Introduction

Hypertrophic cardiomyopathy (HCM) is the most prevalent genetic cardiomyopathy and is one of the most common causes of sudden cardiac death (SCD) in children, adolescents, and young adults.1, 2, 3 An implantable cardioverter-defibrillator (ICD) is recommended for HCM patients with history of sustained ventricular arrhythmias or resuscitated SCD, and for the primary prevention of SCD in a subset of high-risk HCM patients.^4,5 After implantation, ICDs are interrogated routinely during clinic visits or by remote follow-up, generating large amounts of data of ICD-detected atrial and ventricular tachyarrhythmias, and ICD therapies, including shocks and antitachycardia pacing (ATP). In most clinical environments, ICD-recorded data are incorporated in dedicated reports in the electronic health record (EHR) and provide important information to the care teams. These reports include structured and unstructured (free-text) data fields. The absolute counts of events are documented numerically in dedicated fields (structured data), but the interpretation of such data by ICD clinic providers is documented typically in narrative clinical text in these reports (unstructured data).

Atrial and ventricular arrhythmia events are prognosis-defining events in the natural history of HCM. The presentation of large volumes of arrhythmia data in numerous reports accumulating over years of follow-up of HCM patients6, 7, 8, 9 can make it potentially difficult to accurately track and use them in clinical decision-making, particularly for providers without expertise regarding cardiac implantable electronic devices. The accurate and efficient retrieval of information about past arrhythmia events from the EHR is critical for both clinical and research purposes. In this study, we aimed to develop and test a fully automated natural language processing (NLP) using unstructured data as compared to an automated model using structured data for capture of arrhythmia events from ICD interrogation reports embedded in the EHR.

Methods

Patient cohort

The patient cohort was created by using the Mayo Clinic digital data vault. We chose to develop this model in an HCM population because of the longstanding established HCM cohort in our institution and the relatively high rates of atrial and ventricular arrhythmias in HCM that allows for a sufficient number of events for model training and testing. We identified 11,690 patients with an International Classification of Diseases, Ninth Revision (ICD-9) or International Classification of Diseases, Tenth Revision (ICD-10) diagnosis code for HCM between 1995 and 2019 (Table 1). Of them, 1,127 patients had a diagnosis code for presence of ICD or procedure code of ICD implantation. A random sample of 247 patients with HCM and an ICD comprised the study cohort (mean age 50.4 ± 16.2 years; 31% women; Table 2). A total of 687 ICD interrogation reports were analyzed (median 2 per patient, interquartile range 1–3). The study was approved by the Mayo Clinic institutional review board. Patient informed consent requirement was waived owing to the retrospective nature of the study. The study was conducted in accordance with the Declaration of Helsinki.

Table 1.

Criteria used for selection of the patient cohort

Description	Coding system	Specific codes
Patients with HCM	ICD-9-CM	425.11, 425.18
	ICD-10-CM	I42.2, I42.1
AND
Implantable cardioverter-defibrillator diagnosis codes	ICD-9-CM	996.04, V45.02, V53.32
	ICD-10-CM	Z45.02, Z95.810
OR
Implantable cardioverter-defibrillator procedural codes	Current Procedural Terminology	0319T, 0321T, 0326T, 33223, 33230, 33231, 33240, 33241, 33243, 33244, 33249, 33262, 33263, 33264, 33271, 33272, C1721, C1722, C1882
	Healthcare Common Procedure Coding System	C1777, C1895, C1896, C1899

ICD-9-CM = International Classification of Diseases, Ninth Revision, Clinical Modification; ICD-10-CM = International Classification of Diseases, Tenth Revision, Clinical Modification; HCM = hypertrophic cardiomyopathy.

Table 2.

Patient characteristics

Characteristic	Total (N = 247)
Age† (years), mean (SD)	50.4 (16.2)
Female, n (%)	76 (31)
Body mass index (kg/m2)	29.8 (25.3–32.2)
LV ejection fraction (%)	63 (45–70)
LV mass index (g/m2)	141 (113–178)
Septal thickness (mm)	15 (11–21)
Posterior wall thickness (mm)	11.8 (2.8)

SD = standard deviation; LV = left ventricular.

Continuous variables are shown as median (interquartile range) unless they were normally distributed, in which case they are shown as mean (SD).

^†At the time of first echocardiogram.

Extraction of arrhythmia and ICD therapy data

For the extraction of arrhythmia and ICD therapy data from the eligible study population, we used MedTagger (https://github.com/medtagger/MedTagger), an NLP system capable of autonomously extracting clinical events from unstructured, free text when provided with a clinical dictionary and rule set.¹⁰ The system was fine-tuned to use device interrogation reports and extract numerical tabulated information on sustained atrial and ventricular arrhythmias and ICD therapies delivered (ATP or shock), as well as the interpretation of whether the ICD therapy was appropriate (ventricular tachycardia [VT]- / ventricular fibrillation [VF]-terminating) or inappropriate. The processed ICD reports have the same format across all device manufacturers. For the purposes of developing the MedTagger dictionary used for ICD therapy extraction, we randomly selected 70% of the 687 reports for training of the models (n = 480) and the remaining 30% (n = 207) reports were included in the testing dataset. For patients with more than 1 device interrogation report, all reports were included. The randomized assignment of reports to the training and testing datasets was performed at the report level and not the patient level. Thus, 80 of 247 patients had ICD reports in both the training and testing datasets.

Reference standard for model training and testing

Through manual review of the 687 ICD interrogation reports of HCM patients with an ICD, we defined the below reference labels:

• •Arrhythmias: We extracted data on sustained atrial and ventricular arrhythmias. If an interrogation report contained any arrhythmia information (ie, “ventricular fibrillation,” “ventricular tachycardia,” “atrial fibrillation,” “atrial flutter,” “atrial tachycardia,” “supraventricular tachycardia”), these were classified as arrhythmias. Nonsustained arrhythmias did not qualify for inclusion.
• •ICD therapy: Our annotation task for ICD therapies was to determine whether appropriate ICD therapy was delivered or not based on the interpretation by the device clinic provider. We considered an appropriate ICD therapy when an ICD interrogation report indicated that an ATP sequence or ICD shock was delivered for VT or VF (for example, “Patient has had 2 VT episodes treated successfully with ATP × 1 each”). In contrast, ICD therapy was considered inappropriate when it was delivered for arrhythmia other than VT or VF within the programmed therapy rate zones, or owing to oversensing of cardiac or noncardiac signals (for example, “Patient received 3 failed ATP therapies, and electrograms showed atrial fibrillation with rapid ventricular rate”).

Arrhythmia detection was per the detection parameters (such as episode duration and rate) programmed for each individual device. Review of ICD electrograms to ascertain arrhythmia type and appropriate vs inappropriate shock was not performed. Two trained abstractors independently annotated each interrogation for identifiable arrhythmias and appropriate ICD therapy. Any disagreement of annotations was adjudicated by a third abstractor, a board-certified cardiologist (A.A.O.). The inter-abstractor agreement of the annotation was calculated with percentage agreement and the kappa statistic based on a randomly selected sample of 69 interrogation reports (10% of the total 687 interrogation reports).

Model development and evaluation

A process was developed for automated aggregation of structured arrhythmia and ICD therapy data from device interrogation report tables. For development of the separate NLP model based on free-text data, patterns of extracted ICD therapy information as well as the specific type of arrhythmia were identified and incorporated into the MedTagger ruleset. The system extracts concepts and normalizes them to the target phenotypes in our evaluation. For example, “ATP therapy” and “shock” concepts were normalized into “ICD therapy,” whereas “ventricular fibrillation,” “ventricular tachycardia,” “atrial fibrillation,” and “atrial tachycardia” concepts were all normalized into “arrhythmia.” Figure 1 shows examples of structured and unstructured data fields from device interrogation reports. All pertinent information for the development of the structured data and unstructured data (NLP) processes was extracted from the interrogation reports. The data included in these reports (both free-text and structured data) are accessible through the Unified Data Platform. Over the study period there were 2 different databases where ICD report data were stored. The original database (“Historic ICD”) was established in the 1980s and continued until 2014. The more recent database (“Optima”) was established in 2007 and is still in use. Therefore, there are 2 different data formats from which the information was extracted. Data from both formats were aggregated into a common format and the aggregated data were then used to create the structured data and NLP processes. Therefore, we would not anticipate any differences in performance depending on the originating database of the reports. Data aggregation was evaluated iteratively by manual review of random samples of at least 20 reports of each format.

Figure 1.

Examples of structured and unstructured (free-text) data fields in device interrogation reports. AF = atrial fibrillation; APVP = A pace, V pace; APVS = A pace, V sense; ASVP = A sense, V pace; ASVS = A sense, V sense; ATP = antitachycardia pacing; bpm = beats per minute; CRT = cardiac resynchronization therapy; LV = left ventricle; NSVT = nonsustained ventricular tachycardia; RA = right atrium; RV = right ventricle; SVT = supraventricular tachycardia; VF = ventricular fibrillation; VSVT = very slow ventricular tachycardia; VT = ventricular tachycardia.

The performance of the NLP pipeline based on the free-text narrative interpretation of ICD interrogation data and the system based on information aggregated from structured data alone were evaluated against the reference standard in the testing dataset. An exact match with the reference standard annotation (“arrhythmia event” yes/no; “ICD therapy” yes/no) was required for both the NLP and the structured data processes. Diagnostic performance metrics for the NLP tool and for the approach based on structured data were calculated, including sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), and F1-score.

Results

Analyzed ICD interrogation reports

In the overall study dataset of 687 ICD reports, the manual review (reference approach) identified 327 reports with at least 1 arrhythmia event (48% of reports) and 290 reports with at least 1 appropriate ICD therapy event (42% of reports). Of the arrhythmia events, 158 (48%) were supraventricular tachycardias, including atrial fibrillation, and 230 (70%) were VT/VF. In the manual annotation process, inter-reviewer agreement was 91% and the kappa statistic was 0.82 for arrhythmia events, whereas agreement and kappa statistic for ICD events were 91% and 0.80, respectively. Of 687 reports, 481 (70%) were in the “Optima” database and 206 (30%) were in the “Historic ICD” database. In the training set, 332 (70%) reports were in “Optima” and 145 (30%) were in “Historic ICD.” In the testing set, 146 (71%) reports were in “Optima” and 61 (29%) reports were in “Historic ICD.”

Model performance

The fully automated NLP model based on unstructured data exhibited sensitivity and specificity exceeding 90% for the identification of information for arrhythmia events and for ICD therapies in reference to manual reviewer assessment in the testing set. F1-scores were 0.98 and 0.92 for arrhythmia events and ICD therapies, respectively (Table 3). The performance of the approach based on only structured data fields demonstrated a lower diagnostic performance. Sensitivity was 0.33 and 0.76 for identification of arrhythmia events and ICD therapies, while F-scores were 0.45 and 0.78, respectively. There were no differences in the performance of the NLP and structured data processes for reports originating from the older “Historic ICD” database as opposed to the newer “Optima” database.

Table 3.

Diagnostic performance of a process using structured data alone or an NLP system based on unstructured (free-text) data for arrhythmia event and ICD therapy extraction

Model type	Extraction task	TP	FP	TN	FN	Sensitivity	Specificity	PPV	NPV	F1-score
Structured data	Arrhythmia event	33	14	93	67	0.33	0.87	0.70	0.58	0.45
	ICD therapy	61	15	112	19	0.76	0.88	0.80	0.85	0.78
Text data extracted by NLP	Arrhythmia event	98	2	105	2	0.98	0.98	0.98	0.98	0.98
	ICD therapy	79	13	114	1	0.99	0.90	0.86	0.99	0.92

FN = false-negative; FP = false-positive; ICD = implantable cardioverter-defibrillator; NLP = natural language processing; NPV = negative predictive value; PPV = positive predictive value; TN = true-negative; TP = true-positive.

Error classification

The NLP model provided incorrect determination for either arrhythmia (n = 4) or ICD therapy identification (n = 14) in a total of 18 of the 207 (9%) reports in the testing dataset. In Table 4, we categorize the cases of incorrect output by the NLP model and we provide verbatim examples of the free-text fields from the respective ICD interrogation reports. Most cases of incorrect output by the NLP model were due to false classification of defibrillation threshold testing as appropriate clinical ICD therapy event or due to incorrect classification of nonsustained arrhythmias as qualifying arrhythmia events.

Table 4.

Natural language processing model error analysis

Error type	Number of cases	Verbatim examples	Classification
Inconsistent format	7	14J: Good detection, no dropped beats.	False Positive for ICD Therapy (misclassification of DFT testing as appropriate clinical ICD therapy event)
Concept invalidity	3	The available EGM shows ventricular sensed beats at 52 bpm with a sudden rate increase to 120 - 136 bpm and a distinct morphology change which could indicate slow NSVT.	False Positive for Arrhythmia Event
Nonrepresentative pattern	5	Patient broke rhythm before shock delivered.	False Positive for ICD Therapy
Nonrepresentative keyword (eg, typo)	2	Used 200 j externqal [typo of external] shock to convert rhythm to sinus.	False Positive for ICD Therapy
Negation	1	The transmission was reviewed, and showed no new VT/VF episodes.	False Positive of Arrhythmia Event

bpm = beats per minute; DFT = defibrillation threshold; EGM = electrogram; ICD = implantable cardioverter-defibrillator; NSVT = nonsustained ventricular tachycardia; VF = ventricular fibrillation; VT = ventricular tachycardia.

Discussion

In this study, we developed an automated NLP system for the extraction of arrhythmia and ICD therapy data from ICD interrogation reports integrated in the EHR in patients with HCM. Our system achieved F1-scores of 0.98 and 0.92 for arrhythmia and ICD therapy event identification, respectively. The performance of the NLP model was superior to that of an automated system based on extraction of data from structured fields. To the best of our knowledge, NLP has not been applied previously to ICD interrogation reports. This is the first study that uses narrative clinical text in device interrogation reports from EHRs for comprehensive phenotyping of arrhythmia history in patients with ICDs.

We designed this study to test whether an automated NLP system could be used to increase the efficiency of detection of ICD-recorded events. We chose ATP and ICD discharges because they are clinically important arrhythmia endpoints in patients with ICDs and are undisputable events when ascertained by manual review. Our results demonstrate that the automated NLP process can identify ICD events with high sensitivity and could substantially reduce reviewer burden, with implications both for longitudinal clinical care but also for research. While certain types of cardiac arrhythmias may be represented with ICD-9/10 codes in the medical record, ICD device therapies delivered for treatment of specific types of cardiac arrhythmias cannot be represented by ICD-9/10 diagnosis codes. This is because there are no billing codes (ICD-9/10) for ICD device therapy (shock or ATP). Therefore, an NLP system represents an automated way for capture of device therapy events.

Patients with HCM and ventricular arrhythmias constitute a high-risk subgroup susceptible to future major cardiac events and an unfavorable prognosis.¹¹ In addition, atrial arrhythmias are a well-known predictor of adverse clinical course, including stroke, in patients with HCM.¹² Knowledge of a past history of ventricular or atrial arrhythmias is critical in clinical decision-making, and incident arrhythmias also reflect a clinical status change that usually requires action. Even though patient-provided clinical history can reveal prior ventricular and atrial arrhythmia events, many HCM patients may receive an ICD at a young age, resulting in many years of follow-up such that an accurate recollection and documentation of ICD-detected arrhythmias is challenging over long periods of time. In the current clinical environment, data regarding ICD-recorded events are obtained through time-consuming chart review where the clinician reviews clinical reports and ICD interrogations trying to distill the arrhythmia history. The system developed in this study allows the accurate and efficient automated capture of past arrhythmia history without manual intervention. Such a system could be incorporated as a digital phenotyping tool in the EHR, not only for retrospective identification of arrhythmia, but also as a prospective real-time tool for patient care; and it might improve efficiency of patient care, especially for non-electrophysiologists who may find it challenging to review large amounts of complicated arrhythmia reports. It can also help design a clinical decision support system to enable delivery of high value of care. While this cohort consisted of HCM patients with ICDs as a proof-of-concept, the automated system illustrated here may be useful for identifying ICD-recorded events in patients with ICDs implanted for any indication. Finally, in theory such a system may provide a tool for surveillance of device integrity by monitoring lead parameters longitudinally, such as impedance, sensing, and capture thresholds, and alert clinicians before a clinically significant lead failure occurs.

Beyond clinical practice, this system has potential value as a research tool. Current data on arrhythmia history, incident arrhythmia risk, and prognosis of HCM patients with ICDs are largely derived from manually assembled and maintained datasets. Our algorithm can help researchers extract large amounts of data in patients with ICDs and update datasets continuously. Coupled with other automated systems with good performance, these data can be used to identify predictors of ICD events in patients with HCM, investigate the natural history of atrial and ventricular arrhythmias, and assess the prognosis associated with such arrhythmias in HCM and other cardiomyopathies.

For the system using structured data only, we observed that there was significantly lower performance in terms of both sensitivity and specificity compared to the NLP system using unstructured data. A significant contributor to the lower specificity of structured device data lies in the fact that it could not distinguish appropriate ICD therapies for clinical arrhythmias from ICD shocks during defibrillation threshold testing. Also, arrhythmia or ICD therapy counters may not be reset between various device interrogation sessions, thus giving the false impression of recurrent events occurring between device interrogations. Hence, while standardizing structured information, a loss of granularity adversely affects extraction of relevant clinical information.13, 14, 15 The NLP system reported herein mitigated these limitations of structured data analysis.

Limitations

Our system has common NLP challenges, such as dealing with nonrepresentative keyword/pattern (eg, typo) and contextual errors in long sentences (eg, negation or certainty). Structured data elements from the ICD may vary over time owing to changes in format of device reports and corresponding data tables stored in the data warehouse. In mitigation, data aggregation approaches were adapted for report format. The conclusions drawn here remain generalizable regardless of how the structured data were aggregated. In this study, detailed device tracing analysis was not performed for ascertainment of events. However, event tracings are used routinely by device clinic providers for the interpretation of ICD interrogation data and this is reflected in the superior accuracy of the unstructured data model reported herein. Finally, the current model was developed in a single-center clinical environment. The external validity of the model in other institutions/settings would require further study.

Conclusions

Through the comparison of models based on structured data and unstructured text of interrogation reports for extracting phenotyping information of arrhythmias and ICD therapies, we have demonstrated that structured data alone do not represent the underlying information in the real-world clinical context accurately. An automated NLP process based on unstructured data analysis can extract arrhythmia and ICD event information with high accuracy and may provide a precise and efficient tool for longitudinal clinical follow-up of patients with ICDs and for research applications.

Funding Sources

The research reported in this publication was supported by the National Heart, Lung, and Blood Institute of National Institutes of Health (K01HL124045), Mayo Clinic K2R award and by the Robert D. and Patricia E. Kern Center for the Science of Health Care Delivery Augmented Human Intelligence Care of Tomorrow Award. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health. This work was also supported by the Paul and Ruby Tsai and Family Hypertrophic Cardiomyopathy Research Fund.

Disclosures

The authors have reported that they have no relationships relevant to the contents of this paper to disclose.

Authorship

All authors attest they meet the current ICMJE criteria for authorship.

Patient Consent

Patient informed consent requirement was waived owing to the retrospective nature of the study.

Ethics Statement

The authors designed the study and gathered and analyzed the data according to the Helsinki Declaration guidelines on human research. The research protocol used in this study was reviewed and approved by the institutional review board.