Can antiepileptic efficacy and epilepsy variables be studied from electronic health records? A review of current approaches

Abstract

As automated data extraction and natural language processing (NLP) are rapidly evolving, improving healthcare delivery by harnessing large data is garnering great interest. Assessing antiepileptic drug (AED) efficacy and other epilepsy variables pertinent to healthcare delivery remain a critical barrier to improving patient care. In this systematic review, we examined automatic electronic health record (EHR) extraction methodologies pertinent to epilepsy. We also reviewed more generalizable NLP pipelines to extract other critical patient variables.

Our review found varying reports of performance measures. Whereas automated data extraction pipelines are a crucial advancement, this review calls attention to standardizing NLP methodology and accuracy reporting for greater generalizability. Moreover, the use of crowdsourcing competitions to spur innovative NLP pipelines would further advance this field.

INTRODUCTION

Rationale:

Epilepsy affects 60 million people worldwide.¹ Anti-epileptic drugs (AEDs) are first-line therapy for epilepsy and control seizures in two thirds of patients.² More than 25 AEDs are now available and rational “trial and error” often determines drug choice, as comparative data on efficacy between AEDs remains limited. Despite millions of people taking AEDs daily, retrospective and prospective chart review studies comparing AEDs head-to-head are only available for a limited number of medications. Moreover, small samples sizes limit interpretation of these studies. A wealth of information regarding AED efficacy lies within electronic health records (EHRs), yet efficient data extraction has remained a critical barrier to closing this knowledge gap.

Overview:

The goal of this report is to explore how pertinent data can be automatically extracted from the EHR for studies of AED efficacy and other pertinent epilepsy variables relevant to improving patient care. The main exposure variable is AED prescription. The outcome variable of seizure frequency can be assessed by comparing pre- and post-drug seizure frequencies. To compare two or more AEDs, cohorts of patients taking the AEDs of interest can be matched on relevant covariables to minimize confounding and relative changes in seizure frequency over time can be measured.

Patient characteristics that define the clinical context (Table 1) are important to include in the analysis, as they have potential to cause confounding effects in multiple ways. Demographic variables, such as socioeconomic status (SES), comorbidities (including psychiatric conditions such as substance abuse), and medication allergies may play a role in how providers choose which AED to trial, exacerbating nonrandom assignment to treatment groups. Underlying epilepsy etiology, baseline epilepsy severity, and age of onset may also correlate with refractoriness to AED treatment. Furthermore, underlying cause of epilepsy may have interaction effects with particular medications, such as the indication for broad-spectrum agents in the treatment of generalized epilepsy.

TABLE 1: KEY VARIABLES:

If these variables could be extracted from the electronic health record automatically, an antiepileptic drug efficacy study could be performed.

1) Exposure:

a) Prescribed medication extraction (S+U)

2) Covariables:

a) Age, gender, & ethnicity (S)

b) Socioeconomic status by zip code, type of insurance (S)

c) Epilepsy type (U)

d) Seizure type (U)

e) Underlying cause of the epilepsy (S+U):

i) genetic syndrome

ii) structural lesions

(1) strokes

(2) tumors

(3) surgery

(4) bleeding

iii) autoimmune disease

iv) neurodegenerative disease

v) traumatic brain injury

f) Age of onset (S)

g) Medication allergies (S)

h) Co-morbid diseases, aka past medical history (S+U)

i) Refractory epilepsy: as defined by number of other medications tried, or surgery tried

j) Epilepsy risk factors (S+U)

i) Abnormal birth and development (e.g. prematurity, developmental delay, autism)

ii) History of brain infection

iii) Substance abuse

iv) Family history of epilepsy

v) History of febrile seizure

k) Prior EEG abnormalities (U)

3)  Outcome: seizure frequency and surrogates thereof

a) Seizure frequency (U)

b) Use of rescue medications (S+U)

c) Need for increase in therapy: e.g. doses increased, new meds added (S+U)

d) Use of the nursing telephone help-line (S+U)

e) Use of the emergency room (S+U)

f) Hospital admissions (S+U)

g) Total health-care cost (calculated based on the above)

(S) = structured fields; (U) = Unstructured data; (S+U) = structured fields and unstructured data.

In this investigation, we performed a literature review of the currently available data extraction methods for the pertinent variables and examined techniques specific to epilepsy when appropriate to the variable (e.g., seizure frequency).³ For more generalizable variables (e.g., medication), we also reviewed methods developed outside the epilepsy field.

METHODS

Search strategy:

We performed a PubMed search for the variables of interest listed in Table 1. The specific PubMed search phrases are described in detail in the supplementary text. Abstracts were screened for relevance by employing a method of automated EHR extraction for one of the pertinent variables. Some manuscripts focused on the details of a method of extraction and its accuracy, while other manuscripts employed automated extraction methods as a means to describe a clinical outcome; both types of manuscripts were deemed relevant. Once a manuscript was deemed relevant, we reviewed and catalogued elements of the manuscript, including the extraction method summary, accuracy measures, and estimate of generalizability to epilepsy as applicable if the method was not specifically created for epilepsy. Citations within relevant manuscripts were also reviewed in the same manner.

Terminology:

Data in the EHR comprises of both structured and unstructured fields. Structured data fields use controlled vocabulary and limit the variability, which allows more inter-user consistency and accurate data aggregation. As examples of structured fields, blood pressure must be populated by exactly two numerical values (systolic and diastolic pressure), a medication field must be populated by a recognized medication name as selected from a standardized list, or a medical problem list must be populated by selecting from a list of Intelligent Medical Objects (IMOs) or International Statistical Classification of Disease codes (ICD codes).

Conversely, unstructured data components are composed of narrative text or prose written by the provider, usually in the setting of a progress note in various encounter settings. Telephone calls are also often documented with free text. Natural Language Progressing (NLP) uses computer algorithms to extract information from unstructured free text language. Simple forms of NLP use dictionaries (lists of terms or synonyms) and rules (pre-set sentence structures) to extract information. More complex forms of NLP use machine learning to create a classifier, which categorizes a note with the presence or absence of a particular variable. NLP algorithms and machine learning processes are often compiled into larger pipelines.

Accuracy measures are critical to assess the performance of each method. The most commonly reported measures of accuracy are precision, recall, and the F1 statistic for data retrieval. Precision is the proportion of retrieved data that is true (positive predictive value). Recall is the proportion of true data that is correctly retrieved (sensitivity). The F1 statistic is the geometric mean between precision and recall; this statistic ranges from zero to one, where the value of one indicates perfect accuracy. The area under the receiver operator curve (AUC) is a summary measure that combines sensitivity and specificity and ranges from zero to one, where a value of one indicates perfect accuracy. For any extraction algorithm measuring a given variable, precision and recall can be reported on a training set that was used to create the extraction algorithm or on a test set, which is independent and not used to develop the extraction algorithm.

This report will reference specific EHR extraction algorithms and pipelines, including complex NLP machine learning methods (please see table 2), and report their measures of accuracy. In this review, all statistics were assumed to be reported on independent test sets unless specifically stated otherwise. Further elaboration of specific methodology of the pipelines is beyond the scope of this review.

TABLE 2:

Acronyms and abbreviations for data standards and methods of machine learning

ACE score	Adverse Childhood Events scoring system
AUC	area under the curve (receiver operator)
Bi-LSTM	bidirectional LSTM
CNN	convolutional neural network
CRF	Conditional Random Field
cTakes	Clinical Text Analysis, Knowledge Extraction System
CUI	Concept unique identifier
EpSO	Epilepsy-Specific Language
GATE	General Architecture for Text Engineering
GUI	Graphical User Interface
HPO	Human Phenotype Ontology
ICD codes	International Statistical Classification of Diseases
HEDEA	Healthcare Data Extraction and Analysis
HL7	Health Level Seven International
LOINC	Logical Observation Identifiers Names and Codes
LSTM	Long short-term memory
MTL	Multi-task Learning
N-gram	a phrase of n words (e.g. a trigram is a 3word phrase)
NER	Named entity recognition
NLP	Natural Language Processing
NLTK	a python NLP toolkit
RxNorm	normalized names for clinical drugs
SNOMED-CT	a systematically organized computer processable collection of clinical terms
	providing codes, terms, synonyms and definitions
SVM	Support Vector Machines
UIMA	Unstructured Information Management Application
UMLS	United Medical Language System

RESULTS

Over 2000 articles were returned by the PubMed search criteria and screened for relevance. A total of 128 articles were deemed sufficiently relevant for detailed review.

EXPOSURE VARIABLES

Medication:

There are a variety of emerging techniques to extract medications from the EHR, in part due to publicly issued crowdsource challenges which have generated a large number of highly accurate pipelines.⁴ Medications recorded within the EHR have variable interpretations depending on the context, as records may include current medications, past medications, recommended future medications, or medication allergies, which makes extractions challening.⁵ Many analysts suggest methods to account for this, such dividing a clinical note into various sections (i.e., “current medications”, “allergies”, and “recommendations”).^6,7 The best approaches include analyzing a combination of structured fields and unstructured notes.⁸ This approach should account for potential differences in prescribing patterns among institutions and/or medication documentation in the EHR.^6,8

The best pipeline for extraction of active medication ingredients and dosing achieves greater than 97% F1s.^9,10 However, extracting the indication for each medication is more difficult, with the best group achieving F1 of 66%.¹¹ Extracting the indication is particularly important in the field of epilepsy, since many antiepileptic drugs may be prescribed for alternative indications (e.g. topiramate for migraine, oxcarbazepine for mood stabilization, etc.).

COVARIABLES

Age and Sex:

Age and binary sex are extracted reliably from structured fields.¹² Improved methods are needed to account for nonbinary gender designations.

Race and ethnicity:

Race and ethnicity are often extracted from structured fields.^12–16 However, extraction can be incompletely sensitive, and missing data tend to be biased, with underreporting of underrepresented socioeconomic demographics.^17–20 When ethnicity is present in structured fields, Denny et al report over 90% concordance with genetic ethnicity, which can be used as a gold standard for ethnicity extraction when available.^17,21 Sholle et al augmented structured fields with simple NLP to achieve an F1 of 91%.¹⁹

Socioeconomic status (SES):

Some authors were able to extract values from structured fields while others used relatively simple forms of NLP to improve data or population statistics to extrapolate based on geography, such as the use of postal codes to impute SES.²² Bejan et al captured “homelessness” with an AUC of 0.83.²³ Hatef et al showed that a simple NLP algorithm can supplement structured codes for “financial strain” and “housing issues” and increase recall by 10–15-fold over the use of ICD codes alone.¹⁸ Canadian census tract data leveraged by zip code using a combined deprivation index, which combines several ‘material’ and ‘social’ variables (such as income, education, living alone or with a spouse, etc.) has been used to derive a single measure of SES.²⁴ Hollister et al used simple search terms to extract “low education” (PPV 80%), occupation (85%), unemployment (88%), retirement (64%), uninsured status (23%), Medicaid status (82%), and homelessness (33%).²⁵

Epilepsy type and seizure type:

Focal epilepsy (FE), generalized epilepsy (GE), and unknown epilepsy (UE) can be discriminated by n-gram and an SVM classifier, as illustrated by Connolly et al who trained the model on data from one institution and tested it from two other institutions (F1 = 72%), which suggests reasonable generalizability. Performance further improved when trained on data from two institutions and tested on data from a third (F1 = 80%).²⁶ A method defined as PEEP (phenotype extraction in epilepsy) extracted epileptogenic zone, etiology, and EEG pattern from Epilepsy Monitoring Unit (EMU) discharge summaries. F1s ranged between 75% to 85% for exact matches for semiology, lateralizing signs, and EEG pattern, and up to 95% for epileptogenic zone. This is the best method reviewed for specific semiology and etiologies but performs only on EMU discharge summary note types.²⁷

The most comprehensive pipeline specific to the field of epilepsy is ExECT (extraction of epilepsy clinical text).²⁸ This extracts a diagnosis of epilepsy as a binary field (88% precision, 89% recall), focal seizures as a binary field (96% precision, 70% recall), generalized seizures as a binary field (89% precision, 52% recall), and epilepsy type as a trinary field, defined as either focal, generalized, or absence (90% precision, 80% recall).

Underlying cause of epilepsy:

A combination of approaches has been used, which include extracting text from unstructured neuroradiology reports, structured comorbid disease fields, and the unstructured impression section of clinician notes. For certain specific etiologies with no known automated EHR extraction (e.g. cortical dysplasia), raw image processing algorithms may be used.

The ExECT pipeline identified abnormal or epileptogenic imaging findings on CT (56% precision, 59% recall) and MRI (82% precision, 69% recall) from clinical encounters, though these were limited to binary yes/no variables.²⁸ For brain tumors, metastatic brain tumors identified from radiology notes achieved an AUC of 0.92 and best accuracy of 83%.²⁹

For cortical dysplasia, information extracted from raw imaging data successfully identified dysplasia locations in seven out of seven patients, using 30 controls as a reference.³⁰ For mesial temporal sclerosis, hippocampus and amygdala atrophy was identified from raw imaging data to within approximately 10% of the “gold standard” based on structural volume.^31,32 Castro et al improved on an initial screen of ICD codes with a simple dictionary and classifier NLP to achieve 86% PPV for cerebral aneurysm.¹² Combining ICD codes and NLP achieved 100% PPV and 94% NPV to identify venous thromboembolism.²¹ Existing pipelines for flexible information extraction from free-text radiology reports have not been validated in epilepsy but may be repurposed for use within this field.^33–35

Prior EEG abnormalities:

In addition to direct analysis of raw EEG signal, EEG abnormalities can be captured by means of structured fields and unstructured reports. Biswal et al achieved an AUC of 0.99 for detecting reports with seizures and AUC of 0.96 for epileptiform discharges, but they did not differentiate focal from generalized findings.³⁶

Bao reported 94% accuracy in interictal EEG diagnosis from raw EEG signal. However, this does not improve on the accuracy obtained from structured fields and unstructured reports, and increased the computational resources required for analysis as well as the administrative cost to obtain the raw EEG signal data.³⁷

Epilepsy severity:

Epilepsy severity can be estimated by several characteristics, which may be included in a model such as the number of current AEDs and historically prescribed AEDs (see the medications section above) or baseline seizure frequency (see seizure frequency section). Wissel et al used n-grams (up to n=3) and SVM to achieve sensitivity 80%, specificity 77%, PPV 25%, and NPV 98% for determining medically refractory epilepsy.³⁸

Age of epilepsy onset:

Extracting the age of onset in epilepsy remains an unmet challenge. Methods exist to extract the age of onset for other diseases in family history sections (i.e. age of cancer onset in family member).^39,40 However, these methods are not readily convertible to extract age of onset of a patient with epilepsy.

Medication allergies and adverse drug effects:

As noted in the medication section, prior studies reported that typical toolkits for NLP on clinical notes did not work well to abstract medications unless the note was divided into sections.⁴¹ The best reviewed available pipeline to identify medication allergies showed that an SVM model achieved the best average F1 of 89% on test data.⁴²

Comorbidities and past medical history:

Comorbidities are often found in structured problem and diagnosis lists, but these are notoriously underpopulated by physicians, and thus need to be supplemented with free text extraction.^43,44

SEDFE (SEmantics-Driven Feature Extraction) has been used to collect medical concepts from online knowledge sources as candidate features and derived methods to achieve AUCs ranging from 0.90–0.95.⁴⁵ Other pipelines reviewed achieved slightly lower AUCs with widely differing methodologies,^46,47 including the repurposing of a crowdsourcing marketplace.⁴⁸

Capturing common psychiatric comorbidities associated with epilepsy remain challenging. Commonly, comorbidities such as anxiety and depression can be captured as a diagnosis or solely as patient-reported symptoms. Furthermore, these comorbidities are likely underdiagnosed and undertreated within the epilepsy population. Validated depression instruments, such as the Patient Health Questionnarie-9 (PHQ-9) to assess depressive symptoms, can be easily extracted from structured fields. However, Adekkanattu et al implemented an NLP platform to extract PHQ-9 scoring from unstructured clinical text for patients prescribed an antidepressant with high accuracy (F-score 97%) and found that nearly one-third of patients’ charts had a score that clinically indicated major depressive disorder without a structured ICD diagnosis code associated.⁴⁹ In a study to predict advanced care of depression via statewide EHR data, free text extraction from decision tree models yielded AUC scores of approximately 90% for patients deemed high-risk patients versus approximately 80% for the overall patient population, respectively.⁵⁰

Epilepsy risk factors – history of drug abuse:

The main work in this area has been developed for the detection of opioid dependence and smoking.⁵¹ Notably, many authors have reported that ICD codes are insufficient for the accurate diagnosis of opioid dependence or overdose.^52–54 NLP can help improve recall of drug abuse from unstructured fields in clinician’s notes.^55–57 Including nursing notes further improves performance.⁵⁸ NLP has been shown to be more generalizable by publishing the difference in accuracy between their training and test sets.⁵⁴

Variables more applicable to seizure risk include alcohol abuse and stimulant abuse. ^58–61 These variables have not been studied in as much detail as opiates and nicotine. The most accurate study reviewed attained F1s of 90% for alcohol abuse and 85% for drug abuse detection.⁶⁰

Epilepsy risk factors – family history of epilepsy:

Family history statements have been extracted from a variety of note types for a variety of diseases, ^62–66 and it is likely that these can be repurposed to epilepsy. One method was able to achieve precision of 100% and recall of 97% using NLP; however, it was limited to looking at discharge summaries and admission notes.^65,66 Mowery et al achieved precision 96% and recall 94% using NLP on clinicians’ notes.⁴⁰

OUTCOME VARIABLES

Primary Outcome - Seizure frequency:

The ExECT pipeline (methods discussed in Epilepsy Type section above) identified the phrase or sentence within a clinical document that contained the seizure frequency but did not return an exact numeric value, with precision 86% and recall 54%.²⁸ To our knowledge, this is the only study of seizure frequency in the literature.

Secondary Outcomes- Use of rescue medications and the total number of AEDs required:

Surrogate markers of recurrent seizures may be identified by the use of rescue medications or the need to add additional AEDs. AED levels can notably be measured from structured fields. See the medication section above for details on how medications can be extracted.

OTHER GENERAL USE PIPELINES:

Our PubMed search results also found a series of general use pipelines, documented in detail in the supplementary text. One effort worth noting is a proposed strategy for incorporating prospective research-quality data collection into the practitioners’ workflow without burdening practitioners with excessive documentation, which remains the primary barrier to this type of collection.^67,68 They were able to implement prospective cohort building pipelines in 43 chronic diseases.⁶⁷ The implementation of such a framework within the field of epilepsy would allow for prospective data collection which is known to be superior to retrospective studies.

PIPELINES IN OTHER DIALECTS AND LANGUAGES:

For greater applicability and sample size, a multi-center international study would require the use of pipelines in many languages. Currently, convert pipelines from one language to another remains limited. There is one notable study where NLP developed in Europe and the United States was applied to medical notes written in Indian-English to extract medical diagnoses, labs, procedures, demographic information, and outcomes.⁶⁹ Details about pipelines in other languages can be found in the supplementary material.

DISCUSSION

Automated extraction of EHR data has advanced impressively in the last several years, with a plethora of methods published to date. While this review was not systematic, our comprehensive approach returned over 2000 articles by PubMed search criteria of which 128 articles were reviewed in detail. The key studies worth highlighting from this search include: 1) PEEP (phenotype extraction in epilepsy) which is highly accurate for medication dosing and extracts key information regarding epileptogenic zone, etiology, semiology and EEG patterns from EMU discharge summaries^9,10 and 2) ExECT (extraction of epilepsy clinical text) which extracts epilepsy diagnosis, epilepsy type, abnormal findings on imaging, and seizure frequency (albeit with some limitations).²⁸ However, great variability in algorithm performance across variables persists.

Some variables are easily and accurately extracted from the EHR by way of structured and unstructured fields, such as age, sex, and family history. Other variables can be extracted with reasonably high sensitivity but at the price of lower specificity, such as SES, ethnicity, epilepsy risk factors, EEG and MRI results.

Medications can be extracted with high accuracy (precision and recall > 95%) using a number of approaches, thanks in part due to public challenges that awarded prize money to competing teams to crowdsource the best method. Furthermore, techniques that are developed in fields outside of epilepsy can easily be applied to epilepsy.

Yet, several variables cannot be reliably extracted with current published methods, including seizure etiology and epilepsy severity. The problem of seizure etiology may potentially be solved by a multi-modal approach incorporating EEG findings, MRI findings, structured and unstructured fields, and setting up decision trees based on ILAE diagnostic criteria. This approach would be relatively easy to implement if the underlying variables could be extracted reliably (EEG findings, MRI findings, and comorbidities). Epilepsy severity is complex and multivariable, including the number of medications used, seizure frequency, presence of convulsive seizures, and potentially electroencephalographic markers.

The greatest limitation to assessing AED efficacy using EHR data is that seizure frequency cannot be extracted reliably from the record using currently available techniques. Assessing seizure frequency accuracy is crucial and can serve as a surrogate of seizure control efficacy. The American Academy of Neurology has emphasized documentation of seizure frequency as a recent quality measure, which provides further unstructured data to extract these measures.⁷⁰ Unstructured data sources of seizure frequency could include seizure diaries, collateral history, telephone encounters, etc. A recent approach classified and categorized patient portal messages to appropriately triage communication. This approach could be implemented to identify which message types are associated with poor epilepsy control (i.e., seizure reporting) versus routine communication.⁷¹

Our search, described in Section 1 of the supplementary text, yielded 72 results pertinent to “seizure” which included “seizure frequency.” The best reviewed techniques can identify only the seizure frequency text with poor sensitivity (~50%) and only marginal specificity (~80%). As previously mentioned, the ExECT pipeline identified the phrase within clinical text that contained seizure frequency, but was unable to return an exact numeric value in part due to the challenges of reporting variability (F1 66%); this is the only study of seizure frequency to our knowledge in the literature at the time of this review.²⁸ Future approaches to increase sensitivity will undoubtedly cost specificity measures. However, the creation of semi-automated algorithms could be developed, wherein an automated method screens notes comprehensively for key sentences with subsequent manual review to extract seizure frequency quantitatively and with greater accuracy. However, these semi-automated methods would require time intensive human review of each data-reduced chart.

Within the NLP field, the continued use of crowdsourcing will be vital to creating new pipelines and increasing accuracy to optimize data extraction from EHRs. Notably, medication extraction, medication allergy extraction, and opioid use and dependence can all be readily and accurately extracted with one of several available pipelines produced through crowdsourcing competitions.

As automated extraction methods continue to evolve, standards on reporting the accuracy of these pipelines should be followed. This will allow for comparisons to be drawn between methods. We call for, at minimum, the reporting of precision, recall, and F1 statistics for training and test sets (when a test set is available). We also recommend for all studies to use an independent test set when possible, and ideally another independent validation dataset from a different institution. This reporting is crucial because accuracy decrease between the training set and the test set is informative of the algorithm’s generalizability. We also appreciate that most of the studies that we reviewed have provided public access to the extraction algorithms, and we encourage all authors to do the same.

CONCLUSION

In summary, we evaluated the feasibility, availability, and performance of automated data extraction methods to facilitate prospective and retrospective investigation of AED efficacy and pertinent epilepsy variables. The most significant roadblock remains the dearth of algorithms to extract seizure frequency.

HIGHLIGHTS:

• Automated data extraction is rapidly evolving and can be harnessed to efficiently mine the electronic health record.

• Natural language processing (NLP) of unstructured text improves data extraction accuracy when added to ICD coding and structured fields.

• We review these techniques specific to epilepsy and highlight strengths as well as areas of further improvement.