Accuracy of claims-based algorithms for epilepsy research

Abstract

Objective:

To evaluate published algorithms for the identification of epilepsy cases in medical claims data using a unique linked dataset with both clinical and claims data.

Methods:

Using data from a large, regional health delivery system, we identified all patients contributing biological samples to the health system’s Biobank (n=36K). We identified all subjects with at least one diagnosis potentially consistent with epilepsy, e.g., epilepsy, convulsions, syncope, or collapse, between 2014–2015, or who were seen at the epilepsy clinic (n=1,217), plus a random sample of subjects with neither the claims nor clinic visits (n=435); we then performed a medical chart review in a random subsample of 1377 to assess the epilepsy diagnosis status. Using the chart review as the reference standard, we evaluated the test characteristics of six published algorithms.

Results:

The best performing algorithm used diagnostic and prescription drug data (sensitivity=70% [95%CI: 66%−73%]; specificity=77% [95%CI: 73%−81%]; and AUC=0.73 [95%CI: 0.71–0.76]) when applied to patients aged 18 years or older. Restricting the sample to adults aged 18–64 years resulted in a mild improvement in accuracy (AUC=0.75 [95%CI: 0.73, 0.78]). Adding information about current anti-epileptic drug use to the algorithm increased test performance (AUC=0.78 [95%CI: 0.76–0.80]). Other algorithms varied in their included data types and performed worse.

Significance:

Current approaches for identifying patients with epilepsy in insurance claims have important limitations when applied to the general population. Approaches incorporating a range of information, e.g., diagnoses, treatments, and site of care/specialty of physician, improve the performance of identification and could be useful in epilepsy studies using large datasets.

INTRODUCTION

Epilepsy is an incurable, life-threatening neurological disorder that affects an estimated 65 million people worldwide.^1,2 Despite the prevalence and seriousness of the condition, there is limited information on the quality of epilepsy care. This dearth results in large part from the limited availability population-wide databases and questions about the reliability of claims-based algorithms.^3–6

There are significant challenges to creating an algorithm for identifying epilepsy patients in insurance claims databases. Epilepsy is characterized by recurrent, spontaneous seizures, i.e., clinical diagnoses based on symptoms and the exclusion of precipitating causes of seizures. The manifestation of seizures can vary substantially across patients, and the recorded diagnoses can vary across physicians, e.g., with the amount of specialized training of physicians.^7,8 In fact, many patients receive an epilepsy diagnosis without use of standardized evaluation protocols, e.g., receipt of or documentation of an EEG or brain MRI.⁹

As a result, there is uncertainty about even basic information such as the true population incidence and prevalence of epilepsy. Previous efforts to identify epilepsy patients have used multiple data types and medical services to identify epilepsy cases in both the general population or in higher risk subgroups (as the risk increases, so does the likely prevalence, which in turn affects test characteristics such as the positive and negative predictive values).

Unfortunately, the diversity of the healthcare systems being studied represents a serious limitation to the generalizability of the existing algorithms.^4,15,21–23 Validation using a small dataset and cross-validation using the same dataset used to develop the algorithm may not adequately reflect performance in the broader application context. In epilepsy research, the ultimate goal of epidemiologists is to provide accurate predictions for independent samples obtained in different settings. The problem with internal cross-validation is that it may produce inflated discrimination accuracy, when compared to cross-study, cross-population validation.²⁶

In this study, we compare the published, validated algorithms for identification of epilepsy using a large, single dataset, and clearly stated sampling and evaluation criteria.

METHODS

Study design:

This study involved three stages: (1) creation of a validation cohort seen at the Partners HealthCare System (PHS); (2) assessment of epilepsy status using a review of medical records; and (3) evaluation of test characteristics for the published algorithms.^4,5,21,22 The work was part of a larger study on the examination of the interactions of genes, lifestyle, and other factors in the development of epilepsy and other diseases, which focused on patients contributing biological samples to the health system’s Biobank (figure 1). As discussed in the legend of Figure 1, these patients enrolled in the Partners HealthCare Biobank project were implemented in Partners a affiliated ambulatory clinical practice which consists of seventeen centers and departments including Neurology, Psychiatry, Primary Care, Emergency and Internal Medicine. Enrolled patients consented to the following: (1) a dedicated blood draw for preparation of DNA and blood derivatives for storage in the Biobank, (2) means to collect future discarded clinical specimens, (3) linkage of banked samples with their electronic medical record (EMR) and with health information collected through a secure survey, (4) specimen storage/distribution for broad use by IRB-approved Partners Investigators, and (5) willingness to be re-contacted as part of collaborating studies.

Figure 1.

Study design

*PHS: Partners Healthcare System: We used the Partners Healthcare system (PHS) - integrated Epic-based Electronic Health Record or HER. In addition to the two founding academic medical centers in Boston, the PHS includes medical sites in Rhode Island, Connecticut, Massachusetts and Maine, employing both primary care and specialty physicians at community hospitals, managed care organizations, specialty facilities, community health centers, and other health- related entities. Inpatient and outpatient records are collected on every patient in the PHS which includes over 1.5 million covered lives.

**RPDR: The Research Patient Data Registry is a clinical data registry that aggregates all records throughout PHS, including those from the visit narrative (e.g., physician’s clinical notes), test reports (e.g., reports of brain MRI), laboratory results (e.g., anti-epileptic drug level orders and results), or administrative systems (e.g., billing codes in claims). In this manuscript, we used the ICD-9 billing codes in professional claims, which represent the diagnostic judgment of a highly trained healthcare professional (as opposed to professional claims revised by coders or facility claims which represents the judgment of professional coders who are often focused on resource utilization).

***Partners HealthCare Biobank: As a cohort in the validation work, we targeted the patients enrolled in the Partners HealthCare Biobank project implemented in Partners affiliated ambulatory clinical practices. The clinical practices were part of seventeen centers and departments including Neurology, Psychiatry, Primary Care, Emergency and Internal Medicine. As of May 2015, about 36,000 patients consented to be part of the Partners HealthCare Biobank registry. The Partners HealthCare Biobank is a large research program designed to help researchers understand how people’s health is affected by their genes, lifestyle, and environment. Enrolled patients consent to the following: (1) a dedicated blood draw for preparation of DNA and blood derivatives for storage in the Biobank, (2) means to collect future discarded clinical specimens, (3) linkage of banked samples with their electronic medical record (EMR) and with health information collected through a secure survey, (4) specimen storage/distribution for broad use by IRB-approved Partners Investigators, and (5) willingness to be re-contacted as part of collaborating studies. We obtained the necessary institutional ethics review board (IRB) approval as a collaborating study to perform this validation work.

Patients were dichotomized into either confirmed epilepsy diagnosis or unconfirmed epilepsy diagnosis. Epilepsy diagnosis was defined according to the 2014 criteria adopted by the International League Against Epilepsy, including at least one of the following: (a) At least two unprovoked (or reflex) seizures occurring more than 24 h apart, (b) One unprovoked (or reflex) seizure and a probability of further seizures similar to the general recurrence risk (at least 60%) after two unprovoked seizures, (c) Clinical diagnosis of an epilepsy syndrome.

***Compare AUC: The area under the curve (AUC) is a measure of accuracy. It is created by plotting the true positive rate (i.e., proportion of cases identified as epilepsy cases that were confirmed epilepsy cases or “sensitivity”) against the false positive rate (i.e. proportion of cases indentified as epilepsy cases that were not confirmed epilepsy cases or “1- specificity”). Comparatively, the accuracy is greater in the algorithm with greater AUC value.

Patient population and data sources:

This study conducted the use of data from the PHS and the Partners Healthcare Research Patient Data Registry (RPDR), which is a data warehouse populated with data from several source systems, including the hospital and physician billing systems, as well as data from Partners’ Clinical Data Repository (CDR), Epic and the Enterprise Patient Master Index (EMPI). The EMPI assigns a 9-digit reference number to a patient and serves as a mechanism to assign multiple medical record numbers to one reference number, eliminating duplicate patients in the RPDR. This results in a comprehensive database that includes patient demographics, diagnoses (e.g., billing codes in claims), procedures (e.g., reports of brain MRI), inpatient pharmacy data, laboratories (e.g., anti-epileptic drug level orders and results), transfusions, microbiology, inpatient and outpatient encounter information (e.g., physician’s clinical notes), and provider data.

To compare basic test characteristics (e.g., sensitivity, specificity, positive predictive value, negative predictive value, Receive Operative Curves, or C-statistics), we selected four manuscripts that had reported key methodological information in their methods, as detailed in Table 1.^4,5,21,22

Table 1.

Validated claims-based algorithms for identification of incidence or prevalent case of epilepsy

Group	Holden et al. 2005	Reid et al. 2012	Tan et al. 2015	Franchi et al. 2013
Database	Lovelace Health Plan, Lovelace Health System in New Mexico	The Alberta Health Care Insurance Plan Registry (AHCIP)	Health Information Services (HIS) Department at St. Vincent’s Hospital, Melbourne, AU	Drug administrative database of the Lombardy region (Northern Italy)
Validation method	Chart review of sample	Chart review of sample	Chart review of sample	Physician survey (correlate for chart review)
Epilepsy definitiona	Epilepsy	Epilepsy or convulsions	Epilepsy, status epilepticus, or convulsions	Epilepsy
Ageb	0–19,65+	18+	All	All
Time-framec	1996–2001	2002–2007	2012–2013	2000–2008
Emergency room	–	ICD-9-CM (345) or ICD-10-CA(G40 or 41)	–	–
Outpatient claims	1 claim: ICD-9-CM,345.xx, 333.2, 779, 780.3, 780.39, 780.2, 780.31	ICD-9-CM (345) or ICD-10-CA (G40 or 41)	ICD-10 AM (G40 or G41 )d	–
Inpatient claims	–	1) ICD-9-CM (345) 2) ICD-10-CA(G40or 41)	–	–
EEG (CPT)	And 1 CPT: 95812–95958	–	–	At least one
AED	At least one	–	At least one	At least one

ICD, International Classification of Diseases; EEG, electroencephalography; MRI, magnetic resonance imaging; CPT, Current Procedural Terminology; AED, antiepileptic drug.

We reviewed the literature to identify validated computational algorithms with sensitivity and specificity of at least 60%.^{2,5,6,10–12,14–20} Four of the six algorithms included ICD-10 codes, which were not phased into U.S. billing until October 2015. As a result, we converted all instances to ICD-9 equivalents, as determined by current guidelines.¹¹

^aFor consistency in case ascertainment, we used the updated 2014 criteria for epilepsy diagnosis adopted by the International League Against Epilepsy, including documentation of at least one of the following; (1) at least two unprovoked (or reflex) seizures occurring >24 h apart; (2) one unprovoked (or reflex) seizure and a probability of further seizures similar to the general recurrence risk (at least 60%) after two unprovoked seizures, occurring over the next 10 years; or (3) clinical diagnosis of an epilepsy syndrome documented in the medical record.

^bWe performed stratified analysis by age groups to allow for comparison among the different algorithms.

^cWe selected a 2-year time frame (2013 2014) to allow for comparison among the different algorithms.

^dWe replaced ICD-I0-CA (G40 or 41) by ICD-9-CM (345). This substitution was performed because our database contained only ICD-9 codes.

We identified 1,906 medical records of consented patients. Employees of the healthcare system were automatically excluded from the query as per our IRB protocol, yielding a final sample of 1,652 eligible patients. We recognized all subjects with no less than one diagnosis potentially consistent with epilepsy, e.g., epilepsy, convulsions, syncope, or collapse, between 2014–2015, or who were seen at the epilepsy center (n=1,217), in addition to an irregular sample of subjects with neither the diagnosis nor facility visits (n=435). Then we conducted a medical chart review of a random subsample of 1377 to examine the epilepsy diagnosis status. Please find details of the medical records abstraction process in supplementary information 12. A comprehensive description of the sample query, medical records review process and the algorithm selection can be found in supplementary information 1-3.

Statistical Analysis:

We calculated sensitivity, specificity, positive predictive values, and negative predictive values for each algorithm against the reference standard derived from the chart review. Using the validation of the algorithms for epilepsy case definition, sensitivity represented the percentage of total epilepsy cases correctly identified as epilepsy cases by each algorithm. Specificity was defined as the percentage of total non-epilepsy cases correctly identified by the algorithm. Positive predictive values were determined by percentage of algorithm-identified epilepsy cases qualifying as “true” epilepsy cases. Negative predictive values were determined according to the percentage of algorithm-identified non-epilepsy cases qualifying as “true” nonepilepsy cases. The latter two predictive values, however, are influenced by the prevalence of true epilepsy cases in the sample, whereas sensitivity and specificity are not; therefore, we focus our presentation on sensitivity and specificity.

We adapted the study design to allow for comparison among the different algorithms across different patient populations. For instance, Holden grouped age as 0–19 and 65+ years whereas Reid only used adults. We performed the comparative analysis using the entire cohort of adult patients, and repeated the same analysis stratified by age groups: 18–64 and > 64 years old. In addition, we selected a two-year time frame (2013–2014) to allow for comparison among the different algorithms. We also obtained a list of false positives and false negatives from the algorithm of highest performance to describe their demographic and clinic characteristics.

Supporting Information 4 shows a table of the generic and brand names of the anti-epileptic drugs used in this study. Supporting information5 compares the diagnosis and procedure codes obtained using administrative claims. Supporting information6 shows the distribution of prescribed medications abstracted using administrative claims.

RESULTS

The records of 875 (64%) patient’s contained information leading to a diagnosis of epilepsy and 502 (36%) patient records contained data supporting an alternative diagnosis (e.g.: syncope, provoked seizures). Supporting Information 7, describes the demographic and clinical characteristics of the patients with (n=875, 64%) and without confirmed diagnosis of epilepsy (n=502, 36%) based on medical records review. For all analyses we defined the statistical significance level as p < 0.05. We used the SAS® Studio software package (SAS Institute Inc. Cary, NC) to perform statistical analysis.

The algorithm described by Holden had the highest accuracy (AUC area = 0.73 [0.71–0.76]) when applied to this study’s dataset (cohort of patients older than 18 years old). This model used diagnosis and anti-epileptic drug as predictors and had a positive predictive value of 84.1% (exactly the same as published in 2005). This model had a sensitivity of 69.6% [66.4%, 72.6%] and specificity of 77.1% [73.2%, 80.7%]. This model gained accuracy when applied only to the cohort of patients older than 18 years but younger than 65 years (AUC area =0.75 [0.73, 0.78]) and lost accuracy when applied to the population 65 years or older (AUC area = 0.66 [0.61–0.71]). Figure 2A and Figure 2B, table 1 and supporting Information 8 and 9 detail the indicators of model strength between the six algorithms.

Figure 2.

Sensitivity

A: Internal dataset (red) represents the sensitivity of the algorithms applied to the dataset in which they were developed (i.e., as published in the respective manuscripts). External dataset (blue) represents the sensitivity of the algorithms applied to our validation cohort.

Specificity

B: Internal dataset (red) represents the specificity of the algorithms applied to the dataset in which they were developed (i.e., as published in the respective manuscripts). External dataset (blue) represents the specificity of the algorithms applied to our validation cohort.

In Holden 1 applied to the cohort of patients older than 18 years old, 609/875 (sensitivity 69.6%) were true positives and 266/875 (30.4%) were false positive cases (FP). FP indicates that the algorithm incorrectly indicated that the patient had epilepsy but the patient did not have epilepsy based on medical records review. This same algorithm produced 387/502 (specificity 77.1%) true negative and 115/502 (22.9%) false negative cases (FN). FN indicates that the algorithm incorrectly indicated that the patient did not have epilepsy but the patient actually had epilepsy. We obtained the list of false positives and false negatives to describe their clinical characteristics (supporting information10 and 11).

Restricting the sample to require at least two diagnoses suggestive of epilepsy and one prescription of anti-epileptic drug (Modified Holden 1.1) decreased the test performance slightly (AUC area = 0.69 [0.67–0.72]). Finally, including anti-epileptic drug use data (Modified Holden 1.2) increased the test performance (AUC=0.78 [95%CI: 0.76–0.80]).

DISCUSSION

Understanding the impact of epilepsy on patients broadly and assessing the quality of their care requires accurate identification of patients who have epilepsy using large datasets, such as those created by insurance claims. In the first comparison of existing algorithms for claims- based epilepsy identification, we find that the current algorithms perform modestly well at best, and all have important limitations. The most accurate algorithm for identifying patients with epilepsy includes one claim coded with the ICD-9 diagnosis of epilepsy, convulsions, syncope or collapse, plus at least one code for an anti-epileptic drug prescription or use. Incorporating more types of information (e.g., diagnosis, treatment, provider) could improve accuracy.

The scientific community often uses the area under the curve (AUC) for algorithm performance comparison. However, with some analyses, one might want to maximize sensitivity (e.g., screening of patients with possible epilepsy). The sensitivity of algorithms for identification of patients who have epilepsy is also known to vary across different datasets and different algorithms. The sensitivity was highest in the algorithm by Reid et al, perhaps because this algorithm included inpatient and emergency rooms claims, which increases the number of care settings in which a patient may receive the diagnosis code. However, the sensitivity of the algorithm by Reid was substantially reduced when applied to our dataset, perhaps because there has been a high degree of referral leakage in our system (i.e., patients often travel long distances for outpatient epilepsy care but are encouraged to seek emergency care in local hospitals, which are often outside of our system). Interestingly, the sensitivity of the algorithm by Tan did not substantially change when applied to our dataset, likely because our population characteristics were similar to the cohort used by Tan.

A common challenge facing clinical researchers is the timely acquisition of a statistically powerful and representative study sample.²⁷ Medical claims databases have emerged as a method to capitalize on existing, codified data on regional and population-wide scales.¹⁶ Research of this scale has the potential to strengthen epidemiological surveillance as well as monitor the impact of major health policy reform, however its utility rests in the reliability of sample selection models.^28,29 According to the most recent epilepsy quality guidelines, quality care processes is recommended at least annually (e.g., personalized safety counseling, antiepileptic drug side effects query), which would require a minimum of two office visits within a time frame of two years.³⁰ Consequently, we kept the two-year time frame and performed a second exploratory analysis, testing a modification of the best performing algorithm (modified Holden 1.1) to require at least two diagnoses suggestive of epilepsy and one prescription of anti-epileptic drug.

Our study is consistent with prior studies suggesting that current drug therapy and EEG records provide only moderate sensitivity in identifying prevalent cases of epilepsy.⁵ Of note, we compared algorithms based on populations in well-defined geographical regions of Italy, Canada, Australia and the United States. Not surprisingly, Holden’s, which was originally validated in a dataset from New Mexico (state located in the southwestern region of the U.S.), was the best performing algorithm when applied to our dataset,.^4,5,21,22

However, this cross-study comparison was limited by the general lack of congruence in the way the algorithms were developed and validated. In 2012, Reid used only diagnostic codes applied to The Alberta Health Care Insurance Plan Registry (AHCIP) in Canada. They explored 18 algorithms and suggested that the coding algorithm with the best diagnostic accuracy to identify epilepsy cases was 2 physician claims or 1 hospitalization over a 2 year time-frame (Sn 88.9%, Sp 92.4%, PPV 89.2%, NPV 92.2%).⁵ These results were quite different when the same algorithm was applied to our database. A possible explanation to that there was over fitting of the model. In over fitting, a statistical model describes random error or noise instead of the underlying true predictive power. In particular, a model is typically trained by maximizing its performance on some set of training data (i.e., their own dataset, internal dataset). However, its efficacy is determined not by its performance on the training data but by its ability to perform well on unseen data (i.e., external dataset). As an extreme example, a simple model or learning process can predict the training data simply by memorizing the training data in its entirety, but such a model will typically fail drastically when making predictions about new or unseen data, since the simple model has not learned to generalize at all. In order to avoid over fitting, it is necessary to use additional techniques such as cross-validation or early stopping. Another possible explanation is the expected national and regional variation in coding patterns (e.g., secondary to different reimbursement incentives across different health care systems). Along with those lines, our study provides valuable information about each model’s ability to generalize by evaluating their performance on a set of data not used for algorithm development, which is assumed to approximate the range of accuracy of the subsequent studies that used one of the six selected algorithms without prior validation.^26,31

Later in 2013, Franchi et al used a retrospective physician survey as the reference standard. This differs from chart review because it adds the physician recall bias. In fact, the Franchi et al study included a small sample of epilepsy patients (n=71). They reported high accuracy (Sn 85%, Sp 99%, PPV 64%, NPV 99%), which was not replicated when applied to our dataset.⁴ We believe that the robustness of our accuracy results is supported by our validation sample size of more than 700 epilepsy patients. Our results are also intuitively supported by the typical demographic and clinical characteristics of our cohort (e.g.: epilepsy patients often had abnormal brain imaging and neurophysiology studies compared to patients without epilepsy). Unfortunately, Franchi et al did not report the characteristics of their dataset.

Most recently in 2015, Tan used data from the Health Information Services (HIS) Department at a hospital in Melbourne, Australia to validate the algorithm utilizing ICD-10 codes for epilepsy and ≥1 antiepileptic drug (AED), which is essentially the same as in Holden but with ICD-10 codes. They reported a good accuracy (Sn 60, Sp 99.9% PPV 81.4%, NPV 99%). The comparison between this algorithm and the application to our dataset was threatened by our conversion of ICD-10 codes to ICD-9 codes, because ICD-10 codes were not phased into U.S. billing until October 2015. This conversion increased our uncertainty about the validity of the epilepsy identification algorithms.²¹

Accordingly, a comparison between International League Against Epilepsy (ILAE) disease classifications and ICD codes across the most recent iterations, both ICD-9 and −10, showed limited cross-validation strength and considerable variation across studies.¹⁸ Nevertheless, our study builds upon the literature in which suggests that quality of estimations of epilepsy based on claims data depends on the case definition of epilepsy as well as on the demographic and clinical characteristics of the population and the healthcare system.^4,15,21-23The discussion about the secondary findings of this study is provided in supplementary information 13.

A series of limitations of the previously published studies and our study serve as potential directions for further accuracy optimization. First, our study sample may not be well representative of a general population. For instance, the sampling from Partners HealthCare Biobank list of enrollees may have selected patients with higher educational level (e.g.: able to understand the consent form procedures) and with more severe disease (e.g.: willing to provide blood samples for researchers).Well educated patients were able to provide a more accurate description of their events. In addition, more severely ill patients may have yielded more clinical evidence of the diagnosis of epilepsy (and more accurate medical documentation as the reference standard). Admittedly, we had strategically selected the Biobank enrollees in this validation effort as part of a larger project that will examine genetic and biomarker factors in relationship to epilepsy diagnoses and care. Nevertheless, our Supporting Information 7 shows a table that suggests that the clinical and demographic characteristics of the validation sample were representative of a broad cohort of epilepsy patients with multiple types of seizures and etiologies.

Second, in our validation sample, we noted that the majority of patients had documented two or more unprovoked (or reflex) seizures occurring more than 24 hours apart (ILAE criteria “a”). Less often patients had documentation of a clinical diagnosis of an epilepsy syndrome (ILAE criteria “c”) or one unprovoked seizure and a probability of further seizures similar to the general recurrence risk after two unprovoked seizures (ILAE criteria “b”). However, we failed to track how many patients met each criteria or combinations. We also failed to track how many of the non-confirmed cases were due to a lack of evidence to justify a diagnosis of epilepsy (e.g., events of unclear etiology) vs. lack of documentation in the chart to allow for a diagnosis of epilepsy (i.e., loss to follow-up). In addition, it would be natural to consider that future studies may benefit from a validation dataset that includes the likelihood of diagnosis of epilepsy based on more variables such as response to prophylactic therapy, types of EEG and brain MRI abnormalities or the specialty of the physician making the diagnosis (e.g., non-neurologist vs. general neurologist vs. epilepsy specialist). However, adding information about the results of brain MRI and EEG might cause more confusion as abnormalities on these tests alone have low predictive value in many circumstances (e.g., sensitivity of routine EEGs in patients with epilepsy is less than 50%). Symptom resolution after initiation of prophylactic therapy is also known to bias towards error as patients with non-epileptic events often have symptom resolution with the same prophylactic therapies (i.e., placebo effects) and as much as 30% of patients with epilepsy may continue to have seizures despite adequate treatment.³³ Overall, until a biomarker becomes available, we may rely on physician judgment for case ascertainment of epilepsy. With that in mind, our medical records abstraction was performed by a well trained medical student and a neurologist, under the close supervision of an epilepsy specialist in an effort to produce the most accurate categorization.

Similar to previous studies, our study has given little attention to the examination of the accuracy of epilepsy classification (e.g.: whether claims indicating generalized epilepsy accurately represent a patient with generalized epilepsy syndrome). This is particularly important in comparative epilepsy research as efficacy of anti-epileptic drugs often differs across seizure types and syndromes. There are some published categorization methods that also merits cross- validation.^16,34 In addition, the extent to which clinically-scaled measures, such as provider specialty involvement, may also increase accuracy remains unverified.³⁵

Future studies may examine the accuracy of algorithms using sub-decimal ICD 9 codes for epilepsy. For instance, 65% (570/875) of patients with epilepsy received the code 345.1, which is titled “epilepsy, unspecified, without mention of intractable epilepsy”. In contrast, only 22% (112/502) of patients with epilepsy received this code (345.1), as shown in Supplement 5.

Future studies may also refine their algorithms based on the prescription patterns among patients who do and do not have a confirmed diagnosis of epilepsy, which has been described in the supporting information of the present study. For instance, gabapentin is classified as anti-epileptic medication but has been more often prescribed for patients without epilepsy. This is consistent with the existing literature that describes that many anti-epileptic drugs (e.g., gabapentin, carbamazepine, topiramate) are widely used in the treatment of neuropathic pain or headache. When used in patients with epilepsy, gabapentin is often a third or fourth line agent. Based on that, a reasonable nested algorithm that applies different algorithms depending on whether the patient is using gabapentin as monotheraphy (likely not an epilepsy case) vs. polytherapy (a possible epilepsy case) may be valuable.³⁶

Similarly, our analysis of false positive cases (supporting information10) highlights two common problems: mis-coding (i.e.; claims of possible epilepsy for a patient who never had a seizure) and mis-management (i.e., long-term use of anti-epileptic drugs for patients who never had a seizure). Of note, current guidelines for traumatic brain injury (TBI) and aneurismal subarachnoid hemorrhage recommend antiepileptic drugs for a few days to decrease post-traumatic or post-hemorrhage seizure risk.³⁷ Only a few patients may develop seizures, who would then meet the criteria for symptomatic epilepsy and require longer term prophylaxis and establishment of care with a neurologist. However, our medical records review demonstrated that long-term anti-seizure prophylaxis has been widely prescribed for patients with other conditions such as intracerebral tumors, craniotomy, ischemic stroke, or other forms of intracerebral hemorrhage who never had a seizure. Unfortunately, there is no clear evidence to support the long term use of anti-epileptic drugs even for aneurismal subarachnoid hemorrhage patients.

Finally, new-onset epilepsy in older patients is often associated with other neurologic conditions, including Alzheimer’s Disease-Related Dementias (ADRD), stroke and brain tumors.^38,39,40 Therefore, future studies may test the use of cardiovascular diseases and tumor co-morbidities captured in ICD-9 codes to see if this can further increase accuracy of the algorithms applied to older adults. Future studies should also focus on the comparison of our cross-validation findings with the cross-validation of algorithms for identification of other neurological diseases and even other chronic diseases.

In conclusion, this study highlights the immediate need for refinement and cross-validation of algorithms for identification of epilepsy in individuals who may later develop recurrent unprovoked seizures using a combination of data sources.

Key Point Box:

• Accuracy of existing claims-based definitions of epilepsy is modest

• Combining diagnosis codes with medication use increases accuracy

• External validation is an important step in evaluating the performance of claims-based definitions of epilepsy