Abstract
Objective:
Electronic health records (EHRs) are increasingly used to conduct research and evaluate epilepsy quality of care. We examined the accuracy of International Classification of Diseases, 9th and 10th Revision, Clinical Modification (ICD-9-CM, ICD-10-CM)- and antiseizure medicine (ASM)-based algorithms for adult epilepsy.
Methods:
Data from a diverse New York multicenter EHR were queried to identify encounters between January 1, 2012 and September 20, 2018 coded with an epilepsy/seizure ICD-CM code or an ASM. Eight hundred adults were randomly selected (350 with epilepsy-related codes, 150 with an ASM, and 300 with drug-resistant epilepsy codes). With chart review defined as the reference standard, sensitivity (Sn), specificity (Sp), negative predictive value (NPV), positive predictive value (PPV), and Youden index (YI) were calculated to evaluate various ICD-9-CM-, ICD-10-CM-, ± ASM-based algorithms’ accuracy in predicting epilepsy.
Results:
Ninety-four algorithms were tested. A total of 435 (54.4%) patients had definite epilepsy. Estimates ranged as follows: YI = .18–.68, Sn = .59–.95, Sp = .52–.97, PPV = .67–.92, and NPV = .51–.93. The best algorithms were as follows. Highest YI for ICD-9-CM was single encounter with 345 (except 345.2 or 345.3) or 345.2, 345.3, or 780.3 with an ASM (Sn = .95, Sp = .73, PPV = .81, NPV = .92, YI = .68). Highest Y1 for ICD-10-CM was one encounter with G40 in primary diagnostic position or ≥2 encounters with G40 in any diagnostic position (Sn = .82, Sp = .85, PPV = .87, NPV = .80, YI = .67). Highest sensitivity was any encounter with ICD-9-CM 345 or 780.39 or ICD-10-CM G40, G41, or R56.9 (Sn = .96, Sp = .57, PPV = .73, NPV = .93, YI = .53). Highest specificity was ≥1 hospitalization with ICD-9-CM 345.x (except 345.2 and 345.3) or ICD-10-CM G40.x (Sn = .21, Sp = .97, PPV = .89, NPV = .51, YI = .18).
Significance:
We identified ICD-9/10-CM-based case definitions (with and without ASM) that were sensitive and specific for epilepsy. Ultimately, extensive algorithms are provided to help inform case definition selection according to future study aims.
Keywords: accuracy, administrative data, electronic medical records, ICD-9-CM, ICD-10-CM, International Classification of Diseases, medical records, seizure
1 |. INTRODUCTION
Epilepsy is a common neurological disorder accounting for more than 10 million disability-adjusted life-years globally.1 In the United States, the prevalence and mortality of epilepsy has been steadily rising, underscoring the need for accurate disease surveillance.2 Administrative databases are essential tools that can be used to facilitate disease surveillance and conduct population-based outcomes research in people with epilepsy. This is especially important with the uptick in the implementation of electronic health record (EHR) systems globally.3 Using administrative data for these purposes requires the development of algorithms that accurately identify people with epilepsy.
Various International Classification of Diseases (ICD)-based definitions of epilepsy have been previously developed and validated using large administrative databases.4 These studies typically use various combinations of ICD, 9th Revision (ICD-9) codes, ICD-10 codes, Read codes, and procedure codes as well as antiseizure medication (ASM) to define epilepsy. Whereas epilepsy case definitions using ICD-10 codes have been developed and validated for epilepsy populations in Canada, Australia, Denmark, Norway, and Korea,5–11 few ICD-10-based case definitions have been developed in the United States. A 2020 study was the first, to our knowledge, to validate an ICD-10, Clinical Modification (CM) claims-based definition of epilepsy in the United States.12 However, this study tested a limited number of case definitions and did not include data on ASM in their algorithms.
In this study, we evaluate the accuracy of various ICD-9-CM and ICD-10-CM with or without ASM-based case definitions to identify adults with epilepsy in a large New York City health system using a widely used EHR system (EPIC).
2 |. MATERIALS AND METHODS
2.1 |. Study design and data source
This large multicenter study encompassed six tertiary and community hospitals as well as smaller outpatient practices in an urban academic health system in New York City (Mount Sinai Health System) with a large catchment area that includes New Jersey, Connecticut, and other parts of New York State and the United States. Data were provided by the Mount Sinai Data Warehouse and included demographic information (age, date of birth, sex, and insurance status), diagnosis, admission/discharge date, encounters with specialists, ICD codes, and prescription ASMs.
2.2 |. Cohort identification
Adult patients included in the study were identified through programmed Data Warehouse queries in one of three ways. First, all patient encounters between January 1, 2012 and September 20, 2018 that included an ICD-CM epilepsy/seizure code in either the primary diagnostic position, secondary diagnostic position, problem list, or medical history were collated. A code indicative/suggestive of epilepsy had to begin with either 345. xx, G40.x, G41, or R56. For a full list of codes included, see Table S1. All patient encounters within the same time frame that did not include an ICD-CM code suggestive of epilepsy but did include a prescription for an ASM (regardless of indication) were also identified. For a full list of ASMs included, see Table S2. This yielded a total of 1 595 993 encounters among 1 119 212 unique patients.
From these encounters, a random sample of 800 patients were selected for chart review as follows: (1) 350 patients with ICD-9-CM or ICD-10-CM codes indicative of an epilepsy diagnosis were selected (50 randomly selected from each year 2012–2018), (2) 150 patients on an ASM but without any ICD-CM epilepsy-related codes were also randomly selected, and (3) 300 patients who had ICD-9-CM or ICD-10-CM codes indicative of drug-resistant epilepsy (DRE) were randomly selected (38 from 2012 to 2013 [the total number of DRE patients that year], 50 from 2014 [all patients with DRE that year], and 53 from each year 2015–2018; see Table S3 for list of DRE diagnostic codes).
2.3 |. Patient chart review
Chart review was conducted by two medical students (J.G. and H.-H.T.) and a neurology resident (H.N.L.), all of whom were trained by an epileptologist with expertise in health services research (N.J.). The training ensured that the reviewers understood the clinical definition of epilepsy and could accurately extract relevant data from each patient’s record using a standardized form. The definition of epilepsy was based on the 2014 International League Against Epilepsy criteria. To achieve 100% agreement, the three reviewers independently analyzed batches of 15 charts through an iterative process. First, each reviewer independently assessed 15 charts. Any discrepancies among them were discussed and resolved through consensus. An epileptologist (N.J.) also reviewed the charts, corrected errors, and discussed these corrections with the reviewers. This process was repeated for two additional sets of 15 charts. By the end of the third round, full agreement across all 15 charts was reached among the three reviewers and the epileptologist, signifying the completion of the training phase. Following training, the remaining charts were reviewed independently by one of the three reviewers. Any chart categorized as “possible epilepsy” underwent a second review by two epileptologists (L.J.B. or N.J.), who made the final determination of an epilepsy diagnosis. In cases of disagreement, the two epileptologists discussed the case and reached a consensus.
Each patient was ultimately categorized into one of the groups defined below:
Definite epilepsy: A patient with (i) two or more unprovoked seizures in a period of >24 h; (ii) a patient with one unprovoked seizure and a high risk for a second seizure (>60%) based on prior central nervous system insult, seizure occurring at night, abnormal electroencephalographic (EEG) or magnetic resonance imaging (MRI) findings demonstrating prior brain injury, and/or a note (preferably from a neurologist or epileptologist) explaining the nature of the events and workup suggestive of epileptogenesis (abnormal EEG, abnormal MRI, or epilepsy risk factors); or (iii) diagnosis of an epilepsy syndrome.13
Possible epilepsy: Any patient with convulsionlike episodes who does not meet any of the above criteria and in whom epilepsy could not be ruled out.
Epilepsy ruled out or not mentioned: A patient with a history of convulsions who underwent a battery of tests (EEG, epilepsy monitoring unit admission, and/or MRI) with findings and history not suggestive of epilepsy, or a note from a neurologist/epileptologist stating the patient did not have epilepsy. In addition, if seizures, epilepsy, and convulsions were never mentioned in the chart, the patient would be placed in this category.
2.4 |. Claims-based definitions
Previously published ICD-9-CM-, ICD-10-CM-, and ASM-based case definitions were identified based on a recent systematic review.4 Additional algorithms were also developed to determine whether the accuracy of existing case definitions could be improved upon. In total, 94 algorithms were tested (Table S4). Of note, in the United States, claims coding is typically done by the treating physician, who can select any code they feel is appropriate for the visit and/or procedure.
2.5 |. Statistical analysis
Frequency with proportion were calculated to describe the baseline characteristics of included patients. Age and insurance were defined as the value during the first encounter. Chart review was defined as the reference standard. Patients determined to have definite epilepsy were considered positive cases. Patients with possible epilepsy or no epilepsy or for whom epilepsy was not mentioned in the chart were considered negative cases. In a sensitivity analysis, patients with definite or possible epilepsy were considered positive epilepsy cases, and only those patients in whom epilepsy was ruled out or not mentioned were considered negative cases.
We evaluated the performance of each case definition of epilepsy against the reference standard of chart review. We calculated the sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) as well as the 95% confidence intervals for each of the case definitions. To better assess the accuracy of these algorithms, we also calculated the Youden index (YI), which is the sum of sensitivity and specificity minus one. Statistical analyses were performed using SAS 9.4.
This study was approved by the Icahn School of Medicine Institutional Review Board (STUDY-17–01190).
3 |. RESULTS
3.1 |. Primary analysis
Of the 800 patients included in the study, 511 (63.9%) were between the ages of 18–65 years, 407 (51.2%) were female, 182 (27.7%) were Medicaid recipients, 181 (27.6%) were Medicare recipients, and 424 (53%) had been seen in a neurology practice in this health system during the study period (Table 1). On chart review, 435 patients had definite epilepsy, 98 had possible epilepsy, and the remaining 267 did not have epilepsy or had no information on an epilepsy diagnosis.
TABLE 1.
Characteristics of cohort of 800 patients.
| Characteristic | n (%) |
|---|---|
| Age | |
| ≤1 year old | 25 (3.1) |
| 1–18 years old | 121 (15.1) |
| 18–65 years old | 511 (63.9) |
| 65+ years old | 143 (17.9) |
| Sex | |
| Female | 407 (51.2) |
| Male | 388 (48.8) |
| Insurance | |
| Medicaid | 182 (27.7) |
| Medicare | 181 (27.6) |
| Other/self-pay | 48 (7.3) |
| Private insurance | 246 (37.4) |
| Ever been to neurology practice/clinica | |
| Yes | 424 (53) |
| No | 376 (47) |
In the health system during the study period.
The performance of the 94 algorithms assessed can be viewed in Table S4. YI ranged from .18 to .68, sensitivity from .59 to .95, specificity from .52 to .97, PPV from .67 to .92, and NPV from .51 to .93.
The top-performing ICD-10-based algorithms can be viewed in Table 2. The two ICD-10-based algorithms with the highest YI used the ICD-10-CM code G40.x. In one of these algorithms, the G40.x code had to be in the primary position (algorithm identification number [ID] 5), and in the second, the G40.x code had to be either in the primary position or appear in >2 encounters at any diagnostic position (ID 4). The latter algorithm led to improved sensitivity and PPV, but decreased specificity and NPV. YI was .67 in both case definitions. An algorithm requiring at least one G40.x code, without any restrictions on encounter type or diagnostic position, performed similarly, with a YI of .66 (ID 4). Algorithms that required both an ICD-10-CM diagnostic code and an ASM prescription had slightly lower YIs, ranging from .64 to .65 (IDs 59, 73, and 74).
TABLE 2.
Most accurate ICD-10-based algorithms.
| ID | Algorithm descriptiona | ICD-9-CM and ICD-10-CM codes | Youden index | Sn (95% CI) | Sp (95% CI) | PPV (95% CI) | NPV (95% CI) |
|---|---|---|---|---|---|---|---|
| 5 | ICD-10 code | G40 primary position or ≥2 encounters with G40 in any diagnostic position | .67 | .82 (.78–.85) | .85 (.82–.89) | .87 (.84–.90) | .80 (.76–.84) |
| 4 | ICD-10 code | G40 in primary position | .67 | .79 (.75–.83) | .88 (.84–.91) | .88 (.85–.92) | .78 (.74–.82) |
| 59 | ICD-10 code + ASM | G40–41, R56, F80.3 | .65 | .87 (.84–.90) | .78 (.73–.82) | .82 (.79–.86) | .83 (.80–.87) |
| 1 | ICD-10 code | G40 | .66 | .85 (.81–.88) | .81 (.77–.85) | .84 (.81–.87) | .82 (.78–.86) |
| 74 | ICD-10 code + ASM | G40 | .64 | .77 (.73–.81) | .87 (.83–.90) | .87 (.84–.91) | .76 (.72–.80) |
| 3 | ICD-10 code | At least 2 encounters with G40 | .60 | .70 (.66–.74) | .90 (.87–.93) | .89 (.86–.93) | .72 (.67–.73) |
| 10 | ICD-10 code | G40–41, R56 | .51 | .96 (.95–.98) | .52 (.46–.57) | .70 (.67–.74) | .92 (.88–.96) |
Abbreviations: ASM, antiseizure medication; CI, confidence interval; ICD-10, International Classification of Diseases, 10th Revision; ID, algorithm identification number; NPV, negative predictive value; PPV, positive predictive value; Sn, sensitivity; Sp, specificity.
Refers to ICD-9-CM or ICD-10-CM codes.
The algorithms with the highest YI, sensitivity, and specificity are listed in Table 3. The algorithm with the highest YI (.68) required one encounter with ICD-9-CM code 345.xx (excluding 345.2 and 345.3) or 780.3 as well as one encounter with an ASM prescription (ID 78). This algorithm had a sensitivity of .95 and a specificity of .73. When this algorithm was modified so that encounters coded with ICD-10-CM codes such as G40.xx, G41.xx, and R56 would also qualify as epilepsy encounters, there was no impact on the sensitivity or NPV, but the PPV and specificity were lowered slightly, resulting in a YI of .66 (ID 94). The sensitivities of both these algorithms were among the highest of all case definitions assessed. Several claims-based algorithms that did not include a requirement for ASM prescription demonstrated a slightly higher sensitivity of .96; however, the specificities of these algorithms were far lower, ranging from .52 to .57 (see IDs 2, 6, 7, 10, 50, 51, and 52).
TABLE 3.
Algorithms with highest Sn, Sp, and Youden index.
| ID | Algorithm descriptiona | ICD-9-CM and ICD-10-CM codes | Youden index | Sn (95% CI) | Sp (95% CI) | PPV (95% CI) | NPV (95% CI) | |
|---|---|---|---|---|---|---|---|---|
| Highest Youden index | 78 | ICD-9 code + ASM | 345 (.2, .3, + 1 ASM) or 780.3 + 1 ASM |
.68 | .95 (.93–.97) | .73 (.68–.77) | .81 (.77–.84) | .92 (.89–.95) |
| 5 | G40 primary position or ≥2 encounters with G40 in any diagnostic position | G40 | .67 | .82 (.78–.85) | .85 (.82–.89) | .87 (.84–.90) | .80 (.76–.84) | |
| 4 | G40 in primary position | G40 | .67 | .79 (.75–.83) | .88 (.84–.91) | .88 (.85–.92) | .78 (.74–.82) | |
| 94 | ICD-9 or ICD-10 code + ASM | 345.x (except 345.2 and 345.3), G40.x (except G40.xx1) ± ASM or 345.2, 345.3, 780.3x, G40.xx1, G41, R56.x with ≥1 ASM | .66 | .95 (.93–.97) | .72 (.68–.77) | .80 (.77–.84) | .92 (.89–.95) | |
| Highest sensitivity | 50 | ICD-9 and ICD-10 code | 345.x, 780.39 G40.x, G41.x, R56.9 | .53 | .96 (.95–.98) | .57 (.52–.62) | .73 (.69–.76) | .93 (.89–.96) |
| 51 | ICD-9 and ICD-10 code | 345.x, 780.33, 780.39 G40.x, G41.x, R56.1, R56.9 | .53 | .96 (.95–.98) | .57 (.52–.62) | .73 (.69–.76) | .93 (.89–.96) | |
| 2 | ICD-9 code | 345.x, 780.3 | .51 | .96 (.95–.98) | .55 (.50–.60) | .72 (.68–.75) | .93 (.89–.96) | |
| 48 | ICD-9 and ICD-10 code | 345.x (except 345.2 and 345.3), 780.33, 780.39 G40.x (except G40.xx1), R56.1, R56.9 |
.53 | .96 (.95–.98) | .55 (.49–.60) | .72 (.68–.75) | .93 (.89–.96) | |
| 10 | ICD-10 code | G40.x, G41.x, R56.x | .51 | .96 (.95–.98) | .55 (.49–.60) | .72 (.68–.75) | .93 (.89–.96) | |
| 6 | ICD-9 code | 345.x, 333.2, 780.3, 779.0, 781.0 | .48 | .96 (.95–.98) | .52 (.47–.57) | .71 (.67–.74) | .92 (.89–.96) | |
| 7 | ICD-9 code | 345.x, 780.30, 780.31, 780.39, 333.2, 779.0, 779.1, 780.02, 780.2 |
.48 | .96 (.95–.98) | .52 (.46–.57) | .70 (.67–.74) | .92 (.88–.96) | |
| 52 | ICD-9 and ICD-10 code | 345.x, 780.3 G40.x, G41.x, R56.x |
.51 | .96 (.94–.98) | .57 (.52–.62) | .73 (.69–.76) | .92 (.89–.96) | |
| 78 | ICD-9 code + ASM | 345 (.2, .3, + 1 ASM), 780.3 + 1 ASM | .68 | .95 (.93–.97) | .73 (.68–.77) | .81 (.77–.84) | .92 (.89–.95) | |
| 94 | ICD-9 or ICD-10 code ± ASM | 345.x (except 345.2 and 345.3), G40.x (except G40.xx1) ± ASM or 345.2, 345.3, 780.3x, G40.xx1, G41, R56.x with ≥1 ASM |
.66 | .95 (.93–.97) | .72 (.68–.77) | .80 (.77–.84) | .92 (.89–.95) | |
| Highest specificity | 34 | At least 1 hospitalization with an ICD-9 or ICD-10 code | 345.x (except 345.2 and 345.3) or G40.x | .18 | .21 (.18–.25) | .97 (.95–.99) | .89 (.84–.95) | .51 (.47–.55) |
| 35 | At least 1 hospitalization or 3 outpatient visits at least 30 days apart within 1 year with ICD-9 or ICD-10 code | 345.x (except 345.2 and 345.3) or G40.x | .48 | .53 (.49–.58) | .94 (.92–.97) | .92 (.88–.95) | .63 (.59–.67) | |
| 32 | At least 1 hospitalization or 4 outpatient visits at least 30 days apart within 3 years with ICD-9 or ICD-10 code | 345.x (except 345.2 and 345.3) or G40.x | .45 | .51 (.46–.55) | .94 (.92–.96) | .91 (.87–.95) | .61 (.57–.66) | |
| 36 | At least 1 hospitalization or 3 outpatient visits at least 30 days apart within 2 years with ICD-9 or ICD-10 code | 345.x (except 345.2 and 345.3) or G40.x | .51 | .57 (.52–.62) | .94 (.91–.96) | .92 (.88–.95) | .65 (.61–.69) |
Abbreviations: ASM, antiseizure medication; CI, confidence interval; ICD-9- CM, International Classification of Diseases, 9th Revision, Clinical Modification; ID, algorithm identification number; NPV, negative predictive value; PPV, positive predictive value; Sn, sensitivity; Sp, specificity.
Refers to ICD-9-CM or ICD-10-CM codes.
The algorithm with the highest specificity (.97) required a hospital encounter coded with an ICD-9-CM code 345.xx (excluding .2 and .3) or an ICD-10-CM code G40 in any diagnostic position. Outpatient encounters would not meet the criteria for an epilepsy diagnosis in this algorithm. Specificity remained similarly high, at .94, when the same codes were required either with hospitalization, or with multiple outpatient encounters within a specified time frame (IDs 32, 35, and 36). Sensitivities of these high-specificity algorithms were low, ranging from .21 to .57.
3.2 |. Sensitivity analyses
When positive epilepsy cases were defined as those with possible or definite epilepsy on chart review, claims-based definitions generally had improved sensitivity and specificity. However, the best performing algorithms generally remained unchanged. Of the 20 highest performing algorithms, as determined by YI, 19 were the same in both the main analysis and in this sensitivity analysis.
4 |. DISCUSSION
The current study assessed numerous claims-based case definitions for epilepsy using ICD-9-CM codes, ICD-10-CM codes, and ASM prescriptions from adult patient encounters in a multicenter New York City health system. We found that the most accurate ICD-9-CM- and ICD-10-CM-based case definitions performed similarly. Several ICD-9-CM- and ICD-10-CM-based case definitions demonstrated a sensitivity, specificity, NPV, and PPV of >80%. The claims-based case definition of epilepsy with the highest YI was an ICD-9-CM-based algorithm requiring one encounter coded with code 345 (.2, .3) or 780.3 as well as an ASM prescription. This case definition achieved a high sensitivity of 95% but was limited by a low specificity (73%) compared to other high-performance algorithms.
The ICD-10-CM-based algorithm with the highest YI (.67) required a G40.x diagnosis code in either the primary diagnostic position or in ≥2 encounters during the study period. A more inclusive ICD-10-CM-based case definition, which simply required at least one encounter with G40.x in any diagnostic position, achieved a similarly high YI (.66). Both ICD-10-based algorithms achieved a sensitivity, specificity, NPV, and PPV of >80%. The addition of an ASM requirement to ICD-10-CM-based algorithms improved the specificity but resulted in lower sensitivity. Using either ICD-9-CM or ICD-10-CM codes with or without an ASM requirement also allowed for the development of accurate case definitions that could achieve sensitivities, specificities, NPVs, and PPVs all of >80%.
The ICD-9-CM algorithms examined in our study performed similarly to previously published United States-based ICD-9-CM case definitions of epilepsy. For example, a 2005 United States-based study found that combining a single ICD-9-CM code either for epilepsy, convulsions, or a related diagnosis with an ASM prescription resulted in a sensitivity of 82% and a specificity of 94%.14 In 2017, another United States-based study reported that the same algorithm had a sensitivity of 69% and a specificity of 77% when applied to their study cohort.15 When applied to our dataset, this algorithm achieved a sensitivity of 87% that aligned closely with 2017 study15 but a specificity of 75% that was more consistent with the 2005 validation study.14 Although these variations are likely due to differences in sample ascertainment between the three studies, the alignment of our findings with both studies demonstrates that the overall performance of ICD-9-CM-based algorithms in our cohort was largely consistent with past work in the United States. However, it also demonstrates the importance of validating these algorithms in multiple cohorts and databases to ascertain their validity across various populations.
To our knowledge, there is only one other United States-based study that validated an ICD-10-CM-based case definition of epilepsy.12 Their work assessed the performance of two ICD-10-CM-based algorithms: G40.x in any position and G40.x and/or R56.x in any position. The study used outpatient encounters among six providers at an epilepsy clinic, with chart review defined as the gold standard. The reported sensitivities (84.4% for G40.x and 97.7% for G40.x/R56.x) and specificities (79.4% for G40.x and 44.1% for G40.x/R56.x) of these case definitions were similar to those reported in the current study (see IDs 1 and 10). Our study builds on this work by assessing a much wider range of ICD-9-CM- and ICD-10-CM-based case definitions of epilepsy. Additionally, we incorporated ASM data into our case definitions. Lastly, although the above published ICD-10-CM validation draws exclusively from outpatient encounters among six specific providers,12 our study draws from encounters with many different providers in both the inpatient and outpatient setting, providing more generalizable results.
In comparison to validation studies performed outside the United States, these algorithms performed similarly. However, in some cases, certain algorithms were more accurate outside of the United States. For example, a validation study conducted in Canada demonstrated an ICD-9-CM-based algorithm achieving PPV and NPV of 99% and 97%, respectively, which is higher than the best performing algorithm in the current study.5 ICD-10-CM-based algorithms in the Canadian study performed similarly. In an Australian study validating ICD-10-based algorithms, the best performing algorithm demonstrated a YI of .884.6 Despite some variation, the best performing algorithms in the current study demonstrated sensitivity, specificity, PPV, and NPV of >80%, which is aligned with the best performing algorithms inside and outside of the United States.4
Assuring the accuracy of ICD-10-CM-based case definitions of epilepsy is essential for disease surveillance and epidemiologic research.16 Previous work has demonstrated that the transition from ICD-9-CM- to ICD-10-CM-based systematically biased estimates of disease prevalence.17 A retrospective United States-based analysis of the National Inpatient Sample demonstrated significant changes in the prevalence of several neurological diseases during the 2015 transition from ICD-9-CM to ICD-10-CM coding.18 Notably, the authors found a particularly dramatic decrease in the prevalence of status epilepticus during this transition.18 Regardless of the reason for these discrepancies, these findings highlight the challenges presented by the transition from ICD-9-CM to ICD-10-CM and the importance of validating ICD-10-CM-based case definitions for epilepsy within the United States.
A recent systematic review of previous validations of epilepsy case definitions found that the algorithms achieving the highest PPV were those that used disease codes (ICD-9345.xx or ICD-10 G40–41.x) without convulsion codes (ICD-10 R56, ICD-9780.3, 780.39), and included one or more ASMs.4 This was consistent with our findings as demonstrated by the high PPV in algorithms ID 61, 76, and 68. This systematic review also found that achieving a balance between PPV and sensitivity required the use of disease codes alone, without an ASM requirement.4 Once again, this aligned with our finding that several algorithms using disease codes alone (IDs 1, 4, 5, and 9) demonstrated both a PPV and sensitivity exceeding 80%. Lastly, this study reports that algorithms achieving high sensitivity were those that used symptom codes in combination with one or more ASM.4 Although we did not test algorithms that included only symptom codes with an ASM, we did find that algorithms requiring either disease codes or symptom codes in combination with ASMs achieved a sensitivity of 95%, and a specificity of >70% (IDs 78 and 93).
Our study demonstrated a similar performance between ICD-9-CM and ICD-10-CM coding. The algorithm using ICD-9-CM code 345.xx performed almost identically to the algorithm using G40.x regarding sensitivity, specificity, NPV, and PPV (see IDs 1 and 9). This is consistent with a Canadian validation finding that the PPV of these same ICD-9 and ICD-10 codes was 98.9% and 98.6%, respectively, in their study cohort.5
The current study has several limitations. First, the patient population studied was all from a single health system within a single region in the United States and may not be representative of the general US population. However, because EPIC is the most common database used in the United States, these results may be applicable to other hospitals using this EHR.19 A second limitation is our use of chart review as the gold standard for an epilepsy diagnosis. Variations in the diagnostic expertise of providers and the quality of documentation could bias these results. However, the vast majority of prior validation studies used this method as well, which allows for comparison across studies. Lastly, we intentionally queried a population that was enriched for epilepsy, and thus the prevalence of epilepsy in the cohort studied is far higher than the general population. Thus, our findings, especially the PPV and NPV, may not be applicable to the general population.20
5 |. CONCLUSIONS
This study is the first, to our knowledge, to comprehensively assess the performance of multiple ICD-9-CM- and ICD-10-CM-based case definitions of epilepsy in the United States. We identified multiple EHR-based case definitions that were sensitive and specific for a diagnosis of epilepsy. We also identified algorithms that demonstrated a high degree of specificity and others that showed a high sensitivity. Our findings may guide future researchers who may prioritize varying degrees of sensitivity and specify based on their goals; for example, studies where it is important to capture all cases would want to use a definition with high sensitivity, whereas one that prioritizes true cases to examine specific characteristics would prioritize a higher specificity. Overall, the algorithms we identified performed similarly in the United States as they did in other countries. Additionally, ICD-9-CM-based algorithms performed similarly to ICD-10-CM-based algorithms. The application of these claims-based definitions should facilitate disease surveillance and health outcomes research that can benefit people with epilepsy in the United States.
Supplementary Material
SUPPORTING INFORMATION
Additional supporting information can be found online in the Supporting Information section at the end of this article.
Key points.
ICD-9-CM- and ICD-10-CM-based case definitions demonstrated similar accuracy.
ICD-10-based algorithms with the highest YI (.67) used the ICD-10-CM code G40.x.
Encounters with ICD-9-CM 345 or 780.39 or ICD-10-CM G40, G41, or R56.9 had highest sensitivity (.96).
One or more hospitalizations with ICD-9-CM 345.x (except 345.2 and 345.3) or ICD-10-CM G40.x had the highest specificity (.97).
An accurate ICD-10-CM-based case definition of epilepsy is essential for disease surveillance and epidemiologic research.
ACKNOWLEDGMENTS
This work was supported in part through the computational and data resources and staff expertise provided by Scientific Computing and Data at the Icahn School of Medicine at Mount Sinai and supported by Clinical and Translational Science Awards grant UL1TR004419 from the National Center for Advancing Translational Sciences.
Funding information
National Center for Advancing Translational Sciences, Grant/Award Number: UL1TR004419
Footnotes
PATIENT CONSENT STATEMENT
Patient consent was not required for this study.
DATA REPORTING GUIDELINES
This article is consistent with the CONSORT guidelines for ethical publication.
CONFLICT OF INTEREST STATEMENT
L.J.B. receives funding for grants unrelated to this work from the National Institute on Aging (R01NS130119, K23 AG080163). N.J. receives funding for grants unrelated to this work from NINDS (R01NS130119; 4R00HD106060–03). She also receives an honorarium for her work as an associate editor of Epilepsia. None of the other authors has any conflict of interest to disclose. We confirm that we have read the Journal’s position on issues involved in ethical publication and affirm that this report is consistent with those guidelines.
DATA AVAILABILITY STATEMENT
The data that support the findings of this study are available from the corresponding author upon reasonable request.
REFERENCES
- 1.Feigin VL, Abajobir AA, Abate KH, Abd-Allah F, Abdulle AM, Abera SF, et al. Global, regional, and national burden of neurological disorders during 1990–2015: a systematic analysis for the global burden of disease study 2015. Lancet Neurol. 2017;16:877–97. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.DeGiorgio CM, Curtis A, Carapetian A, Hovsepian D, Krishnadasan A, Markovic D. Why are epilepsy mortality rates rising in the United States? A population-based multiple cause-of-death study. BMJ Open. 2020;10(24):e035767. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Evans RS. Electronic health records: then, now, and in the future. Yearb Med Inform. 2016;25(Suppl 1):S48–S61. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Mbizvo GK, Bennett KH, Schnier C, Simpson CR, Duncan SE, Chin RFM. The accuracy of using administrative healthcare data to identify epilepsy cases: a systematic review of validation studies. Epilepsia. 2020;61(7):1319–35. 10.1111/epi.16547 [DOI] [PubMed] [Google Scholar]
- 5.Jetté N, Reid AY, Quan H, Hill MD, Wiebe S. How accurate is ICD coding for epilepsy? Epilepsia. 2010;51:62–9. [DOI] [PubMed] [Google Scholar]
- 6.Tan M, Wilson I, Braganza V, Ignatiadis S, Boston R, Sundararajan V, et al. Development and validation of an epidemiologic case definition of epilepsy for use with routinely collected Australian health data. Epilepsy Behav. 2015;51:65–72. [DOI] [PubMed] [Google Scholar]
- 7.Christensen J, Vestergaard M, Olsen J, Sidenius P. Validation of epilepsy diagnoses in the Danish National Hospital Register. Epilepsy Res. 2007;75:162–70. [DOI] [PubMed] [Google Scholar]
- 8.Lee SY, Chung SE, Kim DW, Eun SH, Kang HC, Cho YW, et al. Estimating the prevalence of treated epilepsy using administrative health data and its validity: ESSENCE study. J Clin Neurol. 2016;12(4):434–40. 10.3988/jcn.2016.12.4.434, 27273925 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Reid AY, St Germaine-Smith C, Liu M, Sadiq S, Quan H, Wiebe S, et al. Development and validation of a case definition for epilepsy for use with administrative health data. Epilepsy Res. 2012;102(3):173–9. 10.1016/j.eplepsyres.2012.05.009 [DOI] [PubMed] [Google Scholar]
- 10.Boesen MS, Cacic Hribljan M, Christensen SK, Klein-Petersen AW, El Mahdaoui S, Sagar MV, et al. Validation of pediatric idiopathic generalized epilepsy diagnoses from the Danish National Patient Register during 1994–2019. Clin Epidemiol. 2022;14:501–9. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Syvertsen M, Nakken KO, Edland A, Hansen G, Hellum MK, Koht J. Prevalence and etiology of epilepsy in a Norwegian county-a population based study. Epilepsia. 2015;56(5):699–706. 10.1111/epi.12972, 25810072 [DOI] [PubMed] [Google Scholar]
- 12.Smith JR, Jones FJS, Fureman BE, Buchhalter JR, Herman ST, Ayub N, et al. Accuracy of ICD-10-CM claims-based definitions for epilepsy and seizure type. Epilepsy Res. 2020;166:106414. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Fisher RS, Acevedo C, Arzimanoglou A, Bogacz A, Cross JH, Elger CE, et al. ILAE official report: a practical clinical definition of epilepsy. Epilepsia. 2014;55(4):475–82. 10.1111/epi.12550, 24730690 [DOI] [PubMed] [Google Scholar]
- 14.Holden EW, Grossman E, Nguyen HT, Gunter MJ, Grebosky B, Von Worley A, et al. Developing a computer algorithm to identify epilepsy cases in managed care organizations. Dis Manag. 2005;8:1–14. [DOI] [PubMed] [Google Scholar]
- 15.Moura LM, Price M, Cole AJ, Hoch DB, Hsu J. Accuracy of claims-based algorithms for epilepsy research: revealing the unseen performance of claims-based studies. Epilepsia. 2017;58(4):683–91. 10.1111/epi.13691 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Iezzoni LI. Assessing quality using administrative data. Ann Intern Med. 1997;127:666–74. [DOI] [PubMed] [Google Scholar]
- 17.Yoon J, Chow A. Comparing chronic condition rates using ICD-9 and ICD-10 in VA patients FY2014–2016. BMC Health Serv Res. 2017;17:572. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.Hamedani AG, Blank L, Thibault DP, Willis AW. Impact of ICD-9 to ICD-10 coding transition on prevalence trends in neurology. Neurol Clin Pract. 2021;11(5):e612–e619. 10.1212/CPJ.0000000000001046 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Eschenroeder HC Jr, Manzione LC, Adler-Milstein J, Bice C, Cash R, Duda C, et al. Associations of physician burnout with organizational electronic health record support and after-hours charting. J Am Med Inform Assoc. 2021;28(5):960–6. 10.1093/jamia/ocab053 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.Monaghan TF, Rahman SN, Agudelo CW, Wein AJ, Lazar JM, Everaert K, et al. Foundational statistical principles in medical research: sensitivity, specificity, positive predictive value, and negative predictive value. Medicina (Kaunas). 2021;57:503. [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
Data Availability Statement
The data that support the findings of this study are available from the corresponding author upon reasonable request.