How accurate do self-reported seizures need to be for effective medication management in epilepsy?

Abstract

Studies suggest that self-reported seizure diaries suffer from 50% under-reporting on average. It is unknown to what extent this impacts medication management. This study used simulation to predict the seizure outcomes of a large heterogeneous clinic population treated with a standardized algorithm based on self-reported seizures. Using CHOCOLATES, a state-of-the-art realistic seizure diary simulator, 100,000 patients were simulated over 10 years. A standard algorithm for medication management was employed at 3-month intervals for all patients. The impact on true seizure rates, expected seizure rates and time-to-steady-dose were computed for self-reporting sensitivities 0-100%. Time-to-steady-dose and medication usage mostly did not depend on sensitivity. True seizure rate decreased minimally with increasing self-reporting in a non-linear fashion, with the largest decreases at low sensitivity rates (0-10%). This study suggests that an extremely wide range of sensitivity will have similar seizure outcomes when clinically treated using an algorithm similar to the one presented. Conversely, patients with sensitivity less than or equal to 10% would be expected to benefit (via lower seizure rates) from objective devices that provide even small improvements in seizure sensitivity.

BACKGROUND:

The most readily available metric for degree of illness in epilepsy is self-reported seizures (SRS)¹. One common use of SRS is clinical treatment with anti-seizure medications (ASMs). Investigators and clinicians worry that SRS has reliability concerns because of poor correlation to intracranial recordings^2,3 or to scalp EEG recordings^4,5. Some have suggested that objective metrics should replace SRS, such as subscalp EEG⁶ or wearable devices⁷. Nevertheless, any seizure detector has an imperfect sensitivity and a nonzero false alarm rate. Thus, the question arises: when would objective tools be better for ASM management, and when is SRS sufficient?

One approach to understand this situation is to use the framework of signal-to-noise ratio (SNR) used by engineers to quantify the utility of a measurement. In the present context, we will use sensitivity and false alarms from SRS to define SNR:

$$Sensitivity(\text{\%})=\frac{reportedtrueseizures}{totaltrueseizures}$$ Eq. 1:

$$Falsealarmrate=\frac{reportednon–seizureevents}{1month}$$ Eq. 2:

$$SNR=\frac{sensitivity}{falsealarmrate}$$ Eq. 3:

SNR can be thought of as overall SRS accuracy. Reports vary widely about sensitivity ^2,4,5, likely because individuals vary widely, estimates as low as 13% and as high as 71% have been found in different subgroups, but a review across multiple studies and modalities suggested the typical sensitivity may be <50%⁴. As for false alarm rate, this quantity is unknown and challenging to study in the absence of a comprehensive gold standard (see Appendix). Here too, wide estimates are possible^8,9, though without proper controls they are difficult to interpret.

With the availability of a statistically realistic seizure diary simulator¹⁰, it is possible to produce a simulation that would address the utility question under various possible scenarios. Our objective was to investigate the impact of different self-report accuracy (SNR) values for SRS on seizure outcomes when treating patients in clinic.

METHODS

The simulations were based on CHOCOLATES¹⁰, open-source software for generating seizure diaries that have characteristics observed in clinical studies. These features are: (1) heterogeneity in average seizure rates across subjects¹¹, (2) a consistent relationship between average and standard deviation in seizure rate across patients (the “L-relationship”)¹², (3) multiple coexisting seizure risk cycles^13,14, (4) seizure clustering features^15,16, and (5) limitations on minimum inter-seizure intervals^11,17,18. The output of CHOCOLATES is a series of seizure counts, representing the number of seizures the synthetic patient reports within a user-specified time window (such as daily, or hourly). CHOCOLATES reproduces typical SRS diaries from a heterogeneous community of patients. Figure 1 illustrates the pipeline used to generate both ground truth clinical seizure diaries as well as self-reported diaries derived from these ground truth diaries. First, a realistic self-reported seizure frequency is chosen with the help of CHOCOLATES. This frequency is then modified to account for the self-report accuracy rate (increased due to sensitivity and decreased due to false alarm rate). This was input to CHOCOLATES to produce the ground truth clinical diary at the modified typical frequency. The ground truth diary was stochastically modified to account for the self-report accuracy (SNR) -seizures deleted based on sensitivity and added based on false alarm rate-, producing the self-reported diary. In this way, two parallel diaries were generated, one which represents ground truth clinical seizures, and the second representing SRS.

FIGURE 1:

Flow diagram for simulation. The upper flow diagram shows how selecting a frequency and adjusting it based on self-report accuracy (signal to noise ratio (SNR)) allows the seizure diary simulator CHOCOLATES and medication effects to produce a “true diary”. That diary is then adjusted to account for the SNR (including under-reporting and over-reporting) to form a self-reported diary. Medications are adjusted based only on the self-reported diary data using the flow diagram on the lower half of the figure (gray box). The seizure rate from the previous 3-month clinic visit is compared to the current rate in the most recent 3 months. If the new rate is 50% reduced or more, then meds are not changed. Otherwise, meds are increased with a certain probability. If meds are unchanged for 2 years, they can be reduced.

Clinic visits were assumed to occur every 3 months for a total of 10 years in a set of 100,000 virtual patients. A standard clinical decision rule (Figure 1 in gray box) was applied at each visit as follows:

Let the current and prior 3-month seizure count be C_CURRENT and C_PRIOR and let probability of ASM increase be P_increase:

1. If C_CURRENT > (½) C_PRIOR then increase ASM dose, probability P_increase.

2. If C_CURRENT = 0 or C_CURRENT <= (½) C_PRIOR then no change.

3. If no change has been true for 2 years, then decrease ASM dose.

For rule #1, ASM adjustments were in half dose increments. The rule used (½) C_PRIOR because often clinicians consider a 50% reduction a significant change¹⁹. P_increase=0.3, consistent with clinical data²⁰. A full dose of medicine was assumed to decrease 20% of true clinical seizures on average, and a half dose was assumed to decrease 10%. These values align with meta-analysis²¹. If an ASM was set to full dose and ASM increase was needed, a new drug at half dose was added to the existing regimen. Similarly, if a patient had a half dose of a medication and a reduction was needed, the final ASM was removed. Thus, patients taking more than 1 ASM would take full strength of all but the final ASM, which would either be full or half dose. ASM count was limited between 1 and 6 after any med was started.

What is the chance an added ASM results in seizure freedom? This has been previously reported²⁰ in a large cohort. What follows is the probability of seizure freedom with the addition of that 1^rst…,6^th ASM, given that the previous medicines failed to produce seizure freedom: 46%,28%,24%,15%,14%,14%. Each patient’s form of seizure-freedom is also subject to a probabilistic pattern²² reported in a large cohort of outpatients: (1) lasting early (37%), (2) delayed last (22%), (3) fluctuating 1-year seizure freedom (16%), (4) no extended seizure freedom (25%). Both probability of seizure-freedom and pattern were built into our medication change model, so that if any new ASM was added, there was a realistic chance of achieving various types of seizure freedom.

To account for different self-report accuracy (SNR) values, we set the false alarm rate to be fixed 1 seizure per month and varied the sensitivity values between 10%… 100%. In this way, SNR (expressed in terms of monthly false alarms) can be numerically equivalent to sensitivity. The number of months until a stable dose of ASM was computed by taking the median number of ASMs during the final one third of the 10 years (at the individual patient level) and determining how long until that individual started taking that number of ASMs. Additional values of false alarm rate were also explored (Appendix).

DATA AVAILABILITY

All data used in this study was synthetically generated using the open-source code.

CODE AVAILABILITY

Github code can be accessed here: https://github.com/GoldenholzLab/WEARsimulator/.

RESULTS

The median number of ASMs needed per patient ranged between 1.5 and 2.1 across all values of self-report accuracy except SNR=0 (0 meds). Median seizure rate ranged 2.3 to 1.9 for all self-report accuracy values (at SNR=0, rate=6.5). The duration until stable ASM dose was 20-24 months for all SNR other than SNR=0, which was 0 months. Results are summarized in Figure 2. Additional simulations for other values of false alarm rates between 2 per day down to 1 per year are included in the Appendix, but the results are approximately the same throughout.

Figure 2:

Result of simulating 100,000 patients for 10 years. Upper left – the daily number of medications are shown as a function of sensitivity (SNR). A red curve is plotted through the median values. Upper right – the average seizure rate per patient as a function of SNR. A red curve is plotted through the median values. Lower left – the months until stable medication dose versus SNR is shown, with a red curve through the median values. There is no dramatic difference for any SNR value. Lower right – the median values from the average seizure rate per patient is plotted (x axis) vs. daily number of medications. These figures indicate that months until stable dose, and typical number of medications does not depend on SNR. Additionally, with the exception of the lowest SNR values, the typical seizure rate decreased only modestly in response to higher SNR values.

DISCUSSION

This study evaluated the impact of different values of self-report accuracy (SNR) values from SRS diaries on simulated patients in clinic. It was shown that virtually any SNR value resulted in similar distribution of patients with few or many ASMs, and similarly the time to achieve a stable dose was roughly 2 years for the typical patient. Conversely, the typical number of seizures seen with different SNR values changed dramatically only for the lowest SNR values. These findings suggest that even if self-reported sensitivity is as low as 10%, a more objective detection device would be unlikely to have a dramatic impact on a large clinic population. Only patients with extremely poor SNR (less than 10% or worse) would be expected to see a significant seizure burden reduction with a high sensitivity wearable. Importantly, the simulator (CHOCOLATES) ¹⁰ accounts for a heterogeneous seizure frequency, so patients of low rates, high rates and everything in between were included, at the rates previously found in outpatients¹¹.

Does this mean that clinicians should “trust” self-reported data? We do not think this question is the correct approach. Rather, we ask: “in the setting of outpatient ASM management, how much confidence can be assigned to SRS?” For most values of self-report accuracy, high confidence can be assigned to SRS when managing ASMs. In other contexts, SRS with low or moderate SNR would pose unacceptable risk, (e.g. SUDEP and injury prevention).

It is worthwhile considering the observation that patients may under-report more at night than during the day^4,23,24. Our present study did not account for day or night, rather simply accounting for total daily counts of seizures. There are several cases to consider: patient A, who only has daytime seizures, patient B, who has only night time seizures, and patient C, who has both day and night events. Statistically, one would assume patient A would have higher SNR than patient B, and one could use the results shown here to derive expectations given their SNR value. In the case of patient C, things are more complex, because the overall SNR reflects a weighted SNR from the daytime and nighttime. From the perspective of treatment in clinic, knowing this overall SNR ultimately is the only value needed, so such details are not required. Conversely, if one wanted to target therapy for a specific time period, (e.g. nighttime), would could consider the SNR of each time period independently. This might mean that nocturnal devices would be recommended to boost the night-time SNR for a patient that had very low nocturnal SNR.

The results here suggest that exact numbers of seizures are not necessary for adequate clinical treatment. Nevertheless, some patients or caregivers would feel greater comfort with apparent increased certainty afforded by objective device monitoring. Due to the many psychological factors involved here, we recommend an individualized approach to patients and caregivers, accounting for perceptions of certainty, false alarms, and general anxiety levels.

These results must be taken in context. The treatment model presented here represents a realistic guess at one possible treatment strategy in a space of nearly infinite options available to clinicians. CHOCOLATES was developed to account for many known statistical features of self-reported seizure diaries¹⁰. That tool represents the current state-of-the-art in seizure-diary simulation, yet many details remain unknown. For example, does the tool equally represent all subtypes of epilepsy²⁵ and all subtypes seizures²⁶? The data used to accommodate the possibility of seizure-freedom was derived from the experience of one center in Scotland²⁰ with 1,795 patients. The data to understand typical medication efficacy comes from meta-analysis of 63 RCTs²⁷ from 11 ASMs. Although a wide variety of data collection techniques and methodologies were used to accumulate the constraints needed to build both CHOCOLATES and the present simulator, could these be insufficient for a more nuanced specific situation or specific treatment style? These questions are currently open for further study. Additionally, assumptions were made for the method to approximate a “true” seizure rate (Figure 1), which cannot be truly known with present technology and therefore represent a best guess. Similarly, seizure-reporting accuracy was by necessity assumed to be constant over time on average but may not be true in many real-world situations. Moreover, the seizure-freedom of patients in clinic was simulated to reflect the rates seen in a large-scale single-center study of 1795 patients over a 30-year period²⁰. When larger multi-center datasets become available, the assumption about seizure freedom may require adjustment. It is therefore fortunate that when new data becomes available, the open-source code can be modified by the research community.

In conclusion, the present study finds that SRS may be sufficient in many cases for clinical ASM management. A surprisingly wide range of SNR values and a standard treatment algorithm result in similar clinical outcomes. Conversely, patients with extremely low reporting SNR tend to suffer dramatically more seizures than those with even modest SNR. Such patients would be expected to clinically benefit from objective seizure detections devices due to the higher SNR.

APPENDIX

Challenges with False Alarm Rate

False alarm rates may also influence the effects described in the main manuscript. However, in order to know the false alarm rate for self-reported seizures, it is necessary to have a gold standard upon which to compare. This has proven extremely challenging. In the setting of the epilepsy monitoring unit, reports exist for how often patients report seizures which turn out to have no EEG correlate. These non-correlated events cannot be automatically presumed to be “false alarms” because many seizures can occur without scalp EEG correlate. These events are better detected with intracranial EEG. On the other hand, studies employing intracranial EEG also cannot be the gold-standard for true seizures, unless there is absolute certainty that all seizure foci are represented by the specific intracranial electrodes. This certainty is extremely challenging to have, and often comes retrospectively after a surgery results in long term complete seizure freedom (several years) which requires no anti-seizure medication. At present, no large-scale study has been conducted with a cohort of these types of patients to look at the false alarm rate of self-reported events compared to a true seizure log derived from intracranial electrodes. Other studies, employing intracranial electrodes either during short term or long term recordings have made anecdotal observations about the discordance of some self-reported events and intracranial recordings, however the interpretation of such reports has the limitations mentioned above.

In the absence of clear data to guide our simulations, we have resorted to a simplifying assumption of a low self-reported false alarm rate of 1 per month. This assumption may be too low or too high, and future studies will help inform that. For now, we have also explored several additional false alarm rates (below) to see what effect this selection has on the overall results.

One speculative false alarm rate model

Currently, it is unknown how frequent patients will self-report events that are not seizures (see above). However, it is logical to assume that perhaps the rate of self-reported false alarm events has two components: a fixed rate, as well as a rate that depends on the true rate of seizures. Thus, a model can be constructed as follows (FPR = false positive rate):

$$\text{Total FPR}=\text{Static FPR}+\text{Proportional FPR}*(\text{Seizure Frequency})$$

Further studies are needed to determine if such a hybrid model reflects the clinical reality of most patients. For the purposes of the present study, we did not explore the above speculative model, because it adds additional degrees of freedom without data driven constraints.

Influence of different false alarm rates

Shown in the figure, higher sensitivity values result in higher drugs/day typically, and fewer seizures per day. However, these trends are relatively small and stable after the lowest sensitivities. Oddly, the number of seizures/day and drugs/day do not follow a linear relationship with false alarm rate. This is due to the impact of having enough self-reported seizures (due to false alarms) that drugs are started. When that happens, the number of true seizures becomes dramatically lower. As seen more clearly in the data table below, when the false alarm rate is 1/180 days then some patients start getting meds, seizure freedom rises from zero and the number of seizures/month falls.

An important note is made here. In the present simulations, the FAR is considered a known value. In that case, the treating physician performs a discounting prior to considering a reported seizure rate. For example, if FAR = 1 per month, and at 3 months a patient reports 4 seizures per month, the physican would consider this number to be 3 = 4 – 1.

If the FAR is not known, this discounting procedure is not possible, and that would increase the number of drugs and decrease the number of seizures for all patients with a FAR>0.

Sens	FAR	szfree	meanDrug	meanSz	how_long
0.0	0.0	0.00000	0.000000	6.495726	0.0
0.1	0.0	0.52847	1.461538	2.341880	20.0
0.2	0.0	0.55001	1.794872	2.162393	22.0
0.3	0.0	0.55838	1.897436	2.068376	23.0
0.4	0.0	0.56783	1.948718	2.017094	23.0
0.5	0.0	0.57000	2.000000	1.982906	24.0
0.6	0.0	0.57301	2.025641	1.991453	24.0
0.7	0.0	0.57604	2.051282	1.957265	24.0
0.8	0.0	0.57949	2.076923	1.982906	24.0
0.9	0.0	0.57948	2.102564	1.948718	24.0
1.0	0.0	0.57850	2.128205	1.940171	24.0
0.0	0.002778	0.00000	0.000000	6.470085	0.0
0.1	0.002778	0.53274	1.538462	2.256410	20.0
0.2	0.002778	0.55719	1.871795	2.068376	22.0
0.3	0.002778	0.56718	1.974359	1.982906	23.0
0.4	0.002778	0.57394	2.025641	1.982906	24.0
0.5	0.002778	0.57508	2.051282	1.940171	24.0
0.6	0.002778	0.57777	2.102564	1.923077	24.0
0.7	0.002778	0.58085	2.128205	1.923077	24.0
0.8	0.002778	0.58298	2.153846	1.923077	24.0
0.9	0.002778	0.58342	2.179487	1.905983	24.0
1.0	0.002778	0.58289	2.179487	1.923077	25.0
0.0	0.005556	0.53923	2.051282	2.427350	28.0
0.1	0.005556	0.56474	2.358974	2.145299	29.0
0.2	0.005556	0.57348	2.461538	2.068376	29.0
0.3	0.005556	0.57592	2.512821	2.025641	29.0
0.4	0.005556	0.58062	2.538462	1.974359	29.0
0.5	0.005556	0.58203	2.564103	1.965812	29.0
0.6	0.005556	0.58409	2.589744	1.940171	29.0
0.7	0.005556	0.58390	2.615385	1.923077	29.0
0.8	0.005556	0.58380	2.641026	1.931624	29.0
0.9	0.005556	0.58471	2.641026	1.914530	29.0
1.0	0.005556	0.58222	2.641026	1.888889	29.0
0.0	0.011111	0.00000	0.000000	6.521368	0.0
0.1	0.011111	0.52441	1.487179	2.341880	20.0
0.2	0.011111	0.54833	1.794872	2.153846	22.0
0.3	0.011111	0.55681	1.923077	2.094017	23.0
0.4	0.011111	0.56564	1.974359	2.051282	24.0
0.5	0.011111	0.57234	2.000000	1.982906	24.0
0.6	0.011111	0.57390	2.025641	1.974359	24.0
0.7	0.011111	0.57763	2.051282	1.957265	24.0
0.8	0.011111	0.57928	2.076923	1.957265	24.0
0.9	0.011111	0.57992	2.102564	1.948718	24.0
1.0	0.011111	0.58030	2.102564	1.905983	24.0
0.0	0.033333	0.00000	0.000000	6.470085	0.0
0.1	0.033333	0.52740	1.461538	2.333333	20.0
0.2	0.033333	0.54722	1.769231	2.128205	22.0
0.3	0.033333	0.55990	1.897436	2.085470	23.0
0.4	0.033333	0.56259	1.974359	2.059829	24.0
0.5	0.033333	0.56873	2.000000	2.008547	24.0
0.6	0.033333	0.57646	2.025641	1.974359	24.0
0.7	0.033333	0.57528	2.051282	1.987179	24.0
0.8	0.033333	0.57969	2.076923	1.931624	24.0
0.9	0.033333	0.57969	2.102564	1.931624	24.0
1.0	0.033333	0.58237	2.102564	1.905983	24.0
0.0	0.142857	0.59477	3.282051	1.846154	30.0
0.1	0.142857	0.58206	2.948718	2.017094	30.0
0.2	0.142857	0.58335	2.974359	2.008547	30.0
0.3	0.142857	0.58248	3.000000	1.982906	30.0
0.4	0.142857	0.58436	3.025641	1.948718	30.0
0.5	0.142857	0.58584	3.051282	1.931624	30.0
0.6	0.142857	0.58539	3.051282	1.914530	30.0
0.7	0.142857	0.58494	3.076923	1.914530	30.0
0.8	0.142857	0.59089	3.076923	1.897436	30.0
0.9	0.142857	0.58886	3.076923	1.914530	30.0
1.0	0.142857	0.58949	3.102564	1.863248	30.0
0.0	1.0	0.49224	1.589744	2.811966	26.0
0.1	1.0	0.54152	1.974359	2.350427	28.0
0.2	1.0	0.55414	2.102564	2.230769	28.0
0.3	1.0	0.56160	2.205128	2.145299	28.0
0.4	1.0	0.56860	2.256410	2.076923	28.0
0.5	1.0	0.57285	2.307692	2.042735	28.0
0.6	1.0	0.57461	2.333333	2.051282	28.0
0.7	1.0	0.57237	2.358974	2.008547	28.0
0.8	1.0	0.57889	2.384615	1.974359	28.0
0.9	1.0	0.57800	2.410256	1.965812	29.0
1.0	1.0	0.58351	2.410256	1.940171	28.0
0.0	2.0	0.50437	1.692308	2.717949	27.0
0.1	2.0	0.53998	1.974359	2.418803	28.0
0.2	2.0	0.55079	2.102564	2.316239	28.0
0.3	2.0	0.55857	2.179487	2.213675	28.0
0.4	2.0	0.56335	2.230769	2.145299	28.0
0.5	2.0	0.56164	2.282051	2.128205	28.0
0.6	2.0	0.56653	2.307692	2.094017	28.0
0.7	2.0	0.56920	2.333333	2.051282	29.0
0.8	2.0	0.57247	2.358974	2.034188	29.0
0.9	2.0	0.57365	2.384615	2.017094	29.0
1.0	2.0	0.57711	2.384615	1.991453	29.0