Confusing Placebo Effect with Natural History in Epilepsy: A Big Data Approach

Abstract

For unknown reasons, placebos reduce seizures in clinical trials in many patients. It is also unclear why some drugs showing statistical superiority to placebo in one trial may fail to do so in another. Using Seizuretracker.com, a patient-centered database of 684,825 seizures, we simulated “placebo” and “drug” trials. These simulations were employed to clarify the sources of placebo effects in epilepsy, and to identify methods of diminishing placebo effects. Simulation 1 included 9 trials with a 6-week baseline and 6-week test period, starting at time 0, 3, 6…24 months. Here, “placebo” reduced seizures regardless of study start time. Regression-to-the-mean persisted only for 3–6 months. Simulation 2 comprised a 6-week baseline and then 2 years of follow-up. Seizure-frequencies continued to improve throughout follow-up. Although the group improved, individuals switched from improvement to worsening and back. Simulation 3 involved a placebo-controlled “drug” trial, to explore methods of placebo-response reduction. An efficacious “drug” failed to demonstrate a significant effect compared with “placebo” (p=0.12), though modifications either in study start-time (p=0.025) or baseline population reduction (p=0.0028) allowed the drug to achieve a statistically significant effect compared with placebo. In epilepsy clinical trials, some seizure reduction traditionally attributed to placebo effect may reflect the natural course of the disease itself. Understanding these dynamics will allow future investigations into optimal clinical trial design and may lead to identification of more effective therapies.

Epilepsy affects at least 2.2 million Americans, costs $9.5–$12.5 billion annually, and has a 10-fold increased risk of sudden death compared to the general public.¹ Approximately 30% of patients remain uncontrolled by anti-seizure drugs.² Surgery results in long-term seizure-freedom for only 50%.³ More effective therapies are needed. The challenge is not only developing novel treatments, but also demonstrating superiority over increasingly “effective” placebos. A major limitation of current epilepsy trials is the large placebo response, making small but real drug effects difficult to detect.^4,5 A similar placebo effect is seen in device trials.⁶ Strangely, a meta-analysis found placebo effects in epilepsy trials increased steadily from 1989 to 2010, with decreasing apparent drug treatment effects.⁷ This placebo effect increase has been noted in other fields, such as depression, anxiety, and smoking cessation.⁸ Large placebo effects may explain the inability of some recent studies to replicate data showing anti-seizure drug effectiveness; ^9,10 others show only relatively small advantages compared with placebo.¹¹ Understanding placebo effects will be very important for development of new, and more effective, drug therapy in epilepsy and other disorders.^8,10

Placebo responses could involve several mechanisms.^12,13 One is regression to the mean.¹⁴ Assuming a quasi-stable baseline frequency, patients would be more likely to enter clinical trials when seizures transiently increase. When frequency regresses to baseline, placebo or treatment takes apparent credit. Under the Hawthorne effect, subjects’ awareness of being studied leads to improvement, regardless of treatment arm.¹⁵ Psychological influences, including classical (Pavlovian) conditioning, and patient expectations, are also important factors.¹³

Ideally, one could remove these factors’ influence through appropriate study design. Seizuretracker.com is a free, online and mobile resource allowing patients to record clinical information about their seizures.¹⁶ Using data from Seizuretracker.com, we retrospectively studied the natural course of seizures over time. These seizures were recorded by patients and caregivers for their own reasons – not to please trial researchers or clinic physicians. Users never received feedback from software, clinicians or scientists about their seizure logs. Without traditional cues, many expected causes of the placebo effect should be absent as well. Therefore, these data were an ideal substrate for trial simulation and searching for improvement potentially attributed to placebos. We hoped to (1) characterize dynamics in the dataset that would, in a clinical trial context, be interpreted as “placebo effects,” and (2) explore methods of modifying the size of the apparent effect.

Data collection

Sequential patient data (16,800 profiles) from Seizuretracker.com’s public release in December 2007 through April 2014 were collected, de-identified, and un-linked prior to transfer, in accordance with NIH Human Research Protection Program’s recommendations. 8,228 patients had documented 684,825 seizures; 8,572 had not documented any. Ages ranged 0–84 years (median 19). Seizure occurrence times were annotated relative to the first recorded seizure per patient (hereafter called time zero). This normalization step allowed seizure diaries to be directly compared across patients. Patients under age 18 were categorized as children.

Placebo Trial Simulations

Each eligible patient was included in several simulated clinical trials, measuring total seizure counts during baseline, and test periods. Counts were normalized as percent change from baseline. To be eligible for the simulation, patients had to have at least 5 seizures during the baseline period, at least 2 seizures in each 3-week period of the baseline, and no 25-day seizure-free period during the baseline.¹¹ To reduce dropout due to lack of recording, eligible patients also needed at least one additional seizure recorded any time after the test period ended.

Outcome Measures

Simulated trial outcomes were assessed using 50% responder rates: the percentage of patients with seizure reduction of ≥ 50% from baseline seizure frequency.^4,7,9,10,11

Experiment 1: Multiple Baseline Simulations

In one simulation set, 6-week baseline and 6-week “test” periods were collected. There was a 1-month delay between baseline and test periods to simulate “ramping up” therapy. The relative change from baseline to “test” seizure frequency was computed. Thus, for example, if start time was 6 months from time 0, then a baseline of 6 weeks was collected (months 6–7.5), a 1-month delay was used, and then a 6-week test period (month 8.5–10) was used. The full simulation was repeated at start times 0,3,6…24 months after time zero to systematically evaluate start time effect on outcome.

Using the criteria described (see Placebo Trial Simulations), eligible patients were selected for each block. Clinical characteristics of patients from the first block are listed in Table 1.

TABLE 1. Clinical characteristics.

The baseline characteristics of the patients selected for each of the three simulations. The three simulations differed somewhat due to inclusion and exclusion criteria.

	EXPERIMENT 1	EXPERIMENT 2	EXPERIMENT 3
N	1266	1767	615
Age range (median)	0.9 – 84, (17)	0.7–84, (17)	12–84 (24)
Male / Female	51% / 49 %	46% / 54%	45% / 55%
Baseline seizure frequency:	>daily: 26% daily-weekly: 74% weekly: 0%	>daily: 27% daily-weekly: 73% weekly: 0%	>daily: 18% daily-weekly: 71% weekly:11%
Focal epilespy*	22%	22%	23%
Non-focal epilepsy**	9%	9%	8%

^*Focal epilepsy were defined as patients who checked any of the list A items AND did not check any of the list B items in the patient profiles.

^**Non-focal epilepsy were defined as patients who did not check any of the list A items AND did check any of the list B items. List A: brain tumors, brain trauma, brain hematoma, stroke, brain surgery, brain malformations, Tuberous Sclerosis. List B: Alzheimer’s, metabolic disorder, genetic abnormalities, electrolyte abnormalities, alcohol or drug abuse, Dravet, Angelman Syndrome, Neurofibromatosis, Down’s, Aicardi, Sturge-Weber, Rett’s, hypothalamic hamartoma.

Figure 1a summarizes results from each multi-baseline experiment, showing changes in proportion of 50%-responder and seizure-free groups with changing start-time. Thus, points on Figure 1a at time 0 represent the results of simulation performed with start time 0, points at time 3 months present start time 3 months, etc. The largest changes occur within the first 3–6 months; improvement then approximates steady-state. Figure 1b shows differences between 50%-responder and seizure-free groups, representing patients experiencing significant improvement but continuing seizures. This “intermediate” group is important because patients continued tracking seizures; the apparently seizure-free group may include patients who stopped recording events.

FIGURE 1.

Experiment 1: Multi-baseline experiments. Each time on the x-axis represents a single experiment, which started a certain number of months after time 0. Each experiment resulted in a certain percentage of patients experiencing 50% reduction in seizures (solid lines, upper graph), seizure-freedom (broken lines, upper graph). The difference between the 50% reduction and seizure free groups are represented in the lower graph. Children are represented in blue, adults are in red.

To evaluate the age categories “child” vs. “adult”, we used a generalized estimating equation (GEE) to model dichotomous response outcome with age category as a fixed factor and subject as a random factor. First, the GEE with additional interaction between age category and time was conducted independently for each of the three sets: the 50% responders (Figure 1a), seizure-free (Figure 1a) and the “intermediate” (Figure 1b) groups, and there was no significant interaction except in the seizure-free group (p=0.046). Within that group, each time point was tested separately with the chi-squared test, identifying only time=6 months as statistically significant (p=0.041) and the other times not differing significantly (p>0.05). Second, the GEE model without the interaction term was applied to the 50% responders and the “intermediate group”. The children’s response was consistently different than adults in both groups (p=0.0027, and p=0.0028, respectively). Thus, the apparent placebo response seen in children was nearly always higher than adults regardless of start time or improvement category (Figure 1).

Experiment 2: Single Baseline Simulation

In a second simulation set, a 6-week baseline was captured starting at time zero. Additional 6-week periods were collected at 3-month intervals, for 24 months, allowing evaluation of relative change in seizure frequency over time (Figure 2). The result of each interval is plotted based on the beginning of that interval. For example, the change from baseline (month 0–1.5) to the first interval (month 3–4.5) is plotted at time 3 months.

FIGURE 2.

Experiment 2: Single baseline experiment. A single baseline was obtained at time zero, and multiple test times (each of 3 month duration) were measured at regularly spaced intervals in time. The resulting 50% responders and seizure-free patients are plotted for each time point, separating out children (blue) from adults (red). The lower tracing shows the difference between the 50% responder rate and the seizure-free rate for each time point.

Clinical characteristics of the full set of patients are listed in Table 1. Each time-period required at least one seizure to occur after the conclusion of the observed period for patients to be included for analysis. Due to different study designs, Experiment 1 and 2 differed in number of eligible patients despite having common inclusion criteria (see Placebo Trial Simulations).

At every time point the groups divided roughly evenly into the four change categories: 0% of baseline (seizure-free), >0 – 50% of baseline (50% responders), >50–100% of baseline (minimal response), and >100% of baseline (worsening) (Figure 3a), although patients may switch from one category to another (Figure 3b).

FIGURE 3.

Experiment 2, Category Switching. A: Response breakdown. A different representation of Figure 2 from the single baseline experiment, showing the percentage of patients who had specific levels of change from their baseline seizure rates. The graphs show 3 times after the baseline (3,12 and 24 months) for children and adults. Roughly a quarter of each graph represents each of the four percentage change categories throughout time. B: Tracking individuals. This plot tracks which category individuals will fall into as a function of time. Initially the patients separate out neatly into 4 categories of baseline change (0%, 0–50%, 50–100%, and >100% change from baseline). However, the figure depicts patients changing categories multiple times, suggesting that a summary figure such as Figure 2, or a single snapshot in time, cannot accurately capture the dynamic process that evolves for individual patients, even if on aggregate patients segregate into these four categories roughly equally.

We investigated the possibility that medication changes in the preceding 3-month block account for individual patient category switching. Approximately 43% of patients had at least one medication change potentially related to switching, however only 27% of all category-switching events (across patients) could be attributed to a medication change.

Similar to the analysis in experiment 1, we used a GEE to show that an interaction between time and age category was non-significant in each of the three groups independently. The GEE without the interaction term showed a significant difference between the two age categories for responder and seizure-free groups (p=0.0026, p=0.0062, respectively) and but not for the intermediate group (p>0.05). Similar to Experiment 1, in most of the measures, a statistically significant difference between children and adults can be seen in Figure 2.

Experiment 3: Simulated Drug Trial

A clinical trial was simulated involving both “drug” and “placebo” arms. The inclusion criteria described above were altered somewhat for this experiment to more closely match that of a recent clinical trial, with the understanding that this would further decrease the number of eligible patients for this simulation.¹¹ Specifically, the “baseline” period was 6 weeks, the “ramp” period was 6 weeks, and the “test” period was 19 weeks. Furthermore, the minimum for enrollment was adjusted to 12 years of age. Using the aforementioned inclusion criteria, patients were randomly assigned to a treatment arm. The placebo arm patients maintained unchanged seizure logs. The simulation assumed that typical seizure drugs act independently at the time of each seizure with a certain probability of successfully blocking individual seizures. The drug arm simulated a “drug” with 15% efficacy: each seizure had a 15% chance of being removed from existing diary entries. The trial was run twice, once at time zero, and again 6 months later. Treatment responses were evaluated for statistical significance using Fisher’s exact test.

Additional analysis was performed to eliminate patients without stable baselines. Using baseline characteristics to identify such patients, both experiments were reanalyzed in subset fashion after removing patients with unstable baselines. Seizure counts in the sequential baseline months were denoted M1, M2, and M3. To choose the subset, patients were excluded if baseline seizure counts satisfied the following equation:

$$\mathrm{M}3<0.75\ast (\mathrm{M}1+\mathrm{M}2)/2$$ (eq. 1)

Equation 1 implies the reduced patient subset did not show large frequency decreases during the third baseline month compared with the first two.

The simulation at time 0 included N=615 patients; 43% of patients responded (i.e. had at least 50% improvement) to “placebo” versus 49% to “drug” (p=0.12). When patients with baseline improvement were eliminated (i.e. the population was reduced using Eq. 1), N=447; 33% responded to “placebo” versus 48% to “drug” (p=0.0028). At time 6 months, with N=325, 21% responded to “placebo” versus 33% to “drug” (p=0.025). When patients with baseline improvements were eliminated from that time 6 month group, N=225, 20% responded to “placebo,” 30% to “drug” (p=0.083). Either intervention (reduced patient set, waiting 6 months, but not both) resulted in improved statistical significance between “placebo” and “drug” due to decreased apparent placebo effects (Figure 4A). Figure 4B shows the full distribution of patients in the “placebo” group for each of the 4 situations. In that figure, the point where each curve crosses the vertical line representing -100% change from baseline ranges from 1.2–7% depending on the curve. In other words, the proportion of “seizure-free” patients decreased from 7% down to 1.2% when the various changes were made – either changed start time or reduced patient subsets.

FIGURE 4.

Experiment 3: Modifying placebo improvement. A) Bar graph comparing “placebo” to “drug” when compared at time 0 and time 6 months, with the full set of patients and with the reduced subset based on Equation 1. Note that either modification (but not both together) results in statistically significant (star) differences between the two groups. B) Breakdown of “placebo” groups from A. Shown are curves that represent the percentage of the total population that have no greater than a specified amount of improvement in seizure frequency compared with baseline. The lower a curve is, the better, as lower percentage improvement with placebo alone is preferred. Thus, the time 6 months curve (red dashed) and the reduced time 0 (black dashed) appear to reduce the apparent placebo effects the most throughout the largest set of patients.

DISCUSSION

We explored dynamics of apparent placebo effects in clinical epilepsy trials, showing reduction of effects is possible. The simulations reflect epilepsy patients’ natural history, showing at least intermittent transient seizure frequency improvement for the overwhelming majority. Although prior studies attribute placebo arm improvement to various factors,^4,17 we found evidence of persistent ongoing improvement, even in the absence of a true placebo (Figures 1 and 2). This may reflect underlying disease processes, inadequate outcome measures, or both. We also found a consistent pattern of greater placebo improvement in children compared with adults over multiple experiments and conditions, an effect seen also in controlled clinical trial settings.⁴ In addition, we found evidence that apparent improvement is not a stable condition over time, rather patients switch categories (Figure 3). Taking advantage of our understanding of these effects, we simulated two example methods for reducing the apparent placebo effect in a clinical trial (Figure 4).

Relationship of Simulation to Clinical Trials

These simulations were based on parameters from a recent clinical trial, in an attempt to clarify the role that non-psychological effects might play in what appeared to be a placebo effect.¹¹ In that trial they report that patients on placebo had a 50%-responder rate of 26.4% (95% confidence interval - 18.6–34.4%) and a 0% seizure-free rate.¹¹ In simulation 3 the parameters were set to more closely reflect the clinical trial; the “placebo” 50%-responders represented 20–42% of the population, with seizure freedom on “placebo” at 1–7%. A meta-analysis of placebo effects in pediatric and adult clinical trials in epilepsy found the placebo treated 50%-responder to range for children 19 +/− 2.3%, and for adults 9.9 +/− 4.6%.⁴ The simulation data and the clinical trial data differ in an important way – clinical trial data is obtained via communication with the trial investigators by motivated volunteers, thus increasing the likelihood for more accurate reporting. This difference may account for the higher “placebo” response in the simulation. Nevertheless, the simulations and true clinical trials have compatible results, suggesting that the natural fluctuations in seizure frequency may predominate the source of “placebo effect” seen in true clinical trials.

Potential Future Directions For Clinical Trials

The increasingly effective placebo arm of clinical trials noted in a recent meta-analysis⁵ is a mystery that likely reflects multiple underlying issues, including the natural history of epilepsy. Future clinical trials may consider adding a “no-intervention” group which could monitor the natural fluctuations seen in epilepsy, and may help disambiguate parts of the problem. Indeed, if such a control arm showed improvement comparable to the placebo arm, that would represent external validation of the findings in this study.

Reducing placebo effects in clinical trials using strategies similar to Experiment 3 could prove useful. It would enhance efficacy demonstration for modest therapies, and could allow re-interpretation of previously completed studies. Previously judged “effective” therapies might, on re-evaluation, prove ineffective compared with placebo, particularly for a marginally effective drug when patients with unstable baselines are removed, leaving behind fewer to create an apparent drug effect. Although smaller placebo effects would increase study effect size, lower patient number would reduce power, potentially outweighing smaller placebo effects. Another strategy explored was waiting for diminished regression-to-the-mean effects (Experiment 3), i.e. recruiting but not measuring patient’s baseline for a delay of 3–6 months. This may prove difficult to achieve, with more patient dropout than in shorter duration trials. However, it would be expected to reduce the apparent placebo effect to the same extent as removal of patients with unstable baselines.

Of note, the simulation in Experiment 3 made use of a simplifying assumption that anti-seizure medication would act on seizures independently with equal probability. Despite conceivable differences between seizure cluster effects and isolated seizure effects, no data exist to support a more advanced model at this time. Future simulations may be enhanced as our understanding of the effects of medications on individual seizures is clarified.

Other strategies for reducing the influence of apparent placebo responders have been proposed, such as the Sequential Parallel Comparison Design, though this method has yet to be validated in epilepsy.⁸ The advantage that our proposed strategies have over more complex study designs is the relative simplicity. One could retrospectively apply Equation 1 to existing clinical trials. Furthermore, one could prospectively recruit patients and delay baseline measurements without compromising the well-established norms for clinical trials, while reaping the expected benefits of decreased apparent placebo effects.

Improvement category switching

Individual’s improvement categories were unstable (Figure 3). Patients and caregivers might misinterpret category switching during standard care as due to AED effectiveness or ineffectiveness. Clinical trials might be particularly susceptible to this effect due to short study duration. Moreover, the modern definition of drug-resistant epilepsy may not capture fully the fluctuating nature of response to any therapy.¹⁸ More studies are needed to clarify the extent of stable therapeutic influence on seizure frequency.

Regression to the mean

Some studies attribute placebo improvement to regression to the mean.^5,7,8 Here this effect may have dominated only in the first 3–6 months; the mere act of recording seizures itself is subject to regression to the mean. Our data suggest little role beyond 3–6 months. Future clinical trials may take advantage of this observation, enrolling patients as usual, but refraining from collecting a baseline for 3–6 months. Such a novel strategy would face obstacles due to patient dropout but might increase dramatically chances of an efficacious therapy achieving statistically significant effect size.

Ongoing improvement

Placebo effects are variously attributed to regression to the mean, natural remission, Pavlovian conditioning, and psychological effects.¹³ Our data imply that regardless of when measured, 20–25% of patients demonstrate at least 50% improvement over baseline (not including seizure-freedom), and that this improvement is present with a recent or remote baseline. Because seizure-freedom is excluded, this is unlikely due to patients abandoning seizure tracking. Similarly, because baselines were evaluated from 0 to 2 years before a test period, and baseline improvement was still observed, regression to the mean effects are unlikely to play a persistent role.¹⁹

The advantage of these simulations lies in a lack of clinical distractors, such as a placebo pill, a white-coated physician, or other psychologically active cues. Absent these cues, patients may be improving due to a different mechanism than the traditional “placebo effect,” or Pavlovian conditioning.

Natural remission from epilepsy remains a possibility when other options are excluded.^19,20 One study in a medically refractory cohort followed prospectively for six years found 24% of patients had a 12-month seizure-free period, which was independent of surgery or medication, though many relapsed after 1 year.¹⁹ The same population had 15% of patients demonstrate at least 6 months of remission during the first 3 years.²¹ In a 15-year study of pediatric epilepsy, 70% reached a terminal seizure-free period of at least 5 years.²² Some studies suggest spontaneous remission may occur in up to 30%.²⁰ Our data support these findings, as we find marked seizure improvement as well as seizure-freedom, and at times even larger gains in children. We elected to exclude patients from our simulations if no seizures were recorded after the trial period, in an attempt to control for patient drop-out due to lack of recording. That exclusion biased our data away from higher seizure-freedom rates. Seizure-free periods can be seen even in patients who eventually become drug-resistant;²³ including them in future studies would be desirable if appropriate bias controls were implemented.¹⁶

Data bias

Seizuretracker.com data include several biases. First, medical professionals do not curate the dataset. Seizure identification is by self-report and may be inaccurate in some cases (e.g. patients with non-epileptic events). Like many clinical trials, this study may suffer from diagnostic misclassification.⁸ Second, there is currently no mechanism for designating a period of time as “seizure-free” in an individual’s log, other than absence of recorded seizures. Patients abandoning the tool will appear equivalent to those becoming seizure-free. Our simulations avoided patients who did not track at least one seizure after the end of the testing period to control partially for this bias. Our analysis biases towards medically refractory patients.

Seizuretracker.com requires basic technological skills and persistence. However, it takes advantage of the prevalence of internet connectivity, and widespread adoption of portable smartphones and tablets. Approximately 87% of US adults use the internet, and 58% own smartphones.²⁴ Interestingly, smartphone ownership is greater in younger populations, 18–29: 83% vs 65+: 19%. The ownership of tablet devices is also increasing; in 2014, 42% of US adults owned one, up from 3% in 2010.²⁴

Video-EEG studies show lack of awareness of seizures, and under-reporting occurs in 23–74% of events.^25–28 Moreover, self-reported events may be subject to recall and response biases.²⁹

Another potential bias in these data comes from medication changes. Seizure frequency ’should’ respond to AED changes. Consequently, the category switching evaluation (Figure 3) may reflect changes in regimen rather than disease natural history. Our analysis (in Experiment 2) of 29,800 medication change entries in the database suggested otherwise – specifically, that medication changes were insufficient to account for the overwhelming majority of seizure frequency category changes. Although additional unrecorded AED changes may have occurred, the number of documented medication adjustments allows a reasonable degree of confidence that this is not likely. Follow-up studies in this area might control more precisely for medication changes, or track them with investigator oversight.

Conclusion

Although our simulation results parallel clinical trials that were able to control for confounders, prospective studies are needed to confirm our findings. Reduction of apparent placebo effects could validate therapies hitherto discarded. The approaches used in this study may have implications for other episodic diseases, such as primary headache disorders, syncope, asthma, congestive heart failure, autoimmune conditions, or blood glucose excursions in diabetes. With the rapid growth of mobile and distributed computing tools, new possibilities are emerging to investigate the natural course of disease in ways not possible before.