ABSTRACT
We estimated the statistical power of studies predicting seizure freedom after epilepsy surgery. We extracted data from a Cochrane meta‐analysis. The median power across all studies was 14%. Studies with a median sample size or less (n ≤ 56) and a statistically significant result exaggerated the true effect size by a factor of 5.4, while the Bayesian estimate of the odds ratio only exaggerated the true effect size by a factor of 1.6. We conclude that Bayesian estimation of odds ratios attenuates the exaggeration of significant effect sizes in underpowered studies. This result could improve interpretation of studies with small sample sizes.
Keywords: Bayesian, Engel outcome, epilepsy surgery, seizure freedom, statistical power
1. Introduction
Low statistical power is a recognized problem in many fields [1], which means true effects will be missed, the false discovery rate will be inflated [2], and discovered true effects will be over‐estimated [3, 4]. We highlighted the issue of low power in a single study [5], but suspected that low power was pervasive in the field of epilepsy surgery. Here, we estimate the median statistical power of studies of epilepsy surgery outcomes. We argue that Bayesian estimation can attenuate the exaggeration of discovered effect sizes.
In neurosciences, the median power to detect a true effect was estimated at 20% [6]. That study defined “ground truth” as the pooled effect size of a meta‐analysis and then calculated the statistical power of each included study to find that pooled effect. We apply that same approach to the Cochrane Library Review of Surgery for Epilepsy, with data from 182 studies with 16,855 participants [7].
A Bayesian approach prevents overfitting in small sample sizes by constraining estimates with prior knowledge. A prior distribution is combined with the likelihood of the observed data to create a posterior distribution. Here, we use Bayesian logistic regression, where the slope is the log of the odds ratio for a prognostic factor.
The Bayesian approach encapsulates prior knowledge as a prior distribution. We derived this from the separate European Epilepsy Brain Bank study, which reported seizure‐free outcomes for 8247 participants stratified by histopathological diagnosis [8]. We performed pairwise comparisons of these diagnoses to produce a prior distribution of plausible odds ratios, which was then used in Bayesian logistic regression on data from the Cochrane review. Finally, we computed the error between the “ground truth” odds ratio and the observed odds ratio, compared to the error between ground truth and the Bayesian estimate of the odds ratio.
2. Materials and Methods
2.1. Cochrane Data
We extracted publication level data from “Comparison 4. Prognostic factors of a good outcome of surgery” from the Cochrane review [7], including the number of participants and the number with a good outcome (seizure free or Engel I) for both the experimental and control groups. We recalculated the meta‐analysis to create a pooled odds ratio, rather than the reported risk ratio, using the package meta in statistical software R (version 4.2.1). The fixed effect estimate for each comparison was used as ground truth for all included studies. We excluded three analyses that duplicated studies from a previous analysis (4.13, 4.15, and 4.17).
2.2. Power Analysis
Our “ground truth” effect size was derived for each prognostic factor by taking the percentage of participants with a good outcome in the control group and applying the pooled odds ratio to derive the percentage with a good outcome in the experimental group. We computed the statistical power (using the R package exact) to detect that ground truth effect size for each study for a given prognostic factor, assuming a Chi‐squared test. We reported the median power for all studies whose pooled effect was statistically significant in the original Cochrane study.
2.3. Bayesian Estimation and Prior Distribution
For Bayesian estimation, we tested the utility of a neutral but informative Gaussian prior for the log odds ratio, derived from prior literature. The prior distribution for plausible log odds ratios was derived from the reported 1‐year seizure freedom by histopathological diagnosis in the European Epilepsy Brain Bank study [8]. We performed all possible pairwise comparisons of 17 non‐overlapping categories that had at least 100 participants. We assumed a zero mean and computed the standard deviation of the log odds ratios of the pairwise comparisons to define a Gaussian prior distribution for the slope. The prior for the intercept was weakly informative (Gaussian with a zero mean, standard deviation 1000). We used these prior distributions to perform Bayesian logistic regression using the probabilistic language Stan [9] (R package rstanarm). We computed the mean squared error of the observed log odds ratio from the “ground truth” log odds ratio and compared this to the error of the Bayesian log odds ratio from the “ground truth” log odds ratio using a paired t‐test.
As a sensitivity analysis, we also tested a diffuse, weakly informative neutral Gaussian prior for the log‐odds ratio, with a zero mean and a standard deviation of 3. This allowed us to verify if deriving the informative prior from the literature showed benefit relative to a non‐specific prior.
2.4. Exaggeration of Statistically Significant Results
We examined the relationship of the sample size to the magnitude of the observed odds ratio in studies that were statistically significant (Chi‐squared test with p < 0.05) and in the same direction as the “ground truth” odds ratio. We added 0.5 to all cells if there was a zero‐cell value to prevent infinite observed odds ratios [10]. We compared this to the relationship of the sample size to the magnitude of the Bayesian odds ratio in studies where the posterior probability of an odds ratio > 1 was more than 95%, and in the same direction as “ground truth.” We estimated the exaggeration ratio between the observed versus the Bayesian odds ratio and the median true odds ratio for studies with sample size less than or equal to the median.
3. Results
We extracted data from 367 binary comparisons of good outcome after epilepsy surgery according to one of 14 prognostic factors. We flipped all odds ratios to be positive, so the experimental group was associated with a better chance of good outcome. The “ground truth” odds ratios ranged from 1.05 (positive history of head injury) to 2.54 (complete resection). The median “ground truth” odds ratio across all studies was 1.72 (presence of mesial temporal sclerosis). The median sample size was 56, with an inter‐quartile range (IQR) of 42–100.
We computed the statistical power of each comparison to detect the pooled effect size for the prognostic factors whose meta‐analysis showed a significant result. The median power across all studies was 14% (IQR 10% – 23%). We expected that underpowered studies with statistically significant results would exaggerate the observed effect size [4]. We found that a study with a median sample size or less (n ≤ 56) and a statistically significant result exaggerated the true effect size by a factor of 5.4 (median odds ratio 9.3 vs. median true odds ratio 1.7).
For the alternative Bayesian approach, we derived an informative prior distribution of odds ratios from the European Brain Bank study. The prior distribution for plausible slopes (i.e., log odds ratios) was a normal distribution with a mean of zero and a standard deviation (STD) of 0.70 (Figure 1A). This implies that 68% of odds ratios are between ½ and 2, and 95% are between ¼ and 4. We found that this qualitatively fit the observed log odds ratios for large (sample size ≥ 100) studies from the Cochrane review (Figure 1B).
FIGURE 1.
Prior distribution of log odds ratios. (A) The prior distribution for the log odds ratio, derived from the European Epilepsy Brain Bank study [8], was normally distributed with mean zero and standard deviation of 0.7 (red). This implies that 95% of the odds ratios are between ¼ and 4. (B) This prior distribution (red) qualitatively matches the observed log odds ratios from large studies (sample size > = 100) from the Cochrane review [7]. All odds ratios are plotted twice to include the reciprocal of the observed odds ratios. There is one outlying study, and a paucity of null studies with log odds ratio of 0 (i.e., odds ratio of 1) The lack of null studies is likely explained by publication bias (researchers are less likely to write up null findings) [11].
We also assumed a weakly‐informative neutral prior distribution for the intercept (see Methods). Given those prior distributions for slope and intercept, we derived the Bayesian estimate of the log odds ratio using a probabilistic statistics package [9]. The mean squared error (MSE) of the observed log odds ratio from the “ground truth” odds ratio (1.0) was significantly higher (p < 0.001, paired t‐test) than the MSE of the Bayesian log odds ratio to “ground truth” (0.27). We compared the observed and Bayesian odds ratios as a function of sample size (Figure 2). While the observed odds ratio of small studies (n ≤ 56) with significant effects exaggerated the true effect size by a factor of 5.4, the Bayesian estimate of the odds ratio with an informative neutral prior only exaggerated the true effect size by a factor of 1.6 (median Bayesian odds ratio 2.7 vs. median true odds ratio 1.7).
FIGURE 2.
Observed versus Bayesian odds ratios. (A) Observed odds ratios are shown as a function of sample size on a semi‐log plot. The median true odds ratio was 1.7, and ranged from 1 to 2.5 (black dotted lines). Observed odds ratios from studies that are statistically significant (Chi‐squared p < 0.05) and in the same direction of the true effect (blue dots) exaggerate the true odds ratio at low sample sizes, as shown by a quadratic trend line (blue line) (B) Bayesian odds ratios are shown as a function of sample size on a semi‐log plot. This analysis used an informative Gaussian prior distribution derived from the literature (see Figure 1). The Bayesian odds ratio estimates from significant studies (greater than 95% posterior probability of odds ratio above 1) that are also in the same direction of the true odds ratio (blue dots) show less exaggeration.
As a sensitivity analysis, we also performed a Bayesian analysis using a diffuse, weakly informative Gaussian prior distribution for the log odds ratio, with mean zero and STD of 3. This diffuse prior was not derived from the literature. It implies that 68% of odds ratios are between 1/20 and 20% and 95% are between 1/400 and 400. The MSE using the diffuse prior (1.17) was significantly higher (p < 0.001) than the MSE of the observed odds ratios (1.0), which were corrected by adding 0.5 to all cells if a zero cell was present. The Bayesian estimate of the odds ratio with a diffuse prior still exaggerated the true effect size by a factor of 4.8 (median diffuse Bayesian odds ratio 8.3 vs. median true odds ratio 1.7).
4. Discussion
In this study, we found that the median power of a typical study of epilepsy surgery was only 14%. We showed that studies with small sample sizes and statistically significant results exaggerated the magnitude of the observed effect. We showed that Bayesian estimation of the odds ratio using an informative prior derived from the literature can attenuate this exaggeration of significant effect sizes in underpowered studies.
We believe Bayesian estimation is relevant for clinicians counseling patients about epilepsy surgery. A recent paper examined whether low voltage fast activity (LVFA) intracranial ictal onset predicted seizure freedom after minimally invasive surgery: they reported a 46% (20/43) chance of seizure freedom with LVFA, but a 0% (0/15) chance without LVFA [12]. That observed odds ratio is infinite. The corrected odds ratio, after adding 0.5 to each cell, is (20.5/23.5) / (0.5/15.5) = 27, which is still very large. In contrast, the Bayesian estimate of the odds ratio is around 3.5 when using the informative prior. Thus, it would be incorrect to exclude a patient without LVFA from minimally invasive surgery based on only one study. Surgery might still be a reasonable option, based on the remaining clinical data and the patient's preferences.
One limitation of Bayesian logistic regression is that it does not have an analytical solution, which makes it less accessible. Rather, state‐of‐the‐art statistical packages are probabilistic [9]. A limitation of our specific approach is that our prior distribution was narrowly defined for predicting seizure freedom after epilepsy surgery. Prediction of another outcome (such as risk of cognitive decline) would require a separate derivation of prior distributions.
There are alternatives for shrinking observed odds ratios, including ridge regression, LASSO, and elastic net [13]. However, the choice of method and degree of shrinkage is itself a hyper‐parameter that must be optimized. The advantage of our Bayesian approach is that the prior distributions were derived directly from existing literature. Notably, using a diffuse prior (not derived from the literature) did not improve performance relative to the observed odds ratio. Whatever the method used, we believe it is critical to shrink observed effect sizes from small studies of surgery outcomes.
We have created a web‐based application (see Data Availability below) to allow a reader to perform this Bayesian shrinkage themselves. Even without making a formal computation, we hope this manuscript will help clinicians interpret small studies with the appropriate amount of skepticism. Researchers should acknowledge the risk of false positives or exaggerated results when publishing small studies. Moreover, we should all strive to publish studies with larger sample sizes through multi‐center collaboration, as this will increase the robustness and generalizability of results from future studies of epilepsy surgery outcomes.
Author Contributions
A.S.D. prepared the figures and performed the statistical analysis, under the guidance of R.T.K. A.S.D. and V.R. extracted raw data from prior publications. A.S.D., V.R., and A.A.R. wrote code in R. A.S.D., V.R., A.A.R., N.P.P., and R.T.K. contributed to the drafting of the text and design of the study.
Funding
This work was supported by the National Institute of Neurological Disorders and Stroke, K08 NS105929, (R21 NS122011). National Institute of General Medical Sciences, (R01 GM113243). National Center for Advancing Translational Sciences, (KL2 TR002381, UL1 TR002378)
Conflicts of Interest
N.P.P. has served as a paid consultant for DIXI Medical USA, which manufactures products used in the workup for epilepsy surgery. The terms of this arrangement have been reviewed and approved by Emory University in accordance with its conflict‐of‐interest policies. The remaining authors have no conflicts of interest.
Acknowledgements
N.P.P. was supported by the Woodruff Foundation, CURE Epilepsy, and NIH NINDS grants K08 NS105929 and R21 NS122011. A.S.D. was supported by NIH NCATS grants UL1 TR002378 and KL2 TR002381. R.T.K. was supported by NIH NIGMS grant R01 GM113243. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.
Funding Statement
This work was funded by National Institute of Neurological Disorders and Stroke grants K08 NS105929 and R21 NS122011; National Institute of General Medical Sciences grant R01 GM113243; National Center for Advancing Translational Sciences grants KL2 TR002381 and UL1 TR002378; National Institutes of Health .
Data Availability Statement
The study used only openly available human data that were previously published by the Cochrane Library and the European Epilepsy Brain Bank. The extracted raw data and an RMarkdown file code which can be used to reproduce the analyses and figures described here will be posted upon publication at https://github.com/AdamSDickey. We have created a web‐based application where users can run Bayesian logistic regression on their own two by two table of good versus bad outcome, stratified by presence or absence of a prognostic factor. It is available at: https://zz0dsr‐adam‐dickey.shinyapps.io/Bayesian_Odds_Ratio/.
References
- 1. Cohen J., “The Statistical Power of Abnormal‐Social Psychological Research: A Review,” Journal of Abnormal and Social Psychology 65, no. 3 (1962): 145–153. [DOI] [PubMed] [Google Scholar]
- 2. Ioannidis J. P. A., “Why Most Published Research Findings Are False,” PLoS Medicine 2, no. 8 (2005): e124. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3. Ioannidis J. P. A., “Why Most Discovered True Associations Are Inflated,” Epidemiology 19, no. 5 (2008): 640–648. [DOI] [PubMed] [Google Scholar]
- 4. Gelman A. and Carlin J., “Beyond Power Calculations: Assessing Type S (Sign) and Type M (Magnitude) Errors,” Perspectives on Psychological Science 9, no. 6 (2014): 641–651. [DOI] [PubMed] [Google Scholar]
- 5. Dickey A. S. and Pedersen N. P., “Low Statistical Power in a Study Predicting Seizure Outcome,” Epilepsia 62, no. 10 (2021): 2565–2566. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6. Button K. S., Ioannidis J. P., Mokrysz C., et al., “Power Failure: Why Small Sample Size Undermines the Reliability of Neuroscience,” Nature Reviews Neuroscience 14, no. 5 (2013): 365–376. [DOI] [PubMed] [Google Scholar]
- 7. West S., Nevitt S. J., Cotton J., et al., “Surgery for epilepsy,” Cochrane Database of Systematic Reviews 6, no. 6 (2019): CD010541. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8. Lamberink H. J., Otte W. M., Blümcke I., et al., “Seizure Outcome and Use of Antiepileptic Drugs After Epilepsy Surgery According to Histopathological Diagnosis: A Retrospective Multicentre Cohort Study,” Lancet Neurology 19, no. 9 (2020): 748–757. [DOI] [PubMed] [Google Scholar]
- 9. Carpenter B., Gelman A., Hoffman M. D., et al., “ Stan: A Probabilistic Programming Language,” Journal of Statistical Software 76, no. 1 (2017): 76.i01. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10. Agresti A., An Introduction to Categorical Data Analysis (Wiley‐Blackwell, 2007). [Google Scholar]
- 11. Franco A., Malhotra N., and Simonovits G., “Publication Bias in the Social Sciences: Unlocking the File Drawer,” Science 345, no. 6203 (2014): 1502–1505. [DOI] [PubMed] [Google Scholar]
- 12. Michalak A. J., Greenblatt A., Wu S., et al., “Seizure Onset Patterns Predict Outcome After Stereo‐Electroencephalography‐Guided Laser Amygdalohippocampotomy,” Epilepsia (2023): epi.17602. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13. Zou H. and Hastie T., “Regularization and Variable Selection via the Elastic Net,” Journal of the Royal Statistical Society, Series B: Statistical Methodology 67, no. 2 (2005): 301–320. [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Data Availability Statement
The study used only openly available human data that were previously published by the Cochrane Library and the European Epilepsy Brain Bank. The extracted raw data and an RMarkdown file code which can be used to reproduce the analyses and figures described here will be posted upon publication at https://github.com/AdamSDickey. We have created a web‐based application where users can run Bayesian logistic regression on their own two by two table of good versus bad outcome, stratified by presence or absence of a prognostic factor. It is available at: https://zz0dsr‐adam‐dickey.shinyapps.io/Bayesian_Odds_Ratio/.