The P Value Problem in Otolaryngology: Shifting to Effect Sizes and Confidence Intervals

Abstract

There is a lack of reporting of effect sizes and confidence intervals in the current biomedical literature. The objective of this paper is to present a discussion of the recent paradigm shift encouraging the use of reporting effect sizes and confidence intervals. Although p-values help to inform us about whether or not an effect exists due to chance, effect sizes inform us about the magnitude of the effect (clinical significance), and confidence intervals inform us about the range of plausible estimates for the general population mean (precision). Reporting of effect sizes and confidence intervals are a necessary addition to the biomedical literature, and these concepts are reviewed in this manuscript.

INTRODUCTION

The use of p-values as a surrogate for clinical importance in the biomedical literature is an ongoing problem.^1,2 Some have gone so far as to call for a complete overhaul of the current paradigm of formulating a null hypothesis in order to accept or reject it.³ A p-value can show whether an effect exists that is not likely due to chance, but it does not reveal the magnitude of the effect. When faced with the unfortunate reality that a majority of published scientific results cannot be replicated,⁴ it becomes clear that there is a problem with current reporting methodology.

If one is to accurately report scientific findings, it is important to not confuse statistical significance with a meaningful or clinically significant difference. With a very large sample size, a clinically meaningless difference may be reported as a “statistically significant difference” via sole reliance on a p-value. Furthermore, calculating a p-value without a prior power calculation to determine the appropriate sample size may yield spurious conclusions, as the difference may eventually be statistically significant if the experiment is repeated enough times.⁵ Effect sizes and confidence intervals allow one to describe the reliability, magnitude, direction, and precision of results. Simply put, the results of the research (e.g. difference between groups, baseline risk) are the effect. The magnitude of that effect is the effect size. Confidence intervals make use of the inherent variability in study results to give a measure of how precise the data are. In other words, there is a 95% chance that the mean of the population will lie within the confidence interval from the study sample, making generalization plausible.⁶

Integrating effect size and confidence intervals into routine practice has proven challenging.^1,2 Perhaps due to habit, training, or the efficiency of a quick glance at a single number (p-value), researchers have been slow to abandon reporting p-values. Our objective is to convey to the reader the disadvantages of relying solely on the p-value, and to encourage instead the use of confidence intervals and effect size when reporting results.

PROBLEMS WITH NULL HYPOTHESIS SIGNIFICANCE TESTING

The gold standard for reporting results of scientific experiments is null hypothesis significance testing (NHST), which estimates whether the results of the study are reliably due to the exposure/intervention or due to chance. Unfortunately, this method is outdated, overused, and is counterproductive to modern clinical research. NHST gives a binary yes/no answer as to whether the results achieve a threshold defined a priori, such as p < 0.05. Under the NHST paradigm, if the p-value is less than the threshold (e.g. p = 0.013), convention suggests that one can be reasonably sure the results are reliable and not due to chance. With a large enough sample size, however, a clinically meaningless difference may be reported as a “statistically significant difference.” Furthermore, in many studies, no null hypothesis is explicitly stated, and when a p-value < 0.05 is obtained, the finding is stated without mention of a power calculation, description of the size of the difference (i.e. effect size), or consideration of the clinical significance of the finding.

INTERPRETING EFFECT SIZE

Examples of effect sizes are the mean, median, mean difference between two groups, odds ratios (OR), and risk ratios (RR). These are unstandardized effect sizes. Standardized effect sizes, such as Cohen’s d, Glass’s delta (Δ), Hedge’s g, are unit-less measures that allow results from different studies to be compared to each other, as is commonly done in meta-analyses (Table 1).^3,7 Because these measures are perhaps harder for the lay person to interpret due to the lack of units, some use the rather arbitrary categories of small (0.20), medium (0.50), and large (0.80) effect sizes as originally described by Cohen.^7,8 While somewhat useful for interpretation, there is controversy about their use. Considering each situation individually is more important than these arbitrary categories, and the appropriate interpretation depends on the context of the comparison.⁷

TABLE 1.

Continuous scale variable standardized effect size

ITEM	FORMULA	STANDARD DEVIATION FORMULA	ASSUMPTION
Cohen’s d	M1 – M2 / SDpooled	SDpooled = √[(SD12 + SD22) /2] New standard deviation by taking square root of the average of the two variances	Similar SDs and sample sizes in both groups
Glass’s delta (Δ)	M1 – M2 / SDcontrol	Use SD of control group	Different SDs but same sample size in each group
Hedge’s g =	M1 – M2 / SDpooled*	Weighted SDpooled*=√ [(nA - 1)SDA2 + (nB – 1)SDB2]/(nA + nB – 2)	Different sample sizes and SDs in each group

Key: SD=standard deviation; subscripts _A,B,1,2 designate groups; n = number in group; M is mean

Reference: Ellis 2010⁷

IMPORTANCE OF EFFECT-SIZE FOCUSED RESEARCH DESIGN

Designing a study based on the alternate hypothesis, rather than the null hypothesis, and using effect size estimates that will be used in reporting the results is ideal. Otherwise, interpretation of a study based on statistical significance using p-values without any consideration of a power calculation is a flawed approach.^3,7,9 It is always wise to consult with an experienced statistician as the study is being designed. If the study is preliminary and without previously published estimates of the expected effect, researchers should use an a priori estimate based on pilot data. This pilot data is then used to determine the minimal clinically important difference (MCID).¹⁰ The study should be sufficiently powered to detect at least this difference. Confidence intervals are used to determine whether the result is due to chance alone or is a true difference, based on whether the interval includes zero (in the case of an absolute difference between groups) or one (in the case of a ratio between groups). If a study is performed without a power calculation (and without using confidence intervals), having a p-value less than 0.05 means nothing, as it is possible the study was over-powered and will yield a p-value less than 0.05 for a small, meaningless difference; alternatively, the study may be under-powered, and having a p-value less than 0.05 may be due entirely to chance, and if the study were repeated, would be found to be greater than 0.05.

CONCLUSION

The aim of a biomedical research study is often to determine the size of an intervention or exposure. How the authors choose to summarize the result is the effect. The magnitude of this effect is the effect size, and accompanying this should be confidence intervals attesting to the reliability and precision of the results. Effect size estimates should be used in a priori sample size calculations, as well as in the interpretation of the study to justify clinically meaningful results. The use of effect sizes and confidence intervals will better allow the reader to reach an objective conclusion based on these numbers.