P Value Problems

To the Editor: Since its invention 90 years ago, the p value has become the standard by which most quantitative research is judged; however, it was never intended for this purpose.¹ Indeed, a 2016 joint statement by the American Statistical Association argued, “By itself, a p value does not provide a good measure of evidence regarding a model or hypothesis.”² A lone p value is uninformative because it is prone to false positives and says nothing about the magnitude or range of an effect.³ Additionally, over-reliance on p values may even encourage unethical research practices.⁴

Before discussing its shortcomings, it is helpful to define a p value. A p value is the likelihood of obtaining one’s data if the null hypothesis is true. Contrary to popular misconception, it is not the probability that one’s results were obtained by chance, the probability that the null hypothesis is true, or the probability of a false positive result.⁵ In fact, the false positive rate associated with a p value of .05 is usually around 30%, but can be much higher.⁶ This discussion may seem pedantic, but accurate false positive rates are a practical matter. A recent study that attempted to replicate 100 psychology experiments was only able to replicate 38.⁷ Amgen was only able to replicate six of its 53 landmark cancer studies.⁸ Bayer could only replicate 25% of the 67 studies that it attempted.⁹ Is there any reason to believe that pharmacy education would fare better?

Even when a p value is interpreted correctly, it is silent on the magnitude and range of an effect. Even the most miniscule effect can be statistically significant if the sample size is large enough.¹⁰ For example, a study of the effect of aspirin on myocardial infarction (MI) collected 22,000 subjects over a 5-year period and found that aspirin reduced the risk of MI, p<.00001. The risk difference for this study, however, was 0.77% and the R² was 0.001. This means that only one-tenth of 1% of the risk of suffering MI could be explained by aspirin.¹¹ Focusing exclusively on the p value when the sample size is large can overstate the practical importance of one’s conclusions. In addition to a p value, researchers should report effect sizes and R² so that readers can properly interpret the magnitude of their findings. Additionally, the p value does not specify the range of probable outcomes; hence, researchers should also report confidence intervals.

Not all of the p values’ shortcomings are mathematical; some are ethical. The publication of accurate and honest results is a moral concern, and overreliance on p values can obstruct this goal.⁴ Making a p value of .05 the sole arbiter of whether or not a manuscript is published can encourage researchers to hunt for small p values using ethically questionable research practices such as: attempting a study multiple times but only reporting the study that produced a significant result; hedging their bets by collecting many variables but only reporting the ones that showed significant effects; dropping outliers or changing screening criteria after analysis has begun; splitting, merging, or transforming variables to produce a significant result; conducting significance tests before data collection is complete and terminating collection if a significant result is obtained. All of these practices are ethically dubious, and can harm the replicability of one’s results.⁴

Given the p value’s limitations, it should not be the sole arbiter of publication, and researchers should always report additional information, especially means, standard deviations, confidence intervals, R², and effect sizes. These additional statistics do not correct p values’ shortcomings – some of which were not mentioned here, but they make results sections more informative and help to hold researchers accountable.