The Value of the P Value

Abstract

Recently, the discussion on the implications of irreproducibility in the sciences has been brought into the spotlight. This topic has been discussed for years in the literature. A multitude of reasons have been attributed to this issue; one commonly labeled culprit is the overuse of the p value as a determinant of significance by the scientific community. Both scientists and statisticians have questioned the use of null hypothesis testing as the basis of scientific analysis. This survey of the current issues at hand in irreproducibility in research emphasizes potential causes of the issue, impacts that this can have for drug development and efforts been taken to increase transparency of findings in research.

THE VALUE OF P VALUE

Researchers use a variety of statistical analytic methods to better quantify the validity and reproducibility of their experiments. A popular method of determining statistical significance is through the use of the p-value, a measure of how well experimental data rejects the null hypothesis. For years, the debate over the value of p-value has been discussed in the literature [1–4]. Critics argue that p-value creates an artificial metric of establishing significance in a data set. Recently, Leek and Peng published a commentary in Nature that discusses limitations of null hypothesis testing.⁵ Rather than focusing their criticism on p-value, the authors argue that data analysis, as a whole, must be more stringently regulated. The null hypothesis is widely used in both the basic sciences and in clinical/translational research, a field in which experimental findings strongly influence the treatment of patients. Therefore, it is vital to understand how experimental design, data analysis, and statistical testing may be optimized to generate studies that more effectively improve human health.

Typical Research Methodology

In most research studies, there is a classical progression of experimental steps (Fig. 1). First, a problem is identified through analysis of evidence seen in prior experimentation and literature. A hypothesis is proposed to address the problem, and experiments are devised to test the hypothesis.⁶ After the generation of data and appropriate statistical filtering, a p-value test is applied to suggest whether the findings are “significant” or “not significant.” P-value is popularly used as a measure of significance because it represents the probability of an event occurring solely by chance or random error. Therefore, a high p-value indicates that there is a higher probability that the event could be explained by chance. In other words, a higher p-value means there is a higher probability that the null hypothesis should be accepted instead of rejected.⁷ Cut off values are arbitrarily assigned for evaluation of significance of the p-value. By definition, a p-value larger than the cutoff value indicates that the event is more likely to have occurred due to chance.⁸ Unfortunately, many scientists misinterpret the p-value as a simple method to determine significance while ignoring other critical factors of a data set that can drastically impact the p-value such as the magnitude of the association and the sample size. These components must be recognized and evaluated for a data set to ensure that the p-value is an accurate measure of deviation from the null hypothesis. The greatest impact on a data set is not the calculation of p value, but rather the manners in which the raw data is altered through cleaning and statistical modeling.⁹ Typically, data is purged of outliers using various statistical algorithms to generate a “clean” dataset. At this stage, the use of null hypothesis testing may generate a p-value that may falsely represent the significance of the study.^10–12

Figure 1.

A pictorial representation of data processing from collection of data to final statistical analysis.

Implications of Misuse of P Value

An underappreciated but critical aspect of analysis of research findings is the correct interpretation of a calculated p value in the context of the sample set of data. There are many factors that define the reliability of a sample set including potential bias and sample size.¹³ For example, having a smaller sample size is less likely to represent the population as a whole. This makes the p value generated from this data less likely to represent the population and therefore less likely to be replicated if the experiment was repeated. Additionally, p value is limited in that it does not correlate to or provide information on the magnitude of an effect.¹⁴

In basic science research where the use of animal models is common, careful planning of experiments is essential to minimize the number of animals required to generate usable data. However, this factor must be balanced with the risk of drawing incorrect conclusions due to an inadequate sample size. Careful selection of animal models and the proper use of controls are critical to picking the appropriate type of data analysis.¹⁵ Another issue that impacts statistical analysis is the inclusion of replicates. There are two types of experimental replicates: biologic and technical. A technical replicate assays a single sample multiple times and serves as an internal control for the experiment. Whereas a biological replicate is when different samples are assayed.¹⁶ Replicates are not independent tests of the hypothesis, and therefore should not be used to calculate p values. However, they are often incorrectly included in statistical calculations.¹⁷

In the clinical sciences, the results of studies often guide clinicians in treatment and can have a more immediate impact on human health. All clinical studies have some type of bias or effects that can alter conclusions away from the truth.¹⁸ It is also important to look at the statistical power of the study. For example, pharmaceutical companies may include hundreds of patients in a clinical trial; this leads to the ability to detect minor differences that are very statistically significant that may not represent a clinical difference.^{19, 20}

A Journal that Publishes Without p Value

The use of p value has been present in literature for years and is a central tenant of evidence-based scientific methods.²¹ The raging debate over the usage of the p-value in science encouraged the journal Basic and Applied Social Psychology to ban the use of null hypothesis significance testing analysis in it’s future publications.²² This journal aims to publish research that integrates basic science research with practical real world applications. Declaring that null hypothesis testing is invalid, the journal now calls for more descriptive statistics to be used like effect size, a measure of the strength of a phenomenon.²³ This is the first journal to institute a ban of this type. This decision has been met with applause from some and criticism from others in the scientific community who believe that this bold move will have little effect on the quality of scientific literature.

Impacts to Drug Development

A common result of the systematic tailoring of data to achieve a p value that demonstrates “significant” is that many studies are irreproducible.²⁴ Validation of scientific data through reproduction of experiments is fundamental to the scientific method. Pharmaceutical companies constantly scan the literature to identify new molecular targets or compounds that show high potential to further study. These targets are often identified and published by academic labs. Amgen, a biotechnology company based in California, identified fifty-three promising cancer studies over the course of ten years from high profile journals. However, only 6 of these studies, a mere 11%, were both reproducible and demonstrated continued promise as a novel target.²⁵ Other pharmaceutical companies have also reported difficulties with validating promising studies, a problem that drastically hinders the progress of therapeutic development.²⁶ This lack of reproducibility has been attributed to a variety of factors, including the chronic misuse and lack of understanding of what the p value represents.²⁷

Future Directions

To help eliminate some of the issues derived from poor experimental design, students-in-training along with experienced research scientists must learn to better critically scrutinize experimental design and identify flaws in published studies. The National Institutes of Health (NIH) is currently developing a training module for postdoctoral students that emphasize the importance of good experimental design.²⁸ The development of similar initiatives in research centers and universities will be positive steps to improve reproducibility of experiments. Publishers of peer-reviewed research also can play a role in improving the reliability and quality of research by enforcing stricter statistical guidelines. Already, high impact journals such as Nature and Science have made changes by eliminating a word count for the methods section and calling for increased transparency of data handling.^{29, 30} These steps are merely the beginning, but provide hope that the scientific community will begin to recognize the dire need for an improvement in the reliability and reproducibility of research.

The scientific community has urged the push for transparency in research presentation. For example should be a continued push to eliminate the use of phrases such as “representative data” and “data not shown,” but rather show the primary data and create a culture of transparency.³¹ Reporting of other statistical values such as confidence interval can help provide a more complete understanding of the statistics by going beyond the scope of a p-value to provide the magnitude and accuracy of an effect.³²

The p value is inherently a valuable statistical measure. However, the manner in which the use of the p value has morphed into a sole determinant of significance has led to an epidemic of irreproducible, vastly over-interpreted research. A complete reform of the experimental process is needed to accelerate the production of scientific research that can more directly and expediently be applied to the treatment and improvement of human health. The actions of the NIH and high profile journals help push in the direction of achieving more research transparency. However, this is only the beginning. Reform of the experimental process must take place at all levels of the scientific ladder, from those students who are first learning about the scientific method to experienced researchers currently training future generations of scientists.