P Values and Confidence Intervals

Some of the critiques raised by Hwang et al. ¹ in their letter concern the risk of type I and type II errors and the implications of such errors for interpretation of results reported by Cookey et al. ² We would like to point out that some of these issues relate to the specific statistical tools chosen by investigators and specifically to the choice of hypothesis testing techniques versus estimation techniques. A P value quantifies the probability of an observed result, or one more extreme, under the assumption of a null hypothesis. P values are usually interpreted in relation to a 5% level of confidence, such that a P value of less than 5% is interpreted as evidence that the null hypothesis can be rejected. However, when a study reports a large number of tests, there is a risk that one or more of the null assumptions will be rejected due to chance, leading to type I error. Type I error can be understood as a “false positive” result of a statistical test since it suggests a difference in the population when one does not exist. When the number of observations is small, it is increasingly possible that an observed difference is compatible with random variation (at the 5% level of confidence) even when the null hypothesis is not true, a type II error. Type II error can be understood as a “false negative” because it suggests a lack of difference between groups in the population when a difference does exist. By the same token, when the sample size is small, any chance effect that is significant will likely be large. ³ For these reasons, the International Committee of Medical Journal Editors (ICMJE) recommends that authors should: “When possible, quantify findings and present them with appropriate indicators of measurement error or uncertainty (such as confidence intervals). Avoid relying solely on statistical hypothesis testing, such as P values, which fail to convey important information about effect size and precision of estimates.” This echoes a recommendation from the Task Force on Statistical Inference of the American Psychological Association, which states “It is hard to imagine a situation in which a dichotomous accept–reject decision is better than reporting an actual P value or, better still, a confidence interval. ⁴ A confidence interval provides an estimate of effect and also a range of values that are consistent with the observed data at the specified level of confidence (usually 95% confidence intervals are reported). A confidence interval for a measure of effect, such as an odds ratio, provides all of the information provided by a statistical test and more. When a confidence interval does not include the null value (e.g., 1.0 in the case of an odds ratio; 0.0 for a standardized mean difference), the null hypothesis would have been rejected had a statistical test been used. However, if the interpretation of a confidence interval leads to a statement of significance, the same issues of multiple “testing” arise. Therefore, carefully planned and thoughtful data analysis is required in addition to the selection of the best statistical tools. The CJP endorses the ICMJE recommendations, and we encourage authors to avoid excessive reliance on P values in reporting their analyses.