3. P values, power, and medical significance for credible results

ABSTRACT

Type I and Type II errors are inherent in any empirical medical research on an antecedent-outcome relationship when it is based on a dataset of a sample of subjects. Type I error is the incorrect rejection of a true null hypothesis, and its probability in a study is the P value. This error is more serious and is kept under control by specifying a cap called the level of significance. The complement of the probability of Type II error, called power, is the probability of not missing a medically significant effect when present. This article concisely explains P values, power, and medical significance in nontechnical terms for our medical colleagues and their implications for assessing the credibility of medical research.

Introduction

Two major types of statistical inferences are (i) estimation and (ii) tests of significance. The former is mostly used in descriptive studies that focus on the profiles of subjects and the extent of disease or a condition, such as prevalence, in a specified population or group. In contrast, the latter generally applies to analytical studies that test a hypothesis on antecedent-outcome relationships. The concepts of P values, power, and medical significance are used in studies that test a hypothesis and not in descriptive studies on estimation. In estimation, confidence intervals are commonly used for assessing reliability.

P values have pervaded medical research to such an extent that they are considered essential for establishing the credibility of results. These values have attained the status of a cliché for many researchers. However, there are limitations, and extensive discussion are ongoing regarding what P values reveal and what they conceal. To appreciate their implications, it is necessary to understand their actual meaning. This communication clarifies and explains P values in simple terms for our medical colleagues.

It has become fashionable these days to power a study to yield some useful results. Our interaction with many medical researchers suggests that the term “power” is often incorrectly interpreted. We strive to provide a lucid explanation of this term so that it is understood from the correct perspective and properly used in medical research.

The third term, “medical significance,” is closely related to the statistical power of a study. The world appears to be shifting from statistical significance to medical significance to ensure that results are applicable to real-life clinical situations. We explain the meaning of medical significance and its close relationship with power. Readers may find the manner in which these intricate concepts are explained in this article refreshing.

All these concepts are based on Type I and Type II errors, which, in turn, depend on the null and alternative hypotheses. A proper understanding of these errors is a prerequisite for appreciating the other three concepts. This understanding may help in conducting research with the highest rigor, which is especially important in medicine, as it deals with the life and health of people. These errors are explained in the next section.

We presume that readers are aware of the process followed in statistical inference. The basic feature of this process is that an inference is made about the target population by studying only one sample. Since one sample can always differ from another due to what are called sampling fluctuations, chance remains an essential component of all statistical inferences. A sample can be misleading if it happens to contain subjects of specific types, such as younger individuals, those with milder disease, or those with unknown comorbidities. Statistical methods are basically used to reduce the chance of this error through proper study design and the selection of a representative sample, as well as to quantify this chance by computing various probabilities.

Type I and Type II Errors

Type I and Type II errors are integral part of any empirical decision-making process that depends on the available evidence, which may vary from sample to sample due to sampling fluctuations. These errors are best illustrated by a trial in a court, where a person can be convicted despite being innocent due to the strong nature of adverse evidence or acquitted despite committing a crime due to a lack of evidence. Acquittal does not necessarily mean that the person has not committed the crime—only that the evidence was not sufficient to convict him. The court starts with the presumption that no crime was committed and allows the prosecution to present sufficient evidence against the accused. Only the evidence presented before the court is examined, while any other evidence is ignored. You can see that convicting an innocent is a more serious error, whereas acquitting a criminal is not as serious. The magnitude of these errors depend on how the evidence is collected, analyzed, and argued in a court of law.

Before statistical errors, consider diagnostic errors, which are easier to understand. Clinicians these days examine a great deal of evidence before reaching a diagnosis, yet errors still occur. Misdiagnosis happens when a person without disease is diagnosed with a disease and unnecessarily prescribed treatment, leading to inconvenience, side effects, and cost. This is more serious and akin to convicting an innocent in a court. Missed diagnosis, which occurs when a person has a disease but is not diagnosed, is less serious. This error is generally transient because the person will be examined again if the complaints persist, and the disease will eventually be detected and treated.

In a medical research setup, the null hypothesis of no effect is setup, similar to the presumption of innocence in a court of law. Sample data is collected and examined to determine whether it provides “sufficient” evidence against the null. Any other evidence is ignored, just as in court proceedings. As in the court example, the decision depends on how large the sample is and how meticulously the data is collected and analyzed. The study is designed to obtain clear and focused evidence. The alternative hypothesis represents a claim, such as the improved efficacy of a regimen, which is analogous to committing a crime. When the null is rejected because sufficient evidence is found against it, the alternative is accepted. Note that the null hypothesis is rejected or retained but never accepted. Retaining the null returns us to the status quo, as though the study was never conducted—similar to acquitting a person of a crime. This is where some researchers misinterpret results—not being able to reject a null hypothesis is often wrongly construed as evidence of no effect in a clinical trial.[1] Instead, it should be stated as “no statistically significant effect could be detected by the sample in the study,” just as a court may find “no sufficient evidence to convict the person.” The study sample could be such that it happens to miss a real effect. We explain the meaning of statistical significance later in this article so that it is correctly interpreted.

Since all medical studies are based on samples, they can mislead depending on the types of observations that happen to be included in the study sample. When the sample data, by chance, wrongly provides sufficient evidence against the null, we unnecessarily reject it and commit what is called a Type I error. This results in a false positive conclusion, similar to convicting an innocent person. This is more serious because an ineffective regimen may be approved for use—leading to unnecessary side effects, increased costs, and a delay in administering the actual effective treatment.

When the data fails to detect a real effect, a Type II error is committed. This error is analogous to acquitting a criminal due to a lack of evidence. In this case, an effective regimen is denied, but this is often a temporary issue, as another trial may eventually establish its efficacy if it truly exists. These errors, along with their court analogies and clinical judgement implications, are illustrated in Table 1.

Table 1.

Type-I and Type-II errors

Court decision	Court example		Clinical decision	Actual		Statistical decision	Statistical null hypothesis
	Crime not committed	Crime committed		Disease absent	Disease present		Effect absent	Effect present
Convicted	More serious	✓	Disease present	Mis-diagnosis	✓	Reject the null of no effect	Type I error	✓
Not convicted	✓	Less serious	Disease absent	✓	Missed diagnosis	Do not reject the null of no effect	✓	Type II error

The probability of Type I and Type II errors in a study depends on factors such as variability across subjects, the distribution (e.g., Gaussian or skewed) of values, and sample size. Only the sample size is easy to control—other factors are intrinsic and can be managed only through proper study design. Note that these errors arise only in hypothesis-testing setups and not in descriptive studies that examine subject profiles or estimate the status of a condition in a population. Second, these errors are relevant only for empirical studies based on a sample of data affected by numerous known and unknown factors; they are not relevant for intracellular studies, where a phenomenon is intensively studied on biological material, or to genetic studies conducted on a smaller number of individuals. Third, these are statistical errors that arise solely from sampling fluctuations and do not include extraneous errors due to incorrect data, selected data, inadequate analysis, etc. Those unfamiliar with these possibilities may inadvertently attribute Type I and Type II errors to extraneous factors.

P Values and the Level of Significance

As mentioned earlier, all medical studies are based on samples, and samples can be misleading. Consequently, there is always a chance of reaching an erroneous conclusion. In analytical studies, the probability of committing the more serious Type I error (because the sample happens to be misleading) is called the P value. Thus, the P value is the probability that our sample data appears inconsistent with the null hypothesis despite the null being true. That is, if the null hypothesis—no difference between groups—is true, our sample may still indicate a difference. Various statistical tests, such as the Student’s t-test and Chi-square test, are used to obtain the P value under certain conditions, such as independence and random sampling. We will discuss some of these tests in a later article in this series. Many researchers ignore the requirement of a random sample, as seen in the work of Horibe et al.[2], and calculate P values for nonrandom samples. Nonrandom samples are likely to provide incorrect P values. Correct calculation of P values also requires that the data for each subject is accurately measured and properly collated, as discussed in the previous article of this series,[3] otherwise, the results can be misleading. The P value can be low in a study, such as 0.008, or high, such as 0.639, depending on the data in the sample. This probability can never be zero due to the omnipresent variations and uncertainties. When this probability is low, the chance of erroneously rejecting a true null hypothesis is small, and we can safely reject it. This is termed statistical significance and is analogous to convicting an innocent person with a small chance of error. Since this error is serious, a cap is set, called the level of significance, denoted by α. Conventionally, no more than a 5% chance of committing a Type I error is tolerated. In this case, α = 0.05. The level of significance is decided beforehand when preparing the study protocol and is not based on the data.

The conventional cap of 5% is being extensively debated and is now considered too high for clinical applications. A low cap of 1% can be advocated, although Benjamin and Berger[4] have suggested an even stricter threshold of 0.5% for novel findings. Reducing the level of significance is a newer approach, and in the context of a clinical trial, it implies that the chance of erroneously labeling a regimen as more effective than another is lower. Consequently, we can be more confident about its efficacy when the P value is less than the lower significance level. Additionally, most journals now require the reporting of exact P values rather than just stating P < 0.05 or > 0.05, as done by Sadeghi et al.[5] and several others, allowing readers to make their own judgements. There is another important implication of using a lower significance level: the likelihood of an effective regimen being incorrectly labelled as ineffective increases. This corresponds to a Type II error, and the following discussion clarifies its implications.

Power and Medical Significance

In analytical medical studies, where an antecedent is investigated for its effect on an outcome, there is always a chance that the sample data, due to sampling fluctuations, fails to detect the required effect when it is present. This results in a false negative conclusion, known as a Type II error, and its probability in a study is denoted by β. Its complement (1 – β) is called power. Thus, power is the probability that our study sample does not miss a specified effect when it is present. “Does not miss” means that the effect is found to be statistically significant at the pre-fixed level of significance. The power is generally set at 80% or 90%.

The effect size is measured in several different ways but is easier to understand as the difference in the efficacy of a regimen between the case and control groups in a clinical trial. The calculation of power also requires a random sample. It is often overlooked that power depends not only on sample data but also on the size of the effect to be detected. For a given sample, power is low for detecting a small effect size and high for detecting a large effect size. For a specific study design, a large sample is required to detect a small effect—similar to searching for a pin in a haystack, which requires extensive effort—compared to searching for a brick, which requires much less effort. Many studies, such as those on the association between serum bicarbonate and 28-day mortality in critically ill patients,[6] ignore the effect size and focus solely on statistical significance with a null hypothesis of no effect. The conventional null of no effect (H₀: Effect = 0, or equal efficacy) will almost always be rejected when the sample size is large. For example, if an existing regimen has an efficacy of 87% and a new regimen has 88%, the small effect size of 1%—measured by the difference in efficacy—would be statistically significant if the sample size is large in each group. However, hardly any clinician would consider switching to the new regimen for just a 1% gain over a well-established existing regimen. A substantial gain, such as at least 3%, could justify switching to the new regimen. The important clinical question is not whether the effect size is zero, but whether the effect size is large enough to justify switching to the new regimen and abandoning the existing one. An effect size that is considered sufficient to change current medical practice is called the medically (or clinically) significant effect and is denoted by δ. Thus, the hypothesis H₀: Effect size ≤ δ is tested against the alternative H₁: Effect size > δ. This is a one-tailed test. Despite their great relevance, not many studies investigate such hypotheses. The specification of a medically significant effect may not be easy, as it can vary from situation to situation. However, present-day research requires that this be specified by the researcher and that the results be interpreted accordingly. Power is calculated to ensure that a study does not miss the specified medically significant effect when it is present. Note that when this effect is not present, no study can detect it except by committing a Type II error.

To avoid confusion, note that a medically significant effect observed in a study is not, by itself, sufficient evidence of its presence. Statistical significance for that effect size is a prerequisite because statistical significance ensures that its presence in the sample is highly unlikely to be due to sampling fluctuations. As mentioned earlier, the P value that determines statistical significance depends on subject-to-subject variations and, to some extent, on study design, but mostly on the sample size, which the researcher can control. Increasing the sample size acts as an insurance to achieve statistical significance for the medically significant effect when present.

Next: The Riddle of Confidence Levels and the Levels of Significance in the Era of Artificial Intelligence.

Conflicts of interest

There are no conflicts of interest.

Funding Statement

Nil.