8. Guard against statistical fallacies in medical research

ABSTRACT

Medicine is a sensitive science as it deals with the life and health of people. A slight error can blow up into an enormous damage when invalid results are applied to the population at large. Thus, extreme care is required in conducting and reporting medical research studies. Since medical research is mostly empirical, variations and consequent uncertainties are integral part. Statistical fallacies occur when data and database research are knowingly or unknowingly poorly done or misreported. This article highlights such instances and illustrates commonly occurring fallacies in medical research. Proper design, collection of the right data on an adequate sample, immaculate analysis, and correct interpretation can minimize such fallacies and increase the credibility of research results.

A fallacy in science is a flawed argument leading to an invalid conclusion. This could be intentional to deceive or unintentional due to carelessness or lack of understanding. Either way, it is harmful. If the students do not ask any questions after a class despite repeated goading, it does not mean that they have understood everything. Fallacies are statistical when the data or database results are misrepresented or misinterpreted. The popular quote “head in an oven, feet in a freezer, and the person is comfortable, on average” epitomizes the statistical fallacies. This ignores the number of observations and the enormous variation. A common misbelief that a couple with four male children is very likely to have a fifth as a female ignores that the sex of each birth is independent and the process has no memory of what happened in the past.

Fortunately for statisticians but woefully for some professionals, most medical research is empirical and draws inference from the data collected from a group of subjects. Varying data from different individuals express uncertainties—at the same time also give an opportunity to select, manipulate, and misinterpret the data. Data are not necessarily facts, although they take people away from belief and opinion, and emphasize evidence. The results may look genuine but may suffer due to subtle omissions. Whereas some such instances are caught in the review process, some remain, and some persist due to inherent and genuine variations. Such fallacies can lead to faulty conclusions and misguided policies. They also affect the credibility of statistical studies.[1] This article describes some of the statistical fallacies, particularly those that are subtle and escape attention, to alert the reader about such fallacies so that s/he uses abundant caution in examining the results. Some of these are mentioned in the previous articles in this series, but now this article highlights them by putting them together in a more succinct manner. The net effect of these statistical fallacies is that errors occur in diagnosis, treatment, and prognosis, and these can jeopardize the life and health of many[2] when the invalid results are applied to the population at large. This article also suggests ways to avoid these fallacies.

Correlation and Regression

Consider Figure 1 on the relationship between height and aspartate aminotransferase level in 15 male adults. For 14 persons, they are randomly scattered with no correlation, but for one person, both these measurements happened to be high. This is an outlier in this example. Just because of one value, the relationship appears significant, and the correlation coefficient shoots from 0.01 to 0.31. Although the outlying value is genuine and there is no error in recording, this has distorted the relationship from almost no relationship to a substantial relationship. Thus, it is important to keep track of outliers and ensure they are not distorting the relationship.

Figure 1.

Overriding effect of an outlier on correlation. AST: aspartate aminotransferase

Now consider the relationship between lung functions and age as illustrated in a previous article in this series.[3] Lung functions increase from childhood to age 30–40 years and decline afterward. They have a strong curvilinear relationship, but the correlation coefficient was only 0.19 in that sample. It is because the Pearson correlation coefficient measures only the linear relationship and fails to measure a curvilinear relationship. A parabolic fit gave a multiple correlation coefficient = 0.88. As another example, consider progesterone and estrogen in the menstrual cycle, which increase and decrease according to ovulation. Investigating linearity in this case would be ridiculous. Wang et al.[4] described the relationship between health factor score and metabolic score (MetS). MetS initially increased when health factor score was low, then steeply declined and settled to a low value for high factor score. Linearity and correlation coefficient in this case would give erroneous results. Thus, the use of correlation coefficient and linear regression should be done with only those variables that are expected to follow a linear trend. For others, a nonlinear regression should be tried, and coefficient of determination (η²) should be used as a measure of the strength of relationship and not correlation coefficient.

Statistical Paradox

The crude death rate in India is nearly 7 per thousand population—so is in Sweden. Do we have similar deaths? The fallacy arises due to a very different age structure. India has nearly 4% of people aged more than 70 years, while Sweden has more than 15%. Since infant mortality is still high in India despite enormous gains,[5] 7 deaths per thousand population in India are substantially children and the young, whereas 7 deaths in Sweden are mostly of old age people who are vulnerable to death anyway. A high proportion of deaths of deaths in old age is a positive feature since death is a certainty.

Consider autoimmune disease cases in patients admitted to two specialty hospitals [Table 1]. Top rows of totals suggest that the percentage of cases is nearly the same in both hospitals for males, females, as well as combined. However, when the cases are divided by severity, serious cases are substantially more in hospital B, especially for females. This illustrates that the totals can be fallacious and hide the substantial differences when divided into subgroups. When a trend in subgroups disappears or reverses in the combined group, this is called Simpson’s Paradox.

Table 1.

Male–female cases of autoimmune disease of different severity in two hospitals

Severity and Sex	Hospital A			Hospital B
	n	Cases	Percent	n	Cases	Percent
Total	800	160	20.0	600	120	20.0
Male	280	40	14.3	340	50	14.7
Female	520	120	23.1	260	70	26.9
Mild + Moderate	500	80	16.0	550	100	18.2
Male	180	20	11.1	310	40	12.9
Female	320	60	18.8	240	60	25.0
Serious	300	80	26.7	50	20	40.0
Male	100	20	20.0	30	10	33.3
Female	200	60	30.0	20	10	50.0

ROC: receiver operating curve

If 70% of diabetes patients attending a clinic are obese, it does not mean that 70% of obese patients are diabetic. In terms of probability, P (C | D) is not the same as P (D | C). If alanine aminotransferase (ALT) is high in 80% of cases of liver disease and if the liver disease is in only 5% of persons in a population and ALT is high in 40%, it can be shown by Bayes’ rule that only 10% of the population has liver disease in those with high ALT. Precisely this kind of fallacy occurs when sensitivity and specificity [or area under the receiver operating curve (ROC) curve] is used to assess the prediction performance of a model. This is a common fallacy.

If 21 frequent users of artificial intelligence (AI) are found to have heart attack, how big is this warning? For media, this is a big catch. If there are 700,000 other AI users who did not have a heart attack, can we say that the use of AI protects from heart attack? This kind of fallacies are common in media reports. Thus, the presentation must be balanced.

Misuse of P values

P value is the probability of incorrectly rejecting a true null hypothesis. This is the probability of Type-I error. Type-II error is failing to reject a false null hypothesis. The former corresponds to convicting an innocent, and is more serious than the latter, which is similar to failing to convict a guilty due to insufficient evidence. Evidence in research is the data collected from the subjects. This could mislead mostly due to sampling fluctuations but also because of data errors and a biased sample. Thus, P value is the composite effect of several such factors—a fact mostly ignored. This is discussed in detail by Wasserstein et al.[6]

P value fallacies occur on at least two counts. First, a statistically significant value, such as P < 0.05, is considered the gospel truth whereas this can be easily achieved just by increasing the sample size—even when the effect is trivial. P value should be calculated for the presence of a pre-specified medically significant effect, which hardly anybody considers. Also, nonsignificance, such as P ≥ 0.05, is sometimes interpreted as absence of effect or equivalence,[7] whereas it only means that the available sample could not detect an effect or difference. This can easily happen when the sample size is small, and the study is underpowered. In addition, a sample of small size has little likelihood of representing the cross-section of the population. If the outcome is rare, such as infection in a modern operation theatre, a small sample may fail to give a valid picture. However, a small sample is more efficient to understand or discover the processes as it can be intensively investigated in depth that minimizes, if not eliminates, all loose ends.

Second fallacy arises when multiple P values are used on the same set of data, and each is interpreted as significant when it is, say, less than 0.05. Multiple tests of hypotheses on the same set of data enormously increase the probability of Type-I and increase the risk of getting false-positive result.[8] The irony is that Type-I and Type-II are universally accepted errors because these are inevitable, but they can certainly be minimized by proper design, collection of correct data on a reasonable number of subjects, and immaculate analysis.

Biased Sample

Inclusion and exclusion criteria define the population under study. However, those criteria also restrict the generalizability. Many studies[9,10] are carried out on hospitalized patients, but the conclusions are implied for the nonhospitalized cases also. In other setups, when inclusion–exclusion criteria are given due consideration in deriving the conclusions, the sample chosen may not be random that can affect the generalizability. Consecutive cases appearing in a facility can be considered as good as random but, in some studies, only those conveniently available are chosen.[11] This introduces unsurmountable bias. In clinical trials, informed consent is mandatory. This itself introduces unintentional bias as the participants providing consent are not similar to those refusing consent with respect to their baseline.[12] This is mostly ignored at the time of drawing conclusions, and the fallacies continue unabated.

Studies done on old age persons have natural bias since only those surviving are available and those not in good health may have expired. This is called survival bias. This is highlighted by Ahmad and Arbel[13] for mortality risk in long-term COVID studies. It is fallacious to imply the findings to all old age people since they apply only to the survivors who are likely to be a healthier segment of the elderly population. A related fallacy occurs when survival, such as 5-year survival, is reported for cancer patients. In this case, age is generally ignored. Note that 5-year survival for a patient of age 60 years has very different implications than for an 80-year-old patient, because degeneration due to ageing is a strong factor in death.

A case in point is the weekend effect. This occurs when more critical cases are admitted during the weekend, and the mortality rate is higher.[14]

Results from meta-analysis are believed to be more credible as they combine results from several similar studies. However, it is oblivious of publication bias since mostly papers reporting significant effects are published. Studies with negative results are rarely accepted, although the editors are now more sensitive to this bias and increasingly providing space to the negative studies. Believe meta-analysis results only if negative studies are not ignored.

Manipulations

Unknown fallacies occur when a study reports only the favorable findings and suppresses the unfavorable or indifferent findings. It is extremely difficult to detect such fallacies since only a finished paper is available—not just to the readers but also to the reviewers. In the literature, this is described as cherry-picking.[15]

Another unknown fallacy occurs when data are analyzed and re-analyzed by different methods to get a favorable result. This is ‘torturing the data till it confesses’ and statistically called data-dredging.[16] This may also involve, in some cases, deliberately deleting subjects with inconvenient data and/or adding some subjects with favorable data. Again, all these manipulations may not be traceable in a draft of the paper for publication. To avoid this, the protocol must be prepared in anticipation of these fallacies and the researcher must stick to the analysis plan initially proposed in the protocol. If a deviation is needed due to unforeseen reasons, this must be explained.

Inadequate Analysis

Inadequate analysis is a common source of statistical fallacies. My experience suggests that ignoring reality is among the most common inadequacies. For example, hardly any medical relationship is linear, but linearity happens to be the relationship of choice in most cases, just because it is easy to study. Young et al.[17] described nonlinear association between sleep duration and plasma p-tau181 in the Framingham Heart Study. There are several other relationships which are nonlinear, including between lung functions and age discussed earlier.

The second related problem occurs when a single regression equation (linear or nonlinear) is obtained for the entire range of the predictor values. Epidemiologic studies suggest that cardiovascular risk decreases as high density lipoprotein (HDL) cholesterol rises from low values and remains low for moderate-to-high values and rises for very high values.[18] A single regression for the entire range of HDL cholesterol values may not be able to capture these subtilities. A spline regression with knots[19] may be more suitable and also requires some knowledge about the points of inflection.

While dealing with a data series, check for abrupt changes. A sudden break or unexpected change may carry medical significance and should not be ignored so long as it is genuine. On the other hand, if the pattern starts to show the target level, reality may be disguised. Such a data should be re-examined for their credibility.

Tests such as analysis of variance (ANOVA) F require homoscedasticity. But it is known that higher values in a group generally have higher variance. For example, people with higher body mass index also have higher variability. This is quite often ignored while running an F-test, leading to fallacious results.

A previous article in this series[20] warned about fallacies occurring in prediction models due to inappropriate assessment and inadequate predictivity because some of the predictors are not considered in the eagerness to build a “parsimonious” model. Parsimony has lost relevance now due to the availability of enormous computing power, and there is no need now to restrict to a few predictors. This is in addition to the common flaw of using sensitivity-specificity area under the ROC curve as indicators of predictivity. They are for classification of the known and not for prediction of the unknown. Positive predictive value and negative predictive value are the right indicators for assessing predictivity.

The prevalent practice is to compare areas under the ROC curve to identify a better prediction model and conclude no difference if the areas are nearly equal. Although this is for classification of the known rather than prediction of the unknown, the two (or more) areas can be the same despite major differences [Figure 2]. In this figure, curve A has high sensitivity when (1—specificity) is low and curve B has higher sensitivity when (1—specificity) is high. Area under the curve masks this useful feature.

Figure 2.

Two different ROC curves with nearly the same area

The same can happen with the area under the concentration curve while comparing the bioavailability of two drugs at different points in time in pharmacologic studies.

For comparing the performance of two hematinics in antenatal women, two kinds of analysis can be done. First, quantitative analysis with Hb levels where a small gain (or loss) after taking the hematinic can be measured. Second, count the anemics, with an arbitrary cutoff such as Hb < 8, before and after, without worrying about the quantitative level of Hb. The two analyses may give entirely different results. Consider a setup where the gain in Hb level is measured after use of a hematinic for 4 weeks and the following values obtained in eight women:

Rise in Hb level (g/dL): 0.4 0.7 –0.9 0.3 0.1 0.5 0.9 0.2

The mean rise in Hb level is 0.275 g/dL and SD = 0.542 g/dL. This gives t = 0.275/(0.542/√8) =1.44 and P = 0.097 with one tail when the fall in Hb with the use of a hematinic is ruled out. This test for small n is valid only when the rise follows a Gaussian distribution, and the null hypothesis is no rise (H₀: µ =0). Since this P value is more than the conventional 0.05, the inference will be that the rise is not statistically significant. However, seven of the eight women showed a rise and one showed a fall. The binomial distribution with null π = ½ (rise as likely as the fall when the hematinic is not effective) in this case gives P(x ≥ 7) = 0.035. This is statistically significant. Note how an analysis can be chosen to give a favorable result yet not cause any suspicion on the findings.

Conclusion

Medicine is a highly sensitive science as it deals with the life and health of people. Thus, extreme caution is required in research by choosing an appropriate design, collecting the right data on all relevant variables, adequate analysis, and proper conclusions. However, this turns out to be a tall order for many of us, and we take shortcuts. Proliferation of predatory journals has worsened this scenario. Medical research must be subjected to the highest rigor. It must provide valid and reliable answers to the pertinent questions set out at the beginning of research, and avoid possible fallacies highlighted in this paper.

Conflicts of interest

There are no conflicts of interest.

Funding Statement

Nil.