Active smoking and severity of coronavirus disease 2019 (COVID-19): The use of significance testing leads to an erroneous conclusion

Dear Editor,

In their letter to the Editor, “Active smoking is not associated with severity of coronavirus disease 2019 (COVID-19)”, Lippi et al. [1] present a timely and important meta-analysis to evaluate if smokers are at greater risk of increased severity of COVID-19 symptoms. This is the first study to assess this question and has important implications for clinical practice and public health recommendations. It is indeed imperative at this time to identify if and how smoking habits can contribute to the progression of COVID-19 and its ensuing complications, such as hospitalisation, admission to an intensive care unit, and ultimately death.

Unfortunately, in their statistical analysis, Lippi et al. use a null hypothesis significance testing (NHST) approach which is known to have fundamental flaws as discussed in recent high profile statements [2,3]. The principal use of NHST leads them to erroneously conclude an absence of effect of smoking, which is incorrect, and could have potentially harmful consequences. Lippi et al. estimate a pooled OR (95% confidence interval) of 1.69 (0.41, 6.92) for the risk of increased severity of COVID-19 symptoms due to smoking. Since the lower limit of the OR overlaps the null value of 1, Lippi et al. are led to conclude that “active smoking is not associated with severity of coronavirus disease 2019” (cf. the title of their letter). While the presentation of confidence intervals is commendable, the comparison of a confidence interval limit with a null value to determine whether or not an effect exists is essentially a surrogate for a null hypothesis test with level α=5% [4]. This practice has been strongly criticized and described as a “misuse” or “debasing” of the confidence interval [5].

The estimation approach has been proposed as a more correct and clinically informative statistical approach for interpreting the results of medical studies [6,7]. In contrast to NHST, this approach correctly recognizes that the entire confidence interval represents a range of plausible values for the OR [6,7]. Thus while the point estimate of 1.69 of the Lippi et al. study represents the expected or mean value, the OR has a probability distribution that extends to either side such that values from 0.41 to 6.92 are also quite probable [7]. In fact the bulk of the confidence interval extends well beyond the null value of 1 (Fig. 1 ). Mathematical integration of the OR probability distribution (assuming a Normal distribution for the log odds ratio) indicates a 77% probability that the estimated OR exceeds 1, while the probability of it exceeding hypothetical clinical thresholds of 1.2 and 2.0 are 68% and 41% respectively, as shown in Fig. 1. These results indicate there is a substantial probability of a clinically important effect of smoking on COVID-19 severity, in contrast with the conclusions presented by Lippi et al.

Fig. 1

The application of the estimation approach to the OR estimated in their sensitivity analysis leads to an even more striking difference in interpretation. Exclusion of the largest study of the five studies results in a pooled OR of 4.35 (0.86, 21.86). While Lippi et al. again perform a null hypothesis test and conclude that there is no association, almost the entire range of plausible values, as indicated by the confidence interval, lies well above the null OR value of 1 (Fig. 2 ). It can be calculated that 96% of the OR probability distribution rests above the null, and that the probability of exceeding hypothetical clinical thresholds of 1.2 and 2.0 are 94% and 83% respectively.

Fig. 2

The inappropriateness of using null hypothesis significance testing to conclude an absence of effect has been well documented in the clinical literature [8,9]. The Lippi et al. study is analogous to an underpowered clinical trial where elevated variability, as reflected in the wide confidence intervals, precludes the ruling out of a non-null or clinically important association. A more appropriate conclusion for this study is that a clinically important association is uncertain due to the lack of sample size, but that the effect of smoking on COVID-19 severity remains highly possible; further and larger studies should be encouraged. The clinical and practical message of this conclusion is substantially different from the negative result asserted in both the conclusion and title of the Lippi et al. letter.

It would be accurate for Lippi et al. to only state that there is no statistically significant association (at the 5% level) based on their meta-analysis, as this is simply a statistical result that makes no claim as to what effect sizes may be plausible. However Lippi et al. also use the phrase “no significant association” in their text, omitting the ‘statistical’ qualifier and leading readers to suppose erroneously that the magnitude of association is clinically unimportant. This distortion of meaning is exacerbated in the title of their letter which emphatically states that “active smoking is not associated with severity of coronavirus disease 2019 (COVID-19)”. The faulty translation of NHST results into statements with clinical significance can mislead readers and lead to biases in future research [2]. Indeed two subsequent meta-analyses on the effect of smoking on COVID-19 have already cited the Lippi et al. study as an example of a negative result [10,11].

Amidst the pressing need to conduct and disseminate new research on COVID-19, there remains more than ever the need for researchers and practitioners of evidence based medicine to apply sound methods of statistical inference in the interpretation of their results. Null hypothesis significance testing, without consideration of the range of plausible effect sizes, is known to produce conclusions that are non-reproducible and represent over-simplistic dichotomies that can impede scientific progress [2,3]. The estimation approach with the proper use of confidence intervals is greatly preferred, and in this case, indicates there is a substantial possibility of an association between smoking and COVID-19 severity.

Declaration of Competing Interest

Re: Active smoking and severity of coronavirus disease (COVID-19): the use of significance testing leads to an erroneous conclusion

The authors Ernest Lo and Benoit Lasnier declare that they have no conflict of interest.