The Need for Robust Epidemiological Evidence During a Pandemic

A feature of the current coronavirus disease 2019 (COVID-19) pandemic has been speculation about factors that may increase or mitigate a person’s risk of infection and severe outcomes [1]. One example that has circulated on social media since March 2020 is the potential for influenza vaccination to increase the risk of coronavirus infection (including severe acute respiratory syndrome coronavirus 2 [SARS-CoV-2]), based on weak evidence from a US Department of Defense (DoD) study [2] and a small randomized trial that reported increased risk of noninfluenza viruses among vaccinated children [3]. Skowronski et al [4] have provided robust evidence from a multiyear analysis of a large Canadian dataset that shows no association between seasonal coronavirus risk and influenza vaccination. Their study underscores the importance of sound epidemiological approaches for identification of disease risks at a time when rigorous review is sorely needed.

The DoD study neglected at least 2 fundamental epidemiological concepts: confounding and selection bias. First, the nature of any association may be spurious if the design fails to adequately control for confounding factors. These are factors associated with both disease and exposure that may induce an association where none exists. When attempting to estimate causal effects, as is the case in vaccine research, it is particularly important to report unbiased estimates [5]. Identification of confounders requires consideration of the sequence of events to avoid unnecessary adjustment (which can also introduce bias [6]) and should be supported by prior knowledge, rather than statistical significance alone [7]. For example, in influenza vaccine effectiveness research, we commonly adjust for age because we know that the propensity to be vaccinated and susceptibility to influenza disease are associated with age [8]. Table 1 in Skowronski et al’s article illustrates this problem: the crude odds ratio suggested a harmful association between vaccination and human metapneumovirus (HMPV) but this disappeared after adjustment for confounding by age [4].

Second, the exposure (vaccination) distribution among controls in a case-control study is supposed to represent the exposure distribution of the underlying source population [9, 10]. However, inclusion as controls persons who tested negative for coronaviruses but positive for influenza induces selection bias. These controls have a different exposure distribution (they are less likely to be vaccinated) than the source population, inflating the observed effect estimate. The re-analysis by Skowronski et al (their Supplementary Table S3 [4]) shows just how influential this selection bias was in the DoD [2] study, with a reversal of the point estimate in some cases (eg, those with rhinoviruses).

Another study (mis)cited to deride influenza vaccines in the face of COVID-19 is a small 2013 influenza vaccine trial from Hong Kong in which there was an elevated risk of noninfluenza virus infections among vaccinated children. This is a curious addition to the evidence base as there were at most 6 seasonal coronavirus cases identified in that study (the exact number cannot be discerned from the information provided), with most infections attributable to rhinoviruses. That study had a small sample size, a very small number of noninfluenza viruses, and, as Skowronski et al [4] note, was underpowered to validate the hypothesis it was seeking to test. The reported 95% confidence interval was consistent with an increased risk of infection that ranged from 1.31 to 14.8 and should be interpreted with caution. Moreover, the study was confined to children aged ≤15 years, making its generalizability (another basic epidemiological concept) questionable, especially with reference to COVID-19 for which we know there is a reduced risk of acute symptomatic infection among children.

The hypothesis underpinning this work posits that infection by one virus leads to a temporary nonspecific immunity against other infections. By preventing influenza infection, influenza vaccines may appear to increase the risk of infection by other respiratory pathogens, such as coronaviruses, because vaccinees are not afforded the temporary cross-protective immune response that might prevent or delay infection by other viruses [11, 12]. These effects are thought to be brief, lasting days or weeks, but that is probably long enough to be meaningful in the context of cocirculating viruses. This phenomenon may not matter for all study designs but does matter for the validity of test-negative studies that are routinely used to estimate influenza vaccine effectiveness because it can introduce selection bias. Were the vaccine to modify the risk of infection by noninfluenza viruses, the exposure distribution among influenza-negative controls would not represent the source population (it would be overrepresented by vaccinated controls). However, in a meta-analysis that summarized 12 test-negative studies, we observed no net effect of influenza vaccination on noninfluenza virus detection, even among children [10]. In meta-regression, younger age was weakly associated with discrepancies in vaccine effectiveness using different control groups, but a more important predictor was whether a study was conducted for a single or multiple seasons. Both the DoD [2] and Hong Kong [3] studies used data from a single season, which can be problematic when making inferences for diseases known to have seasonal variations.

Another important caveat for interpreting this evidence for COVID-19 is the specificity of the outcome. The studies included in our meta-analysis [10] and the Hong Kong study [3] grouped noninfluenza viruses together, making it difficult to know whether virus interference is more important for some viruses than others. Moreover, where individual viruses were assessed, as in the DoD study [2], the increased risk applied to seasonal coronaviruses. We have no information about whether the effects observed remain consistent across all seasonal coronaviruses (OC43, NL63, HKU1, and 229E) and whether observations made for seasonal coronaviruses might also apply to a newly emerged zoonotic coronavirus, SARS-CoV-2.

While COVID-19 causes significant distress to communities and their healthcare systems, we should not forget that influenza remains an important cause of morbidity and mortality, particularly in the elderly and young children [13]. Influenza vaccines, while imperfect, can reduce some of this burden. Calls for the public to be vaccinated against influenza during the COVID-19 pandemic are motivated by expectations that we can potentially reduce the likelihood that severely ill influenza-infected patients will unnecessarily burden our healthcare system. Speculation about the possible harmful effects of influenza vaccination, based on imperfect evidence, could undermine this strategy.

In the current atmosphere of rapid publication and citation of preprint articles, epidemiologists need to be more thorough than ever in their assessment of studies reviewed for publication to identify validity issues. Similarly, journals should heed their reviewers concerns. Despite the rise in so-called arm chair epidemiologists, those actually trained in the nuances of valid study design may identify important limitations that should be addressed prior to publication to prevent misinterpretation and the spread of misinformation.

Notes

Acknowledgments. Thanks to Ben Cowling, Kanta Subbarao, and Ian Barr for their helpful comments.

Disclaimer. The World Health Organization Collaborating Centre for Reference and Research on Influenza is supported by the Australian Government Department of Health.

Potential conflicts of interest. The author: No reported conflicts of interest. The author has submitted the ICMJE Form for Disclosure of Potential Conflicts of Interest.