The ups and downs of observational vaccine research

In The Lancet Infectious Diseases, Chemaitelly and colleagues¹ estimate the relative long-term effectiveness of a third (booster) dose of COVID-19 mRNA vaccine compared with receiving only two doses in preventing SARS-CoV-2 infection and severe disease. Using rich national data from Qatar, the authors perform the estimation in various subgroups, finding that the relative effectiveness is higher in individuals more clinically vulnerable to COVID-19. Estimating the effectiveness over time, the authors found that by 6 months after receipt of the booster, relative effectiveness had mostly waned. The importance of these findings, and particularly of the heterogeneous relative effectiveness in different subgroups, is evident.

This study joins a long line of important observational vaccine studies done during the COVID-19 pandemic. Soon after the vaccines were first introduced in late 2020 following successful phase 3 clinical trials, a deluge of acute scientific questions arose, some of which include: how effective are the vaccines in specific subgroups of high clinical vulnerability (eg, immunosuppression and chronic kidney disease)? How effective are they in pregnancy? How effective are they against emerging variants? Are there safety concerns that were too uncommon to be detected in the clinical trials? Randomised clinical trials, which are by nature slower to be performed and usually limited to specific populations, were not able to provide the necessary answers in time. Observational studies, based on national data or specialised cohorts, rushed in to fill the gap, contributing important knowledge and aiding in formulating public health policy worldwide.² It would probably be reasonable to say that observational epidemiological studies have never been as important as during the COVID-19 pandemic.

However, despite the proliferation of observational studies, researchers must never forget the high risk of bias inherent in them. A specific example from the study by Chemaitelly and colleagues¹ could serve as a good example of this, as the authors estimate negative relative effectiveness starting 6 months after boosting, concluding that immune imprinting from pre-omicron vaccines is probably harming the immune response to omicron variants. Although this conclusion is possible, one must be cognisant of the many possible biases. For example, it is possible that the adjustment performed did not fully account for the differences between the boosted cohort and cohort that did not receive a booster, resulting in residual confounding. Further, it is possible that the cohort that did not receive a booster was less frequently tested if ill, resulting in differential outcome misclassification; it is possible that the use of discrete-time hazards conditioned on survival at least 6 months after vaccination results in selection bias was due to depletion of susceptibles from the cohort that did not receive a booster.³ All of these biases are reasonable explanations for the finding of negative relative effectiveness, probably even more so than the possibility of actual immune imprinting. In fact, considering all these possible biases through a careful lens, I would surmise that the negative relative effectiveness observed in the study, after most of the effect from boosting has waned, is in fact a failed test for a negative control outcome,⁴ pointing to possible bias in the rest of the study findings. Although the authors cite evidence from the immunological literature that supports their assertion, other immunological studies oppose it, instead claiming that the ancestral strain is sufficiently antigenically similar to the omicron variants so that cross-reactivity from the original vaccine is beneficial.⁵

As I mentioned above, observational epidemiology has been instrumental for generating important scientific evidence during the COVID-19 pandemic. But this new-found importance has not lessened its difficulties. Even as the field progresses and becomes more rigorous with the greater application of formal causal inference,⁶ and novel techniques such as target trial emulation,⁷ valid estimation remains highly challenging. With this in mind, authors of observational epidemiological studies should at all times remain careful and modest in their conclusions.

© 2023 Flickr - Marco Verch Professional