Factors influencing estimated effectiveness of COVID-19 vaccines in non-randomised studies

Abstract

Non-randomised studies assessing COVID-19 vaccine effectiveness need to consider multiple factors that may generate spurious estimates due to bias or genuinely modify effectiveness. These include pre-existing immunity, vaccination misclassification, exposure differences, testing, disease risk factor confounding, hospital admission decision, treatment use differences, and death attribution. It is useful to separate whether the impact of each factor admission decision, treatment use differences, and death attribution. Steps and measures to consider for improving vaccine effectiveness estimation include registration of studies and of analysis plans; sharing of raw data and code; background collection of reliable information; blinded assessment of outcomes, e.g. death causes; using maximal/best information in properly-matched studies, multivariable analyses, propensity analyses, and other models; performing randomised trials, whenever possible, for suitable questions, e.g. booster doses or comparative effectiveness of different vaccination strategies; living meta-analyses of vaccine effectiveness; better communication with both relative and absolute metrics of risk reduction and presentation of uncertainty; and avoidance of exaggeration in communicating results to the general public.

Introduction

Vaccines represent a major advance against the COVID-19 pandemic. For several vaccines, phase III randomised trials showed high efficacy for reducing symptomatic infections. Randomised trials had less than ideally desired data on hospitalisations and deaths. For example, in a living systematic review of COVID-19 vaccines versus placebo, among 35 randomised trials with 219 864 participants there was a sum of only 41 deaths.¹ However, subsequent observational studies found also real-world effectiveness for severe outcomes and death.² While many non-randomised observational studies are performed for COVID-19 vaccine research, there are many reasons to be cautious with them, as they can easily be flawed.^3–6 Here, I discuss how to mitigate these flaws.

The framework considers a comparison of the outcomes of vaccinated versus unvaccinated individuals in non-randomised data. Outcomes are grouped here for convenience into three categories: infections (variously defined, for example, symptomatic, severe or any documented (including asymptomatic)), hospitalisations and deaths.

Factors influencing vaccine effectiveness estimates

Pre-existing immunity

Vaccine effectiveness may be adding only a small absolute benefit in people with some pre-existing immunity, while the benefit may be substantially larger in those without pre-existing immunity. The typical reason for pre-existing immunity is prior infection. Prior infection may or may not have been documented, since most infections remain undocumented.⁷ The literature on the additional benefits of hybrid immunity (prior infection plus vaccination) versus only vaccination and versus only prior infection is still contentious and evolving.⁸

People with pre-existing infections are increasingly commonly distinguished in observational studies, but documented infections are only a minority and many more people have been infected without having had positive documentation with PCR or antigen test. Some studies may use serology to document prior infection, but even those may miss infected individuals who never mounted detectable antibodies or seroreverted.

If the vaccine effect (relative risk reduction) is E(prior) and E(notprior) in those with and without prior infection, respectively, the proportion of those who have prior infection is P and the proportion of prior infection is the same in the vaccinated and unvaccinated groups—then the observed effect will be E(prior)P+E(notprior)(1−P). If the ratio E(prior)/E(notprior)=R, then the observed effect will be E(notprior)(RP+1 P). If R is negligible (ie, the vaccine adds negligible value to people previously infected), this simplifies to E(notprior)(1−P). The same concept can be extrapolated also to vaccine-induced pre-existing immunity, for example, where vaccine boosters or other schedules of repeated vaccination are considered. Other things being equal, vaccine effectiveness is expected to decrease over time, as more people get infected or vaccinated. Conversely, vaccine effectiveness may increase again, if natural or vaccine-induced immunity wanes over time.

Pre-existing immunity may differ in the vaccinated versus unvaccinated groups for various reasons. For example, due to health-seeking behaviour, vaccinated people may have been more diligent to protect themselves and thus have lower proportion of prior infection. Meta-analyses show that younger, poorer, less educated people and minorities are less willing to get vaccinated in developed countries,⁹ and typically these populations had higher infection rates before vaccines emerged.^{10 11} In developing nations, conversely, the dynamics of vaccination uptake may be different.¹² Conversely, some policies may prioritise vaccinating people with higher proportion of prior infection. For example, nursing home residents were highly prioritised and such populations already had 4–5-fold higher prior infection rates (as determined by seroprevalence studies) than the general population in 2020.^{13 14}

Vaccination misclassification

In most vaccine programmes, documentation of vaccination is accurate. However, some studies may rely on self-report data. Moreover, vaccination status still needs to be defined and there can be differences on whether someone is defined as vaccinated immediately after vaccination or after some period has elapsed and if so, how long that period is. Different definitions may cause misclassification. Non-differential, independent misclassification spuriously decreases estimates of vaccine effectiveness. However, misclassification may also be differential and may also depend on outcomes, for example, if self-report is influenced by recall bias. The magnitude and sometimes even the direction of such bias are difficult to specify and require careful scrutiny of the specific setting and circumstances.

Exposure differences

The vaccinated group may become more heavily exposed to the virus after vaccination, if they feel liberated to engage in more frequent, massive and high-risk exposures. This phenomenon of risk compensation decreases the benefit of vaccination.¹⁵ Some data have suggested little change in protective behaviour early after vaccination.¹⁶ However, guidance by public health authorities may occasionally encourage risk compensation, for example, if some measures (eg, masks) are removed only for vaccinated individuals. For example, many countries in 2021 offered public health narratives that encouraged the vaccinated individuals to regain their lives, while keeping more restrictions or extra requirements for testing for the unvaccinated. Sometimes, unvaccinated people are entirely banned from some high-risk settings, while vaccinated people have no restrictions. For example, unvaccinated healthcare workers have been paused from their jobs in several countries. Conversely, the vaccinated group may be less heavily exposed to the virus, if they are so health conscious that they engage in fewer exposures compared with the unvaccinated group, despite some liberation of activities after vaccination. Differences in exposure between the vaccinated and unvaccinated groups may extend also to differences in exposure to specific viral strains with different risk-taking behaviours. Furthermore, sometimes vaccines may have genuine differences in efficacy against different viral strains.

Testing

The vaccinated group may be more frequently tested (and thus more likely to document infection) if they are more health conscious and/or have better access to testing compared with the unvaccinated group. Conversely, the vaccinated group may be less frequently tested, if the vaccination reduces the severity of the infection and thus fewer infected people have any symptoms; if the vaccination induces a sense of security and people are less worried to be tested (even if symptomatic) and if public health authorities encourage more frequent testing for the unvaccinated. The same reasons may lead to more or less use of more sensitive testing detection methods as opposed to less sensitive ones (eg, PCR with different circle threshold or PCR vs rapid tests or self-tests).

Testing interpretation may also differ, thus causing misclassification of outcomes. For example, it may be more likely to call indeterminate tests positive for unvaccinated rather than for vaccinated people, although this is a lesser concern for tests that are well standardised and not subjectively interpreted. One may still misclassify outcomes such as ‘symptomatic COVID-19 infection’, since one makes a judgement call based on both testing results and patient symptoms.

Disease risk factor confounding

The vaccinated group may have a higher background risk of developing severe disease after infection compared with the unvaccinated group, if vaccination has been prioritised for people at higher risk of COVID-19 consequences (older age, presence of comorbidities). Early deployment of COVID-19 vaccines in most countries used such priority rules.¹⁷ The opposite situation may also happen, if some extremely high-risk people are preferentially left unvaccinated, for example, if people with terminal illness are spared vaccination because of perceived futility or because of concern about potential vaccine adverse effects with extreme frailty.¹⁸ Adjusting for age and sex, people who choose to be vaccinated are often far more healthy (‘healthy vaccinee effect’), for example, in a US study, vaccinated people had markedly decreased risk of dying from non-COVID-19 causes (adjusted relative risk 0.34–0.54).¹⁹ Matched designs, standard multivariable adjustments, propensity matching, instrumental variables and more complex causal modelling may be used with variable success each time, depending on how well confounders are known, measured and incorporated in the modelling.

Hospital admission decision and treatment use

The vaccinated group may have a higher chance of being admitted to the hospital, if their socioeconomic and demographic profile is such that it leads to better access to and coverage of hospital care than the unvaccinated group and/or if more serious background risk leads to hospital admission more readily. Conversely, the unvaccinated group may have a higher chance to be admitted in the hospital, if for the same clinical presentation and other things being equal, there is a perception that lack of vaccination portends serious outcomes. Hospital admission may then also affect the risk of death. All these considerations are speculative, and one needs to probe empirically if such biases exist in different circumstances and settings. Similar reasons may lead to differential access to and use of treatment modalities that may affect hard outcomes. Biases could also arise from counting people admitted with a COVID-19-positive test versus people admitted due to COVID-19.

Death attribution

The vaccinated group may have a lesser chance of having deaths attributed to COVID-19 rather than to other causes, if vaccines are perceived as very effective, while unvaccinated people may have their deaths attributed to COVID-19 more easily. Conversely, the unvaccinated group may have a lower chance of having deaths attributed to COVID-19, if the unvaccinated group deaths occur in environments that are less able to pursue COVID-19 diagnosis (eg, disadvantaged populations). There is no direct evidence to date for differential death attribution based on vaccination status, but there is evidence that deaths can be overattributed to COVID-19,^{20 21} while in other situations COVID-19 deaths may be missed,²⁰ so the differential attribution according to vaccination status is possible as a form of diagnosis bias.

Outcomes affected

As shown in table 1, all of these factors may impact the estimated effects for deaths. With the exception of death attribution and treatment use choices after hospital admission, all other factors may also impact the estimated effects for hospitalisations. Pre-existing immunity, vaccination misclassification, exposure differences and testing bias may impact the estimated effects for infection outcomes.

Table 1.

Factors influencing estimated COVID-19 vaccine effectiveness: outcomes affected and type of influence

Factors	Outcomes affected*	Influence on effectiveness estimate†
Pre-existing immunity
Same in V and UNV	I, H, D	Genuine
Different in V and UNV	I, H, D	Spurious (selection bias)
Vaccination misclassification	I, H, D	Spurious (misclassification)
Exposure difference
Induced by perceived vaccine protection	I, H, D	Genuine
Pre-existing, carried forward	I, H, D	Spurious (selection bias)
Testing
Typical diagnosis bias	I, H, D	Spurious (selection bias, misclassification)
Affecting treatment	H, D	Genuine
Disease risk factor confounding	H, D	Spurious (confounding bias)
Hospital admission decision
Induced by perceived vaccine protection	H, (D)‡	Genuine
Other reasons	H, (D)‡	Spurious (selection bias, confounding)
Treatment use difference
Induced by perceived vaccine protection	D, (H)§	Genuine
Other reasons	D, (H)§	Spurious (selection bias, confounding)
Death attribution	D	Spurious (misclassification)

*The outcomes considered here are infection (I), hospitalisation (H) and death (D).

†There are two major types of influence: genuine impact represents effect modification (ie, the vaccine effectiveness is genuinely different in different groups, settings, etc) and conversely, spurious impact means bias (ie, the vaccine effectiveness is different from what is being estimated).

‡May affect death outcome if hospital admission affects the risk of death (not necessarily so).

§For treatments that are applied before hospitalisation and may affect the need for hospitalisation.

UNV, unvaccinated; V, vaccinated.

Spurious versus genuine effects

It is useful to separate whether the impact of each factor on estimated vaccine effectiveness is spurious (bias) or genuine (effect modification) (table 1). Selection, confounding or information (misclassification) biases generate effectiveness estimates not representing the true magnitude of vaccine benefits: the true benefit may be larger or smaller than what is observed. These biases should be removed or adjusted for. Conversely, genuine effect modification should be considered (rather than removed or adjusted for) in understanding real-world vaccine effectiveness. In effect modification, the benefit is truly different in different people, settings and circumstances.

For pre-existing immunity, its impact can be either genuine or spurious. When pre-existing immunity is unrelated to vaccination status, its impact represents genuine effect modification: vaccines are more effective when there is no pre-existing immunity and may have no added benefit when pre-existing immunity is saturated. Conversely, when pre-existing immunity levels differ in vaccinated and unvaccinated groups, observed vaccine effectiveness is biased.

Exposure differences that are induced by vaccination (risk compensation) may be seen as effect modification. Conversely, when pre-existing differences in the exposure risk are pervasive (carried forward also after vaccination becomes available), this generates spurious estimates of benefit. Both patterns may coexist.

Vaccination misclassification, testing, disease risk factor confounding, hospital admission decision and treatment use and death attribution generate spurious effect estimates. There are some exceptions, however, that might be seen as effect modification induced by vaccination: for example, vaccinated people may have less testing and thus fewer infections detected early and may be managed less aggressively.

Problems in implementation of vaccination (eg, bad batch effects, spoiled vaccines (eg, storage in inappropriate temperature) or missed/delayed doses) may diminish effectiveness. Genuine heterogeneity of effectiveness across populations may also exist without necessarily obvious, identifiable biological or other reasons.

Assessing and correcting effectiveness estimates

The direction and magnitude of impact for each factor on estimated effectiveness may vary depending on circumstances, specific settings and chosen study design and cannot be generalised. One should think carefully about each factor and, whenever possible, collect information that allows taking it into account in analysing and interpreting results.

If reliable data can be collected, factors that represent biases can be corrected or adjusted for using traditional approaches for selection bias (eg, adjusting for potential background seroprevalence differences, exposure differences and testing intensity) and confounding (eg, adjusting for disease severity risk factors). Seroprevalence studies are doable, but they have many potential biases themselves,²² and require cautious interpretation. Exposure risk and intensity are typically measured by mobility tracking. These data often pertain to whole population group-level estimates with ecological bias and are not very useful for understanding vaccine effectiveness. Individual-level data of exposure risk and intensity may also be captured by questionnaires and by tracking tools, but may still have substantial error (measurement error, reporting bias, response/non-response bias and volunteer bias).²³ COVID-19 testing data are more tractable, but one needs additional information on indications of testing (eg, testing may be performed for symptoms or for routine screening). Vaccination misclassification is best addressed by using reliable register data and avoiding self-reporting. Outcome misclassification may be diminished by assessments of outcomes in audited medical records blinded to vaccination status.

Design considerations for vaccine effectiveness studies predate this pandemic^3–5 and are being revisited in the COVID-19 setting.⁶ For factors that can be measured, various adjustment approaches (including propensity score and marginal structural models) may be used. Their ability to remove bias depends on whether key factors have been measured, measurements are accurate and models are properly constructed. Given that many important factors are difficult to measure reliably, some designs try to circumvent the need for measuring these factors by making cases and controls similar by design choice. Two such options are the test-negative case–control^24–26 and case–crossover designs.^{27 28}

The test-negative case–control design enrols only individuals who seek care and receive testing for the same/similar clinical reasons. In principle, this may remove the impact of differences in health-seeking behaviour that influences several factors discussed above. The design had been applied in influenza vaccines, noting congruence with randomised trial results.²⁹ However, as opposed to influenza, people get tested for COVID-19 for very different reasons, beyond symptoms, for example, contact with known case and repeated work-related screening.⁶ Differences in health-seeking behavior (plus underlying differences in exposure and propensity to get tested and to receive other medical care that affects incidence of symptoms, for example, getting vaccinated for other respiratory pathogens) are not necessarily eliminated in test-negative case–control studies.

Case–crossover studies consider the same individual on different time periods where vaccination status changes, for example, when unvaccinated (control period) and after he/she has been vaccinated. Case–crossover studies had known problems even before the COVID-19 era.²⁸ The design is most suitable if exposure is intermittent, effect is immediate and transient and outcomes occur abruptly. COVID-19 vaccination fits some but not all of these features. Moreover, vaccination induces potentially additional changes in behaviour and protection measures, as discussed above. Perhaps most importantly, epidemic activity may differ between compared time periods. Epidemic wave fluctuations over time may induce large changes both in the background risk of exposure and in what people do to limit exposure, based on perceptions and estimates of epidemic activity.

Improving evidence and inferences

Realisation of the complexity of these factors should lead to great caution in interpreting estimates of COVID-19 vaccine effectiveness from non-randomised studies. Even greater caution is needed when effects or changes in effects are modest. If a vaccine is 95%–100% effective, biases may affect exact estimates of effectiveness, but large benefit is still readily demonstrable. Conversely, claims for different effects for the same vaccine on different outcomes, for different vaccines, for the same vaccine over time, for different viral strains or for different dosing schedules require extra caution, since the comparative treatment effects (if non-null) may be subtle. Moreover, the factors that shape effectiveness estimates could change over time and in different settings.

Table 2 lists some steps and measures that may improve the reliability of inferences in non-randomised studies, their rationale and some challenges that exist in their implementation. They include overarching measures (registration and sharing practices), better data collection, better design for analyses, systematic reviews of the evidence and improvements in communication of information.

Table 2.

Some measures to consider for improving the reliability of evidence from non-randomised studies of COVID-19 vaccine effectiveness

Measures to consider	Rationale	Challenges
Overarching measures
Registration of studies and analysis plans	Allows to know what studies and analysis plans were preconceived and adhered to original plans and reduces degrees of freedom for data dredging	Most observational studies are non-registered or are registered after the analyses are done; there is debate on whether retrospective designs should/could be meaningfully registered; analytical plans are rarely registered in sufficient detail
Sharing of raw data and code	Allows independent validation of analyses and optimises the use of the data in overarching syntheses of data from multiple studies	Sharing has been limited for various reasons (privacy, consent and legal issues, as well as reluctance of primary investigators)
Better data collection
Background collection of reliable information on seroprevalence, exposures, testing, disease risk factors, risk profiles on hospital admission and use of treatments	Allows for better adjustments and exploration of effect modification	Some of this information may be biased or very difficult to collect reliably
Blinded assessment of outcomes, for example, death causes	Allows removing some outcome misclassification biases	Blinding records requires time and resources and a committed effort
Better designs
Use of maximal/best information in properly matched studies, multivariable analyses, propensity analyses and other models	Designs that consider and hopefully address more biases are better	Observational studies are unlikely to ever eliminate all possible biases
Performing randomised trials, whenever possible, for suitable questions (eg, use of booster doses, comparative effectiveness of different vaccination strategies)	Removes many of the biases	Reluctance to perform randomised trials when data suggest large efficacy (but this may be less of a concern for comparative effectiveness), randomised trials also have biases
Systematic review
Living reviews and meta-analyses	Provide bird’s eye view of evolving evidence	Meta-analyses have their own, long list of biases
Better communication
Use of both relative and absolute metrics of risk reduction and presentation of uncertainty	Allows better comprehension of the magnitude of the benefit	Poor ability of many/most people to understand risks and other quantitative metrics
Avoidance of exaggeration in communicating results to the general public	Minimises misconceptions, confusion, panic (eg, from misleading claims of loss of vaccine effectiveness) or dangerous behaviour changes (eg, from misleading claims of retaining high effectiveness even with high exposures)	There is an avid market seeking immediate information on what is new on the pandemic and vaccines and sensationalism is prominent; the anti-vax movement makes confusion worse by adding extra misinformation

Given the complex difficulties faced by non-randomised studies, randomised trials remain indispensable. While placebo-controlled trials are difficult to justify when a vaccine shows large efficacy in short-term trial results, longer-term follow-up is still quintessential. Questions of comparative assessments where effectiveness differences may be subtle or even null (eg, use of booster doses, head-to-head comparisons of different vaccines or of different vaccine strategies) should be approached by randomised trials.

For both observational and randomised designs, transparency and wide availability of the relevant data are essential.³⁰ Finally, collection of reliable information on effectiveness should be coupled with collection of reliable information on adverse events to allow meaningful comparisons of benefits and harms of different vaccination strategies on absolute risk scales.

Ethics statements

Patient consent for publication

Not applicable.

Ethics approval

Not applicable.