Testing behaviour may bias observational studies of vaccine effectiveness

Abstract

BACKGROUND

Recent observational studies suggest that vaccines may have little effect in preventing infection with the Omicron variant of severe acute respiratory syndrome coronavirus 2. However, the observed effects may be confounded by patient factors, preventive behaviours, or differences in testing behaviour. To assess potential confounding, we examined differences in testing behaviour between unvaccinated and vaccinated populations.

METHODS

We recruited 1,526 Australian adults for an online randomized study about coronavirus disease 2019 (COVID-19) testing in late 2021, collecting self-reported vaccination status and three measures of COVID-19 testing behaviour: testing in past month or ever and test intention if they woke with a sore throat. We examined the association between testing intentions and vaccination status in the trial’s baseline data.

RESULTS

Of the 1,526 participants (mean age 31 y), 22% had a COVID-19 test in the past month and 61% ever; 17% were unvaccinated, 11% were partially vaccinated (one dose), and 71% were fully vaccinated (two or more doses). Fully vaccinated participants were twice as likely as those who were unvaccinated (relative risk [RR] 2.2, 95% CI 1.8 to 2.8, p < 0.001) to report positive COVID testing intentions. Partially vaccinated participants had less positive intentions than fully vaccinated participants (RR 0.68, 95% CI 0.52 to 0.89, p < 0.001) but higher intentions than unvaccinated participants (RR 1.5, 95% CI 1.4 to 1.6, p = 0.002).

DISCUSSION

Vaccination predicted greater COVID-19 testing intentions and would substantially bias observed vaccine effectiveness. To account for differential testing behaviours, test-negative designs are currently the preferred option, but their assumptions need more thorough examination.

Introduction

Recent observational studies have suggested that vaccines for the Omicron variant of severe acute respiratory syndrome coronavirus 2 (SARS-Cov-2) may have little or no effect in preventing infection while being effective in preventing hospitalization. For example, a Danish study found low vaccine efficacy against Omicron infections: 55% for the BNT162b2 vaccine and 37% for the mRNA-1273 vaccine (¹). However, for longer follow-up (after 90 d), the effectiveness observed was negative (<0), triggering reports that vaccines might paradoxically increase Omicron infections, which was picked up by multiple blogs and subsequently fact-checked by AP News (²). However, differences in testing behaviour may bias such observational studies of vaccine effectiveness.

As for all observational studies of vaccines, the observed effects may be confounded by other patient factors and preventive behaviours such as social distancing and mask wearing (³). However, vaccine-related differences in coronavirus disease 2019 (COVID-19) testing behaviour may also cause differences in case detection (although this is less likely to affect hospitalizations). For example, if unvaccinated people were half as likely as vaccinated people to get tested for respiratory symptoms, they would also be half as likely to be detected as coronavirus disease 2019 (COVID-19) cases. This differential testing would artefactually dilute estimates of vaccine effectiveness (or could even create a spurious negative effect from vaccines, as seen in the Danish study). The test-negative design—which stratifies for health seeking behaviour by patients who test negative as controls—is a common method to reduce the bias from health-care-seeking behaviour differences between vaccinated and unvaccinated people, but it relies on assumptions about severity and test accuracy (⁴,⁵). To assess the degree of confounding potentially caused by any differential testing behaviour, we aimed to assess any differences in past and intended testing behaviour between unvaccinated and vaccinated populations.

Methods

We included all participants from a nationally representative sample (by age, education, and gender) of 1,526 Australian adults (430 men, 1,064 women, 32 non-binary or not reported) who were recruited for an online randomized study about COVID-19 testing between October and November 2021 (www.anzctr.org.au/Trial/Registration/TrialReview.aspx?id=382318) using methods we have detailed previously (⁶,⁷). We collected self-reported vaccination status (unvaccinated, one dose, or two doses) and three measures of COVID-19 testing behaviour: self-reported testing in past month or ever and intention to test if they woke with a sore throat. Self-reports of influenza vaccination have demonstrated more than 93% agreement against an immunization registry data (⁸). We considered people who had received two vaccine doses at the time of assessment as fully vaccinated because at this time point in Australia, third-dose boosters were only available for high-risk workers (eg, health care workers). We examined the association between testing intentions and vaccination status across the sample with Pearson χ² tests and non-parametric linear-by-linear tests for trend. Analyses were conducted using Stata/BE version 17.0 (StataCorp, College Station, Texas).

Results

The 1,526 participants (mean age 31 y) were from all Australian states; 22% had a COVID-19 test in the past month and 61% ever; 17% were unvaccinated, 11% were partially vaccinated (one dose), and 71% were fully vaccinated (two or more doses). Fully vaccinated participants were twice as likely as those who were unvaccinated (relative risk [RR] 2.2, 95% CI 1.8 to 2.8, p < 0.001) to report positive COVID-19 testing intentions (Table 1). Partially vaccinated participants had less positive intentions than fully vaccinated participants (RR 0.68, 95% CI 0.52 to 0.89, p < 0.001) but higher intentions than those who were unvaccinated (RR 1.5, 95% CI 1.4 to 1.6, p = 0.002). Fully vaccinated participants were also twice as likely (RR 2.1, 95% CI 1.5 to 3.0) to report being tested in the past month than those who were unvaccinated (p < 0.001).

Table 1:

Association between vaccination status and COVID-19 testing intentions and self-reported behaviours (N = 1,526)

Behaviour	No. (%)			Test statistic, p-value
	Both	One	Unvaccinated
How likely is it that you’d get tested for COVID-19 if you woke up with a sore throat tomorrow?*				χ2(4) = 93.8, p < 0.001; ztrend = 9.24, p < 0.001
Likely	600 (55)	64 (37)	65 (25)
Neither	161 (15)	41 (24)	51 (19)
Unlikely	330 (30)	67 (39)	147 (56)
Did you get tested for COVID-19 in the past month?				χ2(4) = 20.6, p < 0.001; ztrend = 4.13, p < 0.001
Yes	258 (24)	41 (24)	29 (11)
No	833 (76)	131 (76)	234 (89)
Have you ever been tested for COVID-19?				χ2(4) = 73.7, p < .001; ztrend = 8.58, p < .001
Yes	733 (67)	90 (52)	104 (40)
No	358 (33)	82 (48)	159 (60)

* Five-point Likert scale response options are very likely, likely, neither, unlikely, and very unlikely; combined into three categories for analyses

COVID-19 = Coronavirus disease 2019

Discussion

For all three measures (intention, self-reported testing in past month or ever), vaccination predicted greater COVID-19 testing intentions and behaviours among vaccinated people compared with unvaccinated people. If confirmed in other studies, this behavioural difference has both policy and research methods implications.

Methods for observational studies of influenza vaccine effectiveness adjust for several important confounders, but the spectrum of confounding behaviours in the current SARS-CoV-2 pandemic may be different, particularly substantial differential testing behaviour. If unvaccinated people were detected as cases at half the rate of vaccinated people (as seen for the “tested in the past month” in Table 1), then a true vaccine effectiveness of 30% would lead to a “negative” observed vaccine effectiveness (for R_vacc/R_unvacc = 0.70, and half R_unvacc cases not detected, then the apparent vaccine effectiveness is R_vacc/0.5 × R_unvacc = 0.70/0.50 = 1.4) of –40% (as seen in the Danish study for the Moderna vaccine at 91–150 d; see Appendix for details). A recent report from the Statens Serum Institut in Denmark also noted lower testing rates of unvaccinated people compared with vaccinated people of a similar magnitude as in our study (⁹).

A recent World Health Organization technical brief on vaccine effectiveness against the Omicron variant did include the Danish study, but the majority of the studies were test-negative designs (¹⁰). For example, the UK Health Security Agency analysis using a test-negative design estimated that the vaccine effectiveness against symptomatic disease was 26%, 2%–18% (2–24 wk versus ≥25 wk post-dose), and 63% for one, two, and three doses of vaccine, respectively (¹¹). However, for hospitalization the vaccine effectiveness was substantially better at 52%, 52%–72%, and 88%, respectively. Such test-negative studies may still be biased by test seeking behaviour if there are differences in severity between cases and non-cases (⁵).

Assessment of vaccine effectiveness against SARS-CoV-2 infection should use methods to account for differential testing behaviours. Test-negative designs are currently the preferred option, but their assumptions should be more thoroughly examined.

Ethics Approval:

The manuscript was reviewed by the University of Sydney Human Ethics (2020/781).

Informed Consent:

N/A

Registry and the Registration No. of the Study/Trial:

N/A

Data Accessibility:

Please email carys.batcup@sydney.edu.au for information about the data.

Funding:

This study was not specifically funded, but in-kind support was provided by National Health and Medical Research Council (NHMRC) Investigator Grant 1080042 (P Glasziou), NHMRC/Heart Foundation Early Career Fellowship 1122788 (C Bonner), and NHMRC Principal Research Fellowship 1121110 (K McCaffery).

Disclosures:

P Glasziou is a member of the Therapeutic Guidelines Board. K McCaffery, E Cvejic, J Ayre, K Pickles, C Batcup, and C Bonner received seed grant funding from the Sydney Infectious Disease Institute at the University of Sydney. The authors declare no competing interest.

Peer Review:

This manuscript has been peer reviewed.

Animal Studies:

N/A

Appendix – Adjusting vaccine effectiveness for differential test behaviour

In the Discussion, we stated that “if unvaccinated people were detected as cases at half the rate of vaccinated people … then a true vaccine effectiveness of 30% would lead to a “negative” observed vaccine effectiveness … of –40%.” This can be calculated from the 30% vaccine efficacy (VE) and 50% detection rate as follows:

R_vacc is the rate among vaccinated people, and R_unvacc is the rate among unvaccinated people. If the true VE is a 30% reduction in cases, that means VE = R_vacc/R_unvacc = 0.70, that is, a 0.30 or 30% reduction. If instead only half of R_unvacc cases are detected, then the apparent VE = (R_vacc/0.5) × R_unvacc = (R_vacc/R_unvacc)/0.5 = 0.70/0.50 = 1.4, or an apparent VE of a 40% increase. (Note: if the under-detection in the R_unvacc is “U,” then this can be written [R_unvacc/R_unvacc]/U.) More generally, if the relative rate of testing and detection is RTR, and the true VE is a T_VE% reduction, then T_VE = 1 – R_vacc/(R_unvacc), but the observed rate among unvaccinated people will be R_unvacc × RTR, so the App_VE = 1 – R_vacc/(R_unvacc × RTR) = 1 – (R_vacc/(R_unvacc))/RTR = 1 – (1 - T_VE/RTR).

Figure A1:

The true and apparent VE for the range of true VE from 0 to 1 for (a) equal testing behaviour among vaccinated and unvaccinated people (solid line) and (b) unvaccinated people undergoing half the testing or detection of vaccinated people (dotted line)

Note: Arrows indicate the 30% T_VE scenario

VE = Vaccine efficacy

Funding Statement

This study was not specifically funded, but in-kind support was provided by National Health and Medical Research Council (NHMRC) Investigator Grant 1080042 (P Glasziou), NHMRC/Heart Foundation Early Career Fellowship 1122788 (C Bonner), and NHMRC Principal Research Fellowship 1121110 (K McCaffery).