Identification of the causal odds ratio in test-negative-design studies of COVID-19 vaccine effectiveness

We thank Drs. Shrier and Stovitz for their letter concerning our article on the potential bias introduced by self-testing in COVID-19 vaccine effectiveness studies at the primary care level [1]. We clarify below our hypotheses regarding the effect of COVID-19 vaccination on self-testing and the selection of study participants based on symptomatology, and discuss their implications.

Effect of vaccination on self-testing

This letter first asks why our directed acyclic graph (DAG; figure 2a) [1] includes a direct arrow going from vaccination to self-testing. We agree that an association can be observed between vaccination and self-testing due to a common cause, which is healthcare-seeking behaviour. We have thus represented arrows going from healthcare-seeking behaviour into vaccination and self-testing, respectively. This is also mentioned several times in our manuscript.

We intentionally drew an arrow going from vaccination to self-testing, and used causal language throughout the text, to reflect our assumption that getting vaccinated could impact one’s propensity to take a self-test prior to/instead of consulting a general practitioner (GP). For instance, once vaccinated, one may be less concerned about potential COVID-19 adverse outcomes and decide to self-test without seeking medical advice.

DAGs enable effective scientific communication by making all assumptions explicit. We nevertheless appreciate that they are partly subjective. Other researchers may disagree with the hypothesized causal structure between variables—and the resulting biases. These hypotheses can only be validated or rejected by appropriately designed empirical studies.

Adjustment for self-testing

Drs. Shrier and Stovitz then suggest that adjusting for self-testing, despite closing the non-causal path of vaccination → self-testing → self-test result ← infection, opens another non-causal path: vaccination → self-testing ← acute respiratory infection (ARI) symptoms ← infection. Although we agree that this applies to other study designs, the test-negative design is only valid if we select participants according to a common case definition—and common indication for consultation [2]. At the primary care level, we only include people with ARI symptoms, regardless of severity. In other words, we select based on ARI symptoms. This is represented by the box around the ARI symptoms variable, which blocks the non-causal path between the vaccination and outcome variables.

We share the authors’ view on Dagitty. This software is a powerful tool to evaluate whether the causal effect measure can be estimated and we commend its recent extension to outcome-based sampling studies [3]. Unfortunately, we could not fully encode our DAG, as we believe that Dagitty only allows selection based on one variable—and we should differentiate between selection and adjustment variables [3]—whereas our study selects participants based on both GP consultation and ARI symptoms. This is illustrated by figure 1a, provided by Drs. Shrier and Stovitz, which includes only one selection node. As a result, the software message states that the model is incorrectly adjusted. Again, we believe that selecting based on ARI symptoms closes an otherwise open non-causal path and allows estimation of the causal odds ratio (assuming no further bias).

We conducted simulations by using parameters reflecting our hypotheses and our code is publicly available. The results demonstrate that adjusting for self-testing does remove bias in vaccine effectiveness. This is also illustrated by numerical examples available in the manuscript supplement.

The authors of the letter further mention that, in a study selecting on GP consultation only, the causal odds ratio may be estimated by adjusting for a (theoretical) mediator on the vaccination → self-testing pathway. This is illustrated in their figure 1b. We do not elaborate on this example, as it does not reflect our causal hypotheses, but we thank them for this important contribution to the reflexion on biases related to self-testing.

We thank the authors and the Editor for this opportunity to openly discuss our scientific work and hope that this correspondence will benefit interested readers.

Ethics approval

Ethics approval was not needed, as simulated data were used for this study.

Author contributions

C.L.D.: conceptualization, formal analysis, methodology, validation, visualization, writing—original draft preparation, writing—review and editing. B.N.: methodology, validation, writing—review and editing. S.M.: conceptualization, methodology, validation, writing—review and editing. M.d.L., G.T., A.M., N.L.-M., I.M., M.L., P.B.R., A.E., N.S., L.D., I.M.-B., M.H., B.O., R.G., M.S., S.K.F., C.P., J.A.L.D., R.D., M.-A.R.-W., A.M., J.C.: writing—review and editing. C.v.H., M.K.: methodology, writing—review and editing. M.K.: project administration, writing—review and editing. E.K.: conceptualization, formal analysis, funding acquisition, methodology, project administration, supervision, validation, visualization, writing—original draft preparation, writing—review and editing.

Conflicts of interest

None declared.

Funding

This study received funding from the European Centre for Disease Prevention and Control (ECDC), through the framework contract Vaccine Effectiveness, Burden and Impact Studies (VEBIS) of COVID-19 and Influenza ECDC/2021/019.

Data availability

There are no new data associated with this article

Use of artificial intelligence (AI) tools

None declared.