Test-negative Designs with Various Reasons for Testing

Abstract

Test-negative designs (TNDs) are widely used for postmarket evaluation of vaccine effectiveness (VE), particularly in cases when randomized trials are not feasible. Unlike classical TNDs, which only include healthcare seekers with symptoms, recent TNDs have involved individuals with various reasons for testing, especially in an outbreak setting. While including these data can increase sample size and hence improve precision, concerns have been raised about whether they introduce bias into the current framework of TNDs, thereby demanding a formal statistical examination of this modified design. In this article, using statistical derivations, causal graphs, and numerical demonstrations, we show that the standard odds ratio estimator may be biased if various reasons for testing are not taken into account. To eliminate this bias, we identify three categories of reasons for testing, namely symptoms, mandatory screening, and case contact tracing, and characterize associated statistical properties and estimands. Based on our characterization, we show how to consistently estimate each estimand via stratification. Furthermore, we describe when these estimands correspond to the same VE parameter and, when appropriate, propose a stratified estimator that can incorporate multiple reasons for testing and improve precision. We demonstrate the performance of our proposed method through simulation studies and a real-data analysis.

The test-negative design (TND,¹) refers to a study in which patients presenting at healthcare facilities are tested for a disease of interest and classified as test positive or test negative. The intervention effect of interest is then estimated as the odds ratio comparing test-positive and test-negative patients. As an observational study design, TND has been increasingly used to study vaccine effectiveness (VE) for infectious diseases such as influenza,² coronavirus,³ and rotavirus,⁴ especially when randomized trials are infeasible or unethical. Compared with a cohort design, TND has the advantage of efficiency and cost-effectiveness and can potentially reduce bias arising from differential healthcare-seeking behaviors.⁵

Classical TNDs recruit only patients who actively seek healthcare and are tested for infection due to disease-related symptoms. However, in many recent applications–particularly COVID-19 vaccine studies–patients tested for other reasons were also enrolled. For example, in a TND study evaluating the effectiveness of face masks for preventing COVID-19,⁶ only 36.4% of patients were tested due to symptoms, accounting for 77.9% of the positive tests but only 16.7% of the negative tests. Other reasons for testing included routine screening at work or school (29.9%), mandatory screening for medical procedures (12.4%), screening out of curiosity (12.2%), and travel screening (8%). In two other COVID-19 TND studies that reported reasons for testing,^7,8 symptomatic individuals comprised a smaller proportion of those tested (25.3% and 29.5%), while contact with positive cases emerged as a major reason for testing (52% and 18%). In addition to these examples, many COVID-19 TND studies involved asymptomatic patients,⁹ mandatory testing for healthcare workers,¹⁰ and massive community screening,¹¹ but none of these tests can be interpreted as testing due to symptoms. Beyond COVID-19, TNDs were also been proposed for evaluating VE in an Ebola outbreak,^12,13 where some participants were identified and tested due to close contact with known cases. Overall, such changes in patient inclusion criteria in TNDs often occur in an outbreak setting, where a substantial proportion of tests are driven by asymptomatic spontaneous healthcare seeking, and the volume of these tests is often too large to ignore.

Despite the emergence of TNDs that include various reasons for testing, appropriate methods for analyzing such studies remain an open question. The status quo is to ignore the reasons for testing and apply a standard analysis, for example, via odds ratios and logistic regression, by aggregating the data across all reasons for testing. However, this method can yield biased estimates of overall VE, as different reasons for testing may involve heterogeneous populations, across which the VE, the likelihood of testing, and covariate distributions can all differ. Lewnard et al.¹⁴ pointed out that reasons other than symptoms for testing may introduce bias, and this issue was conceptualized through causal graphs.^15,16 Both papers and¹⁷ stated that a stratified analysis based on testing reasons could alleviate the bias, but no complete quantification or formal solution was given.¹³ addressed the bias from a naive TND analysis when there were two reasons for testing: (i) symptomatic self-reporting or (ii) testing close contacts of known cases, and proposed a hybrid design to remove such bias. However, their approach relies on a strong assumption that the VE for either recruitment population is the same; in addition, other testing reasons, such as mandatory screening, were not considered in their setting. Finally, a naive solution is to filter out asymptomatic subjects and use only the remaining data to perform a classical TND analysis. Although this procedure is valid, it loses a large amount of information.

In this paper, we combine directed acyclic graphs (DAGs,¹⁸) with additional assumptions to examine the statistical properties of TNDs with various reasons for testing, and propose several estimands along with corresponding unbiased estimators. First, we revisit the classical TND to clarify underlying assumptions and an appropriate estimand, and then identify two additional categories of reasons for testing: mandatory screening and contact tracing, for which we construct corresponding DAGs and derive the target estimands. The target estimands for these three categories are generally distinct, which expands the classical scope of TNDs and provides new perspectives on the generalizability of TND estimates. Second, given multiple reasons for testing, we explain why the odds ratio estimator pooling all testing reasons may be biased. Finally, we propose several estimators to target each VE of interest using the corresponding data. Furthermore, we provide conditions to justify a combined estimator based on multiple reasons for testing, thereby maximizing the use of relevant data to achieve unbiased estimation, valid hypothesis testing, and increased precision. Our proposed methods are not limited to parametric model assumptions; they also offer the flexibility to incorporate nonparametric kernel smoothing to approximate the underlying data-generating distribution. To our knowledge, we provide the first framework to clarify to what extent asymptomatic reasons for testing can be included in TNDs without introducing bias, and our proposed methods are the first to accommodate a range of reasons for testing, which can inform statistical practice for future TNDs.

DEFINITIONS AND ASSUMPTIONS

We consider TNDs to study a specific disease, namely COVID-19, with the goal of estimating VE. For simplicity, we focus on evaluating the effect of a single vaccine dose, although our framework can be extended to accommodate more general vaccination histories represented by the variable $V$. Specifically, for each subject, let $V$ be the vaccination status with $V=1$ if vaccinated and 0 otherwise. Let $I$ indicate the true infection status for severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2), where $I=1$ if the subject is infected and $I=0$ otherwise, as determined by a definitive diagnostic test. Let $S$ be a binary indicator of having COVID-like symptoms, such as fever and coughing. We use $X$ to denote observed confounders, such as age, sex, and calendar time, and let $H$ be a latent (unobserved) healthcare-seeking behavior indicator. Again, for simplicity, we assume $H$ is a binary random variable with $H=1$ indicating actively seeking care if feeling ill. Finally, we define $C$ as a binary indicator of having contact with known cases, which can be impacted by the vaccination status.

For TND studies involving multiple reasons for testing, we define $R$ as a categorical variable encoding the reasons for testing with mutually exclusive categories, with each category associated with a single reason, such as “routine screening for work,” or a pool of related reasons, such as “COVID-like symptoms.” We let $T$ be a binary indicator of being tested at a healthcare setting (and thus eligible for inclusion in a standard TND) with $T=1$ if tested and 0 otherwise. Throughout, we assume all the above variables are measured without misclassification. This setup also applies to studies on other infectious diseases, such as influenza.

We assume the joint distribution of the defined random variables is characterized by the DAG in Figure 1. For the pretest variables $𝒵=\{\mathit{X},H,V,C,I,S\}$, the assumed causal structure is the same as TND literature^5,15,16,19 with the only difference being the new postvaccination variable $C$, which can directly impact infection and symptoms. We introduce this variable for situations when testing is due to case contact tracing. For testing-related variables ($R$, $T$), since $R$ may involve multiple categories of reasons, the joint distribution of ($R$, $T$) can depend on all pretest variables, as shown in Figure 1. However, for a specific reason $r$, $P(R=r,T=1\mid 𝒵)$ can be only a function of a subset of the variables in $𝒵$, and we characterize three categories in the next section.

FIGURE 1.

The data-generating process described by a directed acyclic graph. The dashed arrows represent a simplified depiction of arrows from each node in $𝒵$; for example, $𝒵\to R$ denotes six arrows from all pretest variables to $R$. Here, $\mathit{X}$, $H$, $V$, $C$, $I$, $S$, $R$, $T$ represent observed confounders, healthcare-seeking behavior, vaccination, contact of known cases, infection, symptom, reason for testing, and testing, respectively.

In a TND involving multiple reasons for testing, all individuals tested for SARS-CoV-2 are included in the study. In a prospective design, individuals are recruited, tested, and then categorized as cases or controls. Alternatively, a retrospective design examines electronic health records (EHR) and extracts data related to testing for analysis. For both implementations, we regard the individual data ($𝒵$, $R$, $T=1$) as random samples from the joint distribution characterized by Figure 1, conditional on $T=1$.

THREE CATEGORIES OF REASONS FOR TESTING

Testing Due to Symptoms as in Classical Test-negative Designs

The classical TND only involves individuals who seek care and are tested for infection due to COVID-like symptoms, which we denote by $R=Symptoms$. Accordingly, we assume that

$$\text{Pr}(R=\text{Symptoms},T=1\mid 𝒵)=H\times S\times f_s(\mathit{X},V)$$

for some unknown function $f_s$. This implies that testing due to symptoms requires both healthcare seeking $H=1$ and the presence of symptoms $S=1$ (by definition) and can further depend on observed confounders and vaccination status. Here, we assume testing is not driven by the contact history or the true infection status, which implies that this conditional probability is independent of $C$ or $I$. For example, $f_s(\mathit{X},V)=0.3-0.1V$ implies a 30% probability of testing due to symptoms among unvaccinated symptomatic healthcare seekers, with a 10 percentage-point reduction for vaccinated individuals. Given this assumption on the joint event ($R=\text{Symptome}$, $T=1$), the DAG in Figure 1 implies the DAG for testing due to symptoms in Figure 2.

FIGURE 2.

The directed acyclic graph for classical TNDs. Circle nodes stand for random variables, while rectangle nodes indicate conditioning on specific values.

Beyond the DAG, we assume that test-negative controls are exchangeable across vaccination groups, that is,

$$\begin{gathered} \text{Pr}(I=0,S=1\mid V=1,X=\mathit{x},H=1)= \\ \text{Pr}(I=0,S=1\mid V=0,X=\mathit{x},H=1). \end{gathered}$$

This assumption, also referred to as “control exchangeability,”²⁰ implies the typical TND assumption that vaccination does not reduce non-SARS-CoV-2 causes of symptoms among healthcare seekers.^1,19 These assumptions together motivate an odds ratio estimator for the VE on the relative risk scale:

$$1-\text{Odds-}{\text{Ratio}}_{I\sim V}(\mathit{X}=\mathit{x},R=\text{Symptoms},T=1),$$

where the term $\text{Odds-}{\text{Ratio}}_{I\sim V}$ stands for the odds ratio of $I$ on $V$ conditioning on the values in the parenthesis. With mathematical derivation from eAppendix A; https://links.lww.com/EDE/C291, it follows that

$$\begin{gathered} 1-\text{Odds-}{\text{Ratio}}_{I\sim V}(\mathit{X}=\mathit{x},R=\text{Symptoms},T=1) \\ =1-\frac{\text{Pr}(I=1,S=1\mid V=1,\mathit{X}=\mathit{x},H=1)}{\text{Pr}(I=1,S=1\mid V=0,\mathit{X}=\mathit{x},H=1)}. \end{gathered}$$ (1)

We denote the right-hand side of Equation (1) as $V{E}_{symp}(\mathit{x},1)$, as it evaluates the conditional VE against symptomatic infections, that is, being diseased, among healthcare seekers. Of note, the conditional VE is a function of $\mathit{x}$, which means inference is made on each stratum, such as a specific age group. In summary, classical TNDs eliminate collider bias through their unique design, where controls are healthcare seekers with COVID-like symptoms. With the control exchangeability assumption, the odds ratio identifies VE on the relative risk scale.

Mandatory Screening

Mandatory screening refers to testing due to massive screening, back-to-school prerequisites, international travel, or other factors that are irrelevant to their infection status or symptoms. In other words, tests are motivated by external requirements instead of internal concerns about health. By this definition, there is no direct effect from $H$, $C$, $I$, $S$ to $R$ = Mandatory screening or $T=1$, which implies

$$\text{Pr}(R=\text{Mandatory screening},T=1\mid 𝒵)=f_m(\mathit{X},V)$$

for some unknown function $f_m$. Here, mandatory screening can be directly influenced by observed covariates and vaccination status, given that unvaccinated individuals are often subject to stricter testing requirements.

Given this assumption and Figure 1, we obtain the DAG for mandatory screening in Figure 3 and

$$\begin{gathered} 1-\frac{\text{Pr}(I=1\mid V=1,\mathit{X}=\mathit{x},R=\text{Mandatory screening},T=1)}{\text{Pr}(I=1\mid V=0,\mathit{X}=\mathit{x},R=\text{Mandatory screening},T=1)} \\ =1-\frac{\text{Pr}(I=1\mid V=1,\mathit{X}=\mathit{x})}{\text{Pr}(I=1\mid V=0,\mathit{X}=\mathit{x}}, \end{gathered}$$ (2)

where the left-hand side is an estimable quantity from data under mandatory testing, and the right-hand side defines the conditional VE against infections (covering both symptomatic and asymptomatic infections), which we denote as $VE(\mathit{x})$.

FIGURE 3.

The directed acyclic graph for testing due to mandatory screening. Circle nodes stand for random variables, while rectangle nodes indicate conditioning on specific values.

If COVID-like symptoms are also measured in the data from mandatory screening, following a similar procedure, we can also estimate the conditional VE against symptomatic infections by

$$\begin{gathered} 1-\frac{\text{Pr}(I=1,S=1\mid V=1,\mathit{X}=\mathit{x},R=\text{Mandatory screening},T=1)}{\text{Pr}(I=1,S=1\mid V=0,\mathit{X}=\mathit{x},R=\text{Mandatory screening},T=1)} \\ =1-\frac{\text{Pr}(I=1,S=1\mid V=1,\mathit{X}=\mathit{x})}{\text{Pr}(I=1,S=1\mid V=0,\mathit{X}=\mathit{x}}. \end{gathered}$$ (3)

We denote the right-hand side of Equation (3) as $V{E}_{symp}(\mathit{x})$, which differs from $V{E}_{symp}(\mathit{x},1)$ in classical TNDs by covering healthcare nonseekers.

Case Contact Tracing

Case contact tracing is a common practice of monitoring the spread of diseases in an outbreak setting by testing all contacts of known confirmed cases. Because of the mandatory nature of such contact tracing, testing is unrelated to healthcare-seeking behavior, infection status, or symptoms, but it implies $C=1$ and can be related to covariates and vaccination status. Therefore, we assume

$$\text{Pr}(R=\text{Case contact tracing},T=1\mid 𝒵)=C\times f_c(\mathit{X},V)$$

for some unknown function $f_c$. Given these assumptions, we obtain the DAG for case contact tracing in Figure 4 and have

$$\begin{gathered} 1-\frac{\text{Pr}(I=1\mid V=1,\mathit{X}=\mathit{x},R=\text{Case contact tracing},T=1)}{\text{Pr}(I=1\mid V=0,\mathit{X}=\mathit{x},R=\text{Case contact tracing},T=1)} \\ =1-\frac{\text{Pr}(I=1\mid V=1,\mathit{X}=\mathit{x},C=1)}{\text{Pr}(I=1\mid V=0,\mathit{X}=\mathit{x},C=1}, \end{gathered}$$ (4)

where the left-hand side refers to an estimable quantity from data and the right-hand side represents the conditional VE against infections among contacts of known cases, which we denote as $V{E}_{cct}(\mathit{x})$. Since contacts of known cases may represent a high-risk population of infection,^21,22 statistical inference on this parameter can help estimate VE within this group and potentially guide policy decisions.

FIGURE 4.

The directed acyclic graph for testing due to case contact tracing. Circle nodes stand for random variables, while rectangle nodes indicate conditioning on specific values.

The above procedure can also be adapted to target the conditional VE against symptomatic infections among contacts of known cases by

$$\begin{gathered} 1-\frac{\text{Pr}(I=1,S=1\mid V=1,\mathit{X}=\mathit{x},R=\text{Case contact tracing},T=1)}{\text{Pr}(I=1,S=1\mid V=0,\mathit{X}=\mathit{x},R=\text{Case contact tracing},T=1)} \\ =1-\frac{\text{Pr}(I=1,S=1\mid V=1,\mathit{X}=\mathit{x},C=1)}{\text{Pr}(I=1,S=1\mid V=0,\mathit{X}=\mathit{x},C=1}, \end{gathered}$$ (5)

which we denote as $V{E}_{symp,cct}(\mathit{x})$.

Summary

Overall, Table 1 summarizes the three reasons for testing, their associated assumptions, and the target VE estimands discussed above. Within the DAG framework, each category of reasons for testing targets its own VE estimand. In addition, classical TNDs are limited to healthcare seekers, whereas “mandatory screening” and “case contact tracing” cover both healthcare seekers and nonseekers. Therefore, the odds ratio estimator aggregating across all reasons may target an ambiguous VE estimand, which we discuss in the next section.

TABLE 1.

A Summary of Considered Reasons for Testing with Their Assumptions, Examples, and Target Vaccine Effectiveness (VE) in Test-negative Designs

Reasons for Testing	Main Assumptions	Examples	Target VEs
Symptoms (Figure 2)	Only symptomatic healthcare seekers are tested. Control Exchangeability. Given a symptom, the uptake of testing is independent of true infection status or contact with cases	Fever, fatigue	VE against symptomatic infections for healthcare seekers: $V{E}_{symp}(\mathit{x},1)$
Mandatory screening (Figure 3)	Testing is not directly impacted by infection status, symptom, or healthcare-seeking behavior	Massive tests, travel screening	VE against infection: $VE(\mathit{x})$, or VE against symptomatic infection: $V{E}_{symp}(\mathit{x})$
Case contact tracing (Figure 4)	Testing is not directly impacted by infection status, symptom, or healthcare-seeking behavior	Case contact tracing	VE against infection among case contacts: $V{E}_{cct}(\mathit{x})$, or VE against symptomatic infection among case contacts: $V{E}_{symp,cct}(\mathit{x})$

The VE parameters $V{E}_{symp}(\mathit{x},1)$, $VE(\mathit{x})$, $V{E}_{symp}(\mathit{x})$, $V{E}_{cct}(\mathit{x}$, and $V{E}_{symp,cct}(\mathit{x})$ are defined in Equations (1), (2), (3), (4), and (5), respectively.

In the eAppendix B; https://links.lww.com/EDE/C291, we outline the assumptions required for a causal interpretation of each VE estimand. It is worth noting that $V{E}_{cct}(\mathit{x})$ conditions on a postrandomization variable $C$, and therefore corresponds to the controlled direct effect in the framework of causal mediation analysis,²³ instead of the standard causal VE as in $VE(\mathit{x})$.

Beyond the three testing reasons we discussed, $R$ may encode more categories, for which meaningful VE identification may be challenging. For mathematical completeness in specifying $\text{Pr}(R,T\mid 𝒵)$, we define $R$ = Others as the combination of all other testing reasons and denote $\text{Pr}(R=\text{Others},T=1\mid 𝒵)=f_o(𝒵)$ for some unknown function $f_o$, which imposes no structure on testing due to other reasons. Finally, we require $f_s$, $f_m$, $f_c$, $f_o$ are non-negative functions with summations uniformly upper-bounded by 1 so that $\text{Pr}(R,T\mid 𝒵)$ is well defined.

POTENTIAL BIAS ARISING FROM VARIOUS REASONS FOR TESTING

When a TND involves multiple reasons for testing (including the above-discussed three categories or other reasons), the DAG in Figure 1 can not be simplified. Since the observed data only contain tested individuals, a naive odds ratio estimator with observed data will be biased for VE, for example,

$$VE(\mathit{x})\ne 1-\text{Odds-}{\text{Ratio}}_{I\sim V}(\mathit{X}=\mathit{x},T=1)$$

since the collider bias arising from conditioning on $T=1$ cannot be removed. This bias also exists for estimating $V{E}_{symp}(\mathit{x})$ or $V{E}_{symp}(\mathit{x},1)$ for similar reasons and, additionally, due to the unmeasured confounder $H$. As a numerical example, consider $\text{Pr}(H=1)=0.5$, $\text{Pr}(V=1\mid H)=0.5$, $\log \{\text{Pr}(I=1\mid V,H)\}=-0.5-0.5V-1.5H$, and $\text{Pr}(S=1\mid V,H,I)=0.5$. For simplification, we assumed the independence between $V$ and $H$ and between $S$ and ($V$, $H$, $I$). We consider testing due to symptoms, mandatory screening, or other reasons with

$$\text{Pr}(R=r,T=1\mid 𝒵)=\{\begin{array}{cc}0.2HS\hfill & i\mathrm{f}r=\text{Symptoms},\hfill \\ 0.2V+0.1\hfill & i\mathrm{f}r=\text{Mandatory screening},\hfill \\ 0.3VI\hfill & i\mathrm{f}r=\text{Others}.\hfill \end{array}\phantom{\}}$$

Then, the odds ratio estimator based on aggregated results from tested individuals is −0.02, which substantially underestimates the VE among the whole population, which is 0.39. In addition, we have $VE(\mathit{x})=V{E}_{symp}(\mathit{x})=V{E}_{symp}(\mathit{x},1)=0.39$ in this example that demonstrates that the aggregated odds ratio estimator may not target any VE of interest.

ESTIMATING THE VACCINE EFFECTIVENESS PARAMETERS

Unbiased Estimation by Stratification

For a TND with multiple reasons for testing, we next show how to estimate the identifiable VEs (summarized in Table 1), based on each type of reason $r$ ∈ {Symptoms, Mandatory screening, Case contact tracing}. For this purpose, we model

$$\text{Pr}(I=1\mid V,\mathit{X},R=r,T=1)=h(V,\mathit{X},r),$$

where $h(V,\mathit{X},r)$ is a user-specified learning model. We can assume $h(V,\mathit{X},r)$ follows a parametric model, such as a log-binomial regression model, or a nonparametric model, using kernel smoothing for nonlinear effects of $\mathit{X}$; both approaches are well developed in the literature with software available for estimation. Compared with parametric models, a nonparametric model requires fewer assumptions and can better approximate the truth with the goal of reducing bias, albeit with the disadvantages of requiring larger sample sizes to avoid overfitting and increased variance associated with a slower convergence rate. We provide mathematical details and model fitting in eAppendix C; https://links.lww.com/EDE/ C291. With a user-specified model, we obtain $\widehat{h}(V,\mathit{X},r)$ as the estimate of $h(V,\mathit{X},r)$.

We next construct the following estimators using data from each predefined category of reasons for testing:

$$\begin{gathered} {\widehat{VE}}_{symp}(\mathit{x},1)= \\ 1-\frac{\widehat{h}(V=1,\mathit{X}=\mathit{x},r=\text{Symptoms})\left\{1-\widehat{h}(V=0,\mathit{X}=\mathit{x},r=\text{Symptoms})\right\}}{\widehat{h}(V=0,\mathit{X}=\mathit{x},r=\text{Symptoms})\left\{1-\widehat{h}(V=1,\mathit{X}=\mathit{x},r=\text{Symptoms})\right\}}. \\ \widehat{VE}(\mathit{x})=1-\frac{\widehat{h}(V=1,\mathit{X}=\mathit{x},r=\text{Mandatory screening})}{\widehat{h}(V=0,\mathit{X}=\mathit{x},r=\text{Mandatory screening})}, \\ {\widehat{VE}}_{cct}(\mathit{x})=1-\frac{\widehat{h}(V=1,\mathit{X}=\mathit{x},r=\text{Case contact tracing})}{\widehat{h}(V=0,\mathit{X}=\mathit{x},r=\text{Case contact tracing})}. \end{gathered}$$

For estimating $V{E}_{symp}(\mathit{x})$ and $V{E}_{symp,cct}(\mathit{x})$, we define

$$\begin{gathered} {\widehat{VE}}_{symp}(\mathit{x})=1-\frac{\widehat{g}(V=1,\mathit{X}=\mathit{x},r=\text{Mandatory screening})}{\widehat{g}(V=0,\mathit{X}=\mathit{x},r=\text{Mandatory screening})}, \\ {\widehat{VE}}_{symp,cct}(\mathit{x})=1-\frac{\widehat{g}(V=1,\mathit{X}=\mathit{x},r=\text{Case contact tracing})}{\widehat{g}(V=0,\mathit{X}=\mathit{x},r=\text{Case contact tracing})}, \end{gathered}$$

where $\widehat{g}(V,\mathit{X},r)$ is an estimator for $\text{Pr}(I=1,S=1\mid V,\mathit{X},R=r,T=1)$ and can be constructed in the same way as $\widehat{h}(V,\mathit{X},r)$.

Under our modeling assumptions and regularity conditions listed in eAppendix C; https://links.lww.com/EDE/C291, each estimator converges in probability to its target VE parameter as the sample size $n$ increases, and their normal-approximated 95% confidence intervals based on estimated standard error estimators will have correct coverage in large samples. We also provide variance estimators and technical details in eAppendix C; https://links.lww.com/EDE/C291.

Connecting Vaccine Effectiveness Parameters

In practice, it is common that the above-discussed VE parameters are all distinct. However, with additional assumptions, we are able to connect different VE parameters. For testing due to symptoms, if VE does not vary by the healthcare-seeking behavior¹ and $V$ is independent of $H$ given $\mathit{X}$, then $V{E}_{symp}(\mathit{x},1)=V{E}_{symp}(\mathit{x})$, which motivates us to combine the VE estimation from different testing reasons. For case contact tracing, we will have $V{E}_{cct}(\mathit{x})=VE(\mathit{x})$ and $V{E}_{symp,cct}(\mathit{x})=V{E}_{symp}(\mathit{x})$, if being a contact of known cases does not modify VE and $V$ is independent of $C$ given $\mathit{X}$. Nevertheless, this assumption needs further justification since contacts of known cases might represent a higher level of exposure intensity to the virus, a potential confounder for VE.

For the VEs we have discussed, it is important to note that the VE against symptomatic infection and the VE against general infections are different. Since vaccination may further alleviate symptoms beyond guarding against infection, $V{E}_{symp}(\mathit{x})$ is generally larger than $VE(\mathit{x})$. Therefore, while data from different sources can be combined for a joint analysis, $VE(\mathit{x})$ and $V{E}_{symp}(\mathit{x})$ are estimated separately below.

In fact, whether the above VE parameters are equal is testable from the data. To evaluate whether case contact tracing modifies the VE, we can test the hypothesis $H_0:V{E}_{cct}(\mathit{x})=VE(\mathit{x})$ for any $\mathit{x}$ such that ${\widehat{VE}}_{cct}(\mathit{x})$ and $\widehat{VE}(\mathit{x})$ can be constructed. Since both ${\widehat{VE}}_{cct}(\mathit{x})$ and $\widehat{VE}(\mathit{x})$ are normally distributed (on the root- $n$ scale) and independent in large samples, ${\widehat{VE}}_{cct}(\mathit{x})-\widehat{VE}(\mathit{x})$ also follows a normal distribution in large samples, from which $H_0$ can be rejected if $\mid {\widehat{VE}}_{cct}(\mathit{x})-\widehat{VE}(\mathit{x})\mid$ exceeds the critical value for a given type I error. Following a similar logic, we can also test $H_0:V{E}_{symp}(\mathit{x},1)=V{E}_{symp}(\mathit{x})$ using ${\widehat{VE}}_{symp}(\mathit{x},1)$ and ${\widehat{VE}}_{symp}(\mathit{x})$. This test has greater power if the candidate VEs have substantial differences, the sample sizes are large, and the VE estimates have small variances. However, failing to reject the null does not validate the null hypothesis.

Combining the Stratified Estimators to Improve Precision

In special cases where several VE estimands coincide, we are able to combine the corresponding estimators to yield a more precise estimator. If $V{E}_{cct}(\mathit{x})=VE(\mathit{x})$, we propose a combined estimator for $VE(\mathit{x})$, defined as

$${\widehat{VE}}^{\ast }(\mathit{x})=w_1\widehat{VE}(\mathit{x})+w_2{\widehat{VE}}_{cct}(\mathit{x}),$$

where $w_1$ and $w_2$ are arbitrary positive weights such that $w_1+w_2=1$. The weights can be simply the proportion of samples in each data source, inverse variance-weighting, or other weighting methods from the meta-analysis literature.²⁴ If $V{E}_{cct}(x)\ne VE(x)$, then ${\widehat{VE}}^{\ast }(\mathit{x})$ corresponds to a weighted combination of VE parameters on different populations, which requires additional assumptions for meaningful interpretation.

If $V{E}_{symp}(\mathit{x},1)=V{E}_{symp}(\mathit{x})=V{E}_{symp,cct}(\mathit{x})$, we can construct an estimator combining all three categories of reasons:

$${\widehat{VE}}_{symp}^{\ast }(\mathit{x})=w_1^{\ast }{\widehat{VE}}_{symp}(\mathit{x},1)+w_2^{\ast }{\widehat{VE}}_{symp}(\mathit{x})+w_3^{\ast }{\widehat{VE}}_{symp,cct}(\mathit{x})$$

where $(w_1^{\ast },w_2^{\ast },w_3^{\ast })$ are arbitrary positive weights with $w_1^{\ast }+w_2^{\ast }+w_3^{\ast }=1$, and these weights can be chosen similarly to ($w_1$, $w_2$). When their assumptions hold, combined estimators improve precision by leveraging the additional data. The amount of precision improvement will be greater if we have more data on asymptomatic tests. In the COVID-19 context, asymptomatic tests can often account for a large proportion of data, thus potentially improving precision substantially. This result extends,¹³ which proposed combining stratified estimators for symptoms and case contact tracing (without specifying how to choose the weights), but did not consider mandatory screening or explain the underlying assumptions for estimation validity.

NUMERICAL ILLUSTRATION

Simulation

In eAppendix D; https://links.lww.com/EDE/C291, we provide simulation studies to demonstrate our results numerically. In summary, the simulation studies show that (1) the traditional aggregated odds ratio estimator can be biased for TNDs with multiple reasons for testing, (2) the proposed estimators are unbiased for the target estimands (with the necessary assumptions), (3) leveraging data for asymptomatic reasons for testing can improve precision. Furthermore, supplementary simulations to explore the effect of misclassification of the infection status and missing values in the reasons for testing are also provided in eAppendix D; https://links.lww.com/EDE/C291.

Data Application

We use the EHR from the University of Michigan Health System for a realistic demonstration of our results. Focusing on the period between 1 July and 31 December 2021 (approximately the Delta outbreak), we extract the vaccination history, laboratory results of SARS-CoV-2 tests (positive/negative), COVID-like symptom evaluation (yes/no), and covariates including gender, body mass index, and age. We let $V=1$ if a subject received any vaccine dose. Since reasons for testing are not explicitly recorded in the EHR, we define testing due to symptoms as outpatient testing on symptomatic subjects, and case contact tracing as those with the International Classification of Diseases-10 code Z20.822, meaning “Contact with and (suspected) exposure to COVID-19.” For mandatory screening, while the ICD-10 code Z11.52 matches “screening for COVID-19,” it rarely appeared in the EHR; we therefore exclude this reason for testing from our analysis. It is important to highlight that our definitions of reasons for testing are only for demonstration, which may deviate from the truth and fail in other datasets.

A parametric generalized linear model is used to estimate $V{E}_{symp}(\mathit{x},1)$ based on testing due to symptoms and $V{E}_{symp,cct}(\mathit{x})$ based on case contact tracing. First, we implement the regression model without vaccination-covariate interactions, which results in constant VE estimates across strata. Table 2 summarizes the analysis results and shows a difference in VEs between the two categories, which may reflect the effect modification from healthcare-seeking behavior and contact with known cases. As a result, we do not perform the combination of the stratified estimators. However, we implemented the odds ratio estimator aggregating both reasons and obtained 40 % with standard error 0.04. This estimator shows bias to either VE, which further demonstrates the issue of an unstratified analysis. Furthermore, Figure 5 shows how the VE changes with age through a vaccination-age interaction term in the regression, and the results indicate better VE for older individuals in both categories.

TABLE 2.

Summary of Data Analyses with Logistic Regression

Reason	Target VE	Sample Size	VE Estimate	Standard Error
Symptoms	$V{E}_{symp}(\mathit{x},1)$	1087	0.77	0.05
Case contact tracing	$V{E}_{symp,cct}(\mathit{x})$	22106	0.33	0.05

Since vaccination-covariate interaction terms are not included, the VE estimates are constant across $\mathit{x}$.

FIGURE 5.

VE by age in the data application.

DISCUSSION

As TNDs become increasingly popular for evaluating VE based on observational data, there is a growing demand for valid, robust, and appropriate statistical analyses for TNDs. In this paper, we focus on an emerging challenge that a TND may recruit participants for varying reasons for testing, and we clarify how VE estimates relate to these reasons. Of course, the suggested approach relies on collecting accurate information on the reason for testing at the time of recruitment participation. We note that severe bias can be caused if the reasons for testing are ignored. In summary, while TNDs with heterogeneous testing reasons introduce analytical complexity, they also offer valuable opportunities to obtain more precise and population-specific estimates of VE. Our statistical analysis and proposed methods offer a principled approach to leverage this opportunity and have the potential to inform the design and analysis of future TND studies. Furthermore, our framework also accommodates a meta-analysis scheme, where individual data are first sampled from $\text{Pr}(𝒵\mid R=r,T=1)$ for each reason $r$ and then data across testing reasons are pooled for analysis.

In the design and implementation of TND studies, it is crucial to understand the reasons why individuals undergo testing. When this information is missing, estimates of VE may be biased, and the magnitude and direction of such bias are generally intractable without additional data. Therefore, in settings where asymptomatic testing is expected, we recommend collecting information on reasons for testing at the time of patient enrollment or through the surveillance system. For this purpose, an example questionnaire used by the California Department of Public Health in the United States is available in the Supplementary Material.⁷

Our statistical framework assumes that each subject has a single, accurately recorded reason for testing. In practice, however, individuals may have multiple reasons for testing, or the reason may be unclear or misclassified. While this is less likely to be problematic in a mandatory testing context–such as contact tracing–bias may arise when relying on selfreported information. This issue can be mitigated by adopting survey methods to promote accuracy and honesty,²⁵ and its impact can be further assessed through sensitivity analyses. We leave this topic for future research.

To remove the collider bias, we made a key assumption that the observed covariates include all necessary variables for achieving the conditional independence between $I$ and ($R=r$, $T=1$). Similar to the no-unmeasured confounder assumption in causal inference, evaluating or relaxing this assumption requires additional data or a modified framework. Recently,²⁶ proposed sensitivity analysis methods for TNDs, which can be adopted to evaluate the robustness of our findings when this assumption may be violated.

To derive the VE measures given various reasons for testing, we have mainly focused on a simple setting with a binary vaccination status, a binary outcome, and a binary healthcare-seeking behavior. In the COVID-19 setting, however, the VE may differ by vaccination doses, the specific outcome of interest (infected, diseased, hospitalized, or dead), and the variant of infection. With multilevel $V$ and $I$, our framework can be applied to evaluate the effect of each specific vaccination on each type of outcome separately, provided that vaccination and outcomes are well recorded and the assumptions in Table 1 hold. Furthermore, the healthcare-seeking behavior should reflect the patients’ propensity to seek care, which is unlikely to be fully represented by a binary variable. Therefore, the selection bias from testing may be reduced instead of fully removed by conditioning on a binary healthcare-seeking behavior in classical TNDs.^5,27 On this topic,²⁸ proposed incorporating measures of symptom severity to mitigate selection bias on the odds ratio scale and allow for nonbinary healthcare-seeking behavior. In addition,²⁹ used negative controls to address the bias from healthcare-seeking behavior. A comprehensive investigation of these more complex and realistic scenarios remains an important direction for future research.

Supplementary Material

Supplemental digital content is available through direct URL citations in the HTML and PDF versions of this article (www.epidem.com).