Supplemental Digital Content is available in the text.
Keywords: Causal inference, COVID-19, SARS-CoV-2, Test-negative design, Vaccine effectiveness
Abstract
Observational studies of the effectiveness of vaccines to prevent COVID-19 are needed to inform real-world use. Such studies are now underway amid the ongoing rollout of SARS-CoV-2 vaccines globally. Although traditional case-control and test-negative design studies feature prominently among strategies used to assess vaccine effectiveness, such studies may encounter important threats to validity. Here, we review the theoretical basis for estimation of vaccine direct effects under traditional case-control and test-negative design frameworks, addressing specific natural history parameters of SARS-CoV-2 infection and COVID-19 relevant to these designs. Bias may be introduced by misclassification of cases and controls, particularly when clinical case criteria include common, nonspecific indicators of COVID-19. When using diagnostic assays with high analytical sensitivity for SARS-CoV-2 detection, individuals testing positive may be counted as cases even if their symptoms are due to other causes. The traditional case-control design may be particularly prone to confounding due to associations of vaccination with healthcare-seeking behavior or risk of infection. The test-negative design reduces but may not eliminate this confounding, for instance, if individuals who receive vaccination seek care or testing for less-severe illness. These circumstances indicate the two study designs cannot be applied naively to datasets gathered through public health surveillance or administrative sources. We suggest practical strategies to reduce bias in vaccine effectiveness estimates at the study design and analysis stages.
Nonrandomized studies undertaken after vaccine authorization or licensure provide crucial information about real-world vaccine effectiveness (VE). Such studies may also address questions not answered by clinical trials, such as the duration of protection, the effectiveness of alternative dosing schedules, and protection against emerging variants or within various population subgroups.1,2 Although prospective cohort studies provide an opportunity to compare outcomes among vaccinated and unvaccinated individuals, such studies may be logistically prohibitive due to their large size and the complexity of longitudinal follow-up. Thus, retrospective studies comparing prior vaccination among individuals with known clinical outcomes of disease or no disease (i.e., case-control VE studies) provide an efficient alternative.3
Phase III randomized controlled trials have established efficacy of multiple vaccines against COVID-19,4–6 with others in progress. Observational VE studies have been prioritized by public health7,8 and regulatory9 authorities to inform real-world use. Traditional case-control and test-negative design studies are the primary retrospective frameworks for assessing VE.10 However, these and other observational designs may differ with respect to measures of VE and risks of bias.2
Here, we lay out the theoretical basis for use of the traditional case-control and test-negative design in assessments of vaccine direct effects against SARS-CoV-2 infection and COVID-19. We highlight key assumptions and potential biases underlying VE estimates from the traditional case-control and test-negative design studies and provide practical recommendations to support researchers designing, analyzing, and interpreting these studies.
OVERVIEW OF THE DESIGNS
The traditional case-control and test-negative design derive VE estimates by comparing the odds of prior vaccination among individuals who, at the time of enrollment, are cases (experiencing the clinical endpoint against which VE is to be measured) or noncases (controls). Cases are typically individuals who meet clinical criteria such as the presence of predefined symptoms together with laboratory-confirmed detection of a vaccine-targeted infectious agent.11 Under the traditional case-control, controls are selected from a pool of individuals who are not known to be experiencing the clinical endpoint of interest at the time of enrollment but are members of the same population from which cases are identified. Such individuals may be community controls, selected from among asymptomatic individuals in the community; registry controls, selected from population-based registries; or healthcare controls, selected among patients experiencing an alternate disease unrelated to the pathogen, and vaccine of interest.12
The test-negative design is an alternative to the traditional case-control study that has gained popularity for studies of vaccines against influenza, rotavirus, and other pathogens.13 In the test-negative design, both cases and controls are selected from among individuals who receive diagnostic tests for a pathogen of interest; conventionally, individuals are tested because they experience a clinical syndrome, which may be caused by the vaccine-targeted pathogen or other agents. Cases are those testing positive, and controls are those testing negative, placing a heavy reliance on diagnostic accuracy.2,14,15 Since the test-negative does not require active selection of controls from the community or other settings, it may offer logistical advantages over the traditional case-control design.
Below we lay out the general framework for estimation of vaccine direct effects in traditional case-control and test-negative design studies. We will introduce relevant notation and assumptions and then extend this framework to address observations that may be expected in studies of SARS-CoV-2.
IDENTIFICATION OF VACCINE EFFECTIVENESS UNDER TRADITIONAL CASE-CONTROL AND TEST-NEGATIVE DESIGNS
Consider the instantaneous risk (or prevalence) of a clinical condition of interest to be P (in an unvaccinated population), a value between 0 and 1 (Table 1). Consider that vaccination reduces individuals’ risk of the condition by a factor 
, such that the relative risk of the condition for vaccinated (versus unvaccinated) individuals is 
; we define 
as the vaccine “direct effect” resulting from reduction in biologic susceptibility to the condition of interest due to vaccination.
TABLE 1.
Parameters and Their Definitions
| Parameter | Definition | Range of Values |
|---|---|---|
| P | Prevalence of a vaccine-preventable clinical condition of interest among unvaccinated individuals for the general setting of traditional case–control and test-negative design studies. | 0 to 1 |
 |
Risk ratio for a clinical outcome of interest among counterfactually vaccinated versus unvaccinated individuals; we define  as the risk ratio of shedding the pathogen (owing to differential acquisition among the vaccinated) and  as the risk ratio of disease progression, given infection. Vaccine effectiveness is defined as  in the general case. The combined extent of protection resulting from prevention of shedding and progression is  . |
 corresponds to the scenario of a protective vaccine |
 |
Vaccine uptake, or the proportion of the population being studied who have received vaccination. | 0 to 1 |
 |
Prevalence of the clinical outcome of interest in the population being studied, attributable to causes other than the vaccine-targeted pathogen; ideally, or. To illustrate the impact of this parameter, we consider three values corresponding to differing endpoints. We consider  for the prevalence of any COVID-19-like symptoms among uninfected persons in a community sample.16 We consider  for the prevalence of hospitalization due to causes other than COVID-19, and  for the prevalence of intensive care utilization due to causes other than COVID-19, based the average number of occupied hospital and intensive care beds per capita in the US.17,18
|
0 to 1 |
 |
Prevalence of SARS-CoV-2 shedding among all members of the population being studied. Generally,  throughout the pandemic, although higher values have been reported in certain high-risk populations.16,19–21 To illustrate the impact of this parameter, we plot scenarios with low ( ) and high ( ) values. |
0 to 1 |
 |
Proportion of infected persons experiencing the COVID-19 clinical outcome within the population being studied. To illustrate the impact of this parameter, we consider three values corresponding to differing endpoints. We assume  for any symptoms;  for hospitalization22; and  for ICU admission.23
|
0 to 1 |
 |
Test sensitivity, or the probability of SARS-CoV-2 detection in an infected patient. We further define  as sensitivity for detecting SARS-CoV-2 when the pathogen is not causing symptoms, and  as sensitivity for detecting SARS-CoV-2 when the pathogen is the cause of symptoms. |
0 to 1 |
 |
Probability that individuals who experience the clinical outcome of interest will receive a test. We define  and  for vaccinated and unvaccinated individuals, respectively, to account for potential differences in healthcare access or healthcare seeking associated with vaccination. |
0 to 1 |
 |
Relative risk of SARS-CoV-2 infection among vaccinated (versus unvaccinated) individuals, owing the factors other than vaccination (e.g., differential exposure or susceptibility to infection). |
 corresponds to a scenario of higher risk among those who will receive vaccination;  corresponds to a scenario of lower risk among the vaccinated. |
 |
Proportion of the population that remains uninfected. We define  and  for vaccinated and unvaccinated individuals, respectively, to account for potential differences in risk of infection within these populations, apart from vaccine-conferred immunity. For the case of exponentially distributed infection times, we consider  and  . |
0 to 1 |
As measured, VE is a population-average treatment effect; at two extremes, individuals may experience uniform reductions in risk by a factor of 
or a proportion 
of recipients may experience perfect protection while others experience none. Unless otherwise noted, these differing mechanisms of vaccine action (elsewhere termed “leaky” and “all-or-nothing” protection,10 respectively) do not result in practical differences for measurement of VE under the scenarios we consider.
Defining 
as the proportion of the population receiving vaccine, the probabilities of individuals falling into various case and control categories, given their vaccination status, is laid out in the two-by-two tables in Table 2. Under the traditional case-control design,
TABLE 2.
Two-by-two Tables for Traditional Case-control and Test-negative Design Studies
| Design | Exposure | Outcome | |
|---|---|---|---|
| Traditional case-control | Case | Control | |
| Vaccinated |  |
 |
|
| Unvaccinated |  |
 |
|
| Test-negative design | |||
| Test positive | Test negative | ||
| Vaccinated |  |
 |
|
| Unvaccinated |  |
 |
|
Contents of the table are probabilities of the various listed outcomes, accounting for vaccinated or unvaccinated status. Parameters: 
, proportion of the population vaccinated; 
, risk of the outcome of interest; 
, risk of the negative-control outcome; 
, relative risk of the outcome of interest, given vaccination (see Table 1).
 |
which, for rare outcomes (
), yields an unbiased estimate 
. Alternative design approaches such as incidence density matching of cases and controls24 or inclusive sampling25 may reduce reliance on the rare outcome assumption; here for simplicity, we focus on more general frameworks.
Under the test-negative design, controls instead experience an outcome that is unaffected by vaccination against the pathogen of interest.14,15,26 Defining the risk of this outcome as k, regardless of vaccination (Table 2), and assuming a perfect diagnostic test,
 |
which does not require the rare-disease assumption.
CLASSES OF VACCINE DIRECT EFFECTS
Vaccine-conferred protection may impact multiple aspects of the natural history of infectious disease agents such as SARS-CoV-2, including but not limited to prevention of virus acquisition, reduced viral replication in the upper respiratory mucosa, earlier clearance of infection, and prevention of mild or severe symptoms. The differential effects of SARS-CoV-2 vaccines against infection and disease endpoints remain imperfectly understood27 and may vary by vaccine product and across virus variants.
Typical estimands of traditional case-control and test-negative design studies are VE against symptomatic disease of any severity or VE against severe disease. These are also typical target estimands in individually randomized vaccine efficacy trials.28 Define 
as the relative risk of infection for a vaccinated versus unvaccinated individual, resulting from reduced susceptibility to acquisition of SARS-CoV-2 or accelerated clearance of the pathogen.28 Define 
as the relative risk of COVID-19 (or COVID-19 meeting a particular severity threshold, e.g., hospitalization, intensive care unit admission, or mechanical ventilation) for an infected vaccinated individual versus an infected unvaccinated individual. Protection resulting from prevention of both infection and progression is 
; we assume 
is the desired estimand and describe biases of 
and 
relative to 
. Specialized designs would be required to isolate 
or 
.
OUTCOME MISCLASSIFICATION
Etiologic Detection
Clinical outcomes of interest to researchers may range from laboratory confirmation of infection regardless of symptoms29–31 to laboratory-confirmed mild, moderate, or severe disease manifestations.28 A positive test result may not specifically indicate the etiology of current symptoms, as SARS-CoV-2 genetic material can be shed for prolonged periods after infection.32 Moreover, many acquisitions of SARS-CoV-2 may never cause symptoms.33
Prolonged viral shedding and the presence of asymptomatic infection may introduce misclassification in both traditional case-control and test-negative design studies. For instance, traditional case-control studies enrolling controls selected from the community, from registries, or among patients seeking care for other (e.g., nonrespiratory) diseases, may suffer from misclassification if a proportion of these controls are in fact infected with SARS-CoV-2, but not tested. Inclusion of infected individuals in the control group will diminish the apparent effects of vaccination, provided vaccine prevents infection. Likewise, in both traditional case-control and test-negative design studies, individuals who are shedding SARS-CoV-2 genetic material but experiencing illness due to another cause (e.g., another infection or a noninfectious process with overlapping symptoms such as chronic obstructive pulmonary disease) may be misclassified as cases. Misclassification of asymptomatically infected persons as controls, and of infected symptomatic persons as cases when their illness is not caused by SARS-CoV-2, can be expected to increase with higher prevalence of infection in the community.
To formalize resulting biases, consider separately the risks of SARS-CoV-2 acquisition and disease progression. Defining a as the instantaneous risk (prevalence) of being infected with SARS-CoV-2, at any time, and d as the instantaneous risk of experiencing symptoms, given infection, the risk of symptomatic and asymptomatic SARS-CoV-2 infections are ad and 
, respectively. We extend our contingency tables to accommodate test-positive symptomatic cases, asymptomatic controls, and test-negative symptomatic controls in Table 3; for simplicity, we consider binary disease etiology as SARS-CoV-2 or other factors. Here, the traditional case-control yields
TABLE 3.
Contingency Tables for Traditional Case-control and Test-negative Design Studies With Outcome Misclassification
| Exposure | Outcome | ||
|---|---|---|---|
| Test-positive Case (Symptomatic) | Community Control (Asymptomatic) | Test-negative Control (Symptomatic) | |
| Vaccinated |  |
 |
 |
| Unvaccinated |  |
 |
 |
Contents of the table are probabilities of the various listed outcomes, accounting for vaccinated or unvaccinated status. Parameters: 
, proportion of the population vaccinated; a, prevalence of SARS-CoV-2 infection; d, risk of COVID-19 among those with SARS-CoV-2 infection; k, risk of symptoms due to causes other than SARS-CoV-2 infection; 
, relative risk of SARS-CoV-2, given vaccination; 
, relative risk of COVID-19, among those with SARS-CoV-2 infection, given vaccination (Table 1).
 |
which remains subject to bias owing to the risk of symptoms due to causes other than SARS-CoV-2 (k) and the prevalence of SARS-CoV-2 infection (a). However, these biases may be ignorable under several conditions. If prevalence of other conditions is negligible relative to the risk of COVID-19 for an infected individual (
, if endpoints are chosen with high specificity for COVID-19), the traditional case-control design yields
 |
Such a design may underestimate 
if prevalence of SARS-CoV-2 infection (a) is high owing to inclusion of infected asymptomatic individuals in the control group. If prevalence of SARS-CoV-2 infection in the community is low or negligible (
), 
.
Comparing symptomatic individuals who test positive and negative for infection, the test-negative design yields
 |
The same conditions by which bias can be reduced under the case-control design reduce bias under the test-negative design. With 
,
 |
and with 
, 
We illustrate the quantitative extent of bias under the traditional case–control and test-negative design in eFigures 1 and 2; http://links.lww.com/EDE/B808. Under most realistic situations, misclassification bias is low or negligible for both designs. Although relatively insensitive to changes in prevalence of infection within ranges that have been reported for SARS-CoV-2, bias increases if the association of clinical case-defining symptoms with infection is weak (such that 
is false), potentially leading to substantial underestimation of 
.
Choice of Diagnostic Assays
Multiple assays are available for SARS-CoV-2 detection, and use of assays with differing performance characteristics may influence the extent of biases identified above. Nucleic acid amplification assays such as polymerase chain reaction, for instance, have higher analytic sensitivity (i.e., likelihood of positive test for sample containing viral material at the minimum detectable concentration for which assay is designed), potentially leading to detection of SARS-CoV-2 genetic material among individuals with low levels of virus-shedding. Antigen detection tests, with lower analytic sensitivity, may in contrast yield negative results for individuals with low virus-shedding that remains detectable by molecular testing.34 In comparison to mild illness, both assays may have lower sensitivity for severe disease, which can be delayed in presentation, occurring often during the second week after infection onset when viral shedding is lower.16
Define 
and 
as the sensitivity of an assay for detecting SARS-CoV-2 among individuals experiencing COVID-19 and those not experiencing symptoms attributable to SARS-CoV-2, respectively. We limit our consideration to differences in test sensitivity, considering specificity to encompass the related issue of nonetiologic SARS-CoV-2 detections; under field conditions, antigen, and molecular tests have been found to have similar analytical specificity.34,35 The limiting case of an assay which perfectly rules out SARS-CoV-2 detections that are not causing symptoms (
) yields the contingency table presented in Table 4 for case and control groups (eTable 1; http://links.lww.com/EDE/B808 extends this for nonzero values of 
).
TABLE 4.
Contingency Tables for Traditional Case-control and Test-negative Design Studies With Exclusion of SARS-CoV-2 Shedding From COVID-19
| Exposure | Outcome | ||
|---|---|---|---|
| Test-positive Case (Symptomatic) | Community Control (Asymptomatic) | Test-negative Control (Symptomatic) | |
| Vaccinated |  |
 |
 |
| Unvaccinated |  |
 |
 |
Contents of the table are probabilities of the various listed outcomes, accounting for vaccinated or unvaccinated status. Parameters: 
, proportion of the population vaccinated; a, prevalence of SARS-CoV-2 infection; d, risk of COVID-19 among those with SARS-CoV-2 infection; k, risk of symptoms due to causes other than SARS-CoV-2 infection; 
, relative risk of SARS-CoV-2, given vaccination; 
, relative risk of COVID-19, among those with SARS-CoV-2 infection, given vaccination; 
, sensitivity for SARS-CoV-2 detection among individuals experiencing COVID-19 (see Table 1). We present a more general contingency table for nonzero values of 
(test sensitivity for SARS-CoV-2 infection among those not experiencing COVID-19) in eTable 1; http://links.lww.com/EDE/B808.
With 
, the traditional case-control yields
 |
removing the reliance on the assumption 
, and allowing 
if overall prevalence of infection is low (
). Under the test-negative design,
 |
Here, we obtain an unbiased estimate 
only with high sensitivity of the assay for individuals experiencing COVID-19 (
). Lower values of 
result in greater bias (eFigures 3 and 4; http://links.lww.com/EDE/B808). Nonetheless, under a scenario where 
is high and asymptomatic SARS-CoV-2 detections are minimal (
; eFigures 3 and 4; http://links.lww.com/EDE/B808), the extent of bias remains lower than what arises with perfect detection of SARS-CoV-2 in both asymptomatic infection and COVID-19 (as plotted in eFigures 1 and 2; http://links.lww.com/EDE/B808), particularly for vaccines that primarily prevent infection (eFigures 3 and 4; http://links.lww.com/EDE/B808).
We note that no available assay enables perfect differentiation of COVID-19 from nonetiologic detections of SARS-CoV-2; statistical corrections for imperfect test sensitivity and specificity have been proposed to adjust final VE estimates.36 Further complicating the selection of assays, available data suggest similar viral loads and nucleic acid abundance among individuals with or without symptoms at similar stages of infection.37,38 However, onset of COVID-19 upper respiratory tract symptoms typically coincides with peak viral shedding. Thus, individuals seeking care for new-onset COVID-19 may generally have higher viral load than individuals experiencing symptoms owing to other causes, who may be at later stages of SARS-CoV-2 shedding or molecular detection.39 This may drive an artefactual association of higher SARS-CoV-2 shedding with COVID-19.
HEALTHCARE-SEEKING BEHAVIOR
Differential healthcare-seeking behaviors of individuals who can access and receive vaccines, versus others, may be another source of bias. The test-negative design aims to limit such bias by restricting enrollment to individuals who seek care and receive diagnostic testing for the same or similar clinical indications. In traditional case-control studies, there may be uncertainty about whether controls were truly disease-free, as some may not have sought care given symptoms. This may be particularly true if studies select controls from registries without direct outreach.
We may quantify the resulting bias by considering the number of controls who are misclassified when the probability of seeking care, given symptoms, is 
among the vaccinated and 
among the unvaccinated. Defining eligible controls as individuals who do not become ascertained as cases (Table 5):
TABLE 5.
Contingency Table for Traditional Case-control and Test-negative Design Studies With Differential Healthcare-seeking Among the Vaccinated and Unvaccinated
| Exposure | Outcome | ||
|---|---|---|---|
| Test-positive Case (Symptomatic, Healthcare Sought) | Community Control (Healthcare Not Sought) | Test-negative Control (Symptomatic, Healthcare Sought) | |
| Vaccinated |  |
 |
 |
| Unvaccinated |  |
 |
 |
Contents of the table are probabilities of the various listed outcomes, accounting for vaccinated or unvaccinated status. Parameters: 
, proportion of the population vaccinated; a, prevalence of SARS-CoV-2 infection; d, risk of COVID-19 among those with SARS-CoV-2 infection; k, risk of symptoms due to causes other than SARS-CoV-2 infection; 
, relative risk of SARS-CoV-2, given vaccination; 
, relative risk of COVID-19, among those with SARS-CoV-2 infection, given vaccination; 
, probability of testing for symptomatic vaccinated individuals; 
, probability of testing for symptomatic unvaccinated individuals (Table 1).
 |
The resulting bias generally leads to underestimation of true protection (
) when the probability of seeking treatment is higher among vaccinated than unvaccinated persons, given the same status of being infected or symptomatic, owing to the disproportionately higher likelihood for vaccinated individuals to be ascertained as cases (eFigures 5 and 6; http://links.lww.com/EDE/B808). Studies aiming to mitigate this bias by enrolling health facility controls may still encounter difficulty. If vaccination is associated with higher rates of healthcare seeking, prevalence of vaccination among controls seeking care for less-severe conditions may be inflated, leading to overestimation of 
.
Applying the same formulation under the test-negative design, the terms 
and 
cancel out (Table 5). Nonetheless, in practical contexts, test-negative studies may enroll individuals across a spectrum of clinical severity. This may cause bias to persist if the probability of treatment-seeking differs for vaccinated and unvaccinated individuals with relatively less severe or more severe illness.26 These considerations motivate approaches such as matching of cases and test-negative controls (or cases and alternative-disease controls under the traditional case-control) on clinical severity.
Prevalence of other causes of illness leading to care-seeking (e.g., among alternative-disease controls under the traditional case-control and among test-negative controls under the test-negative design) may vary considerably over time. If risk of disease due to these other causes is also associated with likelihood of vaccination, further bias may arise. Even under normal circumstances, disease etiology among controls can change within a season or from year to year. This variation is likely to be extreme over the time horizon that SARS-CoV-2 vaccines are rolled out, as nonpharmaceutical interventions targeting SARS-CoV-2 have affected circulation of other pathogens.40,41 As nonpharmaceutical interventions are rolled back, other causes of healthcare-seeking may surge. Since many infections causing respiratory symptoms (e.g., influenza, pneumococcus) are vaccine preventable, controls with these infections may be less likely to have sought vaccine. Accordingly, the ratio 
may change over time.
A CAUTIONARY NOTE
Testing for SARS-CoV-2 is commonly undertaken for both clinical diagnostic and public health purposes. Thus, data on SARS-CoV-2 test results may encompass individuals tested because of clinical symptoms as well as those tested due to contact with a known or suspected COVID-19 case, for purposes of screening before work or medical procedures, or for other indications. Stratifying analyses to address fairly homogeneous populations or to individuals who receive SARS-CoV-2 testing with the same frequency or for the same indications, may be of considerable importance.
Variation in the likelihood of seeking testing is expected to be greatest for cases of mild illness or among persons without symptoms. There may be substantial associations to test-seeking with individuals’ healthcare utilization, including the propensity to be vaccinated. In the most extreme case, populations targeted for vaccination may also be populations targeted for frequent testing, such as healthcare providers, residents of long-term care facilities, and essential workers. Infections in these groups will be more likely to be detected relative to infections within groups without similar access to testing, who may also be less likely to be vaccinated. Here 
, potentially leading to substantial underestimation of VE. Similar bias may persist when vaccines are made more widely available if individuals with the greatest propensity to seek healthcare, given the same clinical presentation, including SARS-CoV-2 testing, are also more likely to be vaccinated.
Thus, the test-negative design cannot be applied naively to SARS-CoV-2 testing datasets gathered through public health surveillance or administrative data sources. To adequately correct for potential differences in vaccinated and unvaccinated individuals’ likelihood of being tested, designs or analyses should compare individuals being tested for the same indications (e.g., fulfilling particular clinical criteria, known high-risk exposure to SARS-CoV-2, or routine screening), for instance by matching or stratifying on vaccine eligibility criteria. Bias may be further reduced in studies considering severe disease endpoints for which both vaccinated and unvaccinated individuals would be unlikely to defer treatment, regardless of their propensity to seek healthcare for less severe illness.
ASSOCIATION OF VACCINE UPTAKE WITH DIFFERENTIAL RISK OF INFECTION
Confounding in Prevalence of Infection and Immunity
Limited vaccine supplies are being targeted initially at groups most likely to be exposed to SARS-CoV-2 as well as those at greatest risk of severe COVID-19, if infected. Even when vaccines are more widely available, uptake of vaccination may be highest among individuals who adhered most strongly to nonpharmaceutical interventions. Accordingly, exposure to infection and risk of progressing to severe disease will not be equal between vaccinated and unvaccinated groups. This differential extends not only to rates of new infection but also to the prevalence of prior natural infection and immunity. As available evidence suggests naturally acquired immunity is strongly (~90%) protective against reinfection, there may be substantial differences in risk among vaccinated and unvaccinated individuals owing to factors other than vaccine-derived protection.42,43
We extend the contingency tables laid out previously (Table 6) to incorporate 
, the relative risk of SARS-CoV-2 infection among vaccinated (vs. unvaccinated) individuals due to factors other than vaccination, as well as 
and 
, the proportions of the vaccinated and unvaccinated populations, respectively, that remain uninfected. For simplicity, we assume near-total protection against reinfection conferred by naturally acquired immunity. Under the two designs,
TABLE 6.
Contingency Table for Traditional Case-control and Test-negative Design Studies With Differential Risk of Current and Prior Infection Among the Vaccinated and Unvaccinated
| Exposure | Outcome | ||
|---|---|---|---|
| Test-positive Case (Symptomatic) | Community Control (Asymptomatic) | Test-negative Control (Symptomatic) | |
| Vaccinated |  |
 |
 |
| Unvaccinated |  |
 |
 |
Contents of the table are probabilities of the various listed outcomes, accounting for vaccinated or unvaccinated status. Parameters: 
, proportion of the population vaccinated; a, prevalence of SARS-CoV-2 infection; d, risk of COVID-19 among those with SARS-CoV-2 infection; k, risk of symptoms due to causes other than SARS-CoV-2 infection; 
, relative risk of SARS-CoV-2, given vaccination; 
, relative risk of COVID-19, among those with SARS-CoV-2 infection, given vaccination; 
, relative risk of exposure or infection among vaccinated versus unvaccinated individuals attributable to factors other than vaccine-conferred immunity; 
, proportion of vaccinated individuals who remain uninfected; 
, proportion of unvaccinated individuals who remain uninfected (Table 1).
 |
and
 |
which, with low prevalence of infection (
) and low risk of disease due to other causes (
), yield
 |
Here, differential risk of SARS-CoV-2 infection and prevalence of immunity among vaccinated and unvaccinated individuals lead to counteracting biases. With higher risk among individuals who receive vaccination (
), we may expect higher prevalence of naturally acquired infection in this group and 
; bias is offset fully if 
.
We plot conditions under which bias is driven predominantly by differential exposure to infection or by differential naturally acquired immunity in figure. Considering the context of VE studies undertaken shortly after vaccine implementation, we may assume individuals’ time at risk is contributed primarily in the period before vaccination; with exponentially distributed infection times, and a cumulative force of infection 
exerted over the period at risk, we may define the remaining uninfected populations 
and 
. Bias is fully offset with 
, yielding 
(Fig.). Thus, the prevalence of preexisting naturally acquired immunity needed to offset bias resulting from differential exposure is high, at 33% and 56% for 
and 
, respectively.
FIGURE.
Estimated vaccine effectiveness under the traditional case-control design and test-negative design with differential exposure to infection and prevalence of naturally acquired immunity among vaccinated and unvaccinated persons. We illustrate expected estimates of the odds ratio of vaccination given case versus control (or test-negative) status, under the limiting case of 
and 
, under scenarios where vaccination occurs among individuals with higher (left panel: 
, solid lines; 
, dashed lines) or lower (center panel: 
, solid lines; 
, dashed lines) risk of exposure to SARS-CoV-2. Colors correspond to differing proportions of the unvaccinated population remaining susceptible to infection (
; we define the proportion of the vaccinated population remaining uninfected as 
for given values of 
. Red diagonal lines across the left and center panels illustrate the true effect. The right panel illustrates threshold values of 
, at which the effects of differential exposure to SARS-CoV-2 and naturally acquired immunity among vaccinated versus unvaccinated persons cancel out, resulting in a change in the direction of bias.
Detailed risk factor data are likely necessary to address confounding associated with differential risk of infection among individuals receiving and those not receiving vaccination. Although certain occupations (e.g., healthcare or other essential workers) may be associated with heightened risk of exposure relative to the general public, substantial variation in risk may exist within these categories owing to differences in the specific tasks conducted and availability of as well as adherence to infection control measures. Individuals may modify aspects of their behavior following vaccine receipt, further altering differences in risk of infection among vaccine recipients and nonrecipients. Collection of questionnaire items addressing risk behaviors and exposures is thus of importance to mitigate confounding.
Temporal Considerations
Tiered rollout of vaccination while disease incidence changes concurrently will cause the bias we have described above to vary over time. Thus, in addition to differential exposure or susceptibility, cohorts with earlier or later access to and uptake of vaccines may be exposed to differing risk of infection at equivalent time points after vaccination. Matching, stratification, or statistical adjustment may be necessary to correct for temporal confounding of this nature, and to avoid spurious signals of time-varying VE. Although we have focused here on studies undertaken shortly after vaccine rollout, prior work has considered differential accrual of natural immunity among vaccinated and unvaccinated individuals over longer periods of time.3,26,44,45 For vaccines conferring “leaky” protection, traditional case-control and test-negative design studies may underestimate true protection owing to higher rates of acquisition of naturally acquired immunity among unvaccinated individuals. This bias does not arise, however, for vaccines conferring “all-or-nothing” protection.3,26
DISCUSSION
Our analysis provides several practical insights to inform upcoming traditional case-control and test-negative design studies of SARS-CoV-2 vaccines. Although we identify conditions and reasonable assumptions under which both designs can supply reliable VE estimates, several threats to validity should be taken into account in the design, analysis, and interpretation of traditional case-control and test-negative design studies. Below we offer considerations for researchers undertaking studies of either design to investigate vaccines against SARS-CoV-2:
Use of Severe COVID-19 Clinical Endpoints
Use of severe disease endpoints minimizes risk of misclassification and healthcare-seeking bias. To allow sufficient power, studies may need to enroll patients across multiple healthcare facilities, underscoring the need for standardized clinical criteria and case definitions. Studies addressing VE against common or less-severe clinical endpoints may benefit from using extended analytic methods that have been proposed previously for estimation of VE against diseases with nonspecific diagnostic criteria.11,46,47 As these analysis methods must generally account for the strength of association between SARS-CoV-2 infection and clinical symptoms, enrollment and testing of control groups without symptoms related to SARS-CoV-2 infection may be of value for researchers studying VE against such endpoints.
Use of Molecular or Antigen Tests
Studies using SARS-CoV-2 detection assays with the highest analytic sensitivity may encounter the greatest risk of misclassifying symptomatic individuals at the end of their course of infection as cases, including when SARS-CoV-2 is not the cause of symptoms. This could be more problematic when prevalence of SARS-CoV-2 infection is high and outcomes of interest include nonsevere or nonspecific COVID-19 clinical manifestations, such as in outpatient settings. However, trade-offs associated with risk of bias must be balanced against assay specificity and test performance under varied prevalence and clinical settings. Where molecular assays with high analytical sensitivity are preferred for clinical care provision, researchers may consider analyzing quantitative cycle threshold data or collecting a second specimen for antigen-based testing.
Collecting Detailed Information on Clinical Severity and Testing Indications
Reliance on clinic-based testing under both designs makes it difficult to identify and correct for differences in the likelihood for vaccinated and unvaccinated individuals, given infection and symptoms, to be ascertained as cases. Collection of standardized data on specific clinical symptoms (and their duration) among both cases and controls may reduce the risk of bias related to vaccine-associated differences in healthcare access or healthcare seeking. This strategy further provides an opportunity to identify and exclude, if applicable, registry-based or community controls who were ill but did not seek care. If testing is conducted for multiple pathogens, researchers may also investigate bias by assessing VE against other illnesses not prevented by SARS-CoV-2 vaccination as a “sham” analysis to indicate risk of bias.48,49
Background Immunity
Prevalent naturally acquired immunity may complicate the interpretation of results under both the traditional case-control and TND, potentially contributing to lower risk of infection and disease among individuals with the greatest risk of SARS-CoV-2 exposure. As initial vaccine doses will be prioritized to groups with high infection risk, individuals eligible to receive vaccine at the earliest opportunity may have higher prevalence of preexisting naturally acquired immunity than those in lower-priority tiers. Studies may mitigate this bias, in part, by comparing individuals within the same prioritization groups, by collecting detailed risk factor data to control for variation in exposure, and through use of serological assays (if available) that differentiate vaccine-derived or naturally induced responses (e.g., to nonvaccine antigens).
Supplementary Material
Footnotes
This work was supported by grants R01-AI14812701 from the National Institute for Allergy and Infectious Diseases to N.P.J. and J.A.L., and R01-AI139761 from the National Institute for Allergy and Infectious Diseases to N.E.D.
Code for replication is available from the corresponding author via https://github.com/joelewnard/covidTND.
J.A.L. has received grants and consulting fees from Pfizer, Inc., unrelated to this research. The remaining authors report no conflicts of interest.
The findings and conclusions in this report are those of the authors and do not necessarily represent the official position of the Centers for Disease Control and Prevention.
Supplemental digital content is available through direct URL citations in the HTML and PDF versions of this article (www.epidem.com).
REFERENCES
- 1.Hanquet G, Valenciano M, Simondon F, Moren A. Vaccine effects and impact of vaccination programmes in post-licensure studies. Vaccine. 2013;31:5634–5642. [DOI] [PubMed] [Google Scholar]
- 2.Lipsitch M, Jha A, Simonsen L. Observational studies and the difficult quest for causality: lessons from vaccine effectiveness and impact studies. Int J Epidemiol. 2016;45:2060–2074. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Smith PG, Rodrigues LC, Fine PE. Assessment of the protective efficacy of vaccines against common diseases using case-control and cohort studies. Int J Epidemiol. 1984;13:87–93. [DOI] [PubMed] [Google Scholar]
- 4.Polack FP, Thomas SJ, Kitchin N, et al. ; C4591001 Clinical Trial Group. Safety and efficacy of the BNT162b2 mRNA Covid-19 vaccine. N Engl J Med. 2020;383:2603–2615. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Spector SA, Rouphael N, Creech CB, et al. Efficacy and safety of the mRNA-1273 SARS-CoV-2 vaccine. N Engl J Med. 2020;384:403–416. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Voysey M, Clemens SAC, Madhi SA, et al. ; Oxford COVID Vaccine Trial Group. Safety and efficacy of the ChAdOx1 nCoV-19 vaccine (AZD1222) against SARS-CoV-2: an interim analysis of four randomised controlled trials in Brazil, South Africa, and the UK. Lancet. 2021;397:99–111. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Patel MM, Jackson ML, Ferdinands J. Postlicensure evaluation of COVID-19 vaccines. JAMA. 2020;324:1939–1940. [DOI] [PubMed] [Google Scholar]
- 8.Hodgson SH, Mansatta K, Mallett G, Harris V, Emary KRW, Pollard AJ. What defines an efficacious COVID-19 vaccine? A review of the challenges assessing the clinical efficacy of vaccines against SARS-CoV-2. Lancet Infect Dis. 2020;21:e26–e35. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.FDA Center for Biologics Evaluation and Research. Development and licensure of vaccines to prevent COVID-19: Guidance for industry. 2020. Available at: https://www.fda.gov/regulatory-information/search-fda-guidance-documents/development-and-licensure-vaccines-prevent-covid-19. Accessed 1 February 2021.
- 10.Halloran ME, Longini IM, Jr, Struchiner CJ. Design and interpretation of vaccine field studies. Epidemiol Rev. 1999;21:73–88. [DOI] [PubMed] [Google Scholar]
- 11.Lewnard JA. Uses of pathogen detection data to estimate vaccine direct effects in case-control studies. J R Soc Interface. 2020;17:20200161. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Wacholder S, Silverman DT, McLaughlin JK, Mandel JS. Selection of controls in case-control studies. II. Types of controls. Am J Epidemiol. 1992;135:1029–1041. [DOI] [PubMed] [Google Scholar]
- 13.Chua H, Feng S, Lewnard JA, et al. The use of test-negative controls to monitor vaccine effectiveness: a systematic review of methodology. Epidemiology. 2020;31:43–64. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Jackson ML, Nelson JC. The test-negative design for estimating influenza vaccine effectiveness. Vaccine. 2013;31:2165–2168. [DOI] [PubMed] [Google Scholar]
- 15.Foppa IM, Haber M, Ferdinands JM, Shay DK. The case test-negative design for studies of the effectiveness of influenza vaccine. Vaccine. 2013;31:3104–3109. [DOI] [PubMed] [Google Scholar]
- 16.Lewnard JA, Mora AM, Nkwocha O, et al. Prevalence and clinical profile of SARS-CoV-2 infection among farmworkers in Monterey County, California: June-November, 2020. medRxiv 2020. [Google Scholar]
- 17.National Center for Health Statistics. Hospitals, beds, and occupancy rates, by type of ownership and size of hospital: United States, selected years 1975–2015. 2017. Available at: https://www.cdc.gov/nchs/data/hus/2017/089.pdf. Accessed 1 February 2021.
- 18.Halpern NA, Pastores SM. Critical care medicine beds, use, occupancy, and costs in the United States: a methodological review. Crit Care Med. 2015;43:2452–2459. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Menachemi N, Yiannoutsos CT, Dixon BE, et al. Population point prevalence of SARS-CoV-2 infection based on a statewide random sample — Indiana, April 25–29, 2020. MMWR. 2020;69:960–964. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.Riley S, Ainslie KEC, Eales O, et al. REACT-1 round 6 updated report high prevalence of SARS-CoV-2 swab positivity with reduced rate of growth in England at the start of November 2020. medRxiv 2020. doi: 10.1101/2020.11.18.20233932
- 21.Chamie G, Marquez C, Crawford E, et al. Community transmission of severe acute respiratory syndrome coronavirus 2 disproportionately affects the Latinx population during shelter-in-place in San Francisco [published online ahead of print August 21, 2020]. Clin Infect Dis. doi:10.1093/cid/ciaa1234 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.Salje H, Tran Kiem C, Lefrancq N, et al. Estimating the burden of SARS-CoV-2 in France. Science. 2020;369:208–211. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23.Lewnard JA, Liu VX, Jackson ML, et al. Incidence, clinical outcomes, and transmission dynamics of severe coronavirus disease 2019 in California and Washington, United States: prospective cohort study. BMJ. 2020;369:m1923. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 24.Greenland S, Thomas DC. On the need for the rare disease assumption in case-control studies. Am J Epidemiol. 1982;116:547–553. [DOI] [PubMed] [Google Scholar]
- 25.Rodrigues LC, Smith PG. Use of the case-control approach in vaccine evaluation: efficacy and adverse effects. Epidemiol Rev. 1999;21:56–72. [DOI] [PubMed] [Google Scholar]
- 26.Lewnard JA, Tedijanto C, Cowling BJ, Lipsitch M. Measurement of vaccine direct effects under the test-negative design. Am J Epidemiol. 2018;187:2686–2697. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 27.Lipsitch M, Dean NE. Understanding COVID-19 vaccine efficacy. Science. 2020;370:763–765. [DOI] [PubMed] [Google Scholar]
- 28.Halloran ME, Struchiner CJ, Longini IM., Jr. Study designs for evaluating different efficacy and effectiveness aspects of vaccines. Am J Epidemiol. 1997;146:789–803. [DOI] [PubMed] [Google Scholar]
- 29.Harper DM, Franco EL, Wheeler C, et al. ; GlaxoSmithKline HPV Vaccine Study Group. Efficacy of a bivalent L1 virus-like particle vaccine in prevention of infection with human papillomavirus types 16 and 18 in young women: a randomised controlled trial. Lancet. 2004;364:1757–1765. [DOI] [PubMed] [Google Scholar]
- 30.Alonso PL, Sacarlal J, Aponte JJ, et al. Efficacy of the RTS,S/AS02A vaccine against Plasmodium falciparum infection and disease in young African children: randomised controlled trial. Lancet. 2004;364:1411–1420. [DOI] [PubMed] [Google Scholar]
- 31.Collins AM, Wright AD, Mitsi E, et al. First human challenge testing of a pneumococcal vaccine. Double-blind randomized controlled trial. Am J Respir Crit Care Med. 2015;192:853–858. [DOI] [PubMed] [Google Scholar]
- 32.Cevik M, Tate M, Lloyd O, Maraolo AE, Schafers J, Ho A. SARS-CoV-2, SARS-CoV, and MERS-CoV viral load dynamics, duration of viral shedding, and infectiousness: a systematic review and meta-analysis. Lancet Microbe. 2020;2:e13–e22. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 33.Buitrago-Garcia D, Egli-Gany D, Counotte MJ, et al. Occurrence and transmission potential of asymptomatic and presymptomatic SARS-CoV-2 infections: a living systematic review and meta-analysis. PLoS Med. 2020;17:e1003346. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 34.Pilarowski G, Marquez C, Rubio L, et al. Field performance and public health response using the BinaxNOW TM Rapid SARS-CoV-2 antigen detection assay during community-based testing [published online ahead of print December 26, 2020]. Clin Infect Dis. doi: 10.1093/cid/ciaa1890 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 35.Albert E, Torres I, Bueno F, et al. Field evaluation of a rapid antigen test (Panbio™ COVID-19 Ag Rapid Test Device) for COVID-19 diagnosis in primary healthcare centres. Clin Microbiol Infect. 2021;27:472.e7–472.e10. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 36.Endo A, Funk V, Kucharski A. Bias correction methods for test-negative designs in the presence of misclassification [published online ahead of print July 20, 2019]. Epidemiol Infect. doi: 10.1101/19002691 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 37.Van Vinh Chau N, Lam VT, Dung NT, et al. ; Oxford University Clinical Research Unit COVID-19 Research Group. The natural history and transmission potential of asymptomatic severe acute respiratory syndrome Coronavirus 2 infection. Clin Infect Dis. 2020;71:2679–2687. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 38.Long QX, Liu BZ, Deng HJ, et al. Antibody responses to SARS-CoV-2 in patients with COVID-19. Nat Med. 2020;26:845–848. [DOI] [PubMed] [Google Scholar]
- 39.Zou L, Ruan F, Huang M, et al. SARS-CoV-2 viral load in upper respiratory specimens of infected patients. N Engl J Med. 2020;382:1177–1179. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 40.Cowling BJ, Ali ST, Ng TWY, et al. Impact assessment of non-pharmaceutical interventions against coronavirus disease 2019 and influenza in Hong Kong: an observational study. Lancet Public Heal. 2020;5:e279–e288. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 41.Solomon DA, Sherman AC, Kanjilal S. Influenza in the COVID-19 Era. JAMA. 2020;324:1342–1343. [DOI] [PubMed] [Google Scholar]
- 42.Marsden BD, Cox S, James T, et al. Antibody status and incidence of SARS-CoV-2 infection in health care workers. N Engl J Med. 2020;384:533–540. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 43.Addetia A, Crawford KHD, Dingens A, et al. Neutralizing antibodies correlate with protection from SARS-CoV-2 in humans during a fishery vessel outbreak with a high attack rate. J Clin Microbiol. 2020;58:e02107–e02120. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 44.Farrington CP. The measurement and interpretation of age-specific vaccine efficacy. Int J Epidemiol. 1992;21:1014–1020. [DOI] [PubMed] [Google Scholar]
- 45.Lopman BA, Pitzer VE. Waxing understanding of waning immunity. J Infect Dis. 2018;217:851–853. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 46.De Smedt T, Merrall E, Macina D, Perez-Vilar S, Andrews N, Bollaerts K. Bias due to differential and non-differential disease- and exposure misclassification in studies of vaccine effectiveness. PLoS One. 2018;13:e0199180. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 47.Hasegawa RB, Small DS. Estimating malaria vaccine efficacy in the absence of a gold standard case definition: mendelian factorial design [published online ahead of print December 14, 2020]. J Am Stat Assoc. doi: 10.1080/01621459.2020.1863222 [Google Scholar]
- 48.Lipsitch M, Tchetgen Tchetgen E, Cohen T. Negative controls: a tool for detecting confounding and bias in observational studies. Epidemiology. 2010;21:383–388. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 49.Lewnard JA, Lo NC, Arinaminpathy N, Frost I, Laxminarayan R. Childhood vaccines and antibiotic use in low- and middle-income countries. Nature. 2020;581:94–99. [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.