Discussion on “Estimating vaccine efficacy over time after a randomized study is unblinded” by Anastasios A. Tsiatis and Marie Davidian

Drs. Tsiatis and Davidian (TD) have produced a highly innovative and technically impressive approach to estimating vaccine efficacy (VE) over time after a randomized study is unblinded. The paper was motivated by the trial of the Moderna vaccine against SARS‐CoV‐2 that causes the disease COVID‐19. It has been a great pleasure to read this paper. They cleverly use the potential outcomes framework to estimate potential waning of VE over time and VE at any post‐vaccination time. For those of us in the field of evaluating VE and infectious diseases in general, it is eye opening to see two such talented statistical colleagues throw new light on what seemed to be a settled topic before we even get to the waning. The paper has much worthy of discussion, including the careful time line in calendar time and individual time, the statistical framework, dealing with potential confounding, estimation procedures, and software. Here I raise just five points.

First, the first expression for the infection rate $p(t)c(t)\pi (t)$, where $p(t)$ is the prevalence, $c(t)$ the contact rate, and $\pi (t)$ the transmission probability at time t is a conceptual mechanistic model that we previously considered underlying the hazard rate of infection for infectious diseases. This expression is very familiar in the world of infectious diseases. However, TD chose to express each of the three components as individual potential quantities, moving each into an individual counterfactual framework yielding a conceptually new approach and a new result that I consider in my next point.

It is innovative to set up these common expressions in infectious diseases as individual counterfactuals. An important contribution is that they define the infection rate in the study population as an expectation over the individual infection rates. This is different from the standard result. The relative population‐level infection rate at t becomes the ratio of two expectations (Equation 2). One fine point, are the potential contact rates potential outcomes as they call them? What are they if not?

Second, putting this in the individual counterfactual framework leads to the, for me, initially surprising result that the population hazard rates as in Equation (9) are not equivalent to population‐level infection rates. They show that the population‐level infection rates and population‐level hazard rates are approximately equivalent when the probability of infection under vaccine and placebo are small over the course of the trial. This is comforting, but still conceptually novel. The force of infection is another term for the population‐level hazard rate in the infectious disease world. It is generally thought of as the infection rate, but now there is a clever subtle distinction between the individual infection rate, the population‐level infection rate and the population‐level hazard rate, or force of infection. $VE$ over time then becomes one minus the ratio of two expectations. As they state, in this approach, they have made assumptions regarding individual and population phenomena transparent.

My third point is more a question about the model of how vaccine efficacy wanes over time. In the simulations, they use a step function, that is, VE is suddenly lower after a point in time. If one were analyzing data from a trial, how would one determine the change point on such a VE step function? How would one estimate VE if after some time it wanes continuously over time rather than as a step function?

Fourth, though infection rates can change over time, an assumption is made that the relative effect of vaccine to placebo remains approximately constant. In Section 3, condition (i) separates the biological (transmission probability) from the behavioral aspects (contact rate, prevalence of infection) of the infection rate, and condition (ii) makes the assumption that if variants change transmission probabilities, they remain proportionally the same under vaccine and placebo. There is a limitation of assuming the VE on transmission probability is the same over time, as currently people are very concerned with the VE against variants of concern. TD note this limitation in the discussion, writing that it would not be possible to disentangle personal time from calendar time. This is an important aspect of the work. It is not clear how to get around it.

My fifth point is that there is an unstated further assumption related to the Moderna vaccine trial, as well as most of the other COVID‐19 vaccine trials. The development in this work is on infection, with emphasis on the biologic effect of vaccine on the transmission probability. The authors write in the first paragraph that the primary endpoint in vaccine trials is often viral infection, but later state that the primary endpoint was symptomatic viral infection. In fact, ascertainment of the primary outcome in these trials was on symptomatic disease, which was then biologically confirmed to be the infection of interest. Many COVID‐19 infections are asymptomatic. In addition to each individual having potential transmission probabilities under vaccine and placebo that may change over time, each individual could also have potential probabilities of developing disease given infection under vaccine and placebo, also called the pathogenicity. It would seem that to preserve identifiability, an assumption of some sort of constancy of proportionality, or lack of effect of vaccine on pathogenicity might be required. It would be good to see this addressed within this framework.

In summary, this is a tremendously imaginative and important contribution to estimating vaccine efficacy, with or without the presence of waning and unblinding. It will provide a useful framework in analyzing many VE trials in the future.

Halloran, M.E . Discussion on “Estimating vaccine efficacy over time after a randomized study is unblinded” by Anastasios A. Tsiatis and Marie Davidian. Biometrics. 2021;1–2. 10.1111/biom.13540