Re: Subramanian and Kumar. Vaccination rates and COVID-19 cases

We read with interest the manuscript from Subramanian and Kumar [1] (the authors). We have serious concerns about the methodology employed in this study. We detail these below.

First, the study uses confirmed COVID cases at the country or county level as the primary measure of vaccine efficacy. While the total number of cases remains an important indicator, it does not capture the key component of a successful vaccination strategy, which is a reduction in severe cases, hospitalizations and deaths. Controlling hospitalization is also crucial to limit the burden on the health systems. Therefore, the primary outcome in this study is inappropriate, or at least insufficient, and COVID-related hospitalizations, severe forms and deaths should have been reported. While the authors mention the omission of hospitalizations and severe forms as a potential limitation of their analysis, they do so to highlight that “the CDC reported an increase from 0.01 to 9% and 0 to 15.1% (between January to May 2021) in the rates of hospitalizations and deaths, respectively, amongst the fully vaccinated.” We find this statement misleading. Indeed, this time period corresponds to the beginning of the vaccination campaign, where vaccines were offered to a small high-risk part of the population, mainly the elderly and individuals with serious comorbidities. This is visible in the fact that vaccination for all adults was only available in April in the USA and in May in many other countries (e.g., France or Germany). Furthermore, the effect of an increasing vaccination rate on hospitalizations and deaths figures has been widely explained (e.g., [2]).

Then, the number of confirmed cases is not an accurate measure of the spread of the disease: its accuracy is dependent on the testing capacity, on the national testing policies [3]), on the implementation of Non-Pharmaceutical Interventions (NPIs) [4], on the individual behavioral responses [5], and on the accurate recording of these, none of which were accounted for in the analysis. Not including these factors can lead to biases in the estimation of the effect of any intervention (as explained in [6]). Although this is identified as one of the main limitations of the study, the interpretation of the results was made using causal language without caution, despite the authors’ awareness of the issue.

The timing between the two measurements is also an issue. An arbitrary seven-day time-window for the incidence of COVID-19 cases was used without justification which could lead to include non-representative cases or compare countries over different epidemic phases. Such a short period would only give a cross-sectional view of a phenomenon spanning over months and a seven-day window is not a relevant clinical threshold. Notably, one is considered fully vaccinated 14 days after the second shot. Fourteen days would be the minimum to observe an individual-level effect, but the evaluation of the indirect effect of vaccination on transmission would require an extended follow-up. Vaccinating is a long, continuous process, occurring jointly with successive epidemic “waves”. In addition, while the authors mention a “sensitivity analysis” available in the supplementary materials, it is not available. This seven-day time window thus appears unjustified and does not allow the estimation of the effectiveness of vaccination. Besides, the vaccination status of a population does not capture the population immunisation status, by excluding previously infected individuals. In countries with low vaccination rate but high seroprevalence, the immunisation status of the population remains unclear.

The inclusion/exclusion criteria are either not well defined or were not rigorously followed. The authors have specified that they included “68 countries that met the following criteria: had second dose vaccine data available; had COVID-19 case data available; had population data available; and the last update of data was within 3 days prior to or on September 3, 2021.” These are set without any justification. Furthermore, many countries provide all of this information but are not included in their analysis (such as France, the United Kingdom, Germany, Switzerland, or Spain). In addition, many included countries are low and middle income countries which have less testing capacities and might suffer a higher, yet under-reported, burden from COVID-19 [7].

Moreover, the lack of adjustment for key confounding factors could explain the reported inefficacy of the vaccine. Indeed, the statistical analysis involves an unadjusted linear regression and three descriptive plots. This only allows the readers to gauge raw (confounded) statistical associations. However, the interpretation of these results in the manuscript is causal, which therefore conveys an inaccurate message.

Finally, based on the graphs only, the authors concluded absence of association between the vaccination coverage and the incidence. The categorisation of the proportion of vaccinated people into 15 categories is arbitrary, and we cannot find an empirical justification for the claim that “cases per 100,000 people in the last seven days is largely similar across the categories of percent of the population fully vaccinated”. Yet, if we perform a simple non-parametric Kruskal–Wallis test to compare the distribution of cases across these 15 groups (χ² = 399.39, df = 14, p-value < 0.01), followed by a multiple pairwise Wilcoxon test (Bonferroni corrected), there is a strong evidence that a higher vaccination rate is associated with a lower 7-day incidence. Out of 105 pairwise comparisons, 67 showed a significant difference, with an adjusted p-value < 0.05. Among these, the category (70–100) has a significantly lower seven-day incidence than every category < 50%. This is even clearer from the raw data, where a trend fitted from a generalized additive model shows a decreasing incidence from 50% vaccine coverage onwards. Although this analysis does not account for confounding factors either, it illustrates that the data provided in the manuscript do not support the conclusions drawn by the authors.

We thus would like to highlight that the methodology does not allow the authors to draw the conclusions written in the manuscript. This paper is not up to the standards in epidemiology, and provides a narrative rather than testing hypotheses in a rigorous manner. More critically, the message conveyed in the manuscript may compromise the efforts made to encourage vaccination, despite the numerous valid scientific studies proving vaccine efficacy.

Supplementary Information

Below is the link to the electronic supplementary material.

Declarations

Conflict of interest

The authors declare that they have no conflict of interest.