Patterns of protection, infection, and detection: Country-level effectiveness of COVID-19 vaccination in reducing mortality worldwide

Abstract

Background

The relationship between COVID-19 vaccination and mortality has been established through clinical trials and other investigations at the individual level. In this study, we aimed to investigate the negative relationship between mortality and COVID-19 vaccination at country level.

Study design

We conducted an exploratory, correlational, country-level analysis of open data centralized by Our World in Data concerning the cumulative COVID-19 mortality for the winter wave (October 2021–March 2022) of the pandemic as function of the vaccination rate in October 2021.

Methods

We controlled variables that capture country-level social development and level of testing. We also deployed three segmentation tactics, distinguishing among countries based on their level of COVID-19 testing, age structure, and types of vaccines used.

Results

Controlling for confounding factors did not highlight a statistically significant relationship between vaccination and cumulative mortality in the total country sample. Still, a strong, significant, negative relationship between cumulative mortality (log scale) and vaccination was highlighted through segmentation analysis for countries positioned at the higher end of the social development spectrum. The strongest estimate for vaccine effectiveness at ecological level was obtained for the set of countries that used Western-only vaccines.

Conclusions

COVID-19 testing (log scale) has a significant and positive relationship with cumulative mortality for all subsamples, consistent with patterns of under- and overreporting of COVID-19 deaths at country level, partly driven by testing. This indicates that testing intensity should be controlled as a potential confounder in future ecological analyses of COVID-19 mortality.

What this study adds

• •Our study provides updated information regarding the positive relationship between COVID-19 mortality and testing at country level.
• •Countries with a higher proportion of people aged 65+ present a negative, linear relationship between mortality and vaccination.
• •Vaccine effectiveness estimates were stronger at country-level for the set of countries that used Western-only vaccines in comparison with the average of all vaccines in use.

Implications for policy and practice

• •The level of testing for COVID-19 should be assessed for public health interventions at country level that target COVID-19 mortality.
• •During a pandemic, public health policy should differentiate countries in terms of social development using segmentation tactics.

1. Introduction

It has been proven, through stage III clinical trials and observational studies, that vaccination significantly reduces the risk of dying for people infected with COVID-19 [[1], [2], [3]] and that vaccine protection persists across successive variants [4]. Ecological analyses reported negative correlations between COVID-19 vaccination and mortality at country level [5,6]. Some of these studies focused on highly-developed countries, thus circumventing the positive correlation between Human Development Index (HDI) and mortality. HDI was created by the United Nations Development Programme [7] and it aggregates three dimensions of development: economic prosperity as measured by gross national income (GNI) per capita, a long and healthy life as captured by life expectancy at birth, and human resources for development as measured by a combination of mean and expected years of schooling.

The positive correlation between HDI and mortality is specific for wider global samples, but not for the subsamples of high-income countries [8]. A study of 32 countries in Europe and Israel found a high effectiveness of vaccination against death, through a time series analysis of new COVID-19 deaths during January 2020–April 2021 [9]. An investigation of data from 30 countries of the European Economic Area up to January 2022 reported that low vaccination rates, high proportions of older people, low funding, and inadequate staffing of public health systems were independent risk factors for a higher case fatality rate [10]. A study of the 27 European Union (EU) countries found that countries with higher vaccine coverage improved their relative cumulative mortality profile within the EU [5]. When broadening the geographical focus, a study of 184 countries from December 2020 to December 2021 found that an increased vaccination rate significantly reduced new deaths per million, and that a vaccination rate of 70% additionally contributed to protection from death through herd immunity [11]. A study of the global relationship between vaccination rates and COVID-19 outcomes up to August 2021 did not find proof of vaccination effectiveness in longitudinal data, but observed that cross-sectionally, at a given time point, and especially for the subset of countries with high vaccination rates (more than 60%), there was a negative association between vaccination rates and new deaths per million [12].

Estimating the impact of vaccination on COVID-19 mortality is challenging because many countries with low vaccination levels have also reported reduced rates of infections, case fatality, and mortality [13]. The relatively low intensity of the pandemic in Africa, despite very low vaccination rates, remains a topic of scientific and public debate, being attributed to factors such as a young population, higher proportion of rural populations with lower density and more outdoor living, lower international travel, cross-immunity from previous infections, climate factors, early and drastic lockdown and restriction policies, and possible underreporting, partly derived from low testing rates [14].

The published literature has identified at least three factors that link development and vaccination with mortality in diverging directions [13]. First, as regards the testing level, there may be differential methods of detection – measurement and attribution. Countries with fewer resources available have used fewer tests, and testing is positively associated with HDI and with government capacity [15]. Thus, for countries with lower testing levels, some of the deaths that would have been otherwise identified and attributed to COVID-19 might have been attributed to other causes, or might have remained unreported [16].

Relatively few studies have examined COVID-19 testing as a proxy for country-level measurement strategies concerning deaths, and testing is not, as a rule, controlled as a potential confounder for ecological analyses of COVID-19 mortality. Several studies documented that early-stage pandemic testing capacity was associated with decreased mortality, as testing allowed better diagnosis [17,18] and contact tracing.

As regards the second factor, countries with higher social and economic development have specific patterns of infection because of larger proportions of older people, who have been particularly at risk in the pandemic [19]. The proportion of older people in the general population may also account for the positive correlation between COVID-19 mortality and social and economic development metrics [20].

As regards the third factor, a negative association between social development and mortality may be mediated by patterns of protection, through COVID-19 vaccination and the quality of health care. Since the inception of the vaccination campaign, there has been a strong positive association between vaccination rates and the social and economic development of countries [11]. While social development is negatively linked with COVID-19 mortality through level of vaccination and access to effective health care, it is positively linked with mortality through testing and attribution of deaths, and through higher proportions of older people.

In this study, we aimed to disentangle divergent factors that have shaped COVID-19 mortality and provide estimates for the negative association of vaccination with mortality at ecological level, by examining the relationship between vaccination rates in October 2021 and cumulative mortality for the 2021–2022 winter COVID-19 wave. We combined visual explorations of bivariate associations with multiple linear regression models, and we explored and evaluated three strategies for segmenting the country population – by testing level, age structure, and vaccine type.

2. Methods

We used publicly available data from OWID [21] covering the winter of 2021–2022 wave, from October 2021 to March 2022. We focused on the winter wave because it allowed us to better capture the impact of vaccination, leaving aside cumulative mortality accumulated in the first and second waves when, for many countries, vaccination campaigns were still incipient. For example, a study of the 27 EU countries found that vaccination impact on cumulative mortality only started to become visible with a 4-month lag after July 2021 [5].

We control in our analysis the dimensions of HDI: GNI per capita, life expectancy at birth, and mean years of schooling. Because the proportion of people aged 65+ captures COVID-19 mortality risks better than does life expectancy, we used this variable instead of life expectancy, with which it is strongly correlated (R = 0.752, P-value = 0.000).

Therefore, we included six variables at country level in the analysis, for which descriptive statistics are available in Table S.M. 1 in Supplementary Material. The dependent variable is cumulative mortality for a 6-month period. We computed the difference in average cumulative mortality registered in March 2022 and average cumulative mortality registered in October 2021 (source: OWID), resulting in the cumulative mortality for this interval. The main independent variable is the vaccination rate prior to the studied period, in October 2021. The vaccination rate is estimated through the cumulative share of people (%) vaccinated with two doses reported at a daily rate and averaged across the month of October 2021 (source: OWID). We also controlled for other four confounding factors. The level of testing, measured as total tests per thousand aggregates available testing information from a subset of countries globally [22] (source: OWID, data reported for the first week of January 2022). The proportion of people aged 65+ per hundred, GNI per capita, and mean years of schooling are reported by UNDP in its Human Development Report dataset (2020), also available via OWID [21].

We included countries with a population larger than one million, subject to the availability of OWID data. The list of the 136 countries included in the analysis and the three types of segmentation are presented in Tables S.M. 5, S.M.6, and S.M.7 in Supplementary Material.

Besides exploring correlational patterns and estimating regression models, we also explored segmentation tactics aiming to distinguish countries in terms of pandemic coping strategies. Segmentation rests on the assumption that across countries, a pattern conflates distinct and possibly divergent patterns that could be revealed by separating the population into relevant segments. If the relationship across countries is reversed through segmentation, the situation becomes an instance of the so-called Simpson’s paradox/amalgamation paradox [23]. It is important to keep in mind that any segmentation may work as a proxy for a different confounding variable that differentiates countries, leading to yet another version of the pervasive confounding risks of correlational and ecological analyses.

We first segmented the country set through their approach to testing, comparing the subsample of countries with testing values below the median (758.85 tests per thousand) with those tested at or above median values. This segmentation was based on the assumption that higher testing is conducive to more precise measurement of COVID-19 mortality, and thus it is better suited to indicate the protective impact of vaccination at country level [24].

We then segmented the set of countries according to their proportion of people aged 65+, comparing the subsample of countries with values below the median (6.92%) with countries at or above the median. This segmentation worked on the assumption that the impact of vaccination on cumulative mortality is higher or better visible at ecological level for countries with older populations, which had higher risks of COVID-19 deaths and thus benefitted more from vaccination, compared with countries with younger populations.

Finally, we segmented the country set according to the vaccines they used, comparing countries that administered non-Western vaccines (manufactured in countries such as China, Russia, India, or Cuba), either exclusively or in combination with Western vaccines, with countries that only administered Western vaccines (Pfizer–BioNTech–Comirnaty, Moderna–Spikevax, Johnson & Johnson’s–Janssen, and Oxford AstraZeneca–Vaxzevria).

3. Results

An initial exploration of the correlation patterns between the cumulative mortality in the winter wave of the pandemic (October 2021–March 2022) and the vaccination rate led to inconclusive results (Chart 1.1, Fig. 1). The dispersed scatterplot in Chart 1.2 (Fig. 1) likely resulted from overlapping the negative influence of vaccination on mortality with the positive relationships between social development and COVID-19 mortality.

Fig. 1.

Exploration of covariation patterns between COVID-19 cumulative mortality and both the vaccination rate and the HDI.

Since the mortality rate does not seem to be linearly related to either vaccination rates or HDI, we opted to use a logarithmic scale for cumulative mortality. This transformation led to a clearer visualization of the relationships. As shown in Chart 1.3, the cumulative mortality in the winter wave (log scale) had a nonlinear, reverse J-shaped pattern of association with vaccination rates: an initial positive relationship is followed by a plateau extending to the right, and then by an apparently negative association at the higher vaccination end of the spectrum. The positive linear relationship between cumulative mortality (log scale) and HDI is clearly visible in Chart 1.4.

In Fig. 2 we investigated the association between HDI and its components and COVID-19 cumulative mortality in the winter wave.

Fig. 2.

Exploration of covariation patterns between COVID-19 cumulative mortality and the testing rate, the proportion of people aged 65 and older, GNI per capita, and mean years of schooling with linear vs. logarithmic scaling for all countries.

Based on previous findings, we expected that highly developed countries would have tested their populations more extensively, thus diagnosing more cases and possibly attributing more deaths to COVID-19 than less developed countries [25]. However, the scatterplot of cumulative mortality and tests per thousand (Chart 2.1) does not clarify the association because the relationship is not linear. Transforming both variables through a logarithmic scale shows a clear positive correlation between them (Chart 2.3). The nonlinear positive association of cumulative mortality with the proportion of people aged 65+ in the population (Chart 2.2) is clarified when both variables are visualized through a logarithmic transformation (Chart 2.4). Charts 2.5 and 2.6 present the linear associations of cumulative mortality (log scale) with GNI per capita (log scale) and mean years of schooling (linear scale), respectively.

3.1. Linear regression analysis – all countries

In order to disentangle patterns of vaccine protection from divergent factors that have shaped COVID-19 infection and detection, we used a linear regression model to control for possible confounding variables. Table 1 presents the regression model for all countries included in the analysis. The model has a high predictive value with an adjusted R² of 70%. The only statistically significant coefficients are those that account for the positive relationships between mortality and development, namely the influence of testing, proportion of older people, and mean years of schooling. In the overall cross-sectional picture of the 2021–2022 winter wave of the pandemic, economic capital is not a statistically significant predictor of mortality when controlling for the other factors, nor is vaccination rate.

Table 1.

Regression model for cumulative mortality from October 2021 to March 2022 (logarithmic scale) as a function of the vaccination rate in October 2021, controlling for measurement intensity, age structure, education structure, and GNI per capita.

	Unstandardized Coefficients		Standardized Coefficients	t	P-value
	B	SE	Beta
(Constant)	.548	.808		.679	.499
Vaccination rate Oct. 2021	–.004	.003	–.117	–1.221	.225
Total tests per thousand (logarithmic scale)	.483	.129	.400	3.732	.000
People aged 65 and older (%) (logarithmic scale)	1.240	.206	.514	6.025	.000
Mean years of schooling 2019	.087	.036	.317	2.447	.016
GNI 2019 per capita (logarithmic scale)	–.373	.281	–.210	–1.327	.188

Dependent variable: Cumulative mortality for March 2022–October 2021 (logarithmic scale). N = 90, adjusted R² = 0.703.

3.2. Segmentation analysis

The positive but plateauing shape of the bivariate relationship between cumulative mortality and vaccination, as visualized in Chart 1.3 (Fig. 1), suggests the possibility of segmenting the population of countries into subcategories, attempting to better isolate the negative relationship between vaccination and mortality from other country-level factors. We discuss in the following section the three segmentation tactics we used: testing level, age structure, and vaccine type. The three segmentation criteria were correlated, though not overlapping (see Table S.M. 11 in Supplementary Material).

A synthesis of segmentation results is presented in Table 2, indicating for the sample and each segment the partial correlation between cumulative mortality for the winter wave (log scale) and vaccination rate (October 2021), as well as controlling for the other variables in the model. The segments with values above the median for the differentiating criterion and the Western-only vaccine segment attest negative, statistically significant associations between cumulative mortality and vaccination rate, when confounding factors are controlled. No significant relationships were identified for the unsegmented sample or for the other segments. The strongest partial correlation between cumulative mortality and vaccination rate was determined to be that of the Western-only vaccine set of countries, which was also the most exclusive in terms of membership (listwise N = 25 because of missing values for testing and listwise N = 27 when testing is not controlled).

Table 2.

Partial correlation between cumulative mortality (log scale) and vaccination rate, controlling for proportion of people aged 65 or more (log scale), tests per thousand (log scale), GNI per capita (log scale), and mean years of schooling.

Model	Segments	Also controlling for testing intensity			Without controlling for testing intensity
		Listwise N	Partial correlation	P-value	Listwise N	Partial correlation	P-value
Global analysis	All countries	90	–0.128	0.225	125	–0.124	0.165
Segmented by testing intensity	Testing at or above median	42	–0.304	0.045	43	–0.226	0.135
	Testing below median	42	–0.027	0.862	43	0.049	0.747
	Missing information for testing		N/A		29	–0.128	0.493
Segmented by age structure	People aged 65+ at or above median	51	–0.355	0.09	60	–0.522	0.000
	People aged 65+ below median	33	–0.186	0.284	60	0.038	0.770
Segmented by vaccine type	Western-only vaccines	25	–0.616	0.001	27	–0.527	0.003
	All vaccines	59	–0.013	0.918	93	–0.073	0.484

If we do not control for testing level, we gain more cases in the analysis. The broad pattern of results is unchanged, excepting the partial correlation in the segment with testing above the median, which is no longer statistically significant. The negative correlation of vaccination rate with cumulative mortality (log scale) is still visible for the Western-only vaccine segment and for the segment of countries with older populations.

We compared countries testing below the median with those above the median (countries are listed in Table S.M 4 in Supplementary Material). As Fig. 3 indicates, the countries with testing below the median displayed a positive, non-linear relationship between mortality (log scale) and vaccination, with a plateau toward higher values of vaccination (Chart 3.1), while countries with testing above the median displayed a negative, linear relationship (Chart 3.2).

Fig. 3.

Exploring patterns of covariation between cumulative mortality and vaccination rate in three segmentation scenarios: volume of testing, age structure, and vaccine type.

Segmentation by age structure yields similar results, as shown in the second row of Fig. 3 (countries are listed in Table S.M 6, Supplementary Material). Countries with a lower proportion of older people presented a positive, plateauing relationship between mortality (log scale) and vaccination, while countries with a higher proportion presented a negative, linear relationship (Charts 3.3 and 3.4).

As shown in Charts 3.5 vs. 3.6, vaccine-type segmentation also highlights two different relationships (countries are listed in Table S.M. 8, in Supplementary Material). The segment that used Western-only vaccines included 33 highly-developed countries, which had political and financial access to these vaccines and which, given their high social and economic development, also benefitted from better health care systems and testing policies. This segmentation isolated a subsample with a strong negative association between mortality and vaccination rates (the Western-only vaccines segment, Chart 3.6) from a larger subsample that incorporated high heterogeneity (Chart 3.5).

We further synthesized regression models for each segment (see details in Supplementary Material, in Tables S.M. 11, S.M.12, and S.M.13). We found that the level of testing (total tests per thousand) has a positive and statistically significant relationship with mortality in all analyzed segments, and it is the only predictor with such degree of consistency. Vaccination has a negative relationship with mortality that is statistically significant only for the segments of countries with high levels of testing, high proportions of older people, and Western-only vaccines. GNI has a negative association with mortality that is statistically significant only in countries with high testing levels.

4. Discussion

In this study, we attempted to investigate at ecological level the negative relationship between COVID-19 mortality and vaccination. We conducted an exploratory, correlational, country-level analysis of open data centralized by OWID concerning the cumulative COVID-19 mortality for the winter wave (October 2021–March 2022) as a function of vaccination rate in Oct. 2021, total tests per thousand in early January 2022, and dimensions of social development - the proportion of people aged 65+, GNI per capita, and mean years of schooling.

Linear regression models for segmented subsets of countries highlight the fact that COVID-19 testing (log scale) had a significant and positive relationship with cumulative mortality for all segments. This finding is consistent with the published literature discussing possible patterns of under- and overreporting of COVID-19 deaths, partly driven by testing [16,26]. Nevertheless, excepting studies that have explored the link between early pandemic testing and reductions in mortality, there has been little use of updated testing information in multivariate models on COVID-19 mortality at country level. This indicates that testing intensity should be controlled as a potential confounder in future ecological analyses of COVID-19 mortality.

The stronger estimate for vaccine effectiveness at ecological level for countries that used Western-only vaccines may reflect a higher effectiveness of these vaccines in comparison with the average of all vaccines in use, but it may also derive from the lower social heterogeneity of countries included in this segment. These countries have high social development levels, and they are predominantly located in the EU and North America.

Our study shares the limitations of large-scale, ecological, and correlational analyses. The impact of vaccination on mortality refers to processes that take place at the individual and interpersonal levels, such as preventing transmission, serious illness, and death. Therefore, a study at country level would run the risk of ecological errors caused by successive steps of aggregation in the measurement processes. The high number of countries is conducive to heterogeneous measurements. Despite our goal to control for confounding factors, other sources of amalgamation may persist.

The segmentation strategies that we proposed in this paper are relevant for public health policies and management during global crises. Distinguishing among countries based on public health resources available and population characteristics facilitates the identification of vulnerable regions and informs interventions and policies.

Author contribution

All authors made a significant contribution to the development of this manuscript and approved the final version for submission.

Ethical approval

Ethical approval was not required for this study, because we conducted secondary data analysis on publicly available data.

Funding

This work was supported by the Executive Unit for Financing Higher Education, Research, Development and Innovation (UEFISCDI), Romania, grant PN-III-P4-ID-PCE-2020-1589 ‘‘The manufacture of doubt on vaccination and climate change. A comparative study of legitimacy tactics in two science-skeptical discourses”.

Declaration of competing interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Appendix A. Supplementary data

The following is the Supplementary data to this article: