Effectiveness of the Coronavirus Disease 2019 Bivalent Vaccine

Abstract

Background

The purpose of this study was to evaluate whether a bivalent coronavirus disease 2019 (COVID-19) vaccine protects against COVID-19.

Methods

The study included employees of Cleveland Clinic in employment when the bivalent COVID-19 vaccine first became available. Cumulative incidence of COVID-19 over the following 26 weeks was examined. Protection provided by vaccination (analyzed as a time-dependent covariate) was evaluated using Cox proportional hazards regression, with change in dominant circulating lineages over time accounted for by time-dependent coefficients. The analysis was adjusted for the pandemic phase when the last prior COVID-19 episode occurred and the number of prior vaccine doses.

Results

Among 51 017 employees, COVID-19 occurred in 4424 (8.7%) during the study. In multivariable analysis, the bivalent-vaccinated state was associated with lower risk of COVID-19 during the BA.4/5-dominant (hazard ratio, 0.71 [95% confidence interval, .63–79]) and the BQ-dominant (0.80 [.69–.94]) phases, but decreased risk was not found during the XBB-dominant phase (0.96 [.82–.1.12]). The estimated vaccine effectiveness was 29% (95% confidence interval, 21%–37%), 20% (6%–31%), and 4% (−12% to 18%), during the BA.4/5-, BQ-, and XBB-dominant phases, respectively. The risk of COVID-19 also increased with time since the most recent prior COVID-19 episode and with the number of vaccine doses previously received.

Conclusions

The bivalent COVID-19 vaccine given to working-aged adults afforded modest protection overall against COVID-19 while the BA.4/5 lineages were the dominant circulating strains, afforded less protection when the BQ lineages were dominant, and effectiveness was not demonstrated when the XBB lineages were dominant.

Among 51 017 working-aged Cleveland Clinic employees, the bivalent coronavirus disease 2019 vaccine was 29% and 20% effective in preventing infection while the BA.4/5 and BQ lineages, respectively, were dominant. Effectiveness was not demonstrated when the XBB lineages were dominant.

When the original messenger RNA (mRNA) coronavirus disease 2019 (COVID-19) vaccines first became available in 2020, there was ample evidence of efficacy from randomized clinical trials [1, 2].Vaccine effectiveness was subsequently confirmed by clinical effectiveness data in the real world outside of clinical trials [3, 4], including an effectiveness estimate of 97% among employees within our own healthcare system [5]. This was when the human population had just encountered the novel severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) virus, and the pathogen had exacted a high morbidity and mortality burden across the world. The vaccines were amazingly effective in preventing COVID-19, saved a large number of lives, and changed the impact of the pandemic.

Continued acquisition of mutations in the virus, from natural evolution in response to interaction with the immune response among the human population, led to the emergence and spread of SARS-CoV-2 variants. Despite this, for almost 2 years since the onset of the pandemic, those previously infected or vaccinated continued to have substantial protection against reinfection by virtue of natural or vaccine-induced immunity [6]. The arrival of the Omicron variant in December 2021 brought a significant change to the immune protection landscape. Previously infected or vaccinated individuals were no longer protected from COVID-19 [6]. Vaccine boosting provided some protection against the Omicron variant [7, 8], but the degree of protection was not near that of the original vaccine against the pre-Omicron variants of SARS-CoV-2 [8]. After the emergence of the Omicron variant, prior infection with an earlier lineage of the Omicron variant protected against subsequent infection with a subsequent lineage [9], but such protection appeared to wear off within a few months [10]. During the Omicron phase of the pandemic, protection from vaccine-induced immunity decreased within a few months after vaccine boosting [8].

Recognition that the original COVID-19 vaccines provided much less protection after the emergence of the Omicron variant spurred efforts to produce newer vaccines that were more effective. These efforts culminated in the approval by the US Food and Drug Administration, on 31 August 2022, of bivalent COVID-19 mRNA vaccines, which encoded antigens represented in the original vaccine as well as antigens representing the BA.4/5 lineages of the Omicron variant. Given the demonstrated safety of the earlier mRNA vaccines and the perceived urgency of need of a more effective preventive tool, these vaccines were approved without demonstration of effectiveness in clinical studies. The purpose of this study was to evaluate whether the bivalent COVID-19 vaccine protects against COVID-19.

METHODS

Study Design

This was a retrospective cohort study conducted at the Cleveland Clinic Health System (Cleveland, Ohio) in the United States.

Patient Consent Statement

The study was approved by the Cleveland Clinic Institutional Review Board as exempt research (IRB no. 22–917). Waivers of informed consent and of HIPAA (Health Insurance Portability and Accountability Act) authorization were approved to allow the research team access to the required data.

Setting

Since the arrival of the COVID-19 pandemic at Cleveland Clinic in March 2020, employee access to testing has been a priority. Voluntary vaccination for COVID-19 began on 16 December 2020, and the monovalent mRNA vaccine as a booster became available to employees on 5 October 2021. The bivalent COVID-19 mRNA vaccine was first offered to employees on 12 September 2022. This date was considered the study start date. The mix of circulating variants of SARS-CoV-2 changed over the course of the study. The majority of infections in Ohio were initially caused by the BA.4 or BA.5 lineages of the Omicron variant. By mid-December 2022 the BQ lineages, and by mid-January 2023 the XBB lineages of the Omicron variant were the dominant circulating strains [11].

Study Participants

The study included Cleveland Clinic Health System employees in employment at any Cleveland Clinic location in Ohio on 12 September 2022, the day the bivalent vaccine first became available to employees. Those for whom age and sex were not available were excluded.

Variables

The covariates collected were age, sex, job location, and job type categorized into clinical or nonclinical, as described in our earlier studies [5–7]. Institutional data governance rules related to employee data limited our ability to supplement our data set with additional clinical variables. Employees were considered prepandemic hires if hired before 16 March 2020, the day COVID-19 testing became available in our institution, and pandemic hires if hired on or after that date.

Prior COVID-19 was defined as a positive nucleic acid amplification test (NAAT) result for SARS-CoV-2 any time before the study start date. The date of infection for a prior episode of COVID-19 was the date of the first positive test for that episode of illness. A positive test >90 days after the date of a previous infection was considered a new episode of infection. Since the health system never had a requirement for systematic asymptomatic employee test screening, most positive test results would have been from tests done to evaluate suspicious symptoms. Some would have been tests done to evaluate known exposures or for preoperative or preprocedural screening. The pandemic phase (pre-Omicron or Omicron) during which a study participant had his or her last prior episode of COVID-19 was also collected as a variable, based on which variant/lineages accounted for >50% of infections in Ohio at the time [11].

Outcome

The study outcome was time to COVID-19, the latter defined as a positive NAAT result for SARS-CoV-2 any time after the study start date. Outcomes were followed up until 14 March 2023, allowing for evaluation of outcomes up to 26 weeks from the study start date.

Statistical Analysis

A Simon-Makuch hazard plot [12] was created to compare the cumulative incidence of COVID-19 in the bivalent-vaccinated and nonvaccinated states, by treating bivalent vaccination as a time-dependent covariate. Study participants were considered bivalent vaccinated 7 days after receipt of a single dose of the bivalent COVID-19 vaccine. Those whose employment was terminated during the study period before they had COVID-19 were censored on the date of termination. Curves for the nonvaccinated state were based on data while the bivalent vaccination status of participants remained “nonvaccinated.” Curves for the bivalent-vaccinated state were based on data from the date the bivalent vaccination status changed to “vaccinated.”

Multivariable Cox proportional hazards regression models were fitted to examine the association of various variables with time to COVID-19. Bivalent vaccination was included as a time-dependent covariate [13]. The study period was divided into BA.4/5-dominant, BQ-dominant, and XBB-dominant phases, depending on which group of lineages accounted for >50% of all COVID-19 infections at the time (based on variant proportion data from the Centers for Disease Control and Prevention [CDC]) [11] and which group of lineages was most abundant in our internal sequencing data. Time-dependent coefficients were used to separate out the effects of the bivalent vaccine during the different phases.

The primary model included all study participants. The secondary model included only those with prior exposure to SARS-CoV-2 by infection or vaccination and evaluated the effect of bivalent vaccination with inclusion of time since most recent exposure to SARS-CoV-2 by infection or vaccination, to adjust for the effect of waning immunity on susceptibility to COVID-19. The possibility of multicollinearity in the models was evaluated using variance inflation factors. The proportional hazards assumption was checked using log(−log[survival]) versus time plots. Vaccine effectiveness was calculated from the hazard ratios (HRs) for bivalent vaccination in the models. The analysis was performed by N. K. S. and A. S. N. using the survival package and R software, version 4.2.2 (R Foundation for Statistical Computing) [13–15].

RESULTS

Of 51 982 eligible study participants, 965 (1.9%) were excluded because of missing age or sex. Of the remaining 51 017 employees included, 3294 (6.5%) were censored during the study because of termination of employment. By the end of the study, 13 134 (26%) had received the bivalent vaccine, which was the Pfizer vaccine in 11 397 (87%) and the Moderna vaccine in the remaining 1700. In all, 4424 employees (8.7%) acquired COVID-19 during the 26 weeks of the study.

Baseline Characteristics

Table 1 shows the characteristics of participants included in the study. Notably, this was a relatively young population, with a mean age of 42 years. Among these individuals, 20 686 (41%) had previously had a documented episode of COVID-19, and 13 717 (27%) had previously had an Omicron variant infection; 45 064 (88%) had previously received ≥1 dose of vaccine, 42 550 (83%) had received ≥2 doses, and 46 761 (92%) had been previously exposed to SARS-CoV-2 by infection or vaccination.

Table 1.

Baseline Characteristics of 51 017 Employees of Cleveland Clinic in Ohio

Characteristic	Employees, No. (%)a
Age in years, mean (SD)	42.3 (13.4)
Sex
Female	38 052 (74.6)
Male	12 965 (25.4)
Location
Cleveland Clinic Main Campus	20 495 (40.2)
Cleveland area regional hospitals	12 039 (23.6)
Ambulatory centers	8865 (17.4)
Cleveland Clinic Akron	4301 (8.4)
Administrative centers	4141 (8.1)
Cleveland Clinic Medina	1176 (2.3)
Hire cohort
Prepandemic	34 509 (67.6)
Pandemic	16 508 (32.4)
Human resources job classification
Clinical	25 795 (50.6)
Nonclinical	25 222 (49.4)
Pandemic phase when most recent infection occurred
Not previously infected	30 331 (59.4)
Pre-Omicron	6969 (13.7)
Omicron	13 717 (26.9)
Time since most recent infection, mean (SD), d	287 (220)
No. of prior vaccine doses
0	5953 (11.7)
1	2514 (4.9)
2	14 985 (29.4)
3	23 607 (46.3)
4	3850 (7.5)
5	91 (<1)
6	17 (<1)
Time since most recent vaccine, mean (SD), 3	319 (135)
Time since proximate SARS-CoV-2 exposure, mean (SD)b	263 (142)

Abbreviations: SARS-Cov-2, severe acute respiratory syndrome; SD, standard deviation.

^aData represented no. (%) of employees unless otherwise indicated.

^bExposure by infection or vaccination.

Risk of COVID-19 Based on Prior Infection and Vaccination History

The risk of COVID-19 varied by the phase of the epidemic in which the study participant's last prior COVID-19 episode occurred. In decreasing order of risk were those never previously infected, those last infected during the pre-Omicron phase, and those last infected during the Omicron phase (Figure 1). The risk of COVID-19 also varied by the number of COVID-19 vaccine doses previously received. The higher the number of vaccines previously received, the higher the risk of contracting COVID-19 (Figure 2).

Figure 1.

Cumulative incidence of coronavirus disease 2019 (COVID-19) for study participants stratified by the pandemic phase when the participant's last prior COVID-19 episode occurred. Day 0 was 12 September 2022, the date the bivalent vaccine was first offered to employees. Point estimates and 95% confidence intervals are jittered along the x-axis to improve visibility.

Figure 2.

Cumulative incidence of coronavirus disease 2019 (COVID-19) for study participants stratified by the number of COVID-19 vaccine doses previously received. Day 0 was 12 September 2022, the date the bivalent vaccine was first offered to employees. Point estimates and 95% confidence intervals are jittered along the x-axis to improve visibility.

Bivalent Vaccine Effectiveness

The cumulative incidence of COVID-19 was similar for the bivalent-vaccinated and non–bivalent-vaccinated states in an unadjusted analysis (Figure 3). In a multivariable Cox proportional hazards regression model, adjusted for age, sex, hire cohort, job category, number of COVID-19 vaccine doses before study start, and epidemic phase when the last prior COVID-19 episode occurred, bivalent vaccination provided some protection against COVID-19 while the BA.4/5 lineages were the dominant circulating strains (HR, 0.71 [95% confidence interval (CI)], .63–.79; P <.001), and less protection while the BQ lineages were dominant (0.80 [.69–.94]; P= .005).

Figure 3.

Simon-Makuch plot comparing the cumulative incidence of coronavirus disease 2019 (COVID-19) for the bivalent-vaccinated and non–bivalent-vaccinated states. Day 0 was 12 September 2022, the date the bivalent vaccine was first offered to employees. Point estimates and 95% confidence intervals are jittered along the x-axis to improve visibility.

A protective effect of bivalent vaccination could not be demonstrated while the XBB strains were dominant (HR, 0.96 [95% CI, .82–.1.12]; P = .59). Point estimates and 95% CIs for HRs for the variables included in the unadjusted and adjusted Cox proportional hazards regression models are shown in Table 2. The calculated overall bivalent vaccine effectiveness from the model was 29% (95% CI, 21%–37%) during the BA.4/5-dominant phase, 20% (6%–31%) during the BQ-dominant phase, and 4% (−12% to 18%) during the XBB-dominant phase. The multivariable analysis also found that, the more recent the last prior COVID-19 episode was the lower the risk of COVID-19, and the greater the number of vaccine doses previously received the higher the risk of COVID-19.

Table 2.

Unadjusted and Adjusted Associations With Time to Coronavirus Disease 2019

Variable	Unadjusted HR (95% CI)	P Value	Adjusted HR (95% CI)a	P Value
Bivalent-vaccinated stateb
BA.4/5-dominant phase	.85 (.76–.95)	.005	.71 (.63–.79)	<.001
BQ-dominant phase	.98 (.85–1.14)	.81	.80 (.69–.94)	.005
XBB-dominant phase	1.17 (1.01–1.36)	.04	.96 (.82–1.12)	.59
Age	1.003 (1.000–1.005)	.02	.997 (.995–1.000)	.046
Male sexc	.78 (.72–.84)	<.001	.75 (.70–.80)	<.001
Pandemic hired	.92 (.86–.98)	.01	.96 (.89–1.03)	.24
Clinical jobe	1.12 (1.05–1.18)	<.001	1.15 (1.09–1.23)	<.001
Last prior infection phasef
Pre-Omicron	2.06 (1.85–2.31)	<.001	2.20 (1.97–2.46)	<.001
No known prior infection	2.35 (2.15–2.56)	<.001	2.55 (2.34–2.79)	<.001
No. of prior vaccine dosesg
1	1.91 (1.57–2.32)	<.001	2.07 (1.70–2.52)	<.001
2	2.22 (1.92–2.56)	<.001	2.50 (2.17–2.89)	<.001
3	2.69 (2.35–3.09)	<.001	3.10 (2.69–3.56)	<.001
>3	2.94 (2.50–3.45)	<.001	3.53 (2.97–4.20)	<.001

Abbreviations: CI, confidence interval; HR, hazard ratio.

^aFrom a multivariable Cox-proportional hazards regression model, with bivalent-vaccinated state treated as a time-dependent covariate and time-dependent coefficients used to separate effects during the period of dominance of the Omicron BA.4/5, BQ, and XBB lineages.

^bTime-dependent covariate.

^cReference: female sex.

^dReference: prepandemic hire.

^eReference: nonclinical job.

^fReference: Omicron.

^gReference: 0 doses.

Bivalent Vaccine Effectiveness Among Those With Prior SARS-CoV-2 Infection or Vaccination

Among persons with prior exposure to SARS-CoV-2 by infection or vaccination, HRs for bivalent vaccination for individuals, after adjusting for time since proximate SARS-CoV-2 exposure, are shown in Table 3. Bivalent vaccination protected against COVID-19 during the BA.4/5-dominant phase (HR, 0.78 [95% CI, .70–.88; P <.001), but a significant protective effect could not be demonstrated during the BQ-dominant phase (0.91 [.78–.1.07]; P = .25) or the XBB-dominant phase (1.05 [.85–.1.29]; P= .66).

Table 3.

Associations With Time to Coronavirus Disease 2019 Among Study Participants With Prior Severe Acute Respiratory Syndrome (SARS-CoV-2) Exposure, Adjusted for Time Since Proximate SARS-CoV-2 Exposure by Prior Infection or Vaccination

Variablea	Adjusted HR (95% CI)	P Value
Bivalent-vaccinated stateb
BA.4/5-dominant phase	.78 (.69–.87)	<.001
BQ-dominant phase	.90 (.78–1.05)	.19
XBB-dominant phase	1.06 (.91–1.24)	.43
Age	1.004 (1.001–1.006)	.005
Male sexc	.78 (.73–.84)	<.001
Pandemic hired	1.07 (.99–1.15)	.08
Clinical jobe	1.11 (1.05–1.18)	<.001
Time since proximate SARS-CoV-2 exposuref
91–180 d	1.70 (1.45–1.99)	<.001
181–270 d	1.88 (1.63–2.16)	<.001
271–365 d	2.81 (2.45–3.21)	<.001
>365 d	2.15 (1.86–2.50)	<.001

Abbreviations: CI, confidence interval; HR, hazard ratio; SARS-CoV-2, severe acute respiratory syndrome coronavirus 2.

^aThe number of prior vaccine doses was not included as a variable because its inclusion would have introduced significant multicollinearity into the model.

^bTime-dependent covariate.

^cReference: female sex.

^dReference: prepandemic hire.

^eReference: nonclinical job.

^fReference: ≤90 days; this includes those previously vaccinated within 90 days but not those previously infected within 90 days, as the latter would not have qualified for inclusion until 90 days after their most recent infection.

DISCUSSION

This study found that the current bivalent vaccines were about 29% effective overall in protecting against infection with SARS-CoV-2 when the Omicron BA.4/5 lineages were the predominant circulating strains, and effectiveness was lower when the circulating strains were no longer represented in the vaccine. A protective effect could not be demonstrated when the XBB lineages were dominant. The magnitude of protection afforded by bivalent vaccination while the BA.4/5 lineages were dominant was similar to that estimated in another study using data from the Increasing Community Access to Testing national SARS-CoV-2 testing program [16].

The strengths of our study include its large sample size and its conduct in a healthcare system where very early recognition of the critical importance of maintaining an effective workforce during the pandemic led to devotion of resources to provide an accurate accounting of who had COVID-19, when COVID-19 was diagnosed, who received a COVID-19 vaccine, and when. The study method, treating bivalent vaccination as a time-dependent covariate, allowed vaccine effectiveness to be determined in real time.

The study has several limitations. Individuals with unrecognized prior infection would have been misclassified as previously uninfected. Since prior infection protects against subsequent infection, such misclassification would have resulted in underestimating the protective effect of the vaccine. However, there is little reason to suppose that prior infections would have been missing in the bivalent-vaccinated and nonvaccinated states at disproportionate rates. There might be concern that those who chose to receive the bivalent vaccine may have been more worried about infection and more likely to be tested when they had symptoms, thereby disproportionately detecting more incident infections among those who received the bivalent vaccine. We did not find an association between the number of COVID-19 tests done and the number of prior vaccine doses, however, suggesting that this was not a confounding factor. Those who chose to get the bivalent vaccine could have been those who were more likely to have lower risk-taking behavior with respect to COVID-19. This would have the effect of finding a higher risk of COVID-19 in the nonvaccinated state, thereby potentially overestimating vaccine effectiveness, because the lower risk of COVID-19 in the bivalent-vaccinated state could have been due to lower risk-taking behavior rather than the vaccine.

The widespread availability of home testing kits might have reduced detection of incident infections. This potential effect should be somewhat mitigated in our healthcare cohort because one needs a NAAT to get paid time off, providing a strong incentive to get a NAAT if one tests positive at home. Even if one assumes that some individuals chose not to follow up on a positive home test result with a NAAT, it is very unlikely that individuals would have chosen to pursue NAAT after receiving the bivalent vaccine more than before receiving it, at rates disproportionate enough to affect the study's findings.

We were unable to distinguish between symptomatic and asymptomatic infections and had to limit our analyses to all detected infections. Variables that were not considered might have influenced the findings substantially. Time since last prior exposure to SARS-CoV-2 could not be included in the primary model owing to multicollinearity. It is possible that the association of number of prior vaccine doses with increased risk of infection may have been confounded by time since last prior exposure to SARS-CoV-2. There were too few severe illnesses for the study to determine whether the vaccine decreased severity of illness. Finally, our study was done in a healthcare population, and included no children and few elderly persons, and the majority of study participants would not have been immunocompromised.

A possible explanation for a lower-than-expected vaccine effectiveness is that a substantial proportion of the population may have had prior asymptomatic Omicron variant infection. About a third of SARS-CoV-2 infections have been estimated to be asymptomatic in studies performed in different places at different times [17–19]. If so, protection from the bivalent vaccine may have been masked because those with prior Omicron variant infection may have already been somewhat protected against COVID-19 by virtue of natural immunity. A seroprevalence study conducted by the CDC found that by February 2022, 64% of the 18–64-year age-group population and 75% of children and adolescents had serologic evidence of prior SARS-CoV-2 infection [20], with almost half of the positive serologic results attributed to infections occurring between December 2021 and February 2022, which would have predominantly been Omicron BA.1/BA.2-lineage infections. With such a large proportion of the population expected to have already been previously exposed to the Omicron variant of SARS-CoV-2, it is possible that a substantial proportion of individuals may be unlikely to derive any meaningful benefit from a bivalent vaccine.

The association of increased risk of COVID-19 with more prior vaccine doses was unexpected. A simplistic explanation might be that those who received more doses were more likely to be individuals at higher risk of COVID-19. A small proportion of individuals may have fit this description. However, the majority of participants in this study were young, and all were eligible to have received ≥3 doses of vaccine by the study start date, which they had every opportunity to do. Therefore, those who received <3 doses (46% of individuals in the study) were not ineligible to receive the vaccine but rather chose not to follow the CDC's recommendations on remaining updated with COVID-19 vaccination, and one could reasonably expect these individuals to have been more likely to exhibit risk-taking behavior. Despite this, their risk of acquiring COVID-19 was lower than that that of participants those who received more prior vaccine doses.

Ours is not the only study to find a possible association with more prior vaccine doses and higher risk of COVID-19. During an Omicron wave in Iceland, individuals who had previously received ≥2 doses were found to have a higher odds of reinfection than those who had received <2 doses, in an unadjusted analysis [21]. A large study found, in an adjusted analysis, that those who had an Omicron variant infection after previously receiving 3 doses of vaccine had a higher risk of reinfection than those who had an Omicron variant infection after previously receiving 2 doses [22]. Another study found, in multivariable analysis, that receipt of 2 or 3 doses of am mRNA vaccine following prior COVID-19 was associated with a higher risk of reinfection than receipt of a single dose [7]. Immune imprinting from prior exposure to different antigens in a prior vaccine [22, 23] and class switch toward noninflammatory spike-specific immunoglobulin G4 antibodies after repeated SARS-CoV-2 mRNA vaccination [24] have been suggested as possible mechanisms whereby prior vaccine may provide less protection than expected. We still have a lot to learn about protection from COVID-19 vaccination, and in addition to vaccine effectiveness, it is important to examine whether multiple vaccine doses given over time may not be having the beneficial effect that is generally assumed.

In conclusion, this study found an overall modest protective effect of the bivalent vaccine against COVID-19 while the circulating strains were represented in the vaccine and lower protection when the circulating strains were no longer represented. A significant protective effect was not found when the XBB lineages were dominant. The unexpected finding of increasing risk with increasing number of prior COVID-19 vaccine doses needs further study.