Assessing parameter sensitivity in a university campus COVID-19 model with vaccinations

Abstract

From the beginning of the COVID-19 pandemic, universities have experienced unique challenges due to their dual nature as a place of education and residence. Current research has explored non-pharmaceutical approaches to combating COVID-19, including representing in models different categories such as age groups. One key area not currently well represented in models is the effect of pharmaceutical preventative measures, specifically vaccinations, on COVID-19 spread on college campuses. There remain key questions on the sensitivity of COVID-19 infection rates on college campuses to potentially time-varying vaccine immunity. Here we introduce a compartment model that decomposes a campus population into constituent subpopulations and implements vaccinations with time-varying efficacy. We use this model to represent a campus population with both vaccinated and unvaccinated individuals, and we analyze this model using two metrics of interest: maximum isolation population and symptomatic infection. We demonstrate a decrease in symptomatic infections occurs for vaccinated individuals when the frequency of testing for unvaccinated individuals is increased. We find that the number of symptomatic infections is insensitive to the frequency of testing of the unvaccinated subpopulation once about 80% or more of the population is vaccinated. Through a Sobol’ global sensitivity analysis, we characterize the sensitivity of modeled infection rates to these uncertain parameters. We find that in order to manage symptomatic infections and the maximum isolation population campuses must minimize contact between infected and uninfected individuals, promote high vaccine protection at the beginning of the semester, and minimize the number of individuals developing symptoms.

1. Introduction

Starting in 2020, people worldwide experienced drastic changes to their daily lives as the COVID-19 pandemic swept through communities. In response, communities saw the implementation of COVID-19 mitigation measures such as mask mandates, business closures, quarantines, and school closures (Chappell, 2020; Smith et al., 2020). Included in these changes was the abrupt transition to virtual learning for schools of all levels (Yeung et al., 2020). As COVID-19 restrictions began to ease, in the fall of 2020, and more so in the fall of 2021 schools, especially universities, began to reevaluate their plans of operation (Camera, 2020; Shapiro, 2021). With their dual nature as a place of residence and education, universities had to account for student safety, student education, and their own financial stability.

Arising from this duality, a key challenge in universities' planning was the differing effects COVID-19 can have depending on the group of individuals, or subpopulation, it is affecting. As the COVID-19 pandemic persisted and research provided improved understanding of the virus, SARS-CoV-2, it became clear that the effects of COVID-19 vary depending on the groups of individuals it is affecting. The United States Centers for Disease Control and Prevention (CDC) designate four main groups of individuals who are considered at a higher risk for COVID-19. These are pregnant or recently pregnant individuals, individuals over the age of 65, some immunocompromised individuals, and individuals with certain underlying medical conditions (CDC, 2022). Examples of these types of subpopulations can be seen in current research. Rǎdulescu et al. (Rǎdulescu et al., 2020) investigated the effects age difference can play on COVID-19 spread using a compartment model that split the overall population into four groups based on age and leveraged age dependent infection, recovery, and fatality rates. Research by Lopman et al. (Lopman et al., 2021) expanded this theme further and investigated the effects of an individual's environment on COVID-19 spread. Their research leveraged a model that split a campus population into three main groups: on-campus students, off-campus students, and staff and faculty (Lopman et al., 2021). The diversity of campus stakeholders in terms of multiple different subpopulations poses a challenge for university administrations as they create and update their plans for operation and work to keep their communities safe. As COVID-19 continues, we have seen the introduction of new methods of prevention, such as vaccines, which has naturally created another group of subpopulations for universities to consider: unvaccinated individuals and vaccinated individuals.

Previous work has demonstrated the importance of vaccines. One such study has shown that in certain communities, such as New South Wales, Australia, the use of vaccinations to achieve herd immunity was the only avenue to prevent continued community transmission (MacIntyre et al., 2021). Similarly, research demonstrated that in the U.S. vaccinations led to a reduction in the total number of infections when implemented in many different scenarios and that the vaccination plans originally proposed under the Biden plan would lead to a greater impact than that of the previous administration (Li & Giabbanelli, 2021). Li & Giabbanelli leveraged an agent-based model (ABM) to model disease spread where the previous researchers listed opted for a compartment model approach. ABMs offer an easier path to account for individual variability and the human decision-making process by keeping track of all individuals and their individualized associated characteristics. Conversely, compartment models can be beneficial in disease modeling because they have a smaller computation cost for larger populations. They also allow the spread of a virus to be more generalized. Thus, they are more easily communicable for use in other research with similar viruses. Not only were the strategies for how and when to vaccinate the public important topics for research, but with the vaccine's wide availability to the general public, the next logical questions became the following: How do we account for vaccinated individuals in COVID-19 models and for how long will the efficacy of these vaccines last? Research has found that the vaccines approved for use in the United States wane in efficacy over time (Chemaitelly & Abu-Raddad, 2022; Feikin et al., 2022; Tartof et al., 2021). Raising the question, how this waning immunity affects the spread of COVID-19 especially in environments, such as university campuses, with individuals interacting in close proximity.

Another important factor to consider with any infectious disease spread model is the model output's sensitivity to parameter values. This becomes even more important with a novel disease such as COVID-19, where we are still learning about and fully understanding the disease. Due to SARS-CoV-2's nature as a novel virus, and the fact that its effects on individuals can vary substantially based on age, immunocompromised status, and other conditions (CDC, 2022), in mathematical models for the spread of COVID-19, there are sizable uncertainties in many model parameters. In epidemiological modeling, specifically COVID-19 modeling, one of the main uses of these mathematical models is to provide guidance on the best mitigation strategies (Lopman et al., 2021; MacIntyre et al., 2021; Paltiel et al., 2020; Rǎdulescu et al., 2020). However, in the presence of uncertain parameters, the use of a sensitivity analysis provides important insights into the robustness or fragility of strategies to manage the spread of COVID-19 by highlighting which uncertainties most strongly affect the efficacy of these strategies (Wong et al., 2021).

Our research uses a compartment model to address the questions of how waning vaccine efficacy affects the manner at which COVID-19 propagates on college campuses. We choose a compartment model approach as it allows for large ensemble simulations to be run that enable increased accuracy in our results. We answer the question of, “With this waning immunity, what parameters are certain metrics of interest sensitive to?” to provide college campuses ideas for how to tailor ongoing campus prevention. We develop a model that describes a campus with two subpopulations, vaccinated and unvaccinated. The compartment model with subpopulations approach balances the computational efficiency provided by compartment models with the needed ability to investigate the differences between individuals that an ABM accounts for. We introduce a new model parameter that represents potentially changing vaccine efficacy. We investigate the interaction between the introduction of vaccines to a campus with the number of symptomatic infections and the maximum isolation population on campus. We look at the relationship between the isolation population and symptomatic infections with different testing strategies. Lastly, we complete a global sensitivity analysis to investigate the metrics of interests’ sensitivities to parameters.

2. Methods

2.1. Model compartments and classes

The basis of our model is the compartment model proposed in (Paltiel et al., 2020) which represents the semi-enclosed system of a college campus, with limited possible exchange of illness with the surrounding community. Similarly to models such as those proposed in (Lopman et al., 2021) and (Rǎdulescu et al., 2020), which investigate a medium sized university and a small scale college town respectively, we retain the model assumption of a relatively closed population to investigate the challenging environment of universities. We develop the model from (Paltiel et al., 2020) to represent a population that is split into subpopulations and add a time-dependent vaccine parameter to represent a potentially changing level of protection, additions which more accurately represent the current environment. These model developments allow us to investigate what was not previously possible in the Paltiel et al. model, specifically the efficacy with which vaccines can mitigate the spread of COVID-19 on campuses, and which uncertainties affect the efficacy of this mitigation strategy. Our model development of time-varying vaccine efficacy is responsive to recent work that has noted the importance of waning vaccine immunity (Chemaitelly & Abu-Raddad, 2022; Feikin et al., 2022; Tartof et al., 2021).

The model contains eight compartments: those who are not infected or are otherwise susceptible to new infections (uninfected, U); those who have tested positive but are not actually positive (false positive, FP); those who have been exposed to someone who is infected (exposed, E); those who are infected but not exhibiting symptoms (asymptomatic, A); those who are infected and are exhibiting symptoms (symptomatic, S); those who have tested positive and are infected (true positive, TP); those who have been infected and have since recovered (recovered, R); and those who have become infected and have died as a result of the infection (deceased, D). Each of these compartments fall into one of three compartment classifications: transmission and testing, isolation, and removed. The transmission and testing classification contains any individual who is spreading the virus on campus, those who can be infected by the virus, and/or those eligible for testing (uninfected, exposed, and asymptomatic compartments). The isolation pool classification contains all individuals who have been temporarily removed from the circulating population (true positive, false positive, and symptomatic compartments). The removed classification includes individuals who are permanently removed from the general campus population (recovered and deceased). The movement of individuals between compartments as well as the classifications of each compartment is depicted in the model schematic (Fig. 1).

Fig. 1.

The model schematic including the flow of individuals between compartments. Compartments highlighted in blue represent compartments in the transmission and testing class. Compartments highlighted in orange represent compartments contained in the isolation pool class, and compartments highlighted in gray represent the compartments contained in the removed class. Recovered individuals are still considered a part of campus but since they are assumed to no longer be spreading disease they are considered removed from the system.

2.2. Original model equations and parameters

The movement described in the model schematic (Fig. 1) is described within the model by a set of eight coupled differential equations. Our model conserves humans in the sense that throughout the model the campus population interacts with the community only through exogenous shocks. These exogenous shocks infect only individuals who are already a part of our campus population, thus our campus population remains the same throughout the model. The original model equations we used as a basis are from the model proposed in (Paltiel et al., 2020) (Equation Set (1)). These describe the movement of individuals on a college campus who reside in a single general campus population, but without the presence of vaccinations or preexisting immunity from prior infections.

$$\frac{dU}{dt}=-\beta U\frac{A}{U+A+E}-U\left(t-\Delta t\right)\cdot \tau \cdot \left(1-Sp\right)+\mu FP-X\cdot I$$ (1)

$$\frac{dE}{dt}=-\theta E+\left(\beta U\frac{A}{U+A+E}\right)+X\cdot I$$

$$\frac{dA}{dt}=A\cdot \left(-\sigma -\rho \right)-A\left(t-\Delta t\right)\cdot \tau \cdot Se+\theta E$$

$$\frac{dFP}{dt}=-\mu FP+U\left(t-\Delta t\right)\cdot \tau \cdot \left(1-Sp\right)$$

$$\frac{dTP}{dt}=TP\cdot \left(-\sigma -\rho \right)+A\left(t-\Delta t\right)\cdot \tau \cdot Se$$

$$\frac{dS}{dt}=S\cdot \left(-\rho -\delta \right)+\sigma \cdot \left(TP+A\right)$$

$$\frac{dR}{dt}=\rho \cdot \left(TP+A+S\right)$$

$$\frac{dD}{dt}=\delta S$$

The variables U, E, A, FP, TP, S, R, and D represent our eight compartments: uninfected, exposed, asymptomatic, false positive, true positive, symptomatic, recovered, and deceased respectively. The model parameters (Table 1 and Table 2) dictate how the populations flow through the compartments.

Table 1.

Parameters, their descriptions, their values or the distribution they are sampled from, and their units. Bin(n,p) denotes a binomial distribution with number of trials, n, and probability of success, p. NBin(n,p) represents a negative binomial distribution with number of successes n, and probability of success, p. Tri(a,b,c) denotes a triangular distribution with minimum, a, maximum, b, and peak location, c. Parameter prior distributions, other than how often testing occurs, are from (Wong et al., 2021) time to return a false positive is from (Paltiel et al., 2020).

Parameter	Description	Value or Prior Distribution Used	Units
ζ	Frequency of new infections from community spread and the interaction of students and staff with members in the community.	Bin(n = 14, p = 0.5)	days
X	Number of people who become infected every time an exogenous shock occurs.	NBin(n = 5, p = 0.25)	people
I	Indicator function which is 1 when an exogenous shock occurs and is 0 otherwise	1 or 0 depending on if an exogenous shock occurs	–
η	Number of days it takes from when an individual is initially infected to when they become fully recovered.	Tri(a = 10, b = 21, c = 14)	days
ξ	Percentage of people who are infected by COVID-19 who eventually develop symptoms.	Tri(a = 5, b = 50, c = 30)	%
Rt	Average number of cases caused by an infected individual.	Tri(a = 0.8, b = 2.5, c = 1.004)	–
φ	Ratio of people who are infected with COVID-19 and have symptoms to the number of fatalities from COVID-19.	Tri(a = 0, b = 0.01, c = 0.0005)	–
ψ	Number of days from exposure until symptoms may begin showing,	Tri(a = 3, b = 12, c = 5)	days
		Wong et al. (2021)
ω	Number of days it takes for a test to indicate that the individual's first test was a false positive.	1	days
		Paltiel et al. (2020)
Se	Accuracy in determining true positive cases (Baratloo et al., 2015).	Tri(a = 0.7, b = 0.9, c = 0.8)	–
Sp	Accuracy in determining true negative cases (Baratloo et al., 2015).	Tri(a = 0.95, b = 1, c = 0.98) Wong et al. (2021)	–
φ	How often testing occurs.	0, 7, 14, 30 or	days
		NBin(n = 5, p = 0.25)
		(for Sobol’ Sensitivity Analysis)
α	The amount of time steps per day	3	–

Table 2.

Parameters that are calculated using the original parameters including their symbol, description, equation, and unit (Paltiel et al., 2020).

Calculated Parameter Symbol	Description	Calculation	Units
$\tau$	Testing rate	$\frac{1}{\phi \cdot \alpha }$	1/time step
$\rho$	Recovery rate	$\frac{1}{\mathrm{\eta }\cdot \alpha }$	1/time step
$\sigma$	Rate of symptom onset for infected individuals	$\rho \frac{\xi }{1-\xi }$	1/time step
$\beta$	Infection rate	$R_t\left(\sigma +\rho \right)$	1/time step
$\delta$	Death rate	$\rho \frac{\phi }{1-\phi }$	1/time step
$\theta$	Rate of advancement from exposed to infected	$\frac{1}{\psi \cdot \alpha }$	1/time step
$\mu$	Rate that false positive individuals receive confirmatory test results and return to uninfected	$\frac{1}{\alpha \cdot \omega }$	1/time step

The parameters in Table 1 are either used directly in the equations or are used to calculate the remaining parameters (Table 2) that are present in the model equations.

2.3. Subpopulations and vaccinations

We implement subpopulations in the model by grouping the population into two categories by vaccination status (vaccinated and unvaccinated). Our model framework allows for these groups to be changed to represent other populations and expanded to account for more than just two subpopulations. The type of subpopulation can be changed by tailoring the parameter values, as each subpopulation has its own set of parameters (Table 1, Table 2, Table 3). This generalizable model framework could be applied to (for example) varying age groups, immunocompromised and non-immunocompromised individuals, or varying living environments, for example, on-campus and commuter students. Our model assumes that the subpopulations are disjoint and do not share members. Each model compartment is then split into two subcompartments, one per subpopulation, based on vaccination status. Vaccinated individuals include those who have received one of the three main vaccines approved in the U.S. (Pfizer-BioNTech, Moderna, and Johnson and Johnson). Unvaccinated individuals are considered those who have not received a vaccine.

Table 3.

New parameters introduced in the vaccines and subpopulations model development.

Parameter	Description	Value or Prior Distribution Used	Units
γi(t)	Ratio of infected vaccinated individuals versus the number of infected unvaccinated individuals	γV(t):	people
		Line interpolated from γV(0) to γV(N)
		γU(t):
		1
γi(0)	Initial ratio of the number of infected vaccinated individuals i to the number of infected individuals from the unvaccinated subpopulation	γV(0):	people
		Tri(a = 0, b = 1, c = 0.8) high risk VE,
		Tri(a = 0, b = 1, c = 0.5) medium risk VE,
		Tri(a = 0, b = 1, c = 0.2) low risk VE,
		γU(0):
		1
γi,MOD	Percentage γi(0) increases towards 1 (no vaccine immunity)	Tri(a = 0, b = 1, c = 0.3)	%
γi(N)	End value of the vaccine parameter for subpopulation i	${\gamma }_V\left(N\right)={\gamma }_V\left(0\right)+\left(1{-\gamma }_V\left(0\right)\right){\cdot \gamma }_{V,\mathit{MOD}}$	people
PV	Percent of the total population that is fully vaccinated.	95	%
PU	Percent of the total population that is unvaccinated.	5	%

We add a new parameter γ_i(t) to represent vaccinations. This parameter modulates the flow of vaccinated individuals from the uninfected compartment to the exposed compartment, from the asymptomatic compartment to the symptomatic compartment, and from the symptomatic compartment to the deceased compartment. The parameter γ_i(t) modulates from the uninfected compartment (U) to the exposed compartment (E) because all three vaccines approved in the U.S. have shown to protect against symptomatic infection, thereby decreasing overall infections. Similarly, we use γ_i(t) to modulate the flow from the symptomatic compartment (S) to the deceased compartment (D) as previous work has shown that unvaccinated persons have had at times 53.2 times the risk for COVID-19 associated death (Johnson, 2022). Likewise, studies conducted in the state of New York have also shown the vaccines decrease hospitalizations, thereby decreasing deaths (Johnson, 2022; Self et al., 2021). Lastly, we use γ_i(t) to modulate the flow of individuals from the asymptomatic compartment (A) to the symptomatic compartment (S) because the clinical trials for all three vaccines approved in the U.S. have shown a protection from symptomatic infection (Baden et al., 2021; Polack et al., 2020; Sadoff et al., 2021).

In the new set of equations, each state variable bears a subscript of i, which indexes the subpopulation. We take i = u to represent the unvaccinated subpopulation and i = v to represent the vaccinated subpopulation. The addition of the vaccine parameter γ_i(t) can be seen on the infection, symptom onset, and death terms (Equation Set (2)).

$$\frac{d{U}_i}{dt}=-{\beta }_i{\gamma }_i\left(t\right)U_i\frac{{\mathrm{\Sigma }}_i{A}_i}{{\mathrm{\Sigma }}_i{U}_i+{\mathrm{\Sigma }}_i{A}_i+{\mathrm{\Sigma }}_i{E}_i}-U_i\left(t-\Delta t\right)\cdot {\tau }_i\cdot \left(1-S{p}_i\right)+{\mu }_{i}F{P}_i-X_i\cdot I$$ (2)

$$\frac{d{E}_i}{dt}=-{\theta }_i{E}_i+\left({\beta }_i{\gamma }_i\left(t\right)U_i\frac{{\Sigma }_i{A}_i}{{\Sigma }_i{U}_i+{\Sigma }_i{A}_i+{\Sigma }_i{E}_i}\right)+X_i\cdot I$$

$$\frac{d{A}_i}{dt}=A_i\cdot \left(-{\sigma }_i{\gamma }_i\left(t\right)-{\rho }_i\right)-A_i\left(t-\Delta t\right)\cdot {\tau }_i\cdot S{e}_i+{\theta }_i{E}_i$$

$$\frac{dF{P}_i}{dt}=-{\mu }_{i}F{P}_i+U_i\left(t-\Delta t\right)\cdot {\tau }_i\cdot \left(1-S{p}_i\right)$$

$$\frac{dT{P}_i}{dt}=T{P}_i\cdot \left(-{\sigma }_i{\gamma }_i\left(t\right)-{\rho }_i\right)+A_i\left(t-\Delta t\right)\cdot {\tau }_i\cdot S{e}_i$$

$$\frac{d{S}_i}{dt}=S_i\cdot \left(-{\rho }_i-{\delta }_i{\gamma }_i\right)+{\sigma }_i{\gamma }_i\left(t\right)\cdot \left(T{P}_i+A_i\right)$$

$$\frac{d{R}_i}{dt}={\rho }_i\cdot \left(T{P}_i+A_i+S_i\right)$$

$$\frac{d{D}_i}{dt}={\delta }_i{\gamma }_i{S}_i$$

This model framework, developed to represent university campuses, could be tailored to represent other similar semi-enclosed populations. The most appropriate other contexts would be (1) more or less self-contained such that interaction with the general public can be generalized to exogenous shocks, (2) environments in which some individuals are being tested for COVID-19, and (3) environments in which infected individuals who have been determined from a positive test result or a display of symptoms are placed in isolation. A few possible environments that this work could be further applied to would be, cruise ships, prisons, detention centers, and nursing homes. These environments would require parameter values tailored to these contexts. Our model is also meant to represent a small enough time frame (one academic term) in which it is reasonable to assume those who have been infected with COVID-19 do not become re-infected. However, current research shows that those who have previously been infected with COVID-19 can eventually be reinfected (CDC, 2023). Thus, to extend our model to describe COVID-19 spread in a general community, further model developments would be needed.

2.4. Metrics of interest

We focus on two metrics of interest to investigate the effects of the experiments as well as quantify risk. These are the maximum isolation population reached during the simulation and the total number of symptomatic infections over the course of an 80-day semester. We choose the maximum isolation population because college campuses will only have a finite amount of space available in their isolation housing. The number of symptomatic infections serves as a useful metric to characterize the overall presence of COVID-19 on a college campus.

2.5. Model validation

For validation of our model, we implement a benchmark test based on the model from (Paltiel et al., 2020). We verify that the initial code of our model matched the model in (Paltiel et al., 2020) before adding in the new developments of subpopulations and the vaccine parameter. The original Paltiel model had a corresponding dashboard (Aravamuthan et al., 2020) that portrayed visuals of model outputs and important tallies, such as the cumulative infections. A benchmark test was then created that compares our model outputs to the outputs of the dashboard (Aravamuthan et al., 2020) based on the model from (Paltiel et al., 2020). This benchmark was run after each addition to the model to make sure that the base code continued to match the original model dashboard (Aravamuthan et al., 2020).

2.6. Vaccine efficacy

After the benchmark experiment verifies that our original model framework matches the Paltiel dashboard (Aravamuthan et al., 2020), we examine the importance of vaccine efficacy (VE) on our metrics of interest. We conduct a set of experiments by varying the initial value of γ_V(t) across three vaccine efficacy scenarios: a high-risk scenario (VE of 20%), a medium-risk scenario (VE of 50%), and a low-risk scenario (VE of 90%). The VE values are calculated using the definition of vaccine efficacy (Appendix A). The low- and medium-risk scenarios represent the span of the range of vaccine efficacies seen in the vaccines approved for use in the U.S. (Baden et al., 2021; Polack et al., 2020; Sadoff et al., 2021). The high-risk scenario represents a VE that is lower than all approved vaccines in the U.S.

We introduce the vaccine efficacy parameter, γ_i(t), into the model to represent both steady and decreasing vaccine efficacy. For steady vaccine efficacy the initial vaccine efficacy that is sampled, γ_V(0), is equal to the ending vaccine efficacy, γ_V(N), where N is the number of timesteps. For decreasing vaccine efficacy, the initial vaccine efficacy is sampled from the prior distribution corresponding to the VE scenario chosen. A second value is sampled from a triangular distribution with a lower bound at 0, an upper bound at 1, and peaked at 0.3. This value, γ_V,MOD, indicates the percentage that γ_V(t) increases towards one, representing a decrease in vaccine efficacy. We leverage a triangular distribution peaked at 0.3 because current research has found a decrease in vaccine efficacy to be approximately 20%–40% (Feikin et al., 2022; Horne et al., 2022; Nordström et al., 2022; Tartof et al., 2021). Research currently focuses on decreasing efficacy against infection and symptomatic infection and research that includes decreased immunity against severe COVID-19 shows a significantly smaller decrease than that against symptomatic infection and overall infection (Feikin et al., 2022; Horne et al., 2022; Tartof et al., 2021). Due to this, decreasing vaccine efficacy is implemented only on the infection and symptom onset terms and not the death term. The ending vaccine efficacy, γ_V(N), is then calculated using the values sampled for γ_V(0) and γ_V,MOD (Eq. (3)).

$${\mathrm{\gamma }}_V\left(N\right)={\mathrm{\gamma }}_V\left(0\right)+\left(1-{\mathrm{\gamma }}_V\left(0\right)\right){\cdot \mathrm{\gamma }}_{V,\mathit{MOD}}$$ (3)

The remaining γ_V(t) values are calculated by interpolating between the initial and ending vaccine efficacy values to get a γ_V(t) value for every timestep. For this paper we use decreasing vaccine efficacy in all experiments.

2.6.1. Vaccine efficacy experiment

For this experiment we run three different versions of the model, one for each level of risk scenario. For each scenario the model is run for 10,000 simulations, the parameters are sampled using Latin hypercube sampling (LHS), the surveillance testing of the unvaccinated subpopulation is fixed at no testing, and all metrics of interest are collected. For each risk scenario the initial value of γ_V(t) is sampled from a triangular distribution with a lower bound of zero, an upper bound of one, and with peaks at 0.1, 0.5, and 0.8 for the low-, medium-, and high-risk scenarios respectively.

2.6.2. Decreasing vaccine efficacy experiment

To establish the effects of decreasing versus steady vaccine efficacy, we run an experiment to investigate the difference in effects from a vaccinated population who is experiencing waning immunity versus one experiencing steady immunity. For the steady vaccine efficacy portion of this experiment, the model is run for 10,000 simulations and γ_V(t) is sampled from the medium risk prior, which is centered at a vaccine efficacy of 50%. The parameters are then sampled using LHS. For the decreasing vaccine efficacy portion, the model is run for 10,000 simulations, the initial value for γ_V(t) is sampled from the medium risk prior, and the parameters are sampled using LHS. Both metrics of interest are collected for comparison.

2.7. COVID-19 testing experiment

Another topic we investigate is surveillance testing of the unvaccinated subpopulation. Results from (Paltiel et al., 2020) demonstrated the importance of surveillance testing in order to limit cumulative infection counts and isolation population size. We further investigate these findings with the addition of vaccinations to see if the frequency of testing remains of high importance when 50% of the campus population is vaccinated. Within this paper any reference of surveillance testing is only in reference to the unvaccinated subpopulation. As in the original model of Paltiel et al. (2020), surveillance testing is assumed to occur at regular intervals. We investigate four frequency of testing scenarios: daily testing, weekly testing, monthly testing, and no testing.

We conduct an experiment that investigates how the frequency of administering COVID-19 screening tests to the unvaccinated subpopulation affects the metrics of interest. Similarly, this investigates how different testing strategies can mitigate the spread of COVID-19 on college campuses. This experiment simultaneously allows for the investigation of the relationship between the unvaccinated and vaccinated subpopulations. An attribute that is possible due to our model's development of subpopulations. For this experiment, the medium risk scenario is used to sample the initial γ_V(t) value, the model is run for 10,000 simulations for each testing frequency, and the remaining parameters are sampled using LHS.

2.7.1. Percentage of population vaccinated experiment

From the previous experiment investigating COVID-19 testing frequencies, the question that logically follows is how much of the population must be vaccinated for the frequency of testing of unvaccinated individuals to have minimal effect on symptomatic infections? We investigate this by varying the percentage of the population that is vaccinated as well as the frequency of testing in the following experiment.

For this experiment, we vary the percentage of the population that is vaccinated using the following percentages: 10%, 30%, 50%, 70%, 80%, and 90%. Each scenario that corresponds to a different percentage of the population that has received vaccines is run with each frequency of testing. For each scenario the model is run for 10,000 simulations and LHS is used to sample the parameters. We examine the impact on the number of symptomatic infections.

2.8. Sobol’ sensitivity analysis

For the final experiment we implement a variance-based global sensitivity analysis (Sobol’, 1967) to investigate the metrics of interest, which are model outputs, sensitivities to the parameters. We leverage a Sobol’ sensitivity analysis due to the number (12) of uncertain parameters in our model as well as the non-linear nature of our model. We use this sensitivity analysis to compute the first order and total sensitivities for the total number of symptomatic infections and total maximum isolation population (summing over both the vaccinated and unvaccinated subpopulations). The first order sensitivities (Eq. (4)) represent the percentage of the total model output variance that is attributed to an individual model input (Butler et al., 2014; Reed et al., 2022). For the sensitivities, V represents the variance, E represents expectation, x_i represents the model input of concern, x _∼ i represents all model inputs that are not x_i, and y represents the model output (Butler et al., 2014; Reed et al., 2022).

$$S_i^1=\frac{V_{x_i}\left[E_{x_{\sim i}}\left(x_i\right)\right]}{V\left(y\right)}$$ (4)

The second type of sensitivity that we calculate is the total sensitivities of each of our uncertain model inputs. This total sensitivity (Eq. (5)) represents the total amount that a particular model input affects the model outputs, both by itself and through its interactions with other model inputs (Butler et al., 2014; Reed et al., 2022).

$$S_i^T=\frac{E_{x_{\sim i}}\left[V_{x_i}\left(x_{\sim i}\right)\right]}{V\left(y\right)}=1-\frac{V_{x_{\sim i}}\left[E_{x_i}\left(x_{\sim i}\right)\right]}{V\left(y\right)}$$ (5)

We use Latin hypercube sampling to sample all parameters and run the model for approximately 920,000 simulations. We then use the Sobol’ sensitivity analysis to find each parameter's total and first order sensitivities in regard to the maximum isolation population and symptomatic infections.

3. Results and discussion

3.1. Impacts of waning vaccine efficacy

This experiment investigates the differing effects of steady and decreasing vaccine efficacy on the main metrics of interest: maximum isolation population and symptomatic infections. For all metrics of interest, the density curves shift to the right for the vaccinated subpopulation (Fig. 2B, D) in the case of decreasing vaccine efficacy. We see increases in both means and 90th percentiles (P90) in the decreasing vaccine efficacy cases (Fig. 2, orange curves). For the vaccinated symptomatic infections (Fig. 2D) the mean percentage of the vaccinated subpopulation that becomes symptomatically infected increases from 1.4% with steady vaccine efficacy to 2.0% with decreasing vaccine efficacy (a 43% increase in symptomatic infections). The changes to the upper tail (P90) mirror this increase as well, increasing from 3.2% with steady vaccine efficacy to 4.3% with decreasing vaccine efficacy (a 34% increase). For the maximum vaccinated isolation population (Fig. 2B), the mean increases from 0.44% to 0.70% from steady to decreasing vaccine efficacy respectively (a 59% increase). This is reflected in the P90 which increases from 1.0% for steady vaccine efficacy to 1.5% for decreasing vaccine efficacy (a 50% increase).

Fig. 2.

Estimations of the Probability Density Function (PDF) for the metrics of interest, symptomatic infections and the maximum isolation population, for both the vaccinated and unvaccinated subpopulations. The blue curves represent the density curves for steady vaccine efficacy. The orange curves represent the density curves for decreasing vaccine efficacy. Panels A and B represent the maximum isolation population and panels C and D represent symptomatic infections. Panels enclosed in an orange rectangle belong to the unvaccinated subpopulation and panels enclosed in the blue rectangle belong to the vaccinated subpopulation.

The metrics of interest for the unvaccinated subpopulation similarly increase in mean in the case of decreasing vaccine efficacy as compared to constant vaccine efficacy (Fig. 2A, C). For example, the mean percentage of unvaccinated individuals who become symptomatically infected (Fig. 2C) increases from 3.1% with steady vaccine efficacy to 3.4% with decreasing vaccine efficacy (a 10% increase). The maximum unvaccinated isolation population (Fig. 2A) has an increase in means from 0.97% with steady vaccine efficacy to 1.10% with decreasing vaccine efficacy (a 13% increase). This same pattern is seen in the P90. P90 for unvaccinated symptomatic infections increases from 6.4% to 7.0% for steady and decreasing vaccine efficacy respectively (an 9% increase). The P90 for the maximum unvaccinated isolation population increases from 2.0% to 2.3% for steady and decreasing vaccine efficacy respectively (a 15% increase).

Increases in both means and P90 are found in all metrics of interest for both the vaccinated and unvaccinated subpopulation when vaccine efficacy changes from steady to decreasing. However, the increases are much smaller for the unvaccinated subpopulation than the vaccinated subpopulation. For the vaccinated subpopulation, we see a 43% increase in symptomatic infections when waning vaccine efficacy occurs in comparison to a 10% increase for the unvaccinated subpopulation. Similarly, the maximum isolation population for the vaccinated subpopulation has a 59% increase in the means where the unvaccinated subpopulation has a 13% increase. This is expected, as the population that does not have the vaccine will be less affected by the changes in the vaccine's efficacy in comparison to the subpopulation who has the vaccine. This slight change in means and P90 for both unvaccinated metrics of interest can be attributed to the vaccinated population. For this experiment, the vaccinated subpopulation makes up 95% of the total population (4750 people) where the unvaccinated subpopulation makes up only 5% (250 people). Thus, it is feasible that as the percent of the vaccinated subpopulation who is symptomatically infected or contributing to the maximum isolation population increases that there are more individuals within the system who are able to infect others. This translates to more unvaccinated individuals becoming infected, leading to slightly higher percentages of the unvaccinated population becoming symptomatically infected and isolated. Since the unvaccinated subpopulation only contributes to 5% of the population, however, this increase remains small.

Overall, between both the vaccinated and unvaccinated subpopulations when waning immunity occurs there is a 33% increase in the mean percentage of the total population (5000 people) who become symptomatically infected, which corresponds to 28 more symptomatic infections in the model population. The mean percentage of the total population that contributes to the maximum isolation population increases by 49%, corresponding to an increase of 12 individuals in the model. This demonstrates the significant increase in the metrics of viral presence on campus that occurs when the efficacy of a vaccine wanes.

3.2. Implications of the efficacy of the vaccine

In this experiment, the impacts of varying the efficacy of the vaccine are investigated and as expected, as the VE decreases, the density curve shifts farther to the right (Fig. 3). This leaves the highest risk VE scenario with the farthest right position on the plot. With the density curves being shifted to the right, this corresponds to the highest risk VE scenario having the highest mean and P90. For the maximum vaccinated isolation population (Fig. 3B), the mean and P90 for the low-risk VE scenario are 0.52% and 1.20% of the subpopulation, respectively. These increase to 0.85% and 1.94% respectively for the high-risk scenario (a 63% increase in means and 62% increase in P90). For the vaccinated symptomatic infections (Fig. 3F), the mean and P90 in the low-risk VE scenario are 1.50% and 3.28% respectively. The mean and P90 increase to 2.50% and 5.40% respectively in the high-risk VE scenario (a 67% increase in means and 65% increase in P90).

Fig. 3.

Estimations of the Probability Density Function (PDF; A, B, E, and F) and their corresponding survival plots (C, D, G, and H) for all metrics of interest for both vaccinated and unvaccinated subpopulations. On the kernel density plots the blue line represents the low-risk VE from the low-risk scenario, orange represents the medium risk VE, and gray represents the high-risk VE. Plots corresponding to the unvaccinated subpopulation are contained within the orange rectangle and plots corresponding to the vaccinated subpopulation are contained within the blue rectangle. Panels A, B, C, and D correspond to the maximum isolation population and panels E, F, G, and H correspond to symptomatic infections.

There is a similar trend in the unvaccinated metrics of interest. We see again that the highest risk VE scenario is shifted to the right and is the farthest right density curve (Fig. 3A and E). This is representative of the means and P90 increasing as the risk of the VE scenario increases. For the maximum unvaccinated isolation population (Fig. 3A), the mean increases from 1.0% for the low-risk VE scenario to 1.2% for the high-risk VE scenario (a 20% increase). This same behavior is represented in the P90 which increases from 2.1% for the low-risk VE scenario to 2.7% for the high-risk VE scenario (a 29% increase). Similar behavior is seen for the unvaccinated symptomatic infections (Fig. 3E) whose means and P90 increase from 3.1% and 6.5% respectively for the low-risk VE scenario to 3.7% and 7.7% for the high-risk VE scenario (a 19% increase in means, an 18% increase in P90).

However, there is less change between means and P90 for the unvaccinated subpopulation than the vaccinated subpopulation (Fig. 3B and F). For the maximum vaccinated subpopulation, the mean percentage of the population that makes up the maximum isolation population increases by 63% from low to high risk, where the unvaccinated subpopulation only sees an increase of 20%. For the symptomatic infections, the mean percentage of the vaccinated subpopulation that becomes symptomatically infected increases by 67% from the low-risk scenario to the high-risk scenario, where the unvaccinated subpopulation sees an increase of 19%. Similarly, to the above experiment, this can be attributed to the effects of the vaccinated subpopulation on the unvaccinated subpopulation. When the percentage of vaccinated individuals who are contributing to the maximum isolation population and who are symptomatically infected increases, this indicates an increase in the overall number of individuals who are infected. Thus, if there are more infected vaccinated individuals, there are more individuals circulating in the system who are able to infect any individuals. This, in turn, leads to an increase in the percentage of individuals in both metrics of interest for both subpopulations. This change remains smaller than the vaccinated subpopulation as the vaccinated subpopulation makes up 95% of the overall population and the unvaccinated subpopulation only holds 5% of the overall population.

Overall, for the total population (5000 people), from the low-risk scenario to the high-risk scenario there is an 61% increase in the mean percent of the population contributing to the maximum isolation population. This corresponds to an increase of 17 individuals. For the overall symptomatic infections there is a total increase of 56%, which is an increase of 47 individuals. This demonstrates the significant impact the initial efficacy of a vaccine has. The change from a low vaccine efficacy to a high vaccine efficacy can decrease the mean percentage of the population developing symptomatic infections by approximately 56%.

3.3. Influence of surveillance testing

We analyze the density curves and survival plots for symptomatic infections and maximum isolation population for different frequencies of surveillance testing of the unvaccinated subpopulation. We do this for both the vaccinated and unvaccinated subpopulations. For the maximum unvaccinated isolation population (Fig. 4A), when no testing occurs the mean percentage of the subpopulation contributing to the maximum isolation population is 1.3% and the P90 is 3.0%. When surveillance testing is increased to monthly and weekly surveillance testing, the means increase to 1.9% and 2.2% respectively. Similarly, when the testing frequency is increased to monthly and weekly testing, the P90 increases to 3.8% and 4.1% respectively. Then, when the most frequent surveillance testing is implemented, daily testing, the mean is 3.8% and the P90 is 5.5% (an overall 192% increase in means, 83% overall increase in P90).

Fig. 4.

Estimated PDFs (A, B, E, and F) and their corresponding survival plots (C, D, G, and H) for the four testing scenarios in context of each of the metrics of interest. The vertical black lines in the first and third row of panels (A and E) represents the thresholds for which all the vaccinated estimated PDFs peak to the left of. The blue lines correspond to daily testing of unvaccinated individuals, orange to weekly testing, light gray to monthly testing, and dark gray to no testing. Graphs within the orange rectangle correspond to the unvaccinated subpopulation and graphs within the blue rectangle correspond to the vaccinated subpopulation. Panels A, B, C, and D correspond to the maximum isolation population and panels E, F, G, and H correspond to symptomatic infections.

This increase of 192% in the mean unvaccinated maximum isolation population from the least frequent testing scenario (no testing) to the most frequent (daily testing) corresponds to an increase of 62 people. This 192% increase in means and 83% increase in P90 reflects that as more frequent surveillance testing occurs, more individuals will be detected with COVID-19 than would be detected if less frequent testing occurred. Thus, as more infected individuals are detected they will be isolated, thereby increasing the maximum unvaccinated isolation population. These are infections that would otherwise go undetected, save for those cases developing symptoms.

A positive impact on the unvaccinated symptomatic infections, vaccinated symptomatic infections, and the maximum vaccinated isolation population (Fig. 4E, F, and B) is demonstrated as the frequency with which unvaccinated individuals are tested is increased. As the surveillance testing is increased the peaks of the density curves shift left in tandem, with the most frequent testing scenario, daily testing, presenting as the farthest left peaked density curve. For unvaccinated symptomatic infections (Fig. 4E), when no surveillance testing is occurring, the mean is 4.0% and the P90 is 8.5%. For daily testing the mean decreases to 1.8% and the P90 to 3.5% (a 55% decrease in means, a 59% decrease in P90). For the vaccinated symptomatic infections (Fig. 4F) the mean decreases from 2.2% of the vaccinated subpopulation with no testing to 1.1% with daily testing (a 50% decrease). The P90 displays a similar pattern and decreases from 5.0% with no surveillance testing to 2.2% with daily testing (a 56% decrease). This continues with the maximum vaccinated isolation population (Fig. 4B). When no surveillance testing occurs, the mean is 0.80% of the vaccinated subpopulation. An increase to daily testing corresponds to the mean decreasing to 0.30% (a 63% decrease). The P90 decreases from 1.80% to 0.60% (a 67% decrease) for no surveillance testing and daily surveillance testing respectively.

As the frequency of testing of unvaccinated individuals increases, we see a decrease in the mean percentage of unvaccinated (55%) and vaccinated (50%) symptomatic infections and the maximum vaccinated isolation population (63%). These correspond to a decrease in 55, 27, and 12 people respectively. This represents the relationship that exists between the vaccinated and unvaccinated subpopulations and the effect of these otherwise undetected infections on the vaccinated subpopulation. The act of only testing unvaccinated individuals benefits both the vaccinated and unvaccinated subpopulations. This relationship between the two subpopulations is only visible due to the additions of subpopulations in the model.

3.4. Outcomes of increasing the percentage of vaccinated individuals

Both the unvaccinated and vaccinated symptomatic infections (Fig. 5A and B) demonstrate an increase in the mean when the frequency of testing is decreased. This increase is reduced when the percent of the population vaccinated increases. For the unvaccinated subpopulation (Fig. 5A) when 10% of the population is vaccinated and daily testing occurs the mean percentage of the subpopulation that is becoming infected is 1.1%. When the testing frequency is decreased to no testing the mean increases to approximately 4.8%. Similarly, for the vaccinated symptomatic infections (Fig. 5B), when 10% of the population is vaccinated, the mean percentage of vaccinated individuals becoming symptomatically infected when daily surveillance testing occurs is 0.70%. When surveillance testing decreases to no surveillance testing, the mean percentage of vaccinated individuals becoming symptomatically infected is 2.6%. When 80% of the overall population is vaccinated and daily surveillance testing occurs, the mean percentage of unvaccinated individuals (Fig. 5A) who become symptomatically infected is 2.6%. This mean increases to approximately 3.6% when no surveillance testing occurs. When 80% of the population is vaccinated and daily testing occurs there is a mean of 1.5% of the vaccinated subpopulation (Fig. 5B). When no surveillance testing occurs the mean increases to 2.0%.

Fig. 5.

Panel A shows the unvaccinated symptomatic infections for each testing scenario for each of the different percentages of the population vaccinated. Panel B shows this for the vaccinated symptomatic infections. The boxes corresponding to daily testing are denoted in blue, weekly testing is denoted in yellow, monthly testing in green, and no testing in orange. The lines in the boxes denote the median, the whiskers denote the minimum and maximum values, and the individual points denote the outliers.

As expected, this steady increase in symptomatic infections as frequency of testing decreases occurs in all six of the cases for the size of the vaccinated subpopulation. However, in the cases where 80% or more of the population is vaccinated, the decrease is significantly less. Specifically, when 80% of the population is vaccinated the increase, from daily to no testing, is less than or equal to a one percentage point increase for both subpopulations. This is further demonstrated by the increase in the number of infections. When 10% of the population is vaccinated there is an increase of 175 infections overall. When 80% of the population is vaccinated, there is an increase of only 30 infections. This shows the impact that testing has when less than approximately 80% of the population is vaccinated and stresses the importance of either vaccinating 80% of students or keeping the frequency with which unvaccinated individuals are tested, high.

3.5. Sobol’ sensitivity analysis

We compute first order and total Sobol’ sensitivity indices for all of the uncertain model parameters (Table 1, Table 2, Table 3). For the maximum isolation population (Fig. 6A) and symptomatic infections (Fig. 6B) the three parameters with the largest first order and total sensitivities are the reproductive number (average number of cases caused by an infected individual), R_t, the initial value of the vaccine efficacy parameter, γ_V(0), and the percentage of the infected population advancing to symptoms. The large total sensitivity for the reproductive number, for both metrics, indicates that R_t has a large number of interactions with other parameters. R_t is directly used in the calculation of the infection rate, $\mathrm{\beta }$. The infection rate governs the flow of individuals from the susceptible compartment to the exposed compartment from which they then interact with most of the other parameters. The infection rate also interacts with the vaccine efficacy parameter, ${\mathrm{\gamma }}_V\left(t\right)$, as it directly modulates how many individuals are being infected. This high total sensitivity highlights the practical importance of taking measures to control R_t in order to manage the spread of COVID-19 among the campus population (e.g., through use of masks or contact tracing programs). However, for the symptomatic infections’ metric (Fig. 6B), γ_V(0) and the percentage of the infected population advancing to symptoms have total sensitivities close to R_t. This indicates that in addition to measures to control R_t, effective vaccines are another critical tool to limit the numbers of symptomatic infections on campus.

Fig. 6.

This figure depicts the total sensitivities (S_T) in blue, first order sensitivities (S₁) in orange, and the 90% confidence intervals in black for each parameter in regard to maximum isolation population in panel A and symptomatic infections in panel B.

For both sensitivity metrics, the new infections per exogenous shock parameter has total sensitivity values that are significantly lower than the top three parameters but first order sensitivities that are close to the first order sensitivity values of the top three parameters. That the new infections per exogenous shock parameter has a strong direct influence on both the isolation population and the number of symptomatic infections is unsurprising given that this parameter transfers individuals directly to the exposed model compartment at regular intervals. This is a requisite step in order for individuals to later arrive in the symptomatic or asymptomatic compartments, making them candidates for either the isolation pool (Fig. 6A) or symptomatic infection (Fig. 6B).

For both metrics, the parameters with sensitivities that are not statistically significant are the frequency of screening, test specificity, test sensitivity, and symptomatic case fatality ratio. This indicates that these four parameters do not have a significant impact on these metrics of interest. However, this does not indicate that these parameters lack importance overall. Rather, in this context, conditioned on the parameters we have chosen to include, and the chosen sampling distributions for those parameters, they do not demonstrate significant impact. This could change depending on the context of the model, other potential sets of parameters to vary, or distributions for those parameters. The presence of the frequency of screening parameter in the non-significant sensitivity group of parameters may seem counterintuitive at first. However, this is consistent with our earlier result (Sec. 3.4) that symptomatic infections are dramatically reduced once at least approximately 80% of the campus is vaccinated. These Sobol’ sensitivity analyses were run with a population makeup of 95% vaccinated and 5% unvaccinated, so it follows that the symptomatic infections would not be sensitive to the frequency of screening parameter.

While the initial efficacy of the vaccine is in the top three for both sensitivity metrics, the sensitivities corresponding to γ_V,MOD are relatively low. This indicates that while the maximum isolation population and number of symptomatic infections are somewhat sensitive to the amount by which the vaccine efficacy diminishes over the course of the model semester, they are more sensitive to the initial vaccine efficacy. This demonstrates that campuses should prioritize preventative measures shown to increase the protection of vaccines, such as vaccinations for unvaccinated individuals and booster vaccines, at the beginning of the semester for students (Andrews, Stowe, Kirsebom, Toffa, Rickeard, et al., 2022; Andrews, Stowe, Kirsebom, Toffa, Sachdeva, et al., 2022; Atmar et al., 2022; Bar-On et al., 2021).

4. Conclusion

In this work we describe a mathematical model of COVID-19 spread on college campuses that includes the new developments of subpopulations and vaccines with waning efficacy. These model developments yield a model that better describes the on-the-ground conditions that campuses are faced with. We then implement this model to investigate the impacts of waning vaccine efficacy, the implications of the efficacy of the vaccine, the influence of surveillance testing, the outcomes of increasing the percentage of the population that is vaccinated, and the sensitivity of the metrics of interest to parameters. We find that relative to a scenario with constant vaccine efficacy, decreasing vaccine efficacy leads to a 33% increase in the mean percentage of symptomatic infections and a 49% increase in the mean percentage of the population contributing to the maximum isolation population. Through our experiments with low- or high-risk vaccine efficacy scenarios, we highlight that highly effective vaccines can dramatically reduce the number of individuals who become symptomatically infected and/or are put in isolation.

Importantly, our implementation of subpopulations enables this model to examine the degree to which the two subgroups, vaccinated and unvaccinated, interact in the model world. We observe that increasing the frequency of testing of the unvaccinated subpopulation leads to sizable decreases in the numbers of symptomatic infections in both the vaccinated (50% decrease) and unvaccinated (55% decrease) subpopulations. This result also underscores the importance of frequently testing the unvaccinated subpopulation in order to control overall numbers of infections. However, we also find that once the vaccinated subpopulation makes up at least about 80% of the total population, changing the frequency with which surveillance testing is applied to the unvaccinated subpopulation results in less than or equal to a one percentage point difference in the mean percentage of symptomatic infections in each subpopulation. Finally, the sensitivity analysis highlights the practical importance of beginning the semester with high vaccine efficacy through vaccinations and booster doses, as well as taking steps to reduce the transmission rate through, for example, vaccinations, masking, and the isolation of infected individuals.

Declaration of competing interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Handling Editor: Dr He Daihai

Appendix A. Supplementary data

The following is the Supplementary data to this article: