Stringency of containment and closures on the growth of SARS-CoV-2 in Canada prior to accelerated vaccine roll-out

Abstract

Background

Many studies have examined the effectiveness of non-pharmaceutical interventions (NPIs) on SARS-CoV-2 transmission worldwide. However, less attention has been devoted to understanding the limits of NPIs across the course of the pandemic and along a continuum of their stringency. In this study, we explore the relationship between the growth of SARS-CoV-2 cases and an NPI stringency index across Canada before the accelerated vaccine roll-out.

Methods

We conducted an ecological time-series study of daily SARS-CoV-2 case growth in Canada from February 2020 to February 2021. Our outcome was a back-projected version of the daily growth ratio in a stringency period (i.e., a 10-point range of the stringency index) relative to the last day of the previous period. We examined the trends in case growth using a linear mixed-effects model accounting for stringency period, province, and mobility in public domains.

Results

Case growth declined rapidly by 20–60% and plateaued within the first month of the first wave, irrespective of the starting values of the stringency index. When stringency periods increased, changes in case growth were not immediate and were faster in the first wave than in the second. In the first wave, the largest decreasing trends from our mixed effects model occurred in both early and late stringency periods, depending on the province, at a geometric mean index value of 30⋅1 out of 100. When compared with the first wave, the stringency periods in the second wave possessed little association with case growth.

Conclusions

The minimal association in the first wave, and the lack thereof in the second, is compatible with the hypothesis that NPIs do not, per se, lead to a decline in case growth. Instead, the correlations we observed might be better explained by a combination of underlying behaviors of the populations in each province and the natural dynamics of SARS-CoV-2. Although there exist alternative explanations for the equivocal relationship between NPIs and case growth, the onus of providing evidence shifts to demonstrating how NPIs can consistently have flat association, despite incrementally high stringency.

Introduction

Throughout 2020, non-pharmaceutical interventions (NPIs) were the primary tools employed by governments and public health agencies to slow the spread of SARS-CoV-2 (Ferguson et al. 2020, Koo et al., 2020). In Canada, as in many other countries, common NPIs included border closures, bans on non-essential travel, and mandatory physical distancing measures (McCoy et al, 2020). However, in contrast to many countries—particularly those in Europe—the authority and responsibility to implement these policies fall on provincial and territorial governments, meaning there is no formally coordinated response between them (Cameron-Blake et al., 2021). Instead, provincial and territorial responses, like any other large-scale intervention, were contingent upon local political and social contexts, and thus contained substantial variability (McCoy et al, 2020). These potentially divergent approaches create unique opportunities when seeking to evaluate Canada's pandemic responses in a systematic way.

Composite measures—which combine a set of indicators into an index—help abstract away from the subtleties at the sub-national level (Hale et al., 2021). The main strengths of composite indicators are that they 1) allow for systematic comparisons across different jurisdictions and 2) permit quantitative comparisons between the “intensity” of government responses and spread of infection (Hale et al., 2021). These strengths inherently contain the concept of dose, which can be broadly understood as the “amount” of intervention provided to a target population (Rowbothum et al., 2019). However, most existing research focuses on isolating the effect of common individual NPIs (Haug et al., 2020, Flaxman et al., 2020, Liu et al., 2021, Li et al., 2020, Hsiang et al., 2020, Bendavid et al., 2020, Küchenhoff et al., 2021, Berry et al., 2021).

The few studies that have used composite measures show a dose-response association between incrementally stringent interventions and the value of either the effective reproductive number (Turbé et al., 2021) or the cumulative incidence (Mezencev and Klement, 2021) of SARS-CoV-2. However, an important challenge with these analyses is that they have focused on the early periods of the pandemic—specifically, Europe's first wave—and they have not recognized that outbreak dynamics are time-varying, where slowed transmission will occur without the influence of any NPI (Bendavid et al., 2020).

Even with approved COVID-19 vaccines, NPIs are thought to remain a crucial component of SARS-CoV-2 control strategies (Moore et al., 2021, Gozzi et al., 2021). Thus, quantifying dose and dose-response-like relationships through composite measures will be useful for assessing if NPIs correlate with a decline in the growth of SARS-CoV-2. It will also enhance our understanding of the different types and timing of NPI “packages,” not only across regions within a country but across the temporal span of the pandemic. In this paper, we explore the relationship between a stringency index of SARS-CoV-2 mitigation strategies and the growth of confirmed cases across 5 provinces in Canada, throughout2 epidemic “waves”: spring 2020 (wave 1) and fall-winter 2020/2021 (wave 2).

Methods

The conceptual model behind our approach is that before meaningful levels of population immunity, individual behavior drives transmission and that the “packages” of government-mandated NPIs encourage individual behavior change (Bendavid et al., 2020). We assumed that case growth would incrementally exhibit larger decreases over time as the stringency index increased: If “less stringent” measures provide small nudges to individual behavior, “more stringent” measures will produce larger behavioral effects, and thus larger reductions in the growth of new cases. Specifically, we anticipated these effects to either manifest early—if initial responses were “highly stringent”—or over time—as pandemic responses intensified.

Study Design & Surveillance Data

We conducted a longitudinal study of daily SARS-CoV-2 incidence during the first and second waves in the Canadian provinces of Alberta, British Columbia, Manitoba, Ontario, and Quebec, which comprised 95% of all cases in Canada by the start of March 2021. The data used in these analyses cover the interval February 28, 2020 to February 15, 2021.

We obtained the count of daily laboratory-confirmed cases from provincial surveillance databases to calculate the growth ratio of SARS-CoV-2: in Alberta, from Alberta Health's Interactive Aggregate Data on COVID-19 (Alberta Health, 2020); in British Columbia, from the British Columbia Centre for Disease Control's COVID-19 dashboard (British Columbia Centre for Disease Control, 2021); in Manitoba, from the Province of Manitoba's Interactive Dashboard on COVID-19 (Manitoba Health 2021); in Ontario, from the ICES (formerly the Institute of Clinical and Evaluative Sciences); and in Québec, from the Institut national de santé publique du Québec (Institut national de santé publique du Québec 2021). In contrast to the other four provinces, Québec's case counts also included cases without laboratory confirmation that were epidemiologically linked to known SARS-CoV-2 positives.

Start and end dates of the first wave coincide with each province accumulating more than 10 cases and the last day before the relaxation of restrictions began, respectively; for the second wave, start and end dates coincide with rising case counts in early to late summer through to the trough in case counts after the fall-winter peak. We excluded all province-days with fewer than 10 cumulative cases from this analysis because early case counts could have been driven by imported cases rather than local transmission.

Data on Non-Pharmaceutical Interventions

We used data on all NPIs from the Oxford COVID-19 Government Response Tracker (OxCGRT, Hale et al., 2021). The project has tracked government policies and interventions across a standardized series of indicators for over 180 countries and includes subnational jurisdictions within Canada. The tool contains 8 categories on containment and closures, 4 on economic policies, and 8 on health system policies. Most indicators are reported on monotonic ordinal scales, with others coded on continuous scales, allowing for quantitative analysis of the degree of government response (Hale et al., 2021).

The containment and closures indicators used in this analysis reflect the more restrictive NPIs and include school and workplace closures, cancelling public events, restricting gathering sizes, closing public transportation, restrictions on internal movement, and travel controls. A score for each indicator is created by taking an ordinal value for that indicator and adding an extra 0⋅5 points if the policy applies across the entire jurisdiction as opposed to a specific locality. Each indicator is then re-scaled by its maximum ordinal value and summed to create a stringency index with scores between 0 and 100.

Mobility Data

We used data from Google's Community Mobility Reports (COVID-19 Community Mobility Reports, 2021) to create a mobility covariate for the main analysis (described below). Broadly, these data represent the percent change, from pre-pandemic movement, of where Canadians were going based on a proportion of cell phone “pings” across 6 different categories: residential, workplace, transit stations, parks, groceries and pharmacies, and retail and recreation. Apart from “residential,” we averaged the fractional change from baseline (between February 15 and March 11, 2019) to represent movement outside people's homes.

Because the stringency index and mobility data are likely collinear (Hale et al., 2021), we disentangled their unique contributions by regressing mobility on the stringency index and substituted the value of mobility with the residuals of this simple regression model (Graham, 2003). We linked the surveillance, NPI, and mobility datasets by province and date to generate our working dataset.

Outcome Variable

Similarly to Brown et al. (2021), we hypothesized that “more stringent” NPIs would impact SARS-CoV-2 dynamics in terms of changes in rates, rather than absolute levels of infection. Therefore, our outcome was the daily growth rate, $g\left(t\right)$, calculated as the ratio of the incident SARS-CoV-2 cases on a given day divided by those from the previous day: $g\left(t\right)=\frac{I\left(t\right)}{I\left(t-1\right)}$. Because surveillance data are a product of inherent delays related to both the biology of infection (e.g., incubation period) and reporting (e.g., cases are not reported by the date they were infected), we produced a back-projected version of the daily growth rate for infections using the R package EpiNow2 (Abbott et al., 2020; Figure 1 , panel A).

Figure 1.

Schematic diagrams of: (A) Back-projected cases by their estimated date of infection (blue ribbon) versus their date of report (gray bars). Because of delays between being infected and confirmed as a case, the back-projected cases occur earlier in time. The light and dark blue ribbons are the 90% and 50% credible intervals, respectively; and (B) our calculation of the relative growth ratio. Day t is defined as the t-th day of a stringency period (i.e., since O.S.I changed deciles). Day n represents the last day of the period. Note that different stringency periods could have different numbers of days; O.S.I = Oxford Stringency Index; g(t) = growth rate on day t. Panel (A) is modified from Abbott et al., 2020Figure 1. Panel (B) is a modified version of Figure 1 in Li et al., 2020.

Briefly, EpiNow2 uses the distributions of the incubation period, generation time, and the time from symptom onset to report date as part of an inverse convolution to estimate the number of infections per day from the time series of daily reported cases. The distributions for the incubation period and generation time were based on well-established values for SARS-CoV-2 (Lauer et al., 2020, Ganyani et al., 2020), as well as a log-normal distribution for the reporting delay time with a mean of 0⋅84 days and a standard deviation of 1⋅44 days (Abbott et al., 2020).

Analysis

We have based our approach on the analysis of Li et al. (2020) and defined a stringency period as a time through which the stringency index remained within a 10-point range. We used these periods to sub-divide the timeline of each provincial epidemic into segments based on the status of the containment and closure NPIs. For each period, we defined a daily growth rate, $g\left(t\right)$, as the growth rate on the t-th day of that period (i.e., since the stringency index changed deciles) and $g\left(0\right)$ as the growth rate of the last day of the previous period (i.e., before new NPIs were introduced, Figure 1, panel B).

Because we also expected any association of NPIs with case growth to be relative to previous periods, we calculated a relative growth ratio between $g$ on day t and $g$ on day 0 as a measure of association between NPI stringency and the growth of SARS-CoV-2 cases. A relative growth ratio larger than 1 indicates an increase in transmission since the change in the stringency index, and a growth ratio smaller than 1 indicates a decrease. Based on the change of NPIs between 2 adjacent periods and the corresponding growth ratio, we were able to observe the behavior associated with incrementally stringent NPIs over time.

In a descriptive analysis, we first quantified the extent to which higher starting values of the stringency index correlate with either high or low values of relative growth ratio using Kendall's ${\tau }_B$. We also described any lagged declines in the relative growth ratio as each province entered different stringency periods. Here, the timeline for all provinces and stringency periods (combined) was stratified by the first and second waves. For each wave, we plotted a single loess curve for the longitudinal relative growth ratios.

In the main analysis, we modeled the relative growth ratio using a linear mixed effects modeling approach, with the following equation:

$$y_{ij}\left(t\right)=({\beta }_0+{\nu }_{0i}+{\gamma }_{0j})+({\beta }_1+{\nu }_{1i}+{\gamma }_{1j})\times t_{ij}+{\beta }_2{M}_{ij},$$

where $y_{ij}\left(t\right)$ is the relative growth ratio in the $i$th province, over the $j$th stringency period, on day $t$.

As fixed effects, we entered a time trend, $t_{ij}$, and “residualized” mobility, $M_{ij}$, (without interaction) into the model. As random effects, we had intercepts for province, ${\nu }_{0i}$, and stringency period, ${\gamma }_{0j}$, as well as by-province, ${\nu }_{1i}$, and by-stringency slopes across time, ${\gamma }_{1j}$. Any correlation between relative case growth and the values of stringency index was quantified through the slope term $({\beta }_1+{\nu }_{1i}+{\gamma }_{1j})$ of the trending variable, $t_{ij}$. Using province and stringency period as random effects accounts for 2 sources of non-independence: 1) repeated measures from the same province are more likely to be similar; and 2), the current value of the stringency index depends on its previous values.

Similar to Bendavid et al. (2020), this simple model structure balances the strengths of empirical analyses, whereas accounting for pre-existing trends that naturally accompany outbreak dynamics (Graham, 2003, Kermack et al., 1927, Philipson, 2000). We fit 2 separate models—one for each wave—using the nlme package (Pinheiro and Bates, 2000) from R statistical software, version 4.1.0 (R: A language and environment for statistical computing, 2022).

Results

The epidemic curves in Figure 2 show a first wave of cases concentrated between March and May 2020, with lower numbers of cases detected during the summer months, and a much larger second wave between August and September 2020. When compared with the observed cases in each wave, back-projected infections peaked earlier in time by 6 to 8 days (in the first wave) and 1 to 14 days (in the second). Converting the back-projected infections into growth rates demonstrated a shared cumulative decline across all provinces ranging between 20 and 60% (Figure 3 ). Except for Alberta and Manitoba, these declines stabilized within 1 month of March 11, 2020 and continued into the second wave.

Figure 2.

Epidemic curves of observed cases (gray bars) and the mean back-projected cases (blue line) in 5 Canadian provinces. Because of delays between being infected and being confirmed as a case, the back-projected cases occur earlier in time.

Figure 3.

Growth ratios in cases for the study provinces. The panels below show the daily growth ratio in cases and demonstrate a shared decline in case growth across all provinces. The y-axis re-scales the growth ratios against that on the first day after study inclusion. The dashed line represents the average cumulative reduction in growth ratio across all provinces in the second wave (20%); Str. Period = Stringency Period.

Tables 1 and 2 contain the results of our descriptive analysis. After each province accumulated 10 reported cases, stringency index values ranged from 11⋅1 to 56⋅5 out of 100, with Manitoba introducing the largest array of containment and closure policies (Table 1). When centred on the time to reach 10 cases, there was little correlation between the higher relative starting value of the stringency index and the relative differences in daily case growth between Manitoba and the remaining 4 provinces (Kendall's $\tau$ = -0⋅18, p = 1⋅00, Table 2).

Table 1.

Survey of individual NPIs that correspond to the stringency index at study inclusion for each province. Individual NPI categories (table columns) and the labels linked to the ordinal coding of the intervention (table cells) correspond to those defined in the containment and closure policies of the Oxford COVID-19 Government Response Tracker.

	O.S.I. at study inclusion	SchoolClosures	WorkplaceClosures	Public Events	Ban Gatherings	Stay at Home	Internal Movement	Travel Controls	Public Information
Manitoba	56⋅5	Recommend closing	Recommend closing	Require canceling	11-100 people	Recommend caution in public	Not recommended	Ban arrivals	Urging caution
Alberta	22⋅2			Recommend canceling				Quarantine arrivals	Coordinated campaigns
B.C.	22⋅2			Recommend canceling				Quarantine arrivals	Coordinated campaigns
Ontario	13⋅9			Recommend canceling					Coordinated campaigns
Québec	11⋅1			Recommend canceling					Urging caution

O.S.I. = Oxford Stringency Index; B.C. = British Columbia.

Table 2.

Assessing the correlation between initial values of the stringency index and the daily growth ratio of SARS-CoV-2 cases, adjusted for the time to reach 10 cases.

	O.S.I. at study inclusion*	% Diff. O.S.I. (vs. MB)#	% Diff. g(t)Median (Q1, Q3)||	Kendall's $\tau$ (p-value)
Manitoba	56⋅5
Alberta	22⋅2	-60⋅7	5⋅7 (-8⋅3, 25⋅1)	-0⋅18 (1⋅00)
B.C.	22⋅2	-60⋅7	-0⋅3 (-6⋅2, 17⋅1)
Ontario	13⋅9	-75⋅4	29⋅3 (25⋅5, 31⋅0)
Québec	11⋅1	-80⋅4	3⋅5 (-0⋅7, 54⋅7)

^⁎O.S.I. = Oxford Stringency Index; study inclusion: > 10 cases in each province; B.C. = British Columbia.

^#% Difference in O.S.I. = $\left(O.S.I_{Other}-O.S.I_{Manitoba}\right)/O.S.I_{Manitoba}\times 100$; negative values indicate stringency index values at study inclusion were lower than Manitoba; MB = Manitoba.

^||% Difference in daily growth ratio, $g\left(t\right)=(g{\left(t\right)}_{Other}-g{\left(t\right)}_{Manitoba})/g{\left(t\right)}_{Manitoba}\times 100$; where t = 7,…, 53 are the days in Manitoba's data that correspond to study inclusion and the beginning of their relaxation of restrictions; positive values indicate median daily growth ratios were higher than Manitoba's.

Any association with introducing new (or intensifying) NPIs across each wave was not immediate (Figure 4). In the first wave, a local regression curve demonstrates that (on average) the largest associated decline in the relative growth ratio occurs within the first 9 days of changing stringency periods (left panel). After 10 days, relative growth ratios rebounded and plateaued. For the second wave, the largest associated decline occurs more slowly, approximately 16 days after changing stringency periods before also rebounding (right panel).

Figure 4.

Temporal changes in the relative growth ratio following 10-point changes in the stringency index, in each wave, for all provinces combined. For each stringency period, the reference point is the day before entering a that period. It is important to note that different stringency periods can have different numbers of days. Because of limited data availability, particularly in the first wave, we did not plot timelines longer than 30 days. A locally weighted smoothing function (black line) plots the trend (i.e., the average) across each wave. Shaded regions are 95% confidence intervals. Str. Period = Stringency Period.

Table 3 and Figure 5 present the results of the mixed effects regression analysis. For both waves, a trend model with random slopes and intercepts by province and by stringency period contained more information than a model with only random intercepts (LR = 27⋅03, df = 4, p < 0⋅0001; LR = 16⋅23, df = 4, p < 0⋅0001). Each model explained approximately 46% (for the first wave) and 69% (for the second) of the variation in relative case growth by including mixed effects in our model (Table 3).

Table 3.

The Results of the Linear Mixed Effects Model.

	First Wave		Second Wave
Fixed Effects	Estimate	95% CI	Estimate	95% CI
Intercept	0⋅965	0⋅867, 1⋅064	1⋅026	0⋅981, 1⋅071
Time	0⋅000	-0⋅002, 0⋅002	0⋅000	-0⋅0002, 0⋅0002
“Residual” Mobility	-0⋅072	-0⋅185, 0⋅042	0⋅031	0⋅005, 0⋅057
Random Effects
Province
${\sigma }_{Intercept}$	0⋅056	0⋅005, 0⋅613	0⋅013	0⋅000, 6⋅424
${\sigma }_{Time}$	0⋅001	0⋅000, 0⋅010	0⋅0001	0⋅000, 0⋅001
Stringency Index
${\sigma }_{Intercept}$	0⋅151	0⋅095, 0⋅239	0⋅051	0⋅019, 0⋅132
${\sigma }_{Time}$	0⋅003	0⋅001, 0⋅005	0⋅0002	0⋅0001, 0⋅0005
$R_M^2$	0⋅007		0⋅004
$R_C^2$	0⋅463		0⋅691

$R_M^2$ = “Marginal” and $R_C^2$ = “Conditional” coefficients of determination; each, respectively, represent the amount of variation explained by the fixed effects and with fixed and random effects combined (Nakagawa and Shielzeth, 2013).

Figure 5.

Steepness (slope) of relative growth ratio across different stringency periods (i.e., deciles of the Oxford Stringency Index) for each province (colored dots) as estimated by the mixed effects regression analysis. Values > 0 (dashed line) indicate increased case growth, and values < 0 indicate decreased growth. Here, the greatest decline in growth rate in each province happened with the least stringent measures, at the earliest stages of the pandemic. A locally weighted smoothing function (black dotted line) is plotted to visualize trends in the estimated slopes across all provinces. Shaded regions are 95% confidence intervals.

Linear trends within stringency periods displayed slopes that were more variable in the first wave than in the second across all provinces (Figure 5). For British Columbia, Ontario, and Quebec, the steepest negative slopes were observed for smaller, early values of the stringency index (ranging between 11⋅1 and 29⋅2 out of 100). These trends also decelerated (i.e., became less negative—in Ontario), turned upward (in Quebec), or remained nearly unchanged (in British Columbia) as the stringency index increased. In Alberta and Manitoba, we observed the steepest negative slopes for incrementally higher values of the stringency index ranging from 72⋅2 out of 100 (in Manitoba) and 75⋅9 out of 100 (in Alberta). Across all 5 provinces in the first wave, a local regression line demonstrates that case growth was initially negative, turned positive, and returned to earlier levels across incrementally higher values of the stringency index (Figure 5, left panel). Overall, the steepest declines in case growth correspond to a (geometric) mean stringency index of 30⋅1 out of 100. When compared with the first wave, the stringency periods in the second wave possessed very little association with case growth (Figure 5, right panel).

Discussion

The results presented here demonstrate a limited dose-response association between values of the stringency index and growth of SARS-CoV-2 cases in the first and second waves across 5 Canadian provinces. The minimal association in the first wave and lack of association in the second are compatible with the hypothesis that NPIs, per se, do not lead to a decline in case growth. These results were observed regardless of the initial (or maximum) value of the stringency index and are, overall, contrary to what we anticipated. At first glance, these findings might suggest that Canada's pandemic responses were sub-optimal at slowing the spread of SARS-CoV-2 as the stringency index increased. However, the explanation for our results requires nuance that extends beyond a simple dichotomy of whether government mandated NPIs did, or did not, have their intended effects. Instead, other potential explanations include how NPIs work across risk factors for onward transmission, and people's adherence to interventions over time.

In the first wave, individual NPIs were generally focused on decreasing the risk of community-level spread of SARS-CoV-2. As a result, these early NPIs may have initially benefited those who, with a small reduction in contact, could effectively mitigate nearly all their transmission risks (Mishra et al., 2020, Gomes et al., 2021). When coupled with higher rates of testing and isolation among high-income neighborhoods (Sundaram et al., 2021), the early benefits may have rapidly saturated among those who could work remotely and remain well housed (Public Health England, 2020). Thus, early NPIs may have eliminated a small risk among many, but could not sufficiently mitigate a large risk among the few (Sun et al., 2021, Cevik and Baral, 2021), such as essential workers and those employed in public-facing jobs (Sundaram et al., 2021, Paul et al., 2021, Rao et al., 2021). These same justifications also apply to the results of the second wave, despite starting at higher stringency index values than in the first. Although some provinces introduced new NPIs that targeted “intermediate” per-contact risks of transmission, such as extended family contacts (Sun et al., 2021), the majority of NPIs simply recommitted to a focus on community contacts. Therefore, the impact of many NPIs in the second wave—despite being more stringent—speaks more to a mismatch between where the largest risk of infection was in the population than how restrictive they were.

Our results are also consistent with other important voluntary behaviors not accounted for by government policies (Berry et al., 2021, Goolsbee and Syverson, 2021). In our analysis, we accounted for 2 sources of people's movements: those behaviors linked to the stringency index, and those that were voluntary. The latter was negatively—though not distinguishable from 0 (in the first wave)—and positively associated (in the second) with case growth (Table 2). This observed association has a plausible underlying behavior model indicating that many people were already avoiding busier public spaces—voluntarily—before being mandated to do so (Bendavid et al., 2020, Berry et al., 2021, Goolsbee and Syverson, 2021). For example, increased mobility may have reallocated people from visiting “non-essential” businesses (bars and restaurants) to “essential” ones (groceries and other food sellers) where risk of infection was higher (Goolsbee and Syverson, 2021). Increasingly strict NPIs might also have limited association with case growth if people are less concerned about being infected; reduced concern could lead to a higher likelihood of restrictions getting ignored (Goolsbee and Syverson, 2021). Restrictions on activity toward the type of business also induce large reallocations of movement away from “disallowed” businesses, toward “allowed” ones. Thus, in contrast to recent work (Brown et al., 2021), our findings suggest that the limited association between case growth and greater stringency cannot be fully explained by changes in mobility. However, the consistency across provinces—particularly in fall 2020—is notable, given their diverse geo-political landscapes, which suggests that adherence may have less of an impact than appropriately targeting the risk of onward transmission.

There are several key limitations to consider when interpreting these results. The first is that our estimates of case growth did not account for changes in testing regimes, over time. If a province increased their testing capacity or widened their eligibility criteria, we are likely to observe an artificial increase in case growth. A potential solution is to use other outcomes, such as hospitalization or mortality, which are considered less susceptible to testing-related biases. However, because NPIs are intended to decrease contact rates between individuals in a population, their primary impact, if effective, is on transmission (Flaxman et al., 2020; Ferguson et al., 2020). Any impact on hospitalization and mortality will be delayed in some cases by several weeks. Previous research has also demonstrated that if stricter measures are not correlated with cases, they are unlikely to share that correlation with COVID-19 hospitalizations and mortality (Berry et al., 2021). If our results have been biased by under-reporting, we would expect the trend estimates across stringency periods to overstate the correlation between NPIs and case growth. However, the fact that we estimate incrementally small associations between the stringency index and case growth suggests that the extent of this bias may also be small.

Second, the back-projection method we used is an imperfect recreation of the date of infection because it imparts a distributional form on the incubation period, generation time, and reporting delay that could vary (Küchenhoff et al., 2021), and may have led to temporal inaccuracies in our back-projected infections. Although back-projection methods have been used previously to reconstruct incidence curves of other infections, such as HIV (Mallitt et al., 2012, for a discussion), there are currently no suitable SARS-CoV-2 data which establish the reliability of these methods, particularly given the correlation between testing volumes and case counts mentioned above. However, given that back-projection explicitly accounts for delays between infection and diagnosis, we think it lends itself as a reasonable illustration of how such delays can be the crucial difference between well-intentioned policies missing their target and having their desired effect (Roberts, 1994, Meadows, 2008).

Third, by using the stringency index as the measure of NPI dose, we cannot disentangle which individual NPI is most associated with case growth. However, by measuring several NPIs, the stringency index mitigates the possibility that any one NPI is over- or misinterpreted, while supporting systematic comparisons across sub-national regions (Hale et al., 2021). The stringency index does not measure the effectiveness of NPIs nor their costs. And, like all indices, it makes assumptions about what information counts and may introduce measurement bias (Hale et al., 2021). For example, it focuses on closures and movement restrictions to contain the spread of infection, and thus does not include more “traditional” NPIs, such as testing, contact tracing, and isolation (Fraser et al., 2004). Although this was a limitation, there were consistencies across the 5 provinces, given the federal nature of some laws and recommendations for NPIs including international travel, testing, and contact tracing.

Fourth, the stringency index does not explicitly include information on vaccine programs that started in Canada in mid-December 2020. Although not an NPI, vaccination represents a potential confounder when examining the association between NPI stringency and case growth. However, given that only 2 to 4% of each province's population had received at least 1 dose by early February 2021 (Canadian COVID-19 vaccination coverage report, 2021), the larger effect of the vaccine programs was likely to have been on decreased mortality, not the spread of infection in the second wave.

A fifth concern is that we have not evaluated the impact of NPIs through other quasi-experimental approaches, like difference-in-differences. The most notable challenge to this approach is that inferences would have been made in the absence of sufficient amounts of baseline data in the growth of SARS-CoV-2 cases, and that there is direct feedback between intervention and outcome. This means that at least 2 key assumptions of a standard difference-in-differences design are questionable. Finally, because we regard this study as an ecological time series analysis, our findings do not necessarily imply causation. And although our results are supported elsewhere (Herby et al., 2022), we cannot definitively conclude that incremental increases in the stringency index did not have some independent effect on case growth.

Overall, our analysis provides little evidence that banning public gatherings, closing schools and universities, placing stay-at-home orders, controlling travel, and restricting business hours had any obvious association with reducing the growth of SARS-CoV-2 cases in Canada—an observation that aligns with those of previous pandemics (WHO, 2006; Whitelaw, 1919). However, we emphasize that estimating the effects of NPIs is extremely challenging, and no single approach produces irrefutable results. When placed in the context of other recent large-scale analyses (Herby et al., 2022), our results have led to similar insights, which is reassuring. Nevertheless, we do not discount the possibility that alternative approaches may yield different results. Interpreting the lack of association between greater NPI stringency and reduced case growth could be bolstered through de novo data collection exploring voluntary behavior changes and the network structure of human interactions, rather than the use of routinely collected surveillance data.

Ultimately, these findings suggest that the diminished returns associated with a higher stringency index should signal for better understanding of what “works” and “for whom” when it comes to mitigating the spread of infection. Although there exist alternative explanations for the equivocal relationship between stringency index and case growth (particularly in the second wave), the onus of providing evidence shifts to demonstrating how NPIs consistently have flat associations despite intensified stringency. Continued study of the long-term outcomes of many NPIs, including any adverse costs to society, is needed before similarly aggressive measures can be endorsed for future epidemic control.