Reopening Under Uncertainty

Short abstract

The coronavirus disease 2019 (COVID-19) pandemic required significant public health interventions from local governments. To help in those decisions, researchers used the Robust Decision Making approach to stress-test California's COVID-19 reopening strategy. This article presents lessons learned from these experiments and outlines four characteristics of the best strategies.

Abstract

The coronavirus disease 2019 pandemic required significant public health interventions from local governments. Early in the pandemic, RAND researchers developed a decision support tool to provide policymakers with insight into the trade-offs they might face when choosing among nonpharmaceutical intervention levels. Using an updated version of the model, the researchers performed a stress-test of a variety of alternative reopening plans, using California as an example. This article presents the general lessons learned from these experiments and discusses four characteristics of the best reopening strategies.

As administration of the vaccine for coronavirus disease 2019 (COVID-19) has gotten underway in 2021, policymakers across the United States have been navigating how to reopen their local institutions and economies. They must manage multiple objectives, including preserving the health of constituencies, balancing equity, and limiting the economic consequences of COVID-19 nonpharmaceutical interventions. To guide these decisions, many states have adopted decision rules that link reopening to case counts, test positivity, or hospital capacity. Yet uncertainties—such as vaccination uptake, the extent to which vaccinations reduce transmission, and the impact of variant strains—complicate the path to identifying the best path forward.

In a separate working paper (Nascimento de Lima et al., 2021), we evaluate reopening strategies that California can adopt to handle these uncertainties. Our approach uses models to stress-test COVID-19 reopening strategies under a wide variety of scenarios using the Robust Decision Making approach. Although there is a fast-growing literature on strategies to curb the pandemic with ­nonpharmaceutical interventions and although many modelers sought to explore alternative COVID-19 exit strategies under alternative scenarios, to the best of our knowledge, this is the first analysis that explicitly set out to stress-test California's reopening plan with a focus of finding robust, non-dominated strategies (explained later). This article draws on the analyses in the working paper and discusses the characteristics of successful reopening plans. In addition, we discuss how these characteristics relate to existing plans and California's intention of moving beyond its reopening plan as soon as June 15, 2021 (California Department of Health, 2021b). Our analysis focused on California, but the lessons we learned can be applied by other U.S. states or countries that may want to pursue strategies similar to California's reopening plan.

Key findings

• Poorly designed reopening plans can make society unnecessarily worse off by leading to preventable deaths and longer periods of interventions.

• Robust, non-dominated COVID-19 reopening plans shared common characteristics: They start with higher levels of caution and gradually relax reopening thresholds, contingent on vaccination rates. The timing and pace of relaxing restrictions should be evaluated on a case-by case basis.

• California's reopening strategy contains the structure of robust reopening plans. Different decisions might have been made about stringency levels, but the structure of the state's plan is likely useful for jurisdictions that have no defined long-term strategy to fight the pandemic.

• The consequences of not adopting and not following robust COVID-19 reopening plans can be severe. Our results indicate that poorly designed reopening plans can lead to both unnecessary COVID-19 deaths and days of interventions, especially in scenarios of scarce vaccine availability and high transmissibility.

Reopening Plans

Plans to reopen amid COVID-19 vary widely in the United States. Some states, such as California (State of California, 2021) and Illinois (State of Illinois, undated), have specified reopening plans that rely on rates of positive COVID-19 tests or rates of COVID-19 cases among the population and that are contingent on vaccination levels. Texas has tied local businesses’ capacity restrictions to hospitalization levels (Governor of the State of Texas, 2021). Other states have less-clear guidelines, either avoiding stating specific targets (New York Forward, undated; Pennsylvania Government, 2021) or having no public description of what might cause another lockdown (Florida Department of Health, 2021; Responsible Restart Ohio, undated).

On August 30, 2020, California drafted its Blueprint for a Safer Economy (State of California, 2021), a tiered plan for protecting the state from COVID-19 surges while businesses reopen and social restrictions are lifted on a county-by-county basis. The plan defines four COVID-19 risk levels, based on weekly measures of the number of COVID-19 cases, positivity rates, and other factors. The plan specifies that, as counties meet the criteria for a risk level—for instance, the case rate there falls below a given threshold—certain restrictions will be lifted in that county. This plan is meant to accommodate uncertainty as local pandemic conditions change while also preventing COVID-19 infections from rising unchecked.

California's plan uses COVID-19 case thresholds to communicate risk and justify reopening and lockdown decisions (State of California, 2021). In our analysis, we call these case thresholds the state's level of caution concerning COVID-19 cases.¹ Jurisdictions have used a wide spectrum of restrictions and guidelines based on the amount of caution they are willing to take. Localities adopting a high level of caution require only a small number of cases to impose significant restrictions. In contrast, states adopting a low level of caution allow higher case counts before restrictions are implemented. For example, in Texas, even when COVID-19 hospitalizations are high, counties are unable to restrict business capacities below 50 percent (Governor of the State of Texas, 2021).

Public health officials face many difficult decisions when defining plans to reopen after COVID-19, including defining the initial level of caution and how to manage restrictions as more people are vaccinated. Authorities must find a balance between inviting new surges by relaxing too early and imposing unnecessary constraints by reopening too late. One option to manage reopening plans during vaccination rollout is to simply fix thresholds in the hope that vaccination will bring case numbers down sufficiently to reopen; we label these fixed-threshold strategies. These strategies use the same thresholds until a percentage of the population (e.g., 70 percent) is vaccinated. Another option is to relax restrictions on a certain date, and we label these time-based strategies. A middle-ground approach could account for vaccination, gradually relaxing restrictions as immunity becomes widespread; we call these vaccination-based strategies. After amending its reopening plan on March 4, 2021, California shifted from a fixed-threshold plan so that its reopening thresholds will shift as more individuals are vaccinated (California Department of Health, 2021a). On April 6, California announced its intention to move beyond its reopening plan as soon as June 15, as long as vaccines have been distributed equitably and hospitalization numbers are low (California Department of Health, 2021b).

We set out to stress-test California's strategies to reopen under multiple scenarios through the year 2021. In our framework, a reopening strategy is defined by three things: an initial level of caution (i.e., how aggressively policymakers react to COVID-19 cases), what the level of caution changes in response to, and how much it changes. Details on the mathematical formulation are available in our accompanying paper (Nascimento de Lima et al., 2021).

Simulating the Pandemic

To analyze the interactions among strategies and a variety of uncertainties, we used an epidemiological model to simulate thousands of scenarios describing how vaccination and nonpharmaceutical interventions could shape the pandemic in California during 2021. This theory-based epidemiological model is based on prior work (Vardavas, Nascimento de Lima, and Baker, 2021; Vardavas et al., 2020) and allows us to simulate scenarios that move beyond small changes in current conditions by relying on epidemiological theory. The model simulates both the spread of disease and how populations progress through each stage of the disease, from exposure to recovery or death.

We model the spread of disease by considering heterogeneities in the way people mix across different population groups and locations (such as households, workplaces, or schools), and we consider different mortality rates by population groups. Our model represents nonpharmaceutical interventions by changing how much people have infectious contacts in different settings in response to the restrictions. For instance, we model school closures by removing school mixing and reducing work mixing (because some parents stay home). Our model also includes policy rules that imitate California's Blueprint for a Safer Economy (State of California, 2021) by imposing and removing interventions based on the number of COVID-19 cases. In addition, the model represents the supply and demand of vaccines considering a phased rollout, wherein vaccine supply gradually increases and is distributed to different population groups incrementally.

Because we built the model with these explicit mechanisms, the model allows us to simulate what-if scenarios that describe what could happen if California followed a different reopening strategy. However, no model is a crystal ball. Many model inputs—such as the effect of new variants on transmissibility, vaccine confidence and uptake, and how long immunity will last—are deeply uncertain. The best strategy for one future might not be the best strategy for other futures. Strategies that seem to work during the first phases of vaccination rollout might fail later if vaccination uptake rates decline over time.² Therefore, we need an approach that makes the best use of our model and helps policymakers make good decisions amid uncertainty.

Confronting Uncertainties

RAND has a long history of helping policymakers in situations where uncertainties abound. In such areas as water resource management (Groves, 2006) to terrorism insurance (Dixon et al., 2007), RAND has used the Robust Decision Making approach (Lempert et al., 2013) to help shape policies that affect society's long-term future. The approach works by testing each strategy in many possible futures, each containing a different combination of the values describing these uncertain parameters (Lempert, 2019; Lempert, Popper, and Bankes, 2003; Lempert et al., 2006). Among the results, we look for robust strategies: those that perform well in many futures. This approach requires a large amount of computation—in our case, more than 1 million separate model evaluations. We worked with our colleagues at the U.S. Department of Energy's Argonne National Laboratory to set up this large experimental design and conducted our analyses using their High-Performance Computing clusters. This computing machinery allowed us to stress-test 78 reopening strategies under 20,000 plausible futures, which vary the values of uncertainties. For example, we vary the transmissibility of the virus (given new variant strains) and the level of vaccine hesitancy. These simulations result in a database of model runs that are useful for evaluating how robust each of the 78 reopening plans are when evaluated across these 20,000 futures.

Some futures reflect a more dangerous situation (for instance, new variants with increased transmissibility become dominant, immunity is short-lived, and vaccination rates are low) and systematically result in higher numbers of deaths and economic hardship. And current information might not allow policymakers to distinguish which of those futures are more likely to happen. Instead of looking at absolute outcomes for a best guess of the future, we evaluate how robust each strategy is to many futures using regret as a robustness measure. For example, in each future, there is a reopening plan that minimizes the number of COVID-19 deaths. To find out how robust every other strategy is, we compute the number of COVID-19 deaths that could have been prevented if the best strategy was chosen. The resulting metric is called regret, and less regret is better. Robust strategies are those that are associated with small levels of regret over many plausible futures.³

Reopening plans might result in different types of regret. Society might regret, for instance, not having saved enough lives or allowing COVID-19 cases to overwhelm hospitals. It might regret having exposed populations to COVID-19 unnecessarily, leading to long-term health problems. It might regret not sending children to school sooner or that many people lost their jobs. It is difficult to balance regret across all the dimensions people care about. Achieving zero regrets across all the dimensions is impossible and finding the right trade-offs between economic and health outcomes is a thorny philosophical issue. Thus, in our modeling, we seek strategies that result in fewer COVID-19 deaths and fewer days under nonpharmaceutical interventions. By doing so, we aim to find strategies that minimize the regret associated with COVID-19 deaths while minimizing the regret associated with the social welfare consequences induced by the time spent under nonpharmaceutical interventions.

We call strategies that minimize regret and find efficient trade-offs non-dominated or Pareto-efficient. In other words, a strategy is non-dominated if we could find no other strategy that had fewer COVID-19 deaths and less-stringent restrictions, when judged by the regret the strategy imposes on those outcomes. Dominated strategies are those that create regret on certain outcomes unnecessarily, and these should be avoided, of course. Figure 1 shows a summary of the performance of the strategies we tested that achieve the lowest number of COVID-19 deaths. Each point in the plot represents a reopening strategy. The axes represent two robustness criteria we care about—COVID-19 deaths and days of restrictions (i.e., nonpharmaceutical interventions). The goal is to adopt a strategy that is closer to the origin in that plot; that is, we want to achieve low regret across both dimensions. Plotting strategies in this way reveals that there are dominated (red) and non-dominated (green) strategies. Whenever possible, policymakers should not choose a dominated strategy, because there is a different strategy that results in fewer deaths and fewer restrictions. In other words, dominated reopening plans make everyone unnecessarily worse off.

Figure 1.

Time- or Vaccination-Based Strategies Dominate Constant-Threshold Plans

NOTE: Each point in this plot represents the performance of a reopening strategy. The shape of the point (circle, triangle, or square) represents the type of strategy used. We summarize the performance of strategies using the 75th regret percentile as a robustness metric. For example, a strategy with 20 deaths per 100,000 people in this plot resulted in fewer than 20 preventable deaths in 75 percent of the scenarios we simulated. Ideal strategies combine low regrets across the two dimensions, and these strategies fall in the lower-left corner of the plot. We found that some strategies are systematically worse than others; for example, the strategies represented by circles (constant strategies) were systematically worse than the time-based or vaccination-based strategies. This article explains the characteristics of the best strategies (represented by green squares and triangles) in the figure.

Characteristics of the Best (Non-Dominated) Reopening Strategies

The 1.5 million model scenarios give us a database from which to draw lessons about shared characteristics of strategies that were robust against uncertainty in our analysis. We believe that these findings apply to other states or jurisdictions that are pursuing case-based reopening strategies, not just to California. Most states have not explicitly created reopening plans with all these characteristics, so there is a potential for benefit across many jurisdictions. Moreover, each country in the world will have to make tough decisions as COVID-19 surges emerge where vaccines are scarce.

This section focuses on four high-level, widely applicable characteristics of reopening strategies that we would recommend based on our analysis, and specific results can be found in our working paper. One of our findings is that plans that lacked these four characteristics were dominated; that is, they were outperformed by alternative plans and resulted in unnecessary regret. Therefore, all four recommended characteristics should be considered by policymakers who must craft reopening strategies amid the COVID-19 vaccine rollout.

1. Encourage Inexpensive Adaptation Measures

Across all scenarios we simulated, an increase in transmissibility, caused either by new variant strains or by premature relaxation of relatively inexpensive measures (such as indoor mask-wearing and voluntary social distancing), worsens both health and economic outcomes. Increases in transmissibility require more interventions to control, resulting in higher numbers of days under restrictions. Therefore, decreasing indoor mask use and ending inexpensive mitigation behaviors should be the last step in relaxing restrictions.

2. Begin Cautiously

A common factor among successful plans was a high initial level of caution, meaning that policymakers were quick to lock down before the vaccine was widely distributed. Waiting to impose restrictions allows the virus to spread more widely, leading to later but longer periods of restrictions. It is important to be cautious in the early phases of vaccination rollout because this is when the virus is most deadly: Medical professionals have not yet learned how best to treat the virus (Horwitz et al., 2021), and there is little natural or vaccinated immunity.

3. Adapt as People Are Vaccinated

The timing of reopening decisions matters. When the most vulnerable are vaccinated, those who are infected experience a lower infection-fatality rate. However, if reopening is allowed before sufficient vaccine coverage is achieved, new surges undermine the vaccination efforts. Strategies that relaxed interventions too early (e.g., before March 2021) allowed many individuals to become infected before they were vaccinated. Conversely, plans that do not allow reopening to occur once substantial vaccination coverage has been achieved fail to take advantage of the opportunity afforded by the vaccine. We found that the best strategies tied reopening plans to the percentage of vaccinated individuals or changed reopening thresholds over time. Fixed plans failed to take advantage of changing circumstances. With COVID-19 specifically, vaccination renders the virus less deadly at the population level, especially when the most vulnerable in the population are vaccinated first. SARS-Cov-2 (the COVID-19 virus) is approximately ten times more deadly in those over age 50 than in those under age 50 (O'Driscoll et al., 2020). After older individuals and a substantial percentage of the general population are vaccinated, relaxing restrictions is possible without a corresponding mortality increase that would be observed before the vaccination campaigns. We found that the best reopening plans combined high initial levels of caution while responsibly relaxing later as immunity was more widespread.

4. Gradual Approaches Work Better Than Abrupt Changes

Across all the strategies we tested, abrupt, one-step reopenings (e.g., lifting all restrictions at once after having used fixed case thresholds throughout the vaccination campaign) performed worse than gradual changes did, perhaps because changes in the effective transmissibility and mortality of the virus are also continuous. If the disease characteristics are constantly changing, then the non-dominated policy should change in response. Gradual adjustments enable policymakers to better approximate the most appropriate policy for that time, whereas abrupt changes miss opportunities. In California, gradual approaches should allow the state to cope with threats that might materialize later, such as the emergence of new variant strains.

Our analysis emphasizes that all four recommended characteristics are desirable in a robust, non-dominated reopening plan. Adopting some of these recommendations while overlooking others always led to poor outcomes in our analysis. For example, if a jurisdiction starts with a low initial level of caution and adjusts the plan over time, that is not enough to robustly control COVID-19 deaths. Similarly, starting with a high level of caution and not changing reopening thresholds until the last person is vaccinated was also dominated. Unsurprisingly, our results emphasize that navigating the pandemic responsibly without making society unnecessarily worse off is a difficult task—hence the need to carefully examine policies using the best-available approaches before implementation.

Assessing California's Reopening Strategy

When we began our analysis, California was pursuing a fixed-threshold reopening strategy. However, California's March 2021 changes to its reopening plan shifted policies from a fixed level of caution to one that decreases as more individuals are vaccinated (California Department of Health, 2021a), and state leaders later announced that they will roll back the state's Blueprint for a Safer Economy by June 15, 2021, if the vaccine is equitably distributed and hospitalizations are low (California Department of Health, 2021b). Considering our results, this flexibility should be welcomed. We anticipate that this change will reduce the number of days spent under restrictions. However, these relaxed restrictions may increase mortality. We expect this increase to be relatively small because the easing of rules occurs only after many of the most vulnerable have been vaccinated, and the plan would be phased out contingent on low hospitalizations. Pandemics create difficult trade-offs, and these additional deaths must be weighed against other welfare consequences of restrictions, such as economic hardship among the affected population and lack of consistent in-person education. The state's incorporation of vaccine allocation equity into its reopening plan is an important and welcome addition, though one that our analysis did not consider because of the structure of our model.

Our analysis suggests that California's new decision rules exhibit common features of non-dominated reopening plans. The state's plan avoids abrupt changes by implementing multiple steps, with two vaccination targets at which thresholds change. These smaller steps help ensure that policies are tailored to the current situation. Although policymakers could improve the plan by adding additional steps, there is a trade-off between the reopening plan's complexity and ease of communication.

These results are also relevant to other states and countries. California's reopening plan structure could be useful for other state and local governments that still lack a long-term strategy to curb the pandemic while vaccination is underway. Other states may have to make different choices about the level of restrictions (because of different preferences or local considerations), but the structure of California's decision rules seems appropriate for other jurisdictions. Our results demonstrate that, if local jurisdictions are not able to perform their own stress-testing to evaluate strategies, adopting reopening plans like California's could be a good option.

These results could be particularly relevant to countries for which the trade-offs we analyzed seem even more challenging. Our results imply that the regrets of fragile and dominated strategies will be worse for countries that are already being overwhelmed by the uncertainties that we considered in this analysis. For example, other countries do not have ample access to vaccination and have faced more-transmissible variant strains. These countries effectively face the worst scenarios we simulated and therefore can face the worst consequences of fragile strategies. Although other countries with low vaccination capacity can make use of the structure of California's reopening plan, they should not imitate the timing of California's decisions. For example, California has decided to phase out its plan on June 15, 2021, as a result of its ample vaccine distribution capacity. Other countries should not necessarily follow the timing of that decision because they effectively would expose themselves to a much higher risk of a surge in COVID-19 mortality. Although jurisdictions can and should look for examples of strategies that were successful elsewhere, they should not imitate the timing of reopening decisions, especially in the final phases of the pandemic when the disparities among countries’ vaccination rates are high.

Limitations

Although our analysis considers a wide variety of policies and uncertainties, it also has limitations. First, our model is not a perfect replica of California's reopening plan. California's reopening plan is a highly detailed policy and contains industry-specific regulations that define not only which businesses operate under specific risk levels but also how they operate. Second, we evaluated only decision rules that are a function of case counts. One could also make reopening decisions contingent on deaths or hospitalizations.⁴ Third, we modeled only deaths directly resulting from COVID-19 and did not account for the indirect impact of restrictions on health, such as diminished mental health or health problems stemming from economic hardship. Fourth, we did not consider hospital capacity constraints. The fatality rate may dramatically increase if hospitals have no available beds or oxygen shortages. Finally, although we did consider behavioral uncertainties in our analysis, the current representation of behavior in our model—and in most current epidemiological models—is rather limited. Our analysis also did not consider all potential threats to the reopening strategies, such as increased traveling between states. For example, if travelers from other states reintroduce COVID-19 cases at high rates, that could shift the trade-off curves that we examined. These limitations indicate that it may be beneficial to further relax those assumptions and continue the search for policies that are still robust as more details are incorporated in the model.

Conclusion

We set out to stress-test California's COVID-19 reopening plan and to find reopening plans that were robust to uncertainties and achieved the best possible outcomes (minimizing deaths and days of restrictions). Unsurprisingly, we found that strategies need to start with cautious thresholds. However, we also found that many strategies can be dominated and that the best strategies started with a high initial level of caution and gradually relaxed restrictions as vaccination progressed. Although our conclusions offer support to the structure of California's current strategy, they also illuminate the fact that simple reopening plans with fixed case thresholds might not be the best approach to handle the pandemic; they might make society unnecessarily worse off. Moreover, our conclusions emphasize the message that modelers and epidemiologists have been repeating since the beginning of the pandemic: Policymakers must be careful about when they relax restrictions.

Recent surges in COVID-19 cases and deaths worldwide emphasize how critical these policies are. Brazil, for instance, started seeing its worst COVID-19 surge after vaccination started. As of March 20, 2021, Brazil was going through one of the worst-case scenarios we simulated—a new variant striking while people relax restrictions, vaccination rollout is slow, and vaccine efficacy is relatively low (Andreoni, Londoño, and Casado, 2021). Other countries without access to ample vaccine supply could go through similar scenarios in 2021, and U.S. localities with low vaccine uptake could also see surges. Although these scenarios had not materialized in the United States at the time of writing, they should serve as a cautionary tale of the importance of designing and implementing robust, non-dominated reopening strategies.