Essential information: Uncertainty and optimal control of Ebola outbreaks

Significance

The 2014 Ebola outbreak illustrates the complexities of decision making in the face of explosive epidemics; management interventions must be enacted, despite imperfect or missing information. The wide range in projected caseload generated attention as a source of uncertainty, but debate did not address whether uncertainty affected choice of action. By reevaluating 37 published models, we show that most models concur that reducing funeral transmission and reducing community transmission are robust and effective management actions to minimize projected caseload. Although models disagreed about absolute caseload, this measure has little relevance for evaluating candidate interventions. Our study highlights the importance of projecting the impact of interventions and is applicable to management of other epidemic outbreaks where rapid decision making is critical.

Abstract

Early resolution of uncertainty during an epidemic outbreak can lead to rapid and efficient decision making, provided that the uncertainty affects prioritization of actions. The wide range in caseload projections for the 2014 Ebola outbreak caused great concern and debate about the utility of models. By coding and running 37 published Ebola models with five candidate interventions, we found that, despite this large variation in caseload projection, the ranking of management options was relatively consistent. Reducing funeral transmission and reducing community transmission were generally ranked as the two best options. Value of information (VoI) analyses show that caseloads could be reduced by 11% by resolving all model-specific uncertainties, with information about model structure accounting for 82% of this reduction and uncertainty about caseload only accounting for 12%. Our study shows that the uncertainty that is of most interest epidemiologically may not be the same as the uncertainty that is most relevant for management. If the goal is to improve management outcomes, then the focus of study should be to identify and resolve those uncertainties that most hinder the choice of an optimal intervention. Our study further shows that simplifying multiple alternative models into a smaller number of relevant groups (here, with shared structure) could streamline the decision-making process and may allow for a better integration of epidemiological modeling and decision making for policy.

The devastating 2014 Ebola outbreak in West Africa is the largest ever recorded (1, 2). It resulted in 28,646 cases and 11,323 deaths by March 27, 2016 (WHO report; apps.who.int/ebola/ebola-situation-reports) and engendered an outpouring of concern for those affected. A large number of epidemiological models were developed and published (2–4). To date, we have identified 55 published Ebola models. Most of these models (50 of 55) projected caseloads as the preferred way to predict epidemic trajectory. However, caseload projections varied widely between models, drawing a great deal of attention and causing intense debate (5, 6).

Caseload projection is critical for predicting the size of an epidemic and planning management efforts, and it can vary from model to model for several reasons, such as differences in model structure, parameterization, and other assumptions. Despite model-specific variations in caseload projections, a critical question for decision making is whether different models lead to different management recommendations or different rankings of alternative management actions. If all models agree on the optimal management, then differences in projections are not a critical concern for decision making. Otherwise, if models disagree with respect to the ranking of management recommendations, then the optimal intervention is model-specific, which means that policymakers face the question of which model(s) to rely on to make management decisions; a closer examination of the source of the disagreement is then warranted.

Here, with the objective of minimizing the Ebola caseload, we explored the management recommendations of a large set of published Ebola models. We considered 37 published compartmental Ebola models that varied widely in model structure, parameterization, or both (Table S1). Among them, the SEIR compartment model is the most commonly adopted model framework, where individuals progress through susceptible (S) to exposed (E), infectious (I), and then removed (R) compartments through either recovery or death. Because hospital settings and funerals have been identified as critical transmission sources and targets for intervention, some models explicitly include a hospital (H) or funeral (F) compartment. Based on model structure, we classified 37 models into four categories: models with both H and F explicitly represented (referred to as SEIHFR), models with only H explicitly represented (SEIHR), models with only F explicitly represented (SEIFR), and models with neither H nor F compartments (SEIR). To ensure consistency, we recoded all of the models within the same stochastic environment by simulating the epidemic birth and death processes using the Gillespie algorithm with a tau-leaping approximation (3, 7). We then identified five management actions that are broadly applied to control Ebola: reducing community transmission, reducing hospital transmission, reducing funeral transmission, increasing hospitalization, and reducing case fatality ratio (2, 3, 8, 9). We projected the caseload under five management interventions and identified the optimum management for each model. We then used value of information (VoI) (10, 11) analyses to quantify the potential improvement in caseload outcomes achieved by resolving model-specific uncertainty and identify the key uncertainties to resolve to achieve that improvement.

Table S1.

The 37 published compartmental Ebola models used for this analysis

Model* no.	Model type	Outbreak country	Outbreak year	R0	Ref.	Caseload projected in original publication?
1	SEIHFR	Guinea	2014	1.46	7	Yes
2	SEIHFR	DRC	1976	4.71	13	Yes
3	SEIHFR	West Africa	2015	NA†	24	Yes
4	SEIHFR	West Africa	2014	1.80	25	Yes
5	SEIHFR	DRC	1995	2.70	9	Yes
6	SEIHFR	Uganda	2000	2.70	9	Yes
7	SEIHFR	Liberia	2014	2.22	3	Yes
8	SEIHFR	Sierra Leone	2014	1.78	3	Yes
9	SEIHR	West Africa	2014	2.00	26	No
10	SEIHR	Nigeria	2014	1.50–2.00	27	Yes
11	SEIHR	Liberia	2014	1.90	28	Yes
12	SEIHR	Sierra Leone	2014	1.36	28	Yes
13	SEIHR	West Africa	2015	1.94	29	Yes
14	SEIFR	West Africa	2015	1.60	24	Yes
15	SEIFR	Guinea	2015	1.79	24	Yes
16	SEIFR	Liberia	2015	1.81	24	Yes
17	SEIFR	Sierra Leone	2015	1.32	24	Yes
18	SEIFR	Guinea	2014	1.10	30	Yes
19	SEIFR	Liberia	2014	1.53	30	Yes
20	SEIFR	Sierra Leone	2014	1.33	30	Yes
21	SEIR	Guinea	2014	1.51	31	Yes
22	SEIR	Liberia	2014	1.59	31	Yes
23	SEIR	Sierra Leone	2014	2.53	31	Yes
24	SEIR	Guinea	2014	1.15	32	Yes
25	SEIR	Sierra Leone	2014	1.24	32	Yes
26	SEIR	DRC	1995	1.83	33	Yes
27	SEIR	Uganda	2000	1.34	33	Yes
28	SEIR	West Africa	2014	2.10	25	Yes
29	SEIR	DRC	1995	1.38	34	Yes
30	SEIR	DRC	1995	1.36	34	Yes
31	SEIR	Liberia	2014	NA	35	Yes
32	SEIR	Sierra Leone	2014	NA	35	Yes
33	SEIR	DRC	1995	3.65	36	No
34	SEIR	Uganda	2000	1.79	36	No
35	SEIR	Guinea	2014	1.30	37	Yes
36	SEIR	Liberia	2014	1.69	37	Yes
37	SEIR	Sierra Leone	2014	0.99	37	Yes

DRC, Democratic Republic of the Congo.

^*Model number is as in Dataset S1; parameter values are given in Dataset S1. Links to code are given in SI Text: Parameters.R, Functions.R, and Running models.R.

^†NA means that R₀ was not reported.

The Ebola outbreak highlighted the challenges and opportunities that arise when multiple modeling groups contribute models and projections to help inform decision making. The VoI framework allows us to study the robustness of conclusions in the face of multiple alternative models and discover important model sensitivities with respect to the ranking of interventions. By studying an ensemble of models, we can identify actions that are robust in the face of uncertainty as well as sources of uncertainty that warrant immediate study.

Results

In our simulations, starting with 10 infectious cases in a population of 10,000 individuals, the mean projected caseload was 5,615 ± 2,705 (SD), ranging from 184 to 9,887 cases (Fig. 1). Despite this difference in caseload projections, the majority of the models suggested similar management recommendations (Fig. 1).

Fig. 1.

Unconstrained caseload projections (Upper) and ranks of five management actions (Lower) under 37 published compartmental Ebola models with SEIHFR (representing susceptible, exposed, infectious, hospitalized, funeral, and removed compartments), SEIHR, SEIFR, or SEIR structures. For each model, five management actions were ranked from the worst (with highest caseload projection) as shown in light red to the best (with lowest caseload projection) as shown in dark red. Simulated population size is 10,000 people, and the effectiveness is 30% for each management action.

We define the effect of an intervention as the percentage change compared with a no intervention baseline. The effect of an intervention on underlying rates (e.g., transmission and hospitalization) is unlikely to be known a priori; therefore, we first projected caseload assuming that each intervention resulted in a 30% change in affected parameters. Given a 30% change for each intervention, the majority (22 of 37) of the models recommended reducing funeral transmission as the optimal action, and 29 of 37 or 36 of 37 models ranked it as among the top 2 or 3, respectively. Reducing community transmission was optimal for 10 of 37 models, and 33 of 37 models ranked it in the top 2. Reducing the case fatality ratio was optimal for 3 of 37 models, and reducing hospital transmission was optimal for 2 of 37 models, whereas increasing hospitalization was not optimal in any model (Fig. 1). The optimal management recommendation was closely associated with model structure; for example, all SEIR and the majority (four of five) of SEIHR models recommended reducing funeral transmission as the optimal action, whereas the majority (six of eight) of SEIHFR models recommended reducing community transmission (Fig. 1).

The final epidemic size (i.e., the total caseload) is linked to the basic reproductive ratio, R₀, in SEIR-like compartmental models, such as those analyzed here (12). To provide a deeper mathematical understanding of our individual model results, we therefore conducted elasticity (proportional sensitivity) analyses of R₀ to the parameters associated with five interventions in the full SEIHFR models (in which all parameters could be explicitly perturbed). This analysis revealed that the ranking of the elasticities was the same as that of the associated interventions, in which reducing community transmission ranked as the first (6/8 models recommended it as optimal) and reducing funeral transmission ranked as the second (Fig. 1 and Table S2); if we only have a single model, alternative interventions can be ranked by their effect on R₀.

Table S2.

The elasticity of R₀ to parameters associated with the interventions for the full SEIHFR models

Interventions	Parameters affected	Model 1	Model 2	Model 3	Model 4	Model 5	Model 6	Model 7	Model 8	Average	Rank
Community transmission	βIc1, βIc2	0.86	0.15	0.68	0.46	0.19	0.97	0.76	0.74	0.60	1
Funeral transmission	βFc, βFh	0.14	0.15	0.22	0.31	0.66	0.03	0.19	0.19	0.24	2
Case fatality ratio	δ1, δ2, δH1, δH2	0.45	−0.22	0.67	0.30	0.64	−0.01	0.15	−0.25	0.22	3
Hospital transmission	βIh1, βIh2	0.00*	0.71	0.11	0.23	0.16	0.00	0.05	0.07	0.17	4
Hospitalization	h1, h2	−0.08	0.68	−0.50	−0.06	0.04	−0.71	−0.08	−0.04	−0.10	5

Average gives the mean elasticity across eight SEIHFR models. Rank indicates the rank of the absolute value of the average elasticity. Model number and description of the parameters are as given in Dataset S1.

^*The hospital transmission parameter is not included in this model; thus, the calculated elasticity is zero. The elasticity rank of this intervention is unchanged if this model is left out of the analysis.

To illustrate the sensitivity of the optimal intervention to the intervention effect size, we also compared caseload projections under a particular intervention over a gradient of changes ranging from 10 to 100% in increments of 10% with projections under the rest of the four interventions with the baseline change of 30%. These analyses highlight that whether an intervention is ranked as optimal depends on its effect size. Reducing funeral transmission is optimal in 22 of 37 models if the effect is over 30% and optimal in 31 of 37 models if interventions associated with burials lead to an 80% reduction. Reducing community transmission is rarely optimal when the effect size is less than 30%, but it is recommended as the best intervention in 32 of 37 models if it can be reduced by 50%. Interventions aimed at reducing the case fatality ratio must achieve a reduction of 60% to be optimal in 26 of 37 of the models (Fig. 2 and Fig. S1). Notably, increasing hospitalization and reducing hospital transmission were rarely optimal interventions, even at 100% effectiveness.

Fig. 2.

The number of models for which a particular intervention is recommended as optimal for the interventions of (A) reducing funeral transmission, (B) reducing community transmission, (C) reducing case fatality ratio, (D) reducing hospital transmission, and (E) increasing hospitalization. Evaluation was based on comparisons of caseload projections under each specific management action over a gradient of changes ranging from 10 to 100% against all of the other management actions with a baseline intensity of 30%.

Fig. S1.

The management effect required for each of five management actions to become the optimal action under 37 published compartmental Ebola models. All 37 models are presented within the row for each action reordered by rank for that action. Evaluation was based on comparisons of caseload projections under each specific management action over a gradient of changes ranging from 10 to 100% against all of the other management actions with a baseline intensity of 30% change. The number of models for which a particular intervention is recommended as optimal generally increased with the intervention efficacy. Considering the intervention of reducing funeral transmission as an example, if the efficacy of an intervention to reduce funeral transmission is below 10%, no models recommend that intervention as optimal. When the efficacy increases to 20%, 4/37 models recommend it as optima; when the efficacy increases to 30%, 22/37 models recommend it as optimal. The number of models recommending an intervention to reduce funeral transmission as optimal is positively correlated with the efficacy of the intervention.

We calculated the expected value of perfect information (EVPI) (10, 11), which quantifies the maximum achievable improvement in management that could be obtained by identifying a single model as “best” before the implementation of specific decisions (Materials and Methods has a formal definition). The EVPI analysis showed that the expected improvement in management outcomes caused by resolving all model-specific uncertainties is an 11% reduction in caseload. We further conducted an analysis of the expected value of partial information (EVPXI) (10), which quantifies the expected improvement in management performance by resolving a subset of uncertainties. In particular, we quantified the relative contribution of uncertainty about model structure (SEIR, SEIHR, SEIFR, or SEIHFR) and caseload projection (models that projected low, low intermediate, high intermediate, or high case burden) to expected management outcomes. The EVPXI analysis illustrates that targeting the uncertainties in model structure could improve management (i.e., reduction in caseload) by 9% (82% of total EVPI), but targeting uncertainties in caseload projection could only achieve 1% improvement of management (12% of total EVPI). Thus, the two most important take home messages from our analyses are, although differences in caseload projections from the various models dominated much of the public discussion, (i) that intervention rankings are not affected by this issue and (ii) that resolving uncertainty in model structure is important for identifying optimal response strategies.

Discussion

Identifying the uncertainties that affect the choice of management intervention is critical for focusing scientific inquiry on questions that will improve the management of an epidemic. Conditional on a single model, a conventional approach is to evaluate the sensitivity of outcomes and management recommendations to parametric uncertainty. However, it is increasingly common that there are multiple independent models that can contribute to the evaluation of candidate interventions and policy development (2, 3, 8, 9). Thus, we present a framework for integrating model output to identify actions that are robust to the parametric, structural, and other uncertainties reflected in an ensemble of models. Our study showed that, despite large differences in caseload projections, management recommendations are broadly consistent across 37 published Ebola models; reducing funeral transmission and reducing community transmission are generally ranked as the top two best management options, whereas hospital-associated actions are rarely the best. Focusing on individual SEIHFR models, the same rank order was found for the proportional sensitivity of R₀ to the parameters associated with each of five interventions, in which reducing community transmission ranked as the first and reducing funeral transmission ranked as the second. This result aligns with classical theory (12): if we only have a single model, interventions that most affect the basic reproductive ratio are the best. Both funeral transmission (2, 9) and community transmission (9, 13) were identified as critical transmission sources and, therefore, targets for intervention against Ebola in previous studies. Despite the broad consensus among model recommendations, our EVPI analysis showed that resolving model uncertainties could improve management by 11%. Considering the 2014 Ebola outbreak, which had a caseload of 28,646 and a case fatality ratio of over 50%, this improvement would represent a reduction of 3,266 cases and 1,633 deaths averted. By conducting model–class-specific analyses of EVPXI, we found that the VoI for model structure was far higher than for caseload projections. Thus, the ranking of interventions was not strongly correlated with caseload projections, although expected caseload does provide information on how much effort will be required to halt the epidemic. The ranking of interventions differed more between than within model structures. This result could be a reflection of the inherent differences in the dynamics of different model structures or caused by differences between explicit and implicit representations of interventions within the same model structure. When the target compartments for specific interventions were not explicitly represented in the originally published models, then implementing interventions involved more subtle decisions. Although we tried to achieve the best standardization, our choices (detailed in SI Text) may hamper fair comparisons. An obvious solution is to focus on models that consistently and explicitly represent both compartments and interventions (implying an important role for the integration of operations research and epidemic modeling to ensure that modeled interventions are realistic and reflect real world constraints). However, we garnered important insights by studying all 37 models rather than restricting our analysis to 8 parameterized SEIHFR models.

Our study chose a 30% change to illustrate the management ranking based on caseload; we did not specifically consider the operational cost or constraints inherent in achieving that level of effect with each intervention. In practice, the same percentage change in one intervention might be harder or more expensive to achieve than another. Therefore, it is also important to consider operational and economic constraints (14, 15). Our analysis showed that some interventions, like reducing funeral transmission, can be ranked highly even if they only achieve low effectiveness, but others, such as reducing community transmission and reducing the case fatality ratio, are only ranked high if highly effective. In contrast, interventions targeting hospitalization rate and hospital transmission were rarely optimal, even at 100% effectiveness. Thus, although additional operational or behavioral information may be necessary to evaluate the potential effect of these interventions (e.g., because of interactions with social and cultural processes) (16), the framework that we have presented can be used to estimate minimal levels of effectiveness necessary to consider different classes of interventions. It also needs to be noted that, for purposes of illustration, we evaluated each management action separately in this study. However, management interventions are not mutually exclusive, and combined interventions should be studied. The VoI decision-theoretic approach can easily extend to this situation.

In this study, we focused on minimizing cases as our objective, because most of the published models projected caseload. However, recommended interventions may differ when considering different objectives, such as minimizing mortality or epidemic duration (17). In the case of the 2014 Ebola outbreak, intervention recommendations for the objective of minimizing deaths were the same as for minimizing caseload, because the case fatality ratio was quite similar for different compartments in the published models.

Overall, our study shows that, although differences in Ebola caseload projections were a subject of concern, identification and resolution of uncertainties that hinder management (here, model structure) are more relevant to the selection of optimal response strategies. Our work adds to a growing body of literature on uncertainty in disease dynamics from the standpoint of the decision maker and emphasizes that the uncertainty that is of interest epidemiologically (here, final epidemic size) may not be the same as the uncertainty that is most relevant to management (ranking of candidate interventions).

Our work used VoI to quantify the expected benefit of resolving uncertainty in a decision-making context conditional on model-specific projections. For the published Ebola models, we showed that achieving scientific resolution on model structure achieves 82% of the expected benefit of reducing all uncertainty considered. Scientific understanding (legitimate scientific differences of opinion/data) is frequently described in multiple competing models. Previous methods have assessed potential interventions conditional on individual models or used ensemble prediction (18). Instead, we propose the use of VoI to identify which scientific differences of opinion lead to different management recommendations; research should then prioritize learning about these operationally relevant uncertainties. Thus, our analysis is explicitly not focused on selecting a best model (as is commonly done). Rather, we use all models to identify actions that are robust and uncertainties that are important across the suite of candidate models. For Ebola, simplifying a large set of models into a smaller number of relevant classes (at the level of model structure) may allow for better integration of epidemiological modeling and decision making for policy. In other outbreak situations, resolution of other sources of uncertainty may be more important. More generally, our study shows how VoI analysis may provide “rules of thumb” to guide the decision-making process for other epidemics, such as Zika or avian influenza, where significant uncertainty about individual models remains but timely decision making is required.

Materials and Methods

Literature Survey of Published Ebola Models.

We conducted a literature survey for any published Ebola models on the Institute for Scientific Information Web of Knowledge and through the Google search engine using the search terms Ebola and model. We identified a total of 55 mathematical models in 35 publications and 1 online final report. The majority (37 of 55) of the models were compartmental models, whereas the rest (18 of 55) adopted one or more of a variety of modeling approaches, such as branching process models (19) and spatial models (20). Our study focused on 37 compartment models (a list is given in Table S1); this model type was the most widely used modeling approach for Ebola epidemic projection and management evaluation (2, 3, 9). The consistent framework of compartmental models allows for a general comparison of projected epidemic dynamics among models. Additionally, the widely proposed interventions in practice are either explicitly or implicitly applicable to these models, and therefore, the effectiveness of different interventions can be compared within and among models.

Compartment Models.

In an Ebola compartment model, individuals in a population are classified into different states as represented by different model compartments based on their health status. All individuals remain in the susceptible (S) compartment until they contract the virus through contact with infectious individuals; they then enter the exposed (E) compartment, where they are infected but are not yet infectious. Exposed individuals move to the infectious (I) compartment after a certain latent period, at which point they start to show symptoms and become infectious to other individuals. Infectious individuals will either remain in the community or be hospitalized; both are removed (R) from the chain of transmission through either recovery or death. Deceased individuals may infect others until they are buried.

Compartment models may incorporate different subsets of compartments to explore different transmission mechanisms or evaluate the effectiveness of different interventions. Hospital settings and transmission during funeral practices have been identified as critical transmission sources of Ebola. A number of published Ebola compartment models have addressed these transmission mechanisms by explicitly including a hospitalized (H) compartment and a funeral (F) compartment (3, 8, 9, 13). Based on the model structure, we classified 37 models into four categories: models with both H and F explicitly represented (8 models; referred to as SEIHFR), models with only H explicitly represented (5 models; referred to as SEIHR), models with only F explicitly represented (7 models; referred to as SEIFR), and models with neither H nor F compartments (i.e., these mechanisms of transmission were implicitly incorporated in overall transmission; 17 models; referred to as SEIR models). To ensure consistency, we recoded all of the 37 models within the same stochastic environment (epidemic birth and death processes simulated using the Gillespie algorithm with a tau-leaping approximation; details are given below) (3, 7) using R 3.2.1 (21). A figure of the global model, within which all of the 37 models can be represented as submodels, is presented in Fig. S2. Parameters for 37 models are listed in Dataset S1. Additionally, to ensure correct representation of the published models, we recalculated the basic reproductive number (R₀) (Dataset S1) using the next generation framework (22). Links to code to run all of the models are available in SI Text: Parameters.R, Functions.R, and Running models.R.

Fig. S2.

Flow diagram of the global model to represent each of 37 cited models as a submodel. Compartments in the global model are described as below. A full description of model parameters is in Dataset S1. Links to code to run all of the models are available in SI Text: Parameters.R, Functions.R, and Running models.R. E_c, exposed (infected but not yet infectious) individuals in the community; E_cd, exposed (infected and detectable but not yet infectious) individuals in the community; E_hw, exposed healthcare workers in hospital; E_rh, exposed healthcare workers in the community; E_v, hospital-visiting exposed individuals; F_c, infectious (dead) individuals in the funeral compartment from the community; F_h, patients who died in the hospital entering the funeral compartment; I_c1, infectious individuals in the early infectious compartment in the community; I_c2, infectious individuals in the late infectious compartment in the community; I_ccc1, infectious individuals at the early infectious stage in Ebola community care centers; I_ccc2, infectious individuals at the late infectious stage in Ebola community care centers; I⁻_ccc2, Ebola virus-negative patients with Ebola-like symptoms in Ebola community care centers; I_h1, infectious individuals at the early infectious stage in the hospital; I_h2, infectious individuals at the late infectious stage in the hospital; I⁻_h2, Ebola virus-negative patients with Ebola-like symptoms in the hospital; I_hw1, infectious healthcare workers in hospital; I_rh1, infectious healthcare workers in the community; I_v1, hospital-visiting infectious individuals at the early infectious stage; R, individuals removed from the chain of transmission through either recovery or burial; R_ccc, recovered individuals in Ebola community care centers; R_h, recovered individuals in the hospital; S_c, susceptible individuals in the community; S_hw, susceptible healthcare workers in the hospital; S_rh, susceptible healthcare workers in the community; S_v, hospital-visiting susceptible individuals.

Management Actions.

By surveying the literature, we selected five interventions that were broadly applied to control Ebola outbreaks: reducing community transmission, increasing hospitalization, reducing hospital transmission, reducing case fatality ratio, and reducing funeral transmission (2, 3, 8). Reducing community transmission (i.e., transmission in the community) is a general intervention that is achieved in a variety of ways, such as by providing household sanitation kits, improving contact tracing, improving self-quarantine of sick individuals in the community, reducing individual mobility and border crossing, and increasing community awareness through educational campaigns (2, 3, 8). Hospitalization increase can be realized by improving contact tracing and intensifying campaigns to identify and isolate patients, building more Ebola Treatment Centers, increasing the number of beds, and increasing necessary supplies and public support (3, 9). A reduction in hospital transmission can be achieved by encouraging the use of personal protective equipment for healthcare personnel treating infected cases and reducing hospital visits (8). Hydration of infected individuals has proved to be an important way to reduce mortality of Ebola cases, and various other new pharmaceutical approaches are being explored for future outbreaks. Funeral transmission (i.e., transmission at funerals) reduction can be achieved through improved funeral practices to increase safe burial by reducing risky behavior (2, 8).

Intervention Implementation.

Interventions can be modeled by changing the parameters thought to be influenced by the corresponding management actions. A reduction in transmission in the community, in hospitals, or at funerals can be modeled by reducing the transmission coefficients associated with these classes. Increasing hospitalization can be simulated by increasing the hospitalization ratio, and reducing mortality can be modeled by decreasing the case fatality ratio. Each intervention was assessed in each model in terms of the objective to minimize the Ebola caseload; for each model, we projected caseload under five interventions and then ranked the interventions from the best (lowest caseload) to the worst (highest caseload). All five interventions can be explicitly implemented in the full SEIHFR models; for the other models with some compartments unspecified, the corresponding interventions need to be simulated implicitly. For example, reducing hospital transmission can only be explicitly applied to the SEIHFR and SEIHR models, in which the H compartment is explicitly represented, but cannot be applied in the SEIFR or SEIR models, where the H compartment is unspecified. To be able to evaluate management options and conduct VoI analysis across all of the published models, including those that do not explicitly represent all infectious compartments, we calculated the implicit effect of an intervention via the average proportional contribution of the target transmission to the overall transmission based on the full SEIHFR models. For example, a proportional reduction of ΔH in hospital transmission can be explicitly simulated by multiplying the coefficient of hospital transmission by 1 − ΔH in the SEIHFR and SEIHR models. However, for the SEIFR or SEIR models where the H compartment is unspecified, an implicit simulation method needs to be applied. If P_H is the proportional contribution of hospital transmission to the overall transmission, then a reduction of ΔH in hospital transmission can be implicitly simulated via multiplying the coefficient of overall transmission by the factor 1 − ΔHP_H. A detailed description of the implicit simulation of the other interventions is provided in SI Text. The parameters used for the implicit management simulations were all based on the mean of the parameters across all of the published SEIHFR full models as shown in Table S3.

Table S3.

Parameters that were affected by five interventions, including reducing community transmission, reducing hospital transmission, reducing funeral transmission, reducing case fatality ratio, and increasing proportion of hospitalized cases

Parameter	Biological meaning	Parameters affected by the interventions					Mean value of parameters in full models
		com.tra.	hos.tra.	fun.tra	Fatality	Hospitalization
βC	Transmission coefficient in the community	Yes					0.24
βH	Transmission coefficient in the hospital		Yes				0.10
βF	Transmission coefficient at funerals			Yes			0.53
θ	Proportion of hospitalized cases					Yes	0.67
δC	Case fatality ratio in the community				Yes		0.63
δH	Case fatality ratio in the hospital				Yes		0.55
WC	Weights of βC				Yes	Yes	0.45
WH	Weights of βH				Yes	Yes	0.25
WF	Weights of βF				Yes	Yes	0.29
PC	Contribution of βC to βCHF	Yes	Yes	Yes	Yes	Yes	0.38
PH	Contribution of βH to βCHF	Yes	Yes	Yes	Yes	Yes	0.09
PF	Contribution of βF to βCHF	Yes	Yes	Yes	Yes	Yes	0.53
WCH_C	Weights of βC in βCH					Yes	0.64
WCH_H	Weights of βH in βCH					Yes	0.36
PCH_C	Contribution of βC to βCH	Yes	Yes			Yes	0.81
PCH_H	Contribution of βH to βCH	Yes	Yes			Yes	0.19
DC	Contribution of δC to δCH				Yes	Yes	0.51
DH	Contribution of δH to δCH				Yes	Yes	0.49
WCF_C	Weights of βC in βCF				Yes	Yes	0.61
WCF_F	Weights of βF in βCF				Yes	Yes	0.39
PCF_C	Contribution of βC to βCF	Yes		Yes	Yes	Yes	0.24
PCF_F	Contribution of βF to βCF	Yes		Yes	Yes	Yes	0.53

com.tra., Reducing community transmission; fatality, reducing case fatality ratio; fun.tra., reducing funeral transmission; hospitalization, increasing proportion of hospitalized cases; hos.tra., reducing hospital transmission.

The effect of an intervention, which we define as the percentage change compared with baseline, may not be known a priori. For illustration, we first projected caseload considering a 30% change for each of five interventions. Based on the caseload projection, we evaluated five interventions as well as the outcomes expected without any intervention for each of 37 models. We ranked them from the best (lowest caseload) to the worst (highest caseload). To illustrate how to identify at what effect size a particular intervention shifts to be best, we also compared caseload projections under a particular intervention over a gradient of changes ranging from 10 to 100% (with an interval of 10%) with the projections under the rest of the four interventions with the baseline change of 30%. For example, to assess the intervention of reducing community transmission, we did caseload projections by reducing community transmission from 10 to 100% for all 37 models, compared each projection under each level of change against the other four interventions with a change of 30%, and then ranked the intervention.

We implemented stochastic simulations for all models using Gillespie’s algorithm (7) with a tau-leaping approximation (3) to capture the random nature of epidemic birth and death processes (23). We performed 100 stochastic simulations for each management intervention–model combination, with 10 initial infectious individuals in a population of 10,000 individuals. When the parameters in the original publication were time-dependent, we fixed baseline parameters at values used at the start of the epidemic and/or set them to the no intervention baseline. To be able to conduct this broad comparison across all models, hospitalization capacity was not modeled, because only a few models considered this factor. Links to code to assess all interventions in all of the models in this study are given in SI Text: Parameters.R, Functions.R, and Running models.R. Additionally, to examine how sensitive R₀ is to parameters associated with five interventions, we conducted an elasticity (proportional sensitivity) analysis (Table S2). We calculated R₀ using the next generation framework and estimated derivatives numerically by the method of difference. We limited this analysis to the full SEIHFR models, in which all associated parameters were explicitly represented, thus allowing a perturbation of the full set of associated parameters. We then compared the rank of the elasticity of the parameters with the rank of the associated interventions.

VoI Analysis.

We calculated EVPI, which quantifies the maximum achievable improvement in management that could be obtained by resolving uncertainties before the implementation of specific decisions (10, 11). It is quantified as

$$\mathit{EVPI}={\displaystyle \sum _{\mathit{j}=1}^{\mathit{n}}}{\mathit{p}}_{\mathit{j}}\left(\mathit{op}{\mathit{t}}_{\mathit{a}}{\mathit{C}}_{\mathit{a},\mathit{j}}\right)-\mathit{op}{\mathit{t}}_{\mathit{a}}{\displaystyle \sum _{\mathit{j}=1}^{\mathit{n}}}{\mathit{p}}_{\mathit{j}}{\mathit{C}}_{\mathit{a},\mathit{j}},$$ [1]

where n is the total number of models, C_a,j represents management performance (i.e., caseload in this study) associated with taking intervention a under model j, p_j is the weight associated with model j (i.e., the belief that model j is the true model; subject to the constraint that the p_j sum to one), and opt_a indicates the optimum over all interventions (10, 11). In this initial analysis, we weighed the models equally.

EVPI describes the benefit of resolving all sources of uncertainty. In practice, because of limited time and resources, it may not be possible to collect all of the required information. In this case, it is more realistic to prioritize a subset of uncertainties to resolve to maximize management improvement. Analysis of the EVPXI quantifies how much management performance could be improved by resolving a subset of uncertainties and therefore provides a useful tool to identify which subset of uncertainties should be given priority if time and resources are limited (10). EVPXI is calculated as

$$EVPXI={\displaystyle \sum _{i=1}^q}op{t}_a{\displaystyle \sum _{j\in s_i}}{p}_j{C}_{a,j}-op{t}_a{\displaystyle \sum _{j=1}^n}{p}_j{C}_{a,j},$$ [2]

where n models are grouped into i = 1 … q mutually exclusive and exhaustive sets, set s_i has n_i models in it, p_j is the weight associated with model j, and C_a,j is the management performance under model j and intervention a. Therefore, EVPXI quantifies the improvement in management performance by resolving the uncertainty associated with s_i (10). We conducted EVPXI analyses for four subsets of models with different types of structures (SEIHFR, SEIHR, SEIFR, and SEIR) and also, four subsets of models with different ranges of caseload projections (<2,500, 2,500–5,000, 5,000–7,500, and >7,500 cases) to evaluate the improvement in management by resolving uncertainties in model structure and the range of caseload projections.

SI Text

Implicit Simulation of Interventions.

Our goal here is to provide a comprehensive EVPI analysis of all 37 compartmental models published for Ebola outbreaks. Because not all models have all compartments relevant to all proposed interventions, we were faced with some parameterization choices. Here, we outline our method for intervention simulation by defining a set of weights based on the mean of the parameters of all of the published full SEIHFR models to implicitly represent the contribution of unspecified compartments to the overall transmission in other models.

In a full SEIHFR model, there are three transmission compartments, which are community (C), hospital (H), and funeral (F). The relative contribution of each transmission compartment to the overall transmission coefficients (β_CHF) depends on the absolute magnitude of their transmission coefficients (β_C, β_H, and β_F) and their weights (W_C, W_H, and W_F), which reflect the fraction of infectious individuals in each compartment. Therefore,

$${\beta }_{CHF}=W_C{\beta }_C+W_H{\beta }_H+W_F{\beta }_F,$$ [S1]

and the weights that may be time varying are

$$W_C=\frac{1}{1+\theta +(1-\theta ){\delta }_C+\theta {\delta }_H},$$ [S2]

$$W_H=\frac{\theta }{1+\theta +(1-\theta ){\delta }_C+\theta {\delta }_H},$$ [S3]

and

$$W_F=\frac{(1-\theta ){\delta }_C+\theta {\delta }_H}{1+\theta +(1-\theta ){\delta }_C+\theta {\delta }_H},$$ [S4]

where θ is the proportion of infectious cases that are hospitalized. Parameters δ_C and δ_H are case fatality ratios in the community and hospital, respectively.

The proportional contribution of each transmission compartment to the overall transmission β_CHF can be written as

$$P_C=\frac{W_C\hspace{0.17em}{\beta }_C}{{\beta }_{CHF}},$$ [S5]

$$P_H=\frac{W_H\hspace{0.17em}{\beta }_H}{{\beta }_{CHF}},$$ [S6]

and

$$P_F=\frac{W_F\hspace{0.17em}{\beta }_F}{{\beta }_{CHF}}.$$ [S7]

Correspondingly, we write β_C, β_H, and β_F as

$${\beta }_C=\frac{P_C{\beta }_{CHF}}{W_C},$$ [S8]

$${\beta }_H=\frac{P_H{\beta }_{CHF}}{W_H},$$ [S9]

and

$${\beta }_F=\frac{P_F{\beta }_{CHF}}{W_F}.$$ [S10]

In SEIR models, transmission in heterogeneous compartments, such as H and F, is implicit, and an overall transmission coefficient β_CHF is used to implicitly reflect the combined contributions of β_C, β_H, and β_F. The choice to reduce ΔC in community transmission can be reflected in β_CHF by reducing the contribution of β_C based on the mean of the full SEIHFR models. The new effective transmission β′_CHF is

$${\beta }_{CHF}^{\prime }=\left(1-P_C\right){\beta }_{CHF}+(1-\mathrm{\Delta }C)P_C{\beta }_{CHF}.$$ [S11]

Similarly, the management option of reducing hospital transmission by ΔH can be achieved by recalculating β′_CHF as

$${\beta }_{CHF}^{\prime }=\left(1-P_H\right){\beta }_{CHF}+(1-\mathrm{\Delta }H)P_H{\beta }_{CHF},$$ [S12]

and the management option of reducing funeral transmission by ΔF can be quantified by recalculating β′_CHF as

$${\beta }_{CHF}^{\prime }=\left(1-P_F\right){\beta }_{CHF}+(1-\mathrm{\Delta }F)P_F{\beta }_{CHF}.$$ [S13]

Because the H compartment is unspecified in an SEIR model, we represent the management option of increasing the proportion of hospitalized cases θ by Δh in terms of a change in β_CHF, as a function of θ, and thus, as a change in the weight of hospital transmission W_H. Increasing the proportion of hospitalized cases will also consequently lead to a shift in the number of people who would recover/die in the community who instead recover/die in the hospital. Given that the case fatality ratio might differ between the community and hospital settings (8, 24), changing the proportion of hospitalized cases also consequently changes the weight of funeral transmission (W_F) and the weight of community transmission (W_C), both of which depend on the proportion of hospitalized cases and case fatality ratio as given in Eqs. S2 and S4. We included this to address the possibility that differences in case fatality ratio were important. However, the existing differences in case fatality ratio in this study are small. The new weights of W′_C, W′_H, and W′_F under the intervention of increasing the proportion of hospitalized cases can be calculated by replacing θ with (1 + Δh)θ based on Eqs. S2–S4. Thus, β′_CHF can be calculated based on Eqs. S1 and S8–S10 according to

$${\beta }_{CHF}^{\prime }=W_C^{\prime }{\beta }_C+W_H^{\prime }{\beta }_H+W_F^{\prime }{\beta }_F$$

$$=W_C^{\prime }\frac{P_C{\beta }_{CHF}}{W_C}+W_H^{\prime }\frac{P_H{\beta }_{CHF}}{W_H}+W_F^{\prime }\frac{P_F{\beta }_{CHF}}{W_F}.$$ [S14]

Following the same logic as for hospitalization interventions, the management option of reducing the case fatality ratio in the community (δ_C) and in hospitals (δ_H) both by ΔD in the SEIR model is done by a change in β_CHF occurring as a function of δ_C and δ_H and changes in W_C, W_H, and W_F. The new weights of W′_C, W′_H, and W′_F under the intervention of reducing the case fatality ratio can be calculated by replacing δ_C and δ_H with (1 − ΔD)δ_C and (1 − ΔD)δ_H based on Eqs. S2–S4. Thus, β′_CHF can be calculated based on Eqs. S1 and S8–S10 according to

$${\beta }_{CHF}^{\prime }=W_C^{\prime }\frac{P_C{\beta }_{CHF}}{W_C}+W_H^{\prime }\frac{P_H{\beta }_{CHF}}{W_H}+W_F^{\prime }\frac{P_F{\beta }_{CHF}}{W_F}.$$ [S15]

In an SEIFR model, the funeral transmission coefficient β_F is represented explicitly via the compartment F, whereas the compartments C and H are unspecified, and the transmission coefficients β_C and β_H are implicitly represented by a combined transmission coefficient β_CH. The weights of β_C (written as W_{CH_C}) and β_H (written as W_{CH_H}) can be calculated based on the full SEIHFR models as

$$W_{CH\_C}=\frac{W_C}{W_C+W_H}=\frac{1}{1+\theta }$$ [S16]

and

$$W_{CH\_H}=\frac{W_H}{W_C+W_H}=\frac{\theta }{1+\theta }.$$ [S17]

The overall transmission β_CH can be written as

$${\beta }_{CH}=W_{CH\_C}{\beta }_C+W_{CH\_H}{\beta }_H.$$ [S18]

Therefore, the proportional contributions of community and hospital transmission are

$$P_{CH\_C}=\frac{W_{CH\_C}{\beta }_C}{{\beta }_{CH}}$$ [S19]

and

$$P_{CH\_H}=\frac{W_{CH\_H}{\beta }_H}{{\beta }_{CH}},$$ [S20]

and β_C and β_H can be written as

$${\beta }_C=\frac{P_{CH\_C}\hspace{0.17em}{\beta }_{CH}}{W_{CH\_C}}$$ [S21]

and

$${\beta }_H=\frac{P_{CH\_H}\hspace{0.17em}{\beta }_{CH}}{W_{CH\_H}}.$$ [S22]

The intervention of a reduction of ΔC in community transmission can be achieved via a change in β_CH by reducing the contribution of β_C based on the full SEIHFR models, for which β′_CH is

$${\beta }_{CH}^{\prime }=\left(1-P_{CH\_C}\right){\beta }_{CH}+(1-\mathrm{\Delta }C)P_{CH\_C}{\beta }_{CH}.$$ [S23]

Similarly, the management option of reducing hospital transmission by ΔH involves a change in β_CH by reducing the contribution of β_H. Thus, β′_CH is given by

$${\beta }_{CH}^{\prime }=\left(1-P_{CH\_H}\right){\beta }_{CH}+(1-\mathrm{\Delta }H)P_{CH\_H}{\beta }_{CH}.$$ [S24]

The intervention of reducing funeral transmission by ΔF is direct for the SEIFR model, with β_F as

$${\beta }_F^{\prime }=\left(1-\mathrm{\Delta }F\right){\beta }_F.$$ [S25]

Because the H compartment in an SEIFR model is unspecified, the intervention of increasing the hospitalization proportion θ by Δh involves a change in β_CH, as a function of the proportion of θ, through a change in W′_{CH_C} and W_{CH_H}. W′_{CH_C} and W_{CH_H} are calculated based on Eqs. S16 and S17 by replacing θ with (1 + Δh)θ. β′_CH can be calculated based on Eqs. S18–S22 according to

$${\beta }_{CH}^{\prime }=W_{CH\_C}^{\prime }{\beta }_C+W_{CH\_H}^{\prime }{\beta }_H$$

$$=W_{CH\_C}^{\prime }\frac{P_{CH\_C}\hspace{0.17em}{\beta }_{CH}}{W_{CH\_C}}+W_{CH\_H}^{\prime }\frac{P_{CH\_H}\hspace{0.17em}{\beta }_{CH}}{W_{CH\_H}}.$$ [S26]

Because the case fatality ratio in hospitals may differ from that in the community, changing hospitalization may result in a change in the overall case fatality ratio. Given that the H compartment is unspecified in an SEIFR model, δ_C and δ_H are implicitly represented by an overall case fatality ratio δ_CH. Based on the full SEIHFR models, δ_CH can be written as

$${\delta }_{CH}=\left(1-\theta \right){\delta }_C+\theta {\delta }_H,$$ [S27]

where the contributions of δ_C and δ_H to δ_CH are

$$D_C=\frac{\left(1-\theta \right){\delta }_C}{{\delta }_{CH}}$$ [S28]

and

$$D_H=\frac{\theta {\delta }_H}{{\delta }_{CH}}.$$ [S29]

Accordingly, δ_C and δ_H can be written as

$${\delta }_C=\frac{D_C{\delta }_{CH}}{1-\theta }$$ [S30]

and

$${\delta }_H=\frac{D_H{\delta }_{CH}}{\theta }.$$ [S31]

The new case fatality ratio δ′_CH under the management option of increasing hospitalization is calculated as

$${\delta }_{CH}^{\prime }=[1-\left(1+\mathrm{\Delta }h\right)\theta ]{\delta }_C+\left(1+\mathrm{\Delta }h\right)\theta {\delta }_H$$

$$=\left[1-\left(1+\mathrm{\Delta }h\right)\theta \right]\frac{D_C{\delta }_{CH}}{1-\theta }+\left(1+\mathrm{\Delta }h\right)\theta \hspace{0.17em}\frac{D_H{\delta }_{CH}}{\theta }.$$ [S32]

The intervention of reducing case fatality ratio in the community (δ_C) and in the hospital (δ_H) both by ΔD can be directly modeled by multiply δ_CH by (1 − ΔD).

In an SEIHR model, whereas the hospital transmission coefficient β_H is represented explicitly via the compartment H, the transmission compartments of C and F are unspecified, and the transmission coefficients β_C and β_F are implicitly represented by the compound transmission coefficient β_CF. The weights of β_C (W_{CF_C}) and β_F (W_{CF_F}) are calculated based on the full SEIHFR models as

$$W_{CF\_C}=\frac{W_C}{W_C+W_F}=\frac{1}{1+(1-\theta ){\delta }_C+\theta {\delta }_H}$$ [S33]

and

$$W_{CF\_F}=\frac{W_F}{W_C+W_F}=\frac{(1-\theta ){\delta }_C+\theta {\delta }_H}{1+(1-\theta ){\delta }_C+\theta {\delta }_H}.$$ [S34]

The transmission coefficient β_CF is

$${\beta }_{CF}=W_{CF\_C}{\beta }_C+W_{CF\_F}{\beta }_F.$$ [S35]

Therefore, the proportional contributions of β_C and β_F to the transmission β_CF can be written as

$$P_{CF\_C}=\frac{W_{CF\_C}{\beta }_C}{{\beta }_{CF}}$$ [S36]

and

$$P_{CF\_F}=\frac{W_{CF\_F}{\beta }_F}{{\beta }_{CF}}.$$ [S37]

Correspondingly, β_C and β_F can be written as

$${\beta }_C=\frac{P_{CF\_C}\hspace{0.17em}{\beta }_{CF}}{W_{CF\_C}}$$ [S38]

and

$${\beta }_F=\frac{P_{CF\_F}\hspace{0.17em}{\beta }_{CF}}{W_{CF\_F}}.$$ [S39]

The intervention of a reduction of ΔC in community transmission can be captured by a change in β_CF by reducing the contribution of β_C. Therefore, we calculate β′_CF as

$${\beta }_{CF}^{\prime }=\left(1-P_{CF\_C}\right){\beta }_{CF}+(1-\mathrm{\Delta }C)P_{CF\_C}{\beta }_{CF}.$$ [S40]

The management option of reducing funeral transmission by ΔF is captured through a change in β_CF by reducing the contribution of β_F. Therefore, β′_CF is calculated as

$${\beta }_{CF}^{\prime }=\left(1-P_{CF\_F}\right){\beta }_{CF}+(1-\mathrm{\Delta }F)P_{CF\_F}{\beta }_{CF}.$$ [S41]

The intervention of reducing hospital transmission by ΔH is direct in an SEIHR model, and the β′_H is

$${\beta }_H^{\text{'}}=\left(1-\mathrm{\Delta }H\right){\beta }_H.$$ [S42]

The direct effect of the intervention of increasing the hospitalization proportion by ΔH is explicitly modeled by replacing θ with (1 + Δh)θ in the SEIHR model. The change in the case fatality ratio cannot be directly represented in the SEIHR model given that the funeral compartment is unspecified. It can only be reflected through altering β_CH by changing the weights in community and funeral transmissions. The W_{CF_C} and W_{CF_F} can be calculated based on Eqs. S33 and S34; β′_CF can be calculated based on Eqs. S35–S39 according to

$${\beta }_{CF}^{\prime }=W_{CF\_C}^{\prime }{\beta }_{CF}+W_{CF\_F}^{\prime }{\beta }_{CF}$$

$$=W_{CF\_C}^{\prime }\frac{\hspace{0.17em}{P}_{CF\_C}{\beta }_{CF}}{W_{CF\_C}}+W_{CF\_F}^{\prime }\frac{\hspace{0.17em}{P}_{CF\_F}{\beta }_{CF}}{W_{CF\_F}}.$$ [S43]

Following the same logic, the intervention of reducing case fatality ratio in the community (δ_C) and hospital (δ_H) both by ΔD is achieved implicitly through a change in β_CH by altering the weights of community (W′_{CF_C}) and funeral (W′_{CF_F}) transmission via replacing δ_C and δ_H with (1 − ΔD)δ_C and (1 − ΔD)δ_H based on Eqs. S33 and S34. The β′_CH can be calculated based on Eqs. S35–S39 and S43.

The R Code to Run the Analyses Described Here Is Provided in the Following Links.

• Functions.R: https://scholarsphere.psu.edu/downloads/mp48sc92g.

• Parameters.R: https://scholarsphere.psu.edu/downloads/2514nk66z.

• Running models.R: https://scholarsphere.psu.edu/downloads/0k225b23w.