Population-based simulations of influenza pandemics: validity and significance for public health policy

Abstract

Objective

To examine the validity and usefulness of pandemic simulations aimed at informing practical decision-making in public health.

Methods

We recruited a multidisciplinary group of nine experts to assess a case-study simulation of influenza transmission in a Swedish county. We used a non-statistical nominal group technique to generate evaluations of the plausibility, formal validity (verification) and predictive validity of the simulation. A health-effect assessment structure was used as a framework for data collection.

Findings

The unpredictability of social order during disasters was not adequately addressed by simulation methods; even minor disruptions of the social order may invalidate key infrastructural assumptions underpinning current pandemic simulation models. Further, a direct relationship between model flexibility and computation time was noted. Consequently, simulation methods cannot, in practice, support integrated modifications of microbiological, epidemiological and spatial submodels or handle multiple parallel scenarios.

Conclusion

The combination of incomplete surveillance data and simulation methods that neglect social dynamics limits the ability of national public health agencies to provide policy-makers and the general public with the critical and timely information needed during a pandemic.

Introduction

By 2006, many countries had responded to WHO initiatives to update contingency plans to mitigate the consequences of an influenza pandemic. However, some general concerns arose in connection with these national plans,^1,2 as it became clear that there would be a shortage of antiviral drugs and vaccine³ and that a pandemic would place new demands on public health information systems. At the global level, WHO’s Global Influenza Surveillance Network (FluNet) collects and processes influenza data from 83 countries,⁴ but at the national level few public health surveillance systems can either detect pandemic outbreaks or warn relevant agencies and the public. This inadequacy persists despite a 2005 report to the Government of the United States of America (USA) that identified public health information systems as a priority area for restructuring and investment to secure preparedness for pandemics and bioterrorist attacks.⁵ The development of a National Health Information Infrastructure in the USA had, at the time of the 2005 report, been proposed to detect atypical patterns of health-care use and to provide essential health information to citizens.⁶ This recommendation, however, has not translated into widespread practice, and many health information infrastructure projects remain in the planning stages.

Given that surveillance systems for collecting and analysing pandemic data are not sufficiently robust as a resource for policy planning and decision-making, attention has shifted towards computer-based simulation models. Using artificially generated community models as a basis, workers have forecast the effectiveness of different intervention strategies for containing or delaying the influenza pandemic at its expected source (e.g. rural south-east Asia). Longini et al.⁷ found that if the basic reproductive number (R₀) – the average number of secondary cases that a single case is expected to produce while still infectious in a completely susceptible population – was below 1.60, a prepared response with targeted antiviral drugs would have a high probability of containing the disease. When prevaccination was introduced into the model, targeted antiviral prophylaxis was found to contain an outbreak with an R₀ as high as 2.1. Addressing the same research question, but using an individual-based stochastic simulation model, Ferguson et al.⁸ reported that a combination of geographically targeted prophylaxis and social distancing measures is feasible only if the R₀ is below 1.8. Simulation studies have also included international air transportation patterns in the analyses of the early phases of a pandemic. Colizza et al.⁹ reported that the large-scale therapeutic use of antiviral drugs in all affected countries would mitigate a pandemic effect with an R₀ as high as 1.9 during the first year, if one assumes the antiviral drug supply is sufficient to treat approximately 2–6% of the population and that case detection and drug distribution are efficient. More recently, methods for representing specific social-contact networks in analyses of local influenza transmission have been developed. Using artificially generated social networks grounded in typical American community structures in their analyses, Glass & Glass¹⁰ have suggested that high-school students may form the local transmission backbone of the next pandemic. Therefore, closing schools and keeping students at home during a pandemic would remove the transmission potential in these age groups and could effectively thwart subsequent spread of the disease within a community.

In the absence of reliable pandemic detection systems, computer-based simulations have become an important information tool for both policy-makers and the general public. In this study we examine the validity and usefulness of population-based pandemic simulations from a national-level public health perspective. Specifically, we assess a simulated pandemic influenza outbreak in a Scandinavian community using a non-statistical nominal group technique.

Methods

Case-study simulation

The purpose of the case-study simulation was to investigate two intervention strategies – antiviral drugs and public policy interventions – on influenza transmission in a Swedish municipality. Specifically, we aimed to examine the effects on simulated intervention outcomes of variations in local sociodemographic data, such as alternative population distributions and household structures.

We simulated two different supply situations for each drug and quantified their respective effects as coefficients that modify the basic transmission probabilities assigned to mixing groups. Public policy interventions aimed at reducing the number of contacts were represented as the probabilities that individuals would withdraw from particular mixing groups. In our simulation experiment, we closed schools in an attempt to eliminate interaction within such groups. The resultant information was used to plan a public health response based on the Haddon matrix¹¹ (Table 1). Details of the simulator design, the case study and the results are provided in Appendix A and Appendix B (available at: http://www.crisim.org/documents).

Table 1. Strategic pandemic response framework, based on the Haddon matrix,¹¹ used for planning case-study simulations.

Pandemic timeline	Factors influencing pandemic spread
	Individual	Physical environment	Social environment	Agent/vector
Pre-event phase:	Immunization status	Community structure:	Pandemic policies	Mapping
Underlying risk factors for viral spread		- day care
		- schools
Event phase:	Nutritional state	Information infrastructure	Social networking patterns	Infectivity
Determinants of viral spread		Quarantine possibilities	Sustainability of social order
		Availability of first-aid kits
Post-event phase:	Self-care resources	Health-care facilities	Mobilization of civic resources	Virulence
Determination of final severity and consequences of the epidemic	Antiviral medication	Equipment and supplies	Mobilization of industrial response (production of vaccine and antivirals)

Assessor panel

We formed a multidisciplinary group of experts with skills in the realistic and practical application of simulations in public health policy-making to assess the plausibility, formal validity and predictive validity of the case-study simulation.¹⁴ The panel of nine assessors was made up of an experienced public health manager, a professor of social medicine, a professor of computer science, a professor of social and economic geography, a professor of medical anthropology, a former head of the simulations section at a national department of defence, a software developer with extensive commercial experience, a social forecasting researcher and a cognitive scientist. Two assessors (HE, MM ) designed and implemented the case-study simulation environment, and two (TT, JJ ) contributed to its design.

Data collection

The nominal group technique was used to assess the case-study simulation. A nominal group analysis is the structured use of group processes for systematically soliciting a set of informed judgments on issues described by limited scenarios or case descriptions.

The health-effect assessment structure was used as a framework for data collection.¹⁵ Specifically, we used an adaptation of the scheme suggested by Veerman et al.¹² and focused the assessment on plausibility, formal validity (verification) and predictive validity. Further details of the framework used for assessing the case-study simulation are in Appendix C (available at: http://www.crisim.org/documents). The experts were instructed to “assess the case-study methods and results with reference to the health impact scheme”.

Data analysis

The experts provided the first round of individual comments to the study coordinator, who included them in a case-study assessment document. The data analysis proceeded in cycles during which the experts first individually reviewed the assessment document and then participated in telephone group conference discussions (12 sessions lasting 90 minutes each). When new analysis cycles did not yield significant changes to the document, the assessment findings were considered to be established. In the second step of the analysis, the experts were asked to formulate the implications of the assessment results for simulation strategies on the basis of their own expertise and of the published literature. Specifically, the instructions for the second task were to “analyse the assessment results with respect to practical implications for the application of simulations in public health planning”. The experts first provided individual comments, which were composed by the case-study coordinator. Thereafter implications continued to be formulated in a process in which the experts independently reviewed a preliminary case-study document describing the case-study implications. The comments thus gathered were subsequently circulated to the entire expert group, and a consensus document was iteratively drawn up.

Results

Verification of social assumptions

The assessment came to focus on the intersection between the simulations of biological events and of societal processes in the case-study, particularly on the model of social order. When assessing the formal validity of the simulation, the assessors noted that central aspects of the baseline situation in the case-study community had not been included in the models. The implementation of pandemic response plans presumes a close collaboration between many groups, including public health organizations, commercial companies, and law enforcement agencies, and the successful operation of these plans is critically dependent on the protection of these coordinated processes. Although important lessons were learned during previous influenza pandemics¹⁶ and the 2003 epidemic of severe acute respiratory syndrome (SARS),¹⁷ the interdependence among the broad range of processes occurring in a society under true stress (such as a pandemic) is not fully understood. In other words, even relatively minor breaches of the social order may significantly affect key infrastructural elements underpinning current pandemic response strategies. This frailty in the response strategies was not reflected in the description of the baseline situation in the case-study community.

For instance, assumptions were made about an efficient and sustained distribution of available antiviral drugs. However, in the USA pharmaceuticals are often distributed through retail and food stores, whose employees are at particular risk of becoming infected because of the large number of customers. Moreover, many professionals involved in the pandemic response may fail to report to work during a pandemic for reasons other than actually falling ill, e.g. because of breakdown of transportation systems, deficient law enforcement or having to take care of children due to school closures. ¹⁸

We do not know how an influenza pandemic might affect logistics and staff in the distribution chains for pharmaceuticals, hygienic supplies and sanitary equipment during a catastrophic event.¹⁹ Unforeseen changes in behavioural patterns may also occur among acute care and hospital workers,²⁰ who in particular may be prone to abandoning their tasks if they have shorter tenure, high work stress and already strained social relationships.²¹

Cumulative increase of predictive validity

The global spread of an influenza pandemic has been estimated to accelerate at an exponential rate from around day 50.²² Thus, national governments have a short window of time to plan and implement appropriate response measures. The expert panel concluded that before the detection of an outbreak threatening to become a pandemic, effects on social order may not be predictable and the nature of the infectious agent, the efficacy of vaccines, and the availability of antiviral drugs can only be estimated. Thus, the predictive validity of simulations at early stages of a pandemic will inevitably be poor. To increase the validity of the predictions, the submodels used in simulations need to be modified as more information becomes available about factors such as the virulence of the strain and the efficacy of intervention strategies. In the case study, the simulation submodels were rendered flexible by separating them from each other and from the execution algorithms in the software. This separation saved both considerable programming and programme testing time. However, a major drawback of the flexible software identified in the assessment was the length of time each estimation took. For instance, a single estimation of R₀ in the case-study community (140 000 population) lasted approximately 5 hours on a standard personal computer. This direct relation between flexibility and calculation time is a disincentive to using simulations with high predictive value for national-level policy-making during a pandemic, when time is critical.

Towards socially contingent pandemic simulations

In the case-study simulation, questions were formulated in a standard public health framework under the assumption of a sustained social order during the pandemic. A more valid strategy, however, would have been to alter key social structures and processes as a function of disease effect on the community. Unfortunately, little is known about the theoretical and practical means for integrating the simulation of biological events, such as virus transmission between human hosts, with dynamic models of changes in population behaviour. Multi-level simulations based on “synergetics” and game theory,²³ simulations of policy strategies²⁴ and simulations based on geographically explicit data²⁵ are established fields of population research. Similarly, the simulation of virus dissemination in stable societies is also an established field of research on its own.²⁶ Nevertheless, few frameworks are available to support integrated modelling in these conceptually discrete but practically interrelated areas. Moreover, before biological and societal factors can be integrated, a valid common theoretical basis must be established.

Time geography^27,28 captures concurrent social processes as they unfold in time and space, based on the interaction between individuals and groups within the constraints of the physical and biological environment. This theory can be applied to integrate biological and societal tiers in pandemic simulations. In this way, questions analysed in standard public health planning frameworks can be made dependent on changes in the social order (Fig. 1). However, execution of such two-tier simulations requires even more complex and time-consuming computations. To save scarce planning time during a pandemic, generic (XML-based) software specifications can be used to distribute data and algorithms for parallel computing. Such computing arrangements can technically be administered in routine public health settings in the form of arrays of standard computers.

Fig. 1.

Outline of a second generation framework for planning and analysing pandemic simulations^a

^a The Haddon matrix¹¹ is extended by representation of social-geographical progresses^27,28 to take into account changes of social order during the pandemic. Boxes with full lines denote events in the epidemiological and social-geographic progresses, while broken line boxes represent structural factors affecting pandemic spread and/or potential targets for interventions.

Discussion

Population-based simulations are important sources of knowledge when planning national-level public health responses to pandemic influenza. The results of the case-study simulation assessment primarily apply to the “mixing group” approach to population-based simulations but can be extended to all models in which societies are represented as “compartments” of identical individuals mixing randomly. However, we noted that the formal validity of these simulations is challenged by the failure to take into account the behaviour of people during a pandemic and its effect on the social order. Additionally, the assessment led to questions about how simulation models can be adjusted to reflect dynamic changes in preconditions, e.g. disruptions to drug distribution routines.

Formal methods for macro-level societal forecasting were introduced early in the twentieth century.²⁹ In this research area, disruptions of the normal social organization have been expected, mainly when material resources are scarce or misallocated. When societal forecasting methods are applied to pandemic simulations, the lack of empirical grounding for using formal methods representing social processes in societies under severe strain comes to constitute a dangerous source of error. For example, the estimated effects from quarantine measures in Toronto during the SARS outbreak were diminished because disruptions of the social order led to compliance rates of only 57%.¹³

Furthermore, the application of formal methods to analyse societies under pressure has been strongly opposed by several prominent social scientists. Weber³⁰ and more recently Giddens³¹ have argued that using poorly validated models of social order leads to misunderstandings about the social world and how it operates. Although a central belief during the Enlightenment was that the social order could be controlled if science were sufficiently strong, modern social science, despite being more exact, no longer claims that changes in the social order in societies under severe pressure can be predicted. Moreover, researchers have only just begun to understand how disease shapes behavioural norms, and through them, social structures.³² The limitation of societal forecasting to stable societies is valid for time geography as well. In the context of pandemic simulations, the use of models and concepts from time geography may, however, facilitate the identification of social structures and processes that are likely to remain stable during a pandemic, and those structures that are at high risk of disintegration.

Another issue highlighted by the assessment was the direct relationship between computational time and flexibility in simulation models. In the case study, model flexibility was prioritized above computational efficiency. This strategy rests on the supposition that predictive validity can be increased if assumptions made early during a pandemic can progressively be replaced by validated observations and empirical data. In such situations, however, it is essential to “catalogue” the assumptions to permit accurate changes in the simulation models as new data become available. For example, in the case study it was assumed that asymptomatic infected individuals had a 50% lower risk of transmitting the virus than symptomatic ones. This assumption, which was based on estimates from previous simulation studies^33,34 is to be replaced when empirical data about an actual pandemic strain become available. For each such assumption to be included in a pandemic model, a choice must be made between re-examination of available data and acceptance of a previous estimate.

One possible way to systematically improve the predictive value of pandemic simulations is to include explicit scenario management in the methods. A scenario makes explicit how forecasts are concluded from hypothetical reasoning by overtly drawing out the expected consequences of specific facts and assumptions.^35,36 The use of scenarios is well established as a basis for public health responses to outbreaks of infectious disease.³⁷ Explicit scenario management would make it possible to store and display, at each point in time, the data, information and assumptions employed for a specific simulation model, together with their sources. Flexible simulation models combined with the ability to trace the data included and assumptions to their sources would make it possible to maintain evidence-based scenarios as a basis for improving predictive validity.

Our study has several limitations. First, the expertise of the assessors may not have been broad enough to cover all areas of relevance associated with pandemic simulations. For instance, expertise in legislative issues and statistics might have been useful. However, we kept the number of assessors (nine) to the minimum required by the nominal group technique (n > 8)¹⁴ to facilitate telephone conferences, which we found very useful in bringing together the views of experts in very different locations.

Second, procedural factors may have introduced bias during the case-study review. Originally, we chose the nominal group technique to avoid having particular individuals exert undue influence in face-to-face discussions. However, other researchers contend that participants in a consensus process are prone to reach an agreement if they are given time limits.³⁸ This “bandwagon” effect may be less likely when inspection documents and telephone conferences rather than questionnaires and graphic charts are used for feedback. To avoid premature conclusions, we continued the iterations until no individual group member contributed new comments upon inspecting the documents. Despite these measures, the nominal group procedure cannot be regarded as a guarantee against bandwagon effects or the dominance of individual experts.

Third, the case study did not incorporate the entire scope of pandemic simulation methods and techniques. In simulations of social events using multi-agent models, the behaviour of the agents and the attributes of their environment can both be modelled.³⁹ These multi-agent simulations are not intended to forecast or replicate observables.^23,40 Rather, their aim is to explain the systematic effects of behavioural patterns in explicit “artificial worlds”. Several of the issues raised in the present study may not apply to multi-agent simulations. However, the latter might be able to contribute important insights into the interplay between biological processes and changes in population behaviour during planning for a pandemic if appropriate interpretation frameworks are developed. Similarly, not all issues raised necessarily apply to the evolving infectious disease simulation paradigm with explicit representations of empirically grounded social networks.^41,42 However, baseline data on such networks are still scarce,^43,44 and methods for progressively modifying the network models to represent observed behavioural changes during pandemics are needed.

Our results demonstrate that the current methods applied for population-based pandemic simulations have important shortcomings. Formal validity is poor because assumptions about how human populations respond to stress are not included in the simulation models. Predictive validity suffers because simulation methods do not support the dynamic representation of the interaction between the microbiological, epidemiological and societal progressions during a pandemic.

Pandemic simulation methods need to be reformed to include representation of social dynamics and support for rapid model changes if national public health agencies are to provide policy-makers and the general public with important, valid and timely information. Such reformed methods will require a change of analysis paradigm from forecasting to “near-to-real-time” or “nowcasting” and the use of surveillance data for continual updating of simulation models and parameters. ■