Seven Challenges for Model-Driven Data Collection in Experimental and Observational Studies

Abstract

Infectious disease models are both concise statements of hypotheses and powerful techniques for creating tools from hypotheses and theories. As such, they have tremendous potential for guiding data collection in experimental and observational studies, leading to more efficient testing of hypotheses and more robust study designs. In numerous instances, infectious disease models have played a key role in informing data collection, including the Garki project studying malaria, the response to the 2009 pandemic of H1N1 influenza in the United Kingdom and studies of T-cell immunodynamics in mammals. However, such synergies remain the exception rather than the rule; and a close marriage of dynamic modelling and empirical data collection is far from the norm in infectious disease research. Overcoming the challenges to using models to inform data collection has the potential to accelerate innovation and to improve practice in how we deal with infectious disease threats.

Introduction: What is the role of models in data collection?

When people refer to “models” of infectious disease transmission, they usually mean something far more specific than the word “model” indicates. The term is generally used to refer to a system of equations or computer program that explicitly represents the mechanisms of disease transmission and pathogenesis. This is in contrast to models purely of statistical association that are common throughout the medical literature. By setting forth a mechanistic hypothesis in a mathematically precise form, models become tools for generating (perhaps unexpected) predictions which can be used to test the underlying hypotheses through confrontation with data. Though this use of models has a long tradition throughout all branches of science (including infectious disease epidemiology and ecology), it often takes a back seat to other uses of infectious disease models. The highest profile infectious disease modeling work is often aimed at making predictions or filling in gaps in existing data. These uses of models by definition presume that the hypothesis captured in the model is close enough to the truth to capture the dynamics of the system relevant to the task at hand, and provide answers only conditional on the correctness of the model. While many researchers put great effort into fitting both the structure and parameters of models using existing data, data are rarely collected with the explicit purpose of testing model hypotheses, and many models go unchallenged and untested after they are first presented.

Models are powerful, in part, because they can turn a hypothesis or theory into a tool for making precise predictions. Yet even in this capacity infectious disease models are underutilized in the data collection process. For example, sample size calculations and power analyses are de rigueur for the design of observational studies and clinical trials. However, cases of transmissible infections are non-independent, limiting the utility of standard theory. Using mechanistic models that account for the transmission process can allow robust estimation in the setting of these “dependent happenings” and tell us not only how much data to collect but when to collect it ¹.

Model-driven data collection does occur, and a prime example is the Garki project ². This seminal project studied the transmission of malaria under different control interventions and captured longitudinal data on both disease and vector dynamics. Thirty years after its publication it remains one of the most valuable datasets for parameterizing malaria models (e.g., Griffin 2010 ³), and has been cited over 650 times. In part because input from models was used in the design of the study, it provides an almost unique dataset on the dynamics of malaria across a number of seasons and under different interventions, providing an example for future studies which has rarely been emulated.

Through sensitivity analysis and quantification of uncertainty, models can be used to elucidate the parameters and processes that contribute most to our inability to make predictions with a high degree of confidence, and those which are less important. They can thereby indicate where experimental effort would be best directed to improve the predictive power of a model. An example of where this is being attempted is HPTN 071 (PopART), a community randomized trial of combination prevention packages including early antiretroviral therapy initiation to control HIV transmission in Africa. In the analysis of this trial, models are being used to estimate endpoints (e.g.., community incidence), improve study design, and identify which processes are the main drivers of uncertainty ⁴.

For the modeling of infectious disease to reach its full potential, it must become more tightly integrated with the data collection process. Despite successes such as those described above, significant challenges to successful model-driven data collection remain. Some stem from technical or cultural issues that are, in principle, easily surmountable. However, others are inherent in the role that models are often called on to play in public health response and scientific research, hence may be difficult or impossible to overcome.

In this paper, we start by discussing some fundamental challenges in the relationship between models and data collection, and end with specific challenges in current practice. Throughout we highlight the potential role of “modelers” in the data collection process. Though infectious disease modelling has grown into a distinct specialty, here we refer to anyone who defines and implements a mechanistic model of disease transmission as a “modeler”. In addition to our headline challenges, we have tried to identify specific research avenues under each (mentioned at the end of key paragraphs) that could result in significant progress in confronting the challenge.

1. Ensuring a strong empirical basis for models used to fill gaps in data and knowledge

One primary role of models is to use our mechanistic understanding of a system to fill gaps in available data. Gaps in knowledge may occur because data is difficult or expensive to collect, because we are trying to understand past events which can no longer be directly observed, or because we are confronting a novel disease which is not yet well understood. In each of these contexts models play a hugely important role because of the lack of information, but are at the same time hamstrung by the lack of data with which to test model assumptions or fit parameters. While data will always be scarce in these situations, each presents an opportunity to use mechanistic models to make more effective use of existing data and guide ongoing data collection.

Data on disease incidence is difficult and expensive to collect on a broad scale; and passive clinical surveillance may include only the most severe cases or be clouded by a non-specific clinical profile. However, the burden of disease is one of the most fundamental pieces of epidemiologic information used in setting the public health agenda. Hence, mechanistic models are often used to fill the gap. For instance, transmission models have been used to help estimate the global burden of measles ⁵, using our knowledge of how susceptibility drives epidemic dynamics to infer the true number of cases from what was observed. Likewise, models have been used to help translate observed cases of acute flaccid paralysis to polio incidence through our understanding of transmission and the symptomatic attack rate ⁶. There are opportunities to test and improve models that fill data gaps. Models can be used to identify efficient data collection activities that validate the model but do not require the effort and expense of collecting the data the model is meant to infer. Innovative methods and systems for updating both the model predictions and the model assumptions in real time as new surveillance data becomes available could greatly increase their value in public health practice.

Understanding disease dynamics is often dependent on measuring disease incidence years or decades in the past, making the design of suitable data collection particularly challenging. Past incidence rates may be unmeasured because a disease circulated before it was identified (e.g., HIV before the 1980s), because acute infection often occurs without identifiable symptoms (e.g., dengue, HIV), or because of poor surveillance. In the latter two cases, even recent incidence patterns may be unknown. Dynamic models can often be used to infer past incidence with current cross sectional data or historic samples. Age specific serologies can be used to estimate the past force of infection, and have been used to measure historic patterns of the force of infection for dengue and other diseases ⁷. Phylogenetic models can be paired with simple epidemic models to infer past epidemic dynamics, as has been done with HIV and hepatitis C ⁸. These techniques, particularly phylogenetic inference, have become quite popular, but validation has been largely limited to simulation studies (e.g., Robinson et al., 2013 ⁹, Volz et al., 2012¹⁰). Studies aimed at collecting prospective data specifically to evaluate serologic and phylogenetic approaches to inferring incidence would help to place these inferences on firmer footing.

When responding to emerging epidemics, the problem of missing data is particularly acute. Here we are forced to forecast the course of an epidemic with limited knowledge of the pathogen and burden of disease. Critical data must be collected to carry out this task, some of which can be measured most effectively early on in the process of disease emergence. However, this data is not routinely collected early on, whether due to the difficulty of collection, competing priorities or its value being unrecognized. In particular, the tendency is to focus almost exclusively on cases early in an epidemic, whereas those who were at risk but did not become infected may carry the most information in terms of population susceptibility and disease transmissibility. A notable exception is the 2009 H1N1 influenza pandemic in the United Kingdom, where data collection was guided by the long-term involvement of modelers in the design of control programs. Although not all the data requested was collected, these efforts enabled policy-relevant modelling during the early stages of the epidemic ^11–13. This experience illustrates how integration with the public community can pay off in better inferences to support policy. At the time of writing, modelling is playing an important role in the response to the Ebola outbreak in West Africa, making use of the detailed contact tracing data collected as part of the response (though for purposes other than modeling) ¹⁴. Researchers should challenge themselves to identify the most useful classes of models in an emerging epidemic and the data needed to parameterize them. They should then work with public health officials to integrate collecting this data into epidemic response plans before such a response is needed.

2. Design studies that account for the dynamic nature of disease systems

Mathematical and computational models are most useful in understanding disease systems that are dynamic, often containing structural relationships and feedback loops that lead to behaviors that are unexpected or do not follow simple functional relationships. Furthermore, these dynamics are often driven by stochastic processes which are often hard to predict. These factors present particular challenges for data collection.

While models are extremely useful in understanding disease systems, they cannot capture every factor that plays a role in driving disease incidence. Stochasticity and factors outside the modeled system may play a primary role in determining whether epidemics occur and how they develop once underway. Typical examples include disease introductions, public response to real or perceived disease threats, and pathogen evolution. It is not always desirable, or even possible, to model such exogenous factors. However, when designing studies (or even using existing data) aimed at testing models, parameterizing models, or assessing the impact of a control measure, care must be taken in separating the results of factors lying outside the system from those captured by the model.

The dynamic nature of the disease processes leads to many practical issues in study design. Accurate sample size calculations must take into account disease dynamics, and the resulting non-independence of events. Outside of adjustments for correlation within clusters (i.e., calculating design effects based on intra-class correlation coefficients¹⁵), such calculations are rarely done and may require complex simulations. The study itself may have an effect on the process being studied, and this effect may be magnified by dynamic feedback loops in disease systems. Even “observational” studies may affect disease dynamics, as we may be ethically required to treat the cases of the disease we detect, and this treatment will affect future incidence ¹⁶. Finally, interpretation of our observations is contingent on the current state of the system. It is often assumed the dynamic system is in steady state, but this is rarely the case, and the phenomena we are most interested in often involve perturbations to the system that will guarantee that it is not. For instance, the implications of a particular number of individuals being infected with a disease for broader disease dynamics depends on whether data was collected when disease incidence was growing, receding or in steady state (i.e., the current dynamic regime). Since it is usually impossible to measure the full state of the system, successful model driven data collection must not only measure state variables (e.g., the number susceptible or infectious), but also attempt to determine the dynamic regime in which those variables were collected.

Simulation of trial design is a growing area of research, with numerous applications to vaccine trials (e.g., Van de Velde et al., 2007¹⁷; Yang et al., 2006 ¹⁸) and growing use in other settings (e.g., PopART⁴). The development of standard tools similar to those available for standard sample size calculations, or even a list of best practices, would go a long way to expanding the use of mechanistic models in study design.

3. Identifying critical data collection activities in light of high dimensional parameter spaces and substantial structural uncertainty

Infectious disease systems are often the result of a large number of interacting social and biological processes; hence they can only be fully characterized by complex models. However, model complexity carries a price, since flexibility makes models difficult to fit to data or test in a meaningful manner. Simplification may be a path to creating models that can be fit to data, but may come at the cost of failing to capture essential system dynamics. High dimensional models can fit more complex trajectories, but may have multiple parameter sets that provide almost indistinguishable fits and can be almost impossible to test. For models to drive data collection, modelers face the challenge of identifying the specific parameters or predictions of their model that are most essential in increasing our understanding of the system and testing the structure of the model itself. Fortunately, mathematical and statistical techniques exist for identifying what components have the most influence on the predictions of complex systems (e.g., Saltelli 1999 ¹⁹, Caswell 2007 ²⁰). A fruitful avenue may be to expand these techniques to more directly estimate the value of critical experiments or studies in reducing our uncertainty about critical outcomes.

Even when the critical study to evaluate or improve a model has been designed, that study may be difficult to perform (e.g., outcomes may be hard to measure; sample size requirements may be unreasonably large). When specifying a model reveals these difficulties, the model is serving one of its most important roles in service of data collection: motivating precision in the statement of assumptions and hypotheses. For instance, the hypothesis that a vaccine is “85% effective” at first seems simple to test in an observational study, but once we attempt to put this into a model the subtleties of this statement become clear ²¹: since vaccination can change the course of an epidemic the comparison population must be carefully chosen depending on whether we seek to measure only direct protection or both direct and indirect effects; and our interpretation of results may differ markedly if we believe vaccination prevents infection completely (as with oral polio vaccine) or only stops the development of disease (as with injectable polio vaccine).

4. Collecting data to test and parameterize models that bridge spatial scales

Models of interventions also often rely on assumptions about how interventions work at an individual level. For example, modeling a vaccination intervention involves assumptions about whether protection is homogeneous or heterogeneous in the population, how it affects susceptibility to infection compared to the probability of developing disease given infection, and how it reduces onward transmission ²². Each of these differences at the individual level has important implications for how disease spreads within populations.

Recent developments in individual- or agent-based simulations have helped bridge our understanding of the dynamics of infectious disease transmission at different scales, from the individual to households, schools, towns, cities and countries ^23,24. These models have also served as powerful tools for communicating the insights from models to policy makers. The best examples in this area are based on well validated sub-models fit to data on population distributions, human movement, epidemiological parameters, and within host processes. However, they are rarely tested against epidemic data at the same spatial scale (exceptions include Cauchemez et al., 2008 ²⁵). For these approaches to reach their full potential they require further validation, but how to do that? Collecting data at the spatial and temporal detail of the predictions created by these models is unrealistic, and the situations modeled may be so unique as to make it impossible to identify a comparison set. Evaluation of marginal results (i.e., sets of statistics summarizing more detailed model output) is the clear pathway to validating these models, but work is needed to identify which set of marginal results need to be tested for a model to be considered adequately validated for use in forecasting or inference. Innovative study designs may help to test key model predictions (e.g., the Fluscape study²⁶), and there may be benefit in designing models and validating data collection activities in concert to make validation feasible.

5. Collecting data over time scales commensurate with model predictions

The predictions from mathematical models often play out over many years or even decades. Such predictions as increases in herpes zoster following the introduction of varicella vaccination²⁷ or reductions in cervical cancer incidence following HPV vaccination²⁸ take years to manifest. Likewise, proper parameterization and evaluation of dynamic models often requires decades of data, even for diseases with relatively well observed outcomes and simple dynamics (e.g., measles²⁹). Hence, data collection activities to test model hypotheses or improve model results may need to take place on timescales not in line with the usual process of scientific research or funding. Focusing on studies that are inexpensive but sustainable may help to identify incremental changes to normal surveillance procedures and sustainable data collection activities that will yield results over the course of decades. Starting and sustaining such long term research projects can present a substantial challenge, as they likely must be sustained outside of normal research funding mechanisms (which typically fund in increments of 5 years at best). Investigators may have to content themselves with helping future scientists, and public health and scientific priorities may have changed by the time results come in. However, long running cohort studies such as Multicenter AIDS Cohort Study³⁰, Women’s Health Initiative³¹, the Rakai Health Sciences Program³², and the Framingham Heart Study³³ have shown that long term data collection activities can yield important, and often unexpected, results. If the value of these data can be demonstrated to the public health agencies, they can possibly be included in routine surveillance for many decades to come. Methods that both make incremental use of data to improve models and provide results as data is collected may help to ensure the sustained effort needed to test long term hypotheses.

6. Integrating infectious disease dynamics and modeling into the every-day scientific process

Modelling and the study of infectious disease dynamics is sometimes treated as lying outside the normal scientific process (particularly by those outside of the field), being somehow qualitatively different than laboratory science or other types of epidemiologic studies. Modelers themselves may be in part responsible for this perception due to the frequent use of a “parameters from the literature” approach to building models. However, models should be an integral part in the ongoing cycle of hypothesis generation, data collection, and hypothesis refinement ³⁴. A framework for achieving this goal is to build collaborations that closely integrate modeling, field studies and laboratory experiments.

Experimental studies are an area where relationships between modelers and experimenters have generated questions which can be addressed directly. For instance, the interplay of mathematical models and experimental data in animals has contributed to our understanding of the generation and maintenance of CD8+ T-cell memory ³⁵. As an example: a particular model of homeostasis suggested the total number of CD8+ T-cells remains stable regardless of exposure to additional pathogens. A study inducing new long-lived CD8+ T cells in mice was performed to test this hypothesis. Unexpectedly, the total number of memory CD8+ T-cells doubled to accommodate the new cells³⁶. Thus, a new theory that the number of memory cells in mammals adapts according to immunological experience was developed by an iterative discussion between modelers and experimenters, identifying what data needed to be collected to test a clearly defined hypothesis.

While numerous instances of such collaborations exist, the perception of the scientific importance of infectious disease dynamics is often disconnected from its true importance in disease systems. Vaccines are an excellent example: licensure is based only on individual level effects (though indirect effects may be estimated); while public health decisions are heavily influenced by the indirect effects from vaccination (e.g., herd immunity) and dynamic components. Integrating cross-disciplinary experiences into the training of field, laboratory, mathematical and computational scientists could help make all more effective and ensure a role for dynamic models in the scientific process.

7. Making it common practice to identify data that would test model hypotheses

The presentation of models and model results should provide a golden opportunity for identifying those data or studies that could test the hypotheses underlying the model, reproduce results or show generalizability. However, few take this opportunity, and there is a tendency for models to be presented and then forgotten with little or no reflection on the quality of the results as more information becomes available. For models to play an important role in the progress of science, it is critical to test their qualitative and quantitative predictions and to refine the underlying models over time. Perhaps standard practice in the field should be to have a “what studies need to be done” section at the end of each paper presenting a model or modelling results. As “The Journal on Infectious Disease Dynamics”, Epidemics is well situated to promote such a practice by encouraging or requiring such a section for research manuscripts.

Conclusion

These are broad challenges to strengthen the relationship between models and data collection across experimental, epidemiological and clinical studies of infectious diseases. The breadth of these challenges provides many opportunities to develop new techniques and standards by applying these concepts to studies in all areas of infectious disease research. The core principle behind many of the challenges discussed is that mechanistic models have a role to play early in the process of experimental and epidemiological study design, and not only be used for secondary data analysis. The challenge for the modeler is to identify priorities for what pieces of information are needed and how to collect them within the confines of an experimental or epidemiological study. This discussion will help clarify both the model and study goals, potentially leading to new “Garki projects” that will drive discovery and innovation among infectious disease researchers across disciplines.