Systems Modeling to Advance the Promise of Data Science in Epidemiology

Abstract

Systems science models use computer-based algorithms to model dynamic interactions between study units within and across levels and are characterized by nonlinear and feedback processes. They are particularly valuable approaches that complement the traditional epidemiologic toolbox in cases in which real data are not available and in cases in which traditional epidemiologic methods are limited by issues such as interference, spatial dependence, and dynamic feedback processes. In this commentary, we propose 2 key contributions that systems models can make to epidemiology: 1) the ability to test assumptions about underlying mechanisms that give rise to population distributions of disease; and 2) help in identifying the types of interventions that have the greatest potential to reduce population rates of disease in the future or in new sites where they have not yet been implemented. We discuss central challenges in the application of systems science approaches in epidemiology, propose potential solutions, and predict future developments in the role that systems science can play in epidemiology.

Systems science focuses on the study of complex adaptive systems and includes methodological approaches, such as systems dynamics models (1), network models (2, 3), and agent-based models (ABMs) (4). These approaches are often characterized by nonlinear relations and feedback processes, such that small changes can produce emergent properties that are not entirely predictable by individual components. From a methodological perspective, complex systems approaches use computer-based algorithms to model dynamic interactions between units, or agents, within and across multiple levels. These models have been extensively developed for modeling infectious disease transmission (5–8) and are increasingly used in chronic disease and injury epidemiology (9–15).

HOW DOES SYSTEMS SCIENCE FIT IN WITHIN THE EPIDEMIOLOGY MACHINERY?

The value of systems science models in epidemiology lies in their ability to complement traditional epidemiologic tools when real data are not available or able to answer epidemiologic questions and to answer questions that are ill-suited to traditional epidemiologic methods (6, 16–21).

First, systems science approaches can be used to test assumptions about underlying mechanisms and feedback processes that give rise to the population distribution of complex health behaviors and disease. By assigning a set of behavioral and transition probabilities, we can examine the types of conditions that produce observed disease patterns and provide quantitative bounds around the plausibility of mechanisms that explain these patterns. An understanding of these mechanisms can provide insights into why past interventions may have succeeded or failed. For example, to explain trends in prescription opioid use, misuse, and overdose, Wakeland et al. (22) modeled a complex system in which individuals initiated nonmedical use, transitioned to paying for opioids and tampering with them, and then further transitioned to heroin use. The extent to which drug misuse trajectories were influenced by the availability, accessibility, and physical properties of the prescription opioid supply was tested based on the underlying simulated dynamics that gave rise to historical trends in use and abuse of nonmedical prescription opioids. This simulation of mechanisms suggested that interventions to reduce informal sharing of opioids could reduce opioid abuse to a greater extent than could creating tamper-resistant prescription opioid formulations.

Second, we can use systems science approaches to identify the types of interventions that have the greatest potential to reduce population rates of disease, either in the future or in new sites where they have not yet been implemented. Systems science models can complement evaluations of existing interventions, integrating counterfactual questions with modeling strategies that re-weight effect estimates based on concerns about transportability (23, 24). In this case, systems models can answer research questions about interdependent causal effects that are bedeviled by concerns about feedback loops, interference, and spatial dependence (25). For example, an ABM was used to describe transmission of human immunodeficiency virus (HIV) in a dynamic network of people who injected drugs, those who used non-injection drugs, and those who did not use drugs (26). By simulating a series of hypothetical intervention scenarios, the authors found that a significant scale-up of multiple prevention programs would be necessary to reduce rates of new HIV infections in people who inject drugs over the next decades.

In addition, systems science models such as ABMs that embed systems dynamics and network models can answer questions about the impact of interventions given selection into social networks or spatial contexts, as well as placement of interventions. For example, Keane et al. (27) predicted what “could” happen to opioid overdose rates if they varied the number and spatial location of naloxone distribution sites and the number of naloxone kits provided in a community, given different assumptions about secondary distribution of naloxone within social networks and the local dynamics of substance use.

CURRENT CENTRAL CHALLENGES IN THE USE OF SYSTEMS SCIENCE APPROACHES

Advances in computational machinery and in the use of “big data” for public health have made it possible to answer increasingly sophisticated questions with systems science, yet challenges abound. First, systems science models make strong assumptions about behavioral and disease probabilities, the structure and function of networks, and the dynamics of disease. Hence, model calibration and validation remain a central challenge, because such data are often missing and/or confounded; further, calibration often requires putting together data elements from multiple, disparate data sets. This issue has become especially relevant as there has been a push toward models that replicate a “real-world setting” and rely on multiple assumptions about individuals, their interactions with each other, and their interactions with their environment. Second, concerns exist about the risk for bias, especially with increasingly complex models (28). Because systems models often treat parameters of intervention effects obtained from one population as causal effects in a different population, biases arise because of lack of transportability when 2 populations have different underlying risks for the study outcome or different distributions of unmeasured confounding (29). Further, treating parameters of past intervention effects obtained from one population as causal effects in a different population gives rise to the potential for collider bias in the context of time-dependent confounding (28, 29). Third, the central challenge related to the issues discussed above lies in validation. In the absence of data to calibrate model parameters, estimation of parameters is often carried out by altering parameter values until model summary measures match those of the observed data. Unfortunately, there are multiple sets of parameters that can generate similar model summary measures, making it difficult to conclude whether the model assumptions are valid.

HOW CAN WE OVERCOME THESE CHALLENGES?

Triangulation is critical to address concerns about calibration, bias, and validation in systems science models for public health. First, we need to invest in iterative efforts that combine primary data collection, systems modeling, and implementation of interventions (30). Specification of a systems dynamic, network- or agent-based model forces us to be precise about the source of each model parameter and in that way provides insight into key gaps in data that need to be collected. An ideal approach would involve modeling, identifying gaps in key parameters, investing in primary data collection to fill such gaps, and re-parameterization of a better-calibrated model. In addition, in cases in which systems modeling is used to predict the impact of a specific intervention, such models should form part of a larger evaluation effort in which initial predictions about intervention effects estimated through systems models can be tested through the implementation and evaluation of actual interventions. In turn, this could inform a more well-calibrated and valid model that could be used to answer new questions about the mechanisms through which such interventions might work and their impact on other contexts and time frames.

Second, we can incorporate epidemiologic causal inference methods to address sources of bias in the estimation of systems science models. Some have suggested that methods such as parametric g-computation can be used instead of systems science approaches to examine complex systems (28, 29), as they can handle interdependent causal effects and feedback loops but rely on a single data set to calibrate simulations and are more robust to threats of time-dependent confounding and transportability violations (29). Although such methods are quite valuable, they cannot be used, in isolation, to answer precisely the types of questions that systems science models are best suited to answer, particularly questions that require explicit modeling of the role that space and social networks could play in shaping the impact of new interventions. Yet, they can play a key role in strengthening the validity of systems science models. For example, an ABM could use parametric g-computation to estimate specific causal effects of well-defined interventions within a broader system, adding layers to the model to test how processes of social interaction, interference, and space modify the impact of the intervention. Indeed, g-computation methods excel at identifying causal effects, whereas ABMs require causal effects for valid estimation and can export such effects to illuminate the bounds of potential future intervention and prevention effects. Both are necessary and complementary rather than alternatives (31). Further, we could draw from existing efforts in epidemiology and statistics to develop transportability estimates, devising new ways to incorporate such estimates into systems science models in order to predict the potential impact of new interventions in new contexts (32).

Third, all systems science modeling efforts require careful calibration and validation efforts. This includes standard approaches such as conducting period-by-period calibration-test-recalibration processes, including testing whether the model can forecast study outcomes at a later time point for which there are available data; iteratively excluding study units and testing whether the model can accurately predict the outcome of interventions adopted in the excluded units; and testing how robust the model is to variations in key assumptions about model dynamics (33). Uncertainty analysis approaches, such as Latin hypercube sampling of the space of plausible parameter values, can be used to efficiently test the sensitivity of results to the choice of multiple input parameters at the same time without having to consider all the possible permutations (34). Finally, methods have been applied in infectious disease epidemiology to strengthen the reliability and validity of predictions made from systems science models. These methods include the use of Bayesian Markov chain Monte Carlo and approximate Bayesian computation methods to estimate parameters (8). These methods allow us to conduct statistical inference by searching parameter space to obtain maximum likelihood estimates and 95% credible intervals for parameters of interest (35) and to use deviance information criteria to assist in model selection (36, 37). Because systems science approaches are increasingly used in fields of epidemiology beyond infectious disease, uncertainty analyses and empirical validation methods such as these need to form an integral part of the modeling process.

A VISION FOR THE FUTURE

We predict that 3 key developments will shape the future of systems science modeling in epidemiology. First, the quality and complexity of systems science models will likely advance at an exponential pace because of the acceleration of big data (38) initiatives that provide access to detailed data on individuals, their health histories, and where they live, work, and attend school. One example is the Framework for Reconstructing Epidemiological Dynamics, an open-source agent-based modeling simulation platform (39) that uses census-based synthetic populations that are statistically equivalent to the population in any state or county (40, 41), and that simulates age-appropriate daily activities and interactions of millions of people. Increasingly sophisticated approaches to combine highly detailed location data with health histories (e.g., electronic health records, Medicaid/Medicare data) in user-friendly platforms will make systems modeling approaches more accessible to epidemiologists and will provide the means to base models on higher-quality inputs. Second, machine-learning approaches that incorporate high-dimensional sets of covariates and complex interactions for individual and place-based risk prediction could substantially improve the predictive power of systems science model inputs (42), especially as learning algorithms are increasingly sensitive to real-time data inputs for continuous adaptive learning. They can be used for risk prediction, to account for complex sets of confounders and effect modifiers (42), to predict future hotspots of disease risk (43), and to predict transportability of interventions across sites (32). We envision a scenario where constantly updated sets of big data would inform machine-learning algorithms used for individual and place-based risk prediction, which would in turn serve as inputs into a systems model. Third, the integration of standard validation approaches into systems science modeling will be critical. Although the use of methods such as Latin hypercube sampling for uncertainty analysis and Markov chain Monte Carlo methods for parameter estimation have been used in infectious disease modeling, they are relatively underexplored in the application of systems modeling in other areas of epidemiology.

As data become increasingly “big” and machine-learning systems for predictive modeling become more sophisticated, we can use systems modeling to test assumptions about the types of conditions that give rise to population distributions of disease, as well as the types of interventions that can reduce the population rates of disease, in ways we cannot yet with real data. By combining publicly accessible central data repositories with area- and person-level prediction and causal frameworks, applying careful and systematic validation methods, and integrating systems science modeling into the broader epidemiologic armamentarium through an iterative data collection—modeling—data collection—modeling—intervention—evaluation—modeling process, we can develop valid and reliable systems models that can inform policy priorities.

Abbreviations

ABMs

agent-based models

HIV

human immunodeficiency virus