Skip to main content
NCBI home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2020 May 1.
Published in final edited form as: Comput Environ Urban Syst. 2019 Feb 21;75:170–183. doi: 10.1016/j.compenvurbsys.2019.02.006

Using Multiple Scale Space-Time Patterns in Variance-Based Global Sensitivity Analysis for Spatially Explicit Agent-Based Models

Jeon-Young Kang a, Jared Aldstadt b
PMCID: PMC6855397  NIHMSID: NIHMS1522365  PMID: 31728075

Abstract

Sensitivity analysis (SA) in spatially explicit agent-based models (ABMs) has emerged to address some of the challenges associated with model specification and parameterization. For spatially explicit ABMs, the comparison of spatial or spatio-temporal patterns has been advocated to evaluate models. Nevertheless, less attention has been paid to understanding the extent to which parameter values in ABMs are responsible for mismatch between model outcomes and observations. In this paper, we propose the use of multiple scale space-time patterns in variance-based global sensitivity analysis (GSA). A vector-borne disease transmission model was used as the case study. Input factors used in GSA include one related to the environment (introduction rates), two related to interactions between agents and environment (level of herd immunity, mosquito population density), and one that defines agent state transition (mosquito extrinsic incubation period). The results show parameters related to interactions between agents and the environment have great impact on the ability of a model to reproduce observed patterns, although the magnitudes of such impacts vary by space-time scales. Additionally, the results highlight the time-dependent sensitivity to parameter values in spatially explicit ABMs. The GSA performed in this study helps in identifying the input factors that need to be carefully parameterized in the model to implement ABMs that well reproduce observed patterns at multiple space-time scales.

Keywords: sensitivity analysis, spatially explicit agent-based model, pattern-oriented modeling

1. Introduction

Sensitivity analysis (SA) has been performed to address the challenges related to dynamic associations between input parameters and outcomes in models (Saltelli, Tarantola, & Chan, 1999). Particularly, the usefulness of SA has been well acknowledged in spatially explicit agent-based model (ABM) studies (Ligmann-Zielinska, 2013; Ligmann-Zielinska, Kramer, Cheruvelil, & Soranno, 2014; Perez & Dragicevic, 2009; Thiele, Kurth, & Grimm, 2014). In ABMs, the behaviors and interactions of agents are often specified from incomplete knowledge, which are operationalized through model specification and parameterization (Messina et al., 2008). Consequently, these model building procedures may be responsible for uncertainty in the model outcomes. In this sense, SA helps in identifying the influential factors in model output variability.

Pattern-oriented modeling (POM) (Grimm & Railsback, 2012; Grimm et al., 2005) highlights the usefulness of observed patterns in optimizing the model structure and reducing parameter uncertainty. It is claimed that a single pattern of observations at a particular scale is likely insufficient to choose the model structure and parameters (Grimm et al., 2005; O'Sullivan & Perry, 2013). In addition, there has been increased emphasis placed on the use of spatial patterns to assess the models in agent-based land use studies (Brown & Robinson, 2006; Evans & Kelley, 2004; Parker & Meretsky, 2004). Kang and Aldstadt (2019) also emphasizes the importance of using multiple scale spatiotemporal patterns for validating spatially explicit ABMs. Nevertheless, inadequate attention has been paid to using multiple space-time patterns to comprehend the extent to which parameters in ABMs are responsible for mismatch between model outcomes and observations.

Recent efforts have focused on summarizing the associations between input parameters of interest and the model outcomes arising from varying the input parameters at a particular space-time scale (An, Linderman, Qi, Shortridge, & Liu, 2005; Brown & Robinson, 2006; Happe, Kellermann, & Balmann, 2006; Perez & Dragicevic, 2009; Valbuena, Verburg, Bregt, & Ligtenberg, 2010). Multiple scale space-time patterns of the simulated outcome of interest (i.e., space-time patterns at macro and micro scales) have been rarely employed in SA. Spatially explicit ABMs are used to measure the potential effectiveness of spatially or temporally targeted interventions including models of disease transmission or crime. In this case, it is vital that we understand the relationship between parameters and the spatial patterns of outcomes at multiple scales. Varying input parameters may also have scale-varying impacts on model results (i.e., the total number of cases in the study area as a macro scale pattern and local clustering of risk as a micro scale pattern). Therefore, SA of spatially explicit ABMs at multiple space-time scales is advantageous to investigate the associations between model inputs and outputs in the multiple perspectives (i.e., global or local perspective). Exploring spatially explicit ABMs with SA at multiple space-time scales, however, depends on the research question being addressed.

For SA at multiple space-time scales, we adopt the variance-based global sensitivity analysis (GSA) framework that was conducted by Saltelli and colleagues (Crosetto & Tarantola, 2001; Saltelli et al., 1999), which assesses the ways in which the variance in the model output is attributed to the variances in model input parameters. In addition, using the deviations between simulated and observed patterns at multiple space-time scales as statistics in variance-based GSA, it is possible to decompose the variance in the deviations into the input variations. Therefore, our variance-based GSA approach provides an improved knowledge of the extent that input perturbation contributes to changes in spatio-temporal patterns in model outcomes as well as mismatches between model outcomes and observations.

In this paper, we conducted multiple spatio-temporal scale variance-based GSA using spatially explicit ABMs of a vector-borne disease transmission as the case study. The specific objectives of this study are twofold: 1) to explore uncertainty of space-time patterns in model outcomes, and 2) to measure the sensitivity of the deviations in patterns between outcomes and observations to input parameters. The following section reviews our multiple space-time scale GSA framework. In section 3, the spatially explicit ABM of a vector-borne disease transmission is presented. Section 4 summarizes the results of GSA, and Section 5 concludes with a discussion of advantages of using multiple scale space-time patterns in GSA and suggestions for future research.

2. Multiple space-time scale variance-based global sensitivity analysis

Global sensitivity analysis (GSA) addresses the effect of uncertainty about the input information on model outputs (Crosetto, Tarantola, & Saltelli, 2000). GSA entails systematically varying input elements and measuring how such a variation in the input sources can contributes to a variance in model outputs. It helps in understanding model robustness and sensitivity of outcomes to input parameters in the model (O'Sullivan & Perry, 2013;Paton, Maier, & Dandy, 2013; Rahmandad & Sterman, 2008).

Particularly, variance-based GSA is used to comprehensively study the influence of variability in model inputs on that in outputs, which enables the decomposition of variance in outputs to uncertain input parameters drawn from probability distribution function. As a consequence, the relative importance of a given input factor (Xi) can be measured by the extent to which input factor (Xi) contributes to the variance of model output (V(Y)) (Crosetto & Tarantola, 2001; Ligmann-Zielinska, 2013; Ligmann-Zielinska et al., 2014; Saisana, Saltelli, & Tarantola, 2005; Saltelli et al., 1999).

V(Y)=∑iVi+∑i<jVij+∑i<j<mVijm+…+V12…k

where

Vi=V[E(Y∣Xi)]Vij=V[E(Y∣Xij)]−V[E(Y∣Xi)]−V[E(Y∣Xj)]

Vi denotes the variance of the expectation of Y conditional on Xi.

The first-order sensitivity index (Si) describes the decoupled influence of a single input (Xi) on the model output.

Si=ViV(Y)

The Si indices lie in [0,1], and their sum is less than or equal to one. The sum of Si of all input factors (∑ S) indicates the fraction of variance in model output, which can be explained by the effects of input factor alone. The effects of interactions between input factors are measured by the following formula:

Interactions=(1−ΣS)

A total sensitivity index (STi) explains the overall contributions of a given input factor (Xi) to variance in the model output, including its interactions with any other input factors. Suppose that we have four factors in the model. Each total sensitivity index is computed as follows:

ST1=S1+S12+S13+S14+S123+S124+S134+S1234ST2=S2+S12+S23+S24+S123+S124+S234+S1234ST3=S3+S13+S23+S34+S123+S134+S234+S1234ST4=S4+S14+S24+S34+S124+S134+S234+S1234

where Si is the first-order sensitivity index for factor i, Sij is the second-order sensitivity index for factors the i and j, Sijk is the third-order sensitivity index for factor i, j, and k, and Sijkl denote the highest-order effect considering the interactions of all factors i, j, k, and l. For example, S12 refers to the effects of the interactions between factor 1 and 2.

The first-order and total sensitivity indices are calculated based on Sobol's estimation procedures described in Saltelli (2002). The uses of Sobol's procedures are found in literatures that explore ABMs with variance-based GSA (Hu, Lin, Wang, & Rodriguez, 2017; Ligmann-Zielinska et al., 2014; Ligmann-Zielinska & Sun, 2010; Tang & Jia, 2014). The sum of STi may exceed one, and thus, we will normalize STi to effectively compare the total contribution of a given factor to other factors.

Spatially explicit ABMs aim to simulate and replicate spatiotemporal patterns that characterize the phenomena of interest with the interactions between heterogeneous agents and their environments. Due to input parameters and model specification, ABMs have inherent challenges associated with stochastic uncertainty (Brown, Page, Riolo, Zellner, & Rand, 2005). To address these challenges, SA is used to explore the associations between input parameters and model outputs. The conventional SA is often conducted to comparison of aggregated scalars, such as single basic statistic (e.g., counts or rates) in agent-based disease models (Perez & Dragicevic, 2009; Rahmandad & Sterman, 2008) and multiple fragmentation statistics (e.g., aggregation index and patch density) in agent-based land-use models (Ligmann-Zielinska, 2013; Ligmann-Zielinska & Sun, 2010). However, these approaches to SA may be not possible to identify to what extent the input factors have impacts on discrepancies between simulated and observed patterns at multiple space-time scales. In this paper, we suggest the use of the difference values between model outputs and observations at multiple space-time scales in GSA. Following the POM approach, by comparing patterns in simulated results with reality, our multiple space-time scale GSA approach helps not only examining the applicability of models, but also quantifying to what extent input parameters contribute to variability in mismatch between simulated outcomes and observations. Our GSA requires the multiple available observed datasets, but the number of space-time patterns in which GSA will be performed depends on the data availability.

3. Case study: spatially explicit agent-based model of DENV transmission

3.1. DENV transmission

Dengue is a common mosquito-borne disease usually found in tropical and subtropical countries. It is transmitted primarily by Aedes aegypti (Ae.) mosquitoes. Fifty to one hundred million infections are expected to occur every year in the Asian-Pacific regions (Gubler, 2002; WHO, 2012). A licensed vaccine is being administered on a limited basis in some countries, but current efforts for dengue preventions and control still focus on eradicating the mosquito vectors (Achee et al., 2015; Scott & Morrison, 2010).

Statistically significant case clusters in DENV outbreaks at defined spatial and temporal distance were found (Aldstadt, 2007; Mammen Jr et al., 2008; Yoon et al., 2012) because of the short flight distance of Ae (Harrington et al., 2001; Harrington et al., 2005). Human DENV exposure is likely related to the locations that an individual visits frequently, regardless of the geographical distance from their homes (Reiner, Stoddard, & Scott, 2014; Stoddard et al., 2013; Stoddard et al., 2009). It is clear that human movements are socially constructed. For instance, people are more likely to visit the homes of acquaintances than those of strangers. Considering the mosquito vector’s sedentariness and limited movements (Harrington et al., 2001; 2005; Schafrick, Milbrath, Berrocal, Wilson, & Eisenberg, 2013) and the lifelong immunity that humans can acquire against the four DENV serotypes (Vaughn et al., 2000), human activities and herd immunity play critical roles in the spread of DENV. Herd immunity limits the number of potentially infectious human hosts in a region and provides indirect protections for susceptible community members. Despite the importance of human activities (Reiner, Stoddard, & Scott, 2014; Stoddard et al., 2013) and herd immunity, it is still difficult to accurately estimate their influence on DENV transmission. It is shown that DENV is often introduced by trips between cities, which is responsible for annual oscillations (Schwartz et al., 2008). Epidemiologically significant measures of vector population density are also challenging to obtain. The required periods that mosquitoes become infectious after mosquito bite infectious humans, called extrinsic incubation period, are uncertain. It takes about 10 to 15 days (Aldstadt, 2007). These imperfect knowledges of DENV transmission may be responsible for apparent conflicts in the findings of community-scale studies of DENV transmission. Characterizing the effects of herd immunity, introduction to the study area from outside, mosquito density population, and extrinsic incubation period on the process of DENV transmission would provide important insights for more efficient DENV surveillance and control.

3.2. Data and study area

In our ABM, the study area is based on a portion of Kamphaeng Phet province (KPP), Thailand (Figure 1). There are 3683 houses, 186 workplaces, and eight schools. The locations of buildings were derived from Lidar data. Microdata on households in KPP in 2009 (Thomas et al., 2015) were utilized, and the population of houses were drawn from the sample of households. We assumed individuals commute to the nearest school/workplace from their household. We also included the demographic changes based on the birth and death rates in Thailand, obtained from Department of Provincial Administration (DOPA), Ministry of Interior, Thailand.

Figure 1.

Figure 1.

Open in a new tab

Study Area: a portion of Kamphaeng Phet province, Thailand. The locations of all buildings are projected to Universal Transverse Mercator (UTM) system, zone 47 North.

3.3. Model description

Our ABM of DENV transmission was expanded based on Kang and Aldstadt (2017) and implemented in Anylogic 7.3.5. The ABM consists of individual human agents, infected female mosquito agents, and the environments. For the details, the Overview, Design concepts and Details (ODD) protocol (Grimm et al., 2010) is provided in appendix.

3.3.1. DENV transmission

In the ABM, DENV transmission occurs through interactions between human hosts and infectious female mosquito vectors, as shown in Figure 2. The interactions take place through the bite behaviors of mosquito agents. DENV is transmitted to human hosts by the bite of infectious female mosquitos. The female mosquitoes that bit infectious human hosts become infectious after extrinsic incubation periods. Exposed humans become able to transmit the virus to susceptible female mosquitoes after the incubation period and remain infectious during the viremic period.

Figure 2.

Figure 2.

Open in a new tab

DENV transmission chain

To describe this successive process, we employed a susceptible, exposed, infectious, and recovered (SEIR) epidemic model in which human agents are assumed to undergo four SEIR stages with any other four DENV serotypes: (1) Susceptible (“susceptible to a specific DENV serotype”), (2) Exposed (“exposed to the DENV serotype), (3) infectious (“able to transmit the DENV serotype”), and (4) Recovered (“immune to the DENV serotype”). A human agent who recovered from one serotype is still susceptible to exposure from other serotypes of DENV. We also assumed that infected human agents stay at home until they recover.

3.3.2. Human agents

In our ABM, individual human agents behave differently during weekdays and weekends (Figure 3). During the weekdays, people may contact with others at home, workplace/school, or home of acquaintances during four time periods in a day: the morning (midnight–9 am), the daytime (9 am–5 pm), evening (5 pm–9 pm), and the nighttime (9 pm–midnight). People can be collocated with their household members in their home during the morning evening, and nighttime; with classmates/coworkers at their schools/workplaces during the daytime; and with friends/relatives at the acquaintances’ houses during the evening. On the weekends, colocations with household members happen at home during the morning daytime and evening, and nighttime, while those with friends/relatives occur in the acquaintances’ houses during the daytime and evening.

Figure 3.

Figure 3.

Open in a new tab

Human activities: (a) in spatial frame and (b) in spatio-temporal frame

In our model, human behaviors are age-dependent. On weekdays, an individual spends the daytime period in school (for people aged 5-19) or in the workplace (for people aged 20-64). Both age groups are assumed to spend both morning and nighttime at home. Individuals aged 5-64 sometimes visit the homes of acquaintances during the evening after work/school. On the weekend, individuals may spend leisure time (during both the daytime and evening) in the houses of acquaintances. Individuals outside of those age ranges (that is, those below the age 5 and over 65) are assumed to stay in their houses every day. People visited to houses of friends or relatives eight times in 15 days (53.3 %) (Stoddard et al., 2013), and thus the social activities in acquaintances' houses are assumed to occur with a 0.5 probability in the model. The people who do not visit to acquaintances' houses stay in their houses. Table A2 in appendix provides parameters for human agents in the baseline ABM.

In the epidemic context, given that DENV transmission is likely to occur in the places where people often visit and interact with others (Stoddard et al., 2013; Stoddard et al., 2009), it is important to include people’s social connections in the model. Since the household members were randomly chosen for every simulation, the number of connections may vary, but the general distributions of the number of connections were similar over simulations. In the model, these social connections enable people to visit the houses of others who are socially connected, and people who have many social connections have more opportunity to visit the houses of others. Specifically, each individual has five social connections and can visit only the households of these five individuals in addition to the individual’s own household and school/workplaces (Figure 4). The number of houses that an individuals can visit for his/her social activities was set based on the previous study that identifies the effects of social connections on DENV transmission (Reiner et al., 2014).

Figure 4.

Figure 4.

Open in a new tab

The association of social connections and human movements

3.2.3. Mosquito agents

In our model, only infected female mosquitoes are defined as agents. Mosquito agents can typically reside in and move around nearby buildings (i.e., less than 30 meters) with 15%, but can sometimes move to random locations with a 1% for an occasional long-distance travel. Hazard rates are also defined for the age-dependent survival of mosquitoes (Harrington et al., 2001; Harrington et al., 2008; Harrington et al., 2005). Mosquito bite rates vary according to time of day (Chao, Halstead, Halloran, & Longini Jr, 2012). We assumed the homogeneous mosquito density population that the mosquito abundance in each building is the same, but the abundance of mosquitoes in each building is represented on a month-to-month basis. The capacity for mosquito abundance is maximum in June, and the capacity reaches its peak between May and August due to the hot and humid summer season in Thailand as shown in Figure 5. This capacity changes from 10 to 50 in SA (the details will be addressed in Section 4). Table A3 provides parameters for mosquito agents in the baseline ABM.

Figure 5.

Figure 5.

Open in a new tab

Seasonality of varying mosquito abundance by mosquito population density parameter per building. The maximum number of mosquitoes at each building is assumed to be homogeneous in our model.

3.4. Simulation Design

Figure 6 summarizes the overview of GSA in this paper. To answer the research questions, we used only four factors (level of herd immunity, introduction rates, mosquito population density and mosquito extrinsic incubation period) in GSA. These four factors are model-level factors that are assigned at the beginning of simulation and kept at each simulation run. The other parameters that are not used in GSA remain constant (viremic period, recovery period, PMP, PPM, extrinsic incubation period, hazard rate, biting rates, probability of mosquito migration, and mosquito migration radius). Specifically, four parameters used in GSA represent the environment (introduction rates), interactions between agents and environment (level of herd immunity, mosquito population density), and agent state transition (mosquito extrinsic incubation period). Space-time statistics of model outcomes from 10,000 simulation runs were measured at the macro and micro spatio-temporal scales. The total number of simulation runs (Nsim) is derived from the following equation (Tang & Jia, 2014):

Nsim=(2∗k+2)∗nmcsim

where k denotes the number of factors and nmcsim is the number of Monte Carlo simulation runs. The necessary number of Monte Carlo simulation runs is typically required within the range of [100, 500] or higher (Tang & Jia, 2014). In this paper, we used four input factors in GSA and ran 1,000 times of Monte Carlo simulation runs for enough ABM executions. To perform the GSA, the differences in the space-time patterns from outcomes and those of observed data at macro and micro scale are used. The results from GSA provide quantitative descriptions about the extent to which the input factors contribute to variance in differences to observed patterns.

Figure 6.

Figure 6.

Open in a new tab

Overview of pattern-oriented GSA. D denotes deviation between DENV infection rates of simulated outcomes and those of observations. WD denotes weighted-deviation between cluster-based DENV infection rates of simulated outcomes and those of observations.

To address the impacts of the ecological contexts on the space-time patterns of DENV outbreaks, we performed a variance-based GSA over a period of ten years. The simulations were executed over and over by varying input factors, including the level of herd immunity, mosquito population density, mosquito extrinsic incubation period, and introduction rates (Table 1). The value of baseline is based on Kang and Aldstadt (2017). Following Saltelli et al. (1999), the input factors were drawn from uniform distributions.

Table 1.

Input factor variations

Input
factors		 Description Baseline Probability Density Function
Herd immunity Annual rate of exposure DENV prior to start simulation 0.1 U = [low (0.01), medium (0.05), high (0.1) with equal probability of selection]
Mosquito population density Number of mosquitoes per building 42 U = [ 10 to 50 with increments of 5, with equal probability of selection]
Mosquito extrinsic incubation period The date that mosquito becomes infectious 11 U = [9, 13]
Introduction rates The probability of infections from outside 1.0 *10−6 U = [ 1.0*10−6 to 1.0*10−5 with increments of 1.0*10−6, with equal probability of selection]
Open in a new tab

U - uniform distribution

The level of herd immunity indicates the proportion of immunized people within the community population. Considering the lifelong immunity of individuals, the level of herd immunity reflects the DENV epidemic history. For example, a high level of herd immunity (0.1) assumes that 10% of subjects have been infected by specific serotypes of DENV in the past and are now immune to those serotypes. By contrast, people in regions with low herd immunity are more vulnerable to the exposure of DENV than people in regions with medium and high herd immunity.

Mosquito population density denotes the maximum capacity for the mosquitoes within each building (that is, the schools, workplaces, and houses). Mosquito extrinsic incubation period indicates the required time period during which mosquito vectors become able to transmit DENV to human hosts. Longer periods delay the process of successive DENV transmission between mosquito vectors and human hosts. Introduction rate refers to the proportion of infections introduced from outside. In our model, dengue transmissions begin with the introduction of DENV from outside to the study area.

The scalar outputs in SA are measured by the differences between the patterns of simulated results and those of observations at each space-time scale. At the macro spatiotemporal scale, the space-time patterns of DENV outbreaks are explained by the annual attack rates. According to Endy et al. (2002), about 5.8 percent of the school population (i.e., children aged 4-16 years) in KPP were annually infected. At the micro spatiotemporal scale, the space-time patterns were studied by using geographic cluster investigations (Yoon et al., 2012). Based on a school-based surveillance system, the patterns of DENV infection were measured by the cases among children living nearby a child with an identified infection. The children were included in the geographic cluster investigations were other children who is living within a 100-meter radius of the household in which DENV infection firstly occurred. There were 50 clusters, and each investigation included other DENV infections that happened approximately three weeks prior to and up to 15 days after the first case. Therefore, this method described the spatiotemporal clustering patterns of DENV outbreaks over study area. Figure 7 shows the average rate of DENV infection cases among the children in 50 clustered in each distance interval. Yoon et al. (2012) found the distance decay of the infection rates; 35.3 % in index houses, 29.9 % in houses within 20 meters, 22.2 % in houses within 20-40 meters, 13.2 % in houses within 40-60 meters, 14.4 % in houses within 60-80 meters, and 6.2 % in houses within 80-100 meters.

Figure 7.

Figure 7.

Open in a new tab

DENV infection rates in each distance range (Yoon et al., 2012)

GSA was performed based on overall and cluster-based infection rates. These measurements account for the impacts of input factors on closeness to observed patterns. Specifically, we used deviation (D) for overall infection rates and weighted deviation (WD) for cluster-based infection rates.

Di=(Ri−O)2

where Ri is each of the DENV infection rates over 10,000 Monte Carlo simulations, and O denotes the DENV infection rates (5.8%) observed in Endy et al. (2002).

WDi=Wi(Ri−O)2

where i denotes the index of spatio-temporal patterns (i.e., DENV infection rates in each cluster), and Wi refers the weight (0.5, 0.1, 0.1, 0.1, 0.1, and 0.1). As the significant clustering of DENV cases was found in the case house (Morrison, Getis, Santiago, Rigau-Perez, & Reiter, 1998), the weight of the same household is greater (0.5) than it of other distance ranges (0.1). Ri is the DENV infection rates at the index i, and Oi denotes the DENV infection rates in observed overall DENV infection rates (Yoon et al., 2012).

4. Result

We summarized the space-time patterns of DENV outbreaks with D at macro scale and WD at micro scale. Small values of D and WD reflect the better model fits at each scale. Figure 8 shows examples of the relationships between input factors (incubation period and introduction rate, and incubation period and number of mosquito) in D and, WD at the last 10th simulation year, respectively. There are no apparent relationships between input factors and D, and WD. Specifically, the values of D and WD were not directly changed by varying only two input factors, as shown in each panel of Figure 8. Also, there are no cases that varying a particular factor leads to great changes in D and WD. These results do not indicate a single impact of each input parameter on D and WD. To avoid occupying much space in the paper, the remaining plots of relationships among other input factors are provided in appendix (Figure A1 and A2).

Figure 8.

Figure 8.

Open in a new tab

Relationships between input factors and D, and WD at the 10th simulation year: (a) incubation period and introduction rate in D, (b) incubation period and introduction rate in WD, (c) incubation period and number of mosquitoes in D, (d) incubation period and the number of mosquitoes in WD

Figure 9 illustrates the results of uncertainty analysis of D and WD each year. The black dots represent the average of D and WD at each year, and vertical lines represent the range of D and WD at each year. There was time-dependent uncertainty; the averages and ranges of D and WD changed over year. In detail, the ranges of D became smaller over year, whereas there were no big changes in averages of D. Although the averages of WD decreased over year, no changes in the ranges of WD were found. Therefore, SA needs to be performed to clearly figure out to which input factors may have impacts on variability on the closeness to observations over time.

Figure 9.

Figure 9.

Open in a new tab

Time-varying uncertainty plots

By identifying the influential factors to variance in D and WD, we could gain a better understanding of which factors influence the deviations of multiple scale space-time patterns of DENV transmission to observations. The GSA results in S and ST, which can be interpreted based on Table 2. S indicates an impact of a single factor on the variance in deviations of space-time patterns. ST is interpreted as an influence of a particular factor and its interactions with other input factors on the variance of deviations in space-time patterns. The interpretations was adopted from Ligmann-Zielinska and Sun (2010), with modification to make it applicable to pattern-oriented SA in this study.

Table 2.

S and ST Interpretation

Measurement Interpretation
Relatively high S in D The factor has relatively greater impact on variability of deviance between model outcomes and observations at the macro scale.
Relatively high ST in D The input factor, involved in the interactions with other factors has greater impact on variability of deviance between model outcomes and observations at the macro scale.
Relatively high S in WD The factor has relatively greater impact on variability of deviance between model outcomes and observations at the micro scale.
Relatively high ST in WD The input factor, involved in the interactions with other factors has greater impact on variability of deviance between model outcomes and observations at the micro scale.
Open in a new tab

With S of D and WD, we found the important input factors that would significantly increase the variance in outcomes at the macro (D) and micro (WD) scales. Figure 10 illustrates time series of first-order and total sensitivity indices of D and WD. In Table 3 and 4, the bolded values represent the input factors that have the most significant impacts on the deviations at each year. All the input factors, taken singly, can explain only approximately 18-23% of D at macro scale and 18-25% of WD at micro scale. About 77-82% of D at macro scale and 75-82% of WD are attributed to factor interactions, which occur between the level of herd immunity, introduction rate, mosquito density population, and extrinsic incubation period. In other words, any single factor does not have significant impact on the variance of deviations between model outcomes and observations, but the interactions between all input factors are the most influential to the variance in the deviations at the both scales.

Figure 10.

Figure 10.

Open in a new tab

First order indices S of D and WD, and total effect indicies ST of D and WD

Table 3.

First order indices S and total effect indices ST of D (macro scale)

Indices Input
Factors		 Year
1st 2nd 3rd 4th 5th 6th 7th 8th 9th 10th
S Herd immunity 0.072 0.074 0.073 0.073 0.072 0.069 0.061 0.053 0.043 0.036
# of mosquito 0.077 0.088 0.091 0.102 0.109 0.119 0.118 0.124 0.127 0.126
Incubation period 0.012 0.008 0.008 0.007 0.007 0.006 0.005 0.006 0.006 0.005
Introduction rates 0.026 0.015 0.017 0.019 0.025 0.031 0.041 0.044 0.047 0.060
Interaction 0.813 0.815 0.810 0.798 0.787 0.776 0.775 0.773 0.777 0.773
Normalized ST Herd immunity 0.330 0.339 0.328 0.312 0.299 0.277 0.241 0.228 0.211 0.183
# of mosquito 0.304 0.311 0.305 0.313 0.309 0.316 0.316 0.317 0.318 0.318
Incubation period 0.098 0.092 0.106 0.113 0.126 0.139 0.162 0.177 0.186 0.194
Introduction rates 0.268 0.259 0.261 0.262 0.266 0.267 0.280 0.277 0.285 0.305
Open in a new tab

Table 4.

First order indices S and total effect indices ST of WD (micro scale)

Indices Input
Factors		 Year
1st 2nd 3rd 4th 5th 6th 7th 8th 9th 10th
S Herd immunity 0.041 0.020 0.024 0.025 0.021 0.019 0.015 0.013 0.007 0.007
# of mosquito 0.151 0.172 0.183 0.154 0.182 0.203 0.178 0.184 0.163 0.104
Incubation period 0.022 0.020 0.022 0.020 0.020 0.016 0.010 0.007 0.002 0.000
Introduction rates 0.004 0.006 0.008 0.009 0.009 0.012 0.017 0.024 0.047 0.068
Interaction 0.782 0.782 0.762 0.791 0.769 0.7550 0.780 0.773 0.780 0.821
Normalized ST Herd immunity 0.215 0.163 0.177 0.168 0.168 0.148 0.147 0.123 0.097 0.072
# of mosquito 0.334 0.373 0.366 0.343 0.370 0.393 0.376 0.393 0.370 0.326
Incubation period 0.221 0.234 0.231 0.254 02555 0.261 0.270 0.287 0.182 0.296
Introduction rates 0.230 0.229 0.225 0.235 0.207 0.197 0.206 0.197 0.251 0.305
Open in a new tab

As the variability in model outputs cannot be accounted for directly by individual factors alone, the ST indices are necessary to explain the variance in the deviations between model outputs and observations by varying input factor values. The normalized ST describes an impact of each input factor and its interactions with other factors. With the ST, we identified the time-varying effects of the input factors on the deviations at the macro and micro scales. At macro scale, the normalized ST of mosquito density population demonstrates consistently significant influences for 10 years. In other words, mosquito population density is the most significant input factor that causes mismatch between model outcomes and observations at the macro space-time scale. The level of herd immunity was a moderately influential, especially at the beginning of the simulations (until 4th simulation year), but its impact was less significant through time. The influences of other two factors, including incubation periods and introduction rates, gradually increased after the 4th simulation year.

At the micro scale, mosquito density was the most influential input factor to space-time patterns followed by the introduction rates, mosquito extrinsic incubation period, and the level of herd immunity. We observed the impacts of factors, excluding the mosquito population, were unstable over time. Specifically, the magnitude of an impact of introduction rates was decreased until the 5th simulation year, and that of herd immunity kept decreasing.

5. Conclusion

Despite the recognized importance of characterizing space-time patterns of simulated outputs of spatially explicit ABMs, there has been an inadequate effort to use multiple scale space-time statistics in SA. Parameterization specified from incomplete knowledge also may lead to mismatches between space-time patterns of model outcomes and those of observations. To address this issue, we performed variance-based GSA to assess the individual contributions of each input factor to space-time patterns in deviations between model outputs and observations at multiple space-time scales. We used the spatially explicit ABMs of DENV transmission as the case study. Particularly, we tested the impact of variables such as the herd immunity, mosquito extrinsic incubation period, mosquito abundance, and introduction rates on space-time patterns of the local transmission of DENV. The spatially explicit ABMs of DENV transmission of Kang & Aldstadt (2017) were expanded for this research.

The results from SA indicate the time-varying dynamic interactions between input factors and model outcomes. Particularly, DENV outbreaks are sensitive to local contexts, as captured by the interaction of the model input factors. We also found that the factors regarding interactions between agents and the environment have a relatively greater impact on the patterns of DENV outbreaks. To be more specific, the results from our GSA provide the importance of parameterizing the number of mosquito density populations in DENV models. The improved information about mosquito abundance may enhance the validity of DENV simulation models. Not only that, the level of herd immunity is also shown as one of influential factors responsible for mismatch between observed and simulated patterns at the macro level. This means individual's immunity status also needs to be carefully considered when developing DENV transmission simulation models. Herd immunity may vary widely in the diverse regions where DENV may circulate. A comparison between ST of D and that of WD demonstrates that the impact of a specific factor and its interactions with other factors on variance in the patterns varies depending on the space-time scale. For example, the mosquito extrinsic incubation period has a relatively large impact at the macro scale, but there is no significant role in DENV transmission at the micro scale. These results also empirically highlight the importance of input factors’ time-dependent sensitivity. This has been previously observed in land-use studies (Ligmann-Zielinska, 2013; Ligmann-Zielinska et al., 2014; Ligmann-Zielinska & Sun, 2010), and we show here that it applies to individual-level models of disease transmission.

As spatially explicit ABMs aim to understand and simulate dynamic spatio-temporal phenomena, it is of primary importance to characterize space-time patterns of simulated results. To ensure that models are robust, multiple scale space-time patterns need to be compared (Kang & Aldstadt, 2019). Also, input factors may have varying impacts on model outcomes. Therefore, measuring sensitivities of the deviances between the multiple scale space-time patterns of model outcomes and those of observations would provide an improved understanding of the dynamic associations within an ABM. This type of GSA specifically shows which input factors contribute to the mismatches between model outcomes and observations, identifying influential factors that need to be carefully parameterized in the model.

Using various indicators for space-time interactions (e.g., Knox test (Knox, 1964) and incremental Knox test (Aldstadt, 2007)) and spatial autocorrelations (e.g., Moran's I (Moran, 1950) and Getis-Ord Gi* statistics (Getis & Ord, 1992)) would be an apparent next step in our GSA approach as a means of investigations of dynamic associations of model inputs and their effects on mismatches to observed patterns. Given that variance-based GSA can be used for implementing parsimonious ABMs (Ligmann-Zielinska, 2018; Ligmann-Zielinska et al., 2014), future research will focus on simplification of the model. Through parsimonious models, we can explore the dynamic nature of phenomena with less computational burden and more straightforward interpretations. We restricted the number of social connections of individual humans based on the previous study (Reiner et al., 2014). Employing the number of social connections as an input factor in GSA would help in measuring the effects of social networks on the community-level DENV transmission. It would also be valuable to investigate such social connection effects with other types of social networks (i.e., random (Erdos & Rényi, 1960), scale-free (Barabási & Albert, 1999), and small world (Watts & Strogatz, 1998)).

In addition, it would be interesting to consider the burn-in period in our DENV simulation models. The individual's immunity status tends to be similar to his/her neighbors due to seasonality in serotype-specific dominance in Thailand (Nisalak et al., 2003) and the focality of DENV transmission (Mammen Jr et al., 2008). In our model, the initial state of immunity for individuals was not spatially structured, since the individual's immunity status was assigned based on his/her age. Therefore, the burn-in period may create a more realistic local pattern of community-level immunity and provide an improved understanding of the effects of ecological factors (e.g., mosquito movements, mosquito extrinsic incubation period, mosquito counts) on DENV transmission at the both macro and micro space-time scales.

Highlights.

  • Using multiple scale space-time patterns is advantageous for SA.

  • SA helps in avoiding mismatch between simulated patterns and observed patterns.

  • Input factors embedded in the ABMs have varying effects at each space-time scale.

  • The influential factors need to be carefully parameterized in the ABMs.

Acknowledgements

We would like to express our appreciation for the funding provided by the National Institutes of Health (R01 GM083224 and P01 AI034533).

Appendix

Our model is available in the following link (https://github.com/kang716/SensitivityAnalysisOfABMsDENV). Because of the limited computational power of Anylogic cloud platform, only a small portion of the model has been uploaded. To test the model, please download the source code from the above link and run the simulations using AnyLogic software. The free version of AnyLogic software is available at the link (https://www.anylogic.com/downloads/).

This supplement provides ODD protocol of ABMs. Not applicable elements in the ODD protocol are omitted.

Table A1.

Overview, Design concepts and Details of ABMs

Overview
Purpose To simulate a local-level DENV transmission under model specifications (homogeneous mosquito population, realistic spatial configuration of buildings)
Entities, state variables, and scales ABM consist of three entities: (1) human, (2) infectious female mosquito, and (3) building agents, and each entity has several state variables.
  • (1) Human agent
    • Age
    • Gender
    • Occupation status
    • House location: x-y coordinates
    • School/workplace location: x-y coordinates
    • Current location: x-y coordinates
    • SEIR states for all DENV serotypes
    • Cross immunity state
  • (2) Mosquito agent
    • Age
    • Serotype
  • (3) Building agent
    • Type
    • Location: x-y coordinates
Process overview and scheduling
  • (1) Movement
    • Human:
      • Weekdays: commuting process: school (aged 5-19) and workplace (aged 20-64) during daytime (9 am – 5 pm), social activities (aged 5 – 64) at evening (5 pm – 9 pm)
      • Weekends: spatial activities (aged 5 – 64) during daytime and evening.
    • Mosquito: moving around within 30 meters (15 % of probability) and random locations (1% of probability)
  • (2) The birth, death/out-migration and aging
    • January 1st every year, the certain amounts of individual humans are newly born and died/out-migrated. The newly born humans are randomly assigned to houses.
    • January 1st every year, every individual gets older. The property (age) increases by one.
  • (3) Biting
    • Mosquitoes bite humans with a certain probability
  • (4) Seasonal fluctuation of mosquito population
    • The counts of mosquito population vary to month.
Design concepts
Basic principles The ABMs purpose to explore the impacts of parameters in regard to (1) the level of herd immunity, (2) mosquito population, (3) mosquito extrinsic incubation period, and (4) introduction rates. The model was expanded based on Chapter 4.
Sensing Each mosquito senses the neighboring houses to move around and human to bite in all buildings.
Interaction There is an interaction between humans and mosquitoes by biting process of mosquitoes.
Details
Initialization The model synthesizes human population within 3683 households.
Individual humans’ immune statuses to each serotype are assigned based on their ages with a certain probability (0.14).
For scenarios of heterogeneous mosquito population, the populations are determined by a negative binomial distribution (0.0344, 1.5) where 0.0344 and 1.5 denote number of failure and the probability of success.
For scenarios of synthetic environments, all buildings are randomly arranged.
Input data (1) locations of houses and schools identified from GPS data (Thomas et al., 2015)
(2) household census data (Thomas et al., 2015)
(3) birth and death/out-migration rates obtained from Department of Provincial Administration (DOPA), Ministry of Interior, Thailand.
Parameters The main parameters of human and mosquito agents are provided in Table A2, and A3.
Open in a new tab

Table A2 and A3 provides parameters for human and mosquito agents in the baseline ABM.

Table A2.

Set of parameters for human agents

Parameter Value Note
Incubation period 6 days Time period between exposure and infectiousness
Viremic period 4 days Time period between infectiousness and recover
Recovered period 120 days Required time period human becomes susceptible
PMP 0.25 Probability of transmission from mosquito to person per mosquito’s bite
PPM 0.1 Probability of transmission from person to mosquito per mosquito’s bite
Introduction rates 0.0000001 The proportion of introduced infections from outside of study area
Herd immunity 0.1 The proportion of people who are immune to all population
Open in a new tab

Table A3.

Set of parameters for mosquito agents

Parameter Value Note
Extrinsic incubation period 11 days Required time period that mosquito becomes infectious after biting infectious humans
Hazard Rate 0.09, 0.08 Death rate for mosquito (0.09: younger than 10days, 0.08: older than 10 days)
Biting Rate 0.08, 0.76, 0.13, 0.03 Vary according to the time period (0.08: 08-13, 0.76: 13-18, 0.13: 18-24, 0.03: 00-08)
Probability of Mosquito Migration 0.15, 0.01 Daily movement probability (0.15: neighbors, 0.01: random locations)
Mosquito Migration Radius 30 The movement radius of mosquito agents traveling to neighbors
Open in a new tab

Figure A1.

Figure A1.

Open in a new tab

Relationships between input factors and D at the 10th simulation year: (a) the level of herd immunity and introduction rates, (b) the level of herd immunity and the number of mosquitoes, (c) the level of herd immunity and extrinsic incubation period, and (d) introduction rates and the number of mosquitoes.

Figure A2.

Figure A2.

Open in a new tab

Relationships between input factors and WD at the 10th simulation year: (a) the level of herd immunity and introduction rates, (b) the level of herd immunity and the number of mosquitoes, (c) the level of herd immunity and extrinsic incubation period, and (d) introduction rates and the number of mosquitoes.

Footnotes

Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final citable form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.

References

  1. Achee N, Gould F, Perkins TA, Reiner R Jr, Morrison A, Ritchie S, … Scott T (2015). A critical assessment of vector control for dengue prevention. PLoS neglected tropical diseases, 9(5), e0003655. [DOI] [PMC free article] [PubMed] [Google Scholar]
  2. Aldstadt J (2007). An incremental Knox test for the determination of the serial interval between successive cases of an infectious disease. Stochastic Environmental Research and Risk Assessment, 21(5), 487–500. [Google Scholar]
  3. An L, Linderman M, Qi J, Shortridge A, & Liu J (2005). Exploring Complexity in a Human–Environment System: An Agent-Based Spatial Model for Multidisciplinary and Multiscale Integration. Annals of the association of American geographers, 95(1), 54–79. [Google Scholar]
  4. Barabási A-L, & Albert R (1999). Emergence of scaling in random networks. science, 286(5439), 509–512. [DOI] [PubMed] [Google Scholar]
  5. Brown D, Page S, Riolo R, Zellner M, & Rand W (2005). Path dependence and the validation of agent-based spatial models of land use. International Journal of Geographical Information Science, 19(2), 153–174. [Google Scholar]
  6. Brown D, & Robinson D (2006). Effects of heterogeneity in residential preferences on an agent-based model of urban sprawl. Ecology and society, 11(1). [Google Scholar]
  7. Chao D, Halstead S, Halloran ME, & Longini I Jr (2012). Controlling dengue with vaccines in Thailand. PLoS neglected tropical diseases, 6(10), e1876. [DOI] [PMC free article] [PubMed] [Google Scholar]
  8. Crosetto M, & Tarantola S (2001). Uncertainty and sensitivity analysis: tools for GIS-based model implementation. International Journal of Geographical Information Science, 15(5), 415–437. [Google Scholar]
  9. Crosetto M, Tarantola S, & Saltelli A (2000). Sensitivity and uncertainty analysis in spatial modelling based on GIS. Agriculture, Ecosystems & Environment, 81(1), 71–79. [Google Scholar]
  10. Endy TP, Chunsuttiwat S, Nisalak A, Libraty DH, Green S, Rothman AL, … Ennis FA (2002). Epidemiology of inapparent and symptomatic acute dengue virus infection: a prospective study of primary school children in Kamphaeng Phet, Thailand. American journal of epidemiology, 156(1), 40–51. [DOI] [PubMed] [Google Scholar]
  11. Erdos P, & Rényi A (1960). On the evolution of random graphs. Publ. Math. Inst. Hung. Acad. Sci, 5(1), 17–60. [Google Scholar]
  12. Evans TP, & Kelley H (2004). Multi-scale analysis of a household level agent-based model of landcover change. Journal of environmental management, 72(1), 57–72. [DOI] [PubMed] [Google Scholar]
  13. Getis A, & Ord JK (1992). The analysis of spatial association by use of distance statistics. Geographical Analysis, 24(3), 189–206. [Google Scholar]
  14. Grimm V, Berger U, DeAngelis DL, Polhill JG, Giske J, & Railsback SF (2010). The ODD protocol: a review and first update. Ecological Modelling, 221(23), 2760–2768. [Google Scholar]
  15. Grimm V, & Railsback SF (2012). Pattern-oriented modelling: a ‘multi- scope’for predictive systems ecology. Philosophical Transations of The Royal Society B, 367(1586), 298–310. [DOI] [PMC free article] [PubMed] [Google Scholar]
  16. Grimm V, Revilla E, Berger U, Jeltsch F, Mooij WM, Railsback SF, … DeAngelis DL (2005). Pattern-oriented modeling of agent-based complex systems: lessons from ecology. science, 310(5750), 987–991. [DOI] [PubMed] [Google Scholar]
  17. Gubler DJ (2002). Epidemic dengue/dengue hemorrhagic fever as a public health, social and economic problem in the 21st century. Trends in microbiology, 10(2), 100–103. [DOI] [PubMed] [Google Scholar]
  18. Happe K, Kellermann K, & Balmann A (2006). Agent-based analysis of agricultural policies: an illustration of the agricultural policy simulator AgriPoliS, its adaptation and behavior. Ecology and society, 11(1). [Google Scholar]
  19. Harrington LC, Buonaccorsi JP, Edman JD, Costero A, Kittayapong P, Clark GG, & Scott TW (2001). Analysis of survival of young and old Aedes aegypti (Diptera: Culicidae) from Puerto Rico and Thailand. Journal of medical entomology, 38(4), 537–547. [DOI] [PubMed] [Google Scholar]
  20. Harrington LC, Jones JJ, Kitthawee S, Sithiprasasna R, Edman JD, & Scott TW (2008). Age-dependent survival of the dengue vector Aedes aegypti (Diptera: Culicidae) demonstrated by simultaneous release–recapture of different age cohorts. Journal of medical entomology, 45(2), 307–313. [DOI] [PubMed] [Google Scholar]
  21. Harrington LC, Scott TW, Lerdthusnee K, Coleman RC, Costero A, Clark GG, … Sithiprasasna R (2005). Dispersal of the dengue vector Aedes aegypti within and between rural communities. The American journal of tropical medicine and hygiene, 72(2), 209–220. [PubMed] [Google Scholar]
  22. Hu H, Lin T, Wang S, & Rodriguez LF (2017). A cyberGIS approach to uncertainty and sensitivity analysis in biomass supply chain optimization. Applied energy, 203, 26–40. [Google Scholar]
  23. Kang J-Y, & Aldstadt J (2017). The Influence of Spatial Configuration of Residential Area and Vector Populations on Dengue Incidence Patterns in an Individual-Level Transmission Model. International Journal of Environmental Research and Public Health, 14(7), 792. [DOI] [PMC free article] [PubMed] [Google Scholar]
  24. Kang J-Y, & Aldstadt J (2019). Using multiple scale spatio-temporal patterns for validating spatially explicit agent-based models. International Journal of Geographical Information Science, 33(1), 193–213. [DOI] [PMC free article] [PubMed] [Google Scholar]
  25. Knox E (1964). The detection of space-time interactions. Applied Statistics, 13, 25–29. [Google Scholar]
  26. Ligmann-Zielinska A (2013). Spatially-explicit sensitivity analysis of an agent-based model of land use change. International Journal of Geographical Information Science, 27(9), 1764–1781. [Google Scholar]
  27. Ligmann-Zielinska A (2018). ‘Can You Fix It?’Using Variance-Based Sensitivity Analysis to Reduce the Input Space of an Agent-Based Model of Land Use Change In GeoComputational Analysis and Modeling of Regional Systems (pp. 77–99): Springer. [Google Scholar]
  28. Ligmann-Zielinska A, Kramer DB, Cheruvelil KS, & Soranno PA (2014). Using uncertainty and sensitivity analyses in socioecological agent-based models to improve their analytical performance and policy relevance. PloS one, 9(10), e109779. [DOI] [PMC free article] [PubMed] [Google Scholar]
  29. Ligmann-Zielinska A, & Sun L (2010). Applying time-dependent variance-based global sensitivity analysis to represent the dynamics of an agent-based model of land use change. International Journal of Geographical Information Science, 24(12), 1829–1850. [Google Scholar]
  30. Mammen MP Jr, Pimgate C, Koenraadt CJ, Rothman AL, Aldstadt J, Nisalak A, … Ypil-Butac CA (2008). Spatial and temporal clustering of dengue virus transmission in Thai villages. PLoS Medicine, 5(11), e205. [DOI] [PMC free article] [PubMed] [Google Scholar]
  31. Messina J, Evans T, Manson S, Shortridge A, Deadman P, & Verburg P (2008). Complex systems models and the management of error and uncertainty. Journal of land use science, 3(1), 11–25. [Google Scholar]
  32. Moran PA (1950). Notes on continuous stochastic phenomena. Biometrika, 37(1/2), 17–23. [PubMed] [Google Scholar]
  33. Morrison AC, Getis A, Santiago M, Rigau-Perez JG, & Reiter P (1998). Exploratory space-time analysis of reported dengue cases during an outbreak in Florida, Puerto Rico, 1991-1992. The American journal of tropical medicine and hygiene, 58(3), 287–298. [DOI] [PubMed] [Google Scholar]
  34. Nisalak A, Endy TP, Nimmannitya S, Kalayanarooj S, Scott RM, Burke DS, … Vaughn DW (2003). Serotype-specific dengue virus circulation and dengue disease in Bangkok, Thailand from 1973 to 1999. The American journal of tropical medicine and hygiene, 68(2), 191–202. [PubMed] [Google Scholar]
  35. O'Sullivan D, & Perry GL (2013). Spatial simulation: exploring pattern and process: Chichester: John Wiley & Sons. [Google Scholar]
  36. Parker DC, & Meretsky V (2004). Measuring pattern outcomes in an agent-based model of edge-effect externalities using spatial metrics. Agriculture, Ecosystems & Environment, 101(2), 233–250. [Google Scholar]
  37. Paton F, Maier H, & Dandy G (2013). Relative magnitudes of sources of uncertainty in assessing climate change impacts on water supply security for the southern Adelaide water supply system. Water Resources Research, 49(3), 1643–1667. [Google Scholar]
  38. Perez L, & Dragicevic S (2009). An agent-based approach for modeling dynamics of contagious disease spread. International Journal of Health Geographics, 8(1), 50. [DOI] [PMC free article] [PubMed] [Google Scholar]
  39. Rahmandad H, & Sterman J (2008). Heterogeneity and network structure in the dynamics of diffusion: Comparing agent-based and differential equation models. Management Science, 54(5), 998–1014. [Google Scholar]
  40. Reiner RC, Stoddard ST, & Scott TW (2014). Socially structured human movement shapes dengue transmission despite the diffusive effect of mosquito dispersal. Epidemics, 6, 30–36. [DOI] [PMC free article] [PubMed] [Google Scholar]
  41. Saisana M, Saltelli A, & Tarantola S (2005). Uncertainty and sensitivity analysis techniques as tools for the quality assessment of composite indicators. Journal of the Royal Statistical Society: Series A (Statistics in Society), 168(2), 307–323. [Google Scholar]
  42. Saltelli A (2002). Making best use of model evaluations to compute sensitivity indices. Computer physics communications, 145(2), 280–297. [Google Scholar]
  43. Saltelli A, Tarantola S, & Chan K-S (1999). A quantitative model-independent method for global sensitivity analysis of model output. Technometrics, 41(1), 39–56. [Google Scholar]
  44. Schwartz E, Weld LH, Wilder-Smith A, von Sonnenburg F, Keystone JS, Kain KC, … Network GS (2008). Seasonality, annual trends, and characteristics of dengue among ill returned travelers, 1997-2006. Emerging infectious diseases, 14(7), 1081. [DOI] [PMC free article] [PubMed] [Google Scholar]
  45. Scott TW, & Morrison AC (2010). Vector dynamics and transmission of dengue virus: implications for dengue surveillance and prevention strategies In Rothman A (Ed.), Dengue virus (pp. 115–128). Berlin, Heidelberg: Springer. [DOI] [PubMed] [Google Scholar]
  46. Stoddard ST, Forshey BM, Morrison AC, Paz-Soldan VA, Vazquez-Prokopec GM, Astete H, … Halsey ES (2013). House-to-house human movement drives dengue virus transmission. Proceedings of the National Academy of Sciences, 110(3), 994–999. [DOI] [PMC free article] [PubMed] [Google Scholar]
  47. Stoddard ST, Morrison AC, Vazquez-Prokopec GM, Soldan VP, Kochel TJ, Kitron U, … Scott TW (2009). The role of human movement in the transmission of vector-borne pathogens. PLoS neglected tropical diseases, 3(7), e481. [DOI] [PMC free article] [PubMed] [Google Scholar]
  48. Tang W, & Jia M (2014). Global sensitivity analysis of a large agent-based model of spatial opinion exchange: A heterogeneous multi-GPU acceleration approach. Annals of the association of American geographers, 104(3), 485–509. [Google Scholar]
  49. Thiele JC, Kurth W, & Grimm V (2014). Facilitating parameter estimation and sensitivity analysis of agent-based models: A cookbook using NetLogo and R. Journal of Artificial Societies and Social Simulation, 17(3), 11. [Google Scholar]
  50. Thomas SJ, Aldstadt J, Jarman RG, Buddhari D, Yoon I-K, Richardson JH, … Rothman AL (2015). Improving dengue virus capture rates in humans and vectors in Kamphaeng Phet Province, Thailand, using an enhanced spatiotemporal surveillance strategy. The American journal of tropical medicine and hygiene, 93(1), 24–32. [DOI] [PMC free article] [PubMed] [Google Scholar]
  51. Valbuena D, Verburg PH, Bregt AK, & Ligtenberg A (2010). An agent-based approach to model land-use change at a regional scale. Landscape ecology, 25(2), 185–199. [Google Scholar]
  52. Watts DJ, & Strogatz SH (1998). Collective dynamics of ‘small-world’networks. Nature, 393(6684), 440. [DOI] [PubMed] [Google Scholar]
  53. WHO. (2012). Global strategy for dengue prevention and control 2012-2020. [Google Scholar]
  54. Yoon I-K, Getis A, Aldstadt J, Rothman AL, Tannitisupawong D, Koenraadt CJ, … Jarman RG (2012). Fine scale spatiotemporal clustering of dengue virus transmission in children and Aedes aegypti in rural Thai villages. PLoS neglected tropical diseases, 6(7), e1730. [DOI] [PMC free article] [PubMed] [Google Scholar]

RESOURCES