Strategic directions for agent-based modeling: avoiding the YAAWN syndrome

Abstract

In this short communication, we examine how agent-based modeling has become common in land change science and is increasingly used to develop case studies for particular times and places. There is a danger that the research community is missing a prime opportunity to learn broader lessons from the use of agent-based modeling (ABM), or at the very least not sharing these lessons more widely. How do we find an appropriate balance between empirically rich, realistic models and simpler theoretically grounded models? What are appropriate and effective approaches to model evaluation in light of uncertainties not only in model parameters but also in model structure? How can we best explore hybrid model structures that enable us to better understand the dynamics of the systems under study, recognizing that no single approach is best suited to this task? Under what circumstances – in terms of model complexity, model evaluation, and model structure – can ABMs be used most effectively to lead to new insight for stakeholders? We explore these questions in the hope of helping the growing community of land change scientists using models in their research to move from ‘yet another model’ to doing better science with models.

Introduction

Land systems science is one of many research domains in which agent-based models are being increasingly deployed (Filatova, Verburg, Parker, & Stannard, 2013; Matthews, Gilbert, Roach, Polhill, & Gotts, 2007; Parker, Manson, Janssen, Hoffmann, & Deadman, 2003; Railsback, Lytinen, & Jackson, 2006). The past decade or so has seen an interesting evolution of the community of scientists working on agent-based systems, such that now there is a wide array of ABM applications that have been developed with strong empirical foundations (Janssen & Ostrom, 2006). Indeed, ABM simulation models have been used as an approach to understand a very diverse array of socio-environmental systems of all kinds in geography, urban studies, anthropology, economics, and related social sciences. At the same time, the quickly growing number of case studies and the larger questions they raise led to us and others to jokingly, and then more seriously, use the acronym YAAWN (Yet Another Agent-Based Model … Whatever … Nevermind …) in discussions of where ABM are, or could be going, in land-use modeling.¹ It is an attempt to inject humor into a source of frustration for many agent-based modelers, namely the growing sense that while the profusion of ABM cases is ultimately a sign of research vitality, it is not always apparent how these different cases add up to generalized knowledge about the systems under study.² Here we share a sense of the scope of the problem, potential solutions, and ways forward.

While using agent-based models has become relatively routine – they are now a well-accepted tool in the land systems toolkit judging from publication trends (Figure 1) – there is a danger that the research community is not learning broader lessons from their use. In the face of their growing adoption, it has become pertinent to ask: what is the marginal contribution of additional ABMs of particular social-ecological systems? And related to this concern: what do ABMs offer that other well-understood and powerful methods do not (O’Sullivan & Perry, 2013)? Many ABMs in land systems science (and other fields) are highly specific case studies, focused on particular places at specific times in the context of policy-related questions and concerns (An, Zvoleff, Liu, & Axinn, 2014; Berger & Troost, 2014; Magliocca, Safirova, McConnell, & Walls, 2011; Manson & Evans, 2007; Murray-Rust, Rieser, Robinson, Miličič, & Rounsevell, 2013; Walsh et al., 2013). The profusion of case-studies employing a new and interesting modeling methodology is exciting and continues a long tradition of case studies in geography and cognate disciplines. Indeed, a case-study-driven focus may even be inevitable given the urgent need for better understanding of the complex interactions among diverse decision-makers, disaggregated land-use choices, and the knock-on effects of those decisions on phenomena ranging from the policy arena to the global climate system.

Figure 1.

Number of manuscripts by year involving land and ABM (ISI Web of Science search on the terms agent-based and model and land).

A key drawback to the preponderance of case studies that use ABM is that each model usually brings its own and often unique structures and processes to bear on questions of interest, making cumulative learning challenging. Is there a risk of fatigue with ABMs and to what degree does research using ABMs as a tool position itself as offering methodological contributions versus topical or domain science contributions? We do not mean to suggest that all ABM research should strive to simultaneously satisfy methodological, conceptual, and policy goals, in part because ABMs are now widely accepted as an appropriate tool for land systems science. At the same time, just as statistical analyses of case studies within land systems science came to prominence in the last two decades, we can take advantage of the fact that ABMs are now commonly used to reflect on strategic opportunities for methodological and theoretical advances. There is a good deal of unrealized potential that could arise from more purposeful situation of case studies that use agent-based modeling as a basis for comparison, a call that has been made but only partially satisfied (Parker & Filatova, 2008, Rindfuss, Walsh, Turner, Fox, & Mishra, 2004).

Moving forward ABMs of land change

Here we offer a brief summary of four broad issues on ABMs in land change science, and also on potentially fruitful future research directions that complement those considered in other recent overviews of agent-based models (Filatova et al., 2013; Heppenstall, Crooks, See, & Batty, 2012).

Agent-based models and theoretical development

Initial applications of ABMs to land systems emphasized theoretical dimensions and were largely abstract (Janssen & Ostrom, 2006). We are fortunate now to have a large library of ABM applications that demonstrates the broad utility of this approach. But there has been a drift away from using ABMs to engage with theory, whether to explore the implications of different theoretical frameworks, or to develop new theories. Instead, an increased focus on applications has directed attention to more ad hoc efforts attempting to build realistic models of particular systems. This is not necessarily a criticism, but simply an acceptance of a key corollary of the natural evolution of a new methodology. Whereas initially it was necessary to demonstrate that the novel ABM approach was capable of strengthening analysis and theory, its demonstrated utility has now made it ‘safe’ to use ABM to explore many applications across diverse systems. The natural outcome is a growing library of ABM case studies in land systems science.

This trend is driven in part by the increasingly powerful computers and development platforms available to model builders (Railsback et al., 2006). Creating ABMs offers a range of design challenges, but there are no longer any serious technical limitations preventing researchers from adding detail to models when it is attractive to potential end-users in the policy and governance spheres. Arguably, data limitations impose a more significant challenge than computational power at this point. There may also be difficulties in some longer-term projects where it seems that the simulation model ‘product’ becomes more complicated and detailed as it evolves. In some cases, Lee’s (1973) perils of large-scale models may once again be upon us, in the form of the disutility of highly complicated and data-intensive models that become opaque to their users in comparison to simpler, more abstract models. Also problematic is that while the latter may be easier to learn from, what we learn may be of only limited applicability to real systems.

Thus, the first challenge raised by our discussion is: how do we strike an appropriate balance between empirically rich models and simpler theoretically grounded models, and how can we learn from such ‘mid-level’ models? By mid-level models we mean models that are realistic enough to represent the salient dynamics in a particular system, but do not incorporate so many elements or dynamics that the ability of the modeler or stakeholder to interpret how the model operates is seriously reduced. Indeed, finding this ‘goldilocks zone’ (Larsen, Thomas, Eppinga, & Coulthard, 2014) is relevant to all models, but ABMs perhaps face a particular challenge because even a relatively simple model with a small number of agents can yield a dizzying array of agent interactions. Furthermore, there are often challenges in acquiring sufficient empirical data to validate those agent-interactions (An et al., 2014; Evans, Phanvilay, Fox, & Vogler, 2011; Janssen & Ostrom, 2006; Kelley & Evans, 2011).

Evaluation and sensitivity analysis with agent-based models

Closely related to issues of theory development for ABMs are the challenges involved in model evaluation, which encompasses calibration, verification, and validation (Manson, 2007). There have been interesting and constructive developments in model evaluation methods in other fields, particularly spatial ecology research using agent-based approaches,³ that explore topics including model evaluation at all stages (calibration, verification, and validation), linking pattern to process, and describing models in standard ways to facilitate peer review (Grimm & Railsback, 2013; Grimm et al., 2005). Beyond these engagements, model evaluation remains a challenge, particularly the need for methods that can distinguish between contingent and general effects. Contingent effects are those that depend heavily on the specific geographical and historical development of a particular system, or even an individual model run. General effects are those that we might expect to see in other models of similar systems.

There have been developments in evaluation for models of complex systems (Brown, Page, Riolo, Zellner, & Rand, 2005; Messina et al., 2008) but they are not widely used (Filatova et al., 2013). An increasingly common approach in the model evaluation toolbox for ABM is classic sensitivity analysis, which globally or sequentially tests each model parameter to measure its impact on model outcomes (used variously during calibration, verification, and validation) along with methods to capture sensitivity to interactions between parameters (Ligmann-Zielinska & Sun, 2010; Lilburne & Tarantola, 2009). Beyond sensitivity testing, also highly relevant to model evaluation is the need for methods to assess the structural validity of models, addressing questions such as, ‘does the model represent appropriate system elements in the most appropriate ways?’ This is a larger scale issue that lies at the intersection of calibration, verification, and validation (Manson, 2003). This work stands in contrast to more commonplace efforts to identify the best-fit calibration of an already preferred model structure. What methods and approaches should the research community employ to determine whether or not an ABM appropriately represents salient system dynamics and interactions, both in terms of underlying concepts and external reality? Here we can paraphrase Occam’s razor, where all else equal, simple explanations are generally favored over more complicated ones; or perhaps better still, Einstein’s razor, where the model should be as simple as possible but no simpler. It is difficult to boil down this wide-ranging set of concerns, but we might summarize it as: what are appropriate and effective approaches to model evaluation in light of uncertainties not only in model parameterization, but also in model structure?

Hybrid approaches for agent-based modeling

Another avenue for further investigation in ABM examines the issue of model structure not from the direction of post hoc evaluation, but from the outset. We ought to be exploring more thoroughly the potential for hybrid forms of modeling, or more broadly, developing competing and complementary modeling approaches that enable iterative approaches to scientific inquiry (Filatova et al., 2013; Robinson et al., 2013). We advocate ‘hybrid forms’ in various senses. First, in the degree to which models are integrated, ranging from direct comparisons of different and separate modeling methods applied to the same domain. Second, to settings where different models are coupled or integrated with one another. Third, situations where truly hybrid models are created by tightly integrating or combining two or more approaches. The argument for hybridity is straightforward: ABMs are just one kind of computational model. Cellular automata are a closely related alternative, but many other different types of models can capture complicated patterns and processes. System dynamics models with their rich history of use in diverse fields, and their expressive, readily understood building blocks and associated graphical representation seem particularly suited to the challenges presented by scaling in ABM models of a range of land systems (Feola, Sattler, & Saysel, 2012; Luo, Yin, Chen, Xu, & Lu, 2010).

Hybridity at its simplest would imply either comparing separate models in the same research domain or coupling different models to examine a single domain. Actually implementing either case is more difficult than it sounds, because it requires developing a conceptual framework and a data set amenable to instantiating more than one form of model. Once these concepts and data are in place, it becomes possible to compare multiple approaches to the same problem, either as a form of model-to-model evaluation or as a way to give more insight into the problem (e.g., as seen in the growing amount of research contrasting statistical and ABM approaches to understanding human decisions that give rise to land change). More complete hybridity goes beyond this comparison to actually couple ABM to other kinds of models. There have been a number of methodological accomplishments in this arena including ABMs linked to ecosystem models (Polhill, Gimona, & Aspinall, 2011; Yadav, Del Grosso, Parton, & Malanson, 2008), ABMs coupled to metacommunity models (Gimona & Polhill, 2011), and ABMs coupled with emissions models (Bakam, Pajot, & Matthews, 2012; Heckbert, 2011).

Hybridity can go beyond comparison or coupling to actually integrate modeling approaches. The ability to computationally combine individually separate and discrete agents with continuous dynamics, for example, promises versatility that enables specification of appropriate dynamics at their proper scales, although this flexibility of functional form puts the onus of design choice on the modeler much in the same way as it does for ABM (Metcalf et al., 2013; Rahmandad & Sterman, 2008). Since many spatial scientists are already using such models, there are many opportunities for hybrid models to combine two or more different types of component models. These models can go beyond giving agents static decision rules derived from regression models, for example, to having agents do regression in a dynamic manner on their own as a proxy for decision-making. This form of hybridity uses ABM as a container for, and integrator of, different methods. The goal of such efforts should not be model hybridity for its own sake, but a practical recognition that the most appropriate model in any particular situation is not solely a function of the system being represented, but of the purpose for which the model is being developed.

We can sum up this issue as a call for fuller exploration of hybrid model structures that enable us to better understand the systems under study, recognizing that no single approach is best suited to this task. At the same time, for all its potential advantages, hybridity and integration of disparate model types comes with the challenges of reconciling the advantages of one modeling approach with the limitations of another. For example, system dynamics models are highly effective for overcoming the limitations of distinct time-steps (e.g., the commonly used annual or monthly time-step in land-change ABM models) and representing different types of flows in a system (Metcalf et al., 2013; Rahmandad & Sterman, 2008). Some ABMs can accommodate multiple time-scales but most ABMs are designed with a particular time-step in mind, which poses design limitations. For example, a modeler may make a decision with an ABM of agricultural production to use a one-year interval for a model designed to measure crop production over 20 or 30 years. But this means that the model necessarily aggregates the many complex within-year decisions that a farmer makes. Perhaps these within-year decisions can safely be aggregated to annual time-steps without jeopardizing the ability of the model to produce plausible model outcomes. If this model were then coupled with a system dynamics model of groundwater hydrology, there is something of a design disconnect and it may not be the case that simply by putting these two models together that the coupled model necessarily ‘benefits’ from the systems dynamic capabilities of the groundwater model. Nevertheless, there is certainly tremendous potential in hybrid modeling approaches (McNamara & Keeler, 2013).

Participatory agent-based models

Considering the purpose of models brings us naturally to the last major issue raised, the question of how ABMs (or simulation models more generally) can be used to connect with land-change stakeholders beyond the scientific community (Voinov & Bousquet, 2010). ABMs are often considered as being especially suited to participatory and stakeholder-driven processes with their graphical interfaces and intuitive mapping onto real-world concepts (Barreteau, Le Page, & D’Aquino, 2003; Van Berkel & Verburg, 2012). This issue ties back to each of the three previous ones. Some end-users prefer simpler and more abstract models and lose interest as models become more elaborate and harder to fully comprehend. ABMs perhaps have a tendency towards complexity but ‘a simple model that can be well communicated and explained is more useful than a complex model that has narrow applicability, high costs of data, and more uncertainty’ (Voinov & Bousquet, 2010). But certainly some users prefer to engage with highly complex models because of a desire for greater ‘realism’ and a wish to not leave system elements out of a model. For example, Millington, Demeritt, and Romero-Calcerrada (2011) found that users wanted more explicit representation of subsidies and land prices when presented with an ABM of land-use change and fire regimes. How models are evaluated, particularly model and parameter uncertainties, is a key concern in engaging with model stakeholders. And while model builders may consider the particular model structures adopted to be fascinating (whether for ABM, system dynamics, or others), they are often irrelevant to the purposes for which models are used. Perhaps this issue can be most conveniently summarized by asking: under what circumstances – in terms of model complexity, model evaluation and model structure – can ABMs be used most effectively to lead to new insights for stakeholders?

Brief survey and concluding thoughts

Where does current work on ABMs of land systems stand in regard to these issues? Here we provide a targeted presentation of just a subset of ABM literature by examining the ABM implementations published in the Journal of Land Use Science since inception. We acknowledge that this is a biased sample and not representative of the larger body of work on ABMs of land systems. For example, manuscripts published in Environmental Modeling and Software, another journal that commonly publishes ABM research, may have a more methodological orientation than manuscripts in JLUS. Nevertheless, we present this overview simply as a way to reflect on one particular community of modelers writing to the aims and scope of one particular journal that acts as the standard-bearer for land change science. We note that there are several ABM oriented manuscripts published in JLUS that are overviews, commentaries, or methodological observations in the ABM literature (Luus, Robinson, & Deadman, 2013; Messina et al., 2008; Schreinemachers & Berger, 2006) and we excluded those from the tabular presentation and evaluation we present here.

Based on this limited review of ABM applications in JLUS, it appears that the emphasis tends to be towards models that are not designed for participatory modeling. There are important exceptions, among them (Millington et al., 2011), and we emphasize that ABM models of course need not be participatory to be of value. But the question remains whether land-use modelers using ABM approaches are more or less likely to develop participatory models compared to those using other modeling approaches. Another observation is that few manuscripts explicitly performed a sensitivity analysis, although again there are exceptions (Ligmann-Zielinska & Sun, 2010; Tang, Bennett, & Wang, 2011). We also note that there are varying types of sensitivity analysis deployed, including explicit evaluation of parameter sensitivity as well as expert-informant evaluation, which may perhaps be considered a flavor of sensitivity analysis (Dumrongrojwatthana, Le Page, Gajaseni, & Trébuil, 2011). This is based on a review of the manuscript content and it is possible that the model developers did perform a sensitivity analysis not reported in the JLUS manuscript itself. But even in that case, it suggests that modelers do not routinely describe the sensitivity of their models to various parameters, which may warrant some reflection within the community. A challenging dimension to evaluate is the degree of model engagement with theory. Our evaluation is necessarily subjective but we have attempted to infer the intention of the authors in orienting the content of the manuscript towards description or application of a model as opposed to aiming for insight into specific theoretical issues and questions. There is a clear difference between a manuscript designed to compare decision-making approaches (Schreinemachers & Berger, 2006) versus a model designed to demonstrate the potential of parallel-computing advances (Tang et al., 2011) or the utility of provenance in ABMs (Bennett, Tang, & Wang, 2011). Lastly, we evaluated whether models were conceptual in design or built on empirical data that was used in the design, calibration, or validation of the model. Here we found a broad mix. While the ABM field has its roots in conceptual or ‘toy’ models, the field has quickly evolved to ABMs that embraced empirical data especially for land use/land cover change modeling purposes (Janssen & Ostrom, 2006). Despite this evolution, we found the articles we reviewed still include many that are more conceptually based than built on empirical data. On the other hand, it is clear that many models were published to communicate a particular technical challenge (Tang et al., 2011) and the authors may well have published subsequent or related manuscripts in other journals using the same model that did utilize empirical data in some aspect of a larger project. We also found there are diverse approaches to leveraging empirical data, although they are often used to establish statistical relationships derived from household survey data to inform decision-making dynamics in the model (Entwisle, Malanson, Rindfuss, & Walsh, 2008).

Overall, the papers in aggregate highlight the importance of pursuing the four above-noted research areas.

• Sensitivity analysis and model verification. Judging from these papers, there is a continued need for sensitivity analysis, especially in forms tailored specifically to ABM and not just standard measures such as Kappa that are maybe more suited to static comparison and linear models. Related to this, we see a slow but growing movement towards better model verification in general as ABM increasingly include in-depth explanation of model design either in the manuscript itself (Walsh et al., 2013) or as an appendix (Magliocca, Brown, & Ellis, 2013). There are also repositories for ABMs to facilitate sharing of code such as OpenABM (http://www.openabm.org). This trend towards deep description and availability of code repositories has long been called for in the modeling community and promises to accelerate the development of ABMs.

• Participatory modeling. Most models are not participatory; this is not a problem in the sense that many papers do not claim to have a participatory bent, but the suitability of ABM for this form of model calibration, evaluation, and translation to policy is both promising and under-explored. Particularly interesting is the linking of models in laboratory and experimental settings (Evans, Sun, & Kelley, 2006) in addition to more traditional forms of modeling with explicit engagement with a given community of stakeholders (Voinov & Bousquet, 2010).

• Hybrid modeling. There is definite progress with hybrid models, particularly those that seek to create model ensembles (i.e., linking the model to existing models, such as an ABM of human behavior tied to an established carbon model) and link to complementary approaches like statistical regression (Luus et al., 2013). This work is also focused on leveraging models of ecological of physical systems (Yadav et al., 2008).

• Theoretical engagement. There is growing engagement with theory but more work is necessary to increase the modeling community’s understanding of how to make theoretical connections from ABM to diverse concepts across fields. A larger issue here is the relatively unsettled state of theorization in land change science, but at the same time, there has been new work to outline conceptual ABMs that can serve as platforms for others. These include applications related to domains such as land markets (Parker & Filatova, 2008) and agricultural change (Murray-Rust et al. 2013).

These issues are by no means a comprehensive set of possible directions for researchers to explore with ABMs. The dimensions we have identified are not intended to delineate ‘best practices’ for ABM research but rather a reflection on possible directions for ABM work, to avoid the potential for the field to devolve into one case study after another. Perhaps there is simply a need for case studies to more fully articulate how they contribute to theoretical and/or methodological debates and concerns. Looking ahead, we hope that the research community will more directly target the broad issues we have outlined: the place of mid-level models; new approaches to model evaluation; the need for hybrid models; and the challenges of successful model-user engagement. By highlighting these challenges, we can perhaps mobilize the community of spatial scientists using models in their research to move from ‘yet another model’ to doing better science with models.