Agent-based modeling as organizational and public policy simulators

Abstract

Agent-based models are an increasingly powerful tool for simulating social systems because they can represent important phenomenon difficult to capture in other mathematical formalisms. But, agent-based models have provided only limited support for policy-making because their distinctive abilities are often most useful in situations where the future is unpredictable. In such situations, the traditional analytic methods for applying simulation models to support decision-making are least effective. Fortunately, new analytic approaches for decision-making under conditions of deep uncertainty—emphasizing large ensembles of model-created scenarios and adaptive policies evaluated with the criteria of robustness, rather than with optimality or efficiency—can unleash the full potential of agent-based policy simulators.

Agent-based models are increasingly recognized as powerful tools for simulating social systems. The papers in this session, Agent-Based Modeling as Organizational and Public Policy Simulators, of the October 2001 Sackler Symposium, grapple with the use of such models as policy simulators to provide quantitative support to decision-makers in the public and private sectors. This question has particular relevance because such models have to date only had limited impact on practical decision-making. I will offer my views on why this is so and then discuss how the papers in this session exemplify an emerging new form of decision sciences appropriate for such models.

Let me begin with the question: why use agent-based models to support real-world decisions? There is no shortage of more traditional mathematical representations commonly used for such purposes, including the differential equations of engineering and economic models, statistical forecasters, or systems dynamics models. As expressed eloquently in many papers at this conference, agent-based models are useful because they can represent important information about the world not easily captured with such traditional models. In particular, agent-based models excel at relating the heterogeneous behavior of agents with different information, different decision rules, and different situations to the macro behavior of the overall system.

For instance, in the paper Agent-Based Modeling as a Means of Understanding Extinct Social Behavior (unpublished, presented at the Colloquium), Gumerman and colleagues examine the history of the Anasazi tribes of the American Southwest. They use agent-based models to combine anthropological information on the behavior of individuals and groups in such tribal societies with detailed climatological data on the environmental shifts faced by the Anasazi on the eve of their disappearance from a now well studied valley in present-day New Mexico. Model runs are compared against archeological data on Anasazi settlement patterns, offering deeper understanding of how various behavior patterns combined with climate trends to shape Anasazi society.

The ability of agent-based models to connect heterogeneous microbehavior to different patterns of macrobehavior raises a crucial second question. Once the models are created, how ought they be exercised to support decision-making? The distinctive strengths of agent-based models often emerge in situations where their application presents a particular challenge. Traditional decision analysis has been enormously useful in enabling the systematic application of simulation models to support decision-making. These methods employ simulation models to rank the desirability of alternative actions by predicting their future consequences (1). That is, simulation models are traditionally useful because of their ability to predict reliably the future, either in a best estimate or probabilistic sense. But agent-based models are often most useful under conditions of deep uncertainty† where such predictions are not possible.

Moss' paper, Policy Analysis from First Principles (2), provides an interesting example of this general point. Moss demonstrates that weekly sales data from British supermarkets on several hundred brands of goods show a leptokurtosic, clustered volatility. That is, the distributions have fat tails whose moments do not converge with increasing sample size and thus are unpredictable by statistical forecasting. Moss argues that such clustered volatility does not arise in the equilibrium models commonly used for policy analysis relating to markets. He proposes an agent-model which reproduces this general behavior but cannot predict its specific manifestation in any particular case. Agent-based models frequently excel at describing the behavior of inherently unpredictable systems. Thus, agent-based models may be most important as policy simulators precisely in those situations where the standard methods of predictive policy analysis are least effective.

In recent years, researchers have begun to grapple with the challenge of using such nonpredictive models to inform policy-making. Commonly, “flight simulator” approaches allow the user to play out a small number of often arbitrarily chosen scenarios. Although a tentative first step, such simulators have nothing of the rigor required for systematic decision analysis. For instance, when pilots train in actual flight simulators, it is straightforward to determine the scenarios on which to focus. In contrast, when addressing a complex public policy problem, such as the actions that best address the threat of global climate change, it is not at all obvious which of a huge number of potentially relevant scenarios ought to be considered. At a minimum, rigorous policy analysis requires some means to define and identify the most important scenarios.

Fortunately, the same computer advances that have made agent-based models viable simulators also have enabled new methods of decision analysis suitable for use with nonpredictive models. These multiscenario simulation methods provide systematic, quantitative guidance as to which scenarios to examine and what information to draw from them (3–6).

Multiscenario simulation methods take a variety of forms and are emerging with nonpredictive models—agent-based or others—in many fields. All emphasize one or more key steps. First, they use ensembles of scenarios, with hundreds to millions of members, to capture what is known about the future, rather than any single or small number of runs. Second, they compare alternative policy decisions across this ensemble of scenarios by using criteria such as robustness, resiliency, and stability. A robust strategy is one that performs relatively well, compared with the alternatives, across a wide range of plausible futures. Traditional policy analysis rests on the criterion of optimality or efficiency. Alternative policies are ranked by their expected performance given the probability weighting over alternative scenarios. But, robustness is a better criteria in those situations where it is difficult to predict the future, that is, the likelihood of alternative scenarios is unknown. Not surprisingly, robustness, not efficiency nor optimality, is often the criteria the human decision-makers use in practice under such situations (7). Finally, multiscenario simulation methods explicitly consider policy decisions as an adaptive response that will evolve over time to new information, rather than any fixed set of actions. Policy-makers commonly achieve robustness with such adaptive planning.

Taken together, these steps use the computer as an interactive tool to help humans with what they do best—discover patterns, make hypotheses about the best actions to take, and test those hypotheses against the available data captured in the scenario ensembles—rather than use computers as calculators to support deductive reasoning. For instance, multiscenario simulation often defines, at the start of the analysis, the most important scenarios as those most useful in eliciting information and buy-in from the parties involved in the decision. At the end of the analysis, the most important scenarios can be those most stressing for the recommended policy actions (8).

Each of the papers in this session address important aspects of these new emerging methods for using agent-based models for policy analysis. Gumerman and colleagues focus on the validation of agent-based models, that is, ensuring accurate representation of real systems. The authors claim their model is valid because it reproduces data on Anasazi settlement patterns. Based on this fit, they predict a counterfactual—that environmental stress was not sufficiently severe to force the Anasazi to have vanished from their valley. This work is an important example of the ability to validate agent-based models. But the example would have even more power had the authors expressly grappled with the unpredictability of agent-based models and uncertainty about the precise behaviors of these vanished peoples. In particular, rather than consider a single scenario, Gumerman and colleagues might have considered an ensemble of different behavioral rules, asking whether or not all plausible rules produce a similar fit to the settlement data and whether or not all rules predict the counterfactual of continued Anasazi settlement in the valley.

Carley's paper, Computational Organization Science: A New Frontier (9), provides a compelling demonstration of the unique ability of agent-based models to examine how internal policies, procedures, and technologies affect the performance of commercial firms and other organizations in an information economy. Carley views organizations as networks of intelligent, adaptive agents. She uses agent-based simulators to relate the overall behavior of organizations to data on the knowledge, capabilities, tasks, procedures, and networks of communication for the agents of which they consist. Her models successfully reproduce the actual performance of many real organizations.

Carley describes the policy analysis she conducts with her models as “what if” exercises that allow decision-makers to examine the implications of alternative choices about the design of their organizations. Her results, however, suggest that she implicitly interprets her work with the concepts of multiscenario simulation. Rather than claim accurate forecasts of the future performance of organizations—which, she would be the first to argue, depends not only on internal choices but also on the evolution of the external environment—Carley uses her models to compare the performance of alternative organizational designs over a wide range of scenarios. For instance, her models show that over a wide range of conditions, successful organizations respond to changing circumstances by, first, redesigning internal procedures and, only as a last resort, hiring and firing. In contrast, poorly performing organizations respond first to shifting circumstances with personnel changes. Such arguments implicitly employ robustness criteria—showing how specific organizational designs perform relatively well across many scenarios. Carley's paper would be strengthened if she explicitly discussed the range of scenarios where her recommended policies dominate and those, if any, where they do not.

Moss' paper emphasizes the ability of multiscenario simulation to engage parties to a decision with different views and expectations. Models are used both to elicit information from stakeholders and to help them find policies robust across their different views. In his work on freshwater integrated resource management, Moss builds agent-based models that can reflect stakeholder perceptions. He argues that in situations where predictions are difficult, models must be based on good science, but it is also crucial to include the goals and observations of key stakeholders. When stakeholders disagree, the result is an ensemble of models. Moss' view that policy analysis aims to “develop strategies to mitigate the impacts of clusters of extreme events” is similar to the notion of robustness.

In his closing comments, Moss suggests that agent-based social simulations can play a key role in supporting the development of social processes of policy-making when prediction is not feasible. This point is a crucial one. When faced with conditions of deep uncertainty, actual decision-makers do not rely on forecasts. Rather, they seek robust solutions, often by putting into play adaptive social processes. Prediction-based policy analysis has inhibited the full use of agent-based models as policy simulators because it diverts focus from the questions decision-makers are most prone to ask and agent-based simulators are best positioned to answer. As policy analysts increasingly employ methods of multiscenario simulation, the use of agent-based models for policy analysis will approach its full potential.