Embedding co-production and addressing uncertainty in watershed modeling decision-support tools: successes and challenges

Abstract

Decision-support tools (DSTs) are often produced from collaborations between technical experts and stakeholders to address environmental problems and inform decision making. Studies in the past two decades have provided key insights on the use of DSTs and the importance of bidirectional information flows among technical experts and stakeholders – a process that is variously referred to as co-production, participatory modeling, structured decision making, or simply stakeholder participation. Many of these studies have elicited foundational insights for the broad field of water resources management; however, questions remain on approaches for balancing co-production with uncertainty specifically for watershed modeling decision support tools. In this paper, we outline a simple conceptual model that focuses on the DST development process. Then, using watershed modeling case studies found in the literature, we discuss successful outcomes and challenges associated with embedding various forms of co-production into each stage of the conceptual model. We also emphasize the “3 Cs” (i.e., characterization, calculation, communication) of uncertainty and provide evidence-based suggestions for their incorporation in the watershed modeling DST development process. We conclude by presenting a list of best practices derived from current literature for achieving effective and robust watershed modeling decision-support tools.

1. Introduction

Decision-support tools (DSTs) are used to address the challenges of complex decision making across an array of linked sociological, economical, and environmental issues. Numerous DSTs have been developed over the past several decades in response to diverse stakeholder needs including, for example, quantifying ecosystem services (Grêt-Regamey et al., 2016), economic resilience to climate change (Watkiss et al., 2015), water treatment options (Raseman et al., 2017), water and energy use (Ward, 2007), as well as agricultural management (Rose et al., 2016) and urban planning (Brabec et al., 2002). Social, political, and institutional structures can present barriers toward implementing evidence-based solutions to these and other management problems. Effective DSTs can therefore play an important role in enabling the analysis and communication necessary to navigate these challenges and solve multi-faceted management problems based on scientific evidence (Smits, 2002).

Technical experts (e.g., scientists from academia, governmental organizations, privately funded contractors) are typically responsible for developing and distributing the DST, while stakeholders (e.g., tribes, municipalities, land owners) provide input to the development process of the DST and then either use the DST or its results to inform decision making. Some paradigm obstacles have prevented DSTs from being more widely adopted, many of which stem from a lack of communication between technical experts and the stakeholders (Rogers and Fiering, 1986).

Communication among technical experts and stakeholders is especially important for a class of issues termed “wicked problems” (Rittel and Webber, 1973), where both the benefits and costs have considerable societal impacts. Poor communication among technical experts and stakeholders may be attributed, in part, to the differences in performance criteria and rewards systems of the two groups in their respective employment sectors [see, for example, Winters et al. (2016)]. For example, technical experts in academia are expected to publish new tools and methods, yet they are rarely rewarded based on the usability of a particular tool they create (Yarime et al., 2012). Conversely, stakeholders rarely have the time or resources needed to use new and unfamiliar tools. These differing expectations are not conducive to fostering communication between these groups, and this lack of communication can result in DSTs that do not directly address the relevant issues associated with specific management problems.

It is well known that direct interactions between technical experts and stakeholders can help to more effectively address environmental management problems (Forrester, 1999; Halbe et al., 2017; Prell et al., 2007; Smith et al., 2017). Callon (1999) differentiates three models of interaction between technical experts and stakeholders: (1) the “public education” model, in which technical experts are solely responsible for scientific inquiry and the dissemination of results to the public; (2) the “public debate” model, whereby the technical experts and stakeholders are still separate but there is dialogue between the two sides; and (3) the “co-production” model, which promotes close, direct, two-way interactions between groups to implement the technical aspects of DSTs as well as to convey the utility of that information (Lemos and Morehouse, 2005). While some researchers have discussed the need for public participation in environmental assessments for decades [e.g., Forrester (1999)], recent work has validated that the co-production model improves the efficiency and effectiveness of modeling tools, thereby increasing their use among communities and stakeholder groups (Basco-Carrera et al., 2017a; Huntington et al., 2011; Jones et al., 2015; Mutekanga et al., 2013; Tyl et al., 2015).

In light of these findings, numerous publications have aggregated case studies from the literature and have provided best-practice recommendations regarding the use of co-productive techniques to develop models and modeling support tools. Various terminology has been used to characterize these approaches, including co-production (Ferpozzi, 2017), collaborative and participatory modeling (Basco-Carrera et al., 2017b; Gray et al., 2018; Halbe et al., 2017; Voinov and Gaddis, 2008), translational ecology (Jackson et al. 2017), structured decision making (Failing et al., 2013; Gregory et al., 2012), and generally stakeholder participation (Loucks, 1992; Loucks et al., 2005; Scholten et al., 2001; Voinov and Bousquet, 2010; Voinov et al., 2016). For example, Voinov (2016) and Voinov and Bousquet (2010) demonstrate the importance and demand for stakeholder participation in resource and environmental modeling, extracting information from over 200 papers published in Environmental Modeling and Software (from 2010 to 2016) that referenced stakeholder involvement. This builds upon work several works aimed at promoting stakeholder involvement in water resources modeling [e.g., Van Waveren (1999), Black et al. (2014); Loucks et al. (2005); Van Waveren (1999), Loucks and da Costa (2013)].

Incorporation of uncertainty within the DST development process may lead to more credible and robust tools. The importance of various forms of uncertainty embedded in watershed models that are used for decision support have been widely discussed for several decades in the literature [e.g., Beck (1987), Beven (2010)]. Uncertainty in models can be manifested from input data, model parameters, model structure (e.g., mechanistic or statistical approaches), and simulated output. Rather than only attempting to minimize uncertainty, efforts to characterize, calculate, and communicate uncertainty for problem-solving are widely advocated [e.g., Liu and Gupta (2007), Morgan (2009)]. A wide range of approaches exist for addressing different sources of uncertainty in integrated environmental modeling and DSTs (Matott et al. 2009). Numerous authors provide guidelines for producing quality assurance for environmental modeling (Jakeman et al., 2006; Refsgaard and Henriksen, 2004; Refsgaard et al., 2005; Refsgaard et al., 2007) and addressing uncertainty via frameworks for hydrological and environmental models (Liu and Gupta, 2007; Pianosi and Wagener, 2016).

Despite the extensive number of publications regarding best implementation practices for developing DSTs through the joint involvement of technical experts and stakeholders – as well as separate papers that describe uncertainty analysis in environmental modeling – recent and extensive analysis specifically focusing on watershed models used for DSTs is lacking. Moreover, gaps in the literature remain on the joint consideration of co-production and uncertainty analysis in each step of the DST development pathway.

In this paper, we begin with a simple conceptual model for jointly developing effective DSTs that use watershed models to address a variety of management challenges. The conceptual model involves three basic steps in the DST process (problem identification, method development, and results analysis). We then discuss methods for embedding co-production and systematic approaches for addressing uncertainty into each stage of the conceptual model. We use watershed modeling case studies from the literature to highlight the successful outcomes and challenges associated with integrating both co-production and uncertainty concepts and practices into co-produced watershed modeling DSTs. We conclude by enumerating best practices derived from our literature-based examples that enable successful outcomes.

2. Definitions and Scope

We focus on the development of DSTs that utilize watershed models (Table 1). We operationally define a DST as a tool created by technical experts and used by stakeholders to produce results that can inform management and policy decisions. Informed decisions greatly vary and can include, for example, flood protection and infrastructure planning (Loucks, 1992), placement of conservation practices to improve water quality (Whittaker et al., 2017), and land use planning to promote biodiversity (Lautenbach et al., 2013; Seppelt et al., 2013). For the context of this study, we consider DSTs that are either standalone watershed models or interfaces that allow the user to interact with a watershed model—for example, to create alternative management scenarios, run the model, and view the results.

Table 1.

Watershed models used within (or as) a decision-support tool (DST) that fall within the scope of our presented framework for addressing uncertainty.

Decision-Support Tool (DST)	Watershed Model	Technical Experts	Catchment Delineation	Stakeholder Interaction	Modeling Outputs	Primary Application
Agriculture Non-Point Source Pollution Model (AgNPS)	AgNPS	US Department of Agriculture, Agricultural Research Service and National Resources Conversation Service; www.nrcs.usda.gov go.usa.gov/KFO	Field, Sub-basin	AGNPS GIS Interface	Hydrology, Water Quality	Agricultural
Better Assessment Science Integrating point and Nonpoint Sources (BASINS)	Hydrologic Simulation Program – Fortran (HSPF), Stormwater Management Model (SWMM), Soil and Water Assessment Tool (SWAT)	US Geological Survey; US EPA; www.epa.gov/exposure-assessment-models/basins-download-and-installation	Sub-basins	ArcGIS interface	Hydrology, Water Quality	Agricultural, Urban
Hydrologic and Water Quality System (HAWQS)	SWAT	US EPA; Texas A&M; USDA-ARS; https://epahawqs.tamu.edu	Sub-basins	Web interface, standalone application	Physical Hydrology, Water Quality	Agricultural
Model My Watershed	SLAMM, TR-55	PV Associates; wikiwatershed.org/model	Sewersheds	App; Standalone Application	Hydrology, Hydraulics	Urban, Suburban
National Stormwater Calculator	SWMM	US EPA; www.eoa.gov/water-research/national-stormwater-calculator	Sewersheds	Standalone Application	Economics, Hydrology, Hydraulics	Urban, Suburban
Regional Hydro-Ecologic Simulation System (RhEssys)	RhEssys	University of North Carolina and San Diego State University; www.github.com/RHESSys	Gridded Cells	Standalone Application	Hydrology, Water Quality	Agricultural, Forestry
Sacramento Soil Moisture Accounting Model (SAC-SMA)	SAC-SMA	US National Weather Service; nws.noaa.gov/iao/iao_hydroSoftDoc.php	Sub basins or Basins	R Package	Physical Hydrology	Rural, Urban
Spatially Referenced Regressions on Water Attributes (SPARROW)	SPARROW	US Geological Survey; Water.usgs.gov/nawqa/sparrow/sparrow-mod.html	Sub-basins	SAS (Statistical Analysis System Institute) software; Online interface	Hydrology, Water Quality	Developed from forested reference conditions; Applied for all land cover types
System for Urban Stormwater Treatment and Analysis Integration (SUSTAIN)	SWMM	US EPA; www.epa.gov/water-research/system-urban-stormwater-treatment-and-analysis-integration-sustain	Sewersheds	Standalone Application	Hydrology, Hydraulics, Water Quality	Urban, Suburban
Variable Infiltration Capacity (VIC) Macroscale Hydrologic Model	VIC	UW Hydro Computational Hydrology Group, University of Washington uw-hydro/github.io	Gridded Cells	Various, Python Driver (Open source)	Physical Hydrology	Agricultural, Natural Systems
Visualizing Ecosystem Land Management Assessments (VELMA)	VELMA	US EPA; https://github.com/USEPA/VELMA_Public	Gridded Cells	Standalone Application	Hydrology, Water Quality	Forestry, Agricultural, Grassland, Coastal, Urban
Watershed Management Optimization Support Tool (WMOST)	HSPF, SWAT	US EPA; https://www.epa.gov/exposure-assessment-models/wmost	Sub-basins or Basins	Standalone Application	Economics, Hydrology, Water Quality	Rural, Urban

Watershed models are a particular class of environmental models that simulate various hydrological and biogeochemical components throughout a watershed (Chung and Lee, 2009; Lawler et al., 2010; Rabotyagov et al., 2016). Watershed model simulations can vary across spatial and temporal scales, typically ranging from 0.1 km² to 1000 km² (Golden et al., 2014) and from hourly or event to decadal time scales (Borah and Bera, 2003). We focus on mechanistic (i.e., process-based) watershed models that explicitly simulate the influence of upland rainfall-runoff and biogeochemical processes on downstream hydrology and water quality (Borah and Bera, 2003), in contrast to empirical watershed models that characterize statistical or conceptual relationships between measured watershed properties (e.g., soil infiltration rates, timing to runoff generation).

Throughout the paper and within the conceptual model, we distinguish between two groups of “assets”: stakeholders and technical experts. We consider stakeholders to be those who are actively involved in a decision-making process, are affected by the decision, and who use the tools produced through the DST process for decision making. In contrast, the technical experts are the “tool builders”, typically scientists or engineers from academia, government entities, non-governmental organizations (NGOs), or private sectors enlisted to create a software tool to address one or more decision-making objectives. While there is often overlap between the roles of these groups, it is helpful to illustrate the different assets of each entity when collaborating to construct a DST for watershed models and using it to inform policy decisions and action.

3. A Conceptual Model for Creating Effective Decision-Support Tools (DSTs) for Watershed Modeling

We present a conceptual model for systematically incorporating co-production and addressing uncertainty into the development and deployment of an effective DST for watershed modeling (Figure 1). With established assets (i.e., technical experts and stakeholders), there are three distinct and iterative stages in the conceptual model: (1) define the problem by developing and refining management objectives; (2) select and execute the modeling approach by incorporating all necessary modeling methods to achieve these goals; and (3) assess the results of the DST. Stage 3 involves communicating the results to all assets and determining whether the DST has adequately met the objectives using the selected methods. The final DST and associated DST results may then be used to inform policy and decision making.

Figure 1.

A conceptual model for the development of a decision-support tool (DST). After establishing assets (i.e., stakeholders and technical experts), both groups work together to define the problem, select the approach, and conduct an assessment of the results before producing a final DST and ultimately affecting policy and decisions.

The conceptual model also illustrates how feedback is incorporated throughout the DST-generation process (Figure 1). Various forms of feedback are re-introduced to the stages via the stakeholders and technical experts and serve to modify and refine the three development stages of the model.

In the following sections of the paper, each stage of the conceptual model will be examined from the perspective of watershed modeling case studies to highlight the successful outcomes and challenges found in the literature. In particular, we will emphasize the usage of co-production and highlight methods to incorporate the “3 Cs of uncertainty”—that is, the characterization, calculation, and communication of uncertainty.

3.1. Gather Assets

We have operationally defined a decision-support tool (DST) as a tool produced by technical experts and used by stakeholders to inform management decisions and policy. While in practice the genesis of the DST development process can be prompted from any of the stages of the conceptual model (Figure 1), we begin by describing the project participants, which we call assets. These assets include stakeholders and technical experts associated with the DST and the decision-making objective or management problem that is being addressed. The stakeholders should ideally represent a comprehensive set of all salient interests associated with the project (Grimble and Wellard, 1997; Mostert et al., 2007). Analogously, the technical expert team must collectively possess the ability to incorporate the relevant environmental and management options into the watershed model and DST.

At each step in the conceptual model, the assets offer different yet complementary resources (see Table 2). For instance, stakeholders often have local knowledge that can supplement historical observed data and inform modeling efforts conducted by technical experts. In the Columbia Basin Climate Change Scenarios Project (CBCCSP), for example, technical experts utilized the Variable Infiltration Capacity (VIC) hydrologic model, with extensive inputs from stakeholder groups from the United States (U.S.) and Canada, to construct a hydrologic database that supports climate change planning, impacts assessments, and adaptation preparations in the Pacific Northwest (Hamlet et al., 2013). Early communication between stakeholders and technical experts enabled the resulting publicly available CBCCSP database to be widely applicable and has reduced costs of numerous planning and coordination studies in the Pacific Northwest and Western U.S.

Table 2.

Typical project resources from stakeholders and technical experts

	Problem Definition	Approach: Data, Methods, and Tools	Assessment: Tool Implementation
Stakeholders	Knowledge of local challenges Community vision	Local data and tools Financial resources	Ability to determine feasibility of scenarios
Technical Experts	Scientific and technical knowledge Knowledge of current research gaps	Ability to conduct laboratory and field measurements Existing data and modeling capabilities Research Funding	Hypothetical planning or management scenarios Comparative case studies

The resources from each asset can be maximized if effective mechanisms for providing and receiving knowledge between the two entities are well established. Intermediaries and trained facilitators are oftentimes useful and will be discussed in the following section. In addition, Hall et al. (2014) outline a number of strategies to effectively communicate systems models between technical experts and stakeholders, and, in particular, they demonstrate the use of audience-analysis questions to better understand and characterize the goals of different asset groups.

3.2. Define the Problem

Once assets are assembled, the first stage of the conceptual model is to clearly define the decision-making problem and the respective objectives of the stakeholders (Figure 1). We will now discuss how co-production and uncertainty can be incorporated into this stage and highlight successes and challenges from watershed modeling case studies.

3.2.1. Define the Problem using Co-production

A problem formulation is comprised of objectives (i.e., goals of stakeholders), decision variables (i.e., management actions), and constraints (i.e., limitations on the management actions) (Coello, 1999). Objectives are typically elicited from the diverse group of stakeholders [e.g., Smith et al. (2017)], yet technical experts can also co-produce these objectives. For example, feedback from experts on the ability to quantify objectives proposed by stakeholders based on data availability and model functionality can shape the types of objectives selected for inclusion in the problem formulation (Caminiti, 2004). Selection of management options for inclusion in the problem formulation is oftentimes driven by stakeholders (Prell et al., 2007); however, input from technical experts, especially related to new and innovative management options, may also be important. Finally, constraints are also co-produced by stakeholders and technical experts. Constraints elicited from stakeholders typically relate to restrictions, such as regulations or budget, whereas constraints from technical experts may relate more to physical system or data limitations.

Watershed modeling studies applying participatory modeling concepts have demonstrated methods of facilitating stakeholder discussions (e.g., informal focus groups) that lead to a co-produced set of objectives, decision variables, and constraints (Grayson et al., 1994; Smith et al., 2017). These methods are often employed by federal and state agencies to ensure that environmental assessments are relevant to stakeholder needs. As one example, the U.S. Environmental Protection Agency (US EPA) engaged with local stakeholders to formulate the goals of a DST project using the SUSTAIN tool as part of a Total Maximum Daily Load (TMDL) assessment associated with E. coli in Albuquerque, New Mexico (Shoemaker et al., 2013).

Collaboratively defining watershed-scale management problems and establishing objectives is often a difficult process due to conflicting goals among stakeholders. For example, the goals of agricultural producers interested in optimizing farm-level management decisions generally conflict with the goals of municipal wastewater treatment managers who seek to improve water quality at watershed scales, as noted by case studies using SWAT (Rabotyagov et al., 2010; Whittaker et al., 2017), AgNPS (Young and Onstad, 1990; Young et al., 1989), and WMOST (Detenbeck, 2015). Some conflicts can be especially contentious. One such example is in the Raccoon River watershed in Iowa, in which the water treatment facilities responsible for maintaining water quality for Central Iowa in the United States, including the city of Des Moines, filed a federal suit against three rural counties. Their case purported that drainage tiles in farm fields were contributing to high nitrate levels (Des Moines Register, 2015a, 2015b). Rural stakeholders argued that restricting the use of drainage tiles could have a “devastating effect on agriculture and the economy across the Midwest” (Des Moines Register, 2015b). At such a juncture, arriving at consensus on objectives, if feasible, affords a more efficient DST process. Therefore, efforts that have attempted to simulate nitrate-reduction strategies in the Raccoon watershed using DSTs with SWAT (Jha et al., 2007; Jha et al., 2010) and WMOST (Detenbeck et al., 2015) emphasize the importance of representing stakeholder groups from both sides early in the decision-making process.

In challenging cases like the agricultural example exemplified here, previous watershed modeling case studies have utilized existing intermediary facilitators such as research social scientists, NGOs, or local government agency representatives as a bridge that can actively engage both groups to better understand and communicate the common and conflicting objectives. The facilitator approach has been implemented successfully in numerous examples, including mediated modeling for floodplain management (Metcalf et al., 2010) and facilitated discussions between private industry and environmental groups (Tyl et al., 2015) as well as natural resources management (Etienne et al., 2011). If needed, a trained facilitator can be hired (Mostert et al., 2007). To initiate this type of coordination, Halbe et al. (2017) present a methodological framework associated with participatory modeling to engage technical experts and stakeholders using trained facilitators in model development. Other studies have also emphasized the use of collaborative partnerships to facilitate modeling and decision support for water resource issues (Langsdale et al., 2013; Lubell et al., 2009). All of these studies suggest that building upon positive pre-existing relationships is often the best approach to developing strong partnerships between stakeholders (Lemos and Morehouse, 2005).

Successes from developing well-formulated objectives for the decision-support process are well documented. One promising example is in the Ngenge watershed, Uganda, where soil erosion from steep mountain slopes contributed to water quality issues. After numerous management attempts to control the erosive activity via legislation and policies, an alternative approach was taken to engage stakeholders to generate sustainable and workable objectives and solutions that included their visions of success for decreasing erosion-related water quality issues in the watershed. Indicators embedded within the project approach suggest that this method is proceeding toward effective outcomes (Mutekanga et al., 2013).

A common pitfall of a large and diverse group of technical experts and stakeholders is inadequately defining objectives for success early-on in the DST process. Without defining what common objectives and successful outcomes look like, projects often have “moving goals” throughout the course of the project. As a result, projects either never produce results, or they require an excessive amount of time and energy. For instance, Caminiti (2004) describes instances commonly seen in watershed modeling projects where “increasing demands for data inputs can result in the resource manager serving the model rather than the model serving the manager.” In other words, model development and data collection can be overemphasized to the extent that the DST is no longer useful to the stakeholders. By explicitly stating signs of success at the problem definition stage, the project expectations can be evaluated along the way and iterations/adaptations can be made to more clearly ensure success of the DST.

3.2.2. Addressing Uncertainty while Defining the Problem

Uncertainties exist within the first stage of the DST development conceptual model, and they can be recognized and addressed using the three Cs of uncertainty: characterize, calculate, and convey (Figure 2). Characterizing uncertainty in the problem formulation involves describing the extent to which we understand the representation of the problem and how well it matches with reality. It involves refining the problem based on agreed upon levels of uncertainty that are identified as a result of interactions among stakeholders and technical experts, where both local knowledge and technical expertise are required.

Figure 2.

The 3 Cs of uncertainty applied to problem definition within the conceptual model (i.e. Figure 1) to building an effective DST (TE = Technical Experts; S = Stakeholders).

For example, Schülter and Rüger (2007) describe a project in the Amudarya River Basin where stakeholder understanding of the ecological aspects of water allocation was limited—in particular, regarding uncertainties associated with various processes impacting water allocation (e.g., upstream water management, upstream land use, operational policies and agreement, climate change). The technical experts who developed the DST helped these stakeholders understand the relationship between different management options and the degree of ecological health in order to stimulate and inform discussions to establish ecological rehabilitation goals (Schlüter and Rüger, 2007).

Grouping problems into contexts can also be beneficial (e.g., upland processes, in-stream water quality processes, agricultural vs. urban problems) so that in the following step (i.e., method selection), the problem is already defined based upon how current DST tools using watershed models are constructed. For instance, Montgomery et al. (1995) demonstrated how spatially delineating the physical and biological processes within a watershed can aid in the design of land management scenarios to sustain ecosystems.

Calculating uncertainty during the problem definition includes defining uncertainty thresholds—that is, the acceptable levels of uncertainty for management or decision-making. Oftentimes, the technical experts can assist by providing scientific insights on how to define an acceptable range of uncertainty. For example, a study conducted by Maguire (2003) evaluated interactions between stakeholders and technical experts during a participatory modeling process to establish a TMDL for the Neuse River in North Carolina, USA. During this process, one stakeholder sought assurance that the chlorophyll standard would never be violated, which led to discussions to explain how absolute certainty in variable systems cannot be guaranteed. Model simulations for various scenarios were used to demonstrate variability in chlorophyll concentrations. These efforts helped stakeholders gain a better understanding of the physical system, enabling them to make an informed decision on acceptable levels of uncertainty (Maguire, 2003).

Communicating uncertainty surrounding the problem definition requires contributions from both technical experts and stakeholders. For the stakeholders, this may mean reaching beyond their immediate group to additional stakeholders in the watershed who may be affected by the decision-making outcomes (see, for example, Langsdale et al. (2013), Lubell et al. (2009)). This may involve presenting to communities or local groups, participating in public workshops, or developing and disseminating educational materials. Technical experts may conduct similar outreach to the science and modeling community as well as stakeholder groups. Outcomes from these activities may achieve greater certainty as to whether the problem is appropriately defined or requires revision.

3.3. Select the Approach: Data and Methods

The second stage of the conceptual model (Fig. 1) for co-developing a decision-support tool is to select the appropriate data and methods. We now emphasize how co-production techniques and uncertainty can be included within the conceptual model using watershed case studies from the literature.

3.3.1. Selecting Data and Methods using Co-Production

We will now highlight how previous watershed case studies have incorporated two-way interactions between stakeholders and technical experts when selecting and sharing data and methods. This step includes multiple phases in which (1) the stakeholders share local insights to data sources and types, (2) technical experts choose suitable data sources that match the management problem and data availability, and (3) the two groups communicate and co-produce the best methodological options to address their watershed management challenges.

Oftentimes, technical experts may reference publicly available sources that may provide applicable modeling methods. There are several such sources commonly maintained in the U.S. for and by technical experts. For example, existing models used for assessing TMDLs have been aggregated and presented by the U.S. EPA (https://www.epa.gov/exposure-assessment-models/tmdl-models-and-tools) as well as state environmental agencies (e.g., Oregon DEQ; http://www.oregon.gov/deq/wq/tmdls/Pages/TMDLs-Tools.aspx). However, stakeholders and local groups may have preferences for particular methods or tools based on previous experience. Clear tradeoffs exist between model complexity and the resources (e.g., human, financial, computational) needed to execute a series of scenarios related to particular water management challenge. This is one of the main drawbacks for widespread adoption of DSTs. However, while there are tradeoffs among resources, computational requirements, and complexity with the use of mechanistic models (Golden et al., 2017), they are arguably often the best tools available for guiding management decisions in ecological systems (Cuddington et al., 2013).

Depending on the watershed model used for the DST, either local, regional, or national input data may be most applicable. Data warehouses like the United States Geological Survey (USGS) Data Gateway (https://datagateway.nrcs.usda.gov) are especially useful for beginning discussions of currently available data. While the technical experts may be a more applicable resource for highlighting appropriate national and regional data sources, stakeholder groups and community members may be better able to spatially limit and characterize local data sources.

Many watershed modeling DSTs are publicly available and easily accessible in order to allow for potential feedback loops between technical experts and stakeholder groups. In this way, software errors or limited functionality can be noted by the stakeholders and communicated to the technical experts, and the technical experts, in turn, can fine-tune the DST for efficient use. While traditional software development methods rely on beta-testers—that is, a small group of users that interact directly with technical developers—DST case studies oftentimes expand the idea of beta-testers to include any stakeholders that use the DST, as will be shown in the following examples.

One example of a widely used and tested watershed-based DST is the Hydrologic and Water Quality System (HAWQS; https://epahawqs.tamu.edu), within which SWAT serves as the watershed model. The system requires users to register and use the framework online. As a result, user feedback is collected directly from the website, and simulation errors are forwarded directly to the technical staff. This creates an efficient feedback loop between SWAT stakeholders and technical experts.

Repository databases that host software, as well as comments on the implementation and/or bugs associated with the databases, have also been widely used to support co-production of DSTs using watershed models. Examples of these databases include SVN (https://subversion.apache.org), GitHub (https://github.com), and BitBucket (https://bitbucket.org). Furthermore, public and/or semi-private forums can help to collate diverse users, including both stakeholders and technical experts, from around the world using Sharepoint (https://products.office.com/en-us/sharepoint/collaboration) or Google Groups (https://groups.google.com). Some examples include U.S. EPA’s VELMA watershed model, which has a dedicated Sharepoint site to facilitate collaboration amongst many stakeholder groups (Table 1). Similarly, SWAT has an active ‘SWAT-user’ group where developers and users can coordinate and share resources. The Stormwater Management Model (SWMM) also has an online forum for questions called OPEN SWMM (https://www.openswmm.org), and the National Stormwater Calculator, which uses SWMM as its central model structure, has released a mobile web application that operates on both desktop and mobile devices, such as smartphones and tablets (https://www.epa.gov/water-research/national-stormwater-calculator).

Regardless of the mechanism or resource used, previous watershed modeling DSTs have benefited from co-productive collaboration between stakeholders and technical experts, which ensures that model issues are communicated from the stakeholders (i.e., users) to the technical experts. The technical experts then incorporate this feedback into subsequent versions of the DST to produce a more useful, robust (and therefore effective) DST by fine-tuning features, tools, and visualization.

3.3.2. Addressing Uncertainty in the Data and Methods

Addressing uncertainty related to data and methods (particularly model uncertainty) is one of the most difficult aspects of producing a DST. Yet, these uncertainties can be directly addressed using the 3 Cs and interactions between the technical experts and stakeholder groups (Figure 3). In this stage of the conceptual model, previous studies incorporate diverse groups of technical experts (e.g., field scientists, modelers) and stakeholders (e.g., land managers, homeowners) to select the appropriate data or method for the level of acceptable uncertainties that each group identifies.

Figure 3.

The 3 Cs of uncertainty applied to the second stage (i.e., approach selection) of the conceptual model (i.e., Figure 1) for creating an effective DST. TE = Technical Experts; S = Stakeholders.

Characterizing and calculating uncertainty of watershed models has been a topic of research for several decades. In terms of characterizing uncertainty related to watershed models, consensus in the literature suggests that there are four main types of uncertainty: (1) uncertainty in the forcing information (i.e., input data) used to drive the watershed model, (2) uncertainty associated with model parameters (e.g., soil moisture coefficients and routing parameters), (3) structural uncertainty associated with the process-based equations that are used to represent environmental phenomena, and (4) uncertainty in calibration, that is, the uncertainty associated with tuning particular calibration coefficients so that the model’s simulated output best matches historically observed data (Beven, 2010; Liu and Gupta, 2007; Pianosi and Wagener, 2016). Uncertainties can also occur when scaling localized implementations of management practices (e.g., stormwater management) to watersheds (Golden and Hoghooghi, 2017).

In this stage of the conceptual model, technical experts share their uncertainty expertise with stakeholders regarding input data, parameters, mechanistic formulas, and calibration coefficients. Stakeholders then evaluate which levels of uncertainty are acceptable amidst these constraints. For example, parametric sensitivity and uncertainty analyses have been performed to determine their impacts on overall model results and decision-support outcomes (Barnhart et al., 2017); however, these analyses require additional time and resources from the technical experts.

Equifinality, the notion that multiple unique parameter sets may derive similar modeling results, is pertinent to the calibration of watershed models (Beven, 2006) and are oftentimes explored during DST development. Multi-objective optimization algorithms may be useful for addressing equifinality and demonstrating its potential effects. For example, Ficklin and Barnhart (2014) have explored the impacts of model parameter uncertainty on hydroclimatic projections in snowmelt-dependent watersheds in the western U.S. using SWAT.

Furthermore, multiple mechanistic approaches have been used concurrently within a single watershed model to estimate model structure uncertainty and to verify that the necessary underlying processes are being accurately simulated. For example, Neitsch et al. (2012) tested both the curve number and Green-Ampt methods to simulate rainfall-runoff processes within the widely used SWAT watershed model. If multiple models are used (e.g., groundwater models or in-stream water quality models linked to watershed models), the technical experts typically estimate or quantify uncertainty for each model output.

While the majority of DSTs listed in Table 1 utilize a single watershed model, WMOST (Detenbeck et al., 2015) and SUSTAIN (US EPA, 2017) integrate watershed modeling with economic models to evaluate the cost-benefit trade-offs associated with decision making. In fact, a number of tools integrate economic models with watershed and water quality models to inform decision making and policy [e.g., InVEST (Tallis et al., 2010) or see Ward (2007), Watkiss et al. (2015), or Zoltay et al. (2010)]. While these linked models are efficient in assessing the effects of watershed management on economic outcomes, the approach also adds additional levels of uncertainty.

Communicating uncertainty is particularly important to maintain a transparent DST and provide a consistent rationale for the informed decision making using the DST. Previous studies stress that the technical experts need to clearly communicate to the stakeholders the assumptions and limitations of each approach used to calculate the uncertainty of the DST. In this stage of the conceptual model, stakeholder feedback will help refine which data, methods, or models are necessary for the decision-making process. Therefore, once uncertainty is communicated to the stakeholders by the technical experts, an iterative feedback loop may occur that requires obtaining additional data or modifying the technical approaches to further reduce uncertainty in the DST outcomes. If data limitations cannot be resolved, the problem goals may need to be iteratively refined, which may result in a simpler problem. Jakeman et al. (2006) emphasize that it is better to solve a simpler problem with a reasonable degree of certainty than to answer a more complex question unreliably. Yet, oversimplification of models remains a potential risk, since an overly simplified model may not reliably simulate all of the salient physical mechanisms.

3.4. Assess Results

The third stage of the conceptual model is to assess the results produced by the DST. This stage falls within the development process and represents a model validation stage before the tool is released and ultimately used to inform decision making. Refinement of the other stages can also occur through feedback at this stage. We now discuss how co-production and uncertainty have been addressed in this stage from the perspective of previous watershed modeling case studies.

3.4.1. Assessing DST Outcomes using Co-Production

At this stage of the conceptual model, stakeholders and technical experts work together to co-produce an assessment of the DST outputs. If the results from the DST are deemed invaluable or inaccurate by the stakeholders, the asset groups can collectively decide on new scenarios that should be considered (i.e., re-evaluate management options) or whether the tool itself needs to be refined to meet the management challenge needs. The stakeholders and technical experts can also co-produce alternative definitions of success by revisiting the first stage in the conceptual model (define the problem). Korfmacher (2001) emphasizes the difficulty of including stakeholders in the process of model confirmation or validation. As one example, the Chesapeake Bay Community Watershed Model was made publicly available, and public participation was encouraged in modeling subcommittee meetings. However, Korfmacher (2001) note that because the model is so complex, validation has only been performed by academics and researchers rather than non-scientific citizens. Another challenge with DST assessment is the required resources (e.g., time and money) for using the model. For example, Bagstad et al. (2013) assessed different aspects of 17 different ecoservices models and found that one common deficiency was the unacceptably high resource requirement to use the models effectively by stakeholders.

During and after assessing the results and outcomes of the DST, ideally, the technical experts and stakeholders will work together to prioritize what changes are needed (e.g., new scenarios, model refinement) and balance that with the available resources. For example, with limited resources, it is possible that additional scenarios may be run using the same watershed model for the DST; however, refining the DST may at this stage be too resource-intensive.

3.4.2. Addressing Uncertainty in the DST Outcomes

Thoroughly characterizing the uncertainty of the DST outcomes allows stakeholders to identify whether or not the current state of the DST, and the results produced from it, effectively meet their watershed management objectives. The 3 Cs of uncertainty (Figure 4) can help to systematically address this problem associated with the model outputs and outcomes.

Figure 4.

The 3 Cs of uncertainty applied to the third stage (i.e., DST assessment) of the conceptual model for creating an effective DST.

Characterizing uncertainty related with outcomes of the DST is primarily concerned with whether an effective decision can be made with the suite of DST outcomes. A full analysis characterizing uncertainty can provide the range of potential outputs (e.g., distributions of peak streamflow for each management scenario) and probabilities of whether an output will exceed a specified target or threshold (e.g., streamflow above a specified value). At this point, stakeholders determine whether the DST results are useful and to what extent the original goals of the project were met.

One way to assess uncertain outcomes is to quantitatively compare model results with metrics for success. McKane et al. (1997) present a method for calculating a single metric that summarizes the prediction error across 27 measured output objectives (including process and state variables) for a model analysis of the effects of climate change on an arctic ecosystem. This multi-objective metric provided a quantitative metric to determine how well the watershed model simulated each objective. Similar methods could be used to quantifiably gauge whether or not the uncertainty associated with particular simulations are acceptable to one or more groups.

While perhaps no DST will optimally fit every objective simultaneously, stakeholders oftentimes document these drawbacks and work with the technical community as well as other stakeholders to determine whether additional model simulations or refinements to the DST should be made. Alternatively, technical experts provide a mechanism through which the stakeholders can communicate the utility of the DST outcomes and what modifications are needed to arrive at an effective decision. This mechanism may be in the form of online user groups (as described in the section 3.3), regularly-secured in-person meetings or conference calls, or processes built into the DST for automatic feedback. Without such a feedback system, users are limited in communicating the success (or failures) of the DST and uncertainties associated with its simulated outcomes.

Matott et al. (2009) suggest that approaches to calculating uncertainty within watershed models used for the DST may fall within one of two categories: 1) sampling or 2) approximation uncertainties. Sampling uncertainties may include using statistical distributions of input data which are then translated to output distributions. Approximation uncertainties characterize a statistical moment, such as the mean or variance, of input data and then assess how those translate into model output uncertainty. For example, Harmel et al. (2014) consider measurement uncertainty and how they impact goodness-of-fits in hydrologic and water quality models. While mechanistic watershed models are deterministic and produce a single realization of model simulations (Farmer and Vogel, 2016)], uncertainty can be calculated using Monte Carlo and Bayesian methods that perform multiple instances of watershed models and vary various parameters, input data, or equations. For example, Zhang et al. (2009) used a Bayesian model averaging method to construct ensemble predictions and uncertainty interval estimations for hydrologic simulations using SWAT. Yang et al. (2008) also compared multiple uncertainty techniques while using SWAT and recommended the use of Bayesian methods for hydrologic uncertainty calculations in the Chaohe Basin in China as well as in Switzerland (Yang et al., 2007). Ultimately, the ability to produce multiple watershed scenarios as well as ensemble model simulations are useful to understand and calculate uncertainty.

Communicating uncertainty has been stressed in various watershed modeling case studies when validating model results (Caminiti, 2004; Hall et al., 2014; Smith et al., 2017). Conveying uncertainties typically includes a demonstration of the range of uncertainty in the model outputs for each scenario the DST simulates or assesses, and DSTs often provide a distribution of outcome probabilities for each of the selected decision-support scenarios (O’Hagan, 2012). For example, Abbaspour et al. (2015) built a SWAT model for all of Europe to support the European Water Framework Directive. They presented a protocol for calibrating large-scale models and incorporated multiple model simulations to include uncertainty intervals around predictions for various hydrologic components as well as crop yields and nitrate leaching into groundwater.

Conveying uncertainty is not a one-way process; instead, discussion and refinements help to improve the characterization of uncertainty co-produced by stakeholders and model users. These insights are then reintegrated into the modeling process to better characterize uncertainty. Using what was learned, the stakeholders and technical experts can revisit the problem formulation and acceptable levels of uncertainties.

Public resources have been widely used to document DST outcome uncertainties and how these may impact decision making. For example, the U.S. EPA has developed a green infrastructure modeling toolkit to support comprehensive, community-wide planning approaches to manage stormwater. The toolkit provides information on five EPA water models (including VELMA, SWMM, and WMOST), with a fact sheet and download link, and a video describing the best uses for each of the five models (https://www.epa.gov/water-research/green-infrastructure-modeling-toolkit). This ensures that users will not apply the DST out of context to misinform a particular problem for which the DST was not intended. Transparency about the advantages, limitations, and uncertainties of the DST outcomes will avoid misinterpretations and misuses of DST outputs. Further information regarding best practices of characterizing, calculating, and conveying uncertainty associated with model outcomes can be found in Morgan (2009) and Hall et al. (2014).

3.5. Take Action: Policy and Decision Making

The ultimate intent of DSTs is to remain informative for actionable decision making, including public policy. Inevitably, any form of decision making or policy action is bound to produce curiosity, skepticism, and scrutiny. One reason for this, as suggested by (Makri, 2017), is that the popular media’s portrayal of science can lead to misrepresentations of scientific results. Another may be due to limitations or the impossibility of model validation (Oreskes, 2004; Oreskes et al., 1994). Recognizing the need for systematic model validation, Rastetter (1996) provided four categories of tests that can be used to evaluate complex environmental models (and therefore DSTs which rely upon them). Ultimately, there is an important distinction between general model validation, which may not be fully possible, and explicit evaluations of uncertainties associated with watershed management, which can be directly addressed using the 3 Cs of uncertainty as described throughout this paper.

Nonetheless, in some cases, watershed models have been legally accepted for informing policy and decision making. For example, the Western Washington Hydrology Model (WWHM), which uses HSPF as its back-end, has been widely used in the State of Washington to establish and enforce stormwater policies. At the request of the Governor of the State of Washington, an independent science panel was convened to “evaluate, among other things, the scientific credibility of WWHM for use in Ecology’s Western Washington municipal stormwater permits” (Reiser, 2003), after which the panel endorsed the use of both HSPF and WWHM for continuous stormwater modeling.

In addition, the SWAT (and HAWQS, which is an online version of SWAT) watershed model has been used extensively as part of the USDA’s Conservation Effects Assessment Program (CEAP) to assess the environmental effects of various agricultural management practices and conservation policies in the United States (Gagnon et al., 2008; Maderik et al., 2006). The upland portion of SWAT (without flow routing) was also used as a back-end for a phosphorus risk index that was developed in response to a settlement from a Federal Court lawsuit between the City of Tulsa and a number of defendants, including Tyson Foods, Inc. (2003).

Regardless of the policy action or decision to be made, in addition to the initial step of gathering assets, transparency regarding the development of the DST is key. All documentation regarding the actionable goals that were developed and met throughout the project should be publicly available. This requires project planning documents to be widely available to DST users and to be written in ways that can be easily understood and correctly interpreted. Application of DSTs also may require attention to the legal framework around the management issue and specific legislation associated with the resource (e.g., Endangered Species Act, Clean Water Act, local zoning laws). A review of legal aspects regarding the use of environmental models is provided in McGarity and Wagner (2003).

After development is completed, DSTs may be applied by new stakeholders in new environmental contexts; therefore, it is important to document the limitations and restrictions of use of the DST during the development process. This allows for both potential end users and the general public to clearly understand the context in which the DST was produced and for which problems it can be applied to create informative results that can be used for decision making.

4. Conclusions

Decision-support tools (DSTs) are often produced from collaborations between technical experts and stakeholders to address management issues and environmental problems and inform decision making and policy. However, guidance is needed to produce effective DSTs and avoid miscommunication and disconnection between technical experts and stakeholders. This is particularly true in the context of watershed modeling DSTs and watershed management problems that span local, regional, and national scales.

In this paper, we began with a conceptual model for producing effective DSTs that utilize mechanistic watershed models. In each stage of the conceptual model, we utilized watershed modeling case studies to highlight successful outcomes and challenges associated with integrating the resources of stakeholders and technical experts to co-produce objectives, methods, and results. We also emphasized the use of the “3 Cs”—that is, Characterizing, Calculating, and Conveying—of uncertainty and highlighted their use in various case studies. We did not include other watershed (e.g., conceptual, statistical) or water quality models in our analyses or recommendations, which remains a limitation of the study.

We conclude by highlighting some best practices that may help to foster more effective DSTs, particularly in the area of watershed management.

1. Know Your Assets: Bring together stakeholders and technical experts, i.e., the “asset groups”, to discuss the problem before creating a DST. Also, understand that certain groups may be underrepresented or absent from discussions, and recognize this as a form of uncertainty; provide a mechanism for these groups to become active in the project.

2. Formulate decision-making targets using facts and expert opinions: Each asset group must convey their own objectives as well as uncertainties around these objectives. It is important to refine these objectives as a group to create feasible decision-making targets that can be addressed by the DST.

3. Select the methods for uncertainty analysis based on stakeholder knowledge and information gaps: Quantification of uncertainty is important during each step of the DST process to represent gaps in information and associated assumptions.

4. Allow sufficient time to discuss assumptions, needs, and limitations of approaches: Each asset group will be confined by funding and availability. Create time allocations for each project step to ensure that the project outcomes can be feasibly completed.

5. DSTs should provide options: Each asset group should be treated as equally important; therefore, DSTs should focus on presenting alternative trade-offs of watershed management decisions and allow the stakeholders to use the tool and ultimately make up their own minds.

6. Convey uncertainty of model outputs: The most important place to address uncertainty is in presenting watershed model results within DSTs. Clearly comprehensible graphs and depictions of uncertainty should be given to all model outputs. The definition of comprehensible will ultimately vary with the level of technical aptitude of the audience, and recent research (e.g., Wright et al. (2017)) may help to clarify this issue.

7. Provide convenient mechanisms for DST feedback, and provide feedback. Beta-testers should not be limited to technical experts; rather, DSTs should allow a user to submit feedback within the DST itself. Stakeholder groups should use the DSTs extensively and express their uncertainties openly.

8. Be Transparent. Document all steps of the development process when creating a DST; make these available on a publicly available website.

Successful DSTs for watershed management allow policy makers, communities, and individual stakeholders to make better informed decisions. We systematically describe and review the processes involved to develop these effective DSTs. Those who implement this conceptual model and our literature-derived best practices should document successes and recommend adjustments and iterations for further improvements. Also, future applications of co-production approaches associated with DSTs should identify “lessons learned” and new research directions in this area. Future research should additionally include the development of similar conceptual models for water quality models, as well as integrated modeling frameworks that link environmental and economic models. Through sustained cooperation and collaboration between stakeholders and technical experts, effective DSTs will serve to translate basic science into informed decision making.

6. Disclaimer

The views expressed in this manuscript are those of the authors and do not necessarily represent the views or policies of the U.S. Environmental Protection Agency. Mention of trade names or commercial products does not constitute endorsement or recommendation for use.