Bridging adaptive management and reinforcement learning for more robust decisions

Abstract

From out-competing grandmasters in chess to informing high-stakes healthcare decisions, emerging methods from artificial intelligence are increasingly capable of making complex and strategic decisions in diverse, high-dimensional and uncertain situations. But can these methods help us devise robust strategies for managing environmental systems under great uncertainty? Here we explore how reinforcement learning (RL), a subfield of artificial intelligence, approaches decision problems through a lens similar to adaptive environmental management: learning through experience to gradually improve decisions with updated knowledge. We review where RL holds promise for improving evidence-informed adaptive management decisions even when classical optimization methods are intractable and discuss technical and social issues that arise when applying RL to adaptive management problems in the environmental domain. Our synthesis suggests that environmental management and computer science can learn from one another about the practices, promises and perils of experience-based decision-making.

This article is part of the theme issue ‘Detecting and attributing the causes of biodiversity change: needs, gaps and solutions’.

1. Introduction

Given the urgency of environmental crises and the impending risk of crossing planetary tipping points, developing management strategies in the face of uncertainty is increasingly critical. Decision-making under uncertainty, central to most contemporary environmental policy and practice, is referenced in contexts that span scales and systems, ranging from multilateral initiatives to halt biodiversity loss [1] and mitigate climate change [2] to local measures for habitat protection [3] and water allocation [4]. But uncertainty, and the possibility of triggering regime shifts, persist as challenges to effective conservation decision-making [5], motivating conservation science, policy and practice to focus not only on making decisions given uncertainty but iteratively reducing uncertainty through adaptive decision-making.

Since the coining of the term adaptive management in the 1970s [6], attempts to apply the paradigm of ‘learning while doing’ have proliferated across the environmental domain [7,8]. But the theoretical underpinnings of adaptive management—reducing uncertainty around a discrete set of autonomous models (systems that do not explicitly depend on independent variables) or model parameter values over time to take actions that maximize a notion of expected utility—have proven to be difficult in application [9,10], buckling under the problem complexity and sociopolitical nuance within which real environmental decisions are made [11,12]. Notably, the decision-theoretic methods commonly used to solve adaptive management problems (e.g. Bayesian model updating and dynamic programming) generally assume that uncertainties not only are known but can be precisely quantified in probabilities [13]. This, in turn, hinges on the assumption that models of environmental systems are identifiable and autonomous, which is often not true, particularly in the case of systems with tipping points or in the context of rapid environmental changes due to anthropogenic pressures [14].

In the light of the limitations of decision-theoretic methods, multiple alternative approaches have emerged to inform the management of complex systems under uncertainty (e.g. scenario planning and resilience thinking [5]) For example, statistical early warning signals of tipping points (e.g. critical slowing down) allow us to punt on the issue of model misspecification. Notably, early warning signals, scenario planning, and other ‘resilience thinking’ approaches all shift away from quantifying a decision policy (e.g. a model that suggests a fishing quota of X metric tons) and towards classifying (e.g. ‘the system is/is not approaching a tipping point’, or ‘scenario B is preferable to scenario C’). Hereafter, we will call these ‘classification’ approaches. Despite long-standing calls to better integrate decision theory and classification approaches [13], classification has continued to retreat from the decision problem all together, focusing instead on advanced computational tools—like deep learning—to better predict complex systems dynamics while avoiding the need to explicitly consider actions or the expected utility of a given strategy. For example, Bury et al. [15] use machine learning to classify critical transitions into four possible classes but do not suggest action strategies given a belief state in the system dynamics.

At first glance, focusing on system classification is compelling: if we can identify a proximate tipping point or predict a threshold response in a system—regardless of the exact underlying model (e.g. [15])—surely a manager can leverage that knowledge to compose a good management strategy? But is washing our hands of devising quantitative decision strategies for complex systems really a good idea? Heuristic approaches to decision-making notably fail to design effective sequential decision strategies (in contrast to iterative or static decision strategies) compared with formal approaches [16]. This leads us to ask if the limitations of a heuristic decision are better than those faced by traditional model-based decision theory [17].

If we can predict thresholds more successfully without models, can we also derive quantitative decision strategies for those systems more effectively without models (and the assumptions of model-based decision-theoretic approaches)? Within the field of computer science, a rapidly emerging class of machine learning algorithms has proven remarkably effective at making complex sequential decisions without first learning a model of the system [18]. Interestingly, these model-free reinforcement learning (RL) algorithms often mimic what any good manager does in the same situation: forgoing learning over an explicit set of process-based models and relying instead on knowledge from successes and failures experienced by repeatedly making decisions. But unlike a human manager, RL algorithms can process near real-time and high-dimensional data as well as learn strategies by interacting with a wide diversity of simulations and scenarios [19] that would be infeasible for a human to process. Moreover, RL algorithms show promise to more effectively manage systems with unknown tipping points than do fixed ‘rule-of-thumb’ strategies [20].

In this paper, we first highlight the limitations and assumptions of decision-theoretic approaches that have driven a wedge between the theory and practice of adaptive environmental management. We then explore examples of successful RL applications, highlighting which specific RL methods and concepts might provide a promising path forward to overcome some limitations of model-based adaptive management (table 1). Despite the promise of these emerging methods, the problem of making strategic decisions in non-identifiable and/or non-autonomous (systems subjected to external inputs) systems is not easily solved by even the best algorithms. We simultaneously suggest that RL research might benefit from the insights gleaned from trying to tackle pressing environmental problems (table 1). Finally, we emphasize that while RL approaches may, at last, allow algorithms to help human decision-makers better grapple with real-world complexities, such a transition will raise new challenges for equity, governance and accountability.

Table 1.

Emerging methods and approaches in reinforcement learning (RL) have the potential to help overcome a myriad of challenges in adaptive management, from making effective decisions in complex environments over long planning horizons to learning strategies in high-stakes situations. The following methods help define a research agenda for RL researchers seeking to contribute to environmental management. MDP, Markov decision process.

adaptive management challenge	RL methods and concepts	citation(s)
dealing with uncertainty
Data used to train RL agents may not be precise or generalize to real-world settings.	Robust RL aims to learn policies that perform well across a large class of possible environments, including environments that may not have been explicitly encountered during training.	[21–23]
When data are insufficient to reliably predict the outcome of an action, decision-makers wish to understand the degree of uncertainty.	Uncertainty quantification measures and attempts to reduce uncertainty in predictive systems. These uncertainty estimates can be used to constrain the RL policy to avoid taking actions with high uncertainty in outcome, related to safe RL.	[24–26]
A given model does not precisely describe the true dynamics of the ecological system, regardless of what values are used to instantiate the parameters.	Model misspecification refers to RL settings in which the Markov decision process used to model the environment does not describe reality. Model misspecification is typically addressed with robust RL, including model-free approaches.	[27–29]
For a given set of observed data, multiple sets of parameters used to instantiate the model may have the same probability distribution of being the best-fit model.	Non-identifiability describes the challenge of learning and planning in the presence of unmeasured state variables (confounders).	[30,31]
The reward function may be unknown in advance.	Inverse RL attempts to recover the reward function given an optimal policy and environment dynamics.	[32,33]
limited opportunities to interact with real-world, high-stakes settings
Must learn optimal policies using historical data, without collecting new data. This challenge arises when data collection is expensive—as is often the case in conservation management.	Offline RL (or batch RL) learns the best policy possible given historical observations (a static dataset) without exploration. Requires off-policy evaluation to estimate the performance of policies that were never enacted in the historical data.	[34]
In high-stakes settings, managers wish to be risk-averse to avoid potentially catastrophic settings, such as unintentionally wiping out the population of one species.	RL safety trains an RL policy by limiting explorative actions to those that are unlikely to reach very bad states, for example by imposing additional constraints or avoiding actions with high uncertainty.	[35,36]
Decision-makers wish to know why a policy calls for a specific action.	Explainability in RL aims to provide human-understandable explanations for why an RL method recommends a specific action, in contrast to ‘black box’ methods.	[37,38]
Decision-makers want to have human experts oversee and possibly override an RL agent's decisions.	Human-in-the-loop systems treat humans as experts and defer to these human experts to make decisions when the RL agent is highly uncertain.	[39,40]
complex environments with long planning horizons
The underlying rules of the environment may be changing over time, owing to exogenous factors such as climate change or socioeconomic drivers.	Non-stationarity refers to changes over time in the underlying dynamics of the MDP. In dynamic settings, the state constantly changes, but with non-stationarity the transition dynamics and rewards shift as well.	[41,42]
Management decisions may be highly complex, thus difficult to learn from scratch.	Curriculum learning trains an agent on progressively harder tasks, using transfer learning to build off knowledge learned from previous tasks in subsequent tasks.	[43,44]
Management decisions may require planning challenging multi-step tasks over long time horizons.	Hierarchical RL decomposes long-horizon tasks into more tractable subtasks.	[45]
After we receive a reward, we want to know which action(s) were critical to that outcome. This challenge is exasperated by delayed feedback.	Credit assignment evaluates the utility of individual actions over a long sequence of steps.	[46–48]
Rewards may be sparse, especially with delayed feedback, and the benefit of intermediate actions may not be immediately obvious.	Reward shaping provides more gradual, localized feedback to guide the policy toward high-reward states.	[49–51]
multiple stakeholders, multiple agents
Multiple stakeholders may each have their own objective.	Multi-objective RL learns optimal decisions in the face of multiple conflicting objectives. This challenge is most salient when the relative weighting (importance) of the objectives is not known.	[52,53]
Multiple agents may be acting in an environment simultaneously.	Multi-agent RL trains agents to act in the presence of other agents. In a cooperative setting, these agents share a goal but in a competitive setting, these agents have non-aligned goals that may be in conflict.	[54,55]

2. Adaptive management: translating theory to practice

Given the limitations of model-based approaches to adaptive management (box 1), the practice of adaptive management—and its integration into policy—has deviated from formal definitions of decision theory (figure 1). Rather, implemented adaptive management strategies often take ‘rule-of-thumb’ approaches, substituting human judgment and the experience of decision-makers in place of computationally intensive models [7,61]. While more loosely defined and human-informed adaptive management processes overcome some of the technical shortcomings of model-based adaptive management (e.g. by allowing consideration of greater system complexity and the social context of decisions), these approaches also face several limitations. Beyond the myriad of potential non-technical risks, such as goal slippage, manager turnover and sensitivity to political asymmetries and institutional change [62], ‘rules of thumb’ face technical constraints. For example, the exponentially increasing amount of monitoring data from automated sensors [63] quickly becomes impractical for humans to integrate into a heuristic decision process.

Box 1. Examples of model-based adaptive management and their key challenges.

(A) adaptive flow regimes: Tallapoosa River
For nearly a decade, the United States Geological Survey (USGS) implemented adaptive management on the Tallapoosa River to determine optimal flows under multiple competing stakeholder objectives. The iterative decision model included annual monitoring of ecosystem indicators to vary flow regimes optimally. Despite the adaptive decision framework, the best management strategies to provide both adequate hydrological and thermal habitats while sufficing recreational values remain a central controversy in the system [56].	objective: ensure the conservation of at-risk species and meet ecosystem service objectives
	learning: passive learning of ecosystem dynamics and the valuation of ecosystem services
	states: discrete set of ecosystem indicators, ecosystem values
	actions: four flow allocation strategies
	key challenges faced: multiple competing objectives, noisy observations, delayed feedbacks, stochasticity, high dimensionality
(B) adaptive harvest management: waterfowl
Since 1995, the United States Fish and Wildlife Service has used an adaptive management framework to regulate duck harvests. Harvest quotas rely on an iterative cycle of monitoring, assessment and decision-making. Based on monitoring data, managers continually refine models of the relationships between hunting regulation, harvests and waterfowl abundance. While significant updates to the model weights have occurred over the past 20 years, non-stationarity due to global change challenges the current methodology [57].	objectives: sustainable harvest of waterfowl
	learning: passive learning of population dynamics in response to harvest regimes
	states: waterfowl abundance
	actions: annual harvest quotas
	key challenges faced: non-stationarity, delayed feedbacks, stochasticity, multiple objectives
(C) adaptive harvest management: red knots (Calidris canutus rufas) and horseshoe crab​​s (Limulidae polyphemus)
Following increased harvest of horseshoe crabs in the Delaware Bay during the 1990s, migratory shorebird populations declined steeply. Recognizing this decline, the fisheries commission began regulating the horseshoe crab harvest. Proposed adaptive management frameworks focus on two competing models of red knot population dynamics and horseshoe crab harvests, seeking to iteratively improve harvest policies for both objectives [58].	objective: ensure the conservation of at-risk species while also meeting harvest objectives
	learning: passive learning of multi-species dynamics
	states: red knot abundance and fecundity, horseshoe crab abundance
	actions: harvest quota for horseshoe crabs
	key challenges faced: model set limitations, dimensionality, delayed feedbacks, multiple objectives

Figure 1.

(a) Decision-theoretic approaches to adaptive management—what we refer to as ‘model-based’ adaptive management—formulate problems as Markov decision processes (MDPs) with unknown state transition probabilities (for a helpful overview of MDPs for ecology, see [16,59]). At each time step t, the agent takes an action, on an environment, and the environment transitions from the current state s, to the next state s_t+1 and provides the agent with a reward r. After taking an action, the agent observes the system state and rewards to update its belief of the underlying dynamics of that system to inform the decision strategy (policy) that selects the next action. (b) While theory provides a formalized way to ‘learn’ about systems while acting on them, in practice, adaptive management decisions are usually made by humans who may or may not use the output of models to support decisions but often rely on experience with the complex systems and sociopolitical context not captured by stylized system models. (c) Like classical model-based optimization approaches, the task for a reinforcement agent is to learn a ‘policy’ that maps the state of the system to an action the agent should take to maximize the expected sum of future rewards [60]. This can be done by interacting with historical data or simulators. However, the process by which reinforcement learning (RL) learns optimal policies is fundamentally different from (a). (d) Unlike the classical methods used in model-based adaptive management (AM), RL allows action strategies to be developed without learning over a predetermined set of potential models. (Online version in colour.)

Thorough reviews of the application [7,8], theory [64] and legal implications [65] of adaptive management already exist in the literature. In figure 1a,b, we summarize the adaptive management paradigm. It should be noted that while adaptive management is not appropriate in all settings, the benefits of leveraging adaptive over non-adaptive approaches are particularly apparent in contexts where there is deep uncertainty about system dynamics—a setting in which RL might hold the most promise. In the remainder of this section, we illustrate, through examples, the key gaps between adaptive management as it is expressed in theory and the requirements for its effective implementation in practice. While decision-makers' deviations from scientific and/or algorithmic recommendations and definitions of adaptive management (figure 1b) are often viewed with scepticism, we explore how this deviation frequently reflects the inability of models integrated into decision-theoretic frameworks to sufficiently account for real-world complexity and challenges (box 1).

Take, for example, the Tallapoosa River, where an upstream hydropower dam had uncertain effects on the integrity of the river's species-rich ecological community, prompting a decade-long adaptive management effort [9,56] (box 1A). Monitoring how multiple ecological indicators and stakeholder preferences respond to flow alterations was used to iteratively update a model (a state–space transition model) of the system, which was integrated into a decision-theoretic framework to optimize flow regimes [56]. While the river monitoring effort held promise to instruct more responsive and informed decisions, the high-dimensional system (which included multiple species and ecosystem indicators, recreational interests, dam revenue, seasonal temperature) and nearly unlimited potential flow regime strategies was reduced to three system ‘states’ and four potential flow regime ‘actions'‘ over which the formal adaptive management problem learned and optimized [56]. Would a more realistic representation of the systems, at least mirroring the scope of monitoring data collected, and a more extensive set of possible actions have allowed the solutions to capture nuanced and temporal tradeoffs between human use and ecological demands in this system?

Unfortunately, what adaptive management models add in social and ecological complexity they often lose in the number of models and parameters over which they learn owing to computational constraints. For example, in the case of adaptively managing horseshoe crab harvests in the Delaware Bay (box 1C), proposed frameworks included multi-species population models and a broader set of potential actions but considered only two competing models over which to reduce uncertainty and derive decision strategies [58]. Beyond the challenges of capturing complexity and uncertainty of system states and actions, model-based adaptive management frameworks face limitations to inform decision-making in the context of global environmental change [64]. For example, in the case of adaptively setting quotas for waterfowl harvest in the USA (box 1B), the increasingly non-autonomous (non-stationary) nature of the system (owing to climate change not being included as part of the population model) is beginning to limit the applicability of the model-based decision-theoretic methodology to devising decision strategies [57]. Of course, climate dynamics could be integrated into the models. However, this further complicates both the model and the computational requirements to solve for optimal decision strategies using dynamic programming, as well as introduces additional uncertainty.

In each of the cases in box 1, human decision-makers wrestle with a growing number of complexities, competing needs and pressures left unanswered by the existing decision-theoretic adaptive management paradigms. These mismatches limit the utility of existing model-based optimization approaches to the decision-making process and risk unintended consequences of relying on best-fit models in decision processes. Importantly, proposals to leverage adaptive management for higher-dimensional environmental problems, such as climate mitigation [60] and protected area design and management [66,67], are likely only to widen the gap between model-based theory and the realities of management decisions in practice. While classification approaches enable skirting the decision problem altogether, less constrained approximate methods for finding optimal strategies have also been suggested for dealing with high-dimensional natural resource management problems for decades [68]. In the following sections, we explore how emerging methods from deep RL might allow us to leverage the benefits of both heuristic and computational approaches to adaptive management, improving our capacity to manage systems under uncertainty (table 1 and figure 1c).

3. Reinforcement learning as model-free adaptive management

RL has proven better than humans at making strategic, adaptive and complex decisions across a diversity of problems and domains. While RL's most cited feats are in the context of games (e.g. chess) [43] and robotic tasks [69], RL algorithms are increasingly used to solve planning problems across a variety of noisy and uncertain real-world settings, from healthcare [70] and energy systems [71] to biological systems [72] and economic policy [73].

The problem set-up for RL closely mirrors model-based adaptive management (figure 1). Like classical model-based optimization approaches [16], an RL agent aims to learn a ‘policy’ (decision strategy) that maps the state of the system to the best action to take to maximize the expected sum of future rewards [20]. However, the process by which RL learns optimal policies can be fundamentally different. Unlike classical dynamic programming methods leveraged in adaptive management examples from box 1, which require specifying state transition matrices (state–space models of the system over which to learn) [16] (box 1, figure 1a,d), model-free RL allows action strategies to be developed without a predetermined model (figure 1c), bypassing the need to iteratively learn a ‘best model’ of system dynamics altogether (figure 1d). Importantly, RL learns action strategies through experience, which can include simulated experience, experience derived from historical data, and/or real-world experience.

First, RL can learn from simulated experience. A notable example is Atari, where researchers achieved human-level performance across dozens of Atari video games by training RL agents over millions of time steps (decisions) [19] (box 2A). Instead of fine-tuning a new model for each of the games, a single RL agent using the same neural network architecture and hyperparameters was applied to 49 different Atari games, reaching performance comparable to a professional human game player across a majority of the games. Without the goal of learning a single system transition matrix, the RL agent learns generic concepts that allow strategic decision-making in a diversity of settings. Beyond gameplay, simulators can also model physical environments such as atmospheric wind conditions, as used to build a simulator to train an RL agent to navigate high-pressure balloons in the stratosphere (box 2B). In the context of adaptive management, simulators could be designed to describe water flow, species population dynamics or a changing climate. Leveraging simulated experience to learn policies for adaptive management problems could provide a means to integrate more complex system dynamics and action sets, such as in non-autonomous environments.

Box 2.

Examples of reinforcement learning applications and the key challenges they address.

(A) playing strategic games: Atari
Dealing with high-dimensional inputs to make effective decisions across different tasks and situations remains a key challenge for RL. Atari consists of 49 distinct video games with visual inputs and has become a go-to benchmark for developing RL algorithms. Using deep Q-learning, a single RL agent was trained across dozens of games to outcompete human players [19].	key challenges addressed: high-dimensional input, diverse tasks, long horizon
	objective: maximize score across a set of Atari games
	state: 84 × 84 × 4 colour video frames at 60 Hz
	action: discrete, variable for each game
	reward: rescaled game scores
	RL approach: deep Q-learning
(B) flight control: stratospheric balloons
Stratospheric balloons are high-altitude balloons that can reside 15–60 km.a.s.l. for months at a time. These balloons carry up to 1.1 tons of payload, typically tools used for weather forecasting, satellite navigation, atmospheric chemistry experiments and testing new space technology. Navigation in these high-altitude settings is dependent on stratospheric winds, of which relevant meteorological data are sparse, and solar availability, which is needed to charge the battery. Model-free reinforcement learning enabled effective navigation over the Pacific Ocean over 39 days, using distributed Q-learning [74].	key challenges addressed: incomplete data, noisy observations, unreliable solar availability, safe navigation, long planning horizon
	objective: navigate a super-pressure balloon to float near weather station
	state: 1083 wind variables. Sixteen ambient variables
	action: discrete (Ascend, descend, stay)
	reward: distance from a weather station, with maximum reward within 50 km of the station
	RL approach: Model-free Q-learning with experience replay
(C) clinical decision-making: sepsis treatment
Sepsis is a life-threatening excessive immune response to an infection that may lead to organ failure or death. Treating sepsis involves a complex mix of antibiotics, corticosteroids, timing and dosage of drugs, and intravenous fluids. The treatment regiment for patients in the intensive care unit (ICU) must be customized to each individual patient in response to that patient's response to medical interventions. Deep Q-learning helped learn treatment policies to help reduce patient mortality by 1.8–3.7% [75].	key challenges addressed: continuous state, sparse reward signals, stochasticity, delayed feedback, interpretability
	objective: improve patient survival
	state: continuous. Forty-eight values of demographics, physiological data and vital signs)
	action: discrete. 5 × 5 intervention options with different amounts of IV fluid and vasopressor dosage
	reward: weighted sum of indicators of patient health including the extent of organ failure and changes in blood pressure
	RL approach: dueling double-deep deep-Q-learning

While RL allows us to relax assumptions required by most model-based adaptive management frameworks (e.g. MDPs), managing non-autonomous systems remains a hard problem for both computer and environmental science. We do not suggest that RL will readily overcome this challenge, but rather that in complex socio-ecological settings (e.g. the Tallapoosa River, box 1A), RL might outperform both human heuristics and model-based adaptive management methods through exploring a set of complex simulated environments millions of times, accruing orders of magnitude more feedback than a system that could only interact with the real environment [76].

Notably, simulated experience is not the only way RL can learn decision strategies. Deep RL has been shown to learn effective policies from historical data in the absence of a simulator by stitching together trajectories of system observations (offline RL) [34,77]. For example, in hospitals—which often have decades of historical records about patient status, treatment and outcomes—this offline approach was taken to learn individualized treatments for sepsis patients (box 2C). Of course, the extensive data available in some societal domains like healthcare is less common in environment systems. In some environmental challenges, such as water and air quality monitoring, sensors are already constantly taking samples potentially allowing for the tracking of the impact of specific actions. While this level of monitoring data are not available for many other ecological management problems, improved sensing of everything from vegetation dynamics to species occurrence [78] are trending the field in that direction and likely making RL-based management more feasible.

A key challenge in the environmental domain will be off-policy evaluation, to estimate the performance of policies that were never observed in the offline data [79]. A large body of techniques for off-policy evaluation has been developed for RL for observational healthcare [80]. Additional observations collected through real-world (online) experience may then be used to improve the policy further or ‘adaptively’ update policies while taking actions.

Emerging methods in RL have the potential to address more than just the issue of dimensionality and non-autonomous dynamics in natural resource management problems. From making effective decisions in environments with sparse rewards to addressing systems with multiple competing objectives, in table 1, we map adaptive management challenges to RL methods that might help address them.

4. Possibilities and pitfalls of applying reinforcement learning to adaptive management problems

If real-world complexity has forced the practice of adaptive management beyond the reach of theory, emerging paradigms of RL appear to at last be putting such challenges within reach of algorithms (table 1). But just because this may be possible, is it really a good idea? Adapting RL for adaptive management could open possibilities [20,76] but also introduces new pitfalls while re-surfacing age-old concerns of algorithmic decision processes [81]. We divide these possibilities and perils into three themes. First, the conceptual shift of an RL approach to learning as something based on heuristics and experience rather than rigorous mathematical theorems. This can recapitulate some of the benefits but also the shortcomings of human decision-making. Second, RL still faces the same challenge of any objective-based decision-making: accurately defining the task at hand. RL overcomes some computational constraints but still requires defining a scope of possible rewards, states and actions. We refer to this as world-making. Third, deep RL's technical and computational needs may limit its application to the largest technology institutions with access to these resources. This stakeholder shift exacerbates potential ethical and political consequences. Here we outline both the technical and social components of these opportunities and challenges across the three main themes of (1) learning, (2) world-making, and (3) shifting stakeholders (figure 2). We hope both the RL and adaptive management communities recognize and focus on addressing these challenges when developing and implementing environmental decision-making.

Figure 2.

Traditional adaptive management relies on modeling the environment using Markov decision processes, which mirrors a ‘model-based’ approach to RL (green pathway, left). Model-free RL (grey pathway, center) eschews learning an intermediate model to instead directly estimate the reward for taking specific actions at a given state. As we outline, reinforcement learning brings both promise and new challenges for adaptive management for learning, world-making and shifting stakeholders, which all impact different components of the RL pipeline. (Online version in colour.)

(a) . Learning

The central difference between model-free deep RL and the theory of adaptive management regards learning. In adaptive management, learning is defined as reducing uncertainty over parameters or candidate predictive models¹ of the underlying ecological processes (figure 1a). Learning is expressed in terms of quantitatively precise probability distributions and realized through mathematically precise theorems such as Bayes' rule to dictate how model ‘beliefs’ (probability distributions) narrow in response to actions and new information (figure 1a). By contrast, a human manager does not necessarily need a predictive model of the process to adjust a decision (figure 1b). It is possible to propose a policy without a model based on experience alone. For example, if the estimated waterfowl population (box 1B) decreased too much last year compared to the year before, it's probably a good idea to lower the harvest quota this year. Of course, the theory may give the same answer with more quantitative precision—how much to lower the quota (and also just how much to re-adjust ‘belief’ probabilities towards some more pessimistic growth rates of the species)—but that answer is only as good as the models it considers. The manager's experience may factor in variables ignored by the models—a harvest quota of zero may be socially or politically unacceptable, while past experience of ups and downs may provide an experienced manager with a notion for the right size of adjustment with nary an equation [82,83].

Model-free RL capitalizes on this experience-driven approach to decision-making. The RL agent does not need to predict future states; it decides what action to take given only experience from past states and resulting rewards (or costs). The RL researcher places an untrained agent in a novel environment (usually a computer simulation, e.g. in the Atari example (box 2A)), in which the agent takes exploratory actions while adjusting its policy to improve long-term reward. From repeated simulations over hundreds of thousands of episodes, the agent will extensively explore the space of actions and outcomes. The process is sometimes compared to a newborn first exploring the world around them. Like the newborn, this RL agent is not entirely naive—the researcher must select among a myriad of specific algorithms each with very different approaches to solving the RL problem. The researcher, just like the parent of a newborn, may present a modified system of rewards and costs to coax along the desired learning more efficiently in a process called reward shaping (table 1). Reward shaping becomes particularly useful when there is a big payoff only after a long sequence of actions (e.g. rewarding distance to the end of a maze rather than only completion).

When trained in a single environment, the strategies that RL agents learn rarely generalize to even small deviations from past experience. The agent will often overfit to the smallest details—the units of measurement, the duration of the particular episode. More generally applicable strategies can be found by presenting the agent with a wide variety of environments. For example, one pervasive challenge in learning from simulated experience is the ‘sim2real’ gap, the difference between an RL agent's performance in a simulated environment versus a real environment [84]. Robust RL techniques may help close the sim2real gap and avoid overfitting [85] (table 1). A wealth of emerging approaches seek to improve generalizability. Curriculum learning algorithms seek to provide the most efficient way to interleave different environments (table 1). In adversarial learning, a second agent seeks to learn and propose alterations to the environment that are most likely to fool the focal agent into poor performance.

Most large-scale ecological simulation systems still fall short of capturing the many processes involved [76] but are already beyond the reach of dynamic programming methods of classical adaptive management. Collectively, advances in the ecological realism of simulations and computational RL methods make it feasible to train intelligent agents across a wide variety of simulated environments. Historical observations can only be paired with historical actions, and thus never provide an agent much insight into the outcomes of the actions not taken but can nevertheless be used to supplement and ground-truth training based on simulation (table 1). A more fraught question concerns the role of RL-based learning in real-world contexts. Like the distinction between active and passive adaptive management, RL is typically divided between ‘training’ and ‘evaluation’ loops. In the training loop, the RL agent explores their action space to discover and adjust their decision strategy (active learning). In the evaluation loop, acting is essentially passive, with the RL agent seeking to maximize expected utility without updating their decision strategy. Evaluation need not always be passive in RL (especially in ‘low-stakes’ real-world scenarios, such as a physical robot learning to walk or handle objects) but mirrors the general preference of managers to rely on passive adaptive management in high-stakes scenarios.

(b) . World-making

While RL might allow quantitative adaptive management to consider more realistic state and action spaces, reducing the numerical constraints on problems only refracts the issue of distilling a complex environment: how do we bound an environmental state, define management objectives and determine a set of available actions while ensuring these represent environmental realities and values of those most impacted by the decisions? How do we create sufficient simulations or decide on the appropriate data streams to train algorithms with? RL might expand the scope and range of problems that we can solve, but it does not remove the sociopolitical considerations inherent to how those problems are defined.

Let us imagine reformulating the waterfowl harvest problem as an RL problem. We could simply create a simulation of the states and their responses to actions in alignment with the current model-based formalization (box 1B): the action space is an annual harvest quota to maximize expected long-term yields and the state space is a one-dimensional representation of waterfowl stock. But given fewer constraints, we might represent the waterfowl stock as part of a larger ecological (or socio-ecological) system responding to human land use, climatic shifts, and weather extremes, overcoming shortcomings of current methods, like the consideration of non-autonomous systems. Even if the action space remained the same (harvest quota), the algorithm's optimal policy would change. The action space could similarly be expanded to better capture the possibility of decisions (e.g. to a temporally and spatially dynamic closure rather than a single harvest quota), changing not only the policy and its impact on resource users, but the underlying system trajectory. These changes seem benign, if not beneficial, but it is easy to envision how the imagination and values of the algorithm designer impact not only the conceptualization of the environment (state spaces), but the solutions derived, and actions taken, which ultimately feedback to create the reality of that system [81]. Leaving us to question what parts of the system are included in the simulations and how that might shift the distribution of benefits and costs.

Barriers to the adoption of adaptive management strategies arise not only from a lack of realism in system formalization or capacity to deal with complexity, but also from disagreement over whose values are represented in the decision objectives and the potential risks of following algorithmic suggestions [86]. RL does not sidestep these issues, but methods such as multi-objective RL (table 1) can learn optimal decisions in the face of multiple conflicting objectives, and inverse RL can help align the values as they are represented in formalized rewards with real-world values (value alignment) [32,33] (table 1). Additionally, reward shaping (table 1) can help to ensure RL agents do not myopically take actions that lead to short-term gain over long-term benefits.

Even in light of technical methods to improve the alignment of values and capture multiple objectives, the issues of world-making are social and normative at their core. Specifying state, action and rewards in RL applications will necessarily reflect both epistemic values and contextual values of the developer [81,87]. Which begs the question: who has the power and capacity to define problems and develop RL algorithms?

(c) . Shifting stakeholders

Given both the technical expertise and computational requirements needed to train RL algorithms, industry (specifically big tech) involvement in the development and deployment of these methods is commonplace across environmental application domains[88]. The shift from government-maintained and managed algorithms—as is currently the case in most environmental adaptive management contexts (such as waterfowl adaptive management; box 1B)—to industry-maintained algorithms would create a new set of actors in the environmental regulatory processes. Because political and financial concerns may influence the design of RL environments and agents, developing transparent and inclusive participatory processes will be critical to ethical and equitable development and application of RL to adaptive management problems.

Beyond shifting power and creating new environmental actors, RL-derived environmental decisions risk undermining trust in environmental governance systems by increasing the ambiguity of who is accountable for future environmental degradation [81]. If a decision is derived from an algorithm that relies on trial and error rather than clearly mapping to a model choice, are poor outcomes anyone's fault? Moreover, if an RL agent continues to learn and adapt while interacting with the system (adaptive management, by definition), how do we ensure that its policies are meaningfully overseen [87]? In this way, RL differs from more transparent model-based methods in the relative lack of capacity to query solutions, potentially obscuring biases and compliance with regulations [89]. Lessons from RL applications to other safety-critical domains, such as nuclear fusion [90], and tools from the explainable AI subfield [91,92], might help mitigate these issues (table 1). However, problems of explainability and safety become even more pronounced when RL is proposed for controlling less identifiable and high-dimensional systems, as is the case in many environmental management contexts.

Ethical AI principles provide some guidance to procedure and practice to ensure safe application of algorithms. But these guidelines, like the algorithms themselves, are primarily developed in the Global North, notably missing perspectives from Central and South America, Africa and Central Asia [93]. Moreover, ethical guidelines rarely address the many dimensions of power implicated in world-making; not only the power to make decisions or define objectives, but power to set the agendas (e.g. defining objectives, state and action spaces) and shift ideologies [81]. Applying decolonial theories to AI application and development, as discussed in [94], might help address the shortcomings of AI ethics and recentre the importance of power and representation in procedural and development processes.

While the technical synergies and differences between RL and model-based adaptive management methods are outlined throughout this paper, simultaneously considering the parallels between AI ethics [93], science technology studies [95] and political ecology [96] are critical when considering applications of RL to safety-critical real-world domains like environmental management.

5. Conclusion

To bridge the gap between the science and practice of adaptive management there is a need for decision-centred methods that capture the complexity and uncertainty of ecosystems. Advances in deep learning have positioned RL as a promising approach to solve sequential problems under uncertainty, while sidestepping the need to define a set of candidate models or effectively refine our belief in those models. Here we highlight recent advances in RL methods that overcome several limitations—such as high-dimensional spaces, imperfect models and lack of accurate simulators—that have prevented adaptive management from moving beyond theory in complex situations. Simultaneously we underline key priorities for RL—such as robustness, safety and multi-objective rewards—to enable its effective and responsible deployment for ecological decision-making.

Data accessibility

This article has no additional data.

Authors' contributions

M.C. and L.X.: conceptualization, investigation, resources, writing—original draft, writing—review and editing; M.L.: conceptualization, resources, writing—review and editing; C.B.: conceptualization, resources, writing—original draft, writing—review and editing.

All authors gave final approval for publication and agreed to be held accountable for the work performed herein.

Conflict of interest declaration

We declare we have no competing interests.

Funding

This material is based upon work supported by the National Science Foundation (grant no. DBI-1942280). L.X. was supported by a Google PhD Fellowship.