Toward a computational role for locus coeruleus/norepinephrine arousal systems

Abstract

Brain and behavior undergo measurable changes in their underlying state and neuromodulators are thought to contribute to these fluctuations. Why do we undergo such changes, and what function could the underlying neuromodulatory systems perform? Here we examine theoretical answers to these questions with respect to the locus coeruleus/norepinephrine system focusing on peripheral markers for arousal, such as pupil diameter, that are thought to provide a window into brain wide noradrenergic signaling. We explore a computational role for arousal systems in facilitating internal state transitions that facilitate credit assignment and promote accurate perceptions in non-stationary environments. We summarize recent work that supports this idea and highlight open questions as well as alternative views of how arousal affects cognition.

People and animals undergo dramatic fluctuations in the levels of arousal over time, with some periods marked by heightened alertness, and other periods marked by lower levels of sensitivity to external stimuli. In animals, these fluctuations are marked by altered brain states thought to be controlled via chemical neuromodulation. This raises questions as to the function of such fluctuations – are they simply flukes of an imperfect control system, or might they serve some greater normative purpose? In thinking about this question, it is useful to consider non-biological thinking machines. Recent success of deep learning and recurrent neural network techniques have driven computational neuroscience to think about the brain as a system of connected elements that jointly achieve impressive results on various information processing tasks^1,2. These networks have also served as analogies for natural intelligence have been used to examine a wide range of phenomena observed in both brain and behavior^3–5. These models do not, however, typically undergo the same sorts of state fluctuations observed in the brain. Thus, by comparing animal and artificial intelligences we might better understand the potential computational advantages, if any, afforded by the state fluctuations that are observed in human and animal brains and behavior.

Here we focus on the biological, cognitive, and behavioral consequences of fluctuating brain states related to neuromodulatory signaling. Unlike neural networks, human and animals brains undergo different states, characterized not only behaviorally by different responses to the same input, but also by different biological underpinnings^6–8. One particularly important dimension of these brain states is that of physiological arousal ⁹. Beyond dichotomous sleep wake states humans and animals undergo moment-to-moment fluctuations in their state of arousal that affect their overall perceptual sensitivity as well higher order cognitive processes^7,10. These fluctuations are related underlying changes in brain-wide neuromodulatory signaling, including the levels of norepinephrine (NE) released across the brain in response to activity in the pontine nucleus locus coeruleus (LC)^10–12. We examine the functional role of arousal fluctuations, and in particular the LC/NE system, in facilitating complex cognitive behavior. We focus on differences between problems faced in artificial intelligence, where neuromodulatory states have not yet emerged as useful solutions, and biological ones, where they appear to have done so.

While the goal of this perspective is to engage with questions about function, it would be difficult to do so without first providing a cursory description of the structure of the LC/NE system. The LC is located in the pons anterior to the fourth ventricle and supplies the majority of NE to the brain, including neocortex, allocortex, and thalamus, among other regions. Neurons in LC respond transiently to salient sensory events, irrespective of modality, as well as to goal directed actions ^13,14. In addition to evoked activity, LC neurons fire at a baseline rate around 1–5 hertz ¹⁴, but this rate is modulated by both internal and external factors that are described in more detail below ^10,13,15. Some work has emphasized an anticorrelation between baseline firing rates of LC neurons (or so-called “tonic” activity) and the magnitude of stimulus-evoked transient responses (or so-called “phasic” activity) ¹⁰, particularly when stress is manipulated¹⁶, however this relationship has not been observed in all studies. Neurons in the LC are electrically coupled, but recent work has suggested that there is important functional segregation among LC neurons related to where they project¹⁷. LC spikes result in release of norepinephrine in target regions, which acts on three primary receptor classes (α1, α2 and β receptors) to modulate local processing. Since these receptor subtypes have differential binding affinities (β receptors with the lowest), cellular and subcellular localizations (α2 are prominently expressed on LC neurons and act as autoreceptors), and g-protein coupled mechanisms (α1 = G_q, α2 = G_i, and β = G_s), the effects of NE depend on both NE signaling level and duration ¹⁸. In principle, this makes the distinction between tonic and phasic LC activity very important. However, an exact mapping from firing rate to downstream consequences is difficult for several reasons. For example, the results of NE release may differ depending on brain regions, due to differential expression of other channels that can by turned on or off by the intracellular-signaling cascades triggered by NE ¹⁹. The functional effects of LC signaling may also depend on local cortical activity, in that varicosities of NE projections from the LC contain glutamatergic receptors that could potentially amplify NE signaling when neighboring glutamatergic cells have been highly active, potentially creating a positive feedback loop that could push local levels of NE active pyramidal cells high enough to reach the threshold for G_s - coupled β receptors, facilitating a further increase in local activation¹⁸. In short, the LC-NE system is highly complex, and although we may treat it to be a one dimensional when interpreting our theories, measurements, and manipulations, it is worth keeping this complexity in mind as we try to synthesize experiments with the aim of understanding its potential functional role.

Making sense of a changing world.

The world we live in is dynamic and often undergoes abrupt fundamental changes. Successfully navigating this world requires rapidly adjusting policies in the face of change, for example, learning new masking and social distancing norms after the onset of the Covid-19 pandemic. People and animals are able to recognize and respond rapidly to changepoints in the environment thereby minimizing their reliance on action policies based on out-of-date information. In contrast, standard methods for training artificial intelligence require that training data come from the same distribution of possibilities as the test data. Furthermore, rather than training on one task context at a time, neural networks perform best when all possible contexts are interleaved so as to prevent catastrophic interference, a luxury not afforded to biological brains^20,21. In neural networks, training typically occurs through incremental adjustments of synaptic weights in a network based on prediction errors, which at least matches biological accounts of incremental learning in humans to a first approximation. However, the inability of standard neural networks to solve continual learning problems, particularly those that involve encountering completely new contexts, raises questions about how brains might solve these problems.

One early and influential theoretical account proposes that the brain’s ability to deal with abrupt environmental changes relies on neuromodulatory systems that signal the expected (Acetylcholine) and unexpected (Norepinephrine) sources of uncertainty ²². By this account, Acetylcholine would reflect the inherent uncertainty with which existing environmental cues predict behavioral outcomes within the current context, and norepinephrine (NE) signals the instantaneous probability that the context has changed ²². On the NE side, the theory built on rodent work showing the importance of noradrenergic signaling in effective learning after extradimensional shifts^23–25. Work testing correlative predictions of this model has identified fMRI activations in the basal forebrain, which contains cholinergic cell bodies that project throughout cortex, that correlate with high level prediction errors necessary to estimate expected uncertainty²⁶. On the NE side, LC has been challenging to image with fMRI due to its size and proximity to the fourth ventricle, although a growing number of studies have found success in recent years ^27–30. In addition to fMRI, work in monkeys and rodents suggests that rapid fluctuations in pupil diameter might serve as a proxy for LC activity ^11,31,32 and downstream noradrenergic signaling ³³, albeit a somewhat nonspecific one^33,34.

Pupillometric studies in humans suggest that arousal levels, and by proxy likely noradrenergic signaling, increase dramatically at expectation violations ³⁵. Pupil dilations that occur while processing a stream of sensory information occur at violations of an expected structure, but not to the emergence of new structure ³⁶. Such dilations occur even when participants are not consciously aware that the violation occurred ³⁷. Since such expectancy violations tend to occur at boundaries in temporal context ^38,39, in theory, the changes in arousal state occurring after them could be used to modulate ongoing processing in order to optimize behavior in the face of environmental dynamics. Some evidence for this comes from predictive inference tasks, where participants use new observations to update their predictions about upcoming ones. In such tasks, transient increases in pupil diameter occur at changepoints in the generative process and correlated with the degree to which predictions would be modified in response to a new observation ⁴⁰. In the same paradigm, task-irrelevant manipulations of the arousal system also affect behavior in a predictable manner, with startling sounds increasing the influence of new information on updated predictions, particularly when delivered during periods of low arousal ⁴⁰. Mapping these findings to underlying changes in LC signaling is certainly desirable, however there is some ambiguity about what these findings might mean in terms of tonic/phasic activity profiles. The peak of pupil dilations occurs 1–2 seconds after a surprising stimulus indicative of a changepoint, favoring direct effects on phasic LC activity, however, on the other hand, persisting effects in baseline diameter last several trials, during periods of uncertainty following a changepoint, more consistent with modulation of tonic LC ⁴⁰. Thus, in principle, these events observed through pupil dilation might involve both changes to tonic and phasic LC activation. In perceptual tasks that include changepoints in sensory context pupil dilations occur after context transitions and predict the degree to which participants will discount the weight of prior expectations in their perceptual reports⁴¹. In some perceptual tasks that lack explicit changepoint structure, but instead incentivize robust estimation, discrepant observations are less influential on behavior and tend to be accompanied by reduced pupil diameter ^42,43. A similar behavioral phenomenon, namely a reduction in choice history biases on trials with heightened arousal responses, occurs even in tasks without an explicit experimenter imposed temporal structure^28,44,45. Taken together, these results are consistent with the idea that arousal increases at boundaries in temporal context and affects both perceptual processing and predictive learning.

A second indirect line of evidence has emerged from EEG studies in humans and animals that link the event-related P300 signal to changes in temporal context and transient activation of the LC/NE system. The P300 shares a number of common antecedents with LC single unit activity ⁴⁶ and optogenetic activation of LC elicits an electrophysiological signature in rodents that greatly resembles the P300 in humans, albeit shifted somewhat earlier in time ⁴⁷. Thus, it is possible that the P300 provides another indirect index of LC activity. Consistent with this idea, P300 responses are larger for stimuli that are more surprising^48,49. Like pupil diameter, P300 locked to feedback in decision tasks predicts the degree of behavioral adjustment to that feedback ^50,51. However, recent work has shown that the relationship between P300 and behavior is highly contextual ⁵². When trial outcomes are generated through a process that undergoes occasional changepoints, larger P300 responses tend to precede larger behavioral adjustments^51,52. However, when trial outcomes are generated from a process that includes occasional outliers that should be ignored to optimize predictions, larger P300 responses tend to precede smaller behavioral adjustments, even after accounting for objective measures of outcome surprise ⁵². These findings, along with a parallel set of findings in pupillometry studies that show opposite relationships to behavioral learning in different structural contexts ^40,53, raise questions about the overarching idea that arousal provides a low dimensional controller for optimizing behavior – how can one signal adjust behavior differently in different contexts? Or in different people, since a pharmacology has also highlighted the inconsistent effects of NE reuptake inhibition on behavioral learning rate ⁵¹.

Dynamic representations for contextual learning.

One potential solution to this conundrum requires a slight reinterpretation of the theory. The original theory proposed by Yu and Dayan was derived from a generative model in which the world could change dramatically, providing a mechanism to recognize and respond to context changes, but the generative model lacked the possibility of “returning” to a previously experienced context. On its face, this seems like an important discrepancy from real world environments, where specific contexts (defined, for instance, by the mapping between cues and rewards) are often re-encountered (eg a student might spend the entire summer without encountering a classroom yet displays reasonable behavioral policies when their courses begin again in the fall). Considering this sort of generative structure naturally leads to thinking about continual learning, and, in particular, how people can learn something in one context, store it over long periods, and apply it later down the road in a similar context. This sort of structure, in which contexts can return, also has interesting consequences in terms of the representations required. This slight change to a generative model requires storing information associated with old contexts, rather than simply over-writing learning^38,54. This is interesting with respect to artificial intelligence, since neural networks suffer from catastrophic interference when trained to do continual learning problems, essentially overwriting a learned policy for an old task with weights learned for a new one⁵⁵. From a normative perspective, such context representations are useful if they effectively partition experience into different “latent states” that map onto alternate hidden causes that might be encountered in a complex environment ^56,57. The idea that the brain carves experience into latent states, which are also sometimes referred to ask “task sets”, is supported by behavioral and neuroimaging data in humans ^54,58–61 as well as recording and lesion data in animals ^62,63.

Dynamic control over latent state representations would provide a means for the brain to deal effectively with non-stationarity in the environment, but it also solves other problems, such as helping to avoid catastrophic interference⁶⁴. If learning is associated with an active latent state representation in the brain, then behavior can be modified rapidly in the face of a context change in the world by simply loading a new latent state representation. Loading a new latent state representation would mean that the impact of any associations bound to the previous latent state, including stimulus-action associations governing behavior, can be immediately eliminated³⁸. This idea can be implemented in a simple neural network to explain the dynamics of human learning behavior in many different statistical contexts ⁶⁵. In particular, when latent representations are updated according to a changepoint transition structure, surprising outcomes elicit persisting changes in the active latent state that allow the model to rapidly update beliefs. In contrast, when latent state representations are updated according to a transition structure that assumes that an “oddball” state might occur rarely and transiently, surprising outcomes elicit a brief transition to an oddball latent state that effectively stores learning about oddball events in a separate neural context, thereby preventing such events from interfering with learning done in typical trials. Thus, dynamic latent state transitions, in principle, provide a mechanism to increase flexibility in the face of change, but at the same time avoid interference from atypical events or those typical of an alternate context³⁸.

Implementing dynamic control over latent states requires two key components. First, it requires structural knowledge about the environment, which can be represented as a transition matrix defining the likelihood of transitioning from each latent state to each other latent state. Second, it requires a surprise signal to identify transitions in the environment and trigger neural state transitions to appropriately credit the new observation to the correct context. An interesting feature of this state transition control signal is that it should affect behavior in a context dependent manner⁶⁵. In particular, in changepoint environments where such state transitions tend to persist, promoting a state transition would yield increased learning. However, in oddball environments where state transitions are transient one-off events, identifying an observation as a state transition ensures that it will be stored in a neural context different from that used for past and future learning, thereby minimizing its impact on behavior. In this way, the state transition signal provides a potential computational role for feedback locked P300 signals that were bidirectionally and contextually linked to learning ⁵² as well as pupil signals that correlate positively with learning in changepoint environments ⁴⁰ but negatively in ones that include oddballs ⁵³. This raises a question as to whether noradrenergic signals might serve to partition incoming sensory data streams for them to be assigned to their appropriate latent state, thereby allowing the brain to gain advantages associated with pooling data within a given context but preventing interference by partitioning data across contexts.

Network reset for contextual learning.

While the exact computational formulation of this idea is new, the broader idea that LC-mediated increases in cortical NE might facilitate rapid reconfiguration of neural networks is an old and influential one ⁶⁶. The idea was originally motivated by state-dependent modulation of small invertebrate neural circuits via neuromodulation ⁶⁷ and is supported by a number of different lines of evidence. In particular, LC recordings show that tonic LC firing increases rapidly after a change in reward contingency, even before behavior is altered, presumably at the time when internal networks would need to be reconfigured in order to adaptively alter behavior for the new contingency¹⁵. Work showing that LC/NE activation can improve behavioral flexibility in the face of contingency changes, particularly extradimensional shifts ^23–25,68, hints at functional consequences of this signaling, in particular highlighting the possibility that LC/NE serves to promote updating of active cortical representations that might be linked to an obsolete behavioral policy. This could be achieved through a mechanism in which higher levels of NE in cortex weaken the effectiveness of recurrent excitation in layers 2/3 thereby making networks more sensitive to thalamocortical inputs ^19,69. In some sense, this idea is also compatible with work suggesting that LC/NE activation increases signal-to-noise ratio of representations in sensory cortex ⁷⁰, in particular by incorporating the notion that local recurrence itself in cortical networks may create background “noise” limiting the discriminability of sensory inputs⁶⁹. From a dynamical systems perspective, reducing the local connectivity might be thought to reduce the depth of attractor states, making cortical representations more likely to move from one stable state to another. In principle, by reducing the energy necessary to move cortical population activity out of one attractor state and into another, high levels of NE may serve to reset neural networks to load the latent state most relevant to the current situation. These ideas would also be broadly consistent with pharmacological studies in humans that have manipulated norepinephrine signaling in learning tasks and observed altered behavioral phenotypes^71,72, however the contextual nature of the mechanism ⁵¹has the potential to explain why such effects might differ across tasks, and indeed, individuals.

Destabilization of latent states might also have downstream consequences for perception. Sensory processing relies heavily on regularization, for example, to interpret a two-dimensional view in terms of the three dimensional layout that most likely gave rise to it. This regularization in turn leads to top-down biases in perception, but these biases are contextual, in that they depend critically on the inferred context. In principle, the same latent state representations that serves to carve experience into segments for learning may also be used to control which context is used to regularize perceptual processing. For example, in the case of bistable images, such as the Necker cube, we tend to perceive only one of two possible percepts, and do so stably for prolonged periods. Perceptual switches, at which individuals shift their perception from one to the other percept, are marked by transient pupil dilations ^71,72. Similarly, larger baseline pupil diameter predicts subsequent perceptual stability, with larger pupils predicting more perceptual switching ^72,73. Conceptually similar results have been observed in perceptual decision making tasks, where choice history biases promoting percepts of recently rewarded stimulus categories are reduced during periods of heightened arousal, as marked by dilated pupils^28,44,45.

Comparable results have been observed for more continuous percepts and further supported a normative role for arousal fluctuations in recognizing and responding to changes in latent state. For example, localizing the source of an auditory stimulus in predictable environments can be improved through Bayesian integration of sensory evidence with stored priors⁷⁴. People appear to perform this sort of integration, leading to measurable biases in localization reports towards source locations that were anticipated, but in dynamic environments this bias is substantially reduced at context transitions, when expectations are uninformative with respect to new incoming information ⁴¹. Such context transitions also elicited increased levels of arousal, as measured by pupil dilation. In accordance with these relationships, participants tended to have less perceptual bias on trials where pupil dilations were larger, but perhaps more surprisingly, this effect persisted even after controlling for context transitions themselves, highlighting the possibility that arousal could play an active role in the process⁴¹. These results provide a normative interpretation of arousal in controlling bias; when incoming sensory inputs are consistent with the expected context, arousal remains low and percepts are optimally biased towards expectations, but when contextual expectations are violated, arousal systems become highly active, triggering the need for a new latent state representation, thereby limiting the degree to which prior context representations bias interpretation of new incoming information.

Making cortical representations more labile could potentially have other implications for decision making. For example, recent work has suggested that the brain might make decisions by alternately exploring different decision states. Orbitofrontal neural populations, for example, seem to alternate between distinct states, each encoding the value of an alternative option ⁷⁵. One interesting possibility is that the same computational mechanism that facilitates state switching for more efficient learning in changing environments might also facilitate more alternations in decision states, thereby promoting exploratory behaviors. This idea would be consistent with another prominent theory about locus coeruleus function that hypothesizes that sustained elevation in NE levels promotes exploratory, as opposed to exploitative, behaviors (Aston-Jones & Cohen). This theory is also supported by pupillometry evidence suggesting increased exploration during periods of heightened arousal, although pharmacological tests of the causality of this relationship have been mixed ^76–78. It would also be consistent with recent work showing that pharmacologically enhancing NE signaling by inhibiting its reuptake leads to enhanced decision noise ⁷⁹, consistent with the sort of random exploration that has been theorized to depend on NE ¹⁰. Interestingly, this behavioral result was accompanied by increased connectivity during task performance measured with MEG, and both connectivity and behavioral effects of the pharmacological manipulation could be explained by a simple network model by increasing the excitation/inhibition (E/I) balance of the system, providing a tantalizingly simple way to think about the consequences of NE signaling in target regions ⁷⁹. This matches well with the longer timescale discussion above in terms of continual learning, since higher E/I networks might be more sensitive inputs, thereby enabling rapid switching from a network state maintaining information about an old context to a new network state containing information about a new one. More generally, while the jury is still out on whether rapid decision state switching might share common mechanisms with the longer timescale contextual switches that affect perception and learning, it is certainly an avenue worthy of future research.

Conclusion.

People exist in a dynamic world where they must learn contextual behaviors over time and prevent old learning from being overwritten by new learning. Normative models designed to solve such problems emphasize the need for dynamic latent state representations that serve as substrates for learning and action. Dynamic fluctuations in arousal, which are related to underlying neuromodulatory signaling including that of the LC/NE system, provide a strong candidate for a latent state transition signal. In principle, such a signal might energize latent state transitions in the brain by altering neural processing throughout the brain to drive normative changes in perception and cognition. While there is considerable support for this idea, there is need for more work to better evaluate it. This work will require interventional studies in humans capable of testing the causality of arousal-behavior relationships, ideally coupled with pupillometry and neuroimaging techniques (EEG/fMRI) to better understand how interventions affect biological signaling (phasic versus tonic LC signaling, for example). Beyond this, there is a need for work in animal behavioral paradigms that are sufficiently rich to narrow down the specific computational role played by the LC/NE system, thereby affording the advantages of more precise tools available in animals to measure and manipulate neural activity. Complementary work in animals and humans might sidestep limitations in each, and pave the way to linking dynamic modulation of arousal and its effects on complex human behavior to its underlying pharmacology and systems neuroscience.