Timescales of Cognition in the Brain

Abstract

We live in a world that changes on many timescales. To learn and make decisions appropriately, the human brain has evolved to integrate various types of information, such as sensory evidence and reward feedback, on multiple timescales. This is reflected in cortical hierarchies of timescales consisting of heterogeneous neuronal activities and expression of genes related to neurotransmitters critical for learning. We review the recent findings on how timescales of sensory and reward integration are affected by the temporal properties of sensory and reward signals in the environment. Despite existing evidence linking behavioral and neuronal timescales, future studies must examine how neural computations at multiple timescales are adjusted and combined to influence behavior flexibly.

Introduction

Animals adapt their physiological responses according to the changes in its environment. Environmental changes occur on many timescales, ranging from milliseconds to years, and mechanisms to accomplish this adaptability vary greatly across species. These adaptive mechanisms are shaped by many factors, including the animal’s lifespan, the range and precision of sensors that detect the changes in the environment, behavioral repertoire, and the computational machinery available to identify and select the most desirable response [1]. For many animals, the last element is implemented in the brain, and thus, the spatial and temporal organizations of the brain reflect the evolutionary history of adjustments to changes in the environment.

Learning appropriate behavioral responses is not trivial, because the animal’s environment is always stochastic and a particular behavioral response in a certain environment seldom leads to the same outcome [2]. In rare cases, the probabilities of different outcomes for each response might be fixed, which is referred to as expected uncertainty, and this can simplify and even hard-wire certain learning algorithms in the brain through evolution. This still leaves the main challenge for the animal to properly weigh reward probabilities relative to other reward attributes based on the animal’s current physiological state. By contrast, in a non-stationary environment in which outcome probabilities are unknown, which is referred to as unexpected uncertainty, the animals need to adjust their learning and behavioral strategies [2]. Furthermore, sensory signals provide the information about the animal’s environment only probabilistically. Thus, the timescales for integrating sensory signals can also vary substantially. More importantly, different factors that are crucial for some behaviors, such as the traffic laws and road conditions for driving, must be learned over different timescales.

This review provides an overview of behavioral and neural adaptations on multiple timescales. We consider how animals adjust their behaviors according to timescales of changes in the environment, and how these adjustments rely on integration of relevant information over multiple timescales in the brain. We also examine how hierarchy and heterogeneity of intrinsic timescales in neural response and gene expression throughout the brain could support such adaptive behavior. We conclude with remaining questions about the timescales of brain and behavior and how they can be studied.

Timescales of brain and behavior

Behavioral timescales might be matched to the timescales of important changes in the animal’s environment using several different mechanisms (Figure 1). For example, information about regularities in the environment might be stored in synaptic connections between neurons [3]. Therefore, different types of changes in the synaptic weights across multiple timescales, such as short-term and long-term plasticity, might allow flexible behavioral adjustments to environmental changes [3]. In addition, neuronal activity often displays multiple concurrent timescales that might support behavioral changes at different timescales, although the range of experimentally measured timescales of neural activity is relatively small compared to the full range of possible behavioral timescales. For example, neurons in the primate prefrontal cortex display activity related to multiple timescales of reward integration in that neuronal activity modulated by a reward outcome decays at different rates across neurons [4]. Moreover, these neuronal timescales are correlated with the behavioral timescales for integration of reward feedback during decision making [5, 6]. Concurrent integration of reward feedback on multiple timescales is also manifest in the activity of neurons in the of lateral habenula and dopamine neurons in substantia nigra pars compacta [7], serotonergic neurons [8], and hemodynamic signals in the human anterior cingulate cortex [9].

Figure 1.

Timescales of environmental changes, synaptic plasticity, and neural response. (A) Changes in the environment across multiple timescales. (B) Different forms of synaptic plasticity at different timescales. (C) Different timescales of spiking activity in an example cortical neuron.

Timescale of behavioral changes associated with reward integration is often estimated using the learning rates of reinforcement learning models fit to choices. A single timescale of reward integration is parsimonious and computationally convenient, but often relies on the assumption that the learning rate can be optimized in a given environment. By contrast, multiple timescales of reward integration can be inferred from a better fit of choice behavior by models that incorporate multiple time averages of reward outcomes [10, 11, 12], reward-dependent modulations of value representation [13], or average reward prediction error [14]. Several well-known behavioral observations, such as spontaneous recovery and motor memory, also suggest that multiple memory traces with different timescales might be widespread in the brain [15].

Previous studies also demonstrated a substantial level of heterogeneity in the observed timescales within and across brain areas, suggesting that learning might proceed in parallel at multiple timescales [6]. This implies that the values of different options or actions would vary according to the timescales of different learning algorithms, and therefore requires mediation to match timescales of neural circuits to those of the environment for guiding behavior, similar to the mechanisms proposed for the arbitration between model-based vs. model free reinforcement learning algorithms [16]. It remains an open question how the brain adapts to changing environmental timescales through adaptation of neuronal timescales or selection of an appropriate timescale.

Volatility and reward integration

Reward may not arrive due to the probabilistic nature of reward outcome or actual changes in the environment, but these two scenarios require very different responses from the animal, namely, no update or faster update, respectively. Actual changes in the environment could happen with different frequency or rate, often quantified as volatility [2,17,18]. In hierarchical Bayesian models, volatility can be equated with a parameter to measure the width of distribution for transition probability between different values of reward probability [17, 19]. Experimentally, volatility can be controlled in various ways, but has been mainly manipulated by changing the block length in probabilistic reversal learning tasks [2,17,20]. Although some studies have reported higher learning rates in a volatile environment [17, 21], other studies in monkeys [20] and humans [22] have not observed similar changes in learning rates. The reasons for this discrepancy should be investigated further.

Volatility and uncertainty can modulate learning and choice through means other than a change in the learning rate [12, 22, 23]. For example, a modeling study showed that reward-dependent metaplasticity can allow continuous adjustments in learning without an overall change in the learning rates [23]. In this model, reward integration is performed by transitions between states on multiple timescales, allowing the model to incorporate the history of reward feedback and thus volatility. In addition, more detailed examinations of learning and choice behavior in monkeys and humans have revealed that uncertainty results in fundamental changes in valuation and choice strategies, instead of a change in the overall timescale of reward integration [22]. These results suggest that integration of reward feedback might not happen on a single timescale adjusted by volatility. Instead, reward integration might happen on multiple timescales across many brain areas.

Adaptability-precision tradeoff

Regardless of the complexity of the learning mechanism, there is always a tradeoff between how fast and how accurate new information can be acquired. This adaptability-precision tradeoff has important implications for timescales of reward integration. On the one hand, increasing the timescale of reward integration or, equivalently, reducing the learning rate can improve the precision in estimating the value of a given action or option, but also results in less adaptability (Figure 2A). On the other hand, shorter timescales of reward integration can improve adaptability but at the cost of precision.

Figure 2.

Adaptability-precision tradeoff in the integration of reward feedback. (A) Different timescales of reward integration allow different speeds for updating estimated values (adaptability) and different levels of accuracy for those estimates (precision). (B) Learning about multi-dimensional stimuli/options could be achieved by strategies with different levels of adaptability and precisions. (C) Accuracy in estimated values over time using feature-based or object-based learning in two environments with different levels of generalizability. With more generalizability, it takes longer for the object-based strategy to surpass the level of precision achieved by the feature-based strategy.

It has been suggested that specific structures of metaplasticity [24] or the addition of a surprise-detection system [12] can partially mitigate the adaptability-precision tradeoff. However, neither mechanism can overcome the adaptability-precision tradeoff completely. In general, adaptability or precision must be prioritized at different points relative to a change in the environment. Therefore, the agent should choose adaptability or precision depending on internal or external factors such as hunger or threat, rather than trying to optimize a single learning rate. Integration of reward feedback on multiple timescales could allow additional flexibility in managing the adaptability-precision tradeoff because the brain could adjust its priority on different timescales at different time points.

In addition to integration over multiple timescales, uncertainty requires that reward feedback should be integrated based on different models of the environment. This becomes more important in the real world where stimuli and objects have numerous features or attributes, making it difficult to determine what reliably predicts reward outcomes. For example, one can learn reward values of individual features and combine this information to estimate values associated with each option [25, 26]. Such feature-based learning reduces precision, since individual features often do not predict reward consistently across many stimuli. Nevertheless, this strategy allows much faster learning because values of all features of the chosen option/stimulus can be updated after each feedback (Figure 2B, 2C). Indeed, recent studies showed that learning strategy depends on volatility, generalizability, and dimensionality of the environment [26, 27]. These findings suggests that timescales and strategies for integrating reward feedback are adjusted according to properties of the environment, and that this adjustment depends on how attention is deployed among many features or attributes of a choice option.

Perceptual decision making

Similar to reward learning, perceptual decision is commonly postulated to rely on the integration of sensory signals over time. Although earlier models assumed perfect integration and hence an infinitely long timescale, perceptual decision making in dynamic environments requires the timescale of evidence integration to be flexibly adjusted, not only by reflecting the timescale of the change in sensory signals itself [28], but also by exploiting multiple sources of information other than the sensory signal.

The timescale of evidence integration can be estimated by examining how the weight of evidence on decision varies with the time lag between evidence and decision (Fig. 3A). For time-varying stimuli, the evidence presented close to the decision (late evidence) tended to have stronger influence compared to earlier evidence, consistent with leaky, rather than perfect, integration (Figure 3B) [29,30,31,32]. Although leaky integration can limit the accuracy of decisions by using partial evidence, this enables flexible and strategic modulation of the integration process. For example, the onset of integration can be delayed to make it aligned with the timing of relevant evidence by considering internal processing delays and the temporal structure of dynamic stimulus [31, 33]. Computationally, this can be achieved by time-varying urgency [31]. In addition, the integration gain can be dynamically modulated by the temporal statistics of the evidence, signal duration, and task demands [34,35,36]. Although available descriptive models can account for sensory integration and its interaction with other choice processes such as urgency [31], it remains unknown whether timescales associated with these processes can be distinguished reliably.

Figure 3.

Behavioral and neural timescales during perceptual decision making. (A) Temporal changes in the weight of sensory evidence that reflect different timescales of evidence integration. (B) Recency effect in decision weight may reflect imperfect integration in humans [30]. (C) Heterogeneous intrinsic timescales in rodent dorsal cortex during evidence integration [39].

Despite behavioral evidence for flexible evidence integration, neural mechanisms for adjusting the timescale of neural integration remains poorly understood. Nevertheless, similar to the timescales of reward memory signals observed in the primate cortex, integration of sensory evidence might be performed across multiple timescales in parallel. For example, heterogeneous timescales for evidence integration were reported within and across cortical regions [37,38]. A recent study has also found a hierarchy of intrinsic timescales in the rodent dorsal cortex during a perceptual decision-making task (Figure 3C) [39].

Hierarchy of neuronal timescales

The possibility that the timescales of cognitive computations might be adjusted strategically according to statistics of the environment and task demands raises the question of how a diversity of timescales might be generated in the brain. One possibility is that different brain regions exhibit distinct characteristic timescales in their intrinsic activity, which in turn can shape the functional specialization of regions in terms of reward and sensory integration. In support of this possibility, a growing literature has found that timescales of neuronal spiking activity, related to intrinsic dynamics and cognition, vary across cortical areas. For instance, the intrinsic timescale of spiking fluctuations increases across the cortical hierarchy in the macaque brain, from faster in sensory areas to slower in association areas [4,6]. A similar hierarchical pattern of increasing timescales from sensory to association areas was also found in electrocorticographic recordings in monkeys (Figure 4A, 4B) [40]. These findings are also in line with timescale hierarchies in human cortex measured by various methods including electrocorticography [40,41], magnetoencephalography [42], and functional MRI [43].

Figure 4.

Hierarchical variation in intrinsic timescales across primate cortex. (A) Intrinsic neuronal timescales in spike-train recordings from multiple regions of the macaque cortex, from sensory to association areas [4]. (B) Intrinsic timescales of electrocorticographic recordings in the monkey cortex follow the same hierarchy as those of spiking activity [40]. (C) A spiking-neuron model of an association cortical circuit that contains sub-populations of pyramidal neurons with strong recurrent excitatory connections, parameterized by w₊ [45]. (D) and (E) Intrinsic timescales of spiking fluctuations in model pyramidal neurons (D) and strength of persistent activity related to working memory (E) increases with the strength of recurrent structure.

Regional specialization in intrinsic timescales can arise from regional differences in the strength of recurrent connectivity within cortical circuits, which in turn can support functional specialization for cognitive processes such as working memory and decision making [44]. This circuit mechanism can be demonstrated in a canonical association cortical model that performs working memory and decision-making functions (Figure 4C) [45]. As recurrent structure is strengthened, intrinsic timescales of spiking fluctuations increases as seen in empirical data (Figure 4D). Strength of recurrent structure also controls the working-memory computations in the circuit, as it transitions the circuit among different regimes of persistent activity (Figure 4E). This spiking circuit model provides a mechanistic hypothesis for variation in intrinsic timescales. Different intrinsic timescales reflect differences in local cortical microcircuitry, leading to functional differences across areas, such as their capacity for persistent activity.

In line with this theoretical proposal, studies have characterized intrinsic timescales at the single-neuron level, and found that longer timescales are associated with neurons that support persistent delay activity in working memory tasks [46, 47]. Interestingly, during a decision-making task, single-neuron intrinsic timescales were not correlated with timescales of memory related to rewards or actions [6]. Thus, neuronal and circuit mechanisms for dissociation of intrinsic and functional timescales remain open questions.

Another important neural circuit property that varies across the cortical hierarchy is the relative amount of intracortical myelination in that sensory areas are more highly myelinated than association areas. In structural MRI, T1w/T2w map follows the cortical topography of intracortical myelination [48]. This measure also correlates with cortical hierarchical levels in monkeys and with the dominant topographical pattern of transcriptomic variation in humans that reflects regional specialization of cellular and synaptic processes [49]. One intriguing possibility is that this hierarchical gradient in myelination may contribute to a hierarchical gradient in synaptic plasticity for learning. High myelination in sensory cortical areas means less axonal area available for structural plasticity to form new synaptic connections, whereas the relatively unmyelinated axons in association areas can more readily form new synaptic connections in support of learning and memory.

Conclusion

The diversity of timescales related to cognitive functions across cortical areas raises the question of what neural mechanisms may contribute to these patterns and how signals generated by these diverse timescales are combined to determine choice behavior. Areal specialization for timescales of learning is potentially shaped by gradients in the densities of receptors for neuromodulators thought to play a role in learning and decision making, including dopamine and serotonin [49]. Linking neuronal and regional differences in synaptic and neuromodulatory physiology to functional differences in timescales related to cognitive processes is crucial for understanding neural mechanisms underlying the generation of diverse timescales [50].

Recent findings on the independence of timescales and selectivity to task-relevant signals within individual neurons in a given area suggest that multiple mechanisms must underlie the generation of these timescales [6]. If representations of learning and choice behavior are distributed, independence among timescales in different signals can generate less correlated signals that allow higher dimensional representations and easier decoding of signals from relevant ensembles of neurons. Nonetheless, how such distributed representations can be adjusted according to the reliability of different signals remains unknown.

Future experiments that examine learning behavior under conditions with different timescales of changes in reward environment can be used to address whether behavioral adjustments emerge from adaptation of single or multiple timescales to the environment, or from different arbitrations between systems with different timescales. This requires the development of new methods for estimation of multiple timescales in behavior to accompany multiple existing methods for estimation of neural timescales.