From exploration to exploitation: A shifting mental mode in late life development

Abstract

Changes in cognition, affect and brain function combine to promote a shift in the nature of mentation in older adulthood, favoring exploitation of prior knowledge over exploratory search as the starting point for thought and action. Age-related exploitation biases result from the accumulation of prior knowledge, reduced cognitive control and a shift towards affective goals, and parallel changes to brain networks, attention, and reward circuits. By incorporating these factors into a unified account, the exploration to exploitation shift offers an integrative model of cognitive, affective and brain aging. We review evidence for this model, identify determinants and consequences, and survey the challenges and opportunities posed by an exploitation-biased mental mode in later life.

Exploration versus exploitation: At the crossroads of human mentation

Every volitional act is the product of a choice to either exploit prior knowledge towards a more predictable outcome, or to explore new sources of information with less certain returns. This tension between exploitation and exploration is pervasive in everyday life. Do we return to the same vacation rental again this summer, or take that overseas vacation? Do we watch our favorite cable news outlet or seek an alternative perspective? Do we rely on stereotypes or actively engage with unknown others? This tension is observed across numerous biological, ecological and artificial intelligence systems[1]. In humans, the exploration (see Glossary) versus exploitation trade-off has been extensively studied in young adults[2,3–20]. Yet there is growing evidence that the determinants and criteria governing decisions to explore versus exploit are altered over the adult lifespan, with direct implications for mentation and real-world functioning in later life[21–32].

In this review, we integrate decision science with cognitive and affective aging theories and aging neuroscience research to propose a shifting mental mode hypothesis of aging, anchored in the idea that exploration-exploitation trade-offs are increasingly biased towards exploitation over exploration in older adulthood. This hypothesis is grounded in the recognition that the architecture of human cognition undergoes a striking transition over the course of adult development[33]. Fluid control processes, necessary for fast and flexible engagement of cognitive resources, undergo a linear decline with age. Concurrently, prior knowledge of oneself and the world continues to accumulate[34]. We argue that these dual trajectories of cognitive aging favor an exploitative mental mode, one in which the accumulating weight of prior knowledge representations strengthens the prepotency of exploitation over exploration-based choice behaviors. In addition to cognitive factors, affective shifts that occur with age also produce an exploitation bias in older adults (Box 1; Table 1). We have deliberately chosen the label ‘mental mode’ here to recognize both the cognitive and affective changes that occur with age, which impact older adult mental functioning. With advancing age, decision-making is increasingly biased towards reliance on more positively-valenced information and prior knowledge, resulting in more frequent expression of exploitation-biased behaviors, or a modal shift, from exploration to exploitation-based mentation.

BOX 1: Integration of cognitive and affective factors increase age-related exploitation biases.

In addition to cognitive factors, affective drives and motivations have also been implicated in exploitation-biased mentation in later life [25, 29, 53]. Socioemotional Selectivity Theory [SST, 52, 54] posits that older individuals reweight their goal structures to prioritize emotional satisfaction and meaning. A core prediction of SST is the emergence of exploitation-bias in older adulthood. Indeed, the genesis of SST was the observation that the informational value of forming or fostering new relationships (exploration) declines across the lifespan as experiences accumulate. In contrast, the emotional value of closeness with familiar others (exploitation) increases with age [94]. Thus, SST provides one of earliest theoretical accounts of an age-related exploitation-bias, one grounded in affective changes. This raises important questions about the relative contributions of affective and cognitive factors, and how they interact to produce an exploitation-bias in older adulthood.

We suggest that these two factors can be readily woven together under our shifting mental mode hypothesis. Behavioral research supporting SST [53, 54] converges around the central idea that with age emotional goals are increasingly (chronically) activated and require less cognitive control to be maintained in working memory [95], favoring an exploitative mental mode. With repeated activation, emotional goal hierarchies become consolidated into existing prior knowledge structures or schemas [76]. Schemas, or distillations of prior knowledge, comprise association patterns abstracted from individual events. Critically, they integrate both cognitive (e.g., spatial, perceptual, social associations) and affective elements of past experiences [76]. We suggest that chronically activated schemas (i.e., accumulated prior knowledge), incorporating both affective meaning and cognitive context, bias mentation towards exploitation and away from exploration in older adulthood (Figure 1).

There is early evidence for a neural mechanism supporting such affective-cognition interactions. Medial prefrontal cortex (mPFC) has been associated with schema formation [75, 76] and is a core node implicated in the affective (motivation) circuit [60] with indirect influences on cognitive (attention) circuits through LC connectivity [36]. Both of these circuits underpin the exploration- trade-off in our model (Figure 1). Further, the mPFC, becomes increasingly connected to salience network regions including the anterior insula and anterior cingulate cortex [80, 89, 90, 92], as well as subcortical structures such as the amygdala [96], all regions implicated in exploitation-based search [36]. We suggest that this connectivity profile positions mPFC as a central processing hub, integrating cognitive and affective information into schema [76, 97, 98]. These chronically activated schema, in turn, anchor ongoing mentation in prior knowledge, leading to an exploitation bias in older adulthood.

Table 1.

Biasing mechanisms and predictors of an exploitative mental mode in older adulthood

Biasing mechanism [relevant references]	Predictors of exploitation bias in older adulthood	Relevant references
Declines in cognitive control	Working memory Information compression; integration of fewer decision-options Mental flexibility Reduced capacity to shift search strategies based on changing response contingencies Inhibition Inability to overcome a prepotent exploitative mental mode Future planning and prospection Reduced prospective foraging and mental simulations necessary for directed exploration Learning Poor updating based on novel information and feedback; miscalibration of uncertainty	[13, 18, 26, 29, 36, 47, 104–109]
Gains in prior knowledge	Predictable rewards Increased certainty and reward value for known over unknown choice options Processing efficiency (lower control demands) Schema Heuristics Gist-based knowledge Greater associative processing Anchoring current mentation to prior knowledge stores (schemas)	[10, 25, 26, 39, 43, 50, 110–113]
Greater motivation towards affective goals (life meaning)	Positivity bias Positively-valenced and less detailed (gist-based) recollections promote known over unknown choices. Narrowing time horizons Reduced reward value of unknown information Greater reward value of social intimacy Increased risk of novelty-seeking (exploration) Positive reward expectations Reduced uncertainty of choice outcomes Increased propensity to seek known rewards	[21, 31, 54, 60, 77, 78, 79, 114, 115]

This model (Figure 1) distills the two broad arcs of cognitive aging, declines in fluid processes and gains in prior knowledge, as well as shifts towards affective goals, into a criterion shift, favoring exploitation-based over exploration-driven mentation and action[11,12]. Further, this prepotency for exploitation over exploration in older adulthood is underpinned by specific patterns of age-related brain changes, observable from the level of neurotransmitters to interacting brain networks (Figure 1B). We suggest that a more integrative and holistic perspective on cognitive and affective functioning in aging is urgently necessary to better understand this largely uncharted era of late-life human development, one that can extend nearly half a century and continue beyond the tenth decade of life.

Figure 1. Exploitation-biases in late life development: Reward, attention and associated neural circuits.

Panel A. Reward and motivation circuit. The AIM framework [58] proposes three processes (affective, integrative, motivational) that shape exploitation and exploration decisions. Age-related changes to these circuits provide a mechanism for the emergence of an exploitation-biased mental mode. At the Affective level, reduced sensitivity to predicted negative choice outcomes in older adulthood [79], increases positive reward expectancies, leading to increased DA signaling to attention modulation circuits (Panel C, and see below). At the integration level, greater integration of positively-valenced affective information [54] and prior knowledge coded in medial prefrontal cortex (mPFC) regions [2], reduces uncertainty of known choice options, increasing the relative risk of information seeking (i.e., exploration) behaviors, motivating an exploitation-biased choice. We propose that prior knowledge and affective information are integrated in Schema and instantiated in mPFC [76, see Box 1]. Note that the Motivation level of the hierarchy is dotted, reflecting our lack of specific predictions regarding age-related changes to this circuit. Panel B. Putative brain systems. Medial prefrontal cortex, a core node of the default network, integrates prior knowledge (via connections to default network regions) and affective value information (via dopaminergic inputs from nucleus accumbens into schema. Positive signaling from mPFC to the attention modulation circuit (anterior cingulate cortex (ACC), locus coeruleus (LC)) within the salience network biases noradrenergic functioning, promoting phasic attention. Panel C. Attention modulation circuit. This model describes an integrated neural system for the adaptive regulation of performance [35]. Increased dopaminergic reward signaling from mPFC and associated with positively-valenced outcome expectations and value-based enhances noradrenergic signaling from ACC to LC [and DA signaling to ventral tegmental areas, not shown here, see 36]. Increased noradrenergic signaling leads to sustained phasic firing in LC. According to Adaptive Gain Theory [35], phasic LC signaling promotes focused attention and a sustained bias towards exploitation-biased choice. Abbreviations: dopamine (DA), norepinephrine (NE). Panel A [adapted from 60, Figure 1. See original figure caption for full model details]. Panel C [adapted from 35, Figure 11. See original Figure caption for full model details]

The axis of exploration and exploitation

Exploration and exploitation are ‘control states’[35] and trade-offs between these states are thought to shape all volitional, or goal-directed, thought and action[36]. While trade-offs are typically studied in the context of moment-to-moment decisions, here we view exploration and exploitation through a broader lens, one that extends the conceptual model to consider exploration-exploitation trade-offs as a more pervasive decision-making style, subject to genetic, phenotype, and ontological influences. We refer to a bias towards exploration or exploitation as a mental mode and anchor our model in the rich body of research on search and decision-making[1] and changes with age[31]. When preparing to select from an array of response options, individuals can decide to either select from known options or search to identify new, presently unknown, options. This choice has been classically defined as a theory of rational search[37] and encompasses both external (shopping online for that perfect gift) and internal (recalling your favorite film of all time) search decisions[38]. Exploration-exploitation trade-offs have been investigated across species, from field mice to primates; across disciplines from evolutionary biology to computer science and organizational behavior; and across contexts from foraging, to reinforcement learning and information search[1,38].

While we draw upon the breadth and depth of this field, it is not our intention to review the expansive body of literature. Many excellent reviews have already been published on this topic[1, 8,13,36,39]. Within biological systems broadly, and human mentation more specifically, a mental mode threshold for shifting between exploration and exploitation is determined by environmental, individual and social factors[39]. Environmental factors include resource availability and predictability, extraction costs, win versus loss probabilities, time horizons and whether the search is focused internally (mentalization) or externally (perception/action). In addition to these environmental factors, individual differences also influence exploration-exploitation trade-offs. These include demographics, cognitive capacity, physical and psychological abilities, personality attributes, and neurobiology. Finally, social factors also play a role. These include societal levels of competition and cooperation, as well as availability and communal access to knowledge within a social group[39]. With these determinants in mind, in the remainder of the review we will describe how many of these factors change with age to promote the emergence of an exploitation-biased mental mode in later life.

A shifting mental mode in older adulthood

We propose that by incorporating both dimensions of choice in a volitional act: i.e., “Should I stay or should I go”?[36], exploration-exploitation trade-offs can provide a more integrative, holistic account of aging mentation. The trade-off captures both age-related declines in fluid cognitive ability[34,40], necessary for directed exploration[29] and the accumulation of prior knowledge across the lifespan[41,42], thereby reducing the informational value of exploration, and promoting more exploitative choices (affective factors also play an important role, Box 1; Table 1). We propose that these trends can be consolidated into a unified trajectory, consistent with a shift in preferred, or prepotent, mental mode from exploration to exploitation. On balance, neurocognitive aging research has viewed age-related cognitive changes through the lens of declines in fluid cognitive abilities. However, these accounts often fail to enumerate or consider older adults’ expanded repertoire of knowledge and lived experience and perhaps even more importantly, how this accumulation of knowledge intersects with fluid abilities to guide decision-making behaviors in a variety of contexts (Figure 1). Reconceptualizing these dual cognitive trajectories as a unified overarching shift in mental mode opens the way for a more comprehensive model, one that encompasses and explicitly integrates these dual trajectories of cognitive aging.

Cognitive control processes are critically dependent on prior knowledge, and this dependency increases with advancing age[33,43]. Yet it is difficult to operationalize and reliably measure this transition in aging cognition, as these processes are typically considered as separate trajectories and assessed in isolation[44]. While much is known about these independent trajectories, relatively little is known about how they intersect and combine to shape goal directed thought and action in later life. Measuring their combined influence, as opposed to each capacity independently, requires assessing their relative weighting along a single behavioral dimension. We propose that the exploration-exploitation trade-off provides this dimension. Expressing an exploration- versus exploitation- based choice indexes the summative influence of control processes, prior knowledge (as well as affective influences) in producing a single behavioral outcome: the decision to explore versus exploit. As such, any shift in bias (i.e., mental mode) or more precisely, the threshold at which an exploration-exploitation trade-off occurs, can provide a marker of how these capacities intersect and combine to shape goal directed behavior. We propose that the dual trajectories of cognitive aging bias older adults towards exploitation-based choice, indexed by an elevated threshold for an exploitation to exploration shift in later life.

Reduced cognitive control.

Older adults choose more known versus unknown options during the one-arm bandit gambling task, a reinforcement learning measure of exploratory versus exploitation-based choice. Unlike older adults, young prefer to investigate uncertain possibilities (i.e., explore) that might produce better returns[21]. Notably, this strategy can be altered by changes to the environment that impose additional cognitive load. When control demands increase, younger adults shift their choice strategy away from exploratory search and adopt a more exploitative mental mode[45, but see 12]. A similar shift has also been observed during random number generation, where sequences become increasingly predictable (i.e. exploitation-based) and less random in the context of diminished executive control resources in aging an brain disease [reviewed in 46]. These findings are consistent with the cognitive control hypothesis of search-related behavior; variability in cognitive control demands influence the selection of exploitative versus exploratory search strategies, or flexibly shifting between them, during goal-directed tasks[28,47].

The cognitive control hypothesis has been called upon to explain the shift towards more exploitative behaviors in older adulthood, as control processes show significant age-related decline[29,31]. Older adults spend more time exploiting known information versus exploring for new options on classic search and decision-making paradigms[23,27,31,32,46], although greater exploratory search with age, or an absence of age-differences has also been reported[25, 28]. The trend towards greater exploitative behaviors with age has been observed across many species, consistent with the idea that the emergence of a prepotent exploitative mental mode across development has phylogenetic origins[21,38,46]. However, it is critical to emphasize that both exploration and exploitation may be goal-directed and depend on cognitive control processes. During exploitation, cognitive control is needed to maintain the search cue, sustain attention while searching through available knowledge stores, and suppress irrelevant prior knowledge[12]. Our central premise here is that exploration requires greater cognitive control than exploitation in most contexts. There is significant evidence for this idea. Under conditions of high cognitive control demand, both younger and older adults shift away from exploration and adopt a more exploitative strategy[27,45]. Further, as control processes decline with age, it follows that choice behaviors become increasingly biased towards exploitation as a prepotent mental mode [n.b., random number generation studies reviewed in 46]. As noted above, older adults have shown greater exploratory search on a verbal fluency task which indexes internally-directed search processes[28]. However, this was attributed to the failure to maintain the search cue, resulting in an incidental shift towards exploratory search. This differs from the exploitation biases with age observed during volitional choice behaviors[23,27,32,48].

Increased prior knowledge.

One might ask why exploitation would serve as a prepotent or ‘starting’ mode for older adults. If exploration is the preferred mode when initially learning about the value of various options[29], why wouldn’t older adults calibrate to this mental mode more frequently? Here we turn to the second dimension in the shifting cognitive architecture in older adulthood: An expanding store of prior knowledge. Greater experience with options typically leads to increased exploitation, as the value of unknown options is diminished and thus the risk of exploration is increased[10]. Further, associative learning is increasingly dependent on prior knowledge in later life, with more ready access to a larger repertoire of crystalized knowledge representations in the context of declining control resources[33,49–51]. Older adults thus avail of this expanded store of prior knowledge, rather than facing the risks and costs of exploring for new information (see Box 2 for an interesting parallel in scientific discovery over the lifespan).

BOX 2: Aging and scientific genius - Not all discoveries are created equal.

In Figure 2 we provide a graphic outlining how prior knowledge, and a bias towards an exploitative mental mode, influences performance across different tasks and domains in later life. This shift from exploration to exploitation has a rather striking parallel in the domains of scientific discovery, innovation and creativity. Einstein once quipped that “A person who has not made his great contribution to science before the age of thirty will never do so.” While this idea has gained popular traction, it is not well supported by the data. The most significant scientific discoveries, those later awarded Nobel prizes or scientific medals, occur across the full expanse of middle adulthood with a gradual decline into older age. Indeed, the median age of discovery has continued to increase over the last decades [99]. Critically however, the nature of scientific innovation does change with age. When divided by discovery type, conceptual innovations, those depending on deduction, abstraction and exploration of novel associations occur much earlier than experiential discoveries, which are more inductive and involve exploitation of prior knowledge [99]. By some estimates this conceptual versus experiential ‘discovery gap’ can be more than 20 years. In considering these data it is important to highlight that exploration and exploitation can both lead to innovation and discovery. However, it is the basis of discovery (exploring new associations versus exploiting prior knowledge) that appears to change with age.

We recently reported similar findings in which older adults performed as well as young on a creative (divergent) thinking task [100, 101]. Based on these findings we reasoned that older adults were able to exploit a larger store of prior knowledge to fuel creativity in the context of declining cognitive control resources. Of course, the loss of cognitive control and the weight of prior knowledge may eventually stifle creativity into older age, consistent with the steep declines in innovation into older age reviewed above. This idea of prior knowledge stifling exploration, while offered tentatively here, was eloquently expressed by Sigmund Freud nearly a century ago in his treatise on Civilization and its Discontents (1929): “The conceptions I have summarized here I first put forward only tentatively, but in the course of time they have won such a hold over me that I can no longer think in any other way.”

Shifting motivation and time horizons.

Thus far we have presented evidence that declining control resources and expanded prior knowledge stores promote exploitation as a prepotent mental mode in later life. However, other factors may also play a role (Table 1). Aging is associated with greater neophobia, a reduced interest in, or fear of, novelty. This has been attributed to the fact that the value of novel information gained from exploration is diminished in later life. For older adults, the time horizon to extract the value of novel information, and to amortize the risks and costs of exploration, is shorter[13,21,52]. Thus, the benefit-to-cost ratio of exploration declines with advancing age. These factors directly interact with shifting affective motivations[23,53; Box 1 for a discussion of affective-cognitive interactions]. Increased positive affect is associated with a decline in goal-directed search in young adulthood[29]. The Socioemotional Selective Theory of cognitive aging posits that older adults show a preference for recollecting more positive experiences, an effect known as the positivity bias[54]. Older adults may thus be biased towards exploiting familiar, and positively-valenced, prior knowledge resulting in a curtailed search[25], precluding or delaying a switch to exploration [but there is evidence that the influence of a positivity bias may differ across search contexts 55,56].

Exploitation and exploration shape volitional thought and action across the human lifespan. As such, exploration-exploitation trade-offs may offer a more integrative measure of the shifting architecture of mentation in later life. We have provided converging evidence that a prepotency for exploitation emerges as a mental mode in older adults, attributable to declining control resources, a greater repertoire of prior knowledge, and motivational changes favoring exploitation-based decisions (Box 1; Table 1). However, important measurement challenges remain (Box 3; Outstanding Questions).

BOX 3: Measurement challenges in lifespan studies of exploration and exploitation.

Exploitation and exploration decisions are typically measured in three behavioral domains: information search, foraging, and reinforcement learning [reviewed in 11]. However, a lack of convergent validity was recently reported for behavioral tasks across these search domains, suggesting that specific task characteristics may influence exploration-exploitation trade-offs [102].

This represents a significant measurement challenge for lifespan studies where cognitive (or affective) changes may interact in domain- and/or task specific ways with search decisions. A critical research priority will be to conduct cross-sectional, lifespan convergent validity studies acquiring within-subject measures of exploitation and exploration across domains and search tasks. This would address whether aging is associated with the emergence of a common exploration-exploitation factor, consistent with dedifferentiation theories of cognitive aging [103]. An interesting prediction here would be that the shifting factors highlighted in this review (Table 1) would counter the influence of task-specific demands observed in young [102], thereby promoting an exploitative mental mode in later life. Minimizing task-specific factors is a critical measurement challenge for identifying individual and age differences in exploration-exploitation trade-offs and associated neural mechanisms.

Relatedly, detecting exploitation biases in older adulthood also requires disambiguating the impact of the various contributing factors described in Table 1. Among the most significant measurement challenge in this regard are the cognitive control and learning demands imposed by practically all goal-directed cognitive tasks. This leaves older adult performance susceptible to off-task errors or performance variability that may mimic exploratory responses, and distinguishing these from cognitive control errors (e.g., loss of rule set; poor rule learning) is difficult post-hoc. This was observed by Hills and colleagues [28] who reported that older adults made more exploratory-type category switches than young during a semantic fluency task. However, closer inspection revealed that these switches were attributable to the inability of older adults to hold the goal hierarchy in working memory, resulting in a greater number of unintended ‘exploratory’ responses. Complex learning demands would pose a similar challenge as age differences in the learning of task rules may manifest as greater exploratory-type responses for older adults [29]. As such, another critical measurement challenge for lifespan research will be designing task paradigms that involve low levels of cognitive control and impose low learning demands (or sufficient practice). The objective would be to achieve age-equivalence in the comprehension and application of task rules, necessary to isolate lifespan changes in exploration-exploration trade-off thresholds.

Outstanding Questions.

• What is the optimal (reliable, sensitive) behavioral measure(s) to estimate individual and age-group differences in exploration-exploitation thresholds (c.f. Melhorn et al., 2015)?

• Does the exploration-exploitation trade-off represent a domain-general, common factor that can be reliably associated with specific features of cognitive and brain aging (c.f. von Helversen et al., 2018)?

• What are the relative and/or sequential contributions of cognitive control, prior knowledge and reward motivation to determining exploration-exploitation thresholds in older adulthood? Do these differ by search domain (e.g., foraging, reinforcement learning, information search)?

• How do environmental, social and other individual difference factors interact with age to influence exploration/exploitation thresholds?

• Does an age-related shift toward a prepotent, exploitation-biased cognitive mode manifest as an automatic response, occurring outside of conscious awareness, or as an intentional choice, involving volitional control processes?

• Can differences in exploration-exploitation trade-off thresholds, and associated brain changes, provide reliable behavioral markers of inflection points in adult development, signaling transitions from normative to accelerated and pathological aging?

An exploitative mental mode in the aging brain

The tension between exploitation and exploration, and the calibration of a mental mode, has been related to subcortical and cortical brain systems [1,8,13,31,36]. Here we examine specific neurophysiological changes that may predispose older adults towards a prepotent or preferred exploitative mental mode. The adaptive gain model of exploitative versus exploratory behaviors identifies a neural circuit including medial prefrontal cortex, anterior cingulate cortex and catecholaminergic subcortical nuclei including the locus coeruleus (LC) and nucleus accumbens (NAc)[2,20,35,36]. Additional regions implicated in the exploration-exploitation dilemma also include lateral, orbitofrontal and rostral prefrontal cortices [1,5,57], intraparietal sulcus[17], and the anterior insula[8] as well as limbic structures including the amygdala[58] and the hippocampus[19]. While few direct investigations of large-scale network involvement in exploitation and exploration trade-offs have been reported, the capacity to flexibly shift between exploratory and exploitative behaviors has recently been related to functional connections among distributed brain areas including regions of the frontoparietal control and default networks[59] as well as salience network regions[8]. Below we briefly outline the key circuitry underlying exploratory and exploitation behaviors and how these are altered with age, promoting an exploitation bias in older adulthood (Figure 1).

Attention and reward circuits implicated in exploration and exploitation.

While multiple brain systems have been implicated in the trade-off between exploration and exploitation, we focus on two circuits here, as these offer the greatest promise in understanding the emergence of a prepotent exploitative mental mode in older adulthood (Figure 1). We first describe the adaptive-gain model of LC function and attentional control[35] which has been explicitly identified as a neural mechanism underpinning the exploitation-exploration trade-off[36]. The second is the affective, integrative and motivation (AIM) model of motivation and reward in decision-making behavior[60]. While this model has not been explicitly associated with exploitation and exploration trade-offs, the importance of reward processing, particularly for exploitative decisions (Box 1, Figure 1A), and known changes to these circuits with age, strongly implicate the AIM model as a candidate and complementary neural mechanism underpinning changing exploration and exploitation thresholds with age. These accounts encompass two critical dimensions of the exploitation-exploration trade off, attention modulation and reward expectation, and the role of their respective noradrenergic and dopaminergic circuits in modulating choice behaviors (Figure 1). Critically, both models converge on a pivotal role for the medial prefrontal cortex, and its ventral aspect (vmPFC) more specifically, in the integration and valuation of choice options [and see 2 for a recent converging account]. As we review below, the vmPFC is also a core node of the default network. This network is a functionally connected assembly of brain regions, implicated in the encoding, representation and instantiation of prior knowledge and schema (Box 1), and is selectively vulnerable to changes in aging and brain disease[61,62]. We consider changes to this network, and the mPFC specifically, to be a central feature in the emergence of an exploitation-biased mental mode in later life.

Adaptive gain model.

The adaptive gain model[35] was proposed as a model of attentional control centered on LC, the primary noradrenergic nucleus producing nearly all norepinephrine (NE) in the brain. The LC displays phasic activation during exploitative behaviors, while more tonic activation is associated with exploration[36]. Although specific neurocognitive mechanisms are still being investigated, evidence is converging on goal-directed attentional control as a primary determinant of the shift between exploration and exploitation. Phasic activation of the LC is observed during the occurrence of targets on a goal-directed task, promoting attentional engagement to the current target category, consistent with an exploitative mental mode. In contrast, tonic LC activity expands the window of attentional engagement to include both goal -relevant and -irrelevant stimuli, consistent with a more exploratory mode, which requires shifts in attentional set[7,36]. A critical question is how, and perhaps why, this shift from phasic to tonic signalling (or vice versa) of LC occurs. In part, the answer lies in a parallel catecholaminergic neurotransmitter: dopamine (DA). The dopamine circuit includes the ventral tegmental area, NAc and its dense connections to prefrontal brain regions[36]. Consistent with this model, elevated dopamine-based reward signalling dampens exploratory and motivates exploitative or rewarding behaviors[63–65]. As reward processing in ventromedial and orbitofrontal cortices decreases with ongoing exploitative behaviors, the dorsal anterior cingulate cortex evaluates this altered reward signalling and, in turn, signals a shift from phasic to tonic firing in LC[66]. This perturbation in LC firing shifts goal-directed behaviors from an exploitative to a more exploratory mode.

In the context of this (admittedly simplified) adaptive gain model, we suggest that age-related changes in LC-NE signalling favors exploitation over exploration as a prepotent mental mode of choice (i.e., decision-making) behavior. Age-related changes have been observed in LC, which is considered to be an origin site of tauopathies[67]. Tauopathies occur when the tau protein necessary for sustaining the structural integrity of neurons is altered, leading to neurotoxic changes associated with common brain diseases including AD[68]. Further, changes in LC volume are evident in normal aging, long before the emergence of neurodegenerative disease[69,70]. Age-related structural and functional changes in projection systems from LC to cortical association cortex, including prefrontal and trans-entorhinal cortices, have also been observed[71–73]. Taken together, degraded LC circuitry with advancing age predicts a sluggishness in noradrenergic signaling and less flexible shifting in attentional focus, necessary to alternate between exploration and exploitation based on shifting reward contingencies (Figure 1C). Put simply, environmental and individual difference factors such as shorter temporal horizons, neophobia, and a greater repertoire of known reward values that favor exploitation-based decisions in later life (Table 1; Figure 1A) promote phasic LC signalling. In the context of degraded LC-NE circuitry, flexible shifting from exploitation to an exploration based attentional frame is reduced, increasing the dwell time for older adults in exploitative search.

Affective, Integration and Motivation (AIM) model.

Reduced flexibility in LC-NE functioning and increased dwell time in exploitative search does not necessarily predict the prepotency of exploitation in later life, or as we argued earlier, a different ‘starting point’ for decision-making in older adulthood. To address this prediction, we turn to the AIM model as our lens for considering age-related changes in corticostriatal dopaminergic circuits underpinning reward and motivation[60,74; Figure 1A]. In the AIM model, dopaminergic signaling from the ventral tegmental area combines with noradrenergic signaling from the LC in anterior insula to potentiate anticipated gains and losses. These affective signals are then integrated with other inputs (e.g., prior knowledge and reward representations) instantiated as schema in vmPFC[2,75,76], which, in turn, motivate the configuration of action circuits to implement selected behaviors[60; Figure 1]. In older adulthood, expanding prior knowledge stores[41] increases the probability of exploiting known versus exploring unknown choice options. Further, shorter time horizons in later life also reduce the value of information-focused exploratory searches[21,23,31,77]. Finally, age-related shifts towards more affective goal pursuits[53,54,78] and positively-valenced predicted decision-outcomes[60,79] enhance the relative reward value of exploiting known options, biasing older adults towards a prepotent exploitative mental mode. This shift in reward valuation for exploitation over exploration in turn increases dopaminergic signalling from vmPFC, biasing LC-NE circuits towards phasic signaling which, as predicted by the adaptive gain model, biases attention towards exploiting known options[36].

Cortical networks.

A growing body of evidence suggests that alterations within and among large-scale, functionally connected cortical brain networks are a core mechanism of age-related cognitive change[80,81]. Here we review how age-related changes to the frontoparietal control, default and salience networks converge to promote an exploitation-biased mental mode in late-life development.

First, age-related declines in the integrity of the frontoparietal control network and altered connections to other transmodal networks is associated with reduced executive control[80]. Reduced cognitive control has been postulated as a mechanism underpinning a shift from exploration to exploitation with age[29, Box 2]. While diminished control processes reduce exploration behaviors, we have argued that this is insufficient to instantiate exploitation as a preferred mental mode; here we propose that the accumulation of prior knowledge and schemas, incorporating both affective and cognitive elements (Box 1) plays a crucial role in the shifting exploration-exploitation trade-off[33].

Engagement of prior knowledge has been associated with default network brain regions including inferior frontal gyrus, lateral temporal cortices, temporal parietal junction and temporal polar cortices[82]. The default network also includes mPFC, posterior cingulate cortex as well as medial temporal lobe memory systems[61]. Structural and functional connectivity of the default network declines with normal aging[62,83]. Further, connectivity between the default other large-scale brain networks is increased, following a pattern of network dedifferentiation widely observed with increasing age[80,81]. Greater cross talk between default and frontoparietal and salience networks with age has been associated with memory for meaningful associations[43]; semanticized autobiographical recollection[84], feelings of social closeness[85] and greater vulnerability to financial exploitation[86,87]. Each of these real-world behaviors share a common feature in that older adults are able to draw upon prior knowledge to guide goal-directed thought and action, with both adaptive and maladaptive outcomes (Figure 2). Thus, greater connectivity of the default network, and particularly medial prefrontal cortex (Box 1), to other brain regions may provide a neural conduit, allowing prior knowledge to infiltrate and influence ongoing thought and action[33,88].

Figure 2. Influence of prior knowledge on cognitive performance in older adulthood.

Samples of experimental paradigms where older adults engage prior knowledge, resulting in either adaptive or maladaptive outcomes. [50, 110, 112, 116–118]. For a comprehensive review see [33].

Finally, the salience network includes both dorsal anterior cingulate, and anterior insula regions, associated with value monitoring and reward expectancy[89,90]. These regions are implicated in the adaptive gain and AIM decision-making models and changes to the salience network likely shape the exploration-exploitation trade-off in later life. Age-related changes include reduced within-network connectivity and, critically, increased connectivity to default network regions including mPFC[91,92]. Closer coupling of the salience and default networks, and the mPFC specifically, raise the possibility that the affective valuation of prior knowledge, represented in the vmPFC, exerts greater influence over dopaminergic reward and noradrenergic attention modulation systems, biasing these circuits towards an exploitative mental mode.

Taken together, we suggest that altered cortical network dynamics intersect with attention and reward circuits to bias action and mentation towards an exploitative mode. Age-related declines in the frontoparietal control network reduce cognitive control capacity necessary for exploration[29,45]. Critically, these changes in cognitive control occur in the context of behavioral and brain changes that elevate the relevance, salience and ultimately influence of prior knowledge on decision-making. Default network brain regions, implicated in prior knowledge representations and affective valuation [instantiated in schema,76] become more integrated with both the frontoparietal control and salience networks providing a conduit for knowledge representations to exert greater influence over attention and reward systems that modulate choice behaviors. There is growing evidence for this model. Interactions between vmPFC, and nodes of the salience network, including anterior insular cortex, have been shown to promote exploitative behaviors[93]. These regions also become more functionally integrated with age[80]. Further, greater network connectivity between default and lateral PFC [as proposed in the default to executive coupling of aging, 33] may sustain exploitative search strategies, by instantiating and sustaining value expectancies for prior knowledge in working memory[12], although there is evidence that goal-directed emotional regulation may depend more on relatively preserved mPFC in older adulthood[91]. Finally, changes to the frontoparietal control network, and associated reductions in cognitive control, may preclude the flexible network reorganization necessary to shift from exploitation to exploratory search[8,59].

Concluding remarks

Here we surveyed emerging evidence that the shifting architecture of cognition, affect and brain function leads to the emergence of an exploitation-biased mental mode in later life; one that favors the exploitation of prior knowledge over exploratory search for novel information, solutions, associations, and experiences. This shift in the exploration and exploitation trade-off reflects reduced cognitive control in the context of growing stores (and influence) of prior knowledge, as well as a shift towards affective goals, and parallels age-related changes to cortical brain networks, attention and reward circuits. We argue that by encapsulating the dual trajectories of aging cognition (reduced control, increased prior knowledge) as well as shifts in affectively driven motivations, age-related changes in the exploitation-exploration trade-off provides a more integrative marker, capturing multiple dimensions of aging mentation in a single behavioral outcome.

More importantly, reframing these changes in cognitive, affective and brain architectures as resulting in a fundamentally altered mode of mentation, highlights the possibility, perhaps even the probability, that older adults approach all volitional mentation and action from a different starting position than young; one that carries both the weight and the wisdom from decades of lived experience. While the implications have yet to be fully elucidated (see Outstanding Questions), we argue that the transition towards exploitation-biased mental mode represents a fundamental reshaping of human mentation across the rapidly expanding decades of older adulthood. As such, mapping the determinants, correlates and implications of this largely uncharted epoch of adult development poses a critical challenge for the future of cognitive and affective aging neuroscience.

Highlights.

• Exploration- and exploitation- based search processes underlie goal-directed cognition.

• Older adults rely more on exploitation-based search during goal-directed tasks.

• Emergence of an exploitation-biased mental mode is coincident with declining cognitive control, knowledge accumulation, altered motivational drives, and brain network changes in older adulthood.

• The exploitative mental mode hypothesis may offer a more unitary account of mental functions in late life development.

• The age-related shift in mental mode has both adaptive and maladaptive implications for performance on laboratory-based tasks, as well as real-world functioning in later life.

Glossary

Anterior cingulate cortex

Brain region located along the medial wall of the frontal lobes, immediately posterior to medial prefrontal cortex. Implicated in salience detection and signaling when action outcomes are misaligned with expectations (i.e., prediction error)

Cognitive control

The modulation of thought and action by intended outcomes (i.e., goal states). An umbrella term encompassing specific cognitive processes including working memory, task switching, and inhibition necessary to instantiate, sustain, and update goal hierarchies.

Default network

A functionally connected assembly of brain regions including medial prefrontal cortex, posterior cingulate cortex, as well as lateral and anterior temporal and parietal regions as well as medial temporal lobe structures. Implicated in internally directed mental functions including memory, future thinking, self-awareness and social cognition.

Exploitation

A thought or action involving the utilization of existing resources including both material (e.g., food, social contact) and immaterial (e.g., prior knowledge, memories) where the reward value of choice options is known.

Exploration

A thought or action involving, new learning, a search for new resources, or novel reinforcements where the reward value of choice options is unknown or cannot be reliably predicted.

Locus coeruleus (LC)

Subcortical nucleus located in the brain stem, and the primary source of the neurotransmitter noradrenaline (norepinephrine) in the brain.

Medial prefrontal cortex (mPFC)

Brain region located along the medial walls of the frontal lobes. Comprised of three primary divisions: anterior, dorsal and ventral regions. Implicated a primary site for the integration of affective (emotional), cognitive (mnemonic), and social (self/other) representations.

Mental Mode

A bias, or prepotency, towards exploration or exploitation.

Nucleus accumbens

Subcortical nucleus located in the ventral striatum, and a primary source of the neurotransmitter dopamine in the brain.

Prior knowledge

Accumulation of learned associations, patterns, generalizations and abstractions acquired through intentional or incidental learning events and lived experiences.

Salience network

A functionally connected assembly of brain regions including anterior cingulate cortex, anterior insula with strong connections subcortical brain regions including locus coeruleus. Implicated in detection of meaningful, personally relevant information and interoceptive perception (i.e., felt experience).

Socioemotional selectivity theory

Theory of aging and emotional experience. SST predicts that with age and shrinking time horizons greater priority is afforded to goals related to social and emotional well-being and the avoidance of emotional risk.