Neuromodulation of reward-based learning and decision making in human aging

Abstract

In this paper, we review the current literature to highlight relations between age-associated declines in dopaminergic and serotonergic neuromodulation and adult age differences in adaptive goal-directed behavior. Specifically, we focus on evidence suggesting that deficits in neuromodulation contribute to older adults’ behavioral disadvantages in learning and decision making. These deficits are particularly pronounced when reward information is uncertain or the task context requires flexible adaptations to changing stimulus–reward contingencies. Moreover, emerging evidence points to age-related differences in the sensitivity to rewarding and aversive outcomes during learning and decision making if the acquisition of behavior critically depends on outcome processing. These age-related asymmetries in outcome valuation may be explained by age differences in the interplay of dopaminergic and serotonergic neuromodulation. This hypothesis is based on recent neurocomputational and psychopharmacological approaches, which suggest that dopamine and serotonin serve opponent roles in regulating the balance between approach behavior and inhibitory control. Studying adaptive regulation of behavior across the adult life span may shed new light on how the aging brain changes functionally in response to its diminishing resources.

Learning to choose adaptively between different behavioral options in order to reach goals is a pervasive task in life for individuals of all ages. We are often confronted with complex, uncertain situations that nonetheless require decisive actions to pursue our short- or long-term goals. On the one hand, achieving goals in and of itself is rewarding. On the other hand, monitoring of action outcomes is necessary for goal attainment. Adaptive behavior entails interactions between processes that monitor action–outcome relations and mechanisms that evaluate these relations with respect to goal relevance to modify future actions based on these evaluations.¹ These dynamics require close interplay between cognitive control of actions for the focused pursuit of goals and motivational processes for the evaluation and expectation of action outcomes. These mechanisms are subserved by frontostriatal networks, which are heavily innervated by the midbrain dopamine system as well as other transmitter systems, such as serotonin.^2,3

As adults age, the efficacy of these functional brain circuitries, as well as neurotransmitter systems, decline. Thus, on the one hand, it is necessary to understand how functional changes in these networks contribute to adult age differences in reward-based learning and decision making. On the other hand, one might expect functional changes and strategic adaptations in response to constraints on cognitive resources due to aging. Studying adaptive behavioral control across different life periods may shed light on these adaptations.

This paper provides a selective review of the current literature to highlight relations between age-associated decline in dopaminergic and serotoninergic neuromodulation and adult age differences in adaptive goal-directed behavior. The first section presents an overview on the neural systems involved in adaptive decision making and learning and how they are modulated by dopamine and serotonin. After the overview, evidence on age-related changes in these neurotransmitter systems and how they may affect frontostriatal circuits is reviewed. In the second section, we focus on behavioral contexts that are particularly challenging with respect to older adults’ limited cognitive and brain resources to illustrate the impacts of deficient neuromodulation in old age. Specifically, we review recent findings on adult age differences in three major aspects of learning and decision making: (1) the acquisition of goal-directed behavior under reward uncertainty, (2) the ability to flexibly abandon accustomed actions and learn new behavior during reversal learning, and (3) asymmetries in the valuation of rewarding and aversive outcomes during learning and decision making.

Adaptive behavioral control and neuromodulation of frontostriatal networks

The ability to adapt behavior to changes in the utility of action outcomes is supported by three interrelated brain networks: the substantia nigra (SN) and the ventral tegmental area (VTA) in the midbrain; the basal ganglia, most importantly the ventral and dorsal striatum; and parts of the frontal cortex, including the medial orbitofrontal cortex (OFC)/ventral medial prefrontal cortex (vmPFC), the dorsal anterior cingulate cortex (dACC), and the dorsal lateral prefrontal cortex (dlPFC). The diversity of the involved brain circuits may reflect the different component processes involved in adaptive control of goal-directed behavior.

Activity of midbrain dopaminergic neurons in SN/VTA is thought to contribute the motivation for acquiring behavior (see Ref. 4 for a review) and to code for reward prediction errors during learning.^3,5-7 Furthermore, phasic dopaminergic responses in the midbrain covary with the expected value of reward, presumably integrating factors, such as the magnitude, delay, and probability of outcomes.^8-10 The SN/VTA neurons are reciprocally connected with the striatum.^11-13 Given that the striatum itself receives a major part of its inputs from cortical areas, it might serve as a relay structure for transferring inputs from midbrain dopaminergic neurons to the cortex and vice versa. Furthermore, the nucleus accumbens has been proposed to regulate the balance between limbic and prefrontal contributions to goal-directed behavior.¹⁴

The corticostriatal connections are organized such that the majority of cortical inputs to the ventral striatum stem from regions that are mostly involved in the valuation of outcomes, that is, the vmPFC and medial OFC. In contrast, the majority of inputs to the dorsal striatum originate from cortical areas involved in executive control, such as the dACC and dlPFC.^11,15 This ventral to dorsal gradient is also observable in the connections from the striatum to the midbrain SN/VTA areas. The ventral striatum receives least of the midbrain inputs but targets the most extensive parts of the SN/VTA.^12,16 In contrast, the dorsal striatum receives most of the SN/VTA outputs but targets the SN/VTA less than the ventral striatum.^11,17

In summary, two loops link cortical, striatal, and midbrain structures during reward-based learning. The two loops can be characterized as a reward loop that is involved in the valuation of outcomes (including the VTA/SN, ventral striatum, and vmPFC/medial OFC) and an executive control loop that consists of cognitive control and motor areas involved in action selection (such as the dlPFC, dACC, and dorsal striatum). The specific anatomical setup of these two frontostriatal loops suggests that they play a major role in linking outcome valuation to cognitive and action control and thereby subserve goal-directed behavior.

Other than being richly innervated by the dopaminergic system, these frontostriatal circuits are, in part, also innervated by the serotonergic system.^2,3,7,18 Recent findings from pharmacological studies suggest that serotonin plays a critical role in the adaptation to changes in stimulus–reward contingencies during learning as well as in avoidance learning and the regulation of impulsive behavior.^19-22 Work in animal models indicates that a region in the epithalamus, the lateral habenula, controls serotonergic neuromodulation in the raphe nucleus as well as dopaminergic neuromodulation in the midbrain.²³ The lateral habenula has also been implicated in avoidance learning and shows a preferential sensitivity to negative motivational events.^24-26 Although clearly more research is necessary to replicate these results and to clarify the habenulas’ role in the midbrain dopamine system, the current findings suggest that activity in the lateral habenula to negative motivational cues may trigger serotonergic neuromodulation and lead to adjustments in behavioral control.

Age-related declines in neuromodulation of the frontostriatal-limbic network

Parallel to anatomical shrinkages of cortical and subcortical regions and age-related decline in the integrity of white matter connections,^27-29 the efficacy of various neurotransmitter systems are also compromised by aging. Other than dopaminergic and serotoninergic modulation, cholinergic and noradrenergic receptor mechanisms show age-related declines.^30,31 Whereas the cholinergic system is known to play an important role in hippocampal memory functions (see Ref. 32 for a review), the noradrenergic system is involved in mediating arousal (e.g., Ref. 33) or, as proposed in a recent theory, in regulating the exploration–exploitation trade-off.³⁴

In this review, we focus on dopamine and serotonin because these two systems have been shown to be critically involved in reward-based learning and decision making (e.g., Refs. 7 and 18). Furthermore, the age-related trajectories in these systems are somewhat better documented than is the case for acetylcholine or norepinephrine. Evidence from cross-sectional studies shows that across the adult life span, the dopaminergic and serotonergic systems undergo substantial declines (e.g., Ref. 35; see Tables 1 and 2 for summaries of studies showing cross-sectional estimates of percentage reduction per decade in dopamine D2 and serotonin transporter binding efficacy in various brain regions).

Table 1.

Cross-sectional longitudinal estimates of percentage of decade-by-decade decline in dopamine D2 receptor binding mechanisms in various extrastriatal regions in two independent samples

	Kaasinen et al.36	Inoue et al.37
Brain regions
Frontal (%)	11	14
Temporal (%)	10	12
Parietal (%)		13
Occipital (%)		12
Hippocampus (%)	10	12
Amygdala (%)	7
Thalamus (%)	5	5

Table 2.

SPECT and PET in vivo studies on age-related declines in serotonin receptor availability

	Per decade change	Reference
Brain regions
Frontal	4–6%	Costes et al.38
		Wong et al.35
Midbrain	2–11%	Hesse et al.39
		Kuikka et al.40
		Newberg et al.41
		Pirker et al.42
		Yamamoto et al.43
Thalamus	3–10%	Hesse et al.39
		Pirker et al.42
		Yamamoto et al.43

Thus far, very few studies have directly compared the extent of age-related declines in these two systems. In the rare cases where the aging-related declines in dopamine and serotonin presynaptic and postsynaptic components were investigated in identical samples, the degrees of declines seemed to be comparable in both systems.⁴²

Much of the work on the relationship between aging and the dopaminergic system has focused on the caudate and the putamen, which receive dense dopaminergic innervations from the midbrain. There is clear evidence for age-related loss in pre- as well as post-synaptic biochemical markers of this system. Regarding presynaptic mechanisms, both positron emission tomography (PET) and single photon emission computed tomography (SPECT) studies (e.g., Refs. 44 and 45) indicate age-related loss of the dopamine transporter (DAT) expression in the striatum (see Fig. 1A). For postsynaptic mechanisms, molecular imaging work reveals age-related loss of both striatal D1 and D2 receptor densities of comparable magnitude as found for the DAT (Refs. 46-50; see also Fig. 1B). The average decline ranges between 5% and 10% per decade from early to late adulthood (see Table 1). A similar downward age trajectory is seen for the mesocortical and mesolimbic dopaminergic pathways (see decline in D1 and D2 receptor binding in frontal cortex across the adult life span, as portrayed in Fig. 1C and D, respectively).

Figure 1.

Age-related decline in pre- and post-synaptic dopamine binding mechanisms. (A) Striatal DAT binding (adapted from Ref. 44 with permission from Elsevier Science). (B) Striatal D2 receptor binding (adapted from Ref. 49 with permission from Elsevier Science). (C) D1 receptor binding in frontal cortex (adapted from Ref. 46 with permission from Springer-Verlag). (D) D2 receptor binding in frontal cortex.⁴⁹

In the aging literature, less attention has been focused on serotonin than dopamine. However, available evidence also indicates age-related decline in the serotoninergic system. For instance, an early PET receptor imaging study reported an estimate of about 25% drop in frontal serotonin receptors in adults aged from 19 to 73 years,³⁵ amounting to a decline of approximately 5% per decade. More recent studies reported cross-sectional estimates of 2–11% decline per decade of serotonin receptors in various brain regions, including the hippocampus, midbrain, thalamus, and hypothalamus (see Table 2 for details). Similarly, the binding potential in key elements of the serotonergic system (such as limbic structures and raphe nuclei) has been suggested to decline with a rate of about 3–4% per decade in older adults.⁵¹

Age-related declines in neuromodulation as simulated in computational models

Various neurocomputational models have been proposed to link aging-related decline in dopamine modulation to behaviorally observed cognitive deficits. One of these models relates weakened phasic dopaminergic activity with older adults’ deficits in monitoring-performance errors.⁵² Another model focuses on capturing the relation between deficient dopaminergic modulation and older adults’ compromised prefrontal cognitive control mechanisms.⁵³ Focusing on dopamine’s role in the gain control (G) of neuronal signal-to-noise ratio, a third model has linked age-related increases in random processing fluctuations to less distinctive perceptual, memory, or goal representations in more general terms.^54,55 Specifically, this model captures aging-related decline in dopaminergic modulation by stochastically attenuating the G of the sigmoidal activation function that models dopaminergic modulation of presynaptic to postsynaptic input-response transfer (cf. Ref. 56). Reducing G decreases the slope of the activation function (Fig. 2A), which affects the signal-to-noise ratio of information transmission, and, subsequently, results in increased within-network random activation (Fig. 2B). This, in turn, leads to increased performance variability in a simulated aging neural network (Fig. 2C). If G is increased to excessive values, simulating increased dopaminergic modulation, the activation function becomes a step function, and activation variability depends critically on the amplitudes of inputs. In these cases, activation variability is markedly reduced with large positive or negative inputs, and increased with intermediate inputs. These properties of stochastic G tuning predict an inverted-U function between the levels of dopamine modulation, representational distinctiveness, and working memory capacity (Fig. 2D and E). This inverted-U relation has been confirmed empirically in animal studies and more recently in human functional imaging (fMRI) research.^57-61 The noisier and less distinctive representations in older networks—as simulated in previous work on aging-related working memory and episodic memory deficits^{54, 55,62}—can, in principle, be extended to study the influences of deficient dopamine modulation on reward processing and decision making in old age.

Figure 2.

(A) Simulating aging-related DA modulation by reduction of stochastic gain tuning. Values of G parameters for younger networks were sampled from the range [0.6–1.0], with a mean of 0.9, whereas values for older networks were sampled from the range [0.1–0.5], with a mean of 0.3. (B) Reduced gain tuning increased random activation variability, indicated by activation distributions of one young network’s and old network’s responses to a noise stimulus across 1,000 trials. (C) Reduced gain tuning also increased within-network cross-trial performance variability in simulated old networks in four simulated episodic memory tasks (adapted with permission from Ref. 54 with permission from Elsevier Science). Stochastic gain tuning captures the inverted-U function relating DA modulation and functional outcomes of (D) memory performance and (E) distinctiveness of activation patterns.⁵⁴ To simulate the whole range of dopamine dysfunction from deficient to excessive signaling, mean G values from 0.2 to 4.1 were simulated. Activation patterns presented in (E) correspond to mean G of 0.3 (very low mean G), 1.7 (optimal mean G), and 4.0 (very high mean G).

Specifically, the probability of flexibly switching to other choice options, P(δ_t), may depend on the valuation of a given choice that is acquired during reinforcement learning. Outcome evaluation during learning involves a comparison of the obtained reward with the expected outcomes. In temporal difference models this is quantified by the prediction error (δ_t), which reflects the difference between a received reward and the expected reward.^3,63 Similar to previous accounts the probability of choice switch, P(δ_t), could be modeled by a sigmoidal function.^54,64 In this model, age-related decline in reward-based dopaminergic modulation of choice switches can be captured by attenuating the slope of the function for units in the simulated striatal network. The noisier processing of each choice, due to suboptimal G in simulated old networks, could accumulate during learning and result in less distinctive representations of reward expectations of the choice options. Furthermore, the thus computed P(δ_t), can be used to adapt the representations of choice options in a simulated prefrontal context module to guide (bias) subsequent choices. In the case of simulated old networks, the noisier representations of expectation–outcome contingencies and choice switches would thus also compromise prefrontal executive control of choices. Figure 3 shows the representation of two choice options (one with high reward probability and the other with no reward) across learning. Of specific interest, representations of the two choice options in the prefrontal cortex network are less distinctive in the G reduced network, indicating that deficient neuromodulation of expectation–outcome contingencies in the striatal network may result in less efficient prefrontal control of choice switches and hamper outcome optimization. Such less distinctive representations may render it more difficult for older adults (i) to predict the expected utility of choice options,⁶⁵ (ii) to evaluate the relative payoffs of different options against each other,⁶⁶ or (iii) to flexibly switch between choice options after reversals.

Figure 3.

Activation patterns of two choice options in the simulated prefrontal context module across learning in simulated young (mean G = 1.0) and old (mean gain = 0.3) networks. Across learning, activation levels for choice option with higher reward probability become more distinct from the choice option with no reward. This distinction is, however, less pronounced in the simulated old network.

Based on the anatomical, functional, and computational considerations reviewed, we now consider the potential implications of age-related changes in neuromodulatory systems for adaptive control of goal-directed behavior. For this purpose, we focus on age differences in three major aspects of adaptive behavior: (1) age-related impairments in the acquisition of behavior under reward uncertainty, (2) age differences in the flexible adaptation to changes in the utility of outcomes during learning, and (3) age-related asymmetries in the sensitivity to gains and losses (reward and punishment) during learning. As a conceptual schema, Figure 4 highlights evidence for the triangulation of aging, neuromodulation, and reward-based learning and decision making.

Figure 4.

Schematic overview of the triangle relating aging, neuromodulation, and reward-based learning and decision making.

Reward uncertainty impairs learning in older adults

Recent findings suggest that older adults are impaired in learning if reward is delivered probabilistically and outcome information is partially unreliable. However, if stimulus– (action–)reward contingencies are deterministic, older adults are as well able to learn as younger adults are.^67-69 This is also the case early during learning, suggesting that the lack of age differences is not due to ceiling effects in younger adults.⁷⁰ Consistent with the behavioral findings, the error-related negativity (ERN, the event-related potential response to errors) is comparable in size in older and younger adults if reward information is deterministic. In contrast, if reward is probabilistic the ERN is reduced in older adults, indicating that the elderly perceive less mismatch between errors and their internal representation of the correct response. In line with the computational simulations presented, these findings suggest that older adults have problems in building up distinctive representations of expectation–outcome contingencies, particularly if the outcome information is partially inconsistent. The electrophysiological findings are largely consistent with results from an fMRI study on age differences in reward-based learning.⁷¹ Schott and colleagues⁷¹ found that age differences in reward prediction are associated with reduced ventral striatal activations for probabilistic reward cues. No such effect was observed in the activations to the actual reward outcomes (see also Ref. 72). Similarly, results by Mell and colleagues⁷³ suggest that older adults show reduced ventral striatal activations during learning from probabilistic rewards.

Further support for the idea that older adults have problems in building up reward predictions during learning under uncertainty comes from recent fMRI studies on adult age differences in decision making. Results from these studies suggest that older adults show preserved decision-making capacity if explicit information about the contingencies of the decision outcomes is available.^74,75 In contrast, in the absence of such explicit cues, older adults seem to be impaired in reward-based decision making, especially if they have to constantly update value representations.

Given the limited literature, it is difficult to be more specific about age differences in the underlying mechanisms, despite the fact that they seem to be associated with deficient neuromodulation. In this respect, computational methods, such as the temporal difference learning models,⁷⁶ in combination with neuroimaging and electrophysiological data may help to pinpoint age differences in the component processes involved in reward-based learning. On the basis of recent developmental findings, one might expect to find reduced dopaminergic prediction error-related activity in the striatum in the elderly.⁷⁷ Reduced striatal prediction error signals might impair the formation of distinct value representations (see above). Evidence for less reliable striatal reward signals in the elderly comes from a recent study showing that increased variability in hemodynamic activity in the ventral striatum mediates suboptimal financial decision making in older adults.⁷⁸

To summarize, current findings suggest that reward uncertainty leads to learning impairments in older adults. These impairments may reflect deficits in the ability to form and update value (outcome) representations using prediction error signals. The neurophysiological mechanisms behind these deficits are unclear. Findings by Dreher et al.⁷² point to age-related changes in the association between midbrain dopamine synthesis and prefrontal blood oxygen level-dependent (BOLD) activity during reward processing. This view is consistent with previous PET findings, showing that age-related decline of D2 receptor availability in the striatum is associated with reduced metabolism in lateral and medial prefrontal cortex.⁷⁹ It is also in line with the computational account presented earlier, which suggests a link between deficient dopamine modulation of reward-based learning in the striatal network and less distinctive prefrontal choice representations.

Age-related impairments in the flexible adaptation to changes in reward structure

In the previous section we focused on age differences in the acquisition of behavior and provided evidence for the idea that learning impairments in older adults may reflect deficits in the ability to form and update value (outcome) representations using prediction error signals. However, in more complex environments, individuals not only have to learn which actions to choose but also have to be able to flexibly abandon old behavior and learn new actions if reward contingencies change. Such behavior can be captured in reversal learning paradigms in which participants have to learn stimulus– (action–)reward associations that then get reversed and have to be relearned. Flexible learning and relearning of associations presumably involves an interplay of motivational processes that underlie the formation and updating of stimulus– (action–) reward associations as well as attentional mechanisms such as performance and outcome monitoring during action selection.

Early findings in monkeys, as well as more recent fMRI results in younger adults, suggest that the ventral striatum and the lateral and medial orbitofrontal cortices are critical structures involved in reversal learning.^80-82 The ventral striatum plays a well-established role in the acquisition of stimulus–reward associations via prediction error signals and its major projection areas, the vmPFC, has been associated with the updating of reward value representations.^1,83,84 The lateral OFC seems to play a more specific role in biasing action selection as a consequence of negative feedback on the previous trials.^85,86 Interestingly, there is also increasing evidence for neuromodulatory effects on reversal learning and its neural correlates in younger adults.

Dopaminergic neuromodulation during reversal learning

Pharmacological studies in younger adults showed that dopamine depletion using alpha-methyl-paratyrosine (AMPT) leads to impairments in reversal learning and reduced responding to rewarding stimuli in younger adults.⁸⁷ Evidence from studies in Parkinson patients supports the idea of an involvement of dopamine in reversal learning. In particular, dopaminergic medication with L-DOPA seems to lead to a dopamine “overdose” in Parkinson’s patients.^19,80,88,89 Such an overdose may disrupt the function of the ventral striatum by attenuating negative reward prediction error signals especially during reversals.^19,90 Moreover, very recent data in vervet monkeys suggest that variation in D2-like receptor availability in the dorsal striatum explains individual differences in behavioral flexibility during reversal learning.⁹¹

Serotonergic neuromodulation during reversal learning

In addition to dopamine, current findings also point to a critical role of serotonin during reversal learning. While dopamine is an important neuromodulator for reward-based learning in general, serotonin seems to be involved in reacting to aversive stimuli. Results from serotonin depletion studies indicate that reduced serotonin levels lead to enhanced BOLD signal change to reversal-related negative feedback in the dorsomedial PFC as well as enhanced punishment prediction learning.^{19, 20} In line with these findings, a single dose of citalopram (a serotonin reuptake inhibitor), leads to increased lose-switch behavior after negative feedback during reversal learning, presumably because of a transient reduction of serotonin levels due to presynaptic autoreceptor feedback.⁹² Longer term (seven-day) treatment with citalopram results in reduced neural responses to negative outcomes in the lateral OFC.^93,94 These findings are consistent with the idea that enhanced serotonin levels are associated with a reduced sensitivity to negative outcomes. Moreover, the data in humans line up nicely with work in primates suggesting that decreases in serotonin neurotransmission increase the sensitivity to negative outcomes and reduce reward sensitivity.⁹⁵ Taken together, serotonin seems to be involved in the suppression of actions that would result in negative consequences, thereby controlling processes that reduce the engagement with potentially aversive stimuli.¹⁸

The role of dopamine and serotonin for goal-directed behavior is certainly more complex than this limited review of the literature suggests (see Refs. 96 and 97). However, there is accumulating evidence that dopamine is involved in driving approach behavior, whereas one major facet of serotonin in adaptive learning is its role in punishment prediction and behavioral inhibition. These observations are also consistent with the broader literature on learning and decision making. Recent findings suggest that serotonin depletion affects the ability to predict reward over time during learning, making individuals more sensitive to immediate outcomes.^{22, 98} These results have been taken as evidence for the idea that reduced serotonin levels are associated with an increased devaluation or discounting of reward over time and, thus, lower impulse control.²

Taken together, the computational and empirical work in younger adults points to dopamine and serotonin as two opponent neuromodulatory players that may regulate the balance between approach behavior and inhibitory control and by this support the ability to make flexible and adaptive decisions.¹⁸

Reversal learning in older adults

Findings in older adults and aged monkeys point to age-related impairments in reversal learning that go beyond deficits in the mere acquisition of stimulus– (action–)reward contingencies.^99-103 Whether these deficits reflect greater perseverative tendencies or increased randomness of choice after a reversal in aged subjects is unclear. Findings by Mell and colleagues⁹⁹ suggest that reversal-learning impairments in older adults may not be related to impairments in standard tests of executive function (such as the Stroop task or the Tower of London task). That is, they are not (exclusively) reflecting set-shifting abilities in the elderly. Instead, the authors propose that these deficits reflect age-related changes in neuromodulatory systems (dopamine and/or serotonin) and their projections to the prefrontal cortex.⁹⁹ These deficits may lead to impairments in the updating of reward representations. Findings in animal models support these ideas by showing that reversal learning deficits in aged rats are associated with reduced dopaminergic neurotransmission in the OFC.^104-106 Given these findings and the literature in younger adults, it seems tempting to suggest that reversal learning impairments in the elderly might be due to changes in dopaminergic and serotonergic neuromodulation. Age-related reductions in serotonergic neuromodulation may lead to increased punishment prediction during learning and, hence, a greater randomness of choice after reversals due to difficulties of separating negative feedback to perseveration and reversal errors. At the same time, age-related reductions in dopamine levels may result in reduced prediction error-related activity and more general deficits in reward-based learning. Of course this is an over-simplification given the complicated interactions between the two systems.^18,96 Furthermore, it is unclear how the proposed opponency of dopamine and serotonin relates to the paradoxical effects that are reported with respect to serotonergic neuromodulation (see Refs. 97 and 107). The serotonin paradox describes the fact that, although there is substantial evidence for the idea that low serotonin levels are associated with enhanced processing of aversive stimuli, there is also evidence suggesting that the role of serotonin is to suppress actions that would result in aversive outcomes.^{18, 108, 109} Reduced levels of serotonin would, hence, relate to stronger reactions to aversive outcomes, while at the same time engendering a reduced ability to avoid negative feedback by withholding inappropriate responses such as perseverative responding after reversals.

Age differences in the sensitivity to reward and punishment

Another line of evidence for an increased sensitivity to negative outcomes in older adults comes from recent behavioral and electrophysiological studies.^{66, 110, 111, 112} Results from Frank and Kong¹¹¹ suggest that older seniors (mean age 77) show a bias toward learning from negative outcomes compared to younger seniors (mean age 67, see also Ref. 112). A similar negative learning bias was previously observed in Parkinson patients, which lead to the conclusion that these age-related changes in learning may be due to reduced levels of subcortical dopamine in the elderly.⁹⁰ More recent findings suggest that rather than affecting learning from negative outcomes, dopaminergic medication in Parkinson patients primarily affects learning from rewards.¹¹³ That is, the observed negative learning bias off dopaminergic medication most likely reflects reduced learning from reward rather than enhanced learning from negative outcomes. Whether this is also the case for older adults is still unclear. Recent findings comparing healthy older with younger adults provide behavioral support for an asymmetry in learning from positive and negative feedback.^{66, 110} The study by Haemmerer et al.⁶⁶ showed increased lose–shift compared to win–stay behavior in older than younger adults, indicating that monetary losses had a stronger impact on subsequent choice behavior than gains. In line with these findings, Eppinger and Kray¹¹⁰ found greater individual differences in learning biases in older than younger adults. Moreover, these learning biases were most pronounced in older negative learners. However, in contrast to younger adults, negative learning was not associated with a stronger sensitivity to errors as reflected in a larger error-related negativity. These findings indicate that the subcortical contributions of learning may be relatively unaffected by age, whereas the integration and updating of these learning signals into value representation in the (ventro-) medial prefrontal cortex may be impaired in older age.

Consistent with the neuromodulatory effects on reversal learning reviewed, it seems conceivable that the formation and updating of value representations in the vmPFC during learning is regulated by dopaminergic and serotonergic neuromodulatory processes. Reductions in the efficiency of dopaminergic and serotoninergic neuromodulation with age may lead to reduced reward sensitivity as it has been observed in fMRI studies on learning and decision making,^{71, 114} as well as an increased sensitivity to negative feedback during learning in older adults, as reported in behavioral and electrophysiological studies on learning.^{66, 110, 111, 112} Even though current evidence indicates that age-related cortical changes may underlie older adults’ greater sensitivity to negative action outcomes, strategic adaptations with aging should not be neglected. In particular, if less distinct reward representations are to be assumed with aging, a strategic focus on the more rare and salient negative outcomes might be adaptive for orientation in contexts with probabilistic outcomes.

It is important to note, however, that there are also several findings that point to an age-related bias into the opposite direction. That is, a stronger focus on positive, self-relevant information in older than younger adults. Results of an fMRI study by Samanez-Larkin and colleagues¹¹⁵ using the Monetary Incentive Delay (MID) task suggest that older adults anticipate losses less than younger adults, whereas no age differences were obtained during reward anticipation. The authors interpret their findings in the context of the socioemotional selectivity theory (SST), which suggests that older adults are more engaged in emotion regulation and focus on positive rather than negative cues in their environment.¹¹⁶ Findings from a study by Wood et al.¹¹⁷ using the Iowa Gambling Task (IGT) point to a reduced negativity bias in older adults, which may be due to a more equal weighting of gains and losses in the elderly. Similar findings are reported in a study by Denburg and colleagues.¹¹⁸ The authors applied the IGT in two groups of older adults and measured skin conductance responses. They found that high-performing older adults showed increased anticipatory skin conductance responding to advantageous decks, which was not the case for low-performing older adults.

Piecing together these seemingly inconsistent results in the current literature, there is a need for systematic consideration of potential differences in processing characteristics of the different tasks that may contribute to the two opposing types of age biases across different studies.

First, valence information plays a different role for reward-based learning than it does for performance in the MID task. In the study by Samanez-Larkin and colleagues, participants were explicitly told what they can expect if they press a button within a certain time window. That is, outcomes in this study did not have an impact on subsequent behavior. In contrast, in the reward-based learning studies cited earlier, the acquisition of behavior critically depends on outcome processing.^{66, 67, 111, 112} That is, in a context in which the outcome does not have an impact on subsequent behavior older adults focus less on loss predicting cues. In contrast, in a context in which successful performance depends on outcome processing, they focus more on avoiding negative outcomes than on approaching rewards. Given that during learning, the rare loss outcomes are very salient indicators of erroneous behavior, such a strategy of focusing more on loss outcomes might even be considered adaptive in view of decreasing attentional and working memory resources in older adults.

The studies on age differences in the IGT may not be very conclusive with respect to age-related changes in valence biases. This is because it is not possible to determine whether participants go for the smaller gain decks or avoid the larger punishment decks. Furthermore, the IGT has been criticized because the reward probability structure is (most likely) explicitly accessible. Hence, it could be that in healthy older adults performance in the IGT reflects the ability to calculate the expected utility of rewards rather than adjustments in performance based on rewarding or punishing outcomes (see Ref. 119).

Second, an issue that remains unclear with respect to age differences in the “valence” biases is how valence is defined. The majority of studies that are taken as evidence for the SST are studies that examined age differences in emotional processing, memory for emotional stimuli, or emotion regulation (see Refs. 120 and 121 for reviews). The functions, as well as the neural circuits involved in emotion processing, differ from the mechanisms that are examined in studies on reward-based learning and decision making. Although the two systems interact under specific situations, it does not seem straight-forward to generalize that similar biases should be observed across these different domains of cognitive functioning.

A third aspect that is often ignored in the evaluation of motivational or emotional biases is the question of how explicit or strategic these biases are. The work by Carstensen implies somewhat that the positivity bias is a strategic bias toward positive cues. In contrast, in the probabilistic learning paradigms used by Frank and Kong, as well as others, these biases most likely do not reflect explicit knowledge of the reward structure.^{66, 110-112}

Taken together, it seems likely that biases toward either positive or negative stimuli in older adults reflect adaptations to a specific task context. In a situation in which the outcome does not have an impact on subsequent behavior, older adults may focus more on reward-predicting cues. In contrast, in a context in which successful performance depends on outcome processing, they may focus more on avoiding negative outcomes than on approaching reward. Given the findings reviewed earlier it seems conceivable that older adults may show strategic preferences for positive emotional stimuli when they are not informative for evaluating performance. At the same time, they may be more sensitive to salient loss outcomes that indicate that performance adjustments are necessary to avoid future maladjusted behavior.

Summary

To summarize, the current literature indicates that older adults’ learning impairments under reward uncertainty may reflect deficits in the ability to form and update value (outcome) representations using prediction error signals. In addition to age differences in the reward-based acquisition of behavior, current findings also point to age differences in the flexible adaptation to changes in stimulus-reward contingencies during reversal learning. Furthermore, there is emerging evidence for greater loss (punishment) sensitivity during learning in older adults, at least in contexts, in which the acquisition of behavior critically depends on outcome processing. Computational and empirical work in animal models and younger adults point to the view that two opponent neuromodulatory systems, dopamine and serotonin, may be involved in regulating the balance between approach behavior and inhibitory control during learning. Age-related changes in these neuromodulatory systems may lead to an impoverished representation of expected outcomes and their associations to choice options, as well as a reduced ability to suppress the impact of learned but no longer relevant action–outcome associations. Together, these suboptimal representations may lead to less flexible goal-directed behavior in older adults.

Limitations and caveats

Cohort effects are a notorious problem of age-comparative studies. With respect to the above-mentioned studies on learning and decision making, there are cohort effects that are more general in nature, such as the fact that the younger population is much more used to using computers than the older adults are. However, there are more specific effects as well that pertain to the evaluation and the subjective value of reward. On the one hand, older adults in most western countries grew up in a situation of financial instability and reduced material resources (because of World War II and associated financial crises). These experiences may have shaped their financial decision making as well as their general attitude to monetary reward. On the other hand, older adults have higher income levels and more experiences with financial investment than younger adults, suggesting that they may value reward differently than younger adults. One way to circumvent these problems would be to use primary rewards (such as juice) as a reinforcer and to use more ecologically valid and age-appropriate learning and decision-making scenarios.

Another potential confound is differences in behavioral homogeneity in younger and older adults. Older adults typically show higher between- and within-subject variability than younger adults (see Refs. 122 and 123 for reviews). Future studies should focus more on individual differences in learning and decision making within the two groups. This also necessitates larger sample sizes that allow comparisons of subgroups that are defined by performance or other factors, such as socioeconomic status, risk preference, or amount of variability in performance and brain activation (see Table 3).

Table 3.

Number of participants and age range in aging studies

Reference	Number of participants	Age range in years or mean age ± SD
Serotonergic and dopaminergic neuromodulation during aging
Antonini et al.48	32	21–68
Erixon-Lindroth et al.44	12	34–81
Costes et al.38	53	19–70
Hesse et al.39	22	18–83
Inoue et al.37	27	21–82
Kaasinen & Rinne49	24	19–72
Kuikka et al.40	19	22–74
Moeller et al.51	19	23–73
Mozley et al.45	66	41.1
Newberg et al.41	13	22–56
Pirker et al.42	16	21–60
Suhara et al.46	17	20–72
Wang et al.47	21	22–74
Wong et al.35	44	19–76
Yamamoto et al.43	28	20–79
Learning and decision making during aging
Denburg et al.118	80	26–55; 56–85
Dreher et al.72	13	66 ± 5
	20	25 ± 3.7
Eppinger et al.67	18	20.8
	18	68.5
Eppinger et al.68	17	10–12
	18	19–24
Frank & Kong111	44	60–83
Hämmerer et al.66	44	10–12
	45	13–14
	46	20–30
	44	65–75
Mell et al.73	14	26.48 ± 3.96
	14	67.82 ± 5.01
Pietschmann et al.69	18	23.7
	25	66.1
Samanez–Larkin et al.115	12	19–27
	12	65–81
Schott et al.71	19	62–78
	18	19–28
Simon et al.112	17	18.9
	24	70.3
Weiler et al.100	30	19–33
	30	50–71
Wood et al.117	88	18–34
	67	65–88

Finally, the current literature suffers from a lack of studies that directly investigate the links between aging, learning and decision making and neuromodulation. This review has to rely mostly on evidence linking only two of the components of the aging–neuromodulation–learning and decision-making triangle (cf. Fig. 4). The very few existing studies including all three factors do not yet yield conclusive findings about details of this triangular relationship. They nevertheless provide evidence for age differences in neuromodulatory responses during reward processing. The study by Dreher et al. combined PET and fMRI measurements in younger and older adults during a slot-machine gamble. They observed a change in the association of ventral striatal dopamine synthesis and dLPFC activation with age. Animal studies on self-stimulation of the medial forebrain bundle in young and old rats showed that self-reinforced responding decreased with aging but could be enhanced by administering amphetamine¹²⁴ (see also Ref. 125). Finally, older rats were shown to exhibit smaller increases of dopamine in the amygdala during reversal learning.¹²⁶ Despite these promising first results, more research is clearly needed on age differences in neuromodulatory responses during reinforcement learning and decision making in humans.

Conclusion and outstanding questions

Aging-related deficits in neuromodulation of the frontostriatal-limbic networks, particularly the dopaminergic and serotoninergic systems, may contribute to adult age differences in motivational regulation of behavioral control. Age-related impairments are most apparent in behavioral contexts that are particularly taxing for older adults’ limited cognitive and brain resources. Although the available evidence is still scarce, current findings suggest that deficits in neuromodulation contribute to older adults’ behavioral disadvantages if reward information is uncertain or if the task context requires the flexible adaptation to changes in stimulus–reward contingencies. Beyond these contexts, the temporal aspect of motivational regulation of choice behavior, such as discount rates in intertemporal choice (e.g., Refs. 114 and 127) and multistage Markov decision processes (e.g., Ref. 21) may also be affected by changes in dopaminergic and serotonergic neuromodulation with age. It has to be noted, however, that at this point it is unclear whether, or to what extent, age-related changes in these neuromodulatory systems are indeed functionally detrimental. An alternative view could be that reduced serotonergic or dopaminergic function reflects adaptations to changes in internal or environmental demands or may even serve a compensatory function (see Ref. 128). Hence, it could be that what may be considered an impairment in a specific task context (increased lose–switch behavior or reduced discounting of reward), may in fact be adaptive from an age-specific perspective, considering the higher relevance of accurate performance or possible reduced salience of monetary reward in older adults. Future studies should try to separate strategic or adaptive behavioral changes with age from changes that have their foundation in structural deficits.

In general, more empirical and neurocomputational work is needed to solidify the assumed links between age differences in behavior and neuromodulation as well as neuroanatomy with respect to reward-based learning and decision making. The combination of computational methods such as temporal difference learning or drift diffusion models and neuroimaging approaches seems a particularly promising avenue for studies aiming to address questions about age-related impairments in specific component processes of adaptive behavior.

Second, comparing the effects of interindividual differences across age groups in genetic polymorphisms relevant to neuromodulation might help to investigate the impact of aging-related decline in neuromodulation in a more locally specific manner. Aging may interact with individual differences in genotypes that are relevant for dopaminergic (e.g., the DARPP-32 D1 receptor gene and the DRD2 dopamine receptor gene) and serotonergic (e.g., the 5HTTLPR serotonin receptor gene) modulations, indicating that the neuromodulatory process affected by the polymorphism is related to age differences in decision making or reinforcement learning (see Refs. 129 and 130). Future age comparative studies that use genomic or pharmacological approaches are needed to elucidate the interactions of these two transmitter systems in altering decision making and reinforcement learning across the life span.