Feedback-Based Learning in Aging: Contributions and Trajectories of Change in Striatal and Hippocampal Systems

Abstract

The striatum supports learning from immediate feedback by coding prediction errors (PEs), whereas the hippocampus (HC) plays a parallel role in learning from delayed feedback. Both regions show evidence of decline in human aging, but behavioral research suggests greater decline in HC versus striatal functions. The present study included male and female humans and used fMRI to examine younger and older adults' brain activation patterns during a learning task with choice feedback presented immediately or after a brief delay. Participants then completed a surprise memory task that tested their recognition of trial-unique feedback stimuli, followed by assessments of postlearning cue preference, outcome probability awareness, and willingness to pay. The study yielded three main findings. First, behavioral measures indicated similar rates of learning in younger and older adults across conditions, but postlearning measures indicated impairment in older adults' ability to subsequently apply learning to discriminate between cues. Second, PE signals in the striatum were greater for immediate versus delayed feedback in both age groups, but PE signals in the HC were greater for delayed versus immediate feedback only in younger adults. Third, unlike younger adults, older adults failed to exhibit enhanced episodic memory for outcome stimuli in the delayed-feedback condition. Together, these findings indicate that HC circuits supporting learning and memory decline more than striatal circuits in healthy aging, which suggests that declines in HC learning signals may be an important predictor of deficits in learning-dependent economic decisions among older adults.

SIGNIFICANCE STATEMENT The hippocampus (HC) and striatum play distinct and critical roles in learning. Substantial research suggests that age-related decline in learning supported by the HC outpaces decline in learning supported by the striatum; however, such inferences have been drawn by comparing performance in tasks with fundamentally different structures. The present study overcomes this obstacle by implementing a single fMRI-learning paradigm with a subtle variation in feedback timing to examine differential age effects on memory supported by the HC and striatum. Our results provide converging behavioral and brain-imaging evidence showing that HC circuits supporting learning and memory decline more than striatal circuits in healthy aging and that declines in HC learning signals may predict early deficits in learning-dependent decisions among older adults.

Introduction

Adaptive decision making requires learning to link actions with outcomes and to update value representations. The striatum supports learning from immediate rewards (Schultz, 1998), with subregions such as the ventral striatum playing a key role in responding to prediction errors (PEs) (Pessiglione et al., 2006). Learning from immediate rewards critically depends on inputs from midbrain dopamine neurons, which exhibit reduced responsiveness (Kobayashi and Schultz, 2008) and temporal precision in reward prediction (Fiorillo et al., 2008) when rewards are delayed. When choice outcomes are delayed, learning must rely on links between choices and outcomes based on relational memories mediated by medial temporal lobe (MTL) structures such as the hippocampus (HC). In fact, there is evidence of a double dissociation in which the striatum (and associated regions in the basal ganglia) supports learning from immediate feedback, whereas the HC (and other components of the MTL) supports learning from delayed feedback. For example, in Parkinson's patients with striatal function deficits, learning from immediate feedback is impaired but delayed feedback learning remains intact, whereas the opposite pattern is observed in amnesic patients with MTL lesions (Knowlton et al., 1996; Foerde et al., 2013). In healthy young adults, PE response is greater in the striatum versus HC when the outcome is immediate, but greater in the HC when the outcome is delayed (Foerde and Shohamy, 2011). The current study investigated whether this double dissociation between striatal and HC systems for learning from immediate versus delayed feedback is maintained in healthy aging. We specifically examined the possibility that these functional distinctions between the HC and striatum decline asymmetrically due to differential deterioration in memory systems.

Behavioral research indicates greater age-related decline in MTL-dependent tasks relative to striatum-dependent tasks (Hoyer and Verhaeghen, 2006). Further, structural neuroimaging analyses find age-related volume loss in both regions; however, losses in the HC are more accelerated in late life (Raz et al., 2005, 2010; Walhovd et al., 2011). Prior research suggests that aging has a negative impact on immediate-feedback learning as indicated by performance impairments (Mell et al., 2005; Eppinger et al., 2013) and diminished striatal PE response (Chowdhury et al., 2013; Eppinger et al., 2013; Samanez-Larkin et al., 2014). The relative impact of aging on delayed-feedback learning is unknown, but may be greater than that observed for immediate-feedback learning given apparent asymmetries in decline within different memory systems.

To enhance our understanding of changes to feedback-based learning mechanisms in healthy aging, the present study compared probabilistic learning from immediate and delayed feedback in healthy younger and older adults during fMRI. Using a task adapted from Foerde and Shohamy (2011) (Fig. 1), participants learned to associate symbols with positive or negative outcomes indicated by pictures of indoor or outdoor scenes. In the immediate condition, outcome stimuli were presented immediately after the cue selection and, in the delayed condition, stimuli were presented 7 s after the cue selection. Key outcome measures were learning behavior and neural PE responses in the striatum and HC under different feedback timing conditions. In keeping with previous research (Foerde and Shohamy, 2011), we predicted that PE-related activity would be greater in the striatum with immediate feedback, but greater in the HC with delayed feedback.

Figure 1.

Learning task. Participants selected cues that were probabilistically associated with positive and negative outcomes. Outcomes followed cue selections either immediately (1 s) or after a brief delay (7 s). Outcomes were indoor and outdoor scenes. In a prescan session, participants were trained to associate different scene types with either positive or negative outcomes.

The present study had three goals. Our main goal was to investigate the effects of aging on PE-related responses in different memory systems. Given previous evidence for increased age-related decline in the HC compared with the striatum, we predicted that PE-related activity in the HC in the delayed condition would show greater age-related reductions than PE-related activity in the striatum in the immediate condition. Our second goal was to investigate the effects of aging on episodic memory for the outcome stimuli. Finally, our third goal was to examine effects of age and feedback timing on the ability to subsequently discriminate between cues on indexes of preference, probability awareness, and incentive compatible economic decision making.

Materials and Methods

Participants.

The initial sample included 30 younger adults and 36 older adults from the Durham, North Carolina area. All participants were fluent in English, free of MRI contraindications, and had normal or corrected vision. To reduce the confounding effects of performance differences, we excluded four younger adults and six older adults who were not able to reliably (above chance) select the easiest high-probability positive cues by the end of a practice session (last five presentations). One additional older adult was excluded due to experimenter error during the scan session. Therefore, the final sample included 26 younger adults (12 males; M_age = 26.3; SD = 4.6; M_edu = 16.5; SD = 2.4) and 29 older adults (14 males; M_age = 68.7; SD = 3.7; M_edu = 17.0; SD = 1.7). Older adults completed a Mini-Mental State Exam (MMSE) (Folstein et al., 1975) with a group average score of 29.1 (SD = 1.0; range = 26–30; one participant's score missing due to experimenter error). Participants were compensated with a flat payment for their time ($20/h) and a bonus payment of winnings from the willingness-to-pay (WTP) task (up to $10). All participants provided informed consent under a protocol approved by the Institutional Review Board of Duke Medical Center.

Learning task.

The current study used an adapted version of the probabilistic learning task implemented by Foerde and Shohamy (2011) (Fig. 1). Participants selected cues (Japanese hiragana) that were probabilistically associated with positive and negative outcomes during fMRI data collection. Because none of our participants was familiar with Japanese, the cue stimuli had no associated meaning at the start of the task. Cue pairs were presented in the choice phase and each pair included one cue from each valence condition. Outcome probabilities for “good cues” were 80% positive and 20% negative and, for “bad cues,” 20% positive and 80% negative (pseudorandomized across trials). Participants had up to 3000 ms to select a cue during the choice phase, after which the selected cue remained on the screen for either 1000 ms (immediate condition) or 7000 ms (delayed condition). Then, the outcome screen was presented for 1500 ms. If participants did not make a response within the choice phase, outcomes were replaced with a message that they should respond faster next time. Trials were separated by a jittered fixation with an average duration of 3000 ms (2000–4000 ms, 250 ms intervals, randomly assigned). Participants completed two functional runs of 40 learning trials each. In each run, participants learned about one cue pair from the immediate condition and one pair from the delayed condition (20 interleaved trials for each condition). Cue stimuli were randomly assigned to experimental conditions for each participant. New cue pairs for both conditions were introduced after the first run to increase the amount of time that participants spent actively learning cue–outcome associations.

Before the scan session, participants received instructions and a description of the learning task in a private testing room and practiced the task. They were told that each cue favored certain outcomes (no specific contingencies given) and they were instructed to pick the cue that they believed had the highest chance of positive outcomes. Participants were also explicitly told that the time between their choice and the outcome presentation would vary (no specific delay times disclosed). In addition, they were told that they could earn a cash bonus after the scan session and that performance on the bonus task depended on what they learned during the scanner task.

Optimal choice performance was determined based on outcome history. Specifically, at the start of each block, cues were associated with a 0.5 probability of positive feedback. After each choice outcome, experienced probabilities of positive feedback were updated for the selected cue (e.g., cue yields positive outcomes for 3/4 selections, probability = 0.75). Analysis of choice behavior was based on trials with unequal outcome probabilities, such that selection of the cue with the higher positive-outcome probability within a cue pair was coded as optimal.

Outcome memory task.

Foerde and Shohamy (2011) found that episodic memory for outcome stimuli was enhanced in the delayed condition relative to the immediate condition. The current study sought to replicate and examine potential age differences in this effect. Using the same procedures as Foerde and Shohamy (2011), the current study included a postlearning test of incidental memory for indoor and outdoor scene pictures presented during the outcome phase. During the prescan instruction session, participants were told that one scene type represented positive outcomes and the other, negative outcomes (e.g., indoor = good; outdoor = bad). Then, participants completed a short practice with different indoor and outdoor scenes and categorized them as good and bad with 100% accuracy before advancing to the scan session. The outcome valence conditions assigned to indoor and outdoor scenes were counterbalanced across participants. Immediately after completing the learning task in the scanner, participants were escorted to a testing room to complete additional tasks. First, they completed a self-paced surprise memory test for the outcome scenes. Participants provided memory ratings for each scene that they encountered during the learning task, which were intermixed with an equivalent proportion of new indoor and outdoor scenes. Memory ratings were provided on a 1–4 scale (1 = remember with contextual details, 2 = strongly familiar, 3 = weakly familiar, 4 = new).

To facilitate cross-study comparisons with Foerde and Shohamy (2011), performance on the outcome-memory task was examined using high confidence corrected recognition (“remember” and “strongly familiar” responses). Further, high-confidence recognition responses are associated with HC activity, whereas low-confidence recognition responses are associated with cortical MTL regions (Daselaar et al., 2006; Diana et al., 2007) and could also reflect implicit memory processes (Dew and Cabeza, 2011). Outcome memory scores reflected the difference between the proportion of high confidence responses for “old” and “new” pictures (i.e., hits and false alarms).

Cue contingency awareness tasks.

Next, participants completed several tasks that measured postlearning awareness of cue contingencies. These tasks were included to examine the effects of age and feedback-timing on the ability to subsequently discriminate between cues using different indexes of transfer. These tasks began ∼5–10 min after the conclusion of the learning phase in the scanner (∼20–25 min from first learning trial). In these tasks, each cue from the learning phase was presented individually and participants provided: estimations of time delays between choices and feedback (in seconds), subjective preference ratings (1 = lowest, 7 = highest), estimations of positive outcome likelihood (0–100%), and WTP for a chance at a cash bonus. Data for the delay-to-feedback estimation task was not collected for the first three young adults in the study. In addition, one older adult was excluded from analysis of this task after being identified as an outlier (delay-time estimations >3 SDs above the group mean for immediate and delayed cues). The WTP task followed a Becker–DeGroot–Marschak auction format (Becker et al., 1964) to collect incentive-compatible cue value estimations. For each cue, participants were given $2.00 and were asked to bid an amount from $0.00 to $2.00 for a chance to win an additional cash bonus of $8.00 in a lottery game. In the game, one randomly selected cue would be “played” in a single trial. If the cue led to a positive outcome, the participant would win $8.00, but if it led to a negative outcome, the participant would win nothing. Participants were told that the likelihood of cues giving positive outcomes in the lottery game was equivalent to their likelihood of leading to positive outcomes during the learning task. They were also told that the price to play the lottery would be randomly selected from a uniform distribution from $0.00 to $2.00 and that their bids for individual cues were independent (e.g., they could bid $2.00 for multiple cues). If participants' WTP for the selected cue met the lottery price, they paid for the lottery (keeping the remainder of their $2.00 regardless), observed one “flip” of the selected cue, and either won $8.00 or nothing. If participants' WTP for the selected cue was less than the lottery price, they were given their $2.00 endowment and did not get to play for a bonus. The relative amounts of endowments and bonuses were determined based on behavioral piloting indicating that these values were sufficient to overcome well known loss aversion bias (Kahneman et al., 1991) that would cause participants to avoid gambling on cues with perceived positive values.

Behavioral data analysis.

Behavioral analysis was conducted using SPSS version 24 (IBM SPSS Statistics; RRID:SCR_002865). Optimal choice performance was examined using a 2 × 2 × 4 ANOVA model with age group (younger, older) as a between-subject factor and within-subject factors of feedback-timing condition (immediate, delayed) and trial block (1–5, 6–10, 11–15, 16–20). Outcome memory and estimations of time delays between choices and feedback were analyzed using a 2 × 2 ANOVA with age group and feedback-timing condition as predictors. To evaluate postlearning awareness of cue contingency measures of preference, outcome probability estimation, and WTP, we calculated cue discrimination values for each condition. Cue discrimination values corresponded to the difference between average ratings for good and bad cues (M_good − M_bad) on immediate- and delayed-feedback trials. Postlearning cue discrimination values were examined using a 2 (age) × 2 (feedback timing) ANOVAs. Effect sizes were measured using partial η squared (η_p²) values and errors using SEM.

Learning model.

Learning task data were fit with a hierarchical Bayesian version of a standard reinforcement learning model in which subjects learn the action value of each cue (Q) through experiencing rewards and reward omissions in response to their choices. After learning is complete, the action value should approximate the expected value of each option. Action values are updated according to the standard delta rule: Q_new β Q_old + αΔ, where α is the learning rate and Δ is the reward PE associated with the choice: Δ = R − Q_old, where R is the observed outcome (0 or 1 for incorrect and correct responses). To reflect the fact that the values of cues on each choice were anticorrelated, the unchosen option on each trial was decremented by the same amount. We also assumed a standard softmax form for the choice function: given options A and B with action values Q_A and Q_B, the probability of choosing A over B is given by the following: p(A > B) = e^βQA/(e^βQA + e^βQB) where β is a parameter controlling the sensitivity of choice to the difference in value between the two options. To enhance our ability to make cross-study comparisons, learning rates were determined across feedback-timing conditions as done by Foerde and Shohamy (2011).

The model followed a hierarchical approach in that learning rates for each participant were assumed to be drawn from one of two (β) population distributions, corresponding to younger and older participants, whereas β parameters for each subject were drawn from population-specific gamma distributions. Then, Bayesian inference methods were used to determine posterior distributions for both individuals' learning rates and the population distributions as a whole. By using Bayesian posterior distributions instead of point estimates, this model captured uncertainty about the value of α for each subject and inferences made about the population could be used to improve model fits for each individual subject. Model-based estimates (medians) for participant learning rates (α) were used in the behavioral analysis and PEs (Δ, also medians) were used in the fMRI analysis. Models were fit using Hamiltonian Monte Carlo methods via the Stan probabilistic programming language (Carpenter et al., 2016). Models were fit using two chains of 2000 iterations each, with the initial half of each chain discarded as burn-in. Convergence of models was checked via inspection of trace plots to verify negligible autocorrelation among samples.

Model fits were assessed via log predicted probability, log p(y_new|y_observed) via leave one out (LOO) cross-validation (Vehtari et al., 2017). This approach results in a LOO estimate for each data point, which is a measure of how likely we expect each observation to be under a model trained using all other data. Model parameter estimates are reported in Table 1. Group differences in learning rates and softmax values were analyzed using ANOVAs with age group as a between-subject factor. Learning rates and softmax values from alternative models that separately determined these parameters by feedback-timing condition were analyzed using 2 × 2 ANOVAs with condition as a within-subject factor and age as a between-subject factor. Code used for model fitting, along with model definition files, are available at https://github.com/jmxpearson/bayesrl.

Table 1.

Learning model parameters

	Mean (CI)
Learning rate (α)
Younger	0.14 (0.03)
Older	0.12 (0.03)
Across age groups	0.13 (0.02)
Softmax inverse temperature (β)
Younger	7.98 (1.67)
Older	6.83 (1.58)
Across age groups	7.40 (1.15)
Model fit (mean log predictive probability)
Younger	−0.35 (0.02)
Older	−0.43 (0.02)
Both combined	−0.39 (0.02)

Data are shown as means and confidence intervals (CIs) for distribution medians.

Brain-imaging data collection.

Imaging data were acquired from a 3 T GE Healthcare MR750 scanner with an eight-channel head coil. Anatomical data were collected first using a T1-weighted spoiled gradient recalled echo sequence: 128 oblique slices parallel to the anterior–posterior commissure plane; field of view (FOV): 25.6 cm; voxel dimensions: 1 × 1 × 1.2 mm. FMRI data were acquired using a T2*-weighted SENSE inward-spiral pulse sequence (Glover and Law, 2001) sensitive to the blood oxygenation level-dependent (BOLD) signal: TR: 2000 ms; TE: 30 ms; FOV: 24 cm; 34 oblique slices; voxel dimensions: 3.75 × 3.75 × 3.8 mm. Functional data were collected across two runs, each including 223 volumes, with the first three volumes of each run discarded to allow for stabilization of the T2* signal.

fMRI data analysis.

Brain-imaging data analysis was performed with software from the Functional MRI of the Brain (FMRIB) Software Library (FSL; RRID:SCR_002823). General linear modeling of fMRI data was conducted using FMRIB FSL Feat version 6.00 (Smith et al., 2004). Data were preprocessed using the following procedures: motion correction using MCFLIRT, slice-timing correction (for interleaved acquisition), brain extraction using BET, spatial smoothing (Gaussian FWHM 5 mm), and high-pass temporal filtering (at 100 s). FLIRT was used to register functional images to individual structural images (6 degrees of freedom) and the FSL Montreal Neurological Institute (MNI) template image using an affine transformation (12 degrees of freedom). Z statistic images data were thresholded with a voxelwise threshold of Z > 2.3 and cluster-corrected threshold of p = 0.05 (Worsley, 2001) and result images presented in MNI coordinates. First-level analyses used FILM prewhitening and included head motion parameters as nuisance regressors. All event variables were convolved with a double-gamma hemodynamic response function. Second-level analyses averaged participants' data across runs using a fixed-effects model. Third-level analyses used a mixed-effects model (FLAME1) to average data across and within groups, allowing for determination of group means and differences.

Our primary analysis examined feedback-timing effects on BOLD response to model-derived PE estimates during outcome presentation. The following task events were modeled: choice phase, outcome phase, and outcome phase with PE as a parametric modulator orthogonalized to the main outcome phase regressor. Importantly, the critical manipulation of feedback timing requires standard (nonjittered) delays between choice and outcome phases, leading to nonoptimal modeling of BOLD response to feedback, particularly in the immediate condition. To address this issue and facilitate cross-study comparisons, we followed the basic approach of Foerde and Shohamy (2011) using a two-step process: (1) a regression analysis was first conducted with PEs as parametric modulators of outcome response across feedback-timing conditions and (2) we then extracted BOLD response to PEs in a priori regions of interest (ROIs) and determined average PE response within the immediate- and delayed-feedback trials. We used anatomical ROIs for the striatum and HC drawn from the Harvard–Oxford Probabilistic Atlas' within FSL. Our ROI analysis targeted two regions previously shown to exhibit feedback-timing-dependent PE responses: the ventral striatum and the anterior HC (Foerde and Shohamy, 2011). Specifically, the striatum ROI included the nucleus accumbens (NAcc) and the HC ROI included the HC anterior to y = −23 (as in Foerde and Shohamy, 2011; both bilateral). Foerde and Shohamy (2011) used an ROI threshold of 25%. We increased ROI thresholds to 50% probability to increase certainty that masks were located in our target anatomical regions for both age groups, considering potential age-related variability in brain structure. Trial-level PE-related activation within these ROIs was determined using the single-trial analysis procedure described by Mumford et al. (2012).

To test the primary hypothesis that normal aging is associated with asymmetrical decline in memory systems, we examined age differences in PE response to immediate and delayed feedback in the striatum and HC. Specifically, we examined ROI response to PEs in a mixed 2 × 2 × 2 ANOVA with age as a between-subject factor and feedback-timing and memory region as within-subject factors.

Results

Age difference in effects of feedback timing on learning and memory performance

As shown in Figure 2, optimal choice selection increased by trial block, with results indicating strong linear effects (F_(1,53) = 63.84, p < 0.001, η_p² = 0.55), but also quadratic (F_(1,53) = 5.87, p = 0.02, η_p² = 0.10) and cubic effects (F_(1,53) = 5.18, p = 0.03, η_p² = 0.09). In addition, we observed a group difference (F_(1,53) = 4.20, p = 0.045, η_p² = 0.07), such that older adults had lower optimal choice selection (M = 0.78, SEM = 0.03) across trials and conditions relative to younger adults (M = 0.86, SEM = 0.03). No other significant main effects or interactions were observed (all p > 0.05). Therefore, older adults in our study exhibited intact learning but lower optimal choice selection across feedback-timing conditions.

Figure 2.

Proportion of optimal cue selection by age group and feedback-timing condition across trials. Selection of cues with more positive outcome histories was coded as optimal. Results indicated increased selection of optimal cues across trials, with no differences by condition and only marginally higher overall choice selection in younger adults (YA) relative to older adults (OA). Error bars indicate SEM.

We then examined incidental memory for outcome stimuli (scenes) from immediate- and delayed-feedback trials. We focused on high-confidence recognition responses, which have been previously linked to HC function (Daselaar et al., 2006; Diana et al., 2007). Consistent with previous research (Foerde and Shohamy, 2011), the proportion of corrected high-confidence recognition responses was greater for outcome stimuli in the delayed than in the immediate condition (F_(1,53) = 9.00, p = 0.004, η_p² = 0.15). Older adults exhibited poorer memory performance overall (F_(1,53) = 15.10, p < 0.001, η_p² = 0.22) and no difference between delayed and immediate conditions when their data were analyzed separately (F_(1,29) = 1.11, p = 0.30, η_p² = 0.04). As a result, there was interaction of age and feedback timing (F_(1,53) = 4.16, p = 0.046, η_p² = 0.07; Fig. 3). This effect was explained by differential hit rates for immediate versus delayed outcomes in younger adults (M_immed = 0.38, SEM = 0.02; M_delay = 0.47, SEM = 0.02) and equivalent hit rates across conditions in older adults (M_immed = 0.36, SEM = 0.02; M_delay = 0.38, SEM = 0.02). These data show that the proportion of high-confidence recognition responses was similar across age groups and conditions, with the exception that younger adults had notably better recognition of delayed-outcome images. In addition, overall age differences were also driven by a higher proportion of high-confidence false alarms in older adults (M = 0.26, SEM = 0.02) versus younger adults (M = 0.15, SEM = 0.02; F_(1,53) = 15.10, p < 0.0001, η_p² = 0.22). Therefore, our results indicated that delaying feedback improved subsequent memory for outcome stimuli in younger adults but not older adults.

Figure 3.

Memory for stimuli (scenes) at outcome phase by feedback-delay condition and age group. Across conditions, older adults had poorer incidental memory and younger adults alone exhibited enhanced memory in the delayed condition. Means represent proportion of high confidence hit responses (by condition) versus false alarms. Error bars indicate SEM.

Post hoc correlation analyses across conditions and age groups revealed positive relationships between corrected recognition and measures of cue contingency awareness (preference: r₍₅₃₎ = 0.41, p = 0.002; positive outcome probability estimation: r₍₅₃₎ = 0.37, p = 0.005; WTP: r₍₅₃₎ = 0.30, p = 0.03). These data suggest that individuals with better episodic memory for outcome stimuli had an enhanced ability to transfer stimulus–outcome learning to new contexts, perhaps reflecting subject-level engagement of the HC (Myers et al., 2003; Preston et al., 2004). An alternate explanation, however, is that individuals with better stimulus–outcome learning could allocate more attention to outcome stimuli during the learning phase, allowing for better encoding and subsequent memory (i.e., given an excess of cognitive resources). If true, performance on the outcome memory task should correspond to performance on the learning task. Our results for younger adults do not support this alternate account because enhanced memory for delayed outcome stimuli was observed in the absence of condition effects for learning performance (Fig. 2). Indeed, among younger adults, learning performance in the delayed condition did not differ from the immediate condition across blocks (F_(1,25) = 2.02, p = 0.17, η_p² = 0.08) nor in the first block (F_(1,25) < 1, p = 0.38, η_p² = 0.03).

Postlearning awareness of cue contingencies

The current study included measures of postlearning cue contingency awareness. These measures allowed for examination of age and feedback timing on awareness of relationships between specific cues and their contingencies. After excluding for outliers and incomplete datasets, results for delay-to-feedback estimations revealed significantly higher estimations for cues in the delayed (M = 3.11 s, SEM = 0.20) versus immediate condition (M = 2.74 s, SEM = 0.18; F_(1,49) = 6.78, p = 0.01, η_p² = 0.12). Awareness of delay time did not differ by age (F_(1,49) < 1, p = 0.75, η_p² = 0.00) nor did age interact with condition to affect delay-time estimations (F_(1,49) < 1, p = 0.60, η_p² = 0.01). These results indicated that there was an age-invariant awareness of the feedback-timing contingencies.

In addition, as displayed in Figure 4, we found several consistent results across postlearning measures of cue preference, outcome estimation, and the WTP task. First, we found cue discrimination values (M_good − M_bad) were significantly higher than zero as measured by preference (F_(1,53) = 83.05, p < 0.001, η_p² = 0.61), probability estimations for positive outcomes (F_(1,53) = 118.09, p < 0.001, η_p² = 0.69), and WTP values (F_(1,53) = 107.68, p < 0.001, η_p² = 0.67). These results also revealed age effects indicating that, relative to younger adults, older adults had lower postlearning discrimination performance for good and bad cues on measures of preference (F_(1,53) = 5.30, p = 0.03, η_p² = 0.09), outcome probability estimation (F_(1,53) = 7.11, p = 0.01, η_p² = 0.12), and WTP (F_(1,53) = 4.88, p = 0.03, η_p² = 0.08). Across measures, these results suggest poorer postlearning awareness of outcome contingencies in aging. Third, we found little evidence that feedback timing affected awareness of cue–outcome relationships. The only significant effect of feedback timing was found for preference, such that preference-based discrimination for good versus bad cues was magnified in the delayed condition (F_(1,53) = 4.94, p = 0.03, η_p² = 0.09). Condition effects were not significant for outcome probability estimation (F_(1,53) < 1, p = 0.67, η_p² = 0.00) or WTP (F_(1,53) < 1, p = 0.97, η_p² = 0.00). Interactions between age and feedback-timing condition were not significant for preference (F_(1,53) < 1, p = 0.67, η_p² = 0.00), outcome probability estimation (F_(1,53) < 1, p = 0.67, η_p² = 0.00) or WTP (F_(1,53) < 1, p = 0.67, η_p² = 0.00).

Figure 4.

Cue discrimination for postlearning measures of cue contingency awareness. At the end of the experiment, participants responded to questions relating to their awareness of contingencies associated with individual cues from the learning task. For each cue, participants provided cue-specific preference ratings (1 = lowest, 7 = highest; A), positive-outcome likelihood estimations (0–100%; B), and completed a WTP task with real financial stakes (bids from $0.00 to $2.00; C). A main effect of condition was observed for preferences and age effects were observed for all three measures. Error bars indicate SEM.

Equivalence tests (Lakens, 2017) were used to confirm observed null effects of condition on outcome probability estimation and WTP. A power analysis indicated that our sample of 55 participants had 80% power at an α of 0.05 to reject effect sizes larger than d = 0.395. Equivalence tests revealed that condition differences were significantly within equivalent bounds of dz = −0.395 and dz = 0.395 for outcome probability estimation (t₍₅₄₎ = −2.45, p = 0.009) and WTP (t₍₅₄₎ = 2.83, p = 0.003).

To address the possibility that differences in experienced probabilities for good and bad cues may have affected performance on measures of cue discrimination, we conducted a post hoc analysis with experienced probabilities. Here, we extracted measures of experienced probabilities (i.e., percentage positive outcomes) by feedback-timing condition and cue valence. We then calculated the difference in experienced probabilities between good and bad cues (analogous to the calculation of cue discrimination) for trials in the first block and across all blocks. Five participants did not have outcome data for all cells in the first block and were excluded from that analysis. Results revealed no significant correlations for either measure of experienced probability with any measure of postlearning cue discrimination in the immediate or delayed condition (all p > 0.13). Therefore, cue discrimination performance did not appear to differ by experienced probabilities.

Finally, a follow-up correlation analysis found strong within-subject relationships between measures of postlearning cue discrimination (Table 2). Late learning performance for each cue pair was marginally related to each of the postlearning measures. Postlearning cue discrimination abilities appeared to be best indexed by estimations of positive outcome probability, which were most strongly correlated with the other measures of cue contingency awareness, as well as optimal choice selection in the last trial block.

Table 2.

Correlations (r) between measures of postlearning cue discrimination and late-learning performance

	WTP	Probability estimation	Preference
Last block optimal choice	0.26a	0.26a	0.24a
Preference	0.75b	0.83b
Probability estimation	0.84b

^ap < 0.1.

^bp < 0.001.

Model fitting

Consistent with the aforementioned analysis of trialwise change in optimal choice selection, we observed no significant age difference in model-derived learning rates (F_(1,53) = 2.07, p = 0.16, η_p² = 0.04; Table 1). Softmax values were also similar across younger and older adults (F_(1,53) = 2.07, p = 0.16, η_p² = 0.04; Table 1), indicating that the degree to which choices were value driven (or random) was similar in younger and older adults. Age differences were observed in distributions of model goodness of fit across trials (Mann–Whitney U = 1931873, p < 0.00001, two-tailed), with higher fit values in younger versus older adults (Table 1). Critically, subsequent fMRI analyses did not reveal overall reduced PE response in older versus younger adults, indicating that group differences in model fits did not significantly diminish our ability to detect PE-related brain activation in older adults.

Lower fit values in older adults were not driven by differences in learning rates or softmax estimates across feedback-timing condition. Furthermore, we ran an alternative model that estimated learning rates separately for the immediate and delayed condition. This alternative model did not yield significant differences in learning rates by age (F_(1,53) = 0.96, p = 0.33, η_p² = 0.02), condition (F_(1,53) = 1.38, p = 0.25, η_p² = 0.03), nor their interaction (F_(1,53) = 0.68, p = 0.41, η_p² = 0.01). To investigate the possibility that younger and older adults' choices differed in softmax values by condition, we ran another version of our alternate model wherein softmax parameters were determined by condition. Softmax values under this model did not differ by age (F_(1,53) = 1.84, p = 0.18, η_p² = 0.03), condition, (F_(1,53) = 0.01, p = 0.93, η_p² = 0.00), nor their interaction (F_(1,53) = 0.39, p = 0.54, η_p² = 0.01).

fMRI whole-brain analysis: PE signals in younger and older adults across feedback-timing conditions

The initial whole-brain analysis examined PE signal combined across feedback-timing conditions. Results indicated robust PE-related activation of the striatum and HC in both younger and older adults with no significant age differences observed anywhere in the basal ganglia or MTL (Fig. 5, Table 3). Therefore, when examining neural learning signals across feedback-delay conditions, we found no age differences in the striatum or HC. Age differences were observed elsewhere in the brain such that younger adults had greater PE-related activation of the right lateral occipital cortex, whereas older adults had greater activation in the right frontal pole and paracingulate gyrus (Table 3).

Figure 5.

Whole-brain analysis of response to model-determined PEs across feedback-delay conditions in younger and older adults. Results indicated robust activation in a priori ROIs, specifically the NAcc (y = 67) and HC (y = −20) bilaterally. No age differences were observed in the striatum or MTL regions.

Table 3.

PE-related activation across feedback delay conditions (whole-brain analysis)

PE response	x	y	z	Z-max	Voxels
Younger adults
R lateral occipital cortex	26	−78	28	5.3	36975
R caudate	6	14	−8	5.0
L hippocampus	−28	−20	−22	5.0
L inferior frontal gyrus	−38	30	4	3.8	433
Older adults
L inferior temporal gyrus	−50	−52	−12	5.0	16368
L ventromedial prefrontal cortex	−10	12	−14	4.6
L caudate	−10	16	−10	4.6
Younger > older
R lateral occipital cortex	56	−68	6	3.6	400
Older > younger
R frontal pole	34	46	8	4.3	456
Paracingulate gyrus	−2	32	38	4.3	393

Locations are presented as MNI coordinates.

fMRI ROI analysis: effects of feedback timing and age

The critical test of our hypothesis required examining PE-related brain activation in the striatal and HC ROIs among younger and older adults when probabilistic feedback was either immediate (1 s) or delayed (7 s). This ROI analysis yielded no overall age differences (F_(1,53) < 1, p = 0.76, η_p² = 0.00), but did reveal a significant three-way interaction (F_(1,53) = 6.22, p = 0.02, η_p² = 0.11; Fig. 6), indicating that PE response depends on memory region, age, and feedback-delay condition. Results for younger adults yielded a replication of previous findings (Foerde and Shohamy, 2011). Specifically, we found that in young adults immediate feedback led to increased PE response in the striatum compared with the HC, whereas delayed feedback led to the opposite pattern (region × feedback timing: F_(1,25) = 21.04, p < 0.001, η_p² = 0.46). Among older adults, regional PE response did not differ by feedback timing (region × feedback timing: F < 1).

Figure 6.

PE-related activation by memory region, feedback-timing condition, and age group. As previously reported (Foerde and Shohamy, 2011), younger adults exhibited relatively greater response to PEs in the NAcc with immediate feedback and in the HC with delayed feedback. In older adults, response to PEs with immediate feedback were similar to younger adults in the NAcc and HC, but with delayed feedback there was no elevation in HC responses. Younger adults are represented by solid bars and older adults by striped bars. Error bars indicate SEM.

Further inspection of PE-related signal in the NAcc indicated that both age groups had enhanced activation for immediate relative to delayed feedback (F_(1,53) = 6.52, p = 0.01, η_p² = 0.11), with no main effect of age (F_(1,53) = 1.25, p = 0.27, η_p² = 0.02) nor an interaction of age and feedback timing (F < 1). Therefore, older adults appeared to retain feedback-timing sensitivity in NAcc-based learning signals. In contrast, feedback-timing sensitivity in the HC was only observed in younger adults, whereas older adults' HC responded similarly to immediate and delayed PEs (age × feedback-timing: F_(1,53) = 4.42, p = 0.04, η_p² = 0.08; main effects nonsignificant). These findings suggest that aging is associated with differential change in HC versus striatal mechanisms of trial-and-error learning.

Discussion

This study provides converging behavioral and brain imaging evidence that aging is associated with asymmetric decline in memory systems that support feedback-based learning. A large body of research indicates that learning from immediate feedback is supported by response to PEs in striatal dopaminergic neurons (Schultz, 2013). Delays in feedback appear to increase reliance on learning mechanisms in the declarative memory system, including the HC (Foerde et al., 2013). Indeed, the present study and previous research (Foerde and Shohamy, 2011) found that, when choice feedback is briefly delayed, response to PEs are more robustly represented in the HC compared with the striatum among healthy young adults. The present study adds to this literature by suggesting that these region–function relationships break down in normal aging and that the trajectory of age-related change is accelerated in the HC relative to the striatum.

The study yielded three main findings. First, we found that older adults' PE responses in the striatum showed greater feedback-timing sensitivity compared with PE responses in the HC. These results suggest a relative preservation of neural learning signals in the striatum versus HC in normal aging. Our behavioral analysis also suggests reduced engagement of the declarative memory system during reinforcement learning among older adults. Second, unlike younger adults, older adults failed to exhibit enhanced episodic memory for outcome stimuli in the delayed-feedback condition. Finally, although older adults had similar rates of learning to younger adults across feedback-timing conditions, they exhibited an impaired ability to subsequently transfer their learning to measures of cue preference, outcome probability awareness, and WTP. The three findings are discussed in separate sections below.

Relative preservation of striatal learning signals in healthy aging

The current study showed that striatal PEs are relatively well preserved relative to HC PEs in healthy older adults. Indeed, older adults' striatal PEs responses to immediate feedback were indistinguishable from those of younger adults. This finding appears to conflict with some previous reports of age-related declines in the striatal PEs to immediate feedback (Chowdhury et al., 2013; Eppinger et al., 2013; Samanez-Larkin et al., 2014). However, a closer examination of these previous studies suggests that decline in striatal PE responses is a characteristic of a subpopulation of older adults who also exhibit impaired behavioral learning rates relative to younger adults.

In particular, the study by Chowdhury et al. (2013) provided compelling evidence for the maintenance of striatal PEs in high-functioning older adults. They examined effects of L-DOPA on reinforcement learning performance and reward PEs in high- and low-performing subgroups of older adults using a task with trial-by-trial variation in reward probabilities based on a Gaussian random walk. One subgroup exhibited impaired task performance on placebo but performance equal to young adults on L-DOPA. Correspondingly, reward PE signal in low-performing older adults was found to be incomplete due to decline in the representation of expected value, but PE signals in this group were restored to their canonical form with L-DOPA. A second, high-performing group of older adults showed no deficits in learning performance or PE signal on placebo, but the application of L-DOPA caused deficits in both. Like this latter group, older participants in the present study did not significantly differ from younger adults in their learning performance or striatal representation of the PE signal.

Together with the previous findings, our results suggest that the integrity of striatal PE signals is maintained among healthy older adults who also exhibit intact feedback-based learning performance. This conclusion fits with prior reports that age effects on learning may include different trajectories of change (Denburg et al., 2007) and understanding age-related cognitive decline requires attention to individual differences. Our findings also suggest that the integrity of neural learning signals in the HC decline more rapidly than learning signals in the striatum, even in healthy older adults with learning performance equal to young adults. Future research should investigate whether deficits in HC learning signals among healthy older adults predicts later impairments in learning from delayed feedback, as seen in patients with MTL lesions (Foerde et al., 2013).

Age-related dedifferentiation in the HC

Substantial behavioral research suggests that age-related decline in declarative memory outpaces decline in nondeclarative memory (Craik, 1994; Nilsson, 2003; Hoyer and Verhaeghen, 2006) and such findings imply differential age effects for the neural correlates of these types of memory. A limitation of most prior work is that age differences in learning and memory have been observed using tasks with fundamentally different structures. This presents challenges for interpreting direct age group comparisons of regional engagement associated with these tasks. The present study overcomes this obstacle by implementing a single learning paradigm with only a subtle variation in feedback timing to demonstrate differential age effects on learning supported by the MTL and striatum. Under very similar conditions, we found that, compared with younger adults, older adults' neural learning signals were intact for immediate-feedback learning mediated by the striatum, but were impaired for delayed-feedback learning mediated by the HC. Therefore, our results extend the evidence of asymmetry in age-related change to different memory systems using a single probabilistic learning paradigm.

As illustrated by Figure 6, we found that aging eliminated the double dissociation between the roles of the HC and the striatum in immediate- versus delayed-feedback learning. This finding contributes to the literature on age-related dedifferentiation or an age-related weakening of functional distinctions displayed by younger adults. Dedifferentiation has been observed in the localization of processes (less localized activations in older adults; Cabeza, 2002), in the selectivity of cortical activation patterns (reduced specificity for different type of stimuli categories; Park et al., 2004), and in the dissociation between functional systems. In the memory domain, for instance, older adults show less distinctive activation patterns for explicit versus implicit learning (Dennis and Cabeza, 2011; Rieckmann et al., 2010), autobiographical versus semantic memory retrieval (St-Laurent et al., 2011), boundary- versus landmark-based spatial learning (Schuck et al., 2015), and item versus associative memory encoding (Saverino et al., 2016). As an example of dedifferentiation in memory, Dennis and Cabeza (2011) compared younger and older adults' responses to implicit learning (serial response time task) and explicit learning (semantic categorization task) in the striatum and MTL. They found that younger adults exhibited preferential recruitment of the striatum for the implicit task and preferential recruitment of the MTL for the explicit task, whereas older adults exhibited no task-by-region effect.

Our findings are consistent with respect to age-related dedifferentiation in the HC; however, we did not observe dedifferentiation in the ventral striatum. One likely explanation for this discrepancy is that, within brain regions, dedifferentiation effects are task dependent. Direct comparisons of striatum-dependent tasks (e.g., stimulus–outcome vs motor sequence learning) could clarify which tasks are most sensitive to early signs of dedifferentiation in aging.

Postlearning discrimination deficits in high-functioning older adults

Despite the fact that older adults in the current study exhibited similar learning performance to younger adults, they nonetheless showed impaired cue discrimination performance across three different measures of contingency awareness: preference, outcome probability, and WTP. Reports of age-related declines in learning generalization are common; however, older adults in such studies typically exhibit learning deficits as well (Weiler et al., 2008; Simon and Gluck, 2013; Lighthall et al., 2013). Data from the present study indicate that, even among high-functioning older adults with intact learning, age-related decline may be detected in the generalization of learning to new but closely associated contexts after only a short period of time. Such findings hold implications for research on “brain training” in aging, for which generalization and transfer effects have remained elusive (Green and Bavelier, 2008; Simons et al., 2016). Therefore, an important avenue for future research will be to identify more effective methods of enhancing and measuring learning generalization in aging populations.

Conclusions

Our results show that, whereas older adults with high cognitive function exhibit striatal learning signals that are indistinguishable from younger cohorts, the integrity of their HC learning signals shows notable age-related decline. Across behavioral and neuroimaging measures, we find that aging is associated with reduced engagement of the HC during feedback-based learning and evidence shows that changes to HC learning signals may be an early predictor of decline in learning-dependent economic decision making. Future research should identify the origins of decline in HC function among high-functioning older adults, which may include loss of structural integrity in the HC and undetected Alzheimer's disease pathology.