Stress Modulates Reinforcement Learning in Younger and Older Adults

Abstract

Animal research and human neuroimaging studies indicate that stress increases dopamine levels in brain regions involved in reward processing and stress also appears to increase the attractiveness of addictive drugs. The current study tested the hypothesis that stress increases reward salience, leading to more effective learning about positive than negative outcomes in a probabilistic selection task. Changes to dopamine pathways with age raise the question of whether stress effects on incentive-based learning differ by age. Thus, the present study also examined whether effects of stress on reinforcement learning differed for younger (age 18–34) and older participants (age 65–85). Cold pressor stress was administered to half of the participants in each age group and salivary cortisol levels were used to confirm biophysiological response to cold stress. Following the manipulation, participants completed a probabilistic learning task involving positive and negative feedback. In both younger and older adults, stress enhanced learning about cues that predicted positive outcomes. In addition, during the initial learning phase, stress diminished sensitivity to recent feedback across age groups. These results indicate that stress affects reinforcement learning in both younger and older adults and suggests that stress exerts different effects on specific components of reinforcement learning depending on their neural underpinnings.

Good choices require predicting how each option is likely to turn out. These predictions are often based on past experience with the same or similar options. Experiencing stress while making choices and learning about options is not uncommon; an estimated 80% of Americans report at least moderate levels of stress on a daily basis (American Psychological Association, 2008). In addition, learning about decision options can be stressful (e.g., comparing complex health care plans) and high-stake choices (e.g., which job offer to accept) may elicit stress given the weight of their potential outcomes. An emerging literature indicates that a brief episode of stress affects subsequent decision behavior. In particular, behavioral studies suggest that stress alters reward-motivated decision making (Cavanagh, Frank, & Allen, 2010; Lighthall et al., 2011; Lighthall, Mather, & Gorlick, 2009; Mather, Gorlick, & Lighthall, 2009; Petzold, Plessow, Goschke, & Kirschbaum, 2010; Preston, Buchanan, Stansfield, & Bechara, 2007; Starcke, Wolf, Markowitsch, & Brand, 2008; van den Bos, Harteveld, & Stoop, 2009).

In the real world, the outcomes of decision options are often uncertain – requiring that people make choices based on the perceived likelihood of positive and negative results. Thus, decisions often rely on previous learning. For instance, reinforcement learning involves making associations between choices and subsequent outcomes based on experience. Laboratory studies with young adults indicate that acute stress may improve learning about potential rewarding outcomes, but impair learning about potential aversive outcomes. For example, experiencing a psychosocial stressor impaired learning which cues were more frequently associated with negative feedback but did not impair learning which cues were more frequently associated with positive feedback (Petzold et al., 2010). Using the same probabilistic selection task, another study found that social stress was associated with better positive feedback learning, but poorer negative feedback learning, in young adults with low punishment sensitivity compared to those with high punishment sensitivity (Cavanagh et al., 2010). In addition, a study with college-age adults found an association between higher basal stress hormone levels (cortisol) and reward dependency (greater monetary gains) on the Iowa Gambling Task (van Honk, Schutter, Hermans, & Putman, 2003).

These stress effects on reinforcement learning may be related to stress and stress-hormone-related enhancements to dopamine in key reward processing regions (Abercrombie, Keefe, DiFrischia, & Zigmond, 1989; Anstrom & Woodward, 2005; Feenstra, Botterblom, & Mastenbroek, 2000; Imperato, Puglisi-Allegra, Casolini, & Angelucci, 1991; Kalivas & Duffy, 1995; Piazza et al., 1996) together with a concomitant degradation of prefrontal cortical function (Cavanagh et al., 2010; Qin, Hermans, van Marle, Luo, & Fernández, 2009). Stress has also been found to alter synaptic plasticity in dopamine neurons (Saal, Dong, Bonci, & Malenka, 2003), increase drug craving (Sinha, 2008), and, in positron emission tomography (PET) studies, enhance striatal dopamine responses to painful stressors (Scott, Heitzeg, Koeppe, Stohler, & Zubieta, 2006) and psychological stressors in stress-vulnerable individuals (Pruessner, Champagne, Meaney, & Dagher, 2004; Soliman et al., 2008). In sum, these findings support the hypothesis that stress can affect reward-related processing by increasing dopamine activity (Mather & Lighthall, 2012).

Dopamine appears to promote reward learning in part by facilitating dopamine-dependent plasticity in corticostriatal circuits, but impair aversion learning by preventing dips in dopamine that are necessary for learning to avoid negative feedback (Frank & O’Reilly, 2006; Frank, Santamaria, O’Reilly, & Willcutt, 2007; Frank, Seeberger, & O’Reilly, 2004; Waltz, Frank, Robinson, & Gold, 2007, see also Cohen & Frank, 2009 for review). Central to the current study, increases in striatal dopamine support slow, habitual learning by determining the long-term probability of positive and negative outcomes with experience and related changes to synaptic plasticity (for review Doll & Frank, 2009). Thus by increasing dopamine activity in the striatum, stress may strengthen long-term memory for positive stimulus-feedback associations but impair memory for negative stimulus-feedback associations. Different stress effects may be expected for components of reinforcement learning primarily supported by the prefrontal cortex (PFC). In particular, the PFC affects sensitivity to recent feedback as it supports rapid response to outcomes through maintenance of recent reinforcement history (Doll & Frank, 2009) and prefrontal dopamine modulates such working memory-based functions (Seamans & Yang, 2004). Previous research has found stress-related impairments to working memory (Schoofs, Preuss, & Wolf, 2008; Schoofs, Wolf, & Smeets, 2009), which appear to be mediated by stress effects on the PFC (Qin et al., 2009). Thus, stress may affect specific components of reinforcement learning differently depending on their neural mechanisms.

In this study, we were particularly interested in how stress affects reinforcement learning for older adults compared with younger adults. Dopamine function and related cognitive processes decline as people age (Bäckman, Lindenberger, Li, & Nyberg, 2010; Bäckman, Nyberg, Lindenberger, Li, & Farde, 2006) and attenuated dopamine responses to stress have been observed in older versus younger rats (Del Arco et al., 2011). These declines suggest the possibility that, in late life, dopaminergic systems will be less responsive to reward outcomes as well as to the effects of acute stressors, leading to a reduction in stress effects on reinforcement learning with advancing age.

However, there are some suggestions from previous research that reward processing is relatively well-maintained in aging. Older adults show robust brain activation in striatal regions in response to reward outcomes, indicating that processing rewards engages dopaminergic pathways in older adults (Cox, Aizenstein, & Fiez, 2008; Mell et al., 2009; Samanez-Larkin et al., 2007; Samanez-Larkin, Kuhnen, Yoo, & Knutson, 2010; Schott et al., 2007). Enhanced incidental memory for pictures seen with, or following, positive feedback has been observed in younger and older adults, with no age differences in the strength of the feedback effect (Mather & Schoeke, 2011). However, some previous findings also suggest that stress may have as much or more of an effect on behavior in late life as it does earlier. For example, older rats show similar impairing effects of stress on working memory despite attenuated dopamine responses to stress (Del Arco et al., 2011), and dopamine agonist administration affects memory and related brain activation more in older adults than in young adults (Morcom et al., 2010). Thus, even if stress-related enhancements to dopamine are diminished in older adults, they may still show robust effects of stress on dopamine-dependent cognitive processing. These findings highlight the possibility that younger and older adults may exhibit similar stress effects on reinforcement learning.

This study is the first to examine age differences in stress effects on reinforcement learning in humans. Specifically, the present study investigated performance on a probabilistic reinforcement-learning task after healthy younger and older adults experienced the cold pressor stress task (Lovallo, 1975) or a control task. Cortisol levels and ratings of stress and pain were used to evaluate reactivity to the stressor. Importantly, the cold pressor has been shown to result in similar cortisol and subjective stress responses in younger and older adults (e.g., Mather et al., 2009). The current study was designed to achieve two primary goals: 1) test the hypothesis that cold pressor stress would result in improved reward learning and impaired avoidance learning on a probabilistic selection task, and 2) determine whether stress-related alterations to reinforcement learning are similar or different for younger and older adults.

Method

Participants

The present study included 96 healthy adults; 48 younger adults age 18–34 (23 female, M_age = 23.12 yrs, SD = 4.7, M_Edu = 14.44) and 48 older adults age 65–85 years (25 female, M_age = 72.58, SD = 5.7, M_Edu = 16.44). Basic demographics and measures of cognitive function are included in Table 1. Participants were recruited for a study of stress and cognition from the University of Southern California (USC) campus and Los Angeles community via flyers, online and print advertising, snowball recruitment, and from the USC Healthy Minds Participant Pool (http://healthyminds.matherlab.com/). Individuals were not recruited if they reported using hormone birth control, sex hormone replacement medications, corticosteroid medications, beta-blocker medications, or cigarettes. None of the younger female participants reported having been pregnant within the previous six months and all reported having regular menstrual cycles (every 26–30 days). To control for the possible influence of sex or menstrual cycle on stress effects (Diamond, 2007), the number of men and women was approximately balanced, and younger female participants also reported on their current day of the menstrual cycle. Young women were then categorized as early follicular (days 1–7), late follicular (8–13), or mid-luteal (18–24). Potential participants were not recruited if they had any of the following medical conditions: coronary artery disease, angina, arrhythmia, peripheral vascular disease, diabetes, Raynaud’s phenomenon, cryoglobulinemia, vasculilitis, lupus, tingling or numbness in hands and/or feet, previous stroke, or Alzheimer’s disease. The Mini Mental Status Exam (MMSE; Folstein, Folstein, & McHugh, 1975) was administered to older participants to assess their global cognitive functioning. All older participants in the present study met a cutoff score of 24 points on the MMSE (see Table 1 for means).

Table 1.

Group characteristics

Group	Age M(SEM)	Years of Education M(SEM)	WTAR M(SEM)	MMSE M(SEM)
Younger Control	23.4 (1.0)	14.3 (.5)	41.5 (1.2)	-
Younger Stress	22.9 (1.1)	14.6 (.6)	42.4 (1.3)	-
Older Control	72.5 (1.1)	16.0 (.5)	41.5 (1.2)	27.4 (.3)
Older Stress	72.7 (1.1)	16.9 (.5)	42.1 (1.3)	27.7 (.4)

Notes. WTAR = Wechsler’s Test of Adult Reading (assessment of verbal intelligence; Wechsler, 2001). MMSE = Mini-Mental State Exam (assessment of global cognitive function; Folstein et al., 1975). Characteristics did not differ by stress group nor were there interactions of age and stress group (ps > .05).

Protocol

Study sessions were conducted between 2 and 6 pm. The study was restricted to these times to reduce the impact of diurnal variations in cortisol (Kudielka, Schommer, Hellhammer, & Kirschbaum, 2004). In terms of cognitive performance, this time of day may have been more ideal for younger than older adults overall (Yoon, May, & Hasher, 2000), but study time was consistent across stress conditions and age groups to minimize confounding effects of test time on potential age differences in stress effects. Eligibility was determined over the phone. After completing informed consent, participants drank an 8-oz bottle of water to rinse their mouths and improve hydration. Next, participants completed questionnaires for at least 10 min, then provided baseline pain ratings and saliva samples (t0). Immediately after the baseline saliva sample, participants completed either the stressor or control task followed by a second pain rating (worst pain experienced during the hand immersion task). Thereafter, participants completed a 5-min paced breathing task (results not included here) and a 9-min selective working memory task (adapted from Gazzaley, Cooney, Rissman, & D’Esposito, 2005; results not included here), followed by a second saliva sample (t1; M = 20.42 min, SD = 2.77 from manipulation start), then the learning task, and finally a third saliva sample (t2; M = 44.40 min, SD = 10.35 from manipulation start). At the end of the session, participants provided ratings of psychological stress experienced during the stress manipulation.

Stress Manipulation: Hand Immersion in Water (Cold Pressor)

Participants were randomly assigned to either the stress or control condition and did not know their condition assignment until the start of the manipulation. The percentage of participants in each condition by demographic group was similar (stress: 24.4% younger female, 24.4% younger male, 26.7% older female, 24.4% older male; control: 23.5% younger female, 27.5% younger male, 25.5% older female, 23.5% older male). The cold pressor (Lovallo, 1975) is a widely used laboratory stress manipulation in which participants are exposed to a cold stimulus for a few minutes. In the present study, cold pressor participants were asked to hold their non-dominant hand in ice water (0–5°C) for as long as possible up to 3 min. An experimenter was present during the hand immersion task and let participants know when 3 min had elapsed. All participants in the stress condition completed the cold pressor task for at least 60 sec. The control task was conducted in the same manner with warm water at 37–40° C.

Salivary Biomarker Collection and Assay

Salivary cortisol was used as our primary measure of physiological response to the stressor. To reduce variation in cortisol, participants were asked to refrain from exercise and food within 1 hr, sleep within 2 hrs, caffeine within 3 hrs, and alcohol within 24 hrs prior to the experiment. Passive drool samples (1 ml) were collected at three time points during the test session and each was assayed for cortisol. One sample was collected immediately before the stress manipulation (t0), a second sample was collected immediately before the learning task (t1), and a third sample was collected immediately after the learning task (t2). Samples were frozen immediately after testing at −30°C. Levels of cortisol found in saliva are considered good estimates of unbound levels (bioavailable) in plasma (Kirschbaum & Hellhammer, 1994). Samples were transported frozen to CLIA-certified analytical laboratories where cortisol levels were determined with high-sensitivity enzyme immunoassay kits (Salimetrics, LLC, State College, PA). Duplicate assays were conducted for each sample interval and the mean of the two values was included in the final analysis. For three sample interval values, hormone values could only be determined for one of the two duplicate assays. In these cases, the single available value was included in the final analysis.

Three participants were unable to provide 1 ml of drool at some or all three intervals (one younger control male, two older stress females). In these cases, saliva was collected using two small sponges (sorbettes).

Pain and Subjective Stress Ratings

Participants completed pain ratings on a visual analogue scale just before and immediately after the manipulation, reporting on their present pain level and the worst pain they experienced during the hand immersion task, respectively. Ratings were made on a line 10-cm long, such that 0 cm represented no pain and 10 cm represented worst possible pain. Pain response to the hand immersion task was measured by change from baseline pain (e.g., 7 cm hand immersion task – 1 cm baseline). At the end of the session, participants indicated the amount of stress they experienced during the hand immersion task on a 7-point Likert scale, such that 1 represented no stress at all and 7 represented a great deal of stress.

Probabilistic Learning Task

Participants completed a probabilistic learning task (Frank et al., 2004; Frank & Kong, 2008) involving three pairs of symbols. For each pair, they had to learn which symbol (Japanese Hiragana characters) more often predicted positive feedback and which more often predicted negative feedback (Figure 1). After meeting a learning criterion or completing six training blocks, participants completed a test without feedback. In the test, they were instructed to select the symbol they thought had the highest chance of leading to a positive outcome for novel pair combinations and previously learned symbol pairs. Symbols appeared in black 72-point font on a white background (Figure 1). The participants pressed a designated key on the right or the left side of the keyboard to indicate which stimulus they chose to be “correct.” After each choice, “Correct!” appeared in blue text or “Incorrect” in red text. If no response was made within 4 sec, the words “no response detected” appeared.

Figure 1.

Probabilistic selection task. Three symbol pairs were presented during training; each symbol was associated with a unique probability of positive and negative feedback (e.g., symbol A: 80% Correct! and 20% Incorrect). In the test phase, participants selected symbols from previously learned and novel pair combinations. Critical outcome measures from the test phase included accuracy at selecting the most rewarding symbol ( Choose -A) from novel pair combinations and accuracy at avoiding the least rewarding symbol (Avoid-B) from novel pairs.

During training, three stimulus pairs were presented (Figure 1); each pair was presented 20 times per training block. Valenced outcomes followed stimulus selections in a probabilistic manner (AB pair = 80%-correct and 20%-incorrect for A and 20%-correct, 80%-incorrect for B; CD pair = 70%-correct and 30%-incorrect for C, 30%-correct and 70%-incorrect for D; EF pair = 60%-correct and 40%-incorrect for E, 40%-correct and 60%-incorrect for F). With training, participants typically learn to choose symbols with higher probability of positive outcomes (A, C, and E) and to avoid selecting symbols with higher probability of negative outcomes (B, D, or F). Participants who met the performance criteria after any 60-trial training block advanced to the test. The criterion differed for each pair in the following manner: 65% A in AB, 60% C in CD, 40% E in EF. In the EF pair, stimulus E is correct 60% of the time, but this is especially difficult to learn. Therefore, a 40% criterion was used for this pair (simply to ensure that participants do not have a robust preference for the incorrect stimulus).

Outcome measures from the training phase were number of training blocks completed (training required to meet learning criterion for all symbol pairs) and sensitivity to recent feedback during initial training (trial-by-trial adjustments to feedback in the first block of training).¹ This second measure evaluated tendency to select the same symbol following a win (win-stay) and select the alternative symbol following a loss (lose-shift). As symbol pairs are presented randomly, the reoccurrence of a given pair is usually not immediate. For this reason, selection of a previously rewarded cue or avoidance of a previously punished cue is thought to require maintenance of recent reinforcement experiences in working memory. Note that, as sometimes the most recent feedback is correct and sometimes it is incorrect, win-stay/lose-shift behavior is neither optimal in the long run nor an effective measure of long-term learning. However, this strategy is common when uncertainty about stimulus-outcome contingencies is high (e.g., in the first training block) and provides information about the influence of feedback in a short-term context. Indeed, optimal learning implies that the impact of any one observation is highest when uncertainty is high (e.g., Behrens, Woolrich, Walton, & Rushworth, 2007), but that as learning progresses and uncertainty diminishes, the learner should rely more on their cumulative history than on any one observation. The tendency to learn faster during periods of uncertainty has been shown to relate to prefrontal cortical function dissociable from cumulative learning in this task given genetic and pharmacological manipulations (Doll et al., 2011; Frank, O’Reilly & Curran, 2006; Frank et al., 2007;) and other tasks using functional imaging (Badre, Doll, Long, & Frank, 2012; Behrens et al., 2007).

The test to assess learning from positive and negative feedback was administered immediately after the training period. In this phase, participants were tested with the same symbols in familiar and novel combinations with pairs presented at random (4 times per pair). In this way, the test phase examined generalization of the learned stimulus values. Participants were told they would not receive feedback during this phase and should select the symbol that “feels” more correct based on what they learned. Positive feedback learning was assessed by preference for the most positive stimulus A when paired with more neutral stimuli (C, D, E, F) which have on average a neutral (50%) value, whereas negative learning was assessed by avoidance of the most negative stimulus B when paired with those same neutral stimuli.

Statistical Analysis

Group effects for cortisol levels, subjective pain and stress ratings, and learning outcomes were assessed with ANOVAs that included stress condition (cold pressor, control), age (younger, older), and sex (male, female) as between-subject factors. Valence of feedback (positive, negative) was included as a within-subject factor for sensitivity to recent feedback (win-stay, lose-shift behavior) and the learning generalization test (Choose-A, Avoid-B acccuracy). Cortisol values were measured in ug/dL. To determine the impact of between-subject factors on cortisol levels, we conducted repeated-measures ANOVAs for individual cortisol levels with sample interval at three levels: t0, t1, and t2 (corresponding with baseline, post-manipulation and pre-task, and post-manipulation and post-task). Values for each sampling interval were tested for skewed distributions against a normal distribution using one-sample Kolmogorov-Smirnov (K-S) tests. Based on these tests, normal distributions could not be assumed for cortisol values (one-sample K-S tests, all three sample intervals ps < .05). Thus, log-transformed values were used for all cortisol analyses; however, raw values were included in results and figures to allow for comparisons across studies. Cortisol change scores were also calculated by subtracting individual log-transformed t0 levels from the mean of t1 and t2 levels. Partial eta squared (η_p²) values were reported to provide measures of effect size and means are presented with their standard errors (SEM). Greenhouse-Geisser values were reported for any analyses in which the homogeneity of variance assumption was not met. Additional analyses of stress effects on learning were conducted with younger females alone that included menstrual cycle phase as a covariate. Correlations were used to determine the relationship between behavioral outcome measures and cortisol change values.

Results

Cortisol

All cortisol analyses were conducted on log-transformed cortisol values. Baseline (t0) cortisol values were higher in younger adults compared with older adults, F(1,88) = 4.69, p = .03, η_p² = .05, but there were no other significant group differences or interactions in t0 cortisol values (ps > .05). We observed a main effect of stress condition on cortisol across the three sample intervals, F(1,88) = 6.80, p = .01, η_p² = .07, such that cortisol levels were higher in stressed participants. Importantly, we found a significant interaction of stress with sample interval, F(2,176) = 8.12, p = .001, η_p² = .09 (Table 2, Figure 2), confirming that the stress manipulation affected cortisol levels. This two-way interaction of stress and sample interval was not further qualified by any interactions with age or sex group, thus, the impact of stressor on cortisol was similar for younger and older adults.² Consistent with the repeated-measures analysis, a main effect of stress group was found for cortisol change ([mean of cortisol at t1 and t2] – t0 cortisol), F(1,88) = 11.03, p = .001, η_p² = .11, with no significant interaction between age and stress group, F(1,88) < 1. Examination of means indicated that cortisol increased in the stress group (M = .11, SEM = .06) and decreased in the control group (M = −.15, SEM = .06). The lack of a significant age-by-stress interaction on cortisol change observed here is consistent with a previous research eliciting salivary cortisol responses with the cold pressor (Mather et al., 2009) and laboratory psychosocial stress tasks (Kudielka, Schmidt-Reinwald, Hellhammer, & Kirschbaum, 1999; Kudielka, Schmidt-Reinwald, Hellhammer, Schürmeyer, & Kirschbaum, 2000; Nicolson, Storms, Ponds, & Sulon, 1997; Rohleder, Kudielka, Hellhammer, Wolf, & Kirschbaum, 2002).

Table 2.

Stress response and reinforcement learning outcomes by group

Group	t0 cortisol M(SEM)	t1 cortisol M(SEM)	t2 cortisol M(SEM)	Hand immersion task pain M(SEM)	Hand immersion task stress rating M(SEM)	Sensitivity to Recent Feedback M(SEM)	Choose-A Accuracy M(SEM)	Avoid-B Accuracy M(SEM)
Younger Control	.11 (.02)	.09 (.01)	.09 (.01)	−.16 (.40)	1.16 (.30)	.61 (.02)	.65 (.04)	.74 (.04)
Younger Stress	.15 (.02)	.15 (.01)	.13 (.01)	4.46 (.44)	4.64 (.32)	.53 (.02)	.80 (.05)	.66 (.05)
Older Control	.11 (.02)	.10 (.01)	.10 (.01)	−.15 (.41)	1.40 (.30)	.57 (.02)	.53 (.04)	.63 (.04)
Older Stress	.11 (.02)	.13 (.01)	.14 (.01)	5.55 (.42)	3.91 (.31)	.52 (.02)	.67 (.05)	.60 (.05)

Notes. Cortisol in ug/dL and non-log transformed for interpretability. Hand immersion task pain = Δ in pain rating from baseline to hand immersion task. Sensitivity to recent feedback represents performance from the first training block only (initial learning acquisition).

Figure 2.

Mean cortisol levels by stress condition across sample intervals. Analysis of log-transformed cortisol levels revealed that baseline values (t0) did not differ between control and stress groups, but diverged after the stress manipulation. Raw cortisol values used in display; error bars represent SEMs.

Tests for stress and control groups separately indicated a linear decrease in cortisol values for the control group across samples, F(1,47) = 10.47, p = .002, η_p² = .18, but a quadratic increase in cortisol values for the stress group such that mean cortisol levels were highest at the pre-learning task sample (t1), F(1,41) = 6.00, p = .02, η_p² = .13. Within the stress group alone, we found no age differences in cortisol change from t1 to t2, indicating that there was no significant difference in cortisol trajectories between stressed younger and older adults from the beginning to end of the learning task.³ Further examination of cortisol responses indicated that 46.7% participants in the stress group had a cortisol response ≥ 10% of their baseline level, while 25.4% of participants in the control group experienced such a cortisol change. This difference in the percentage of responders by group was significant, χ²(1, N = 96) = 7.82, p < .01. Stress group differences did not interact with age or sex (ps > .05). These results indicate that, across age and sex groups, cortisol levels were higher in the cold pressor group than in the control group during the learning task.

There was also an interaction of age and sample interval, F(2,176) = 8.17, p = .001, η_p² = .09. To better characterize the nature of this interaction, repeated-measures ANOVAs were conducted on younger and older adult groups separately. There was a linear decrease in cortisol values for younger adults across stress conditions, F(1,44) = 12.30, p = .001, η_p² = .22 (M_t0 = .13, SEM = .01; M_t1 = .12, SEM = .01; M_t2 = .11, SEM = .01), but the effect of sample interval was not significant in older adults across stress conditions, p > .05 (M_t0 = .11, SEM = .01; M_t1 = .12, SEM = .01; M_t2 = .12, SEM = .01). While the reason for this age difference is unclear, it may be related to a stronger diurnal decline in younger adults than older adults (e.g., Deuschle et al., 1997; Heaney, Phillips, & Carroll, 2010; VanCauter, Leproult, & Kupfer, 1996), or it may be due to age differences in how stressful participants found the general experimental context or the selective working memory task (before the stress manipulation).

Pain and Subjective Stress

The stressor increased ratings of physical pain relative to baseline ratings, F(1,87) = 155.85, p < .001, η_p² = .64 (Table 2). There were no other main effects or interactions by group (ps > .05). Post-experiment ratings of psychological stress experienced during the hand immersion task were higher for those in the stress condition than for those in the control condition, F(1,87) = 95.44, p < .001, η_p² = .52 (Table 2), without any interactions of age or sex with stress condition.

Number of Training Blocks Completed during Learning Acquisition

Older adults required more training blocks to learn the stimulus-outcome associations (M = 3.88, SEM = .28) than did younger adults (M = 2.85, SEM = .29), F(1,88) = 6.50, p = .01, η_p² = .07. Consistent with stress effects observed in a study of only younger adults (Petzold et al., 2010), exposure to the stressor did not modulate the number of training blocks completed. We also found no interactions of stress condition with age or sex (ps > .05). A separate analysis of younger females alone, which included cycle phase as a covariate, also found no effect of stress condition on learning acquisition (p > .05). Due to age differences in the amount of training completed, post-hoc tests for generalization of learning (test phase performance) were conducted that included number of training blocks was included as a covariate, but inclusion of this covariate did not alter the nature of any result (i.e., significance status or direction of mean differences).

Sensitivity to Recent Feedback: Win-stay and Lose-Shift Behavior

For trial-to-trial behavior in the first training block, participants were more likely to select previously correct cue (win-stay) than to avoid selecting previously incorrect cue (lose-shift), F(1,88) = 198.00, p < .001, η_p² = .69 (M_win-stay = .72, SEM = .02; M_lose-shift = .39, SEM = .01). Additionally, stress reduced sensitivity to recent feedback, F(1,88) = 17.70, p < .001, η_p² = .17 (M_stress = .52, SEM = .01; M_control = .59, SEM = .01), but did not interact with age, sex, or feedback valence (ps > .05; Table 2). That is, stress had a similar impairing effect on win-stay and lose-shift performance. We found no other significant main effects or interactions (ps > .05). We conducted follow-up analysis to determine whether stress effects on the tendency to stay with a previously selected cue depended on whether the previous choice was the better option in the long-term (i.e., accuracy; selected A) or not. For stay behavior, there was an interaction between the accuracy of the previous choice and feedback received with the previous choice, F(1,88) = 4.91, p = .03, η_p² = .05, such that staying with a previously selected cue was most likely for positive cues that yielded positive feedback that last time they were selected, (M_{pos cue-pos feedback} = .74, SEM = .02; M_{pos cue-neg feedback} = .61, SEM = .02; M_{neg cue-pos feedback} = .61, SEM = .02 M_{neg cue-neg feedback} = .56, SEM = .02); however, stress did not modulate this interaction (p > .05). Thus, during initial training, stress appeared to reduce sensitivity to recent feedback regardless of whether the previously selected cue was associated with long-term positive or negative contingencies. These results suggest that stress effects on sensitivity to recent feedback during initial training were not isolated to the selection of either just positive or negative cues.

Generalization of Learning: Choose-A (positive) and Avoid-B (negative) Accuracy

There was no main effect of stress condition on cue selection in the test phase across feedback valence types, F(1,88) = 1.89, p = .17, η_p² = .02; however, there was a significant interaction of stress condition and feedback valence for cue-selection accuracy, F(1,88) = 9.93, p = .002, η_p² = .10. Age did not modulate the effect of stress, F(1,88) = .11, p = .74, η_p² < .001, nor did it modulate the stress-by-valence interaction, F(1,88) = .10, p = .76, η_p² < .001. Separate ANOVAs for Choose-A and Avoid-B confirmed that stress enhanced Choose-A performance, F(1,88) = 10.23, p = .002, η_p² = .10, but did not significantly affect Avoid-B performance, F(1,88) = 1.63, p = .21, η_p² = .02. Age and sex did not significantly interact with stress condition to influence Choose-A or Avoid-B performance in these separate ANOVAs (Table 2; Figure 3). In addition, the stress-by-valence effect on learning was significant in young women alone whether menstrual cycle phase was included as a covariate, F(1,20) = 4.55, p = .045, η_p² = .19, or not, F(1,21) = 4.63, p = .04, η_p² = .18. Differences in performance by feedback valence were significant in ANOVAs conducted with stress, F(1,41) = 5.40, p = .03, η_p² = .12, and control groups individually, F(1,47) = 4.60, p = .04, η_p² = .09; such that performance was better for positive feedback in stressed participants but better for negative feedback in controls. Age and sex did not interact with valence type in these additional analyses, nor were there any three-way interactions of these factors. Although older adults had lower accuracy across feedback types, F(1,88) = 10.09, p = .002, η_p² = .10, stress effects on learning about positive and negative outcomes were similar in younger and older adults. Providing further support for this assertion, significant interactions of stress and feedback valence – with similar effect sizes – were obtained for both age groups when their data was analyzed separately, F(1,44)_younger = 5.33, p = .03, η_p² = .11; F(1,44)_older = 4.60, p = .04, η_p² = .10. No other significant effects or interactions were found (all ps > .05); indicating that stress selectively enhanced probabilistic learning involving positive outcomes in both younger and older adults.

Figure 3.

Stress by valence interaction for learning performance in younger and older adults. Error bars represent SEMs. Results indicate that stress-related improvements to reward learning were significant for both younger and older adults. Decreases in negative feedback learning under stress were not significant.

Additional analyses were conducted on control participants alone to address the possibility that a lack of age differences in stress effects were the result of valence-specific changes to feedback learning in aging (e.g., to examine the possibility that reward learning is less affected by aging than avoidance learning). These analyses examined whether there were age differences in reinforcement learning in non-stressed conditions, and whether these age differences depended on feedback valence. While performance was lower in older controls overall, F(1,47) = 5.90, p = .02, η_p² = .11 (Table 2 for means), there was no interaction of age group and feedback valence (p > .05). Consistent with a lack of age-by-feedback valence effects, Pearson’s correlations did not yield any significant associations between age in years and any of the valenced outcome measures within the older adult group across stress conditions or in the control condition alone (ps > .05).

Correlations for Cortisol Change and Learning Performance

Correlations were conducted for log-transformed cortisol change ([mean of cortisol at t1 and t2] – t0 cortisol) and our primary learning performance measures, which included number of training blocks completed and accuracy measures for the win-stay, lose-shift, Choose-A, and Avoid-B components of the task. Across groups, there were no significant relationships between cortisol change and these learning outcomes (ps > .05); however, there was a marginal positive correlation between cortisol change and Choose-A performance. We then examined relationships between cortisol change and learning performance in individual groups. There were no significant correlations for separate analyses of the stress group, control group, and younger adult group (ps > .05). In older adults, we observed a significant positive relationship between cortisol change and Choose-A performance across stress conditions, r(46) = .36, p = .01. This result indicated that older adults who experienced greater increases in cortisol from baseline learned better from positive reinforcement. Although not significant, the relationship between cortisol change and Choose-A performance was also positive for younger adults, r(46) = .17, p = .26 (all other correlations, p > .05). A follow-up test for differences between correlations for independent samples confirmed that these correlations were not significantly different (Fisher’s r-to-Z transformation, Z = .97, two-tailed p = .33). Thus, the positive relationship between cortisol change and positive reinforcement learning was similar across age groups, but only significant in older adults. In general though, reinforcement learning performance appeared to be more sensitive to stress exposure (as reflected in the stress group differences) than to inter-individual differences in cortisol change.

Discussion

Previous research indicates that stress increases dopamine levels in reward-processing regions of the brain and increases cravings and the likelihood of relapse among drug addicts (Mather & Lighthall, 2012). Such findings suggest that stress amplifies the impact of rewarding cues. The present study provides further evidence along these lines by showing that stress enhances learning about associations with positive outcomes, but does not enhance learning about associations with negative outcomes.

In this study, we investigated whether inducing stress with the cold pressor task would alter subsequent reinforcement learning and whether effects of stress would depend on age. During the initial block of learning acquisition, when uncertainty about stimulus-outcome associations was greatest, stress exposure resulted in reduced sensitivity to recent feedback (i.e., lower win-stay and lose-shift accuracy) and this effect was similar across age groups. Stress also altered performance on the post-learning test, such that selection of cues associated with positive feedback was enhanced by stress in younger and older adults. There was no age difference in this effect (Figure 3); selection accuracy for stimuli with a high probability of positive outcomes increased an average of 23% in younger adults and 28% in older adults. Older adults’ enhanced learning from positive reinforcement under stress could not be explained by an overall age-related decline or maintenance in positive reinforcement learning, as in the control condition there were no age-by-valence interactions in learning. Further, while we must exercise caution in interpreting a lack of age-dependent stress effects on learning, the absence of such effects did not appear to be attributable to age differences in stress response, as the magnitude of stress responses to the cold pressor was similar across age groups. We also found that, although individual cortisol responses were not strongly related to performance, there was a significant correlation between cortisol change from baseline and positive reinforcement learning in older adults, with young adults exhibiting a similar, albeit non-significant, correlation. Together these findings suggest that, in early and late adulthood alike, acute stress can enhance learning which cues are most associated with positive feedback.

This study is the first to test the effects of acute stress on reinforcement learning in older adults, but similar effects of stress on feedback-based learning have been observed in studies with young adults that involved cortisol administration or exposure to a psychosocial stressor. For example, our finding that cold pressor stress enhanced positive feedback learning is consistent with an earlier study with college-age adults that found an association between higher basal cortisol levels and reward dependency (greater monetary gains) on the Iowa Gambling Task (van Honk et al., 2003). In addition, using the same probabilistic selection task implemented in the current study, others have found that psychosocial stress leads to impairments in negative feedback learning in young adults (Petzold et al., 2010). Using the same task, another recent study found that social stress was associated with better positive feedback learning, but poorer negative feedback learning, in young adults with low punishment sensitivity compared to young adults with high punishment sensitivity (Cavanagh et al., 2010). Thus, across studies, stress enhances positive feedback learning and impairs negative feedback learning, although which effect is stronger varied depending on the study.

Although the neural underpinnings of stress effects on reinforcement learning have yet to be clarified, animal research and nuclear imaging studies with humans are in agreement that acute stress increases dopamine activity in the striatum and PFC (e.g., Abercrombie et al., 1989; Feenstra et al., 2000; Kalivas & Duffy, 1995; Scott et al., 2006). Dopamine action in the striatum appears to be critically important for generalization of stimulus-outcome learning on this task (i.e., test phase performance; see Doll & Frank, 2009 for review). For example, increasing phasic dopamine release in the striatum enhances selection of reward-associated cues but impairs selection of punishment-associated cues (Frank & O’Reilly, 2006), and enhances striatal response to prediction error during feedback-based learning, with increased activation predicting better performance in a later test phase (Jocham, Klein, & Ullsperger, 2011). Dopamine treatment also increases the difference in ventromedial PFC response to positive versus negative feedback cues after learning acquisition (Jocham, et al., 2011). Thus, by increasing dopamine action in the striatum and PFC, stress may have opposite effects on learning from positive versus negative outcomes.

In contrast with the slow accumulation of learning about cues that relies on striatal regions (Yin & Knowlton, 2006), trial-to-trial win-stay and lose-shift decisions appear to depend more on prefrontal working memory processes, as they require maintenance of recent selections and outcomes during intervening trials. The dorsolateral PFC appears to support win-stay behavior in decision making when potential outcomes are unpredictable, while the anterior cingulate is important in mediating lose-shift behavior at both high and low levels of predictability (Paulus, Hozack, Frank, & Brown, 2002). Although age differences have not been examined, stress appears to alter brain activation in both of these regions (Pruessner et al., 2008; Wang et al., 2005), and stress-related impairments to working memory have been linked to dorsolateral PFC disruption (Qin et al., 2009). Accordingly, we found that stress diminished the expression of win-stay and lose-shift behavior during the initial training block of the probabilistic selection task with no differences in stress effect by feedback valence. Others have found similar stress-related impairment to working memory tasks without reinforcement (e.g., Schoofs et al., 2008; Schoofs et al., 2009). The current study adds the novel finding that older adults also experience stress-related impairments to working-memory-related performance during reinforcement learning acquisition; however, these effects do not appear to be valence specific. Given these earlier findings, our results suggest that acute stress may interfere with initial reinforcement learning in younger and older adults by altering maintenance of stimulus-outcome associations, perhaps via PFC disruption.

The current study found no age differences in physiological responses to stress (cortisol reactivity) or in stress effects on reinforcement learning between healthy younger and older adults. The present finding suggests that, despite age-related declines in various aspects of the dopamine system, healthy older adults still experience robust effects of stress on reward learning. Other studies have observed greater stress-induced changes to risk taking (Mather et al., 2009) and an age-related increase in sensitivity to pharmacological dopamine manipulation on episodic memory (Morcom et al., 2010). These earlier human studies, taken with the findings from the current study, present the possibility that the magnitude of stress effects and other dopamine-modulators on cognition and underlying neural circuitry may be maintained or increase in normal aging. This may be due to adaptations of the system to lower levels of dopamine, such that less of a perturbation is needed to have the same impact. However, given that we did not measure dopamine in our study, we must also leave open the possibility that the stress effects on learning about positive outcomes operate via mechanisms unrelated to dopamine. Such other mechanisms may include stress effects on information processing related to emotional goals (e.g., regulating mood by focusing on rewarding outcomes).

With respect to cortisol, we found that change in cortisol was weakly related to positive reinforcement learning in older adults. The correlation between cortisol change and positive reinforcement learning was also positive, although not significant, in young adults alone, and the correlations for the older and younger group were not statistically different from each other. As these correlations were weak and observed across stress conditions, we cannot make strong claims that inter-individual differences in cortisol change affected performance. However, the lack of age differences in the relationship between cortisol change and memory effects is consistent with a previous study reporting similar effects of cortisol administration on younger and older adults’ declarative memory (Wolf et al., 2001). We did not find any significant relationships between cortisol change and any of our other cognitive outcomes.

A caveat to the current study is that the design may not have been optimal for observing age differences in stress effects on reinforcement learning. For example, it is possible that the cold pressor can elicit age-by-stress interactions for some cognitive tasks (e.g., risk tasks under time pressure, Mather et al., 2009) but not others. The cold pressor effect is moderate (the effect size of the cortisol difference between stress groups was η_p² = .09); it may be that more powerful stressors or direct pharmacological manipulation of stress hormone levels will elicit age differences in the impact of stress on reinforcement learning. It is also possible that older adults experienced additional stress from the selective working memory task (completed prior to the reinforcement-learning task). However, any age differences in stress elicited by the selective working memory task would have been present across stress conditions. Thus, differences in reinforcement learning between the cold pressor stress condition and the control condition are due specifically to the stress induced by the cold pressor, not the selective working memory task. A lack of power may have also prevented observation of age differences in stress-by-feedback valence effects; a limitation that can be addressed in future studies. However, the conclusion that the interaction of stress and feedback valence does not differ for older and younger adults is supported by the very small effect size obtained for the age-by-stress-by-feedback valence interaction (η_p² = .001). Power calculations were conducted with G-Power version 3.1.2 for a repeated-measures design with 2 levels, 4 groups, and a correlation between repeated measures of .01 (for Choose-A and Avoid-B scores), and the effect size for the age-by-stress-by-feedback valence interaction of .001 (the latter two parameters obtained from the current study). For α = .05 and power = .90, a sample size of 7012 subjects would be required to detect an interaction of age, stress condition and feedback valence. Thus, it seems that stress exerts different effects on learning about positive compared to negative feedback, and these stress effects are similar in early and late adulthood.

A remaining issue, however, is that the present study could not determine whether effects of stress are linear or not, as we included a moderate stressor and did not manipulate the intensity of the stress induction. Some previous research suggests that cognitive performance and stress, as well as cognitive performance and dopamine, have inverted-U relationships (see Bäckman et al., 2006; Lupien, McEwen, Gunnar, & Heim, 2009). Future studies may help determine whether stronger manipulations of stress lead to a decline in reward learning performance, consistent with an inverted-U relationship. With regard to stressor type, the two previous studies (Cavanagh et al., 2010; Petzold et al., 2011) examining effects of stress on the probabilistic selection task used psychosocial stressors and observed significant impairing effects of stress on aversion learning (however, only in those with low-punishment sensitivity in Cavanagh et al., 2010), whereas we only found trends in that direction for aversion learning. One possibility is that psychosocial stressors have a greater impact on affect and emotion regulation goals than physiological stressors like the cold pressor. Specifically, psychological stressors like the Trier Social Stress Test (Kirschbaum, Pirke, & Hellhammer, 1993) involve negative social feedback. Aftereffects may include a motivation to avoid attending to negative feedback – resulting in poorer aversion learning. Furthermore, psychological and painful stressors affect dopamine transmission in different aspects of the striatum (Scott et al., 2006), which may result in different behavioral biases (e.g., toward reward versus away from punishment). PET studies conducted with healthy populations may help determine the specific neural mechanisms of stress effects on learning from positive and negative feedback, and whether there are age differences in these mechanisms.

Although not the main focus of our study, it is also interesting to compare the relative influence of positive versus negative feedback on learning in younger and older adults in the control condition. Research on this topic is mixed (for review, Eppinger, Hämmerer, & Li, 2011), with some previous studies finding better learning from positive than negative feedback in older adults relative to younger adults (Nashiro, Mather, Gorlick, & Nga, 2011; Wood, Busemeyer, Koling, Cox, & Davis, 2005), others finding no age differences in learning from positive and negative feedback in younger and older adults (Eppinger, Herbert, & Kray, 2010; Pietschmann, Endrass, Czerwon, & Kathmann, 2011), and still others finding better learning from negative relative to positive feedback for older adults compared with young adults (Eppinger & Kray, 2011; Hämmerer, Li, Müller, & Lindenberger, 2011) and for older compared with younger seniors (Frank & Kong, 2008). Recent studies have also observed age differences in brain blood oxygen level-dependent activation during loss anticipation (Samanez-Larkin et al., 2007), and in the electrophysiological correlates of feedback-related negativity signals (Eppinger, Kray, Mock, & Mecklinger, 2008; Hämmerer, Li, Müller, & Lindenberger, 2011). The current findings are consistent with those studies reporting no significant relationship between age and whether learning was better from positive or negative outcomes. Although age is associated with dopamine declines and lower dopamine levels are associated with better learning from negative than positive feedback, there are some reasons older adults may not exhibit this pattern of avoidance learning dominance. First, striatal response to rewarding feedback is maintained in older adults (Cox et al., 2008; Mell et al., 2009; Samanez-Larkin et al., 2007; Samanez-Larkin et al., 2010; Schott et al., 2007), indicating that the dopamine system continues to respond to rewarding feedback in older age. And second, aging is associated with an increased focus on positive information (Mather & Carstensen, 2005), which may selectively improve memory for positive stimulus-outcome associations in the face of dopamine declines. Further research is needed to sort out when age differences in valence bias for reinforcement learning will be seen and when they will not. For example, neuroimaging research may reveal whether structural integrity of the corticostriatal pathway mediates age differences in reinforcement learning.

In summary, this study revealed that acute stress enhanced learning involving positive feedback in both younger and older adults. Stress is present in everyday life, and thus, often accompanies everyday decision making. These findings suggest that stress may affect abilities to quickly adapt to positive and negative feedback when learning out choice outcomes. In addition, choices made under stress may be biased toward associations with positive or rewarding experiences – a bias which may hold implications for decisions about finances, social interactions, and health behaviors. Decisions in these domains can significantly impact well being and must be made from early adulthood until very late in life; thus, further research on the mechanisms driving the effect of stress on reinforcement learning is likely to have application potential across adulthood.