Necessary Contributions of Human Frontal Lobe Subregions to Reward Learning in a Dynamic, Multidimensional Environment

Abstract

Real-world decisions are typically made between options that vary along multiple dimensions, requiring prioritization of the important dimensions to support optimal choice. Learning in this setting depends on attributing decision outcomes to the dimensions with predictive relevance rather than to dimensions that are irrelevant and nonpredictive. This attribution problem is computationally challenging, and likely requires an interplay between selective attention and reward learning. Both these processes have been separately linked to the prefrontal cortex, but little is known about how they combine to support learning the reward value of multidimensional stimuli. Here, we examined the necessary contributions of frontal lobe subregions in attributing feedback to relevant and irrelevant dimensions on a trial-by-trial basis in humans. Patients with focal frontal lobe damage completed a demanding reward learning task where options varied on three dimensions, only one of which predicted reward. Participants with left lateral frontal lobe damage attributed rewards to irrelevant dimensions, rather than the relevant dimension. Damage to the ventromedial frontal lobe also impaired learning about the relevant dimension, but did not increase reward attribution to irrelevant dimensions. The results argue for distinct roles for these two regions in learning the value of multidimensional decision options under dynamic conditions, with the lateral frontal lobe required for selecting the relevant dimension to associate with reward, and the ventromedial frontal lobe required to learn the reward association itself.

SIGNIFICANCE STATEMENT The real world is complex and multidimensional; how do we attribute rewards to predictive features when surrounded by competing cues? Here, we tested the critical involvement of human frontal lobe subregions in a probabilistic, multidimensional learning environment, asking whether focal lesions affected trial-by-trial attribution of feedback to relevant and irrelevant dimensions. The left lateral frontal lobe was required for filtering option dimensions to allow appropriate feedback attribution, while the ventromedial frontal lobe was necessary for learning the value of features in the relevant dimension. These findings argue that selective attention and associative learning processes mediated by anatomically distinct frontal lobe subregions are both critical for adaptive choice in more complex, ecologically valid settings.

Introduction

Optimal decision making requires attending to cues that reliably predict reward, often on a background of distracting or even misleading information. A trip down the grocery aisle reveals the daunting nature of this problem. For example, fruits vary on multiple dimensions (e.g., color, texture), each with their own features (e.g., red or green, soft or firm) that could guide choice. However, some dimensions are more relevant than others, or may be physically salient, but entirely irrelevant (e.g., the color of the packaging). How learning is optimized in such multidimensional settings has long been a challenging problem for normative computational models (Sutton and Barto, 1998) and animal learning theory (Pearce and Mackintosh, 2010), and is increasingly recognized as an important question in decision neuroscience (Niv et al., 2015).

Multidimensional learning requires attributing outcomes to features that are predictive of rewards, while ignoring nonpredictive features. However, stimuli that have been spuriously correlated with outcomes, or are more salient than the predictive stimulus, may be maladaptively credited with predictive value (Pavlov, 1927; Wilson and Niv, 2011). Attentional mechanisms aid learning in such settings by selecting between relevant and irrelevant features (Mackintosh, 1975; Kruschke, 2003; Rombouts et al., 2015).

Within the frontal lobes, the lateral prefrontal cortex (PFC) has been implicated in selecting between relevant and irrelevant features of the environment for goal-directed behavior (Desimone and Duncan, 1995; Miller and Cohen, 2001). Lesions to this region in monkeys and humans disrupt attentional shifting between stimulus dimensions (Milner, 1963; Dias et al., 1996) and attention to nonsalient, but task-relevant, stimulus dimensions (Rossi et al., 2007; Gläscher et al., 2012; Tsuchida and Fellows, 2013). Lateral PFC lesions in humans or monkeys do not affect learning of stimulus–reward associations when cues are simple and unvarying (Dias et al., 1996; Tsuchida et al., 2010). However, hemodynamic signals in the lateral PFC correlate with attentional demands in a multidimensional learning environment (Niv et al., 2015). Whether this region is required for reward learning under these attentionally demanding conditions is unknown.

In contrast, the ventromedial frontal lobe [VMF, here referring to both orbitofrontal (OFC) and ventromedial PFC (vmPFC)] is necessary for optimal learning of dynamic stimulus–reward relationships. VMF damage impairs the ability of monkeys and humans to learn probabilistic and reversing stimulus–reward associations for simple predictive cues (Butter, 1968; Fellows and Farah, 2003; Hornak et al., 2004; Tsuchida et al., 2010). Unlike lateral PFC damage, VMF damage does not affect attentional set shifting, or the ability to ignore salient, task-irrelevant stimulus dimensions (Milner, 1963; Dias et al., 1996; Gläscher et al., 2012; Tsuchida and Fellows, 2013). This region has thus not been considered as playing a role in attention.

However, recent work suggests that the VMF may contribute to attentional selection during value-based learning and decision making (Walton et al., 2015). VmPFC value signals measured with fMRI are sensitive to the behavioral relevance of option dimensions during choice (Lim et al., 2013; Hunt et al., 2014), and the VMF is critical for attentional priming of rewarded stimulus features (Vaidya and Fellows, 2015). The interaction of the vmPFC and lateral PFC is also correlated with selection of relevant stimulus features during decision making (Hare et al., 2011; Chau et al., 2014). Together, these studies raise questions about the necessary contributions of these regions during value-based choice between complex stimuli.

The aim of this study was to test the necessary contributions of frontal lobe subregions to optimal learning in a multidimensional task environment. Patients with frontal lobe damage completed a probabilistic reversal learning task where stimulus options were defined by three dimensions, only one of which predicted feedback. We focused on trial-by-trial behavior, testing the effects of frontal lobe damage on feedback attribution. Left lateral frontal damage increased attribution of rewards to irrelevant dimensions, and decreased attribution to the relevant dimension. VMF damage also affected learning in the relevant dimension, but not reward attribution to irrelevant dimensions. These results demonstrate apparently distinct roles for these regions during learning in a complex environment.

Materials and Methods

Subjects.

Forty-five patients with focal lesions involving the frontal lobes were recruited for this study, 36 from the Cognitive Neuroscience Research Registry at McGill University, and nine from the Center for Cognitive Neuroscience at the University of Pennsylvania (Fellows et al., 2008). They were eligible if they had a fixed lesion primarily affecting the frontal lobes. Patients were categorized into groups a priori following standard anatomical boundaries (Stuss et al., 2005), based on the location of their damage, assessed on their most recent MR or computed tomography imaging, by a neurologist blind to task performance. Patients with lesions primarily affecting the VMF were identified first, as the primary region of interest. The remaining patients were then subdivided further into dorsomedial frontal (DMF), left lateral frontal (LLF), and right lateral frontal (RLF) groups. Lesions were manually registered to a common brain space (MNI brain) to allow overlap images to be generated. The overlap images for the four anatomically defined groups are shown in Figure 1.

Figure 1.

Representative axial slices and sagittal view of overlap of lesion tracings on the MNI brain. Rows show overlap in the DMF, LLF, RLF, and VMF groups. Numbers above slices indicate z coordinates of axial section in MNI space. Colors indicate number of subjects with overlapping lesions, as indicated by the color bar. L, Left; R, right.

One VMF patient found the task too difficult and asked to stop the experiment after the first block. Three other patients were excluded from further analysis because their choices indicated that they were following idiosyncratic rules unrelated to the task instructions or feedback: one patient with DMF damage appeared to be deliberately avoiding previously rewarded features, one patient with VMF damage chose the green stimulus on nearly every trial, without any regard to feedback or any other task dimensions, and one DMF patient simply chose stimuli in one color throughout the first two and half blocks of the experiment, alternating the color between blocks. A fifth patient was excluded from analysis when it was found that the boundaries of her lesion could not be accurately established. The final sample included 40 patients with frontal lobe damage, 19 males and 21 females.

Damage to the DMF was caused by tumor resection in nine cases and ischemic stroke in one case. Damage to the LLF was caused by ischemic stroke in five cases, aneurysm in one case, and tumor resection in one case. Damage to the RLF was caused by tumor resection in four cases and ischemic stroke in two cases. Damage to the VMF was caused by tumor resection in nine cases, hemorrhagic stroke in four cases, aneurysm in three cases, and ischemic stroke in one case. Patients were tested in the chronic phase of brain injury. The median time since brain injury (defined as the onset of symptoms for stroke or aneurysm rupture, and the date of surgery for tumor resection) was 8.25 years (range, 3.5 months to 48.1 years). Nineteen patients were taking ≥1 psychoactive medications, most commonly an anticonvulsant or antidepressant. There was a marginally significant difference in the frequency of psychoactive medication usage between groups (χ² test of independence: χ² = 7.82₍₃₎, p = 0.05), with more patients in the DMF group taking such medications (80%) compared with other patient groups (VMF, 47%; RLF, 33%; LLF, 14%).

Age-matched and education-matched healthy control subjects (N = 21; 8 males and 13 females) were recruited through local advertisement in Montreal. They were free of neurological or psychiatric disease and were not taking any psychoactive medication. They were excluded if they scored ≤26 on the Montreal Cognitive Assessment (Nasreddine et al., 2005). Mean performance on this test was 28.3 (SD = 1.5). All subjects provided written informed consent in accordance with the Declaration of Helsinki. The study protocol was approved by the institutional review boards of both participating centers.

Neuropsychological screening.

All patients completed neuropsychological screening to test general cognitive functioning. Patients at both institutions underwent screening for hemispatial neglect, using the Posner cueing task (Posner, 1980), and a circle cancellation task (Marsh and Hillis, 2008). These patients also completed tests of working memory (backwards digit span; Lezak et al. (2012)), and semantic and phonemic verbal fluency (Fluency-F, Animals; Benton et al., 1989). Patients recruited from the Cognitive Neuroscience Research Registry at McGill University also completed a test of visual memory for faces without explicit instructions (incidental memory; Bower and Karlin, 1974), and a test of the ability to understand and follow one-step, two-step, and three-step verbal instructions (sentence comprehension, similar to the Token Test; De Renzi and Vignolo, 1962).

Apparatus.

The experimental task was programmed using E-Prime 1.2 (Psychology Software Tools). Twenty-nine patients and all 21 controls were tested at the Montreal Neurological Institute. They saw stimuli presented on a 19 inch monitor (Dell) and responded using the leftmost and rightmost keys on a serial response box (Psychology Software Tools). Eleven patients tested at the University of Pennsylvania or during in-home visits in the greater Philadelphia and Montreal areas performed the experiment on a 13.5 inch laptop (Fujitsu) and used the left and right arrow keys of the laptop keyboard for their responses. Two patients with VMF damage used the index and middle fingers of their right hands due to weakness in the left hand. All other subjects responded bimanually.

Procedure.

Subjects completed a multidimensional, probabilistic reversal learning task. In this task, subjects chose between two compound stimuli, described in the instructions as “cards,” that were defined by a shape (a backward-facing or forward-facing C) and color (blue or green) and appeared on the left and right sides of the screen. Subjects chose between these two options on each trial by pressing the corresponding right or left key. Critically, only the shape inside the card was relevant to whether or not subjects would be rewarded. One shape was associated with a 75% chance of reward, while the other shape was associated with a 25% chance of reward. Once subjects had chosen the probabilistically more rewarding shape in 10 of the previous 12 consecutive trials (i.e., once more than would be achieved by simply using a win–stay, no win–shift strategy, on average), the reward probabilities for the two shapes would reverse. The color and side of the stimulus had no predictive value for determining feedback. Mirror shapes were chosen as the relevant stimulus to place greater demands on selective attention. Mirror shapes are relatively difficult to discriminate (Cooper, 1975; Corballis and McLaren, 1984), and thus likely to be less salient than the features of the color and side dimensions, a notion supported by associative learning work in monkeys (Baxter and Gaffan, 2007).

At the beginning of the experiment, subjects were given the following instructions: “You are going to play a card game. You will see two cards on either side of the screen. Your job is to choose one of the cards. You will either receive points or get nothing after making a choice. Only the shape inside the card is relevant to whether you will be rewarded. The color is irrelevant. One shape is better than the other, however no shape is always rewarding. So, you should try to STICK WITH one shape. Be careful, because the better shape may also change. The points you receive will be converted to money at the end of the experiment.” These instructions were deliberately very similar to instructions used in previous work from our group using a probabilistic reversal learning task (Tsuchida et al., 2010; Camille et al., 2011), with additional information added regarding the relevant and irrelevant stimulus dimensions unique to this experiment. After reading these instructions, subjects were asked to explain what they were supposed to do to the experimenter to check their understanding.

On each trial, subjects would first see a central fixation point for 500 ms, followed by presentation of the two options (i.e., cards) on either side of the screen for 600 ms. These stimuli were subsequently replaced by a mask (a black “O” card) for 500 ms. Subjects were allowed to respond at any point in this 1100 ms window, from stimulus presentation to the end of the mask presentation. Following the mask, subjects were shown a feedback screen that lasted 800 ms. Subjects were explicitly instructed that they could still respond after the cards had disappeared, until they saw feedback. The relatively brief stimulus presentation was intended to increase the selective attention demands of the task.

On the feedback screen in each trial, subjects saw the number of points they had won in the trial (either 0 or 40) and a running total of the number of points they had earned thus far in the center of the screen. The option selected on that trial was presented above the total. On trials where subjects won points, they would hear a high-pitched “ding” sound and see the text “Win!” written in yellow below the total. On trials where no points were won, subjects would not hear any sound or see any other text. If no response was made, or if the response did not occur in time, subjects simply saw the total number of points with the text “Respond faster” written above. At the end of the experiment, subjects received a small monetary reward based on the proportion of trials in which they chose the high-reward shape, in addition to compensation for their time.

Subjects completed four blocks of this task, with 200 trials in each block for a combined 800 trials total. Two control subjects, two VMF patients, one DMF patient, and one RLF patient chose stimuli in only one color in nearly all trials of the first block, without regard to the feedback. These subjects were reinstructed after the end of the block, and responded to feedback in subsequent blocks. The data from this first block were excluded from analysis in these subjects.

Voxel-based lesion symptom mapping.

In a secondary analysis, we used voxel-based lesion symptom mapping (VLSM) to test where brain damage was associated with reduced win–stay behavior for the relevant shape dimension, and increased win–stay behavior for irrelevant color and side dimensions, as measured by parameters from a multiple logistic regression analysis. Given that VLSM analysis removes matching for demographic factors that may be related to performance, we used multiple linear regression to test whether these three variables were related to age or education in control subjects, and applied corrections to parameters where necessary (see Statistical analysis). These parameters were then used in the VLSM analysis using Non-Parametric Mapping (version June 6, 2013) software (freely available at http://people.cas.sc.edu/rorden/mricron/index.html). This analysis compared patients with damage at each voxel with the rest of the patient group using nonparametric Brunner–Munzel (BM) tests (Brunner and Munzel, 2000). Only voxels damaged in ≥3 patients were included in this analysis, in line with past work from our group and from others (Coulthard et al., 2008; Haramati et al., 2008; Tsuchida et al., 2010). Due to the large number of comparisons involved in this test, a null distribution of BM Z-scores was calculated from the same dataset using permutation tests (3000 permutations) to find an appropriate threshold for the adjusted α rate (Nichols and Holmes, 2002). Permutation tests provide an assumption-free means for controlling the rate of false positives after multiple comparisons, with more statistical power than overly conservative methods, such as the Bonferroni correction (Kimberg et al., 2007). Images of the results of this analysis were created using the software MRICron. A cluster extent threshold of k = 50 voxels was applied to the statistical maps from this analysis, and all reported effects.

Statistical analysis.

Demographic measures for patient groups (age, years of education, and Beck Depression Inventory-II) and performance on neuropsychological screening tests were compared with controls using uncorrected between-subjects t tests. American National Adult Reading Test (AMNART)-estimated IQ and lesion volume in patient groups were compared with uncorrected, nonparametric Mann–Whitney U tests, as these values were not normally distributed.

Group differences in performance on neuropsychological screening tests were assessed using one-way ANOVAs, or nonparametric Kruskal–Wallis tests (for circle cancellation, sentence comprehension). Where there were any significant, or trending, effects of group status, post hoc uncorrected between-subjects t tests, or Mann–Whitney U tests, were used to assess these differences.

To assess basic task performance, a χ² test of independence was used to examine whether the overall frequency of trials (misses and choices of the high-reward or low-reward probability shapes) differed between groups. Effects of group status on the frequency of choices of the more rewarding shape (excluding missed responses), and the frequency of reversals per block, were tested with a Kruskal–Wallis test. The effect of experimental block on the frequency with which subjects chose the high-reward probability shape was tested using a mixed-measures ANOVA. An arcsine transformation was used to ensure that these frequency values were normally distributed. Postreversal accuracy was tested using a mixed-measures ANOVA to examine the effects of group status, and postreversal trial number on the frequency with which subjects chose the high-reward probability shape. Subjects who achieved <3 reversals were omitted from this analysis (three controls, two DMFs, one LLF, one RLF, and one VMF). Again, an arcsine transformation was used to ensure that these frequency values were normally distributed.

Effects of the trial-by-trial repetition of the color and side of the previously rewarded shape on reaction time were tested with a mixed-measures ANOVA, with the color and side repetition of the chosen shape as within-subjects factors and group status as a between-subjects factor.

Generalized estimating equations (GEEs) as implemented in SAS (version 9.4, SAS Institute) were used to examine trial-by-trial behavior. Like mixed-regression models, this method takes full advantage of the relatively large number of observations per subject in this experiment while estimating group-level parameters. While the results of mixed-effects models are sensitive to the assumptions made about the underlying correlation structure of the data, GEE analyses are comparatively more robust and hence preferable for analysis of this type of data (Hubbard et al., 2010). To test the effects of group status on staying with the rewarded features in each stimulus dimension, we modeled the probability that subjects chose the left option in trials immediately following a reward, as a function of whether the shape or color chosen in the last trial were on the left, and whether subjects chose the left side in the last trial. We first estimated parameters for this model in the control group separately to examine trial-by-trial effects of reward in healthy subjects in this task. Next, we analyzed the full dataset, including main effects of group status and interactions between each of these variables and group status. All effects of group status and interactions with group status were referenced to the parameter estimates for control subjects. An identical analysis was used to test trial-by-trial effects for switching away from features paired with negative feedback (i.e., absence of rewards) in the previous trial.

Effects of positive and negative feedback history were also analyzed using a GEE model. We calculated the frequency with which features in the left and right options in a given trial were associated with positive (rewards) and negative (no reward) feedback in the last four trials. We then estimated how the difference in frequency of positive and negative feedback between the features in the left and right options was related to the probability of choosing the left or right option. This analysis tested how the relative value within each dimension, as learned in recent trials, influenced subjects' choices. As in the trial-by-trial analysis, we first estimated these parameters in the control group alone before testing the interaction between group status and positive and negative feedback history for each stimulus dimension. Any significant differences between patient groups and controls in the main GEE analyses were followed up by assessing differences between GEE parameter estimates of patient groups using two-way Bonferroni-corrected t tests (α = 0.008 for p = 0.05).

We also explored whether the information value within each dimension affected trial-by-trial attribution of feedback to the same dimension and to other stimulus dimensions. We operationalized information value here as the absolute value of the difference in reward frequency for features within each stimulus dimension in the last four trials (similar to choice history). Given that subjects were influenced most by the association of features with rewards, rather than the absence of reward, and the effects of group status were principally related to changes in reward attribution, we focused on information value of positive feedback, not negative feedback. In three separate GEE models, we estimated the probability that subjects would stick with a previously rewarded shape, color, or side, given the information value within each dimension. As in the previous GEE analyses, we first tested these effects in healthy control subjects before analyzing the effects of group status.

To acquire individual estimates of win–stay behavior for each subject for use in VLSM analysis, we performed multiple logistic regression analyses on subject-level data. Win–stay behavior was modeled the same way as in the GEE analysis at the group level (i.e., estimating how the probability of choosing the left or right option depended on whether these options contained features previously rewarded in the last trial). Patient parameters for win–stay behavior in the side and shape dimension were adjusted for age and education level, as these parameters were significantly predicted by these factors in the control group in multiple linear regression analyses. Education was categorized as high school or less, some college to a college degree, and some graduate education to a graduate degree. To adjust for these effects, we calculated predicted parameters for win–stay behavior in these dimensions for each patient based on the relationship between these parameters and education level and age in the healthy control group. The residuals between these predicted parameters and parameters estimated from individual patients' behavior were then used in the VLSM analysis.

Using nonparametric Spearman ρ correlations, we also tested for associations between these measures of trial-by-trial task performance and neuropsychological screening measures to ensure that performance was not explained by any coexisting language or long-term memory deficits that might affect the ability to follow instructions.

Data availability.

The complete behavioral dataset for this experiment is available for download (see Notes).

Results

Demographic information and lesion volumes are provided in Table 1. VMF and DMF groups scored higher than controls on the Beck Depression Inventory-II test, but lesion groups did not differ from each other on this measure. VMF-damaged patients also scored lower than controls and DMF-damaged patients on the AMNART-estimated IQ test (though this measure was not available for all subjects). There were no differences between groups in age, years of education, or lesion volume.

Table 1.

Demographic information for controls and for patients with PFC damage^a

Group	Age (years)	Sex (male/female)	Education (years)	BDI-II	AMNART IQb	Lesion volume (cc)	Handedness (right/left/both)
Control (N = 21)	59.7 (10.9)	8/13	15.1 (3.4)	4.4 (4.3)	119 (4)	—	18/2/1
DMF (N = 10)	57.8 (5.1)	5/5	15.0 (4.3)	8.9 (6.2)*	120 (5)	14 (3–83)	8/2/0
LLF (N = 7)	63.3 (10.8)	4/3	14.7 (3.0)	4.7 (3.3)	117 (10)	17 (5–47)	7/0/0
RLF (N = 6)	57.5 (6.2)	3/3	15.3 (3.6)	8.5 (8.1)	120 (6)	24 (22–96)	5/1/0
VMF (N = 17)	60.9 (10.2)	7/10	14.5 (3.2)	9.8 (6.9)*	110 (7)∧	20 (7–192)	15/1/1

^aValues represent means with SDs in parentheses, except for lesion volume where the median and range are provided.

^bNot all subjects were able to complete the AMNART.

*P < 0.05, two-tailed t test against control scores, uncorrected.

∧P < 0.05, two-tailed Mann–Whitney U test against control and DMF scores, uncorrected.

Neuropsychological screening results are reported in Table 2. There were no group differences in performance on tests of language comprehension, memory, executive function, or spatial attention (P's > 0.05, uncorrected between-subjects t tests, Mann–Whitney U tests). The RLF group scored higher than other patient groups on a test of verbal fluency.

Table 2.

Performance on neuropsychological screening tests for controls and for patients with PFC damage^a

Group	Posner Cueing (uncued–cued) left/right (ms)	Circle cancellation % missed (left/right)	Circle cancellation % false alarms	Fluency: animals	Fluency: F	Backwards digit span	Incidental memory P (Correct)∧	Sentence comprehension P (Correct)∧
Control (N = 21)	81.2 (37.8)/54.2 (34.5)	—	—	—	—	—	—	—
DMF (N = 10)	77.6 (59.5)/72.8 (49.9)	0.4 (0.8)/1.1 (2.4)	0.06 (0.2)	19.1 (8.6)	8.9 (5.8)*	2.3 (1.3)	0.78 (0.13)	0.99 (0.88–1.00)
LLF (N = 7)	65.9 (31.4)/86.8 (32.0)	0.5 (1.0)/ 0.7 (1.2)	0.71 (1.9)	17.8 (7.9)b	8.2 (3.7)b*	2.7 (1.2)	0.81 (0.12)	0.96 (0.83–1.00)
RLF (N = 6	50.6 (56.3)/32.8 (39.0)	1.7 (2.9)/1.0 (2.0)	0.31 (0.5)	22.0 (2.7)	14.5 (3.6)	2.8 (1.2)	0.82 (0.11)	0.95 (0.77–1.00)
VMF (N = 17)	72.8 (37.5)/ 57.2 (37.5)	1.6 (2.0)/1.9 (2.7)b	0.08 (0.2)	17.7 (3.5)c	10.2 (4.4)c*	3.2 (1.3)	0.85 (0.11)	0.95 (0.77–1.00)

^aValues represent means with standard deviations in parentheses, except for sentence comprehension where the median and range are given instead.

^bData missing from one patient.

^cData missing from two patients.

*P ≤ 0.05, against RLF two-tailed t test, uncorrected.

∧Montreal patients mostly.

Subjects performed the probabilistic reversal learning task shown in Figure 2. This task was designed to test how subjects attributed feedback to stimulus features at the trial-by-trial level. In each trial, subjects chose between two options defined by three dimensions: a relatively nonsalient shape (forwards or backwards C), a color (green or blue), and the side of presentation (left or right). Note that side of presentation refers here to both the position of the stimulus on the screen and the response required to select it (i.e., left or right button press). However, only features within one dimension (shape) were predictive of whether or not an option was rewarding (75 or 25% chance of reward). Subjects were informed that only the shape was relevant and that rewards were probabilistic before beginning the task.

Figure 2.

Task design. In each trial, subjects were shown a fixation point for 500 ms, followed by two “card” options on either side of the screen for 600 ms. These cards were then replaced by a mask for 500 ms. Subjects could respond at any point from the onset of the option slide to the termination of the mask slide. After a choice was made, subjects received feedback, a win (40 points) or no win (0 points), that was probabilistically dependent on the shape (forwards or backwards C) inside of the card. In the next trial, the color and side of these shapes would randomly stay the same or switch.

Once subjects met a criterion indicating that they had learned the shape–reward association, this relationship was reversed. This criterion condition ensured that the frequency of reversals was matched to subjects' performance level. Reversals in this task were not themselves of primary interest, but were included to ensure that the task remained difficult for all subjects and to prevent the adoption of a simple rule-based response strategy.

The task consisted of four blocks, each lasting 200 trials. In each trial, the two options comprised combinations of shape, color, and side features such that all features were represented in every trial, and all combinations of option features were equally represented in each block. Thus, from trial to trial, the features defining each option could either remain the same or differ along multiple dimensions. Optimal performance in this task required subjects to continuously track the changing values associated with these shapes, while filtering out information from other stimulus dimensions that was not predictive of rewards. Thus, this task taxed subjects' ability to direct selective attention for the optimization of reward learning in a dynamic, challenging environment.

Studies of visual search have shown that similar trial-by-trial manipulations prime attention to task-irrelevant features of previously selected targets (i.e., priming of pop-out; Maljkovic and Nakayama, 1994; Kristjánsson and Campana, 2010), and that this priming effect is modulated by feature–reward associations (Hickey et al., 2010b; Kristjánsson et al., 2010). We also have recently shown that attentional priming for reward-associated visual features is affected by VMF damage (Vaidya and Fellows, 2015). Here, we asked whether such trial-by-trial attentional effects relate to the flexible reward learning deficits previously observed after VMF damage (Tsuchida et al., 2010). Stimuli were more briefly presented than is typical for reward learning tasks, and the predictive stimulus feature was less salient than nonpredictive (“distracting”) task features to place particular demands on the relatively rapid selective attention processes that were of interest here (Sigurdardottir et al., 2008; Hickey et al., 2010a).

Overall task performance

While the task was designed primarily to study highly local interactions between attention and learning on a trial-by-trial basis, we nonetheless examined overall task performance in a first step to gauge group performance at a coarser level and connect to the existing literature. We first tested whether subjects consistently chose the high-reward shape by comparing the frequency of choices of the high-reward and low-reward probability shapes, as well as missed responses (i.e., failure to respond by the deadline) between groups across the whole task. There was a significant difference in the proportion of these responses between groups (χ² test of independence: χ₍₈₎ = 295.39, p < 0.001), with the frontal groups missing more responses compared with healthy controls (Table 3). After excluding missed responses, there was no difference in the proportion of choices of the high-reward versus low-reward shape between groups (Kruskal–Wallis test: H₍₄₎ = 3.67, p = 0.4; Fig. 3A). Given the challenging nature of this task (multiple reversals, probabilistic reward, multiple stimulus dimensions), we expected that overall performance would be far from optimal. Nonetheless, a majority of subjects in each group chose the high-reward shape more often than expected by chance overall, though the proportion of the LLF group meeting this criterion was lower than in controls and other patient groups (percentage of subjects above chance: Controls, 76.2%; DMF, 70.0%; LLF, 57.1%; RLF, 83.3%; VMF, 75.5%).

Table 3.

Overall task performance^a

Group	Choice high-probability shape (%)	Choice low-probability shape (%)	Missed (%)
Control (N = 21)	55.2 (7.8)	40.1 (7.8)	4.7 (2.6)
DMF (N = 10)	51.2 (9.5)	40.8 (7.2)	7.9 (5.8)
LLF (N = 7)	49.8 (4.7)	46.4 (3.1)	3.7 (2.4)
RLF (N = 6)	50.0 (6.9)	43.6 (5.9)	6.4 (6.4)
VMF (N = 17)	51.2 (8.3)	40.9 (5.9)	7.8 (6.3)

^aValues represent mean with SD in parentheses.

Figure 3.

Measures of overall task performance. A, Overall probability of choosing the shape with a high probability of reward. B, Probability of choosing the shape with a high probability of reward over task blocks. C, Average number of reversals per block (i.e., frequency that subjects met the learning criterion). D, Probability of choosing the shape with a high probability of reward in trials immediately following a reversal. Dashed line indicates chance-level performance. Box plots show the 10th, 25th, 50th, 75th, and 90th percentiles of data. Error bars indicate SEM. Healthy controls (CTL).

We next tested whether subjects' performance improved over the course of the experiment by comparing the frequency with which subjects chose the high-reward shape across blocks (Fig. 3B). The first block of the experiment was dropped from analysis in some subjects (see Materials and Methods). In these cases, the second block was considered the start of the experiment and only three blocks were considered in the analysis. For the majority of subjects, the third and fourth blocks were collapsed in this analysis. Across groups, performance improved over the course of the experiment (mixed-measures ANOVA: F_(2,112) = 8.53, p = 0.0004). There was no significant effect of group status (F_(4,56) = 0.94, p = 0.4), or interaction between block and group (F_(8,56) = 1.15, p = 0.3), in the frequency with which subjects chose the high-reward shape, though LLF and DMF groups did not show the same degree of improvement as controls.

We also examined the number of reversals per block to assess how often each group encountered reversals, and to test whether these groups differed in the frequency with which they met the learning criterion (Fig. 3C). All subjects met the criterion that triggered a reversal at least once over the course of the experiment. There was wide variance in the number of reversals subjects experienced, and a trend toward a difference between groups (Kruskal–Wallis test: H₍₄₎ = 7.96, p = 0.09), with frontal groups, particularly LLF-damaged subjects, meeting the learning criterion less often than controls.

Flexible learning requires rapid adaptation to changing reward associations. Prior work has shown that VMF damage leads to prominent learning deficits under such conditions (Dias et al., 1996; Fellows and Farah, 2003; Walton et al., 2010). We tested for group difference in the selection of the high-reward shape in trials immediately following reversals in subjects who achieved ≥3 reversals over the course of the experiment (Fig. 3D). Overall, most subjects chose the shape (newly) associated with a high probability of reward at well below chance rate in the first trial after a reversal, consistent with a deliberate selection of the previously rewarded shape, and then increasingly more often in subsequent trials (mixed-measures ANOVA: F_(11,539) = 8.33, p < 0.0001), consistent with learning the new contingencies. While the VMF group was numerically similar to controls, there was a modest trend toward an interaction between group and trial number (F_(44,539) = 1.26, p = 0.1), driven by the poorer performance of LLF-damaged patients immediately following the reversal, in contrast to controls and the other patient groups. There was no significant main effect of group (F_(4,49) = 1.29, p = 0.3) on this measure.

To summarize, these overall measures of task performance revealed that all groups struggled to consistently choose the most rewarding shape, i.e., that the task was sufficiently difficult to require dynamic adjustments in behavior in response to feedback. Most subjects improved in overall performance over the course of the experiment; all met the learning criterion that triggered a contingency reversal at least once. The LLF group performed numerically worse than controls and other patient groups in all measures of overall task performance related to choosing the high-reward shape, though no significant effects of group status were found in these measures.

Trial-by-trial behavior

We were primarily interested in how frontal lobe damage affected the attribution of feedback to relevant and irrelevant stimulus features on a trial-by-trial level. We predicted that the VMF was critical for attributing feedback to features within the relevant stimulus dimension. To assess this prediction, we used GEEs to test how feedback in the immediately previous trial (n − 1) influenced subjects' choices on the current trial (trial n). This analysis maximized our sensitivity to detect effects at the level of trial-by-trial behavior, taking full advantage of the relatively large number of unique observations per subject, and the structure of the task itself, with random changes in the features composing each decision option from trial to trial. The GEE analysis estimated the probability that subjects would choose the left or right option, as predicted by the shape, color, and side chosen in the last trial (i.e., whether or not the previously chosen shape or color appeared in the left or right option, and whether subjects chose the left or right option in the last trial). For each parameter, we report an odds ratio (OR) and 95% confidence interval (CI) representing how each factor affected the probability of choosing the left or right option from trial to trial on a logarithmic scale.

We first tested how the association of relevant and irrelevant features with rewarding feedback (i.e., “wins”) affected the trial-by-trial behavior of healthy control subjects. Control subjects were significantly more likely to choose a shape (OR, 16.13; 95% CI, 6.77–38.47; p < 0.0001), or color (OR, 2.18; 95% CI, 1.52–3.12; p < 0.0001) that had been rewarded (win) in the previous trial, but not the rewarded side (OR, 1.08; 95% CI, 0.66–1.76; p = 0.8). We next examined the influence of “no win” feedback on controls' trial-by-trial behavior. Absence of reward did not affect the probability that controls would repeat a choice of a previously chosen shape (OR, 1.01; 95% CI, 0.65–1.58; p = 0.9), or color (OR, 0.94; 95% CI, 0.81–1.10; p = 0.5) on the current trial, but did significantly decrease the likelihood of choosing the same side (OR, 0.50; 95% CI, 0.33–0.75; p = 0.0009). Thus, while controls were strongly inclined to choose the shape that had been rewarded on the previous trial, they also frequently chose previously rewarded colors, despite task instructions and the fact that these features were not consistently associated with reward feedback across trials (i.e., this dimension had no overall predictive value). Healthy controls also tended to avoid picking a previously unrewarded side, though this dimension was also, in fact, irrelevant (nonpredictive). Thus, healthy control subjects' choices were influenced by feedback associated with relevant, predictive features, as well as irrelevant features with no predictive value.

We next tested interactions between group status, to investigate the effects of regional frontal lobe damage. These interactions capture the likelihood that lesion groups repeated a choice of features that were rewarded, or unrewarded, in the previous trial compared with the control group. This analysis thus tested how frontal groups were influenced by the trial-by-trial association of positive and negative feedback with stimulus features. Group effects are reported as ORs for the comparison of patients to the control group.

We first tested whether frontal groups differed from controls in staying with previously rewarded shapes (i.e., win–stay behavior), and whether these subjects were more likely to choose previously rewarded irrelevant features (color or side; Fig. 4A). The full results of this GEE analysis can be found in Table 4. VMF-damaged and LLF-damaged groups were significantly less likely than controls to select a previously rewarded shape. A similar trend was also seen in the DMF-damaged group. Thus, both VMF-damaged and LLF-damaged groups showed a reduced tendency to stay with the previously rewarded feature in the relevant dimension.

Figure 4.

Analysis of trial-by-trial performance. A, Probability of staying with a feature after a rewarded choice (win). B, Probability of staying with after an unrewarded choice (no win). Dashed line indicates chance-level responding. Box plots show the 10th, 25th, 50th, 75th, and 90th percentiles of data. *p < 0.05, **p < 0.0001, generalized estimating equation coefficients, referenced to controls. ∧p < 0.05, Bonferroni-corrected t test.

Table 4.

ORs, 95% CIs, and respective P values for trial-by-trial win–stay behavior within all three dimensions

Effect	OR (95% CI)	P value
Intercept	0.14 (0.08–0.26)	<0.0001
Main effects
DMF	1.29 (0.58–2.85)	0.5
LLF	0.79 (0.38–1.62)	0.5
RLF	2.70 (1.13–6.45)	0.03
VMF	1.32 (0.64–2.73)	0.4
Shape	16.14 (6.77–38.47)	<0.0001
Color	2.18 (1.52–3.12)	<0.0001
Side	1.08 (0.66–1.76)	0.8
Feature win–stay × group (referenced to controls)
Shape ×
DMF	0.34 (0.10–1.19)	0.09
LLF	0.12 (0.04–0.34)	<0.0001
RLF	0.38 (0.08–1.71)	0.2
VMF	0.29 (0.09–0.90)	0.03
Color ×
DMF	0.81 (0.47–1.38)	0.4
LLF	2.49 (0.83–7.48)	0.1
RLF	0.78 (0.49–1.24)	0.3
VMF	1.08 (0.66–1.79)	0.7
Side ×
DMF	2.02 (0.73–5.57)	0.2
LLF	5.13 (1.64–16.02)	0.005
RLF	0.48 (0.22–1.03)	0.06
VMF	1.11 (0.59–2.10)	0.7

We also examined the tendency of patient groups to choose features that were transiently associated with reward on a given trial, but had no predictive value overall. The LLF group was significantly more likely than controls to choose the previously rewarded side. The RLF group showed a modest trend in the opposite direction (i.e., less likely to choose a previously rewarded side), as well as a modest, but significant, tendency to choose the left side more often overall. Post hoc tests between frontal lobe-damaged groups revealed a significant difference between RLF and LLF groups in their likelihood to choose a previously rewarded side (p = 0.04, Bonferroni-corrected t test), and no other significant differences. No significant differences were observed between patient groups and controls in the probability of choosing a previously rewarded color, although there was large variance in this effect within the LLF group.

Similarly, we tested whether subjects with frontal lobe damage differed from controls in switching away from a shape, color, or side after a choice was unrewarded (i.e., no-win–stay behavior; Fig. 4B). The full results of this GEE analysis can be found in Table 5. We found no differences between controls and any frontal lobe-damaged groups in this behavior for either the relevant shape dimension, or in the irrelevant color, or side dimensions. Thus, frontal lobe-damaged groups differed principally in their tendency to stay with previously rewarded features in this task.

Table 5.

ORs, 95% CIs, and respective P values for trial-by-trial no win–stay behavior for all three dimensions

Effect	OR (95% CI)	P value
Intercept	1.20 (0.83–1.75)	0.3
Main effects
DMF	1.43 (0.86–2.39)	0.17
LLF	0.83 (0.49–1.40)	0.5
RLF	1.30 (0.69–2.44)	0.4
VMF	0.90 (0.57–1.44)	0.7
Shape	1.01 (0.65–1.58)	0.9
Color	0.94 (0.81–1.10)	0.5
Side	0.50 (0.33–0.75)	0.0009
Feature no win–stay × group (referenced to controls)
Shape ×
DMF	0.75 (0.35–1.64)	0.5
LLF	0.97 (0.60–1.57)	0.9
RLF	1.14 (0.54–2.43)	0.7
VMF	0.86 (0.53–1.42)	0.6
Color ×
DMF	1.31 (1.01–1.71)	0.04
LLF	1.53 (0.84–2.78)	0.2
RLF	0.98 (0.76–1.28)	0.9
VMF	1.05 (0.78–1.42)	0.7
Side ×
DMF	0.59 (0.31–1.12)	0.1
LLF	1.01 (0.49–2.07)	0.9
RLF	0.59 (0.26–1.33)	0.2
VMF	1.09 (0.62–1.92)	0.7

Recent choice history

We were interested in whether the influence of feedback on choices of irrelevant features depended on accumulated feature–reward history, or were driven primarily by highly local (i.e., one-trial back) feature–reward pairings. We tested how the frequency of rewards (positive feedback), or absence of rewards (negative feedback), associated with recent choices of relevant and irrelevant stimulus features affected current decisions. In each trial, we calculated the difference in the frequency of positive and negative feedback for stimulus features within each dimension, based on choices in the past four trials. We then used a GEE model to estimate the probability of subjects choosing the left option as a function of the positive and negative feedback history associated with stimulus features on the left and right in all three dimensions.

We first fit this model in control subjects to test the effects of feedback history in each stimulus dimension. Control subjects were more likely to choose shapes that had been rewarded more frequently in the past four trials (OR, 1.52; 95% CI, 1.37–1.71; p < 0.0001). A similar, but weaker effect was found in the color dimension (OR, 1.14; 95% CI, 1.07–1.22; p < 0.0001), but not in the side dimension (OR, 1.01; 95% CI, 0.97–1.06; p = 0.5). Healthy controls' choices were unaffected by the relative frequency of negative feedback for left and right shapes (OR, 0.94; 95% CI, 0.85–1.05; p = 0.3), or colors (OR, 0.99; 95% CI, 0.95–1.03; p = 0.7). However, these subjects showed a small, but consistent, tendency to avoid choosing a side that was more frequently associated with negative feedback (OR, 0.90; 95% CI, 0.85–0.95; p = 0.0001). Thus, control subjects chose shapes and colors associated with a greater frequency of reward and avoided sides that had been frequently unrewarded in past choices, similar to the pattern of behavior revealed by the trial-by-trial analysis.

We next tested the interaction of group status with the history of positive (reward) and negative (absence of reward) feedback for each of these stimulus dimensions. The full results of this analysis can be found in Table 6. The relative frequency of positive, but not negative, feedback in the relevant shape dimension had a weaker influence on the choices of DMF, LLF, and VMF groups compared with controls, but not the RLF group (Fig. 5A). No frontal lobe-damaged group was affected more than controls by the relative frequency of positive feedback history in the color dimension (Fig. 5B), though the DMF group showed a marginally significant tendency to choose frequently unrewarded colors compared with controls (Fig. 5E). Notably, the LLF group alone was more likely than controls to make choices based on the frequency of positive feedback associated with the left and right sides (Fig. 5C). The RLF group was significantly less likely to choose the side with a greater frequency of either positive or negative feedback (Fig. 5C,F), essentially switching away from a side repeatedly chosen over several trials. Post hoc tests on these coefficients between frontal-damaged groups revealed a significant difference between RLF and LLF groups in the influence of reward frequency on the side dimension (p = 0.009, Bonferroni-corrected t test), and no other significant differences in the influence of positive or negative feedback history. Thus, the LLF group was influenced more by the reward history within the irrelevant side dimension, while VMF, DMF, and LLF groups were all less sensitive than controls to the history of positive feedback within the relevant (shape) dimension.

Table 6.

ORs, 95% CIs, and respective P values for effects of the difference in reward history of features in the left and right options

Effect	OR (95% CI)	P value
Intercept	0.17 (0.08–0.34)	<0.0001
Main effects
DMF	2.11 (0.66–6.73)	0.2
LLF	0.42 (0.08–2.15)	0.3
RLF	2.34 (0.58–9.43)	0.2
VMF	1.30 (0.52–3.24)	0.6
Shape reward	1.53 (1.37–1.71)	<0.0001
Color reward	1.14 (1.07–1.22)	<0.0001
Side reward	1.01 (0.97–1.06)	0.5
Shape no reward	0.95 (0.85–1.06)	0.3
Color no reward	0.99 (0.95–1.04)	0.7
Side no reward	0.90 (0.86–0.95)	0.0001
Feature positive reward history × Group (referenced to controls)
Shape reward ×
DMF	0.84 (0.71–0.99)	0.04
LLF	0.75 (0.64–0.88)	0.0004
RLF	0.93 (0.76–1.15)	0.5
VMF	0.84 (0.72–0.97)	0.02
Color reward ×
DMF	0.98 (0.87–1.10)	0.7
LLF	1.16 (0.96–1.40)	0.1
RLF	0.97 (0.88–1.07)	0.5
VMF	1.00 (0.91–1.09)	0.9
Side reward ×
DMF	1.08 (0.99–1.17)	0.1
LLF	1.25 (1.10–1.43)	0.0006
RLF	0.90 (0.83–0.97)	0.01
VMF	1.03 (0.95–1.11)	0.5
Feature negative reward history × Group (referenced to controls
Shape no reward ×
DMF	1.01 (0.87–1.17)	0.9
LLF	1.06 (0.95–1.19)	0.3
RLF	1.10 (0.92–1.30)	0.3
VMF	1.02 (0.90–1.16)	0.8
Color no reward ×
DMF	1.08 (1.00–1.16)	0.04
LLF	1.10 (0.93–1.31)	0.2
RLF	1.02 (0.95–1.09)	0.6
VMF	1.02 (0.96–1.09)	0.5
Side no reward ×
DMF	0.87 (0.74–1.01)	0.07
LLF	0.95 (0.81–1.12)	0.6
RLF	0.89 (0.82–0.97)	0.008
VMF	1.00 (0.93–1.07)	0.9

Figure 5.

Influence of recent feedback history within each stimulus dimension on choice. A–C, Probability of choosing the left item as a function of the relative frequency of positive feedback for past choices of the left and right; A, shape, B, color; C, side. D–F, Probability of choosing the left item as a function of the relative frequency of negative feedback for past choices of the left and right; D, shape; E, color; F, side.

Feature information and learning

Theories of associative learning in multidimensional settings suggest that attention helps sculpt learning by focusing on features that are informative about feedback contingencies (Mackintosh, 1975; Pearce and Hall, 1980). While subjects were explicitly instructed about which stimulus dimension was informative, all groups, including healthy controls, were sensitive to the recent feedback history of features within ultimately uninformative stimulus dimensions. Although these irrelevant dimensions were not predictive, they may have gained some apparent information value through spurious correlation with rewards in subsets of trials. We tested whether the apparent information value within each dimension affected attribution of feedback to the relevant and irrelevant stimulus dimensions (i.e., if greater apparent information in the irrelevant dimensions reduced attribution of reward to the relevant shape dimension and vice versa), and whether sensitivity to information value within relevant and irrelevant dimensions was affected by frontal lobe damage.

We first established whether trial-by-trial performance in healthy control subjects was affected by the information in each stimulus dimension. The information in each dimension was calculated as the absolute value of the difference in the frequency with which each feature was associated with reward in the past four choices. This measure captured the extent to which the features in each dimension carried apparent information about reward associations based on recent trial history.

We used separate GEE models to test how the probability of staying with previously rewarded relevant and irrelevant features was affected by the information in each dimension. With greater information in the shape dimension, healthy controls stayed with previously winning shapes significantly more often (OR, 1.44; 95% CI, 1.32–1.58; p < 0.0001), and previously winning colors significantly less often (OR, 0.90; 95% CI, 0.84–0.97; p = 0.006). Information in the shape dimension did not affect the likelihood of controls choosing the previously rewarded side (OR, 0.95; 95% CI, 0.89–1.03; p = 0.2), which was unsurprising given that these subjects were not overall inclined to choose the previously rewarded side, as described earlier. Controls did not choose a previously winning side more, even as the side dimension became more informative (OR, 1.06; 95% CI, 0.99–1.12; p = 0.1). Nor did the information in the side dimension affect the probability of sticking with a rewarded color (OR, 1.00; 95% CI, 0.95–1.05; p = 0.9) or shape (OR, 1.00; 95% CI; 0.90–1.02; p = 0.2). Similarly, control subjects did not stick with a rewarded color more as this dimension became more informative (OR, 1.06; 95% CI, 0.97–1.14; p = 0.2). However, as the color dimension became more informative, these subjects were less likely to stick with the rewarded shape (OR, 0.86; 95% CI, 0.80–0.92; p < 0.0001), though the probability of choosing a rewarded side was unchanged (OR, 0.96; 95% CI, 0.91–1.02; p = 0.2). Thus, healthy control subjects scaled the extent to which feedback was attributed to features in the shape and color dimensions based on the information value within these dimensions derived from recent reward history.

We next examined how frontal lobe damage affected the influence of information in relevant and irrelevant stimulus dimensions on reward attribution. GEE models were used to test the interaction between group status and reward information on win–stay behavior in all three dimensions (see Notes). There was a trend for information in the relevant shape dimension to have less influence on win–stay behavior for these features in the LLF group (OR, 0.84; 95% CI, 0.70–1.01; p = 0.07), but no significant differences between patient groups and controls.

We next examined the influence of dimension information on win–stay behavior for the irrelevant color dimension. The LLF group was significantly more likely to choose to stay with rewarded colors when this dimension was informative compared with controls (OR, 1.15; 95% CI, 1.01–1.30; p = 0.03). Post hoc tests revealed no significant difference between frontal-damaged groups in the influence of color information (P's > 0.1, Bonferroni-corrected t tests). There was also a slight trend for the LLF group to stick with a winning color less often as reward information in the side dimension increased (OR, 0.87; 95% CI, 0.74–1.03; p = 0.1), but no other significant differences.

As in the color dimension, LLF-damaged subjects showed increased win–stay behavior for the rewarded side as information in this dimension increased compared with control subjects (OR, 1.27; 95% CI, 1.09–1.49; p = 0.002). However, as information in the color dimension increased, this group also chose the rewarded side less often (OR, 0.85; 95% CI, 0.77–0.94; p = 0.001). Similarly, the DMF group showed a reduction in the probability of choosing the rewarded side as information in the relevant shape dimension increased (OR, 0.86; 95% CI, 0.75–1.00; p = 0.04). Post hoc tests revealed no significant differences between frontal lobe-damaged groups in the influence of information within any of these dimensions on the probability of staying with a rewarded side (P's > 0.1, Bonferroni-corrected t tests).

Overall, the LLF group alone was more inclined to attribute rewards to features in irrelevant dimensions as the information value of these dimensions increased. This group was also more likely to switch between attributing rewards to the irrelevant side or color dimensions, depending on the information value of these dimensions extracted from recent trials.

Reaction times

We anticipated that the random switching of the color and side of the relevant shapes might result in trial-by-trial interference effects reflected in choice reaction times. Indeed, we found that subjects were slower to choose the previously rewarded shape on the next trial when it changed colors (F_(1,56) = 4.07, p = 0.05) or sides (F_(1,56) = 42.66, p < 0.0001). There was also a significant interaction between side and color repetition (F_(1,56) = 14.44, p = 0.0004), with larger effects of color repetition when the previously rewarded shape stayed on the same side. However, there was no significant effect of group (F_(4,56) = 0.91, p = 0.5) nor any interaction between group and color or side repetition (F's_(4,56) ≤ 1.04, P's ≥ 0.4), nor any three-way interaction between color and side repetition with group (F_(1,56) = 0.93, p = 0.4; Fig. 6). The interference effect from the side dimension was marginally larger than interference from the color dimension across groups (mixed-measures ANOVA: F's_(1,56) ≤ 3.96, p = 0.05). This difference might reflect the relative salience of these dimensions, motor interference specific to the side (response) dimension, or be due to task instructions to ignore the color dimension. Altogether, changes in the color or side of a shape significantly affected reaction time when choosing a previously rewarded shape, and this effect was similar between groups.

Figure 6.

Influence of trial-by-trial changes in irrelevant stimulus dimensions on reaction times for choosing a previously rewarded shape.

VLSM

The above analyses focus on differences in learning between healthy controls and frontal lobe-damaged subjects, grouped according to relatively coarse anatomical regions of interest. We followed up these findings with VLSM, which is not constrained by predefined anatomical boundaries, and can provide insights into whether more narrowly defined subregions are driving the effects. This method compares the behavior of subjects with damage at each voxel to all other lesioned subjects in the sample to test the association of a behavioral deficit with the damaged voxels.

Multiple logistic regression analyses were used to estimate parameters for individual subjects' win–stay behavior in each stimulus dimension. Given that VLSM removes the age and education matching designed into the primary region-of-interest analyses, we used multiple linear regression to first test whether parameters for win–stay behavior in each of these stimulus dimensions were related to age or education in the healthy control group. A significant positive relationship was found between parameters for win–stay behavior in the shape dimension and education level (OR, 2.50; 95% CI, 1.56–4.00; p = 0.002), but not age (OR, 1.00; 95% CI, 0.97–1.03; p = 0.9). Neither education level (OR, 0.80; 95% CI, 0.39–1.63; p = 0.6) nor age (OR, 1.02; 95% CI, 0.96–1.07; p = 0.4) was related to parameters for win–stay behavior in the color dimension. Education level was modestly associated with parameters for win–stay behavior in the side dimension (OR, 2.30; 95% CI, 1.15–4.58; p = 0.02), though age was not (OR, 1.03; 95% CI, 0.98–1.08; p = 0.2). To correct for these relationships, we calculated the residuals of parameters for win–stay behavior in the shape and side dimensions in frontal lobe-damaged patients by subtracting the values predicted by age and education level based on these multiple linear regression analyses in the healthy control group. No correction was applied to coefficients for win–stay behavior in the irrelevant color dimension, as this measure was not significantly related to these demographic variables.

Figure 7A shows the voxels where there was sufficient lesion overlap to test for lesion–performance relationships using VLSM, and the power to detect significant effects (i.e., maximum detectable z-score based on Wilcoxon nonparametric tests; Gläscher et al., 2009). No voxels passed the permutation-corrected threshold for significance at p < 0.05 for win–stay behavior in the shape dimension (Z > 3.35), or for win–stay behavior in the side dimension (Z > 3.43). However, increased win–stay behavior for the color dimension was significantly associated with damage in the left inferior frontal gyrus in two clusters of voxels (Z = 3.72, p = 0.01, permutation-corrected; Fig. 7B). Effects were also found at an uncorrected threshold (Z ≥ 1.92, two-tailed) in several areas within the frontal lobes (see Notes).

Figure 7.

VLSM analysis. A, Power map of voxels with sufficient lesion overlap for VLSM methods. The color scale indicates the maximum possible Z-score detectable at a given voxel, an indication of the power for detecting effects. Top row shows this map overlaid on the MNI brain in three-dimensional views. Bottom row shows representative axial slices. Numbers above the axial slices correspond to z-coordinates in MNI space. R, Right; L, left. B, VLSM statistical map for increased win–stay behavior in the color dimension overlaid on MNI brain in axial slices (top) and three-dimensional views (bottom). Color scale indicates Brunner–Munzel Z scores. Voxels in pink represent where significant effects were found at p = 0.01, permutation corrected.

Psychoactive medication

The proportion of patients taking psychoactive medication was not evenly distributed across lesion groups (see Materials and Methods). We tested for any effects of psychoactive medication on the influence of the difference of positive and negative feedback history in each of the three stimulus dimensions with a GEE model, collapsed across patient group and medication type. We found no significant difference between patients on and off psychoactive medication on the influence of positive feedback history for the shape (OR, 0.98; 95% CI, 0.86–1.11; p = 0.7), color (OR, 0.92; 95% CI, 0.82–1.10; p = 0.7), or side (OR, 0.99; 95% CI, 0.89–1.11; p = 0.9) dimensions. There was likewise no effect of psychoactive medication on the influence of negative feedback history in these stimulus dimensions (shape: OR, 0.99; 95% CI, 0.90–1.08; p = 0.8; color: OR, 0.93; 95% CI, 0.85–1.01; p = 0.09; side: OR, 0.97; 95% CI, 0.87–1.07; p = 0.5).

Relationship with neuropsychological screening tests

Instructions were important to performance of this task. We therefore investigated whether impairments in verbal ability or memory affected results. To test this possibility, we examined the relationship between coefficients for win-stay behavior in each of the three stimulus dimensions with verbal fluency (adjusted for age and education using population norms found in Mitrushina et al., (2005; N = 36), and episodic memory for faces (N = 32) in the frontal lobe damaged subjects who completed these screening measures. FAS fluency was not related to win–stay behavior in the relevant shape dimension (Spearman's ρ = −0.09, p = 0.6), or the irrelevant color (ρ = −0.08, p = 0.6), or side dimensions (ρ = −0.25, p = 0.1). There were also no significant correlations between memory task performance and win–stay behavior in the shape (ρ = 0.19, p = 0.3), color (ρ = 0.22, p = 0.2), or side dimensions (ρ = −0.26, p = 0.2).

Discussion

We tested the necessary contributions of four frontal lobe subregions to reward learning in a multidimensional environment where only one dimension was predictive of feedback. We focused on the interaction of attention and learning, examining how relevant and irrelevant reward associations influenced trial-by-trial behavior. LLF-damaged subjects showed the greatest impairments in this task, with deficits in learning about the relevant stimulus dimension and an increased tendency to make choices based on rewards associated with a second, irrelevant dimension. Subjects with VMF damage were also impaired in staying with the relevant stimulus dimension, as expected, but were not more influenced by the feedback history within irrelevant dimensions compared with controls. These distinct behavioral patterns argue that these frontal subregions have different functional roles in optimally navigating a multidimensional task environment.

The impairment of LLF-damaged patients in this task is notable, given the intact performance of such patients in simpler dynamic reward learning tasks (Fellows and Farah, 2003; Tsuchida et al., 2010). Here, stimulus options were more complex and more briefly presented, presumably putting more demand on top-down, selective attention. Importantly, the impairment of the LLF group could not be explained by a simple perseverative bias to a rewarded side or color, as these subjects took into account the recent history of feedback within these irrelevant dimensions in their choices. The LLF group was also not completely insensitive to the history of feedback in the relevant dimension, and decreased their win–stay behavior in the irrelevant dimensions when reward history in the relevant dimension was more informative. Thus, LLF-damaged subjects appeared to discriminate between these stimuli, but failed to attribute feedback to the appropriate dimension in the face of distracting dimensions.

Attentional learning theories argue that the locus of attention determines the strength of learned stimulus–reward relationships (Kruschke, 2003), and that outcomes can be easily attributed to salient, but irrelevant, stimuli (Pavlov, 1927; Mackintosh, 1976). The mirror shapes used here were less discriminable, and likely less visually salient, than the colors or side of the stimulus, taxing selective attention. The deficits of the LLF group are consistent with impairment in selectively attending to the relevant, but nonsalient, shape dimension and ignoring salient distractors, resulting in the misattribution of rewards in this learning context. This view is consistent with this region's putative contribution to selective attention in other settings, like visual search (Buschman and Miller, 2007; Suzuki and Gottlieb, 2013; Tremblay et al., 2015), and the classic Stroop task (Perret, 1974; Gläscher et al., 2012; Tsuchida and Fellows, 2013).

These findings have relevance to theories of lateral PFC function. Our findings provide causal support for the work of Niv et al. (2015), arguing that the frontoparietal attention network, including the lateral PFC, facilitates the selection of stimulus dimensions for feedback attribution. In addition to misattributing rewards, LLF-damaged subjects also showed a greater tendency to fluctuate between attributing rewards to irrelevant color and side features, depending on local correlations with feedback. This increased sensitivity to local changes in the apparent information value within these irrelevant dimensions might reflect a reduction in attentional filtering (Chrysikou et al., 2014). Recent imaging studies in humans and monkeys have argued that interactions between anterior ventrolateral PFC and the amygdala mediate the fidelity of feedback attribution (Chau et al., 2015; Jocham et al., 2016). Future work studying the connections necessary for appropriate feedback attribution might shed further light on the brain networks involved in multidimensional learning of this kind.

The nature of the stimuli used here might also be relevant to our findings. Mirror letter stimuli are specially processed by the left hemisphere (Pegado et al., 2011; Nakamura et al., 2014), and left hemisphere lesions sometimes affect reading and writing of mirror letters (Schott, 2007). LLF-damaged subjects therefore may have had particular difficulty tuning attention to the letter-shaped stimuli. Further work is necessary to establish whether the lateral PFC makes lateralized, material-specific contributions to associative learning, as we have shown in conventional cognitive control tasks (Geddes et al., 2014). Relevant and irrelevant features were also perceptually bound together as single objects, with optimal performance requiring a dimensional approach that generalized across stimuli. The object-level presentation of these options might have encouraged attention to irrelevant features (O'Craven et al., 1999), promoting incidental learning about irrelevant dimensions. The binding of these features might therefore have placed greater demands on lateral frontal-dependent attentional processes.

In contrast, we did not find evidence that VMF lesions affected the attribution of feedback to irrelevant stimulus dimensions compared with controls. Thus, learning deficits in VMF-damaged subjects are likely not the result of “distracted” learning due to misattribution of feedback to irrelevant features. This result is in line with work demonstrating that VMF damage does not affect the ability to select between stimulus dimensions, to ignore irrelevant, salient features in conventional attention tasks, such as the Stroop task (Gläscher et al., 2012; Tsuchida and Fellows, 2013), or to ignore changes in irrelevant stimulus dimensions in an associative learning task (Chase et al., 2008).

We observed impaired learning within the relevant dimension in the VMF group, consistent with other work suggesting this region facilitates attention to reward-predictive stimuli (Chase et al., 2008; Vaidya and Fellows, 2015). The current results are in line with a role for the VMF in tuning attention to reward-predictive stimulus features within this relevant dimension. These findings also resonate with recent imaging studies showing that vmPFC activity reflects the strength of subjectively relevant cues, and predicts choice repetition (Boorman et al., 2013; Akaishi et al., 2016), suggesting that this region facilitates selection of stimuli perceived to be preferable. Notably, VMF-damaged subjects in this study scaled reward attribution based on the information value of the relevant and irrelevant features, like controls. However, the choices of these patients were influenced less by the accumulated evidence from reward history within the relevant dimension. These results suggest that VMF damage may affect the allocation of attention based on an integrated sense of predictive value, but not local changes in the apparent information value of stimulus dimensions.

DMF-damaged subjects showed a trend toward attributing rewards to the irrelevant side, and decreased sensitivity to the history of positive feedback in the relevant dimension. The VLSM analysis indicated that left DMF damage affected these measures more than right DMF damage (see Notes), suggesting some commonality with the LLF group, possibly related to damage to underlying white matter. As in most real-life situations, the side of the stimulus (i.e., left or right side of the screen) was completely correlated with the action needed to acquire it (i.e., left or right button press). Prior work has shown that DMF damage in monkeys and humans impairs learning about the value of actions from feedback in action-selection tasks without visual stimuli (Rudebeck et al., 2008; Camille et al., 2011). Increased win–stay behavior for previously chosen actions is, at a glance, counterintuitive given this previous work, though staying with previously rewarded, but nonpredictive, actions could itself be a sign of impaired action-value learning. However, it is impossible to distinguish spatial-based or action-based accounts for this behavior in the current dataset.

In this task, subjects were verbally instructed regarding the relevant dimension. This prior information likely mitigated against greater attention to irrelevant stimulus dimensions. Learning about the information value in each stimulus dimension may be especially important when subjects are forced to use evidence from reward history to infer which dimensions are relevant based on feedback (Pearce and Mackintosh, 2010). Future work addressing how focal brain damage affects learning in a completely uninstructed multidimensional task will be helpful for establishing the neural mechanisms involved in forming an attentional set de novo based on reward feedback.

While only patients with well characterized, focal lesions were included in this study, lesion damage may affect both the cortex and underlying white matter, including fibers of passage. Rudebeck et al. (2013) have argued that learning impairments associated with OFC damage in macaques may relate to the interruption of such fibers. We cannot rule out such a possibility here, though VLSM analysis could, in principle, reveal whether behavioral deficits were associated with a consistent pattern of white matter damage. As with any method, converging evidence from other techniques will be important to fully address this limitation.

In summary, we examined the effects of frontal lobe damage on learning about reward-predictive and irrelevant stimulus dimensions in a dynamic environment. The findings argue that, within the frontal lobes, selective attention processes dependent on the LLF become critical under these more naturalistic conditions, in contrast to the narrow reliance on the VMF subregion when learning from probabilistic reward predicted by simpler stimuli. A complete understanding of value-based learning must address not only how reward is assigned to relevant features, but also how irrelevant features are suppressed. This work is a contribution to this effort, and suggests a framework for relating well studied aspects of lateral PFC function to models of value-related behavior centered on the VMF.

Notes

Supplemental material for this article is available at http://www.mcgill.ca/decisionlab/publications: Additional tables for dimension information analysis and VLSM, as well as the full behavioral dataset. This material has not been peer reviewed.