Frontostriatal and Dopamine Markers of Individual Differences in Reinforcement Learning: A Multi-modal Investigation

Abstract

Prior studies have shown that dopamine (DA) functioning in frontostriatal circuits supports reinforcement learning (RL), as phasic DA activity in ventral striatum signals unexpected reward and may drive coordinated activity of striatal and orbitofrontal regions that support updating of action plans. However, the nature of DA functioning in RL is complex, in particular regarding the role of DA clearance in RL behavior. Here, in a multi-modal neuroimaging study with healthy adults, we took an individual differences approach to the examination of RL behavior and DA clearance mechanisms in frontostriatal learning networks. We predicted that better RL would be associated with decreased striatal DA transporter (DAT) availability and increased intrinsic functional connectivity among DA-rich frontostriatal regions. In support of these predictions, individual differences in RL behavior were related to DAT binding potential in ventral striatum and resting-state functional connectivity between ventral striatum and orbitofrontal cortex. Critically, DAT binding potential had an indirect effect on reinforcement learning behavior through frontostriatal connectivity, suggesting potential causal relationships across levels of neurocognitive functioning. These data suggest that individual differences in DA clearance and frontostriatal coordination may serve as markers for RL, and suggest directions for research on psychopathologies characterized by altered RL.

Introduction

Learning to select behaviors that lead to positive outcomes is fundamental to survival, and prior research has suggested that the neurobiological mechanisms of successful reinforcement learning (RL) are shared across species, environments, and contexts (Seger 2009). In particular, striatal dopamine (DA) signaling for unexpected rewards (reward prediction errors (RPE)) is believed to play a key role in coordinating activity among striatal and orbitofrontal regions, thereby updating the value of environmental cues and action plans (Schultz 2015). Evidence for RPE encoding by DA has been found using electrophysiological measurement of DA cell firing (Cohen et al. 2012; Eshel et al. 2015; Schultz 2015), optogenetic stimulation of DA neurons (Tsai et al. 2009; Kravitz et al. 2012), and measurement of DA release in striatal terminal field regions (Flagel et al. 2011; Hart et al. 2014). Similarly, evidence linking frontostriatal circuit strength to RL has been documented in research ranging from preclinical studies (Bailey and Mair 2007; Braz et al. 2015) to human developmental (van den Bos et al. 2012) or lesion studies (Bellebaum et al. 2008). Together, this prior research provides support for the idea that DA release in frontostriatal circuits is a critical mechanism of successful learning.

A critical protein in the DA signaling pathway is the DA transporter (DAT), which facilitates rapid clearance of extracellular DA within the striatum. According to theoretical accounts of DA RPE signals, phasic release of DA in response to unexpected rewards should persist in the synapse long enough to engage post-synaptic targets but be cleared rapidly enough to maintain the requisite temporal precision for prediction error encoding. Moreover, abundant preclinical evidence indicates that, in addition to DA clearance, DAT shapes the signal-to-noise ratio of DA neurotransmission and can affect presynaptic DA levels during neuronal activity (for review, see Sulzer et al. 2016). If correct, this model would predict that modulation of DAT function would affect RPE signaling and by extension, reinforcement learning. Consistent with this framework, pharmacological manipulations that block or reverse DAT function, such as psychostimulants, have been shown to produce marked change in DA-dependent behaviors, including enhanced instrumental conditioning (Taylor and Robbins 1984; Everitt and Robbins 2005; 2016). However, these drugs often possess noradrenergic and serotonergic effects as well, and some studies using more selective DAT inhibitors (e.g., GBR12909) have failed to detect clear effects on RL behavior (Costa et al. 2014). Consequently, the role of DAT function for RL remains unclear.

In addition to its potential influence on DAergic RPE signals, individual differences in DAT function may also influence connectivity within frontostriatal networks, for example, the magnitude of positive functional connectivity between key nodes within the brain reward pathway, including regions implicated in reward prediction errors (such as the nucleus accumbens, NAc) and regions involved in updating action plans (such as the orbitofrontal cortex, OFC) (Yeo et al. 2011; Smith 2012; Smith et al. 2013). Given that one of the major effects of post-synaptic DA receptors on striatal medium spiny neurons (MSNs) is to potentiate or attenuate the strength of excitatory cortical and limbic inputs (Floresco 2015), it is plausible that DAT availability may also be associated with downstream effects on the level of large-scale network functioning. For example, lower DAT availability corresponding with increased synaptic DA may be related to enhanced coordination among frontostriatal regions. However, to our knowledge these possibilities have not yet been tested.

One strategy for investigating RL and frontostriatal DA is to capitalize on individual differences in these functional domains. People vary considerably in their RL behavior, and prior research has demonstrated that such variability corresponds with distinct profiles of DA and network-level functioning. For example, individuals characterized by DA deficiencies are shown to exhibit impaired learning (Frank et al. 2004; Wilkinson et al. 2009) and altered frontostriatal recruitment (Nakamura et al. 2001). An individual differences approach to the associations between DAT availability, frontostriatal network activity, and RL behavior, may be useful in revealing naturally occurring covariance across levels of functioning.

The present study was designed to examine how individual differences in RL manifest in frontostriatal DA systems in healthy humans, with a particular focus on DAT binding, as indexed by DAT binding potential (BP_ND). We took a multi-modal approach that included evaluation of individual differences using positron emission tomography (PET), functional magnetic resonance imaging (fMRI), and behavioral testing. We used the highly selective DAT ligand [¹¹C]altropane to assess DAT availability in the NAc within ventral striatum. Motivated by research suggesting the reliability of slow-wave intrinsic connectivity (Geerligs et al. 2015), frontostriatal circuit activity was evaluated using resting-state functional connectivity. Directly relevant to the current study, prior research has shown that pharmacological manipulation of DA enhances resting-state functional connectivity between NAc and ventral frontal regions (Kelly et al. 2009), and individual differences in frontostriatal resting-state functional connectivity have been associated with DA concentrations (Horga et al. 2016). Therefore, in the present study, resting-state functional connectivity of bilateral NAc was interpreted as a circuit-level index of effective coordination among DA systems (although other neurochemicals may contribute to individual differences in the same or overlapping circuits (Felger et al. 2016)). Reinforcement learning behavior was indexed using a validated task that has been used to measure individual differences in implicit reward sensitivity in previous research (Santesso et al. 2008). We predicted that better learning task performance would be related to 1) lower DAT BP_ND in ventral striatum (interpreting DAT BP_ND as an index of DAT clearance capacity); and 2) stronger resting-state functional connectivity in a frontostriatal circuit including ventral striatum and areas of orbitofrontal cortex.

Materials and Methods

Thirty-four healthy adults (ages 19–44, mean age = 26.81, SD = 7.00; 24 females) were recruited from the Boston metropolitan area through local websites, flyers, and advertisements. All participants completed a Structured Clinical Interview for the DSM-IV-TR to confirm the absence of current or history of psychiatric illness. In a behavioral testing session, participants completed a task designed to assess implicit RL; in a separate session, participants completed PET scanning. Next, a subset (n = 25, ages 19–44, mean age = 25.48, SD = 7.04; 15 females) of participants completed a session involving magnetic resonance imaging (MRI) scanning that included structural imaging and a resting-state paradigm. The average interval between sessions was 15.12 days; inter-session interval did not covary with experimental variables. In previous independent studies, these sample sizes were shown to be adequate for examining dopamine transporter (DAT) (Yeh et al. 2012) and for investigating the neural and behavioral indices of reinforcement learning used in the present study (Pizzagalli et al. 2008; Santesso et al. 2008, 2009). In light of prior evidence that DA functioning changes with age (Volkow et al. 1996), all analyses controlled for participants’ age in months. No participant was taking psychoactive medications, and all participants reported no history of neurological impairment, head injury, or MRI counter-indications. This study was approved by the Partners Healthcare Institutional Review Board, and written informed consent was collected. Data were stripped of identifying information, encrypted, and saved to password-protected servers. Data from the present study are available upon request.

Behavioral Testing and Analysis

Probabilistic Reward Task for Assessment of Reinforcement Learning Behavior

Individual differences in RL were measured in an implicit learning task that takes a signal detection approach to measuring sensitivity to rewards, the Probabilistic Reward Task (PRT) (Pizzagalli et al. 2005). For each trial of this task, the participant was presented (for 500 ms) with a drawing of a face on which either a short or long mouth stimulus (or a short or long nose stimulus), was displayed (for 100 ms). The participant was instructed to respond as quickly as possible to indicate which stimulus was displayed, and correct responses either resulted in reward feedback ($0.20 and the phrase “Correct! You won $0.20”) or null feedback (blank screen). The reinforcement schedule was asymmetrical: one “rich” stimulus was rewarded for correct responses 3 times more frequently than the other “lean” stimulus, unbeknownst to the participant. In total, the participant received 40 reward outcomes, 30 of which were elicited by correct response to rich stimulus, 10 of which were elicited by correct response to lean stimulus. Accordingly, in each block 30 of the 50 rich trials (60%) but only 10 of the 50 lean trials (20%) could be followed by a reward feedback. For the PET analyses, participants were pooled across 2 separate studies, which used the identical PRT paradigm, but different numbers of blocks. Specifically, 67% of participants completed 3 blocks of 100 trials/block, whereas the remaining participants completed 2 blocks of 100 trials/block. In order to merge these databases, performance scores were z-transformed within subgroups that performed either the 2-block or the 3-block version of the task, before pooling z-scores for a unified RL factor in subsequent analyses; in addition, task version was included in the analyses as a covariate. The primary index of reinforcement learning behavior was z-transformed change in response bias (ΔRB) from the first to the last block of trials [ΔRB = Response Bias (final block) – Response Bias (first block)]. Response bias (RB) was computed with the equation:

$$\begin{array}{c}\mathrm{Response}\mathrm{bias}:\log b\hfill \\ =\frac{1}{2}\log \left(\frac{(\mathrm{Ric}{\mathrm{h}}_{\mathrm{correct}}+0.5)\ast (\mathrm{Lea}{\mathrm{n}}_{\mathrm{incorrect}}+0.5)}{(\mathrm{Ric}{\mathrm{h}}_{\mathrm{incorrect}}+0.5)\ast (\mathrm{Lea}{\mathrm{n}}_{\mathrm{correct}}+0.5)}\right).\hfill \end{array}$$

In this equation, the variables Rich_correct and Rich_incorrect correspond to the number of correct and incorrect responses to identify the rich stimulus, respectively, and the variables Lean_correct and Lean_incorrect correspond to the number of correct and incorrect responses to identify the lean stimulus, respectively. Consistent with previous studies using this task, 0.5 was added to each of the above variables to permit calculating response bias in cases in which one of the raw variables was equal to zero (Santesso et al. 2008; Vrieze et al. 2013). Positive ΔRB over the course of the task indicates reinforcement learning proficiency (i.e., increased bias to respond accurately to “rich” compared with “lean” stimuli over time). Individual differences in this measure of reinforcement learning have been shown to correspond to symptoms of anhedonia (Pizzagalli et al. 2005), response to dopaminergic drugs (Pizzagalli et al. 2008), and neural response to reward (Santesso et al. 2008). Participants who performed poorly (<55% accuracy, or >10% outlier trials with RT < 150 ms or RT > 2500 ms, or failure to achieve an overall reinforcement schedule of approximately 3:1) (Pizzagalli et al. 2005) were excluded from analyses (all n = 34 eligible for analysis). See Figure 1 for summary of the PRT; no outliers on learning performance were detected in the present sample (i.e., ΔRB scores within 3 standard deviations of mean).

Figure 1.

Behavioral Tasks. (A) The Probabilistic Reward Task (PRT). For each trial, a cartoon face stimulus was presented, after which a mouth (or a nose, depending on task version) appeared briefly (100 ms) on the face and the participant was instructed to indicate (via buttom press) whether the mouth (or nose) was long or short. After the participant responded (or 2500 ms elapsed) feedback was presented. Correct responses for one (“rich”) stimulus were rewarded 3 times more frequently than correct responses for the other (“lean”) stimulus. The asymmetrical reward schedule induced a response bias in which participants were more likely to respond accurately to the “rich” stimulus than the “lean” stimulus. (B) Participants either completed a 2-block (100 trials/block) or a 3-block (100 trials/block) version of the task (task version was entered as a covariate in the analyses). Displayed is average response bias for each block of trials, for the subgroup of participants who completed each version of the task. (C) For analyses, to enable collapsing across task versions, participants’ change in response bias from the first (block 1) to the last (either block 2 or block 3) was computed, and z-scored within the subgroup of participants who completed each version of the task. Z-scored response bias was then collapsed across task versions; however, task version was a covariate in analyses. Displayed is the distributions of z-scored response bias across all participants.

PET Acquisition and Analysis

To investigate DA clearance, we used the radiotracer [¹¹C]altropane with DAT binding potential (BP_ND) as the primary outcome parameter. [¹¹C]altropane was selected as the PET tracer for this study because it has rapid and specific striatal binding (rapid kinetics in DA-rich striatal regions) and high selectivity for DAT (e.g., 28 times more selective for DAT than serotonin transporter (Fischman et al. 2001; Madras et al. 1998)). [¹¹C]altropane binding was assessed using an ECAT EXACT HR+ (CTI, Knoxville, TN) PET camera (3D mode, 63 contiguous 2.4 mm slices, 2.06 × 2.06 mm transaxial grid). For each participant, (approximately) 10 mCi of [¹¹C]altropane was administered intravenously over 20–30 s. Images were acquired in 39 frames, with the duration of each frame increasing over time (8 frames of 15 s, 4 frames of 60 s, 27 frames of 120 s) for a total duration of 60 min. A filtered back-projection algorithm was used to reconstruct PET images with physical corrections applied for photon scatter and attenuation, random coincidences, system deadtime, and detector inhomogeneity. The motion-corrected frames were summed and coregistered to a common reference space (Montreal Neurological Institute, MNI) using FSL (http://fsl.fmrib.ox.ac.uk/fsl/fslwiki/) and by computing the deformation field on the basis of the participant’s structural MRI scan for individual warping. Transformations were then applied to the dynamic PET images.

To calculate regional BP_ND (Innis et al. 2007), we used the multilinear reference tissue model (Ichise et al. 2003) with the reference region defined as the cerebellum, excluding the vermis (Alpert and Yuan 2009; Fang et al. 2012). BP_ND was estimated in left and right NAc, with regions defined by an anatomical atlas (Tzourio-Mazoyer et al. 2002) in MNI space (AAL atlas publically available as an SPM12 toolbox, http://www.gin.cnrs.fr/en/tools/aal-aal2/). The NAc ROI for PET imaging was structurally defined to ensure adequate coverage of ventral striatum, which is necessary to gain a sufficiently strong radiotracer signal in resting PET imaging studies. See Table 1 for report of DAT BP_ND in left and right NAc across the sample; the range of DAT BP_ND is consistent with previous studies, and no outliers were detected in the present sample (i.e., DAT BP_ND scores within 3 standard deviations of sample mean). For analyses and in all figures, DAT BP_ND scores were residualized for age and z-scored.

Table 1.

Descriptive statistics for DAT BP_ND

	DAT BPND
	Min	Max	Mean	STDEV
Left NAc	0.98	2.90	2.08	0.45
Right NAc	1.08	3.04	2.19	0.45
Bilateral NAc	1.03	2.94	2.14	0.44

Note: DAT BP_ND was indexed by [¹¹C]altropane. For analyses and in all figures, DAT BP_ND scores were residualized for age and z-scored.

To test the association between individual differences in reinforcement learning and DAT binding in ventral striatum, we performed a correlation between (z-scored ΔRB) and DAT BP_ND (residualized for age) in left and right NAc. Putative hemispheric differences in correlation coefficients were tested with the Meng test (Meng et al. 1992), which tests for differences between correlation coefficients while taking into account dependency between predictor variables in each correlation.

MRI Acquisition and Analysis

Data Acquisition

A Siemens Tim Trio 3T scanner and 32-channel head coil were used to collect MRI data, including a high-resolution T1-weighted anatomical image (TR = 2200 ms, TE = 4.27 ms, flip angle = 7, 144 slices, field of view = 230 mm, matrix = 192 × 192, voxel size 1.2 × 1.2 × 1.2 mm) and eyes-open resting functional data (TR = 3000 ms, TE = 30 ms, flip angle = 85, 47 slices, field of view = 216 mm, matrix = 72 × 72, voxel size 3 × 3 × 3 mm, total duration = 6.2 min, total volumes = 124). Resting-state fMRI data were collected immediately following collection of anatomical data, and prior to other functional scanning.

Resting-state: General Image Preprocessing

We discarded the first 6 s of each participant’s functional data to allow for stabilization of the magnetic field. Preprocessing of functional data was performed in SPM8 using the standard spatial preprocessing steps of slice-time correction, realignment, normalization in MNI space, and smoothing with a 6-mm kernel.

Resting-state: Head Motion and Artifact Detection

Motion correction is of special importance for resting-state functional connectivity analysis (Buckner et al. 2013). We used SPM8 to assess head motion by translation and rotation in x, y, z directions. Next, Artifact Detection Tools (ART, www.nitrc.org/projects/artifact_detect/) were utilized to calculate time points of significant head motion or fluctuations in the magnetic field (>1 mm motion from previous frame, global mean intensity >3 standard deviations from mean intensity across functional scans) for each participant. Then, outlier images were modeled in each participant’s first-level general linear model (as a vector the length of the time series, with 1 for outlier time points and 0 for non-outlier time points) to remove the influence of outlier time points on estimates of functional connectivity while maintaining the temporal structure of the data. Thus, motion correction included the regressing out of not only residual head motion parameters (3 translation and 3 rotation parameters, plus 1 composite motion parameter reflecting the maximum scan-to-scan movement), but also outlier volumes (as calculated through artifact detection).

Resting-state: Denoising

We performed voxelwise seed-based functional connectivity analyses using the CONN toolbox (https://www.nitrc.org/projects/conn/; Whitfield-Gabrieli and Nieto-Castanon 2012). We estimated physiological and other sources of noise using CompCor (Behzadi et al. 2007), a method that estimates physiological noise from white matter and cerebrospinal fluid for each participant using principal component analysis. The first 5 components were then regressed out of each participant’s functional data on the first level of analysis. In addition, a temporal band-pass filter of 0.01–0.10 Hz was applied to the time series. This range was selected to remove high frequency activity related to cardiac and respiratory activity (Cordes et al. 2001) and low frequency activity that may be related to scanner drift.

Together, the corrections performed on the time series included: detrending, outlier correction, motion regression, and CompCor correction (which were performed together in a single first-level regression model), followed by band-pass filtering. These corrections produced a residual BOLD time course at each voxel that was used for subsequent analyses.

Resting-state: First-level Functional Connectivity Analysis

For first-level resting-state analyses, we computed the Pearson’s correlation coefficient between the full time course of a bilateral seed region of interest (ROIs) in left and right NAc and the time course of all other voxels, yielding a correlation map for each participant. Correlation coefficients were normalized using the Fisher’s z-transformation. Functionally defined NAc ROIs were used for fMRI analyses in order to restrict the seed ROI to voxels that have been shown to correspond with regions responsive to rewards (vs. null feedback) in previous BOLD imaging research from an independent sample (Admon and Pizzagalli 2015). (For alternate views of functionally defined and structurally defined NAc ROIs, see Supplementary Fig. 2). Although primary analyses used a bilateral seed, to explore potential laterality effects, follow-up voxelwise analyses were performed using left and right NAc seeds independently.

Resting-state: Group-level Functional Connectivity Analysis

For group-level analyses, first-level normalized correlation maps were entered into a whole-brain regression analysis and group-level statistics were performed at each voxel. To identify regions in which NAc functional connectivity was associated with individual differences in reinforcement learning behavior, mean-deviated ΔRB was entered as the independent variable predicting the magnitude of correlations in activity between NAc and other regions of the brain (mean-deviated age of participant was entered as a covariate). Regions in which functional connectivity with NAc was associated with ΔRB were considered significant if they exceeded a peak amplitude of P < 0.05 (2-sided, i.e., P < 0.025 in each tail), cluster corrected within an intrinsic brain mask that restricts the search space to the SPM MNI template brain to False Discovery Rate of P < 0.05. Analyses were also repeated including sex and number of outlier images as group-level covariates; because controlling for these variables did not affect results, and these variables did not relate to ΔRB, simple analyses (with age as the only covariate) are reported. See Table 2 for report of resting-state functional connectivity in significant clusters of effect; no outliers were detected in the present sample (i.e., estimates of functional connectivity within 3 standard deviations of sample mean). For mediation analyses and in all figures, estimates of resting-state functional connectivity were z-scored.

Table 2.

Implicit reinforcement learning ability predicts resting-state functional connectivity of nucleus accumbens

	Peak Coord	Vol	Average FC of cluster		Correlation between implicit reinforcement learning and FC of cluster
			Mean	STDEV	r	Cluster P
Bilateral NAc seed
OFC	−16, 20, −10	1214	0.21	0.08	0.69	<0.001
Parietal cortex	54, −36, 56	1070	0.01	0.09	−0.66	0.001
Left NAc seed
OFC	8, 60, −24	1611	0.14	0.10	0.60	0.003
Right NAc seed
OFC/subcallosal	−20, 4, −16	2463	0.09	0.10	0.61	0.002
SMA	6, −16, 52	941	0.05	0.02	0.44	0.042
MFG	36, 40, 24	1292	0.02	0.13	−0.49	0.020
Parietal cortex	40, −42, 36	1704	0.01	0.09	−0.54	0.010

Note: Coord = coordinates in MNI space, Vol = volume in 1 × 1 × 1 mm voxels, FC = resting-state functional connectivity. Peak thresholded at P < 0.05 2-sided, cluster corrected to false discovery rate (FDR) of P < 0.05.

Mediation Modeling

The mediation model included the following variables: 1) DAT BP_ND (residualized for age) in the NAc, 2) functional connectivity between NAc and OFC (Fisher’s z-transformed correlation coefficients from the OFC region in which ΔRB predicted increased functional connectivity between NAc and OFC, extracted using REX (https://www.nitrc.org/projects/rex/), and 3) individual differences in reinforcement learning (ΔRB). The model tested the indirect relationship between DAT BP_ND and individual differences in RL behavior as mediated by NAc-OFC functional connectivity. To estimate the standardized regression coefficients for the associations between DAT BP_ND in NAc, resting-state functional connectivity of NAc-OFC, and reinforcement learning behavior, we performed regression analyses. Mediation was tested using a bootstrapping (5000 iterations) approach (Preacher and Hayes 2008) to estimate the indirect effect. Mediation and regression analyses were performed using SPSS19.

Results

Individual Differences in Reinforcement Learning Behavior are Associated with Striatal Dopamine Transporter Binding Potential

Correlation analyses tested the associations between individual differences in RL (z-scored ΔRB) and (z-scored, age-residualized) DAT BP_ND in left and right NAc (controlling for RL task version). Results showed that ΔRB was negatively associated with DAT BP_ND in bilateral NAc, r(31) = −0.43, P = 0.01, indicating that—as hypothesized—lower DAT availability was associated with better reinforcement learning (Fig. 2). Visual inspection of the scatterplot suggested potential outliers (data points exerting undue influence on the correlation); Cook’s D (Cook 1977) was calculated, and the correlation was repeated omitting (n = 3) data points with Cook’s D above the standard threshold (4/n; Bollen and Jackman 1985). Omitting these data points did not alter the overall pattern of results; although the association dropped to trend-level, the magnitude of the correlation remained a medium effect size (adjusted r = −0.32, adjusted P = 0.08) (Cohen 1992). Follow-up correlations to examine putative laterality effects revealed that ΔRB was negatively associated with DAT BP_ND in both left NAc, r(31) = −0.49, P < 0.01, and (at the level of a trend) in right NAc, r(31) = -0.34, P = 0.06.

Figure 2.

Individual differences in reinforcement learning behavior are associated with dopamine transporter binding potential (DAT BP_ND) in nucleus accumbens (NAc). In healthy adults, better reward learning performance (increased response bias towards a frequently rewarded stimulus) was related to lower DAT BP_ND in (A) bilateral NAc, r(31) = −0.43, P = 0.01. Follow-up analyses confirmed this inverse association independently in (B) left NAc, r(31) = −0.49, P < 0.01, (C) and (at the level of a trend) in right NAc, r(31) = −0.34, P = 0.06. Displayed on the x-axis are individual differences in z-scored change in response bias towards a reward-rich stimulus; on the y-axis are DAT BP_ND estimates (residualized for age, z-scored); all statistical tests were 2-sided.

Individual Differences in Reinforcement Learning Behavior are Associated with Frontostriatal Resting-State Functional Connectivity

Next, a voxelwise functional connectivity analysis was performed using seed ROIs in left and right NAc to identify regions in which functional connectivity with ventral striatum was associated with individual differences in reinforcement learning (z-scored ΔRB, controlling for age). Results showed that ΔRB was positively associated with functional connectivity between bilateral NAc and regions of medial OFC (Fig. 3), and negatively associated with functional connectivity between NAc and areas of posterior parietal cortex (Supplementary Fig. 3). Of note, we had no a priori hypotheses with respect to cortical systems beyond OFC, however, in light of interesting recent work showing that parietal systems may be especially relevant to reinforcement learning under conditions of high attentional control (Niv et al. 2015), these findings are displayed in greater detail in the Supplement and discussed below (Table 2).

Figure 3.

Correlation between reinforcement learning behavior and voxelwise resting-state functional connectivity of left, right, or bilateral NAc. Whole-brain resting-state functional connectivity (RSFC; Fisher’s z-transformed correlation coefficients) was computed for seed regions in either left or right NAc, or bilateral NAc, and correlated with ΔRB (i.e., the increase in response bias towards a more frequently rewarded stimuli across blocks). (A) Better learning behavior was related to increased RSFC between bilateral NAc and regions of orbitofrontal cortex (OFC) (peak −16, 20, −10, k = 1214). Follow-up analyses confirmed that better learning behavior was independently associated with (B) increased RSFC between left NAc and regions of OFC (peak 8, 60, −24, k = 1611), and (C) increased RSFC between right NAc and regions of medial cortex including OFC and (peak −20, 4, −16, k = 2463). Note: Cluster correction performed using peak P < 0.05, FDR; P < 0.05. Displayed are scatterplots showing the correlation between individual differences in z-scored change in response bias towards a reward-rich stimulus and z-scored RSFC between NAc seeds and clusters of effect indicated in black squares.

Follow-up voxelwise analyses revealed positive correlations between ΔRB and functional connectivity of left NAc with distributed regions of medial and lateral OFC, and of right NAc with regions of medial OFC and mid-cingulate (Table 2).

Frontostriatal Resting-state Functional Connectivity Mediates the Relationship Between Striatal Dopamine Transporter Binding Potential and Individual Differences in Reinforcement Learning Behavior

Taken together, the above results suggest that individual differences in RL may be driven by striatal DAT expression and frontostriatal circuit coordination. These neurobiological mechanisms may act in concert to shape or reflect RL behavior: prior data indicate that DA firing together with co-activation by cortical and limbic glutamatergic afferents drives striatal responses (Floresco 2015). Lower clearance of DA by DAT may therefore increase the impact of striatal RPE signals on frontostriatal pathways, resulting in better learning. To examine this hypothesis, we tested the indirect effect of DAT BP_ND on individual differences in reinforcement learning performance through resting-state functional connectivity of NAc (Fig. 4). Variables entered into the mediation model included DAT BP_ND in bilateral NAc (age-residualized and z-scored), resting-state functional connectivity (z-scored Fisher’s z-transformed correlations) between bilateral NAc and OFC, and RL behavior (z-scored ΔRB). Bootstrapped path-analysis (Preacher and Hayes 2008) revealed that frontostriatal resting-state functional connectivity significantly mediated the relationship between bilateral DAT BP_ND and learning behavior (confidence interval: −0.26 to −0.01). However, in a separate mediation model, resting-state functional connectivity between striatal and parietal regions did not significantly mediate the effect of bilateral DAT BP_ND on learning behavior (confidence interval: −0.36 to 0.005). Follow-up analyses showed a significant indirect effect of left NAc DAT BP_ND through left NAc-with-OFC functional connectivity on learning behavior (bootstrapped 95% confidence interval: −0.24 to −0.02); and a trending effect of right NAc DAT BP_ND through right NAc-with-OFC functional connectivity on learning behavior (bootstrapped 95% confidence interval: −0.23 to 0.00).

Figure 4.

Dopamine transporter binding potential (BP_ND) in bilateral nucleus accumbens (NAc) has an indirect effect on reinforcement learning ability through resting-state functional connectivity of NAc with orbitofrontal cortex (OFC). Variables entered in the mediation model were: DAT BP_ND in bilateral NAc (DAT BP_ND measured with the DAT radiotracer [¹¹C]altropane, residualized for age, z-scored), bilateral NAc resting-state functional connectivity (z-scored Fisher’s z-transformed correlation coefficients) extracted from OFC (FDR, P < 0.05), and reinforcement learning (z-scored change in response bias towards a frequently rewarded stimulus). Shown are standardized regression coefficients for each path in the model; the standardized regression coefficient between DAT BP_ND and reinforcement learning, controlling for NAc-OFC connectivity, is in parentheses; all statistical tests were 2-sided.

Discussion

The present study provides evidence that individual differences in RL are associated with DAT BP_ND and intrinsic frontostriatal functioning. Consistent with a wide range of evidence supporting the role of striatal DA in RL across people and species (Flagel et al. 2011; Cohen et al. 2012; Eshel et al. 2015; Hart et al. 2014; Schultz 2015), we observed that individual differences in RL in healthy humans were related to DAT BP_ND in ventral striatum and resting-state functional connectivity of a frontostriatal circuit linking ventral striatal regions with areas of orbitofrontal cortex involved in updating action plans. Moreover, mediated effects support a model in which DA clearance capacity may shape learning performance by contributing to the intrinsic strength of frontostriatal circuits. Collectively, these data suggest that DA re-uptake and frontostriatal circuit integrity are a key source of individual differences in reinforcement learning.

In the striatum, phasic DA release in response to better-than-expected outcomes is believed to facilitate learning for reward-predictive cues via enhanced long-term potentiation or depression of cortical glutamatergic inputs to striatal medium spiny neurons (Reynolds et al. 2001; Frank 2005). This well-supported model leads to several predictions for understanding individual differences in RL. First, lower DAT availability—and hence greater sensitivity to RPE signals owing to higher levels of DA—is predicted to promote reinforcement learning. In support of this idea, we observed that individuals characterized by lower DAT BP_ND showed higher reinforcement learning scores in behavioral testing. This finding contrasts to some prior research suggesting that DAT blockade does not influence learning, but instead, has effects on novelty seeking (Costa et al. 2014). However, differences in the nature of tasks used to probe RL behavior may contribute to mixed findings (modeling RL behavior using a modified multiarm bandit task vs. an implicit reward sensitivity task as in the present study), and the present findings are consistent with other research indicating that DAT has a role in RL (Everitt and Robbins 2005). Second, because the expected effect of larger DAergic RPE signals is to increase coupling between striatum and glutamatergic afferents, we predicted that lower DAT binding would be associated via frontostriatal connectivity with reinforcement learning behavior. Supporting this prediction, we observed that individuals who exhibited lower DAT BP_ND not only performed better in a reinforcement learning task but also this association was mediated by stronger resting-state functional connectivity between NAc and the OFC. Taken together, these results support the idea that striatal DA signaling contributes to the coordinated activity of striatal and orbitofrontal regions to update learned behaviors (Wilson et al. 2014; Schuck et al. 2016), and that individual differences in striatal DA can be linked to individual differences in RL.

Some limitations to the current study warrant additional comment. First, the present study was designed to target “intrinsic” individual differences in frontostriatal functioning, by investigating molecular (DAT signaling) and systems-level (functional connectivity) activity at rest. We note that prior research suggests that approximately half of variability in resting-state functional connectivity is attributable to trait-like individual differences (Buckner et al. 2013). However, a complementary approach would be to investigate individual differences in frontostriatal functional connectivity and molecular signaling during active demands for reinforcement learning. Efforts to extend the scientific questions of the present study to dynamic task-based neuroimaging, using high-resolution neuroimaging sequences, are underway. Second, the present study evaluated DAT concentrations and frontostriatal functional connectivity in separate sessions. This approach assumes stability over time of individual differences in molecular functioning and functional connectivity, an assumption that has been supported (Zuo and Xing 2014; Chen et al. 2015) but may have exceptions (Somandepalli et al. 2015). Therefore, capturing individual differences in molecular and systems-level functioning at the same time point, for example, through simultaneous PET/fMRI (Riedl et al. 2014; Bailey et al. 2016), may provide more precise information about the correspondence between DAT and frontostriatal activity. Third, the negative relationship between learning rate and DAT potential was reduced to a trend when 3 participants characterized by data points with Cook’s D above the standard threshold were excluded (adjusted r = −0.32, adjusted P = 0.08); clearly, independent replications of the current findings are needed. A final caveat to the described findings is that the behavioral measure of reinforcements learning used (Probabilistic Reward Task) was relatively simple, and only involved learning actions in response to a single stimulus dimension that were associated with possible gains. The relationship between DAT and other aspects of reinforcement learning, for example, learning from penalties or with more complex multidimensional stimuli, was not examined. Consequently, it is possible that the observed associations between DAT and performance would not generalize to these other experimental designs. It is also possible that other RL tasks would be useful to clarify the present finding of a negative association between RL behavior and resting-state functional connectivity in a parietal corticostriatal circuit. In this context, it is interesting to note that prior research that has shown that parietal and dorsolateral regions of an attention control network are important for selecting and updating which stimulus dimensions are relevant to reinforcement learning (Niv et al. 2015). Thus, it is possible that in the present study, poor learners responded to the task as more attentionally demanding than good learners; this possibility should be examined using an RL task that varies cognitive load.

Conclusion

In spite of the above limitations, our results highlight—we believe for the first time in humans—the importance of DAT mechanisms putatively involved in DA clearance and frontostriatal circuit functioning as markers of individual differences in RL. While prior studies have used measures of DA cell firing, pharmacological and optogenetic manipulation, and terminal efflux to demonstrate the role of DAergic RPE signals during reinforcement, these data provide novel evidence for DA re-uptake as a critical source of individual differences in human reinforcement learning. Future research may build upon these findings to investigate linkages between frontostriatal DA functioning and dimensions of personality (e.g., impulsivity, reward dependence), psychiatric health (e.g., anhedonia), and daily functioning.

Authors’ Contributions

All authors have approved the manuscript. D.A.P was responsible for study conceptualization and funding. F.G., L.M.M., M.B., and A.L.C. collected the data. R.H.K., M.T.T., D.W.W., P.P., and D.A.P. designed the analytic methods. R.H.K., D.W.W., and A.W conducted formal analysis. N.M.A., G.E.F., M.D.N., and D.A.P contributed resources for the present study. The manuscript was written by R.H.K. and M.T.T., and edited by D.W.W., P.K., M.D.N., and D.A.P.

Funding

Supported by National Institute of Mental Health grants R01 MH068376 and 1R01 MH101521 (DAP). Dr. Kaiser and Dr. Treadway were supported by 1F32 MH106262 and R00 MH102355, respectively.

Materials and Correspondence

Direct requests for materials and correspondence to Dr. Diego A. Pizzagalli Department of Psychiatry, Harvard Medical School, and Center for Depression, Anxiety and Stress Research, McLean Hospital, dap@mclean.harvard.edu