Modulation of Associative Learning in the Hippocampal-Striatal Circuit Based on Item-Set Similarity

Abstract

Mounting evidence suggests that the medial temporal lobe (MTL) and striatal learning systems support different forms of learning, which can be competitive or cooperative depending on task demands. We have previously shown how activity in these regions can be modulated in a conditional visuomotor associative learning task based on the consistency of response mappings or reward feedback (Mattfeld & Stark, 2015). Here, we examined the shift in learning towards the MTL and away from the striatum by placing strong demands on pattern separation, a process of orthogonalizing similar inputs into distinct representations. Mnemonically, pattern separation processes have been shown to rely heavily on processing in the hippocampus. Therefore, we predicted modulation of hippocampal activity by pattern separation demands, but no such modulation of striatal activity. Using a variant of the conditional visuomotor associative learning task that we have used previously, we presented participants with two blocked conditions: items with high and low perceptual overlap during functional magnetic resonance imaging (fMRI). As predicted, we observed learning-related activity in the hippocampus, which was greater in the high than the low overlap condition, particularly in the dentate gyrus. In contrast, the associative striatum also showed learning related activity, but it was not modulated by overlap condition. Using functional connectivity analyses, we showed that the correlation between the hippocampus and dentate gyrus with the associative striatum was differentially modulated by high vs. low overlap, suggesting that the coordination between these regions was affected when pattern separation demands were high. These findings contribute to a growing literature that suggests that the hippocampus and striatal network both contribute to the learning of arbitrary associations that are computationally distinct and can be altered by task demands.

1. Introduction

Many studies of memory function suggest that memory is not accomplished by a single system, but that there are multiple, anatomically distinct neural systems that support dissociable forms of learning and memory. For example, medial temporal lobe (MTL) structures are critical for the ability to rapidly form new long-term memories for facts, events, and relationships – often referred to as ‘declarative’ or ‘relational’ memory (Eichenbaum & Cohen, 2001; Squire, Stark, & Clark, 2004). A key feature of the declarative/relational memory system is its ability to quickly form new associations between arbitrary stimuli (Eichenbaum & Cohen, 2001). In contrast to the MTL, the striatum is known to be involved in a more incremental form of ‘habit’ or ‘procedural’ memory, tied to the gradual acquisition of stimulus-response associations using reward prediction error signals (Graybiel, 1995; Knowlton, Mangels, & Squire, 1996; Packard & Knowlton, 2002; Robbins, 1996; White, 1997). Classic studies of multiple memory systems have shown that the functions of these two areas can be doubly dissociated (Packard, Hirsh, & White, 1989; Packard & McGaugh, 1992). Some studies suggest that in certain situations, these two systems may compete for control of behavior (Moody, Bookheimer, Vanek, & Knowlton, 2004; R. A. Poldrack et al., 2001), while other studies have found that they can coordinate their activity on a given task (Doeller, King, & Burgess, 2008; Hartley & Burgess, 2005; Mizumori, 2008; Voermans et al., 2004).

Historically, the striatum was thought to contribute only to habit learning, but it is now recognized that the striatum is a heterogeneous structure that is also involved in flexible action-outcome learning (Bornstein & Daw, 2012; Diuk, Tsai, Wallis, Botvinick, & Niv, 2013; Tricomi, Balleine, & O’Doherty, 2009; Yin & Knowlton, 2006). The striatum is organized into three distinct, though partially overlapping anatomically defined divisions (Joel & Weiner, 1994, 2000; Parent, 1990; Seger, Peterson, Cincotta, Lopez-Paniagua, & Anderson, 2010) (Figure 1). The limbic division of the striatum includes the ventral striatum/nucleus accumbens and receives input from the amygdala, hippocampus, limbic, and paralimbic cortical areas. The associative division of the striatum includes the anterior caudate and anterior putamen and receives inputs from prefrontal, temporal, parietal, and cingulate cortices. The sensorimotor division of the striatum includes the posterior putamen and receives inputs from sensorimotor cortex. The vast majority of the neurophysiological studies of conditional visuomotor association learning have focused on the associative division of the striatum (Foerde, Braun, & Shohamy, 2013; Foerde & Shohamy, 2011; Miyachi, Hikosaka, & Lu, 2002). Striking learning-related signals have been observed in the caudate and putamen during a visuomotor task, particularly in the early and middle phases of learning (Buch, Brasted, & Wise, 2006; Romero, Bermudez, Vicente, Perez, & Gonzalez, 2008). For example, spiking in the caudate correlated with the learning rate while activity in the putamen correlated with the learning curve during a conditional associative learning task in monkeys (Williams & Eskandar, 2006). Research from our lab has shown that anterior regions of the striatum, particularly the caudate, tracked the learning rate of new associations (Mattfeld & Stark, 2015, 2011). These observations are consistent with the role of the associative striatum in flexible, goal-directed learning, similar to that observed in the medial temporal lobe (Yin & Knowlton, 2006).

Figure 1.

Schematic diagram of the connections and possible interactions between the Striatum and the MTL, based on those by Hikosaka and colleagues (1999; 2002). Based on the anatomical organization of the striatum, together with previous physiological findings, we propose that the strong interactions will be seen between the MTL and the associative division of the striatum, which may be modulated by pattern separation demands. Abbreviations: Asso (associative striatum); SM (sensorimotor striatum); EC (entorhinal cortex); HPC (hippocampus); PH (parahippocampal cortex); PR (perirhinal cortex).

The MTL is important for conditional visuomotor associative learning, particularly during the “flexible” early phase of learning and specializes in learning new associations between arbitrary elements in memory. Several computational models of the MTL have suggested that the hippocampus employs a process known as pattern separation to assist in the rapid storage of arbitrary associative information (McClelland, McNaughton, & O’Reilly, 1995; Treves & Rolls, 1994). This pattern separation process transforms similar representations into dissimilar representations (orthogonalization) so that learning can be rapid while avoiding deleterious interference effects that would result if the same connections used in prior learning were reused here. The cost comes in the form of a reduction in the ability to abstract regularities over the course of varied experience. It has been suggested that this set of constraints and trade offs may be a reason why the brain evolved multiple memory systems with different computational properties (McClelland et al., 1995; Sherry & Schacter, 1987).

The dentate gyrus (DG) subfield of the hippocampus has been heavily implicated in pattern separation processes within the MTL system. The sparse firing rate of the DG may result in unique, nonredundant representations ideal for this orthogonalization that then feed forward to the CA3 auto-associative network (Rolls, 2013). Activity consistent with pattern separation processing has been observed in firing rates in the DG to small changes in the environment that are insufficient to produce firing changes elsewhere in the hippocampus (Leutgeb, Leutgeb, Moser, & Moser, 2007; Neuneubel & Knierim, 2014). Likewise, functional neuroimaging studies in humans have shown the DG to be more responsive to small changes in input than other hippocampal subfields (Bakker, Kirwan, Miller, & Stark, 2008; Lacy, Yassa, Stark, Muftuler, & Stark, 2011). Thus, the DG has a pronounced role in the pattern separation processing occurring within the hippocampus.

We have utilized the conditional visuomotor associative learning task to study the nature of the interaction between the MTL and the striatum, in which subjects learn to associate a visual cue with a particular motor response. Performance on this task is dependent on the integrity of the MTL (Gaffan, 1992; Murray, Baxter, & Gaffan, 1998; Murray & Wise, 1996), with neurophysiological studies reporting strong MTL learning-related activity during task performance (Cahusac, Rolls, Miyashita, & Niki, 1993; Law et al., 2005; Mattfeld & Stark, 2015, 2011; Wirth et al., 2003). Previous studies have shown that while anterior striatal areas (associative striatum) are critical for the learning of sequential motor tasks, more posterior areas (sensorimotor striatum) are critical for the representation of well-learned sequences (Miyachi et al., 2002). Regions in the associative striatum were engaged during the learning of new arbitrary associations (Boettiger & D’Esposito, 2005; Toni, Ramnani, Josephs, Ashburner, & Passingham, 2001) and activity for both the learning of new and expression of familiar associations in the caudate nucleus suggests a dual role for this region in associative learning (Grol, de Lange, Verstraten, Passingham, & Toni, 2006). Subsequent studies have shown that anterior regions in the striatum were robustly modulated by reward prediction error signals during associative learning (Brovelli, Coquelin, & Boussaoud, 2007; Haruno & Kawato, 2006) while posterior regions of the putamen were modulated by how well an association was known (Haruno & Kawato, 2006). Increasingly, there is evidence for interactions between the hippocampus and the striatum during reward-based learning (G. E. Wimmer, Braun, Daw, & Shohamy, 2014; G. Elliott Wimmer & Shohamy, 2012), though it is not clear as to whether these interactions are inherently competitive or collaborative (Shohamy & Turk-Browne, 2013).

In an earlier report, we employed a manipulation of the consistency of feedback (100% vs. 80% valid) and of the consistency of the physical, motoric response (constant location vs. constant label and variable location) to alter the viability of MTL and striatal learning for solving the task (Mattfeld & Stark, 2015). Here, we tested their putative contributions to learning by altering the demands placed on learning via manipulating the similarity of the input stimuli. Using a similar conditional visuomotor associative learning task as before, we constructed sets of kaleidoscopic images with low or high perceptual overlap across the learning set (Figure 2). If a contribution of the hippocampus is the ability to learn different associations for even highly similar inputs via the process of pattern separation, the high-overlap condition should differentially tax the hippocampus. It should be noted that the hippocampus is expected to continue to show activity that is modulated by how well associations have been learned across both overlap conditions similar to our prior work, simply more so in the high overlap condition due to an increased demand for pattern separation processing. Specifically, we predicted that the dentate gyrus subregion should be more sensitive to the overlap condition than the CA1 subfield due to its unique role in pattern separation processing. In contrast, we predicted that learning in the associative striatum should equally contribute to associative learning in both the high and low overlap conditions. We further investigated the interaction between the hippocampus and the associative division of the striatum to assess the how the connectivity between these regions may be modulated by the high versus low overlap condition.

Figure 2.

Examples of A) High and B) Low perceptual overlap items used in the conditional paired associates task. C) The task structure involves the presentation of a stimulus, followed by a brief delay and then a trial-and-error response phase with feedback. Learning is calculated over time and represented as memory strength. D) Scan structure: 12 total scan runs in 3 phases. The high overlap items in the first and last block were not repeated in the last block.

2. Materials & Methods

2.1. Participants

A total of 24 young adults were recruited from the University of California, Irvine, provided written consent, and were compensated for their participation. One participant was excluded due to scanner error resulting in a loss of data, two subjects were removed for excessive data loss due to motion because they had >10% of TRs censored using a threshold of >1mm framewise displacement, and 1 was removed for moving 2cm between runs, resulting in the ROIs shifting out of the frame of view. Thus, we had a total of 20 usable participants (10 female; mean age 21 years; range 18–26 years). All participants had no history of neurological disease or psychiatric illness and normal or corrected-to-normal vision. This research was performed in compliance with the UCI Institutional Review Board.

2.2. Materials

Stimuli were computer-generated kaleidoscopic images that we have used previously (Law et al., 2005; Mattfeld & Stark, 2015, 2011) presented in the center of the screen against a black background. We created multiple sets of images with high perceptual overlap (Figure 2A) and low perceptual overlap (Figure 2B) that we drew from during the experiment. Low overlap sets contained 12 items to be learned concurrently and were constructed such that the colors used, base shape, and complexity (depth of recursion) were all random. High overlap sets contained 6 items to be learned concurrently and were constructed by maintaining the colors, complexity, and color at depth in the algorithm. Behavioral pilot testing revealed that this balance of number of items to be learned concurrently resulted in a comparable learning rate across low and high overlap sets, effectively matching difficulty.

2.3. Conditional Visuomotor Associative Learning Task

In the conditional visuomotor associative learning (CVAL) task, participants were informed that each kaleidoscopic image was associated with a specific response corresponding to one of the four squares on the screen, learning through trial-and-error feedback (Figure 2C). Each trial began with the presentation of a kaleidoscopic image and four horizontal superimposed square outlines (500 ms) representing their button choices. A brief delay (700 ms) followed the presentation of the stimulus in which the kaleidoscopic image was removed, leaving only the fixation cross and four square outlines on the screen. The response period (700 ms) began when the fixation cross was replaced by the cue “Go!”, during which the participants selected one of the four square outlines with an MR-compatible button box. The selected square outline filled with white, indicating which response had been recorded. Feedback (800 ms) was provided following the response period: a green “yes” if they were correct, a red “no” if they were incorrect, or a “?” if they failed to respond in time. The inter-trial interval (300 ms) consisted of a central fixation cross.

In addition to the CVAL trials, we included baseline trials as a contrast condition that shows reduced hippocampal activity compared to rest (Stark & Squire, 2001). Participants were shown a fixed random visual static pattern with four squares filled with transparent white. One of the squares was randomly assigned as the target on each baseline trial and set to a slightly greater opacity than the three remaining options. Participants were instructed to identify the brightest box and press the corresponding button. Following their responses, the selected button turned fully-opaque white, indicating their selection and deterministic feedback was provided. The opacity of the target and foil boxes were initially 0.3 and 0.2 respectively. Difficulty was titrated to maintain above chance, but below ceiling performance. If performance was 80% or better across a 10-trial moving window, then the target opacity was reduced by 10% of its current value. Conversely, if performance was below 50%, the target opacity was increased by 10% of its current value.

2.4. Prescan Training

Participants were trained on an independent set of kaleidoscopic images 24–48 hours prior to scanning in order to familiarize themselves with the task. They performed two runs with different sets of 12 low-overlap items, each presented 6 times, and one run with 6 high-overlap items, each presented 12 times, for a total of 72 trials for each run. Each run also contained 20 baseline trials, randomly intermixed.

2.5. fMRI Imaging

All scanning was performed on a Phillips 3.0 Tesla Scanner (Best, the Netherlands), using a 32-channel sensitivity encoding (SENSE) head coil at the Research Imaging Center at UC Irvine. During each scanning run, 138, T₂*-weighted, single-shot echo-planar volumes were acquired along the long-axis of the hippocampus covering the basal ganglia and the majority of the MTL in 34 slices (the temporal-polar portion of the MTL was not covered). Each slice was 1.8 mm thick separated by a 0.2 mm gap. Functional pulse sequences had a repetition time (TR) of 2000 ms, an echo time (TE) of 26 ms, a flip angle of 70°, an acquisition matrix size of 128 × 128 mm, a field of view (FOV) of 180 × 180 mm, and a SENSE factor of 2.5, resulting in an in-plane resolution of 1.5 × 1.5 mm. The first four functional volumes were discarded to accommodate for T1 equalization. Additionally, T1-weighted whole-brain anatomical images were acquired using a sagittal magnetization-prepared rapid gradient echo (MP-RAGE) scan (TR 11ms; TE 4.6 ms; flip angle 18°; matrix size 320 × 320 mm; FOV 240 × 150 mm; resolution 0.75 mm isotropic; 200 slices). Each fMRI run contained a total of 72 CVAL trials and an additional 20 baseline trials, for a total of 92 trials per run.

Functional runs were divided into a high overlap block of three scan runs, a low overlap block of six scan runs, and another high overlap block of three scan runs (using a different set of stimuli for each set of scans). Each run, regardless of overlap, had 72 behavioral trials. For high-overlap runs, this consisted of six stimuli presented twelve times each within a run. For low-overlap runs, this consisted of twelve stimuli presented six times each within a run. Thus, in the first set of three high-overlap runs, participants would have had 36 trials (12 times each run for 3 runs) to learn the associations for each of six stimuli. In the set of six low overlap runs, participants would have had 36 trials (6 times each for 6 runs) to learn the associations for each of 12 stimuli. Finally, in the last set of three high overlap runs, participants would have had 36 trials (12 times each for 3 runs) to learn the associations for each of a separate set of six stimuli. Thus, the total number of trials to learn each association was constant (36) and the number of associations learned (12) was constant across high vs. low-overlap. What varied (in addition to the overlap) was the number being learned concurrently.

2.6. Behavioral Data Analysis

Consistent with our previous work (Law et al., 2005; Mattfeld & Stark, 2015, 2011), for each kaleidoscopic image, we used a logistic regression algorithm to calculate a trial-by-trial probability correct estimate as well as its 95% confidence interval (Smith, Stefani, Moghaddam, & Brown, 2005; Wirth et al., 2003). The algorithm uses a state equation (Gaussian random walk model) and an observation equation (Bernoulli model) to calculate trial-by-trial probability correct estimates for each stimulus based on behavioral performance (1 correct; 0 incorrect). These stimulus specific learning curves provide a measure of trial-wise learning. We then grouped the trial-specific probability correct estimates into five equivalently spaced bins to create discrete memory strength indices (Str1 to Str5). We then used these memory strength indices to assess learning related changes in BOLD fMRI activity, based on how well each item to response association has been learned (with 5 being the best learned). By combining trials of similar memory strengths across stimuli, we increase our signal to noise when examining learning curves. In addition, this approach mitigates concerns over variations in learning rates across conditions because trials in each memory strength bin share the same approximate probability correct.

2.7. fMRI Pre-processing

We used Analysis of Functional Neuroimages (AFNI; version 17.1.09)(Cox, 1996) software to perform most of the imaging data analyses. Functional data for the univariate analyses were slice-time and motion corrected using rigid-body transformation using the function align_epi_anat.py (Saad et al., 2009) and high pass filtered (f>0.016 Hz) using the 3dBandpass function. Time points exceeding 3° rotation, 1 mm translation, or 2 standard deviations from the mean within run global signal intensity, as well as immediately adjacent TRs were removed. Each subject’s anatomical image was segmented into grey matter, white matter, and cerebral spinal fluid probability (CSF) maps using Freesurfer (Dale, Fischl, & Sereno, 1999; Fischl et al., 2002). In order to reduce the effects of physiological noise in the BOLD signal, we passed the filtered data through ANATICOR (Jo, Saad, Simmons, Milbury, & Cox, 2010) using the white matter and CSF maps created from Freesurfer. Each run was then concatenated into a single time series for each participant.

We used the ANTS toolkit (Avants, Epstein, Grossman, & Gee, 2008) to normalize each participant’s T1-weighted MP-RAGE to a MNI template. ANTS combines a 12-parameter affine registration with a diffeomorphic 3D vector field mapping (SyN) to perform invertible, smooth mappings between the original participant space and the model template space. We used a hand segmentation of the MTL, in template space, from previous work in our lab (Lacy et al., 2011) to define the anatomical ROIs for hippocampus, perirhinal, parahippocampal, and entorhinal cortices for both hemispheres and subsequently reverse warped the anatomical masks into the participant’s original space using ANTS’ multi-label interpolation. Similarly, we segmented the hippocampus into 3 regions: a combined dentate gyrus and CA3 (DG/CA3), CA1, and subiculum, based on our previous work (Stark & Stark, 2017) and warped these onto the individual. For the striatal regions of interest, Freesurfer segmentation routines (Dale et al., 1999; Fischl et al., 2002) were employed to identify the nucleus accumbens, the caudate, and the putamen regions for each participant in that participant’s original space. Further separation of the putamen into anterior and posterior regions was completed by splitting the putamen at slice y = 0. In the end three seed ROIs per hemisphere were created: the limbic division of the striatum was represented by the nucleus accumbens, the associative division included the caudate and the anterior putamen, and the sensorimotor division was represented by the posterior putamen. The resulting transformation parameters were then applied to the functional data.

2.8. fMRI Memory Strength Analyses

Behavioral design matrices included regressors for trials from the five memory strength indices, as well as the first presentation of each stimulus. We chose to include all five memory strength bins in our analysis, consistent with our previous studies (Law et al., 2005; Mattfeld & Stark, 2015, 2011). Using 3dDeconvovle (AFNI), we used a deconvolution approach based on multiple linear regression to analyze each participant’s data. The hemodynamic responses for each event of interest was estimated using 9 time-shifted tent functions, estimating the BOLD activity from 0–16 seconds after trial onset. The resulting time-shifted beta coefficients represent activity versus the perceptual baseline for each regressor of interest at a given time point in each voxel. To estimate each regressor of interest in the model, we combined the beta coefficients, summing over 4 to 12 seconds after trial onset. This experimental design does not allow us to isolate activity related to distinct periods within each trial (e.g., action selection, feedback, etc.). Thus, the summed estimate for each regressor of interest cannot be used to interpret specific within trial processes, but instead, reflects the activity over the entire trial.

Based on our prior work with this task (Law et al., 2005; Mattfeld & Stark, 2015, 2011), we hypothesized that changes in hippocampal learning would exhibit a monotonic increase, consistent with a roughly linear trend that varied as a function of high or low overlap. In contrast, we predicted that the associative striatum would demonstrate a quadratic relationship with memory strength. Therefore, we performed an anatomical analysis on the following bilateral ROIs: hippocampus, with a specific comparison between the DG/CA3 and CA1 subfields), and the associative striatum (the combined anterior putamen and anterior caudate). In exploratory analyses, we also examined activity in the entorhinal cortex (ERC), perirhinal cortex (PRC), parahippocampal cortex (PHC), posterior caudate, posterior putamen, pallidum, and nucleus accumbens, evaluating the betas for each of the 5 memory strengths. We then calculated a 2×5 repeated-measures ANOVA with overlap (high or low) and memory strength (1,2,3,4,5) as variables for each region. One participant was a clear outlier, with beta values greater than 3 standard deviations greater than the mean in the hippocampus and was therefore removed from that analysis.

2.9. Functional Connectivity Analyses

To assess functional connectivity between the striatum and medial temporal lobe, we employed ROI-to-ROI correlation analyses. Functional connectivity refers to the temporal correlations of the time series between spatially separated brain regions, which may be modulated by task demands. The functional connectivity analysis pipeline differed from the previous preprocessing pipeline in the following ways: we used a temporal bandpass filter of 0.009 to 0.1 Hz (Fox et al., 2005) and we used a stricter criterion to censor out TRs due to excessive motion (0.5mm and/or 0.5 degrees). From the regression analysis using 3dDeconvolve (AFNI), the average residual time-series were then extracted separately for the high and low overlap runs for each ROI. Pearson’s r correlation coefficients were then calculated between regions. We first wanted to examine functional connectivity between the hippocampus (and additionally, the dentate gyrus) and the associative division of the striatum. We compared the correlations for the high and low conditions using paired t-tests. Then we examined functional connectivity between the three striatal divisions (associative, limbic, and sensorimotor) and the hippocampus to evaluate the specificity of this functional connectivity pattern. These correlation values were Fisher’s r-to-z transformed and 2 (interference: high or low) × 3 (striatal division: limbic, associative, sensorimotor) repeated-measures ANOVAs were calculated on the transformed values. All statistical analyses were performed using Prism 6.0h (www.graphpad.com).

3. Results

3.1. Behavioral Results

The estimated onset of learning was defined as the trial when the lower 95% confidence interval exceeded chance performance (25%). This measure was well-matched across conditions as the 9.6 median trials required in the high-overlap condition (SD = 3.5) was not reliably different than the 8.3 median trials required in the low-overlap condition (SD=4.2; paired-t(19) = 1.12, p = .28). Additionally, there was no difference between conditions when the high overlap condition was separated into the first (median: 10.45) and second (median: 11.8) blocks (F(2,19) = 2.1, p = .15) A more robust measure of learning is the area under the curve measuring the probability correct for each item (or in its quantized version, the sum of the five strength bins per item). Here, we could resolve a small difference in performance despite our pilot testing to match difficulty across conditions as there was a greater area under the curve for the low-overlap condition (mean = 27.7, SD = 3.5) than the high-overlap condition (mean = 24.5, SD = 3.7; paired-t(19) = 4.2, p<.01), reflecting slightly more rapid learning in the low-overlap condition. A comparison of the two high overlap conditions revealed no difference between the first (24.2) and second (24.9) blocks (paired-t(19) = .53, p=.60). We observed similar learning behavior across the conditions (Figure 3). Learning curves were created by averaging the trial-wise probability-correct estimates across all stimuli within a block and then across subjects. The shading indicates the averaged 95%-confidence interval of this estimate.

Figure 3.

Average learning curves across stimuli and subjects for the low and high overlap blocks.

3.2. fMRI Memory Strength Results: Hippocampus & Associative Striatum

In the hippocampus, a 2×5 repeated-measures ANOVA (condition: high vs low overlap and memory strength: strengths 1–5) revealed differences in activation in the high compared to low-overlap condition (main effect of condition: F(1,17) = 4.3, p=.05), that changed with memory strength (main effect of memory strength: F(4, 68) = 27.1, p<.0001) similarly across conditions (no significant interaction: F(4,68) = 2.0, p=.10) (Figure 4A). Post hoc analyses revealed that the main effect of condition was driven by greater activation in the hippocampus for the high than the low condition (t(18) = 2.1, p=.05). Trend analyses supported the observation that the main effect of memory strength was the result of both linear (F(1,88) = 18.7, p < 0.001) and quadratic (F(1,87) = 11.9, p < 0.01) increases in activation across learning. Note, that this quadratic component is driven heavily by the first memory strength bin where the estimate of the probability correct on that trial is well below chance (0–0.2, chance=0.25). Without this memory strength bin, the trend is better fit by a line (extra-sum-of-squares F-test). We chose not to overly interpret the value of this bin and include it in other analyses presented here but draw attention to the potential issue by lightly shading out this point in relevant figures.

Figure 4.

A) Bilateral anatomical analysis of the hippocampus (in red) revealed both an effect of memory strength and effect of overlap, with higher beta values for the high overlap condition. The dashed lines show the nonlinear fit curves, which differed for the high and low conditions. B) A bilateral anatomical analysis of the associative striatum (anterior caudate and anterior putamen in yellow, posterior putamen (not part of associative striatum) in red) also showed a quadratic relationship with memory strength, but in the opposite direction than in the hippocampus, with no effect of overlap. Memory strength bins are evenly spaced based on probability of correct performance making the Strength 1 bin (0–0.2 probability correct]) reflect numerically below-chance performance, which is shaded in these graphs.

In contrast, in the associative striatum, we observed activity that varied by memory strength, but not by overlap condition (Figure 4B). A 2×5 ANOVA found changes in activation across memory strength (main effect of memory strength: F(4,76) = 2.6, p<.05), but no differences in activation in the high and low overlap conditions (no main effect of condition (F(1,19) = .03, p = .87) and no interaction (F(4,76) = .34, p = .85)). Post hoc trend analyses showed that the main effect of memory strength was best captured by a negative quadratic trend (F(1,97) = 3.7, p < 0.06). In order to concretely address whether these two regions varied in their activity due to overlap, we averaged the data across the 5 memory bins and entered them into a repeated-measures 2×2 ANOVA (condition: high vs low overlap and region: hippocampus and striatum). We observed greater activity in associative striatum than the hippocampus (main effect of region: F(1,17) = 21.91, p<.001), and critically, a trend towards a significant interaction between overlap and condition (F(1,17) = 3.89, p =. 06). Sidak’s multiple comparisons test revealed an effect of high vs low overlap for the hippocampus (t(17) = 2.46, p = .049), but not for the associative striatum (t(17) = 0.33, p =. 94).

In summary, activity in the hippocampus was greater during the high overlap condition but increased similarly for both conditions across memory strength. While activity in the associative striatum was not sensitive to overlap, it showed a decrease in activity across memory strength when associations were well-learned. Thus, as predicted, pattern separation demands modulated learning related activity in the hippocampus, but not in the associative striatum.

3.3. fMRI Memory Strength Results: DG/CA3 and CA1 Hippocampal Subfields

We hypothesized that the dentate gyrus (DG/CA3; combined with the CA3 here due to limitations in resolution allowing for accurate differentiation) would demonstrate a difference in activity for high vs low overlap, consistent with pattern separation processing, while the CA1 would be agnostic to this manipulation. Consistent with this hypothesis, the DG/CA3 showed a difference in activation in the high and low overlap conditions (main effect of condition: F(1,17) = 6.0, p=.02) that changed across learning (main effect of memory strength: F(4,68) = 14.0, p<.0001), but no interaction (F(4,68) = 1.0, p = .39). Post hoc analyses revealed that the main effect of condition was driven by greater activation in the DG/CA3 for the high than the low condition (t(18) = 2.4, p<.05). Post hoc trend analyses also revealed quadratic increases across memory strength for both high (F(1,87) = 3.6, p=.06) and low (F(1,87) = 7.2, p<.01) overlap conditions and an extra-sum-of-squares F-test found these to curves to differ (p<0.05). Meanwhile, the CA1 showed a change in activity across learning (main effect of memory strength: F(4,68) = 16.16, p<.0001), but no modulation by low and high overlap condition or interaction. Post hoc trend analyses revealed a quadratic increase in activity across memory strength (F(1,87) = 12.2, p<.001). These findings (Figure 5) are consistent with previous reports showing greater activity in the DG/CA3 when pattern separation demands are high (Bakker et al., 2008; Kirwan & Stark, 2007; Lacy et al., 2011).

Figure 5:

A) Representative slices (and corresponding anterior-posterior position in Talairach space) demonstrating the segmentation of the hippocampus and surrounding medial temporal lobe cortex. Hippocampal subfields: CA1 (fuschia), DG/CA3 (yellow), and subiculum (pink). Medial temporal lobe: entorhinal cortex (purple), perirhinal cortex (green), and parahippocampal cortex (blue). B) The DG/CA3 showed a main effect of high and low overlap with quadratic increases across memory strength. In contrast, the CA1 subfield also showed a quadratic increase with memory strength, but no modulation by overlap condition.

3.4. fMRI Memory Strength Results: Exploratory Analyses

After examining our a priori regions of interest, we were interested in whether other medial temporal (entorhinal (ERC), perirhinal (PRC), and parahippocampal cortex (PHC)), subiculum subfield of the hippocampus, or striatal regions (sensorimotor striatum = posterior putamen, and limbic striatum = nucleus accumbens) showed a modulation of activity by high and low overlap. We implemented 2×5 repeated-measures ANOVAs (condition: high vs low overlap and memory strength: strengths 1–5) and Bonferroni-corrected for multiple comparisons of 6 ROIs for a modified p-value threshold of p<.001. We found an effect of learning in the PRC (main effect of memory strength: F(4,76) = 5.6, p=.0005) and the effect of memory strength was best modeled by an increasing linear trend across memory strength (F(1,93) = 5.48, p<.05). Additionally, we found an effect of learning in the subiculum (main effect of memory strength: F(4,68) = 8.5, p<.001) but no effect of high or low. In a number of these regions, we found some evidence for an effect of overlap similarity, but they were less reliable as they failed to pass the threshold set when correcting for multiple comparisons, which can be seen in Table 1.

Table 1.

Results of Exploratory Analyses (shaded areas passed statistical threshold for multiple comparisons).

Region	Condition	Strength	Interaction
Entorhinal Cortex	F(1,19) = 0.6, p=.45	F(4,76) = 1.8, p=.l4	F(4,76) = 2.0, p=.10
Perirhinal Cortex	F(1,19) = 4.1, p=.06	F(4,76) = 5.6, p=.0005	F(4,76) = 0.6, p=.63
Parahippocampal Cortex	F(1,19) = 6.2, p=.02	F(4,76) = 3.0, p=.03	F(4,76) = 0.8, p=.53
Subiculum	F(1,18) = 6.3, p=.02	F(4,68) = 8.5, p<.001	F(4,72) = 1.8, p=.l4
Sensorimotor Striatum	F(1,19) = 1.2, p=.29	F(4,76) = 3.5, p=.01	F(4,76) = 0.9, p= .44
Limbic Striatum	F(1,19) = 0.2, p=.70	F(4,76) = 1.2, p=.32	F(4,76) = 0.7, p= .61

3.5. Functional Connectivity Results: Associative striatum

We hypothesized that if the associative striatum and hippocampus shared a cooperative relationship, then the functional connectivity of these two regions should be modulated by the high vs low overlap conditions. Indeed, in this planned comparison, we found evidence for greater functional connectivity between the hippocampus and associative striatum in the high (mean: .26) than the low condition (mean: .21) (paired-test: t(18) = 1.97, p = .06), driven in large part by functional connectivity in the DG/CA3 (paired t-test: t(18) = 2.69, p=.02; high mean: .22; low mean: .14). While these findings are consistent with our hypothesis, it is possible that the functional connectivity between the hippocampus and other regions of the striatum would be similarly affected.

Therefore, we followed up this planned comparison by exploring the functional connectivity between the hippocampus and all three divisions of the striatum (associative, limbic, and sensorimotor) post hoc. We entered the data into a 2 (overlap: low or high) × 3 (striatal division: limbic, associative, and sensorimotor) repeated-measures ANOVA (Figure 6). However, there was no main effect of striatal division (F(2,36) = 2.2, p=.13) or overlap (F(1,18) = 2.7, p=.12), and no significant interaction (F(2,36) = 1.2, p=.31). We repeated this analysis with functional connectivity for just the DG/CA3 and found a marginal main effect of overlap (F(1,18) = 3.66, p=.07; high mean: .20, low mean: .16), and a marginal interaction (F(2,36) = 2.94, p = .07), but no effect of striatal division (F(2,36 = 1.4, p=.25). In follow-up paired t-tests, we observed greater modulation of high and low overlap in the DG/CA3 for the associative striatum than the sensorimotor striatum (t(18) = 2.43, p<.05) and marginally for limbic striatum (t(18) = 1.99, p=.06). These results reinforce the planned comparisons, showing modulation of the functional connectivity difference between high and low overlap for the associative striatum compared to the limbic or sensorimotor divisions of the striatum, although some of these effects are only marginally significant.

Figure 6.

Means and standard error of the mean (SEM) separately for the high (red) and low (blue) overlap conditions. Values represent Fisher’s z transformed r values for the functional connectivity ROI-to-ROI analysis. Planned paired-comparisons of the low and high condition for each striatal division resulted in a marginal difference for functional connectivity between the hippocampus and associative striatum (A) and reliable modulation of connectivity between the DG/CA3 subfield and associative striatum (B). *** p<.050020

4. Discussion

The goal of the current study was to evaluate hippocampal and associative striatal contributions to learning a conditional visuomotor associative learning task when the stimuli had high or low perceptual overlap. Consistent with our prior findings (Law et al., 2005; Mattfeld & Stark, 2015, 2011), we predicted that activity in the hippocampus would track learning (e.g., memory strength), which might be exacerbated when pattern separation demands are high (e.g., the high overlap condition). We found that the hippocampus does indeed track learning in this task. More interestingly, activity was higher in the hippocampus for high overlap than low overlap trials, perhaps reflecting the greater demands on pattern separation supported by the dentate gyrus. Indeed, the combined DG/CA3 hippocampal subfield showed this pattern, while the CA1 subfield did not. In contrast to the hippocampus, we predicted learning-related activity in the associative striatum would not be modulated by high or low overlap, which is what we observed here.

These results support a growing number of studies and models that propose that the hippocampus (and the dentate gyrus in particular) uses pattern separation and pattern completion mechanisms to be able to rapidly store and later retrieve arbitrary associative information (Norman & O’Reilly, 2003; Rolls, 2016; Yassa & Stark, 2011). While pattern separation involves the orthogonalization of overlapping representations into distinct memory traces, pattern completion refers to the process by which stored memory representations may be reactivated based on noisy or degraded cues (often thought of as driven by attractor dynamics). Evidence for the role of these subfields in pattern separation and completion has been mounting in both animal and human studies. Rodent studies have demonstrated that lesions to the dentate gyrus (DG) result in failures of pattern separation behavior (Gilbert, Kesner, & Lee, 2001; Goodrich-Hunsaker, Hunsaker, & Kesner, 2008) and that DG cells have been found to alter their firing in response to small changes in input that are insufficient to alter firing patterns elsewhere in the hippocampus (Leutgeb et al., 2007; Neuneubel & Knierim, 2014). Likewise, in humans, BOLD fMRI activity consistent with pattern separation has been observed in the DG or combined DG/CA3 subfields (Bakker et al., 2008; Berron et al., 2016; De Shetler & Rissman, 2017; Lacy et al., 2011). In the current data, greater DG/CA3 activity in the high overlap condition may be driven by pattern separation processes occurring in the dentate gyrus.

The hippocampus receives inputs from the perirhinal (PRC) and parahippocampal (PHC) cortices via the entorhinal cortex. These regions have traditionally been associated with the ‘what’ and ‘where’ pathways, with the PRC involved in object recognition memory (Ahn & Lee, 2015; Watson, Wilding, & Graham, 2012) and the PHC involved in spatial memory processing (Bohbot et al., 2015; Duzel et al., 2003; R. Epstein, Harris, Stanley, & Kanwisher, 1999; Russell Epstein, Smith, & Ward, 2009). In an exploratory analysis of these medial temporal regions, only the PRC showed learning-related activity, with increasing values across memory strength (consistent with object-like learning), but no modulation by overlap condition.

The striatal learning system is comprised of a variety of networks also involved in learning, but with unique property dynamics. We divided the striatum into three divisions following the work of others (Hikosaka et al., 1999; Yin & Knowlton, 2006): 1) the sensorimotor division of the striatum (including the posterior putamen) is responsible for the execution of motor sequences; 2) the associative division of the striatum (including the anterior caudate and anterior putamen) is predominately involved in the early stages of learning; and 3) the limbic division of the striatum (including the ventral striatum/nucleus accumbens) contributes to performance of well-learned sequences. We focused on the associative striatum, which coordinates with prefrontal cortex (PFC) during learning and is strongly modulated by action-outcome contingencies that are critical during the early stages of learning (Poldrack et al., 2005; Yin & Knowlton, 2006). Here, we observed that the associative striatum showed a modulation of activity by memory strength, consistent with prior studies (Mattfeld & Stark, 2015, 2011), where we described a quadratic relationship across memory strengths, which could be modulated by stochastic feedback (Mattfeld & Stark, 2015). Activity levels were relatively flat over memory strengths 1–4 and then dropped for memory strength 5. This pattern mirrors the rise in beta values in the hippocampus for these very well-learned items, raising the possibility that these regions may be cooperatively interacting during the course of learning.

While we are unable in this design to disentangle cue and feedback responses, these results are also consistent with single unit recordings in the monkey caudate (Williams & Eskandar, 2006) and fMRI in humans (Reavis, Frank, & Tse, 2015) that are correlated with learning rate and prediction error in which the difference between the expected and received reward is greatest early in learning and declines once the associations are well-learned (McClure, Daw, & Montague, 2003; O’Doherty, Dayan, Friston, Critchley, & Dolan, 2003; Pagnoni, Zink, Montague, & Berns, 2002; Schiffer, Ahlheim, Wurm, & Schubotz, 2012). Surprisingly, we did not observe significant learning-related activity in the limbic striatum, which has shown a strong relationship with the hippocampus (Mattfeld & Stark, 2015).

While activity in the associative striatum was modulated by learning, it was not sensitive to the degree of similarity between the stimulus sets. These findings again promote the utility of the hippocampus for pattern separation of similar representations, while the associative striatum is more sensitive to action-outcome contingencies. We were interested in evaluating the dynamics between these two memory systems during learning, so we turned to functional connectivity analyses. Interestingly, we found greater functional connectivity between the hippocampus and associative striatum during the high overlap than the low overlap condition. Further, this relationship was emphasized when isolating the DG/CA3, again consistent with the unique role of this region in pattern separation processing. These findings suggest coordination between the hippocampus and the associative subdivision of the striatum when pattern separation demands are high, requiring greater coordinated activity between the hippocampus and striatum during learning of the task. We observed greater modulation of functional connectivity by overlap between the hippocampus and the associative striatum than the sensorimotor or limbic striatum, emphasizing a unique relationship between the associative striatum and hippocampus during learning. Indeed, these results are consistent with findings that habit formation involves a shift from the associative striatum learning the action-outcome contingencies early in learning to stimulus-response habits supported by the sensorimotor network later in learning (Yin & Knowlton, 2006).

To our knowledge, these are the first data to report coordination between the DG/CA3 and the striatum. These data support observations that these brain regions interact, dynamically gating information from the hippocampus to the anterior striatum through dopaminergic mediated plasticity (Goto & Grace, 2005; Lisman & Grace, 2005; O’Donnell & Grace, 1995), which is no longer needed once the associations are well-learned. Diffusion tractography in humans have also identified connectivity between the anterior putamen (along with the anterior caudate makes up the associative striatum) and the hippocampus (Lehéricy et al., 2004). Further, greater functional connectivity between the hippocampus and ventral striatum has been reported for overlapping compared to non-overlapping sequences of faces (Ross, Sherrill, & Stern, 2011). These regions, along with the orbitofrontal cortex (OFC) which may mediate activity between the two, may be important for a network involved in the selection of the appropriate response depending on contextual information (Ross et al., 2011). Here, we suggest that increasing the pattern separation demands in the high overlap condition biased learning in the hippocampus that modulated the association-outcome mapping in the striatum. Specifically, the increased connectivity of the hippocampus (and DG/CA3 in particular) and associative striatum during high-overlap trials may be a reflection of the hippocampal retrieval of the correct response association, which reinforces the correct association-outcome response exhibited by the associative striatum. In the low overlap condition, the associative striatum may be capable of creating the association-outcome contingencies with less hippocampal contribution because there is less competition between items.

One possible limitation of the current study involves the role of working memory in the study design. Here, we chose to equate difficulty across the low and high overlap conditions by presenting a different number of concurrent stimuli (12 low overlap and 6 high overlap). However, this design does introduce a confound of differing memory load and spacing between repeated items (e.g. both in terms of actual time and number of intervening items). There is evidence that working memory load contributes to behavioral performance on a reinforcement task very similar to this one (Collins & Frank, 2012). Further, working memory has been shown to interact with reinforcement learning in the striatum, showing blunted signals in low set sizes compared to higher set sizes (Collins, Ciullo, Frank, & Badre, 2017). However, in this study, their set sizes ranged from 1–6, with the blunted response occurring in set sizes of 1 and 2. In contrast, both our low and high overlap conditions corresponding to their higher set sizes (6+) that places them outside the range of an exclusively working memory strategy. Additionally, we did not find any difference in striatal activity for the low and high overlap conditions, further reducing the likelihood that any possible differences working memory contributed to these results. It is worth noting though that the rate of learning was more rapid in the low overlap than the high overlap condition, which may have contributed to some of the differences in activity that we observed here. Future study designs should either explicitly explore the possible role of working memory contributions to these results or address matching of behavior using another study design.

In this study, we focused on a hypothesis-driven contrast between the hippocampus and associative striatum. To that end, we restricted our data acquisition region to the medial temporal lobe and striatum, with limited coverage of other brain regions. This approach allowed us to scan at a higher-resolution in order to examine hippocampal subfield contributions to this learning. However, future studies should focus on a more holistic approach to better map the whole-brain circuitry and contribution of the PFC to this learning. In addition, it would be interesting to observe the activity differences across regions in a design with greater specificity that could isolate the cue and feedback stages independently. Finally, we were unable to resolve a relationship with memory strength for the functional connectivity analyses, possibly because we were underpowered for this type of analysis. We would predict greater coordination between the hippocampus and associative striatum early in learning, particularly for the high overlap trials, which may require pattern separation processes in the hippocampus.

5. Conclusions

In the present study, we examined the functional roles of the hippocampus and associative striatum under conditions of high and low perceptual overlap during a conditional visuomotor associative learning associates memory task. We found differential activity in the hippocampus for high vs. low overlap items, presumably because the images with high perceptual overlap taxed pattern separation processes, a primary function of the dentate gyrus subregion of the hippocampus. In contrast, the associative striatum showed activity that tracked learning in the task but was not modulated by the similarity of the stimuli. Functional connectivity between the hippocampus and the associative striatum was differentially modulated by high vs. low perceptual overlap, suggesting that the coordination between these regions was affected when pattern separation demands are high. These findings contribute to a growing literature that suggests that the hippocampus and striatal network both contribute to the learning of arbitrary associations that are computationally distinct and can be altered by task demands.

Once learned however, the regions likely play distinct roles in the maintenance of conditional visuomotor associations.