Neural Interactions Underlying Visuomotor Associations in the Human Brain

Abstract

Rapid and flexible learning during behavioral choices is critical to our daily endeavors and constitutes a hallmark of dynamic reasoning. An important paradigm to examine flexible behavior involves learning new arbitrary associations mapping visual inputs to motor outputs. We conjectured that visuomotor rules are instantiated by translating visual signals into actions through dynamic interactions between visual, frontal and motor cortex. We evaluated the neural representation of such visuomotor rules by performing intracranial field potential recordings in epilepsy subjects during a rule-learning delayed match-to-behavior task. Learning new visuomotor mappings led to the emergence of specific responses associating visual signals with motor outputs in 3 anatomical clusters in frontal, anteroventral temporal and posterior parietal cortex. After learning, mapping selective signals during the delay period showed interactions with visual and motor signals. These observations provide initial steps towards elucidating the dynamic circuits underlying flexible behavior and how communication between subregions of frontal, temporal, and parietal cortex leads to rapid learning of task-relevant choices.

Introduction

The remarkable ability to rapidly and flexibly link sensory stimuli with appropriate motor responses underlies essential aspects of intelligent behavior and is thought to be orchestrated by circuits that include frontal cortex (Miller and Cohen 2001; Buch et al. 2006; Koechlin and Summerfield 2007; Botvinick 2008). A simple example linking sensory stimuli with motor outputs involves learning to interpret traffic lights: red means stop, and green means go. Flexible learning maps between sensory inputs and motor responses has been studied using Arbitrary Stimulus-Response Association tasks, wherein subjects learn to associate visually presented stimuli with arbitrary motor responses. The relationship between stimuli and motor response is learned by trial and error through reinforcement. This type of learning ability also underlies the association of sounds (motor action) with letters (visual stimuli) during learning how to read, the hierarchical learning of conditional rules (e.g., if in the UK drive on the left, if in the US drive on the right), and high-level cognitive rules including numerical and mathematical abilities (Murray et al. 2000; Badre et al. 2010; Hare et al. 2011).

Arbitrary stimulus-response associations allow subjects to learn flexible, abstract relationships between any stimulus and any type of response. In monkeys, different variations of such arbitrary mapping tasks involve activation of neurons in the prefrontal cortex (PFC) (Asaad et al. 1998; Johnston et al. 2007; Badre et al. 2010; Durstewitz et al. 2010), dorsal premotor cortex (Buch et al. 2006), supplementary and frontal eye fields (Chen and Wise 1995), caudate (Pasupathy and Miller 2005), putamen (Buch et al. 2006), globus pallidus (Inase et al. 2001; Sheth et al. 2011), and the hippocampus (Wirth et al. 2003; Mattfeld and Stark 2015). In humans, functional neuroimaging studies have reported frontal cortex and hippocampus activation in this type of task (Boettiger and D’Esposito 2005; Hare et al. 2011; Mattfeld and Stark 2015), and more anterior frontal cortex structures in tasks involving abstract rules (Badre et al. 2010; Badre and Frank 2012).

While several studies have separately documented the involvement of multiple brain areas in adaptive learning, little is known about the relationships between those circuits and how different brain circuits interact with each other to instantiate learning. Here we set out to investigate the dynamic interactions that give rise to learning new maps between visual stimuli and motor responses in humans. Visual information reaches dorsolateral PFC from the inferior temporal cortex (ITC) (Ungerleider et al. 1989; Logothetis and Sheinberg 1996). Several investigators have documented visual shape selectivity along ITC in monkeys (Logothetis and Sheinberg 1996; Connor et al. 2007) and humans (Liu et al. 2009). Dorsolateral PFC is reciprocally connected with ventral premotor cortex (Wang et al. 2002; Takada et al. 2004), which in turn is interlinked with dorsal premotor cortex and primary motor cortex (Dum and Strick 2005). There is extensive literature describing the selectivity for output movements in different parts of motor cortex in monkeys (Picard and Strick 1996) and humans (Kubanek et al. 2009; Fifer et al. 2013). We asked whether visuomotor associations would be reflected in dynamic interactions between visually responsive areas and frontal cortex and between frontal cortex and motor cortex. We conjectured that those interactions would be contingent upon learning adequate visual to motor mappings.

To evaluate these questions, we simultaneously recorded invasive intracranial field potentials from multiple locations along visual, frontal and motor areas in patients implanted with subdural electrodes for clinical reasons. We found electrodes that responded to visual stimuli along ventral visual cortex and other electrodes that responded to motor outputs along motor cortex. We also observed electrodes whose responses directly reflected the visual-motor mappings, but only after subjects learnt the appropriate behavior. These mapping selective electrodes were found in frontal cortex, and also in posterior parietal and anteroventral temporal cortex. After learning, these mapping selective electrodes showed enhanced interactions in the gamma frequency band with the visual electrodes and with the motor electrodes. These results provide an initial spatiotemporal model of neural circuits and their corresponding dynamic interactions that could underlie rapid and flexible learning.

Materials and Methods

Ethics Statement

All the experiments described here were approved by the Institutional Review Boards at each hospital and were carried out with each subject’s informed consent.

Data Availability

All the raw data and source code for the current study will be available at: http://klab.tch.harvard.edu/resources/Madhavanetal_rulelearning.html

Subjects

Subjects were 12 patients (9 females, 2 left handed, 10–52 years old) with pharmacologically intractable epilepsy. The subjects were admitted to Children’s Hospital Boston, Brigham and Women’s Hospital, Massachusetts General Hospital, or Johns Hopkins Hospital to localize their seizure foci for potential surgical resection.

Intracranial Field Potential recordings

The recording procedures were as described previously (Liu et al. 2009). Briefly, subjects were implanted with intracranial electrodes (Ad-Tech, Racine, WI, USA; 2.3 mm diameter, 1 cm separation between electrode centers, impedance <1 kOhm) to localize the seizure foci. All recordings were conducted during seizure-free epochs. The total number of recording sites per subject ranged from 56 to 186 (103.3 ± 34.6, mean ± SD). A notch filter was applied at 60 Hz and harmonics, and the mean across all electrodes at each time point was subtracted from each electrode’s response (common average reference). Throughout the text we refer to the recorded signal as “intracranial field potential” (IFP). Electrode localization was performed by aligning each subject’s MRI into Talairach space, coregistering CT and MR images, and assigning each electrode to one of 75 different regions in Freesurfer software based on the 2009 atlas (Destrieux et al. 2010).

Stimulus Presentation and Task

A scheme of the task is shown in Figure 1A. Subjects performed an arbitrary stimulus-response mapping task (Asaad et al. 1998; Buch et al. 2006). A stimulus was presented for 0.5 s (Stimulus), followed by a delay period consisting of a grey screen with a fixation cross for 1 s (Delay 1). After this delay, two targets appeared on the screen instructing the subject to make a response. The response consisted of either a left or a right button press on a gamepad (Logitech Cordless RumblePad 2). The subject’s response was followed by a post-response delay of 0.5 s (Delay 2). After this delay, feedback was presented to the subject in the form of a “smiley” face on correct trials, and a “devil” face on incorrect trials. Subjects were presented with a diverse set of visual stimuli, which were mapped to each response according to a random but fixed rule. Subjects learned to map each visual stimulus with the correct motor response by trial and error: stimuli were arbitrarily divided into Group 1 (example stimuli in blue boxes in Fig. 2A) or Group 2 (example stimuli in red boxes in Fig. 2A). There was no visual similarity within the Group 1 or Group 2 stimuli. The number of stimuli was the same in both groups. The presentation order was randomized.

Figure 1.

Task description and behavioral performance. (A) Each trial started with the presentation of a fixation cross for 1000 ms, followed by a visual stimulus presented for 500 ms. The stimulus was followed by a 1000 ms delay period (Delay 1). Next, a display screen with two targets appeared and served as a go signal for the subjects to press one of two buttons on a gamepad. After a 500 ms post-response delay (Delay 2), feedback was presented for 500 ms. Subjects learned to map images to left or right button presses. The number of mappings (image and response combinations) in each block was progressively increased, making the task more difficult with time (Methods). (B) Fraction of mappings learnt by each subject in each block. The color of each bar denotes the total number of mappings per block. The numbers above each bar show the average number of trials per mapping in each block. For subjects 4–12, the images in the previous block were removed and new images were introduced in each block whereas for subjects 1–3 each new block included and expanded on the images in the previous block. The fraction of mappings learnt decreased as the total number of mappings increased (P < 0.01, permutation test). Only blocks with more than 6 trials/mapping were included in the analyses. Note that the y-axis does not show the overall performance in each block but rather the fraction of mappings learnt.

Figure 2.

Mapping selective electrodes differentiated between the two groups of stimuli. A. Example images used for Group 1 and Group 2 stimuli. The blue and red borders are shown here for reference to the physiological responses below but were not included in the actual experiment. B–F. Example intracranial field potential (IFP) responses during correct trials from an electrode located in the intraparietal sulcus (subject 5, Talairach coordinates = [31.2, −72.2, 52.0], location shown in part F). B. Responses averaged across trials aligned to stimulus onset for each of 6 mappings; the colors correspond to the image borders in A. The gray rectangle denotes the stimulus period. Error bars denote ±SEM (shown only every 100 ms for clarity). During the delay period, this electrode showed a significantly greater response for mappings in Group 2 (red) compared with those in Group 1 (blue). The horizontal gray bar indicates time points with a statistically significant difference between the two groups (two-tailed t-test, P < 0.01, Methods). C. Responses averaged across mappings within each group aligned to stimulus onset. The lack of visual responses during the stimulus period in B–C, combined with the similarity in responses to visually dissimilar images in B, and the long latency of responses argue against a purely sensory interpretation of these responses. D. There was no significant response when the signals were aligned to button press (two-tailed t-test, P > 0.05). E. Stimulus-aligned responses during error trials. The lack of differences around button press (D), combined with the lack of differences between groups in error trials (E) argues against a purely motor or motor preparation interpretation of these responses. F. Location (arrow) for the example electrode in B–E. G. Thirty-five electrodes showed significant differences in IFP responses between Group 1 and Group 2 mappings (Methods). Note that these electrodes were pre-selected based on the differences between groups and are therefore expected to be off the diagonal (they could be either above or below the diagonal based on the selection criterion). They are shown here only to summarize the magnitude of the differences for all the mapping selective electrodes. The average root-mean squared difference (RMSD) between mappings was higher than the RMSD within mappings in 31/35 (88%) of mapping selective electrodes. Error bars denote ±SEM. The arrow indicates the example electrode in B–F. H. Histogram of Mapping tuning index (MTI), which measures whether differences between responses are larger within mappings (negative values) or between mappings (positive values). The distribution of the MTI for the 35 mapping selective electrodes (gray) was significantly different from the null distribution obtained by shuffling the mapping labels (black, P < 10^–14). The arrow indicates the example electrode in B–F.

The task was organized into blocks. Mapping rules and the number of stimuli were fixed within a block. The number of stimuli presented in each subsequent block was incremented after overall performance reached 80% in a block. The images were either incrementally added to an initial set of 4 images in each new block (Subjects 1–3), or completely replaced with new images in each new block (Subjects 4–12). There were 326.1 ± 135.4 (mean ± SD) trials per subject. Figure 1 reports the number of mappings learnt; the overall block performance criterion cannot be directly read from this figure (e.g., overall performance in a block can be >80% correct even when 75% of the mappings are learnt).

Data Analyses

All data were analyzed off-line using MATLAB (Mathworks Inc., Natick, MA).

Learning Criterion

We defined a learning criterion to establish when a given mapping was learnt by considering an ordered set of N trials in which the mapping was presented (where there could be other stimuli presented in between, which were ignored for the purpose of this definition). Within those N trials, the subject responded correctly in n consecutive trials (0 ≤ n ≤ N), where the term “consecutive” only refers to the specific mapping under consideration (and there could be other stimuli in between). For example, considering a block with two mappings, mapping 1 and mapping 2, using lower case letters to denote incorrect trials and upper case letters to denote correct trials, the following sequence contains N₁ = 10 instances of mapping 1 with a maximum of n₁ = 6 consecutive correct trials:

$$\begin{array}{l}{\mathrm{m}}_{\mathrm{1}}{-\mathrm{m}}_{\mathrm{2}}{-\mathrm{m}}_{\mathrm{1}}{-\mathrm{m}}_{\mathrm{1}}{-\mathrm{M}}_{\mathrm{2}}{-\mathrm{M}}_{\mathrm{1}}{-\mathrm{M}}_{\mathrm{1}}{-\mathrm{m}}_{\mathrm{2}}{-\mathrm{M}}_{\mathrm{2}}{-\mathrm{M}}_{\mathrm{1}}{-\mathrm{M}}_{\mathrm{1}}\\ {-\mathrm{M}}_{\mathrm{2}}{-\mathrm{M}}_{\mathrm{1}}{-\mathrm{M}}_{\mathrm{2}}{-\mathrm{M}}_{\mathrm{1}}{-\mathrm{m}}_{\mathrm{2}}{-\mathrm{m}}_{\mathrm{2}}{-\mathrm{m}}_{\mathrm{1}}{-\mathrm{M}}_{\mathrm{2}}{-\mathrm{M}}_{\mathrm{2}}\end{array}$$

(and N₂ = 10 instances of mapping 2 with a maximum of n₂ = 3 consecutive correct trials). A stimulus-response mapping was defined as learnt if there was a probability P < 0.05 of obtaining n consecutive correct trials out of N by randomly pressing the response buttons. For example, if there were N = 10 total trials, the subject had to perform 6 trials in a row correctly, whereas if there were N = 50 total trials then 9 consecutive correct trials were required. In the example sequence above, mapping 1 reached learning criterion where mapping 2 did not. Figure 1B shows the fraction of stimulus-response mappings learnt for each subject and block. Note that this is different from the criterion used during the experiment to move from one block to the next (80% overall performance in a block).

Visual Responsiveness

An electrode was labeled visually responsive if the amplitude of the IFP |max(IFP)-min(IFP)| in the [100;250] ms window after stimulus onset was significantly different from the baseline period ([−100;50] ms with respect to stimulus onset; P < 0.01, Wilcoxon rank-sum test, minimum difference in amplitudes of 10 μV (Bansal et al. 2012)). An example visually responsive electrode is shown in Figure S2. As demonstrated in other publications, some electrodes showed differential responses between the stimuli (visual selectivity) but those visually selective electrodes still showed a visually evoked response above baseline for all the stimuli (see examples and discussion of visual selectivity in previous work (Liu et al. 2009)). Therefore, the set of visually responsive electrodes includes the set of visually selective electrodes. As discussed in the text, none of the visually responsive electrodes showed responses that differentiated the two groups of stimuli such as the ones illustrated in Figure 2 (the latency at which differences between groups arose for mapping selective electrodes was well past the narrow window used to define visual responsiveness).

Motor Responsiveness

An electrode was labeled motor responsive if the IFP responses in the [−200; 200] ms window centered on the button press were significantly different from the baseline period ([−100;50] ms with respect to stimulus onset, P < 0.01, Wilcoxon rank-sum test, minimum difference in amplitudes of 10 μV (Bansal et al. 2012)). An example motor responsive electrode is shown in Figure S4. Some electrodes showed differential responses between the two motor outputs (motor selectivity, n = 30 electrodes) but those motor selective electrodes still showed a response above baseline for all button presses (right/left). Therefore, the set of motor responsive electrodes includes the set of motor selective electrodes.

Mapping Selectivity

We separated the visually responsive and motor responsive electrodes (defined above) to distinguish neural responses related to purely sensory or purely motor aspects of the task. For the remaining electrodes, we evaluated whether there was a differential response to stimuli in Group 1 (G1) versus Group 2 (G2). Unless otherwise indicated, we aligned the responses during correct trials to visual stimulus onset and focused on the window from 0 to 1500 ms (Stimulus and Delay 1 periods). An electrode was considered to be mapping selective if it satisfied the following two conditions:

1. The IFP responses to Group 1 stimuli were significantly different from those to Group 2 stimuli (two-tailed t-test, P < 0.01) for at least 100 consecutive milliseconds.

2. The difference in mean IFP response between Group 1 and Group 2 stimuli was at least 15.97 μV.

The 100 ms window and the voltage threshold (15.97 μV) were chosen based on the corresponding distributions of shuffled data (wherein the group labels were randomly chosen in each trial) to ensure a global false discovery rate (FDR) below 1%.

Permutation Tests

When comparing two distributions X = {x₁,x₂,x₃,…,x_n} and Y = {y₁,y₂,y₃,…,y_m} throughout the text, we used a non-parametric permutation test. For example, in Figure 3, X = RMSD_between before learning and Y = RMSD_between after learning. We compute the difference between the means of the two distributions: d_actual = <X> − <Y>. According to the null distribution, we expect that this difference should be centered around 0. For each iterations i = 1,…,10000, we randomly permute the labels, thus creating two new distributions _iX* and _iY* and compute the null difference _id*. We compute the P value by comparing d_actual to the distribution of_id*: P = #{i | _id* > d_actual}/10000.

Figure 3.

Mapping selectivity increased with learning. Responses during correct trials from the electrode in Figure 2 plotted separately before (A) and after (B) reaching learning criterion (Methods, figure format as in Fig. 2C). There were 30 trials before and 12 trials after learning. Here and in part C, the number of trials before and after learning was balanced by random subsampling. C. Root mean squared difference between groups after (y-axis) vs. before (x-axis) learning for all mapping selective electrodes. Error bars denote SEM. The slope of the regression line was significantly different from 1 (P = 9.5 × 10⁻⁸). Note that n = 31 here, and not 35, because in one subject (4 electrodes) there were not enough trials before learning. The arrow indicates the example electrode in parts A–B.

Mapping Tuning Index

To summarize and visualize the responses from multiple electrodes in Figure 2, we defined a mapping tuning index. This index was only used for visualization purposes in Figure 2G–H, but all of the subsequent quantitative analyses, discrimination of mapping selective electrodes, and conclusions depend on the definitions of mapping selectivity in the previous section and not on the metrics defined in this section. We refer to the average IFP responses within each group as ${\overline{r}}_1(t)=\frac{1}{\#G1}\sum _{i\in G1}^{\#G1}{r}_i(t)$ and ${\overline{r}}_2(t)=\frac{1}{\#G2}\sum _{i\in G2}^{\#G2}{r}_i(t)$ where r_i(t) denotes the IFP response at time t in trial i and #G1 and #G2 indicate the number of trials in each group. To evaluate the change in the IFP response between groups and compare it to the variability in the IFP responses within groups, we defined the root-mean squared difference (RMSD) between groups as (Fig. 2G, y-axis):

$$\mathrm{R}\mathrm{M}\mathrm{S}{\mathrm{D}}_{\mathrm{b}\mathrm{e}\mathrm{t}\mathrm{w}\mathrm{e}\mathrm{e}\mathrm{n}}=\sqrt{\frac{\sum _{\mathrm{t}\ge 0}^{\mathrm{t}<1500\mathrm{m}\mathrm{s}}{({\overline{\mathit{r}}}_1(\mathrm{t})-{\overline{\mathit{r}}}_2(\mathrm{t}))}^2}{{\mathrm{t}}_{\mathrm{s}}}}$$

where the denominator normalizes by t_s, the number of points sampled in the (0,1500) ms interval. To compute the variability within each group, for each electrode we considered two non-overlapping equal sized random partitions of the total of G1 trials. Let $r_{G1}^{\prime }\left(t\right)$ ($r_{G1}^{″}\left(t\right)$) denote the average response to the first (second) partition. A similar random split was considered for the G2 trials. The RMSD within groups was defined as (Fig. 2G, x-axis):

$$\begin{array}{l}\mathrm{R}\mathrm{M}\mathrm{S}{\mathrm{D}}_{\mathrm{w}\mathrm{i}\mathit{t}\mathrm{h}\mathrm{i}n}=\frac{1}{2}(\sqrt{\frac{\sum _{\mathit{t}\ge 0}^{\mathit{t}<1500\mathrm{m}\mathrm{s}}{(r_{G1}^{\text{'}}(\mathit{t})-r_{G1}^{\text{'}\text{'}}(\mathit{t}))}^2}{{\mathit{t}}_{\mathrm{s}}}}\\ +\sqrt{\frac{\sum _{\mathit{t}\ge 0}^{\mathit{t}<1500\mathrm{m}\mathrm{s}}{(r_{G2}^{\text{'}}(\mathit{t})-r_{G2}^{\text{'}\text{'}}(\mathit{t}))}^2}{{\mathit{t}}_{\mathrm{s}}}})\end{array}$$

To summarize the response differences between the two groups, we defined a Mapping Tuning Index (MTI):

$$\mathrm{M}\mathrm{T}\mathrm{I}=\frac{\mathrm{R}\mathrm{M}\mathrm{S}{\mathrm{D}}_{\mathrm{b}\mathrm{e}\mathrm{t}\mathrm{w}\mathrm{e}\mathrm{e}\mathrm{n}}-\mathrm{R}\mathrm{M}\mathrm{S}{\mathrm{D}}_{\mathrm{w}\mathrm{i}\mathrm{t}\mathrm{h}\mathrm{i}\mathrm{n}}}{\mathrm{R}\mathrm{M}\mathrm{S}{\mathrm{D}}_{\mathrm{b}\mathrm{e}\mathrm{t}\mathrm{w}\mathrm{e}\mathrm{e}\mathrm{n}}+\mathrm{R}\mathrm{M}\mathrm{S}{\mathrm{D}}_{\mathrm{w}\mathrm{i}\mathrm{t}\mathrm{h}\mathrm{i}\mathrm{n}}}$$

The distribution of MTI for mapping selective electrodes was compared with the null distribution obtained by 100 shuffles of the group labels (Fig. 2H). When shuffling the group labels, the stimulus identity was retained, and therefore individual trials corresponding to the same stimulus could not end up in both groups at the same time (which would have occurred if the group for each trial was shuffled independently). In contrast, in the definition of RMSD_within, different trials corresponding to the same stimulus could end up in the two different partitions. The conclusions were similar if the two partitions were constrained to have trials from different stimuli because there was no sharp selectivity between different stimuli within the same group (e.g., Fig. 2B, see also (Liu et al. 2009)). We ensured by random subsampling that any differences between RMSD_between and RMSD_within could not be ascribed to different numbers of trials used in the calculations.

Latency Analysis

The latency of mapping selectivity was defined as the first time-point when the IFP response was significantly different between Group 1 and Group 2 trials for at least 100 consecutive ms (P < 0.01, two-tailed t-test).

Before vs. After Learning Comparisons

To assess whether the physiological responses changed as the subject learned the mapping between stimuli and responses (Figures 3, 7, S5, S6), we compared the IFP responses from trials before performance crossed 65% (before learning trials), with those from trials after the learning criterion was reached (after learning trials). The 65% threshold defining the before-learning trials was chosen to ensure that there was a large enough buffer before the learning criterion was reached. All the before learning trials were combined and all the after learning trials were combined in the corresponding curves in Figures 3, 7, S5, S6.

Figure 7.

Coherence in the gamma band during the delay period increased with learning. A−B. Average coherogram during the delay period for an example pair of electrodes before (A) and after (B) learning criterion was reached. One of the electrodes was mapping-selective (right intraparietal sulcus) and the other one was visually-responsive (right fusiform gyrus). The format and conventions are the same as those in Figure 6B. Increased coherence was observed in frequencies >30 Hz in the delay period after learning compared to before learning (P = 0.002, permutation test). n = 86 trials. C. Cumulative distribution of the learning index (LI) showing the change in the coherence after learning versus before learning in the [30,100] Hz frequency band for all pairs of task-selective electrodes (blue, n = 1754) or all other electrode pairs (black, n = 27 560). The distribution of LI_30–100^Delay in task-selective electrode pairs was significantly different from the shuffled distribution (LI’) obtained by randomly permuting the label of before/after learning trials (P < 0.001, permutation test, Methods). D. Cumulative distribution of LI_30–100^Stim (dotted), LI_30–100^Delay (solid) and LI_30–100^Response (dashed). The distribution of LI_30–100^Stim and LI_30–100^Response for task-selective electrode pairs were not significantly different from the shuffled null distributions (P = 0.4). E. Cumulative distribution of LI in low frequencies, LI_0–30^Delay (cyan), versus high frequencies, PC_30–100^Delay (blue). The distribution of LI_0–30^Delay for task-selective electrode pairs was not significantly different from the null distribution (P = 0.3). F. Cumulative distribution of LI_30–100^Delay for pairs of electrodes between mapping-selective and visually-responsive electrodes (Map-Vis, red), mapping-selective and motor-responsive electrodes (Map-Mot, green) and visually-responsive and motor-responsive electrodes (Vis-Mot, blue). The distribution of LI_30–100^Delay between Map-Vis and Map-Mot electrode pairs were significantly different from the shuffled null distribution (P < 10⁻⁵, permutation test). G. Cumulative distribution of LI_30–100^Delay for pairs within mapping-selective electrodes (Map-Map, magenta), visually-responsive electrodes (Vis-Vis, black) and motor-responsive electrodes (Mot-Mot, cyan). The distribution of LI_30–100^Delay between Map-Map and Vis-Vis electrode pairs were significantly different from the shuffled null distribution (P < 10⁻⁴, permutation test).

Coherence Analysis

For each electrode, we calculated local bipolar differences between IFPs by subtracting the immediately neighboring electrode to remove any potential synchronization due to the common average reference. Additionally, data for each electrode and time period were normalized by subtracting the mean and dividing by the standard deviation across trials. The resulting bipolar normalized data were used for all analyses of coherence.

In Figure 5C, we show an example of the coherence between two electrodes as a function of time. In Figures 6 and 7, each trial was split into 4, non-overlapping task periods of 0.5 s duration: Baseline, [−0.5 0] s before the onset of the visual stimulus; Stimulus, [0 0.5] s from the onset of the visual stimulus; Delay, [0.8 1.3] s from the onset of the visual stimulus; and Response, [−0.25 0.25] s around the time of key press.

Figure 5.

Example of changes in pairwise coherence interactions. A. Example IFP response averaged across the two groups from a mapping selective electrode (right inferior parietal angular gyrus, subject 4, Talairach coordinates = [38.8 −73.3 52.2]). The horizontal gray bar indicates time points with a statistically significant difference between the two groups of stimuli. Error bars denote ±SEM. n = 208 trials. B. Example IFP response averaged across the two groups from a visually responsive electrode (right lingual gyrus, subject 4, Talairach coordinates: [0.7−83.3 9.5]). C. Average coherence between the pair of electrodes in (A–B), showing the percentage change with respect to baseline in different frequency bands as a function of time from stimulus onset (Methods, see color scale on right). The degree of coherence in the gamma frequency band increased approximately 800 ms after stimulus onset.

Figure 6.

Coherence in the gamma band increased during the delay period in task-selective electrode pairs. A–C. Average coherogram for an example pair of electrodes during Stimulus (A), Delay (B) and Response (C) periods. One of the electrodes was mapping selective (right intraparietal sulcus) and the other one was visually responsive (right occipito-temporal gyrus). The dashed black line in C indicates the time of key press. Increased coherence with respect to baseline was observed in frequencies >30 Hz during the delay period (P = 0.005, permutation test). D. Cumulative distribution (CDF) of percentage change (PC) in coherence with respect to baseline in the [30,100] Hz frequency band during the delay period (PC_30–100^Delay) for all pairs of task-selective electrodes (blue curve, n = 1754 pairs including all mapping-selective, motor-responsive and visually-responsive electrodes) or all other electrode pairs (black trace, n = 27 560 pairs). The distribution of PC_30–100^Delay for selective electrode pairs was significantly different from the shuffled distribution (PC_30–100^Delay’) obtained by randomly permuting the label of baseline/delay period (P < 0.001, permutation test, Methods). There is no curve fitting here, the plots show the actual cumulative distributions. Here and in subsequent plots, the x-axis was cut at 15% change in coherence for visualization purposes; the CDF for all curves reaches 1 upon extending the x-axis. The arrow shows the example electrode pair from parts A–C. Note that the “instantaneous” change in coherence can reach higher and lower values than the time-averaged values reported in D. E. Cumulative distribution of PC for Stimulus (dotted), Delay (solid) and Response (dashed) periods. The Delay period curve is the same curve shown in D. The distribution of PC_30–100^Stim (dotted blue line) and PC_30–100^Response (dashed blue line) were not significantly different from the shuffled null distribution (P = 0.3). F. Cumulative distribution of PC in the gamma frequency band (blue) versus low frequency bands (cyan) during the delay period (gamma frequency band curve is the same as in D). The distribution of PC_0–30^Delay (low frequency bands) was not significantly different from the shuffled null distribution (P = 0.7). G. Cumulative distribution of PC in the gamma frequency band during the delay period separated based on the response characteristics of the constituent electrodes: mapping and visual pairs (Map-Vis, red, n = 578), mapping and motor pairs (Map-Mot, green, n = 34), visual and motor pairs (Vis-Mot, blue, n = 54). The distributions of PC_30–100^Delay between Map-Vis and Map-Mot electrode pairs were significantly different from the shuffled null distribution (P < 0.0001 and P < 0.01 respectively, permutation test). H. Cumulative distribution of PC_30–100^Delay for pairs within mapping-selective electrodes (Map-Map, magenta, n = 92), visually-responsive electrodes (Vis-Vis, black, n = 993) and motor-responsive electrodes (Mot-Mot, cyan, n = 3). The distribution of PC_30–100^Delay between Vis-Vis electrode pairs was significantly different from the shuffled null distribution (P < 10⁻⁵, permutation test).

For each time period, the coherence in the responses of two electrodes x and y at frequency f was calculated as:

$${\mathit{Cxy}}_f=\mathrm{}\frac{\left|S_{\mathit{xy}}\mathrm{}(f)\right|}{\sqrt{S_x\mathrm{}(f)\mathrm{}{S}_y(f)}}$$

where S_xy(f) is the cross spectral density between the IFP time-series, while $S_x$ and $S_y$ denote the spectra of x and y respectively. Spectral power and coherence were calculated using a fast Fourier transform with multitapering using 5 tapers (Mitra and Pesaran 1999). Given the epoch length of 0.5 s, this resulted in spectral smoothing of ±6 Hz. To avoid artifact coherence values caused by passive current spread, electrode pairs within a distance of ≤1 cm or pairs in the same parcelation area were excluded from these analyses. To statistically compare the degree of coherence across different analyses or conditions, we repeated the calculations after randomly shuffling the corresponding labels (e.g., selective versus non-selective electrode pairs in Fig. 6D), correcting for multiple comparisons with a false discovery rate FDR < 0.01.

We focused the analyses on two frequency bands: (i) low frequency band including theta, alpha and low-gamma ranges ([0–30] Hz) and (ii) gamma band [30–100] Hz after removing the frequencies from [50–70] Hz to rule out potential artifacts due to 60 Hz line noise. For each trial, the coherence in a given frequency band from [f1–f2] Hz and period E (Baseline, Stimulus, Delay, Response) is denoted as ${\mathit{Cxy}}_{[f1,f2]}^E$. For each electrode pair, the mean coherence across trials was calculated: $\overline{C}x{y}_{[f_1,f_2]}^{\mathrm{p}\mathrm{e}\mathrm{r}\mathrm{i}\mathrm{o}\mathrm{d}}=1/{\mathit{N}}_{\mathrm{t}\mathrm{r}\mathrm{i}a\mathrm{l}\mathrm{s}}\sum _{\mathrm{i}=1}^{\mathrm{i}={\mathit{N}}_{\mathrm{t}\mathrm{r}\mathrm{i}a\mathrm{l}\mathrm{s}}}{}_{\mathrm{i}}Cx{y}_{[f_1,f_2]}^{\mathrm{p}\mathrm{e}\mathrm{r}\mathrm{i}\mathrm{o}\mathrm{d}}$. The percentage change (PC) with respect to the baseline period was computed: $P{C}_{[f1,f2]}^{\mathrm{p}\mathrm{e}\mathrm{r}\mathrm{i}\mathrm{o}\mathrm{d}}xy=100*\left(\frac{\overline{C}x{y}^{\mathrm{p}\mathrm{e}\mathrm{r}\mathrm{i}\mathrm{o}\mathrm{d}}-\overline{C}x{y}^{\mathrm{b}\mathrm{a}\mathrm{s}\mathrm{e}\mathrm{l}\mathrm{i}\mathrm{n}\mathrm{e}}}{\overline{C}x{y}^{\mathrm{b}\mathrm{a}\mathrm{s}\mathrm{e}\mathrm{l}\mathrm{i}\mathrm{n}\mathrm{e}}}\right)$ where [f₁,f₂] were dropped inside the parenthesis for clarity. This is the quantity shown in Figures 5C, 6A–C and 7A–B.

The number of possible electrode pairs in Figures 6 and 7 were very different between conditions (mapping-visual pairs, mapping-motor pairs, and visual-motor pairs, Table S3, Figs 6 and 7). The cumulative distributions reported in those figures normalize by the total number of pairs. Additionally, we repeated the analyses by randomly subsampling to equalize the number of possible pairs and the conclusions were qualitatively similar.

To assess whether there were changes in coherence with learning, we separately computed the coherence in those trials before $\left({}_{before}\overline{C}x{y}_{[f1,f2]}^{period}\right)$ versus after $\left({}_{\mathrm{a}\mathrm{f}\mathrm{t}\mathrm{e}\mathrm{r}}\overline{C}x{y}_{[\mathrm{f}1,\mathrm{f}2]}^{\mathrm{p}\mathrm{e}\mathrm{r}\mathrm{i}\mathrm{o}\mathrm{d}}\right)$ learning as defined in the previous section. We defined the learning index as $LIx{y}^{\mathrm{p}\mathrm{e}\mathrm{r}\mathrm{i}\mathrm{o}\mathrm{d}}=\frac{{}_{\mathrm{a}\mathrm{f}\mathrm{t}\mathrm{e}\mathrm{r}}\overline{C}x{y}^{\mathrm{p}\mathrm{e}\mathrm{r}\mathrm{i}\mathrm{o}\mathrm{d}}-{}_{\mathrm{b}\mathrm{e}\mathrm{f}\mathrm{o}\mathrm{r}\mathrm{e}}\overline{C}x{y}^{\mathrm{p}\mathrm{e}\mathrm{r}\mathrm{i}\mathrm{o}\mathrm{d}}}{{}_{\mathrm{a}\mathrm{f}\mathrm{t}\mathrm{e}\mathrm{r}}\overline{C}x{y}^{\mathrm{p}\mathrm{e}\mathrm{r}\mathrm{i}\mathrm{o}\mathrm{d}}+{}_{\mathrm{b}\mathrm{e}\mathrm{f}\mathrm{o}\mathrm{r}\mathrm{e}}\overline{C}x{y}^{\mathrm{p}\mathrm{e}\mathrm{r}\mathrm{i}\mathrm{o}\mathrm{d}}}$. This is the quantity shown in Figure 7C–G. For each electrode pair, we performed the following permutation test. We randomized the labels of before / after trials and recomputed the learning index. This process was repeated 500 times. The shuffled learning index is denoted LI’. The distribution of the actual LI was compared with the null distribution of LI’ to determine statistical significance using the Wilcoxon’s rank sum test.

Results

Subjects Learnt to Associate Visual Stimuli with Behavioral Responses

Twelve subjects learned to map an image (visual stimulus) onto a left or right button press (motor response) (Fig. 1A). Visual stimuli were randomly allocated to one of the two possible groups, corresponding to two possible behavioral responses. This sensory to motor mapping was learnt through trial and error based on feedback (Asaad et al. 1998; Buch et al. 2006). We refer to a mapping as a stimulus/response pair. On average, across all blocks, subjects required 10.4 ± 7 (mean ± SD) trials to learn a given mapping (Methods).

When subjects correctly learned the mapping in a given block, they moved on to a new block that had a larger number of mappings (Methods). Task difficulty increased with the number of different stimuli per block: as the number of simultaneous visual-motor mappings within a block increased, the fraction of mappings learnt decreased (Fig. 1B, P < 0.01, permutation test on the slope of fraction of mappings learnt versus number of mappings per block); the number of trials required to learn each mapping increased (Fig. S1A, P < 0.01, permutation test on the slope of number of trials to reach criterion versus number of mappings per block), and the overall performance decreased (Fig. S1B, P < 0.01, permutation test on the slope of performance versus number of mappings per block). A total of 108 stimulus-response mappings reached learning criterion (see Methods for definition of Learning criterion).

Visual and Motor Responsive Electrodes

Intracranial field potentials (IFP) were recorded from 1240 electrodes implanted in 12 epilepsy subjects. Table S1 reports the electrode locations (rendered on a common brain in Fig. S8A–B). We evaluated the broadband responses (0.1–500 Hz) as well as the responses in the following frequency bands: 4–8 Hz, 8–12 Hz, 12–35 Hz, 35–50 Hz, and 70–100 Hz (Table S2). As reported in previous studies (e.g., Liu et al. 2009), presentation of a visual stimulus elicited a strong response in areas within visual cortex. Figure S2 shows an example electrode located on the left inferior temporal gyrus demonstrating a significant change in the broadband IFP response to the visual stimulus onset compared with pre-stimulus baseline (Fig. S2A, P < 0.01, Wilcoxon rank-sum test, see Visual responsiveness in Methods). The visual stimuli were randomly allocated to each group and there was no feature similarity among stimuli within a group (see example stimuli in Fig. 2A). The responses from this visually responsive electrode did not distinguish between the two groups of stimuli (Fig. S2A–B, P > 0.01, Wilcoxon rank-sum test). This lack of visual selectivity stands in contrast to other studies documenting differential responses to specific groups of similar stimuli (e.g., Liu et al. 2009). The lack of selectivity in Figure S2 can be expected as a consequence of the visual dissimilarity among stimuli within a group. Additionally, this electrode’s responses did not change around the button press nor did they discriminate between the two behavioral responses (Fig. S2C, P > 0.01, Wilcoxon rank-sum test). A total of 87 electrodes showed significant visual responsiveness in the broadband signals during the stimulus period; these electrodes were mostly distributed over the ventral visual cortex (Methods, Fig. S8C). Visual responsiveness was also evident when considering other frequency bands (Table S2). In particular, several investigators have documented visual responsiveness in the IFP gamma frequency band (e.g., Privman et al. 2011; Bansal et al. 2012) and the current results are consistent with those previous observations.

As reported in previous studies (e.g., Miller et al. 2007; Wang et al. 2009), another set of electrodes, disjoint from the visually responsive electrodes, showed a differential response triggered by the button press. Figure S3 shows an example electrode on the left postcentral gyrus demonstrating an enhanced response around the time of key press (Fig. S3, P < 0.01, Wilcoxon rank-sum test, see Motor responsiveness in Methods). In contrast with the visually responsive electrode from Figure S2, this electrode did not show any change in activity when the stimulus was presented (Fig. S3C, P > 0.01, Wilcoxon rank-sum test). A total of 67 electrodes showed motor responsiveness in the broadband signals during the 400 ms period centered on the key press; these electrodes were mostly located around motor cortex and the postcentral gyrus (Methods, Fig. S8D). Motor responsiveness was also evident when considering other frequency bands (Table S2). Several investigators have documented motor responsiveness in both the beta (Miller et al. 2007) and gamma bands (Miller et al. 2007; Wang et al. 2009; Gunduz et al. 2016) and the current results are consistent with those previous observations.

Mapping Selective Electrodes Distinguished Between the Two Behaviorally Relevant Groups of Stimuli

The transformation of the visual signals (Fig. S2) into the motor signals (Fig. S3) requires the type of visual-motor mapping behaviorally described in Figure 1. We investigated these visual-motor associations by evaluating whether the physiological signals revealed the behaviorally relevant and visually arbitrary grouping of stimuli (referred to hereafter as mapping selectivity). Figure 2 illustrates the responses of an example electrode located in the intraparietal sulcus (Fig. 2F) that showed a differential signal in response to stimuli from Group 1 versus Group 2 (Figs. 2B–C). The intracranial field potential signal during the delay period in correct trials was larger for stimuli from Group 2 compared with Group 1 (Fig. 2C, P < 0.01, Wilcoxon rank-sum test). This differential signal between the two groups was consistent across the different individual stimuli within each group (Fig. 2B). Additional examples from other subjects are shown in Figure S4.

This mapping selective physiological signal is unlikely to constitute a purely visual response, given that: (i) the physiological signal was modulated by the group the stimulus belonged to even though stimuli within each group were highly heterogeneous in terms of visual shape properties (Fig. 2B); (ii) there was no physiological change during the stimulus period; and (iii) the physiological change commenced several hundred milliseconds after visually triggered responses (c.f., Figs. 2B–C versus the visually responsive electrode in Figure S2A–B; see also latencies for visual signals in Liu et al. (2009)).

This mapping selective physiological difference cannot be ascribed to a purely motor response either, given that: (i) the signal difference was manifested during the delay period well after the disappearance of the stimulus and well before the motor response (Fig. 2B–C, in contrast to the motor responsive electrode in Fig. S3); (ii) there was no significant difference between the responses to the two groups when we aligned the physiological signals to the key press (Fig. 2D, P > 0.01, Wilcoxon rank-sum test), and (iii) this differential signal observed during correct trials in Figure 2B was absent in error trials (Fig. 2E, P > 0.01, Wilcoxon rank-sum test).

Thirty-five electrodes in 6 subjects showed mapping selectivity defined by a significant differential response in the IFP between Group 1 and Group 2 during correct trials as illustrated by the example electrode in Figure 2 (Methods). We were concerned that this number of mapping selective electrodes was a small fraction of the total number of electrodes (total = 1086 electrodes, after removing the 87 visually responsive and 67 motor responsive electrodes). However, this number of electrodes is over 3 times the number expected by chance, after controlling for multiple comparisons, given the false discovery rate (FDR) of 1% (Methods). To further evaluate whether the low number of electrodes could be a statistical selection artifact, we repeated the analyses after randomly shuffling the group labels in each trial. In this shuffling procedure, we compared random group allocation against the special stimulus-to-group mapping that subjects learnt and which was pre-assigned at the beginning of each experiment (Fig. 2A, Methods). This shuffling procedure showed that the probability of obtaining 35 mapping selective electrodes by chance was P < 0.01. We ascribe this low number of mapping selective electrodes primarily to the fact that the electrode locations are dictated by clinical criteria and, therefore, most electrodes do not show any relevant response during the task (as opposed to other techniques that only examine or report the electrodes in specific areas of interest). In the next two sections when we discuss the effect of learning and inter-areal interactions, we present further evidence that argues against the possibility that this set of mapping selective electrodes could be the result of a statistical selection artifact.

We compared the root mean square difference in the IFP response between groups (RMSD_between) to the variability of the IFP response within groups (RMSD_within, Fig. 2G). To summarize the differences in IFP responses between the two groups, we defined a Mapping Tuning index (MTI, Methods). If the difference in the IFP response between groups were comparable to the response variability within groups, MTI would be close to 0. MTI > 0 indicates that the difference in response between groups was higher than the variability within groups. The mean MTI for mapping selective electrodes was 0.17 ± 0.07 (Fig. 2H). Because these electrodes were selected based on the differences between groups, we expected them to show more distinct responses between groups compared to within groups; Figure 2G–H serve to summarize and quantify those differences and for comparison with the next section.

We performed the same analyses after filtering the data in different frequency bands including theta, alpha, beta, and gamma. Despite the presence of abundant visually responsive electrodes and motor responsive electrodes in other frequency bands, particularly in the gamma band, there was only a small number of mapping selective electrodes when restricting the responses of individual electrodes to different frequency bands, particularly to high frequency bands (Table S2). The contribution of different frequency bands to the mapping between visual and motor responses will be revisited when we examine the communication between brain areas.

Mapping Selectivity Emerged After Subjects Reached Learning Criterion

The signals described in Figure 2 included correct trials while subjects were learning the mappings as well as those trials after learning. Because the mapping between stimuli and groups was arbitrarily pre-defined in each experiment, and because this mapping was unknown to the subjects before the onset of the experiment, we would not expect to observe any grouping among the stimuli before the behavioral manifestation of learning the visual-to-motor associations. To test this prediction, we compared the IFP signals before versus after the visuomotor mappings were learned. For this analysis, we focused on those trials before subjects reached 65% correct performance versus those trials after the learning criterion was reached (Methods). Figure 3A–B follows the example electrode from Figure 2B–F. There was no significant difference between the physiological responses to Group 1 and Group 2 during correct trials before learning (Fig. 3A, P > 0.05, Methods). Mapping selectivity emerged after learning (Fig. 3B, P < 0.01). Of the 35 mapping selective electrodes, 31 had a sufficient number of trials before learning (at least 5 trials for each mapping). Of these 31 electrodes, 28 showed a differential response after learning but not before learning (P < 0.01). Using the same metric introduced in Figure 2G, we visualized the degree of mapping selectivity before versus after learning. As illustrated for the example electrode in Figure 3A–B, the difference between the physiological responses to Group 1 and Group 2 after learning was higher than the difference before learning for all the mapping selective electrodes (Fig. 3C, P < 0.01, permutation test, number of before-learning and after-learning trials balanced by random subsampling). It should be noted that the selection of electrodes that differentiate between the two groups, such as the example electrode in Figure 2, is independent of the analyses in Figure 3. Therefore, the consistent differences between the physiological responses before versus after learning reported in Figure 3 are not a trivial consequence of mapping selectivity. The effect of learning on each electrode was not correlated with the number of trials required to reach learning criterion (Fig. S7). Mapping selectivity arose in each electrode as a consequence of learning, irrespective of whether learning was fast or slow.

In contrast with the mapping selective electrodes, the visually responsive electrodes only showed weak changes with learning (Fig. S5, P = 0.03, permutation test) and the motor responsive electrodes showed no systematic changes with learning (Fig. S6, P > 0.05, permutation test). The differences between mapping selective electrodes and the visually responsive or motor responsive electrodes with respect to learning further dissociate rule learning signals from purely sensory or output signals. Furthermore, the specificity of learning effects rules out the possibility that changes accompanying learning merely reflect the passage of time, or increased attention over time or other non-specific explanations.

Mapping Selective Electrodes were Concentrated in three Anatomical Clusters

The distribution of all electrode sites across 12 subjects in this study is shown in Figure S8A–B and Table S1. The anatomical locations of the mapping selective electrodes are shown in Figure 4. The mapping selective electrodes clustered into 3 spatially distinct, non-overlapping regions in the frontal cortex, in the anteroventral temporal cortex and in the posterior parietal cortex (Fig. 4). In contrast, visually responsive electrodes were largely clustered along the ventral visual stream (Fig. S8C) as previously described (Liu et al. 2009), and motor responsive electrodes were mostly restricted to anterior parietal and motor cortex (Fig. S8D), as expected from previous studies (Wang et al. 2009). We calculated the latency at which mapping selectivity emerged in the IFP responses after the onset of the stimulus (Methods). We compared the latency to mapping selectivity across the 3 spatial clusters of electrode locations (Fig. S9). Mapping selectivity appeared earlier in frontal regions (labeled G1 in Fig. 4, mean ± SD = 0.56 ± 0.29 s, n = 8) compared with posterior parietal (labeled G2, n = 17, 0.81 ± 0.15 s) and temporal regions (labeled G3, n = 10, 0.68 ± 0.16 s). The differences in latencies between G1 and G3 or between G2 and G3 were not statistically significant (P = 0.8); whereas the latencies in frontal cortex (G1) were significantly lower than those in posterior parietal cortex (G2) (P = 0.006).

Figure 4.

Mapping selective electrodes were clustered in 3 regions. Lateral view of the brain showing anatomical locations of electrodes that showed mapping selectivity (red, n = 35). All electrodes were mapped onto one subject’s brain (subject 6). The electrode locations spatially clustered into 3 regions: G1, with electrodes over the frontal cortex; G2, the posterior parietal cortex and G3, the anteroventral temporal cortex. The location of all electrodes examined in this study is depicted in Figure S8A–B and Table S1.

Interactions Between Brain Areas Underlie Mapping Selectivity

Linking the visual signals (Fig. S2) to the behavioral output (Fig. S3) requires an interaction between the sensory inputs and motor commands. We conjectured that the mapping selective electrodes are ideally positioned to implement this link. We therefore sought to directly examine the interactions between the mapping selective electrodes, the inputs (as conveyed by visually responsive electrodes) and the outputs (as conveyed by motor responsive electrodes). To evaluate these putative interactions, we computed the degree of coherence between the signals derived from pairs of electrodes (Mitra and Pesaran 1999) (Methods). Figure 5 shows the responses aligned to stimulus onset of a mapping selective electrode (subject 4, right inferior parietal angular gyrus, Fig. 5A) and a visually responsive electrode (right lingual gyrus, Fig. 5B). We computed the degree of coherence between the signals from these two electrodes in different frequency bands (Fig. 5C) and followed the changes in coherence with respect to baseline over the course of the trial. There was an increase in coherence, particularly noticeable in high frequencies, spanning several hundred milliseconds and starting at about 800 ms after stimulus onset (30–100 Hz, 800–1300 ms, P = 0.02, permutation test).

To further investigate the dynamics underlying these putative interactions, we considered the subset of mapping selective electrodes and separately examined the degree of coherence in 3 periods of 500 ms duration: the stimulus period (0–500 ms after stimulus onset, Fig. 6A), the delay period (800–1300 ms after stimulus onset, Fig. 6B) and the response period (±250 ms around key press, Fig. 6C). The definition of the delay period from 800 to 1300 ms was based on ensuring that the window was well after the stimulus presentation (which ended at 500 ms), well before the motor response (which started at 2000 ms) and had the same duration as the stimulus and response windows. Figure 6A–C shows the change in coherence between a mapping selective electrode (right intraparietal sulcus, subject 5) and a visually responsive electrode (right occipito-temporal gyrus) during these 3 periods (different subject from the one in Fig. 5). There was an increase in coherence with respect to baseline (peak change of 24%) during the delay period; this increase in coherence was broadly concentrated in the high frequency band (30–100 Hz, 800–1300 ms, Fig. 6B, P < 0.005, permutation test). In contrast, there was no change in coherence during the stimulus period (Fig. 6A) or during the response period (Fig. 6C). Additional examples from other subjects are shown in Figure S10A–F. We extended this analysis to the population of electrode pairs for those subjects with mapping selective electrodes (Table S3), excluding adjacent electrodes or electrodes in the same anatomical parcel (Table S1) to avoid potential effects of volume conduction. The increase in coherence in the gamma frequency band demonstrated for the example electrodes in Figures 5 and 6A–C was consistent across the population of electrode pairs with task selectivity, including mapping, visual or motor electrodes (mean ± SD = 2.7 ± 4.8%, Fig. 6D, blue line, P < 0.001, permutation test) but was not observed in the population of electrodes that did not show any task selectivity (0.1 ± 2.6%, Fig. 6D, black line,). This increased coherence was restricted to the delay period and was absent in the stimulus and response periods (stimulus period: −1.4 ± 4.0%, response period: −1.3 ± 2.8%, Fig. 6E, P > 0.05, permutation test; compare also the time-averaged coherograms in Fig. S10G versus Fig. S10H). This increased coherence was concentrated in the [30–100] Hz frequency band and absent in the [0–30] Hz frequency band (1.8 ± 2.1%, Fig. 6F, P > 0.05, permutation test).

We separated all electrode pairs across functional clusters into 3 groups in Figure 6G: (i) a mapping selective electrode and a visually responsive electrode (red); (ii) a mapping selective electrode and a motor responsive electrode (green); and (iii) a visually responsive electrode and a motor responsive electrode (blue). There were significant differences in the changes in coherence among these 3 groups (P < 0.001, permutation test). The largest changes in coherence were manifested in the mapping-visual electrode pairs: the distribution of gamma band coherence changes with respect to baseline during the delay period between mapping and visual electrodes was significantly different from the shuffled null distribution (P < 10⁻⁴, permutation test). The gamma band coherence between mapping and motor responsive electrodes showed a weaker but still statistically significant increase (P < 0.01, permutation test). When restricting the electrodes to pairs within one of these functional clusters (mapping, visual, motor), the visual pairs showed significant changes in coherence (Fig. 6H, P < 10⁻⁵, permutation test).

The distribution of electrodes (and consequently electrode pairs) across subjects was highly non-uniform (Table S3). In particular, subject 6 had more mapping selective and visually responsive electrodes, and hence, a much larger number of possible mapping-visual electrode pairs, than any other subject. This is not atypical in this type of studies where the electrode locations are purely dictated by clinical criteria (e.g., Tang et al. 2016). We re-analyzed the coherence interactions separately in the other subjects (4 other subjects for mapping-visual pairs and 1 other subject for mapping-motor pairs). The conclusions were qualitatively similar in this subset of subjects. Furthermore, we showed examples of mapping selective electrodes from multiple subjects (Fig. 2, Fig. S4) as well as examples of interactions between mapping and visually responsive electrodes in multiple subjects (Figs 5, 6A–C, Fig. S10A–F).

Changes in Brain Interactions During Mapping were Manifested Only After Reaching Learning Criterion

We next asked whether the increase in coherence in the gamma frequency band during the delay period was dependent on whether subjects had learned the visuomotor contingencies or not. To address this question, we separated the coherence computations into those trials before versus after learning as we did previously for the single electrode analyses in Figure 3. These analyses were also restricted to correct trials. An example electrode pair composed of an electrode in right intraparietal sulcus (mapping selective) and another electrode in the right fusiform gyrus (visually responsive) is shown in Figure 7A–B. The percent change in coherence compared with baseline during the delay period in the [30–100] Hz frequency band was significantly larger after learning (Fig. 7B) compared to before learning (Fig. 7A, P = 0.002, permutation test). To quantify the changes in coherence before versus after learning, we computed a learning index defined as the difference between coherence after learning and before learning, normalized by their sum (Methods). This learning index takes a value of 0 if there are no differences before versus after learning. As the example electrode pair in Figure 7A–B illustrates, there was a significant shift in the learning index towards positive values across the population of task-selective electrodes (Fig. 7C, P < 10⁻³, permutation test). This shift was absent in the population of electrode pairs that did not show task selectivity (Fig. 7C, P > 0.05, permutation test). These learning dependent changes were manifested only during the delay period and not during the stimulus and response periods (Fig. 7D, P > 0.05, permutation test; see also the time-averaged changes in coherence as a function of frequency in Fig. S10G–H). Additionally, these learning dependent changes were apparent in the gamma frequency band but not in the lower frequency band (Fig. 7E, P > 0.05, permutation test). The learning index was particularly prominent for those electrode pairs consisting of a mapping selective electrode and a visually responsive electrode (Fig. 7F, P < 10⁻⁵, permutation test). The learning index was weaker but still statistically significant for electrode pairs that included a mapping selective electrode and a motor responsive electrode. (Fig. 7F, P < 0.01, permutation test). When restricting the electrodes to pairs within one of these functional clusters (mapping, visual, motor), both the visual pairs and the mapping pairs showed a significant learning index (Fig. 7G, P < 10⁻⁵ and P < 10⁻⁴, respectively, permutation test). In sum, the putative interactions demonstrated here by computing the coherence between electrode responses in different frequency bands were only evident after subjects correctly learnt the visuomotor associations.

Discussion

To successfully perform the task in Figure 1, subjects needed to visually identify the image and learn the appropriate motor response based on an associative rule learnt by trial-and-error. In accordance with previous studies, we observed electrodes that responded shortly after the visual presentation (Fig. S2) and other electrodes that responded around the motor response (Fig. S3). We identified a separate group of electrodes that were neither purely visual nor purely motor, but instead responded according to the visuomotor mapping rule during the delay period of the task (Fig. 2). These mapping selective signals were only apparent after learning (Fig. 3) and were concentrated in 3 clusters in the frontal, anteroventral inferior temporal and posterior parietal regions (Fig. 4), arising first in the frontal cluster (Fig. S9). Furthermore, those mapping selective electrodes showed an interaction, defined by the coherence between the corresponding physiological signals, with visually responsive electrodes and motor-responsive electrodes (Fig. 5–7). These interactions were manifested in the broadband gamma frequency range (Fig. 6), were restricted to the delay period (Fig. 6), exhibited a short temporal lag from visual to mapping to motor electrodes (Fig. S12C–D), and were only evident after learning (Fig. 7).

The mapping selective electrodes are distinct from the purely visually responsive electrodes or the purely motor responsive electrodes in terms of their anatomical location, their response timing, and their response properties. Visually responsive electrodes were clustered along the ventral visual cortex (compare Fig. S8C versus Fig. 2), did not differentiate between the stimulus groups, had short latency responses during the stimulus period (compare Fig. S2 versus Fig. 2B–D), and showed a weaker effect of learning (compare Fig. S5 versus Fig. 3), separating them from the mapping selective electrodes. Motor responsive electrodes were clustered along motor cortex (compare Fig. S8D versus Fig. 4), the responses were centered around the button press (compare Fig. S3 versus Fig. 2B–D), and did not change their responses with learning (compare Fig. S6 versus Fig. 3), separating them from the mapping selective electrodes. While mapping selectivity during the delay period could be interpreted as motor preparation (ultimately, mapping selectivity implies the transformation of a sensory representation into a behavioral response), several observations make a purely motor interpretation unlikely: (i) a purely motor preparation signal would be expected to extend into the motor response time as demonstrated for the motor responsive electrodes, in contrast to mapping selective responses (Fig. 2D); (ii) a purely motor preparation signal would be expected to maintain the differences between the two groups when aligning responses to key press, in contrast to the mapping selective responses (Fig. 2D); (iii) a purely motor preparation signal would be expected to differentiate between the two groups during error trials, in contrast to the mapping selective responses (Fig. 2E); (iv) a purely motor preparation signal would not be expected to change with learning, in contrast to the mapping selective responses (Fig. 3).

Previous studies have shown evidence for visuomotor rule-selective responses in frontal cortex in monkeys (Asaad et al. 1998; Miller and Cohen 2001; Wallis and Miller 2003; Muhammad et al. 2006) and humans (Miller and Cohen 2001; Bunge 2004; Badre et al. 2010; Badre and Frank 2012). Additionally, recordings from the macaque monkey premotor cortex (Wallis and Miller 2003; Muhammad et al. 2006), dorsal striatum (Muhammad et al. 2006) and parietal cortex (Freedman and Assad 2006) have also identified correlates of rule selective responses. In the current report, we observed mapping selectivity in frontal and posterior parietal cortex, consistent with those locations in previous work. We also observed multiple mapping selective electrodes in the anterior parts of inferior temporal cortex, even though rule-selective responses seem to be less prominent in the macaque inferior temporal cortex (Freedman et al. 2001; Meyers et al. 2008). Comparisons across these studies are challenging due to the differences in tasks, recording techniques and species. Based on the relative timing between the responses in these anatomical clusters, the mapping selective responses in inferior temporal cortex and posterior parietal cortex may be the result of top-down signals from the frontal cortex cluster of electrodes.

While many of the neurophysiological studies in monkeys discussed above report the responses after the animals have been extensively trained in the task, here we show that mapping selectivity arises during the course of a session and is contingent on behavioral learning. These learning-dependent changes are reminiscent of the activity of neurons in macaque hippocampus during learning of new associations (Wirth et al. 2003).

Such visuomotor associations require communication between brain areas involved in visual, motor and mapping selectivity during learning. To understand the interactions between vision, motor and mapping responses and how this communication might change during learning, we computed the degree of coherence in the field potential signals across pairs of electrodes. Coherence analyses have been used in many studies to document putative interactions between areas (Womelsdorf et al. 2007; Pesaran et al. 2008; Baldauf and Desimone 2014). We observed a significant increase in coherence in the gamma band (30–100 Hz); this frequency band has also been reported in other studies of pairwise coherence (Womelsdorf et al. 2006; Jutras et al. 2009; Korzeniewska et al. 2011; Vidal et al. 2012; Baldauf and Desimone 2014). We report enhanced coherence both between visually responsive and mapping selective electrodes and also between motor responsive electrodes and mapping electrodes. We interpret these results to suggest that information from visual areas is routed to motor outputs through the mapping selective regions. These interactions were restricted to the delay period and could represent a link between interpretation of the visual input and action execution.

These interactions were not exclusively restricted to mapping electrodes: we also observed increased coherence between visually responsive electrodes (Figs 6H and 7G). These interactions are particularly intriguing, given that the individual visually responsive electrodes did not reveal any appreciable change during the delay period (e.g., note the essentially flat baseline response between 800 and 1300 ms after stimulus onset for the example electrode in Fig. S2). Combining the coherence changes within visually responsive electrodes (Figs 6H and 7G), the weaker but still evident effects of learning on individual visually responsive electrodes (Fig. S5), and the fact that the location of some of the mapping electrodes (group G3 in Fig. 4) partially overlapped with that of some of the visually responsive electrodes (Fig. S8C), these results suggest that visual areas may also play a central role in the arbitrary mapping task. On a speculative note, the long delay of the responses in group G3 (Fig. S9) and the interactions between visually responsive electrodes taking place hundreds of milliseconds after the visually evoked responses, might argue that the rule-dependent signals and interactions along ventral visual cortex might be the result of top-down feedback signals from frontal areas.

Previous studies have found that changes in gamma-band coherence can be accompanied by modulation in gamma power in the individual signals (Fries et al. 2008; Chalk et al. 2010). However, multiple lines of evidence show that the changes in coherence described in Figures 5–7 cannot be explained as a direct consequence of increased power in the gamma band in the individual electrodes: (i) The coherence metric is normalized by power in each frequency so that changes in coherence do not merely reflect a change in power from one or both of the component electrodes. (ii) We found significant increase in gamma power during the delay period, compared to pre-stimulus baseline for some, but not all, of the task-selective electrodes: 50.5% of visually responsive electrodes, 25.7% of mapping-selective electrodes and 28.5% of motor-responsive electrodes (P < 0.01, Wilcoxon rank sum test) and therefore there were many electrodes that showed interactions without accompanying changes in power. (iii) Significant changes in coherence during the delay period were also observed for pairs of electrodes that did not demonstrate any change in gamma band power during the delay period (see the small number of mapping selective electrodes in the gamma band in Table S2). (iv) There was no correlation between the change in coherence in the gamma band for a given pair of electrodes and the change in gamma power of the individual electrodes (Fig. S11). (v) There was a small increase in power-power coupling (Vidal et al. 2012) in the gamma band in the mapping-visual electrode pairs during the delay period and in the mapping-motor electrodes during the response period, but these changes were not statistically significant (Fig. S12A–B, P > 0.5, Wilcoxon, rank sum test). In sum, while an increase in gamma power of individual electrodes provides insights into the contribution of gamma band oscillations during the different phases of the task, here we focused on how these selective areas communicate during a visuomotor task and how these interactions change during learning.

Further dissociation between changes in the responses of individual electrodes and the coherence analyses can be derived from considering the interactions between visually responsive electrodes (black curve in Figs 6H and 7G). Visually responsive electrodes demonstrated transient responses shortly following the stimulus onset rather than during the delay period, either in the broadband signals or in gamma band signals (there was no visual stimulus on the screen during the delay period). Yet, pairs of visually responsive electrodes showed a significant increase in coherence during the delay period. The interpretation of this observation is unclear; on a speculative and post-hoc note, it is conceivable that these interactions between visually responsive electrodes might represent a correlate of working memory required to maintain information about the stimulus during the delay period in the context of previous responses. Independently of whether this interpretation is correct or not, these interactions between visually responsive electrodes during the delay period provide a strong dissociation between power changes in individual electrodes and coherence measurements.

The small magnitude of the coherence changes reported here is consistent with previous studies. We provide a few examples only as a coarse order-of-magnitude estimation of coherence changes in the gamma band in the literature: (i) Baldauf and colleagues reported changes of approximately 5–10% in gamma band coherence between the inferior frontal junction and either the fusiform face area or the parahippocampal place area in an attention task in humans using a combination of fMRI and MEG (Figure 3A in (Baldauf and Desimone 2014)). (ii) Jutras and colleagues reported changes in gamma band coherence of about 10% between spikes and local field potentials (LFPs) within the macaque hippocampus (Figure 2C in Jutras et al. (2009)). (iii) Womelsdorf and colleagues report changes of about 5–10% in gamma band coherence between spikes and LFPs within macaque area V4 (Figure 3B,F in Womelsdorf et al. (2006)). Pesaran and colleagues showed larger changes in gamma band coherence between spikes and LFPs within the macaque lateral intraparietal area during a working memory task (Figure 7 in Pesaran et al. (2002)). It should be noted that it is complicated to directly compare coherence changes in studies that used different recording techniques, different species, different tasks, and compared different brain areas.

The interactions demonstrated here are consistent with studies showing that transcranial magnetic stimulation of frontal cortex or posterior parietal cortex leads to activation of visual cortical areas (Silvanto et al. 2006, 2009; Karabanov et al. 2012) and also with studies showing interactions between frontal cortex, parietal cortex and visual cortical regions in monkey neurophysiology (Pesaran et al. 2008; Bichot et al. 2015; Siegel et al. 2015) and human functional neuroimaging (Gazzaley et al. 2007; Karabanov et al. 2012; Baldauf and Desimone 2014; Mackey and Curtis 2017). Whereas some investigators have argued that coherence interactions between regions provide evidence of putative functional connectivity, the biophysical implications of the changes in coherence demonstrated here are not clear and should be interpreted with caution. The interactions demonstrated in Figures 5–7 need not necessarily reflect direct neural communication. It is possible to observe enhanced coherence between two brain areas even if the communication is mediated by another area. Yet, there exist anatomical substrates for potential direct communication between the areas described here. There are strong and direct anatomical connections between lateral frontal cortex and the anterior parts of inferior temporal cortex as well as between lateral frontal cortex and motor cortex (Fuster 2001; Miller and Cohen 2001). While the connections from anterior inferior temporal regions to frontal cortex and to parietal areas have been extensively documented (e.g., Ungerleider et al. 1989; Distler et al. 1993), to our knowledge, direct projections from inferior temporal cortex to motor cortex have not been documented.

The small differences in the phase lags (Fig. S12) are suggestive of a directionality of information transfer from visually responsive electrodes to mapping selective electrodes to motor responsive electrodes. There has been recent exciting work based on macaque neurophysiological recordings (Bastos et al. 2015) and human magnetoencephalography (Michalareas et al. 2016) showing that different frequency bands are involved in conveying feed-forward versus feed-back information. In particular, those studies ascribe feed-forward propagation to gamma band signals such as the ones used in the current study, consistent with the tentative directionality from visual to mapping to motor signals. Yet, the results presented here do not demonstrate causal interactions. Future work should evaluate such putative causality through further analyses such as Granger causality metrics.

Rapid and flexible learning of new rules is critical for adaptive behavior. The results presented here provide initial steps towards elucidating the dynamic interactions across brain areas that may be critical to instantiate such flexibility by mapping visual inputs into adequate motor actions.

Authors’ Contributions

The experiment was designed by A.K.B. and G.K. The surgical procedures were conducted by J.R.M., W.S.A., A.J.G., T.S.T. and E.N.E. The data were collected and analyzed by R.M. and A.K.B. The manuscript was written by R.M., A.K.B. and G.K. with feedback from all authors.

Funding

This work was supported by grants from National Science Foundation (1745365, CCF1231216), McKnight Foundation, and National Institutes of Health (R01EY026025).