Selective Modulation of Early Visual Cortical Activity by Movement Intention

Abstract

The primate visual system contains myriad feedback projections from higher- to lower-order cortical areas, an architecture that has been implicated in the top-down modulation of early visual areas during working memory and attention. Here we tested the hypothesis that these feedback projections also modulate early visual cortical activity during the planning of visually guided actions. We show, across three separate human functional magnetic resonance imaging (fMRI) studies involving object-directed movements, that information related to the motor effector to be used (i.e., limb, eye) and action goal to be performed (i.e., grasp, reach) can be selectively decoded—prior to movement—from the retinotopic representation of the target object(s) in early visual cortex. We also find that during the planning of sequential actions involving objects in two different spatial locations, that motor-related information can be decoded from both locations in retinotopic cortex. Together, these findings indicate that movement planning selectively modulates early visual cortical activity patterns in an effector-specific, target-centric, and task-dependent manner. These findings offer a neural account of how motor-relevant target features are enhanced during action planning and suggest a possible role for early visual cortex in instituting a sensorimotor estimate of the visual consequences of movement.

Introduction

A prominent organizational feature of the primate visual system is its extensive, highly interconnected web of feedback projections. As a case in point, the feedback that primary visual cortex (area V1) receives from higher-order cortical areas is far more extensive than the input is receives from the retina (Felleman and Van Essen 1991; Carandini et al. 2005; Muckli and Petro 2013). Such a feedback architecture provides early visual cortex with access to the output of operations performed at higher stages of visual and/or cognitive processing (Gilbert and Li 2013). To date, this well-known organization has been predominantly investigated in the context of working memory and visual-perceptual tasks. For instance, in the domain of visual working memory, these feedback projections are thought to help maintain specific information about stimulus features in early visual areas when those stimuli are no longer in view (Harrison and Tong 2009; Christophel et al. 2012). In the context of visual-perceptual processing, different models argue that feedback to V1 sharpens perceptual representations (Jehee et al. 2007), enhances consciously available visual information (Super et al. 2001), tunes, and anticipates the response to perceptual stimuli (Ress and Heeger 2003), or cancels out “predicted” visual signals, thereby allowing only unexpected sensory information to arrive at subsequent stages of processing (Murray and Wojciulik 2004; Bastos et al. 2012; Clark 2013). While there is some debate about which subset of these theories or models is correct, it is nevertheless widely accepted that top-down projections to visual cortex influence perception by modulating early retinotopic representations of visual stimuli. Beyond visual-perceptual processing and working memory, however, the role—if any—of these feedback projections in the planning and control of goal-directed movements remains poorly studied.

One possible role of top-down projections to early visual areas during planning is to enhance object features critical for motor control (Craighero et al. 1999; Bekkering and Neggers 2002; Fagioli et al. 2007; Gutteling et al. 2011, 2013; Perry and Fallah 2017), which are generally different than those critical for perception (Brouwer et al. 2009). For example, prior to an impending reaching movement, there is enhanced processing at the target’s spatial location (Baldauf et al. 2006; Baldauf and Deubel 2008) and, likewise, prior to object grasping, there is enhanced processing of the target’s orientation (Gutteling et al. 2011, 2013). Such attentional enhancement likely explains recent results showing that activity in human visual cortex can be shaped by impending pointing and grasping movements, which have different spatial attentional requirements (Chapman et al. 2011; Gutteling et al. 2015). A second possible role of these top-down projections is to filter very early sensory information so as to allow the sensorimotor system to separate expected versus unexpected sensory outcomes of action (Wolpert and Flanagan 2001; Flanagan et al. 2006; Franklin and Wolpert 2011). Specifically, sensorimotor prediction of the visual consequences of movement at the level of early visual cortex could enable the more rapid detection of movement errors and their subsequent correction (Wolpert and Ghahramani 2000; Wolpert et al. 2011). At the neuroanatomical level, both the motor-related enhancement and sensorimotor filtering of information could, in principle, be mediated by top-down projections to early visual cortex from parietal and frontal areas involved in action planning and attention-orienting (Moore and Fallah 2004; Borra and Rockland 2011; Greenberg et al. 2012; Takemura et al. 2015; Perry and Fallah 2017).

Here, using human functional magnetic resonance imaging (fMRI) and multivoxel pattern analysis (MVPA), we evaluated the general hypothesis, in three separate experiments, that early visual cortex is selectively modulated by motor-specific, top-down projections. Specifically, we tested two key predictions, related to neural coding, that have been previously used to map and characterize motor representations in parietal cortex (Snyder et al. 1997; Andersen and Buneo 2002). First, we tested the prediction that, during movement preparation, information about that effector to be used, as well as the particular action to be performed by that effector, can be decoded from patterns of activity in early visual cortex. Second, we tested the prediction that decoding of the intended action—unlike the effects of global attention, which uniformly impacts the whole of visual cortex (Serences and Boynton 2007)—would be primarily observed in corresponding neural representations of the target object(s) to be acted upon in the upcoming movement. Confirmation of these two predictions would suggest that early visual cortex, rather than being a passive purveyor of sensory information, is more involved in the initial visual-to-motor transformation stages than previously expected.

Materials and Methods

Overview

Here we sought to determine if, and to what extent, motor-related processing during planning influences early visual cortical representations in a top-down manner. To investigate this issue, we performed a new analysis on two previously published experiments (Gallivan, McLean, Flanagan, et al. 2013; Gallivan et al. 2016) and conducted an additional, third experiment. In brief, the present investigations all required participants to perform several different types of target-directed movements while maintaining central fixation. In each trial, participants were first visually presented with the target object(s) and cued to the specific action to be performed; then, following a delay period, they executed the prepared action. Importantly, the target object(s) were always presented throughout the full trial sequence and positioned in the same location(s) (in peripheral vision) for the duration of each experiment. Thus, the visual presentation of the object(s) remained constant both within and between all trials (with respect to central fixation). The timing of our tasks enabled isolation of the delay period activity prior to movement (Delay epoch) from the later, movement execution responses (Execute epoch). This allowed us to examine, using neural decoding techniques (Tong and Pratte 2012), whether we could predict the upcoming action to be performed, on a given trial, from delay period voxel activity patterns in early visual cortex that correspond to the retinotopic location of the target object. To demonstrate the specificity of these effects, we also performed the same analyses in control regions of visual cortex that correspond to the location of non-target objects on a given trial (i.e., an object location that was not to be acted upon).

Subjects

Fourteen subjects (seven females; age range: 20–28) participated in Experiment 1, 11 subjects (five females; age range: 22–33) participated in Experiment 2, and the same eleven subjects participated in Experiment 3. All experiments were undertaken with the understanding and written consent of each subject, obtained in accordance with the ethical standards set out by the Declaration of Helsinki (1964) and with procedures approved by the University Health Sciences Research Ethics Board. Experiment 1 was performed at Queen’s University (Ontario, Canada) and Experiments 2 and 3 were performed at the University of Western Ontario (Canada). The complete methods for Experiments 1 and 2 have been previously described in detail elsewhere (Gallivan, McLean, Flanagan, et al. 2013; Gallivan et al. 2016). As such, here we provide more concise descriptions of the methods relevant for our new analyses.

Experimental Design

Experiment 1

Subjects were scanned in a head-tilted configuration (allowing direct viewing of the hand workspace) while they performed an object-directed delayed movement task (see Fig. 2A,B for an overview of the experimental setup and timing). During the experiment, the participant’s workspace was illuminated with bright white light-emitting diodes (LEDs) attached to flexible plastic stalks. Experimental timing and lighting were controlled with in-house software created in MATLAB and C++. To control for eye movements during MRI scanning, a small fixation LED, attached to a flexible plastic stalk, was placed above and behind the target objects and participants were required to always foveate the fixation LED during experimental testing. A cube object, left cup, and right cup were positioned at ~7°, 12°, and 11° of visual angle with respect to the fixation point (for further methodolocial details, see Gallivan et al. 2016). Throughout the experiment, the subject’s arm movements were recorded with a magnetic resonance-compatible infrared-sensitive camera (bore camera, MRC Systems; not shown in Fig. 2A). The videos captured during the experiment were then analyzed off-line in order to exclude error trials from analysis.

Figure 2.

Delay period decoding in primary visual cortex (area V1) for Experiment 1 is linked to the task-relevant object locations on a given trial. (A) Experiment 1 task. Left, subject point of view (POV) during the Delay epoch. Red star denotes the fixation LED and the hand, to the right of the cube object, is shown at its starting position. Right, The three object-directed action sequences, each involving the centrally located cube object, performed by participants (during Execute epoch). (B) Timing of each event-related trial. Trials began with the hand workspace being illuminated while, simultaneously, participants received an instruction, via headphones, to prepare one of the three possible hand movements. This initiated the Delay epoch of the trial. After a jittered delay interval, participants were then cued, via an auditory signal (“Beep”), to execute the instructed hand movement. This initiated the Execute epoch of the trial. Two seconds following this Go cue, illumination of the workspace was extinguished, cueing participants to return their hand to the starting position. Participants then waited for the following trial to begin (16 s, intertrial interval, ITI). Subjects were required to maintain fixation on the LED over the entire duration of the trial. (C) Selective decoding in V1 of object-directed action sequences directed towards the task-relevant object locations (cube, left and right cups), separately mapped using the procedures shown in Fig. 1A–D. Each ROI is associated with two plots of data, with the corresponding legends shown at bottom right. Top, percentage signal change time course activation. The activity in each plot is averaged across all voxels within each ROI and across participants. Note that due to jittering of the delay period in the event-related design, to allow alignment, only time courses for 5-volume (10-s) delay periods are shown and averaged. The vertical dashed line corresponds to the onset of the Execute epoch of the trial. Shaded gray bars indicate the 2-volume (4-s) windows that were averaged and extracted for pattern decoding. Bottom, Pairwise decoding accuracies, shown for each time epoch. To examine the extent to which intention-related information could be recovered from visual cortex brain activity, decoding from voxel patterns during the pre-movement time window (bordered in red) was of particular interest. Note that accurate classification is primarily attributable to the voxel activity patterns associated with different action sequences and not to differences in the overall signal amplitude (i.e., the time courses are generally overlapping during the Plan epoch). Error bars represent ±1 SEM across participants and dashed horizontal black lines denote the chance accuracy level (50%). Black asterisks indicate statistical significance with two-tailed t-tests across participants with respect to the chance level of 50% correct. Red asterisks indicate statistical significance using an FDR correction of q ≤ 0.05, based on the total number of regions examined in this experiment. The presentation of both levels of statistical significance (black and red asterisks) allows for results that did not make the FDR correction threshold to be inspected. (D) No decoding of action sequences from pre-movement signals in sub-regions of V1 corresponding to the task-irrelevant object locations (top left and right positions), separately mapped using the same procedures shown in Fig. 1A–D. Percentage signal change time courses and decoding accuracies are plotted and computed the same as in C. (E) No decoding of action sequences from pre-movement signals in parafoveal retinotopic cortex, separately mapped using the procedures shown in Fig. 1E,F. Percentage signal change time courses and decoding accuracies are plotted and computed the same as in C. Note that color-bordering around POV images in C–E are meant to correspond with object locations in Fig. 1C.

For each trial, subjects were required to perform one of three actions upon the target object: (1) grasp, lift, and replace the cube object, (2) grasp, lift, and place the cube object in the left cup, or (3) grasp, lift, and place the cube object in the right cup (see Fig. 2A). These actions were cued via the auditory commands, “grasp,” “left,” or “right,” respectively. Other than the execution of these hand actions, participants were instructed to keep their hands still and in a pre-specified “home” position throughout all other phases of the task.

Each trial began with the Delay epoch, in which, concurrent with the auditory cue instructing the upcoming movement required (mentioned above; delivered through headphones), the subject’s workspace was illuminated, revealing the object locations. Following a jittered delay interval (6–12 s in duration; randomly selected from a Gaussian distribution centered on 9 s), a 0.5-s auditory beep cued participants to immediately execute the planned action (for a duration of ~2 s), initiating the Execute epoch of the trial. Two seconds following the beginning of this Go cue, the illuminator was turned off, providing the cue for subjects to return their hand to its starting position. Subjects then waited (16 s) in the dark (intertrial interval, ITI) for the following trial to begin (see Fig. 2B). The three trial types, with six repetitions per condition (18 trials total), were pseudorandomized within a run and balanced across all runs so that each trial type was preceded and followed equally often by every other trial type across the entire experiment. Each subject participated in nine functional runs and two object retinotopic mapping runs, the latter being performed at the end of each participants’ respective testing session (for further details about the object retinotopic mapping runs, see below; see also Gallivan et al. 2014). Note that we did not conduct eye tracking during this or any of the other scan sessions because of the difficulties in monitoring gaze in the head-titled configuration with standard MRI-compatible eye trackers (due to occlusion from the eyelids). Nevertheless, multiple behavioral control experiments done at Queen’s University (Gallivan et al. 2014, 2016) and the University of Western Ontario (e.g., Gallivan, McLean, Flanagan, et al. 2013) for the majority of the fMRI data sets analyzed here, have demonstrated that the same groups of subjects tested with MRI can reliably maintain fixation during behavioral testing. Two of the 14 subjects were removed from analysis (leaving N = 12) due to data collection issues associated with the object retinotopic mapping scans (see below).

Experiment 2.

This study was similar to Experiment 1 with the exception that (1) participants performed four different hand movements (grasp left, grasp right, reach left, and reach right, see Fig. 3A), (2) the Delay epoch was preceded by a Preview epoch (6 s), and (3) the Delay epoch was slightly longer (a fixed duration of 12 s). [Note that, although there were no visual differences between the Preview and Delay epochs, only in the Delay epoch did participants have the information necessary to internally prepare the upcoming action]. Here, the target object was located ~8° with respect to central fixation (for further methodological details, see Gallivan, McLean, Flanagan, et al. 2013). The four trial types, with five repetitions per condition (20 trials total), were pseudorandomized as in Experiment 1. Each subject participated in eight functional runs and two object retinotopic mapping runs, the latter performed at the end of each participants’ respective testing session (for further details, see below).

Figure 3.

Delay period decoding in primary visual cortex (area V1) for Experiment 2 is both effector-specific and goal-dependent, and is linked to the task-relevant object location. (A) Experiment 2 task. Left, subject POV during the Delay epoch. Green star denotes the fixation LED and the hands, to the left and right of the object, are shown at their respective starting positions. Right, the four object-directed hand movements performed (during Execute epoch). (B) Timing of each event-related delayed movement trial. (C) Decoding of each hand action from pre-movement signals in the sub-region of V1 corresponding to the task-relevant object location, separately mapped using the same procedures shown in Fig. 1A–D. Each ROI is associated with three plots of data, with the corresponding legends shown at bottom. Top, percentage signal change time course activation. The activity in each plot is averaged across all voxels within each ROI and across participants. The vertical dashed line corresponds to the onset of the Delay and Execute epochs of the trial. Shaded gray bars indicate the 2-volume (4-s) windows that were averaged and extracted for pattern decoding. Bottom left, pairwise decoding accuracies, shown for each time epoch. Bottom Right, cross-decoding accuracies, shown for each time epoch. Effector-specific, action-independent accuracies were computed from training classifiers on grasp left versus grasp right trials and testing on reach left versus reach right trials and then averaging the resulting accuracies with those obtained from the opposite train- and test-ordering, within each subject. Action-specific, effector-independent accuracies were computed from training classifiers on grasp left versus reach left trials and testing on grasp right versus reach right trials (again, averaging these resulting accuracies with those obtained from the opposite train- and test-ordering, within each subject). Error bars, horizontal black lines, black asterisks, and red asterisks are plotted and computed the same as in Fig. 2. (D) No decoding of hand actions from pre-movement signals in the sub-region of V1 corresponding to the task-irrelevant object location, separately mapped using the same procedures shown in Fig. 1A–D. Percentage signal change time courses and decoding accuracies are plotted and computed the same as in C. (E) No decoding of hand actions from pre-movement signals in parafoveal retinotopic cortex, separately mapped using the same procedures shown in Fig. 1E,F. Percentage signal change time courses and decoding accuracies are plotted and computed the same as in C.

Experiment 3.

This experiment was similar to Experiment 2, with the exception that participants performed either movements of the eye or the hand towards two different spatial targets (look left, look right, grasp left, grasp right, see Fig. 4A). The left and right target objects were located at ~15° and 12° with respect to central fixation, respectively. The four trial types, with five repetitions per condition (20 trials total), were pseudorandomized as in Experiments 1 and 2. Each subject participated in eight functional runs and two object retinotopic mapping runs, the latter performed at the end of each participants’ respective testing session (for further details, see below).

Figure 4.

Delay period decoding in primary visual cortex (area V1) for Experiment 3 is both effector-specific and spatially selective, and is linked to the task-relevant object locations. (A) Experiment 3 task. Left, subject POV during the Delay epoch. Green star denotes the fixation LED and the hand, centrally located, is shown at its starting position. Right, the four different movements performed (during Execute epoch), two of which involved the hand and two of which involved the eyes. (B) Timing of each event-related delayed movement trial. (C) Selective decoding in V1 of movements directed towards the upcoming task-relevant object location, separately mapped using the same procedures shown in Fig. 1A–D. Each ROI is associated with three plots of data, with the corresponding legends shown at bottom. For the cross-decoding accuracies, effector-specific, direction-independent accuracies were computed from training classifiers on look left versus grasp left trials and testing on look right versus grasp right trials and then averaging the resulting accuracies with those obtained from the opposite train- and test-ordering, within each subject. Direction-specific, effector-independent accuracies were computed from training classifiers on look left versus look right trials and testing on grasp left versus grasp right trials (again, averaging these resulting accuracies with those obtained from the opposite train- and test-ordering, within each subject). Error bars, horizontal black lines, black asterisks, and red asterisks are plotted and computed the same as in Fig. 2. (D) No decoding in V1 from pre-movement signals in task-irrelevant locations, separately mapped using the same procedures shown in Fig. 1A–D. Percentage signal change time courses and decoding accuracies are plotted and computed the same as in C. (E) Selective decoding of eye but not hand movements from pre-movement signals in parafoveal retinotopic cortex, separately mapped using the same procedures shown in Fig. 1E,F. Percentage signal change time courses and decoding accuracies are plotted and computed the same as in C. Note that color-bordering around POV images in C–E are meant to correspond with general object locations in Fig. 1C.

Localizer Scans

Retinotopic Mapping of Object Locations

To retinotopically map the location of the target object(s) in Experiments 1–3, at the end of each motor testing session (i.e., with subjects in the same head-tilted experimental setup), hollow semi-opaque illuminable objects were presented in (1) locations at which the target objects appeared throughout the testing session (task-relevant locations), and (2) locations outside of reach that were never acted upon throughout testing (task-irrelevant control locations; see Fig. 1C for examples). Each object contained two super-bright LEDs in the center, which could flicker on-and-off at 5 Hz, one at a time, within the scanner. Other than the periods of object illumination/flickering, the scanner environment was completely dark, except for a fixation point LED that was too dim to illuminate the scene. Each of the experimental runs was composed of eight stimulus epochs per object location (12 s each), with each stimulus block separated by an ITI (10 s each, in which subjects simply maintained fixation in the dark), and two baseline epochs (12 s each; also fixation in the dark) placed at the beginning and end of each run. For the entire duration of these experimental runs, participants were required to maintain their gaze on the fixation point.

Figure 1.

Methods and representative results from retinotopic mapping of task-relevant (TR) and task-irrelevant (TI) object locations (A–D), as well as parafoveal cortex (E and F). (A) Examples of target objects for hand actions (left) and corresponding illuminable objects used for mapping of retinotopic locations (right), taken from Experiment 1. (B) Example setup from Experiment 1, showing the three task-relevant object locations (cube and left and right cup locations) (see Fig. 2 for how these objects were interacted with during the task). (C) Example setup for retinotopic mapping of five separate object locations (always done at the end of experimental testing). The three bottom illuminable objects (cube, bottom left, and bottom right) were positioned in the same locations as the target objects in B. The top left and right objects were positioned in locations not acted upon during the actual experiment. These latter TI locations allowed us to map retinotopic cortical locations that were used for control analyses; that is, these regions allowed us to directly test the idea that planning-related modulation in V1 is specific for the retinotopic cortical representations of the target objects (TR locations) and does not extend into retinotopic cortical representations of positions in visual space that were not actually acted upon during testing (TI locations). (D) Early visual cortical representations of the five different object locations shown in C, displayed on the flattened cortex of a representative participant. The activations are color-coded to correspond with the object positions (and border colors) shown in C and are based on a contrast of the object flashing at each respective location versus baseline (each activation is shown at t = 15, P < 7.0 × 10⁻⁴⁵). The dotted white lines indicate delineations of early visual areas, V1–V3, derived from standard retinotopic mapping procedures performed in a separate testing session for the same participant. That is, areal boundaries were defined at the appropriate horizontal and vertical meridia based on phase mapping procedures. Spheres (of 5-mm radius) were placed at the peak voxel locations of each object-related activation in V1 and provided inputs for pattern classification analysis. (E) Contrast used for retinotopic mapping of the foveal confluence. “Traveling wave” stimuli, consisting of expanding rings (presented during a separate testing session), were binned based on their peripheral versus central positions in the visual field and input into a general linear model for the purposes of localizing parafoveal retinotopic cortex (shown in orange/yellow activation in F). (F) Parafoveal and peripheral visual representations, shown on the medial surface of the inflated hemispheres of a representative participant (same individual as in D). The dotted white lines delineate the calcarine sulcus. For analysis, spheres with a 5-mm radius, as in D, were placed at the peak voxel locations of the parafoveal representations in both the left and right hemispheres. LH = left hemisphere; RH = right hemisphere.

The task-irrelevant, control location objects were placed at the following visual angle distances with respect to central fixation: Experiment 1, top left and top right locations, ~9°; Experiment 2, top location, ~6°; Experiment 3, top left and top right locations, ~8°. All visual angles (including those of the task-relevant objects) were computed (via the arctangent) by placing a clear Plexiglas screen just in front of the fixation point (of known distance from the eye) and having participants, using their peripheral vision, manually estimate, with their right index finger, the visual center of each task-relevant and task-irrelevant object location(s). Note that although the top position objects were of the same size as the bottom position objects and thus, subtended a smaller visual angle (due to greater distance from the eye), the associated regions-of-interest (ROIs) were selected based only on the peak voxels of activity and were fixed in size (i.e., did not scale with activation extent, see below). This, along with the important observation that certain types of movement-related information could also not be decoded from the bottom object retinotopic locations in Experiments 1 and 3 (cases in which the objects subtended approximately the same visual angle), validates our use of the top object locations as control ROIs.

Retinotopy Mapping of Early Visual Areas

In a separate testing session for each subject in Experiments 1–3, early visual areas (i.e., V1, V2, and V3) were mapped and delineated using standard phase-encoded protocols and retinotopic mapping procedures (Sereno et al. 1995; DeYoe et al. 1996; Engel et al. 1997). During this testing, subjects maintained fixation while viewing “traveling wave” stimuli consisting of rotating wedges and expanding rings (Swisher et al. 2007; Arcaro et al. 2009, 2011; Gallivan et al. 2014). The rotating wedge, used for polar angle mapping, was 45° in width and extended to the edges of the screen (~8.5°). The expanding ring, used for eccentricity mapping, increased logarithmically as a function of time in both size and rate of expansion, so as to match the estimated human cortical magnification function (for details, see Swisher et al. 2007). The duty cycle of the annulus was 12.5° (i.e., any given point on the screen was within the aperture for 12.5% of the stimulus period). Each stimulus type (wedge or ring) was presented in a separate scan of 12-min duration, and was composed of 18 wave cycles, each lasting 40 s. To encourage participants to maintain fixation during scanning, subjects performed a detection task throughout, whereby responses were made, via a right-handed button press, whenever they detected a slight dimming of the fixation point (on average, every 4.5 s). Cross-correlation analyzes were used to construct phase-encoded retinotopic maps of polar angle and eccentricity and early visual areal boundaries were delineated using field-sign mapping procedures (Sereno et al. 1995). Retinotopy stimuli were rear-projected with an LCD projector (NEC LT265 DLP projector; resolution, 1024 × 768, 60 Hz refresh rate) onto a screen mounted behind the subject. The subject viewed the images through a mirror mounted to the head coil directly above the eyes.

MRI Scanning

Scanning was done on 3-Tesla Siemens TIM Trio scanners located at Queen’s University and the University of Western Ontario. Functional MRI volumes were acquired using a T2*-weighted single-shot gradient-echo echo-planar imaging sequence (TR = 2 s, echo time = 30 ms, flip angle = 90°, 3-mm isovoxel resolution). During the motor testing sessions, functional volumes were collected using a combination of parallel imaging coils to achieve a good signal/noise ratio and to enable direct object viewing without mirrors or occlusions. We tilted (~20°) the posterior half of the 12-channel receive-only head coil (six channels) and suspended a four-channel receive-only flex coil over the forehead (10 channels total). During the retinotopic mapping of early visual areas scan session, functional volumes were collected using a conventional setup (i.e., the participant was supine and a standard 12-channel receive-only head coil was used). For both testing sessions, each volume comprised 34 contiguous (no gap) oblique slices acquired at a ~30° caudal tilt with respect to the anterior-to-posterior commissure (ACPC) line, providing near whole-brain coverage (for further details, see Gallivan, McLean, Flanagan, et al. 2013; Gallivan et al. 2016).

Statistical Analysis

fMRI data analysis was conducted using Brain Voyager QX v2.8 and in-house custom MATLAB scripts. The preprocessing steps included slice-scan time correction, 3D motion correction, high-pass temporal filtering, and co-registration to each participant’s anatomical image, aligned in ACPC space. Other than the trilinear-sinc interpolation performed during realignment, and the sinc interpolation performed during reorientation, no additional spatial smoothing was applied to the data (i.e., the individual subject data were not transformed into a standard brain space). To localize regions of interest (ROIs) for MVPA, we used a general linear model (GLM) with predictors created from boxcar functions, aligned to the onset of each stimulus block with its duration dependent on stimulus block length, that were then convolved with a standard two-gamma hemodynamic response function. All regression coefficients (betas) were defined relative to the baseline activity during the ITI. For all data, the time course for each voxel was converted to percent signal change before applying the GLM. All reported statistics are Greenhouse-Geisser corrected.

Experiments 1–3

We first delineated, at the single-subject level, retinotopic regions V1, V2, and V3 by identifying reversals in the systematic representation of visual space with respect to the polar angle maps from the “rotating wedge” Retinotopy scans. We additionally used the eccentricity maps to ensure that their phase progressions were essentially orthogonal to these polar angle phase progressions (Sereno et al. 1995; Arcaro et al. 2009). Next, using the separate data from the scans associated with the retinotopic mapping of object locations, we then conducted a voxel-by-voxel univariate analysis, to identify, in each participant and in each visual sub-region (V1–V3), the object-defined area for both the target (task-relevant) and non-target (task-irrelevant) object locations (e.g., identifying a representation of both the target and non-target objects in V1). A sphere of 5-mm radius was placed around the voxel peaks of activity and all the voxels within each sphere provided inputs for pattern classification (this resulted in 33 voxels per sphere). This sphere size was used as it not only allowed for the inclusion of multiple voxels for pattern classification (33 per ROI) but it also ensured that adjacent ROIs in early visual cortex did not overlap (note that similar results to those reported here were observed with the sphere sizes of 4 and 6 mm). The use of a constant sphere size across retinal locations ensured that no retinotopic location was more likely to decode information simply because it contained more voxels. Voxels within V2 and V3 were combined for analysis (i.e., creating a single V2/V3 region) due to the immediate proximity of, and difficulty in reliably separating across subjects, some of the associated object-related activations. Activation foci corresponding to each object position were defined using the object location retinotopic mapping localizer data by the contrast of a single object flashing at a specific position (e.g., left cup location, see Fig. 1) versus baseline (ITI).

In addition, for each participant, we also delineated, using the “expanding ring” retinotopy scans, the foveal confluence (Schira et al. 2007) of areas V1, V2, and V3 (Wandell et al. 2007) (see Fig. 1E,F). This was done by decomposing the duration of each traveling wave cycle (40 s) into ten separate epochs (of 4 s each), creating a GLM with convolved predictors aligned to the onset of each epoch, and then contrasting activations associated with parafoveal (epochs 1–3; corresponding to 0.1–1° of visual angle) versus peripheral (epochs 6–8; corresponding to ~2–6° of visual angle) rings. This contrast robustly identified parafoveal cortex in each subject and hemisphere, at the very posterior extent of the calcarine sulcus at the occipital pole (see Fig. 1E,F for a representative participant). As above, a 5-mm-radius sphere was placed around the voxel peak in the left and right hemisphere and these parafoveal voxels were then used for the ROI-based multivoxel analyses.

For this current study, we did not perform a searchlight-based analysis (Kriegeskorte et al. 2006) for several reasons. First, neuroanatomical variability in the location of the calcarine sulcus, and associated retinotopic cortex, presents particularly unique challenges for across-subject normalization and group-level averaging, both of which are required for group-level statistical inferences in searchlight analyses (Etzel et al. 2013). Indeed, this is why retinotopic analyses, as performed here, involve ROI procedures implemented at the single-subject level. Second, as noted above, the exact placement of the target object locations was tailored to each participant based on their body size and extent of reachable space, as well as general personal comfort. As such, even if the neuroanatomical variability of retinotopic cortex were not a confounding factor, this would still mean that the same target object location (e.g., left cup location) would occupy slightly different retinotopic zones in each participant. We confirmed both of these above points by performing a group-level random effects analysis in which, using the same univariate contrasts outlined above (in the section Experiments 1–3), we functionally identified the retinotopic representation of the object positions used in each experiment (at P < 0.0001, cluster-size threshold corrected). While each of these object locations could be easily identified in visual cortex at the group-level, their retinotopic correspondence was both imprecise and highly overlapping. For instance, we observed that the activation associated with each object location bled across the calcarine sulcus (i.e., dorsal and ventral V1) and, in some cases, across the left vs. right hemispheres. Taken together, these factors would have made the searchlight approach highly problematic in both implementation and interpretation. Our ROI-based approach avoids these limitations while still providing a robust characterization of patterns of retinotopic activity.

Multivoxel Pattern Analysis

MVPA was implemented using a combination of in-house software (with MATLAB), a support vector machines (SVM) binary classifier (libSVM, https://www.csie.ntu.edu.tw/~cjlin/libsvm/), with a constant cost parameter, C = 1, and NeuroElf analysis tools (http://neuroelf.net). BOLD percent signal change values for each subject, ROI and experimental condition provided inputs to the SVM classifier. The percentage signal change response was computed from the time course activity at time points of interest (the windowed average of the time points denoted by the gray-shaded bars in Fig. 2, for example) with respect to the time course of a run-based averaged baseline value (−1, the imaging volume prior to the start of each trial), for all voxels in the ROI (Gallivan, McLean, Smith, et al. 2011; Gallivan, McLean, Valyear, et al. 2011, 2013; Gallivan, McLean, Flanagan, et al. 2013; Gallivan et al. 2014, 2016). These time points corresponded to the activity patterns that form in the two imaging volumes prior to the movement onset cue (i.e., Delay epoch activity) and the two imaging volumes corresponding to the peak of the BOLD response following the movement onset cue (i.e., Execute epoch activity). Following the extraction of each trial’s averaged activity pattern, we rescaled these voxel patterns between −1 and +1 for each trial within an ROI (Misaki et al. 2010). A “leave-one-run-out” cross-validation procedure was used for all classifier training and testing. Decoding accuracies were computed separately for each subject, ROI, trial epoch (Delay or Execute) and pairwise discrimination, as an average across train-and-test iterations (Duda et al. 2001).

Multiclass and Pairwise Discriminations

SVMs are designed for classifying differences between two stimuli and LibSVM (the SVM package implemented here) uses the so-called “one-against-one method” for classification (Hsu and Lin 2002). With the SVMs we performed two complementary types of classification analyses; one in which the multiple pairwise results were combined in order to produce multiclass discriminations (distinguishing among multiple trial types; see Supplementary Figs 2, 6, and 9) and the other in which the individual pairwise discriminations were examined and tested separately.

The multiclass discrimination approach allowed for an examination of the distribution of the classifier guesses through visualization of the resulting “confusion matrix”. In a confusion matrix, each row (i) represents the instances of the actual trial type and each column (j) represents the predicted trial type. Their intersection (i,j) represents the (normalized) number of times a given trial type i is predicted by the classifier to be trial type j. Thus, the confusion matrix provides a direct visualization of the extent to which a decoding algorithm confuses (or correctly identifies) the different classes. All correct guesses are located in the diagonal of the matrix (with classification errors represented by non-zero values outside of the diagonal) and average decoding performance is defined as the mean across the diagonal. The values in each row sum to 1 (100% classification). If decoding is at chance levels, then classification performance will be at 1/number of conditions (i.e., 33.3% in Experiment 1 and 25% in Experiments 2 and 3). For all multiclass discriminations, we statistically assessed decoding significance across participants (for each ROI and trial epoch) using two-tailed t-tests versus chance decoding (see Supplemental Material).

Examination of pairwise discriminations allowed us to identify brain regions that exhibited a region × pairwise decoding interaction. That is, in accordance with our hypotheses that neural decoding in retinotopic cortex should vary as a function of the spatial direction/location of the prepared movement, they permitted examination of whether the decoding of certain pairwise experimental conditions is directly linked to the representation of the target object. It is important to note that such an effect would be largely obscured using the multiclass discrimination approach, which only provides a single decoding accuracy that summarizes the classification of all conditions. For pairwise discriminations, we statistically assessed decoding significance across participants using two-tailed t-tests versus 50% chance decoding. For both the pairwise and multiclass discriminations, we applied a false discovery rate (FDR) correction based on the total number of regions examined per experiment.

Results

Experiment 1

As a first test of the hypothesis that movement planning selectively modulates early visual activity in a top-down manner, we examined the extent to which the action being prepared—but not yet executed—can be decoded from early visual cortical signals. We predicted that, rather than upcoming movement information being fed back uniformly across the whole of visual cortex (as shown in the effects of feature-based attention, e.g., Serences and Boynton 2007), top-down action-related modulations might be generally constrained to zones of retinotopic cortex corresponding to the spatial location of the target object to be acted upon in a given trial (though see Gutteling et al. 2015). To test this prediction, we scanned fourteen subjects while they performed one of three delayed object-directed sequences of hand movements, which varied either in their movement complexity or final spatial goals: (1) grasp a cube object, grasp trial, (2) grasp the cube object to place it in a left cup, place left trial, or (3) grasp the same cube object to place it in a right cup, place right trial (see Fig. 2A,B).

Identification of task-relevant and task-irrelevant locations in retinotopic cortex

To localize the retinotopic sub-regions in early visual cortex (within areas V1 and V2/V3) that correspond to the spatial locations of the target objects, at the very end of the testing session, we placed hollow, semi-opaque illuminable objects, of the same size and shape as the target objects, at (1) locations at which the target objects appeared (cube, bottom left cup, bottom right cup; task-relevant (TR) locations, see Fig. 1C) and (2) control locations, outside of participants’ reach, that were never acted upon during the experiment (top left cup, top right cup; task-irrelevant (TI) locations, see Fig. 1C) (for similar methods, see Gallivan et al. 2014). In a block-design protocol, these illuminable objects alternated flickering on-and-off (at 5 Hz; one at a time), resulting in robust and reliable identification of the five different aforementioned object positions in V1 and V2/V3 within each subject (see Fig. 1D). (Note that the boundaries of V1–V3 were defined in a separate localizer testing session using standard retinotopic mapping procedures, see “Materials and Methods” section.) Due to data collection issues in two participants during these scans, they were removed from further analyses.

Delay Period Decoding Is Linked to Task-Relevant, Not Task-Irrelevant, Locations

If the feedback of action-related information during the Delay epoch is linked to the object location(s) to be acted upon in any given trial, then successful decoding should be observed in the retinotopic representations of those target objects (i.e., task-relevant locations) and absent in the zones of retinotopic cortex not actually acted upon during testing (i.e., task-irrelevant locations). To test this idea, we extracted, from our separately defined object representations in V1, the trial-related spatial voxel activity patterns associated with the Delay (and Execute) epoch for the object-directed action sequence task and then used these as inputs to a SVM pattern classifier (see gray-shaded bars in Fig. 2C,D for the time windows that were averaged and used as inputs for classification). An analysis on classification accuracies revealed that the decoding of the upcoming actions to be performed (i.e., during the Delay epoch) was constrained to the retinotopic zones in V1 corresponding to the object locations to be acted upon and, importantly, was not observed in retinotopic zones of V1 corresponding to the control object locations. That is, we found significant decoding, during the Delay epoch, in retinotopic sub-regions corresponding to the task-relevant (Fig. 2C) but not task-irrelevant (Fig. 2D) locations (see Supplementary Fig. 1 for similar results in the corresponding V2/V3 regions).

Moreover, we found that the precise nature of this retinotopic modulation was even more saliently revealed in the movement-related decoding from the cube, left cup, and right cup object representations in V1 (i.e., within the task-relevant locations themselves). Pairwise decoding in the sub-region of V1 corresponding to the left cup location was found to be linked only to the experimental conditions that involved actions to be executed towards that leftward spatial location (grasp vs. place left, t₁₁ = 2.970, P = 0.013; place left vs. place right, t₁₁ = 3.390, P = 0.006; but not grasp vs. place right, t₁₁ = 1.235, P = 0.242), and the same, but opposite, was also true for the sub-region of V1 corresponding to the right cup location (grasp vs. place right, t₁₁ = 3.266, P = 0.007; place left vs. place right, t₁₁ = 2.515, P = 0.029; but not grasp vs. place left, t₁₁ = 1.156, P = 0.272). Notably, in the sub-region of V1 corresponding to the cube location, we found decoding for upcoming actions that only became differentiated relatively early in the action sequence, near that particular spatial location (i.e., whether the individual would move the cube object towards a future cup location or simply replace it at that location; grasp vs. place left, t₁₁ = 3.890, P = 0.003; grasp vs. place right, t₁₁ = 2.684, P = 0.021) and not for those actions that only became reliably differentiated much later into the action sequence, outside of the cube object’s visual spatial location (i.e., place left vs. place right, t₁₁ = 1.205, P = 0.253). To complement these pairwise decoding results above, we also performed multiclass discriminations to visualize the pattern of misclassifications, the results of which are shown in Supplementary Fig. 2 (see also Supplemental text). Together, these results show that even though the first step of each of the three action sequences is identical (i.e., grasping the cube object), pattern information in V1 fully predicts the final set of movement goals to be performed (i.e., whether the object will be simply replaced at the cube location or placed into the left or right cup). This suggests that delay period activity in V1 is not simply linked to a low-level coding of where the fingers will be placed on the cube object but, rather, ultimately how the object will be interacted with by the individual. In other words, this result indicates that feedback projections carry information about the whole action plan and do not just modulate visual neural activity corresponding to the very initial action location.

Delay Period Activity Does Not Appear to Encode Imagined Visual Events

One possible interpretation concerning the specificity of these above results is that individuals might be simply imagining the movements to be executed mere moments later, and that this visual imagery is responsible for driving Delay epoch decoding. According to previous work, such imagery involves the reactivation of neural activity patterns associated with actual viewing and/or perception (Ishai et al. 2002; Polyn et al. 2005; Slotnick et al. 2005; Buchsbaum et al. 2012; Johnson and Johnson 2014; Naselaris et al. 2015; Wing et al. 2015). Considerable evidence implicates a role for V1 in such imagery (for a recent meta-analysis and review of this material, see Winlove et al. 2018), though it is often in cases in which imagery wholly constitutes the task itself (i.e., tasks in which individuals are instructed to explicitly perform these imaginings). To directly test this visual imagery account of our current findings, we performed a cross-decoding analysis in which, employing a leave-one-run-out cross-validation procedure, we used the Execute epoch voxel patterns for training the SVM classifier and the Delay epoch voxel patterns for testing the classifier. The logic of this analysis is that if the Delay epoch activity is simply a neural reinstatement of the activation associated with the subsequent execution of the movement, then the Execute and Delay epoch activity patterns should be similar enough to allow for cross-decoding. Notably, this analysis revealed no reliable cross-decoding (see Supplementary Fig. 3), suggesting that a pure reactivation-based visual imagery explanation of our findings is unlikely. In the sensorimotor research domain, this observation is entirely consistent with mounting neurophysiological work (Churchland et al. 2010, 2012; Shenoy et al. 2013; Kaufman et al. 2014) showing that the neural responses of a brain region during planning are not simply a subthreshold representation of its neural responses during execution. The extent to which the same is also true of certain types of visual imagery remains a topic for future investigation.

Delay Period Activity Is Unlikely to Reflect Eye Movement Confounds

Despite our requirement that participants maintain fixation throughout the task (see “Materials and Methods” section), one alternative explanation of the present results is that the decoding of action-related information in visual cortex, rather than the result of feedback signals from higher-order areas, simply reflects systematic differences in the patterns of eye movements across the three conditions. If this were the case, then we should be able to use activity patterns from the zone of retinotopic cortex corresponding to the fovea—which is displaced during eye movements—to similarly decode differences between the three types of trials. To test for this potential confound, we independently mapped each participant’s foveal confluence during a separate testing session (see Fig. 1D,E) and extracted the corresponding voxel pattern responses during the Delay epoch from our action sequence task. Critically, when we did this, we observed no decoding of prepared actions from this region (see Fig. 2E). This is consistent with our previous observation that participants’ gaze behavior in this task, when tested outside the scanner, did not systematically differ between the trial types (see Gallivan et al. 2016). Together, these control findings suggest that the feedback of action-related information to early visual cortex is largely selective to the representations of the target object(s), and is not widely distributed throughout the entire early visual cortex.

Delay Period Decoding Is Driven by Multivariate, Not Univariate, Information

As an additional control analysis, we further wondered whether the pairwise decoding results observed above were actually driven by the underlying multivariate voxel pattern, as assumed, or instead primarily a result of the mean univariate signal within each ROI. With respect to the latter, past neuroimaging work in humans (Tootell et al. 1998; for review, see Silver and Kastner 2009) has shown that shifts in spatial attention can reliably influence univariate activity in cortical topographic maps of visual space. To examine whether such effects also explain our data we computed, for each ROI, the mean % BOLD signal across each trial’s voxel pattern (for each of the Delay and Execute epochs) and used these single features as inputs for pattern classification. The results of this analysis demonstrate that, during the Delay epoch, the statistically significant decoding reported above is overwhelmingly the result of the distributed information contained in the multivariate voxel patterns rather than the information contained in the mean univariate signal across the pattern (see Supplementary Fig. 4).

Experiment 2

Despite the absence of univariate effects in driving our Experiment 1 results, those findings could still be interpreted as being generally consistent with an attention-based enhancement of the target object (Tootell et al. 1998; Noesselt et al. 2002; Ress and Heeger 2003), wherein feedback from higher-order areas improves the processing of the stimulus’ location in retinotopic cortex. Indeed, such feedback-related attentional modulation may explain recent observations of changes in visual cortex activity during the preparation of pointing vs. grasping movements (Gutteling et al. 2015). However, our hypothesis that these top-down projections carry motor-specific information predicts that this modulation should be directly reflective of movement-related parameters, such as the motor effector to be used in an upcoming action (Snyder et al. 1997; Quian Quiroga et al. 2006). To directly test this idea, in Experiment 2, we scanned eleven subjects while, in any given trial, they performed one of four delayed object-directed hand movements towards a centrally located target object: (1) left hand grasp, (grasp left), (2) right hand grasp (grasp right), (3) left hand reach (reach left), or (4) right hand reach (reach right; see Fig. 3A,B). Using the same object location retinotopic mapping procedures as in Experiment 1, at the end of the Experiment 2 testing session we localized sub-regions in early visual cortex corresponding to the retinotopic location of the target object (task-relevant location), as well as a location not actually acted upon during the task (task-irrelevant location) in order to provide a control region for analyses. In addition, we independently mapped each participant’s foveal confluence, allowing us to assess any systematic effect of confounding eye movements (as in Experiment 1).

Delay Period Decoding Is Linked to the Task-Relevant Location

As in Experiment 1, an analysis on classification accuracies revealed that decoding of upcoming actions (i.e., during the Delay epoch) was linked to the sub-region of V1 corresponding to the target object location (task-relevant location, Fig. 3C) and did not extend into the sub-region of V1 corresponding to the control object location (task-irrelevant location, Fig. 3D). [For a similar pattern of results in V2/V3, see Supplementary Fig. 5]. In particular, from the task-relevant retinotopic zone, we found that we could not only decode the motor effector to be used in the upcoming action (grasp left vs. grasp right, t₁₀ = 2.979, P = 0.014; reach left vs. reach right, t₁₀ = 8.800, P < 0.001) but also the hand action to be performed using the same effector (grasp left vs. reach left, t₁₀ = 5.207, P < 0.001; grasp right vs. reach right, t₁₀ = 4.209, P = 0.002). Notably, here, as in Experiment 1, we also did not observe any significant decoding from parafoveal retinotopic cortex (Fig. 3E). This makes it unlikely that the present results can be attributed to differential eye movements across the conditions, and is consistent with our prior findings showing that participants had no difficulty maintaining stable fixation throughout this task (see Gallivan, McLean, Flanagan, et al. 2013). Finally, similar to Experiment 1, we also performed decoding on the mean univariate signal associated with each trial pattern and again found that the pairwise decoding reported above is primarily a result of the distributed information contained in the multivariate voxel pattern rather than in the mean signal (see Supplementary Fig. 7). Taken together, these findings suggest that, during planning, it is primarily the retinotopic representation of the target object’s spatial location that receives motor-related feedback modulation from higher-order cortical areas.

Delay Period Activity Robustly Encodes Effector-Related Information

To further elucidate the nature of this target-centric modulation in early visual cortex, we next tested for main effects of both motor effector (left vs. right hand) and action goal (grasp vs. reach) in the early visual activity patterns. To do this, we trained an SVM classifier by using one set of action trials and tested their accuracy in classifying a different set of action trials (i.e., cross-decoding, see Formisano et al. 2008; Harrison and Tong 2009; Gallivan, McLean, Smith, et al. 2011; Gallivan, McLean, Flanagan, et al. 2013). Specifically, to test for a main-effect of limb (i.e., effector-specific, action-independent encoding), we trained a classifier on grasp left vs. grasp right trials and then tested it on reach left vs. reach right trials, and vice versa. Likewise, to test for a main-effect of movement goal (i.e., action-specific, effector-independent encoding), we trained a classifier on grasp left vs. reach left trials and tested it on grasp right vs. reach right trials, and vice versa (for specific details, see Fig. 3 caption).

Using this cross-decoding approach, we found that Delay epoch activity patterns in the task-relevant retinotopic zone of V1 showed significant coding for the effector to be used when the hand action changed (pink bar in Fig. 3C; t₁₀ = 2.491, P = 0.032) but not the hand action to be performed when the effector changed (cyan bar in Fig. 3C; t₁₀ = −1.015, P = 0.334). Examination of the confusion matrices associated with the multiclass discriminations further support this effector-specific nature of encoding. Specifically, Supplementary Fig. 6A shows that the task-relevant retinotopic zone of V1 exhibits a checkerboard-like pattern of misclassifications during the Delay epoch wherein grasp left trials were more likely to be misclassified as reach left trials than other trial types (and vicevice versa) whereas grasp right trials were more likely to be misclassified as reach right trials than other trial types (and vice versa). This pattern of effects may reflect the fact that the motor efference copy signals are likely to be more similar when the same motor effector (left vs. right hand) is to be used rather than when it differs. Critically, consistent with the within-trial decoding results reported above, we did not find significant cross-decoding during the Delay epoch in the task-irrelevant control object location (see Fig. 3D, for similar results in V2/V3, see Supplementary Fig. 5).

Summary of Experiment 2 Findings

Taken together, the ROI results of Experiment 2 show that representations of the target object location in early visual cortex are modulated in a top-down manner based on fundamental motor features of the task (effector used) and are subtlety tuned based on the precise kinematic details of the actions to be performed (whether the fingers require preshaping for object contact or not; i.e., grasping vs. reaching).

Experiment 3

To further establish the Experiment 2 finding that early visual cortex encodes motor effector information during planning, and that this motor information is linked to the target object’s representation in visual cortex, in the third, and final, experiment we adapted a task from primate neurophysiology frequently used to dissociate motor versus sensory coding in parietal cortex (Snyder et al. 1997, 2006; Quian Quiroga et al. 2006; Cui and Andersen 2007). In our modified version, participants (N = 11) first prepared, and then executed, one of four movements: (1) left object-directed eye movement; look left, (2) right object-directed eye movement; look right, (3) left object-directed grasp; grasp left, or (4) right object-directed grasp; grasp right (see Fig. 4A,B). Using the same retinotopic mapping procedures as in Experiments 1 and 2, at the end of experimental testing, we again mapped visual cortex locations corresponding to the target objects (bottom left and right, Fig. 4C) and control, non-target object (top left and right, Fig. 4D) locations (i.e., task-relevant and –irrelevant locations, respectively). In addition, in a separate testing session, we again mapped each subject’s foveal confluence.

Delay Period Decoding Is Linked to the Task-Relevant Locations

As in Experiments 1 and 2, an analysis on classification accuracies revealed that decoding of upcoming actions (i.e., during the Delay epoch) was linked to the sub-region of V1 corresponding to the task-relevant, but not task-irrelevant, locations (compare Fig. 4C,D) (for a similar pattern of results in V2/V3, see Supplementary Fig. 8). Specifically, we found that from the V1 retinotopic representation of the left target object we could decode only those sets of upcoming actions, one of which was to be performed, towards that particular spatial location (look left vs. look right, t₁₀ = 5.757, P < 0.001; grasp left vs. grasp right, t₁₀ = 4.179, P = 0.002; look left vs. grasp left, t₁₀ = 7.553, P < 0.001; but not look right vs. grasp right, t₁₀ = -0.333, P = 0.746), and the same, but opposite, was true for the V1 retinotopic representation of the right target object (look left vs. look right, t₁₀ = 7.003, P < 0.001; grasp left vs. grasp right, t₁₀ = 5.019, P < 0.001; look right vs. grasp right, t₁₀ = 6.569, P < 0.001; but not look left vs. grasp left, t₁₀ = 1.758, P = 0.109). Further control analyses revealed that this pattern of effects was primarily driven by the multivariate voxel patterns rather than mean univariate signal associated with each ROI (see Supplementary Fig. 10). Together, these results demonstrate both motor and spatial (i.e., directional) specificity in the feedback-related modulations of V1.

Delay Period Activity Multiplexes Both Spatial- and Effector-Related Information

We also further assessed, as in Experiment 2, each of the main effects in the V1 neural patterns through cross-decoding analyses (in this case, the motor effector and direction of movement). We observed significant cross-decoding for the spatial location to be acted upon (when the effector changed) but not the effector to be used (when the target location changed; cyan and pink bars in Fig. 4C, respectively) for each of the left (t₁₀ = 3.165, P = 0.010; t₁₀ = −0.414, P = 0.687, respectively) and right (t₁₀ = 3.282, P = 0.008; t₁₀ = 0.835, P = 0.423, respectively) target object locations. This result suggests that Delay epoch response patterns in each of these V1 sub-regions were similarly modulated by the upcoming spatial direction of the movement (left vs. right), with some invariance for the actual effector to be used at that location (eye vs. hand). A similar effect can be observed in the confusion matrices associated with the multiclass discriminations. For instance, Supplementary Fig. 9A shows that, for each of the bottom left and bottom right sub-regions of V1, look left trials were more likely to be misclassified as grasp left trials than other trial types (and vice versa) whereas look right trials were more likely to be misclassified as grasp right trials than other trial types (and vice versa). Such direction-specific, effector-independent responses are consistent with a basic “receptive field”-type encoding, whereby neural responses are primarily modulated by whether a movement will be made towards (vs. away from) a given spatial location. (Note that the failure to observe significant effector-specific, direction-independent cross-decoding in Fig. 4C should not be unexpected given the failure to decode upcoming eye vs. hand movements when they were to be made into the ipsilateral visual field.) Notably, significant cross-decoding was never observed during the Delay epoch in any of the task-irrelevant ROIs (Fig. 4D).

Delay Period Activity in Parafoveal Cortex Encodes the Locations of Future Eye Movements

Lastly, we investigated neural decoding in parafoveal retinotopic cortex, which recall had served in the previous Experiments (1 and 2) as a control region to rule out effects of eye movements on our data. Consistent with this, here, as in Experiments 1 and 2, we did not observe decoding from this region for upcoming trials involving hand movements (i.e., grasp left or grasp right trials). Strikingly, however, we now found significant decoding from this region for trials involving upcoming eye movements (left parafovea: look left vs. look right, t₁₀ = 6.401, P < 0.001; look left vs. grasp left, t₁₀ = 7.529, P < 0.001; look right vs. grasp right, t₁₀ = 3.378, P = 0.007; but not grasp left vs. grasp right, t₁₀ = 1.258, P = 0.237; right parafovea: look left vs. look right, t₁₀ = 4.160, P = 0.002; look left vs. grasp left, t₁₀ = 4.900, P = 0.001; look right vs. grasp right, t₁₀ = 9.280, P < 0.001; but not grasp left vs. grasp right, t₁₀ = 0.091, P = 0.929). This pattern of decoding from retinotopic parafoveal cortex is consistent with recent fMRI work showing an oculomotor remapping of visual information to foveal retinotopic cortex (Knapen et al. 2016), as well as a great wealth of neurophysiological evidence demonstrating a modulation of neural responses, immediately prior to a saccadic eye movement, at the future, post-saccadic receptive field location (i.e., the new location to be stimulated by the object following the eye movement, Wurtz 2008; though see Zirnsak and Moore 2014). Our observation may also relate to the finding, from perceptual discrimination tasks, that stimulus information presented to the visual periphery can be selectively fed back to foveal retinotopic cortex (Williams et al. 2008).

Notably, further investigation with cross-decoding methods of the parafoveal activity patterns revealed an opposite effect to that observed in the sub-regions of V1 corresponding to the target locations: We now found significant cross-decoding for the effector to be used (when the location changed) but not the spatial location to be acted upon (when the effector changed; pink and cyan bars in Fig. 4E, respectively) in both left (t₁₀ = 3.068, P = 0.012; t₁₀ = 0.779, P = 0.454, respectively) and right (t₁₀ = 3.747, P = 0.004; t₁₀ = −0.634, P = 0.541, respectively) parafoveal retinotopic cortex. Such effector-specific, direction-independent pattern responses is consistent with the selective representation of information related to prepared eye, but not hand, movements in this region, and suggests some degree of invariance in the activity patterns with respect to the final position of the target object (i.e., left vs. right target location). With respect to the latter, however, our results show that these position-invariant signals in parafoveal cortex are intermixed with direction-related signals, as we were able to decode the spatial direction of the upcoming eye movement (left vs. right object) through our pairwise classifications (see Fig. 4E).

Summary of Experiment 3 Findings

To summarize, the ROI results in Experiment 3 not only replicates (1) the target-centric nature of the action-related modulations described in Experiments 1–2 (i.e., decoding being primarily linked to the zones of retinotopic cortex that spatially represent the upcoming targets of action) and (2) the top-down motor-specificity described in Experiment 2 (i.e., feedback of movement effector information), but it also (3) clarifies the task conditions that can bring about pre-movement neural coding in retinotopic parafoveal cortex (i.e., the preparation of eye but not hand movements).

Discussion

Across three separate fMRI experiments we have documented the different ways in which movement preparation modulates neural representations in early visual cortex via top-down projections. First, we have shown that, for upcoming hand movements, feedback-related modulations of delay period activity in early visual cortex are largely target-centric. That is, we demonstrate through our ROI-based analyses that an impending hand action selectively modulates activity patterns corresponding to the retinotopic representation of the target object’s location (i.e., task-relevant location), and not at other retinotopic zones irrelevant to the action (i.e., task-irrelevant locations). Second, we find that this target-centric modulation is both effector-specific and motor goal-dependent. That is, from activity patterns corresponding to the retinotopic representation of the target object we are able to decode both the motor effector to be used (i.e., left vs. right hand, in Experiment 2, and arm vs. eye, in Experiment 3) and the goal of the action to be performed (i.e., whether the object will be involved in a sequential action, in Experiment 1, or grasped versus reached, in Experiment 2). Third, as a departure from that observed for hand movements, we find that, for impending saccadic eye movements, parafoveal retinotopic cortex contains target-related information about saccades to be performed outside of the fovea, in the visual periphery. This coding of saccade, but not hand, movement-related information in a different cortical region (parafovea) from the retinotopic representation of the actual target object may reflect an anticipatory updating from the pre-saccadic to future, post-saccadic location (e.g., Duhamel et al. 1992; Tolias et al. 2001; Nakamura and Colby 2002; Merriam et al. 2007; Knapen et al. 2016) and/or covert shift of attention to the impending saccade target’s location (e.g., Tolias et al. 2001; Zirnsak et al. 2014; Neupane et al. 2016). Taken together, the current set of findings reveal how movement preparation shapes neural response patterns at the earliest levels of visual cortical processing (see also Steinmetz and Moore 2014; Gutteling et al. 2015) and provide a unique characterization of this feedback modulation in terms of basic movement-related parameters, such as the effector to be used and the motor goal to be performed in the upcoming action.

Given the organization of feedback connections in visual cortex (Felleman and Essen 1991) and visual-perceptual models concerning its architecture (Gilbert and Li 2013; Muckli and Petro 2013), our finding that movement preparation primarily modulated retinotopic representations of the target object(s) is noteworthy. At the neuroanatomical level, there exists a direct overlap between attention-based topography and stimulus-based retinotopy in visual cortex (Tootell et al. 1998; Noesselt et al. 2002), and the target-centric modulations described here are consistent with direct structural connectivity (Greenberg et al. 2012) between attention-based topographic maps in the intraparietal sulcus (Sereno et al. 2001; Silver et al. 2005) and corresponding retinotopic maps in visual cortex (see Wandell et al. 2007 for review). Such feedback connections, according to most models, aid perceptual processing by providing visual cortex with high-level contextual information (Smith and Muckli 2010; Muckli et al. 2015) and by enhancing or anticipating incoming retinal information (Ress and Heeger 2003; Muckli and Petro 2013). With regards to sensorimotor processing, psychophysics studies support the idea that motor-relevant features of a target object (e.g., its location, orientation) are enhanced via spatial attention mechanisms during movement preparation in accordance with precisely how the object will be manipulated (e.g., Baldauf et al. 2006; Gutteling et al. 2011). At the neural level, this attentional enhancement could explain why, in recent work, prepared but not yet executed, grasping vs. pointing actions can be decoded from visual cortex (Gutteling et al. 2015) or why, prior to object grasping, activations in visual cortex are modulated by the degree of interference from an adjacent obstacle (Chapman et al. 2011). It could also explain why, in our Experiment 2 for instance, we were able to decode, from retinotopic representations of the target object, the grasp vs. reach actions to be performed. Grasping requires the extraction (and enhancement) of object features relevant for hand preshaping and finger placement (Brouwer et al. 2009) in a way that simple reaching (or pointing, as in (Gutteling et al. 2015)) does not.

An attention-based account may also partly explain why, for a single spatial target location, we were able to decode information related to (1) the upcoming hand to be used (left vs. right limb; Experiment 2) and (2) the eye versus hand (Experiment 3). Indeed, in the case of the former (Experiment 2), approaching the object with one hand versus the other likely requires a slightly different allocation of attentional resources to the different sides of the target object (despite the exact same thumb placement and thus, precision, being required for both hands). Likewise, in the case of the latter (Experiment 3), while the attentional enhancement of the target’s spatial location is presumably equal across the eye and hand (Andersen and Buneo 2002), the precision requirements of each movement may involve subtly different kinds of retinal and extra-retinal processing about the object. However, we would argue that such differences in task-related attentional requirements emerge only as a byproduct of motor planning. That is, “spatial attention” in our task cannot be construed as some sort of abstract cognitive resource that is separable from motor planning, but rather is part and parcel to the movement planning and control process itself (e.g., providing visual information about target locations and detecting errors). Indeed, it has been argued that using “attention” as a key explanatory factor in all modulations of neural activity (visual or otherwise) has the effect of ignoring its specificity and serves only to diminish its usefulness as a construct (see Andersen and Buneo 2002; Andersen and Cui 2009).

Previously, the presence of motor effector-related information in neural population activity (e.g., in parietal cortex) has often been taken as evidence that the brain region in question performs some type of sensorimotor computation during movement planning; that is, that such an area codes an action “intention” or early plan for movement (see Snyder et al. 1997, though this has been a matter of robust debate, see Bisley and Goldberg 2003). Consistent with this interpretation, we and others have reported considerable evidence demonstrating the coding of planned movements in a variety of conventional “sensorimotor” regions throughout human parietal and frontal cortex (Gallivan, McLean, Smith, et al. 2011; Gallivan, McLean, Valyear, et al. 2011; Ariani et al. 2015, 2018; Tucciarelli et al. 2015; Wurm and Lingnau 2015; Turella et al. 2016; Gertz et al. 2017). However, unlike areas in posterior parietal cortex (PPC), which lay directly at the interface of sensory- and motor-related processing and where intention-related signals have been previously characterized (e.g., Snyder et al. 1997), early visual cortex is the primary entrance stage for the cortical processing of visual input. Thus, it is unreasonable to interpret the present set of findings in this same light, or apply similar interpretations to our other findings demonstrating the decoding of action-related information from conventional higher-order “non-sensorimotor” areas in occipitotemporal cortex (Gallivan, Chapman, et al. 2013; Gallivan et al. 2014, 2016).

Instead, one possibility, given that PPC sends top-down structural projections to early visual cortex (Borra and Rockland 2011; Greenberg et al. 2012), is that the present results reflect some type of modulation related to sensory prediction. In the visual-perceptual literature, such prediction is often likened to a form of “visual” or “mental” imagery, wherein participants imagine visual stimuli and/or events. In conventional visual imagery tasks, however, imagery is either encouraged or wholly constitutes the task itself (i.e., imagery is the experimental manipulation) (for a recent meta-analysis and review of this material, see Winlove et al. 2018). This is very different from the current set of sensorimotor tasks wherein participants are only instructed about what movements to perform. Indeed, when we directly tested a visual imagery account of our findings in Experiment 1, we observed no evidence that individuals were actually reinstating, during the Delay epoch, the neural activation patterns that would subsequently manifest during the Execute epoch. This would suggest that the observed feedback-related modulations of visual cortex activity in our tasks are likely occurring at a more automatic, implicit level.

One such potential mechanism is through efference copy signals related to upcoming motor commands. If shared with the early visual system, these efference copy signals would allow it to anticipate incoming retinal information (Haarmeier et al. 1997; Gallivan and Culham 2015; Hutchison and Gallivan 2018). At a perceptual level, such prediction would allow our early visual system to immediately disentangle movements of one’s own body from movements of the world (von Helmholtz 1866; Holst and Mittelstaedt 1950; Haarmeier et al. 2001). Moreover, at a motor level, given the delay of incoming visual (and other sensory) signals (Wolpert and Miall 1996; Desmurget and Grafton 2000; Franklin and Wolpert 2011; Gallivan et al. 2018), anticipating the sensory consequences of movement (i.e., via an internal forward model) is critical for rapid motor error detection and the subsequent correction of ongoing visually guided movements (Wolpert and Ghahramani 2000; Wolpert et al. 2011). As an example, for a simple sequential manipulation task like grasping an object, lifting it, holding it in air, and then replacing it (as in Experiment 1), different action phases are marked by discrete sensory events (e.g., object contact, lift off) that occur in both the visual and tactile modalities, and which signify subgoal attainment. By predicting these sensory events, the brain can monitor task progression and smoothly produce corrective actions if a mismatch between predicted and actual sensory events is detected (Flanagan et al. 2006; Johansson and Flanagan 2009). The PPC, which receives a convergence of sensory inputs from multiple modalities and which has been linked to forward model estimates of the sensory consequences of movement (Mulliken et al. 2008), is well situated to perform the reafference-canceling computations that allow disambiguation of self-induced versus external visual motion. However, the typical visual response latencies associated with PPC neurons (~90 ms, Mulliken et al. 2008) are considerably lagged to those of extrastriate cortex (~30–50 ms, Maunsell and Gibson 1992). Thus, there is presumably a temporal (and computational) advantage in initiating visual reafference-canceling mechanisms at even earlier levels of the visual processing cascade, before reaching PPC.

In summary, here we have shown, from delay period neural activity patterns, the existence of motor effector-related and movement goal-dependent information in early retinotopic visual cortex. While the encoding of task-dependent action information from visual cortex during planning is consistent with recent observations from human fMRI (Chapman et al. 2011; Gutteling et al. 2015), we are unaware of any reports demonstrating the encoding of diverse motor effector information. We submit that by relaying such information to the earliest cortical levels of visual processing this may allow the sensorimotor system to (1) enhance processing of the motor-relevant dimensions of the target object(s) during planning (e.g., object contour and grasp points) and (2) instantiate an internal estimate of the consequences of movement, thereby helping to overcome sensory processing delays (and aiding the internal forward model). These findings add to growing evidence (Muckli and Petro 2013) that visual cortex, rather than being a passive propagator of incoming sensory information, plays an important role in sustaining higher-level cognitive operations.

Funding

Canadian Institutes of Health Research (CIHR) (MOP126158 to J.R.F. and J.P.G. and MOP84293 to J.C.C.); a Natural Sciences and Engineering Research Council (NSERC) Banting Postdoctoral fellowship to J.P.G.; a CIHR postdoctoral fellowship to to J.P.G.; an Ontario Ministry of Research and Innovation Postdoctoral fellowship to J.P.G.; a Canadian Foundation for Innovation award to J.P.G.; a NSERC Discovery Grant to J.P.G.