Decoding Internally and Externally Driven Movement Plans

Abstract

During movement planning, brain activity within parietofrontal networks encodes information about upcoming actions that can be driven either externally (e.g., by a sensory cue) or internally (i.e., by a choice/decision). Here we used multivariate pattern analysis (MVPA) of fMRI data to distinguish between areas that represent (1) abstract movement plans that generalize across the way in which these were driven, (2) internally driven movement plans, or (3) externally driven movement plans. In a delayed-movement paradigm, human volunteers were asked to plan and execute three types of nonvisually guided right-handed reaching movements toward a central target object: using a precision grip, a power grip, or touching the object without hand preshaping. On separate blocks of trials, movements were either instructed via color cues (Instructed condition), or chosen by the participant (Free-Choice condition). Using ROI-based and whole-brain searchlight-based MVPA, we found abstract representations of planned movements that generalize across the way these movements are selected (internally vs externally driven) in parietal cortex, dorsal premotor cortex, and primary motor cortex contralateral to the acting hand. In addition, we revealed representations specific for internally driven movement plans in contralateral ventral premotor cortex, dorsolateral prefrontal cortex, supramarginal gyrus, and in ipsilateral posterior parietotemporal regions, suggesting that these regions are recruited during movement selection. Finally, we observed representations of externally driven movement plans in bilateral supplementary motor cortex and a similar trend in presupplementary motor cortex, suggesting a role in stimulus–response mapping.

SIGNIFICANCE STATEMENT The way the human brain prepares the body for action constitutes an essential part of our ability to interact with our environment. Previous studies demonstrated that patterns of neuronal activity encode upcoming movements. Here we used multivariate pattern analysis of human fMRI data to distinguish between brain regions containing movement plans for instructed (externally driven) movements, areas involved in movement selection (internally driven), and areas containing abstract movement plans that are invariant to the way these were generated (i.e., that generalize across externally and internally driven movement plans). Our findings extend our understanding of the neural basis of movement planning and have the potential to contribute to the development of brain-controlled neural prosthetic devices.

Introduction

In daily life, we continuously select which movements to plan and execute. Parietofrontal regions have been implicated in the planning, execution, and online control of eye and hand movements in a number of human (Binkofski et al., 1999; Connolly et al., 2002; Tunik et al., 2005; Beurze et al., 2009; Cavina-Pratesi et al., 2010; Filimon, 2010; Gallivan et al., 2011a, 2011b, 2013a; Glover et al., 2012; Barany et al., 2014; Brandi et al., 2014; Fabbri et al., 2014; Leoné et al., 2014; Gallivan and Culham, 2015) and monkey (Hoshi and Tanji, 2006; Andersen and Cui, 2009; Fattori et al., 2010; Afshar et al., 2011; Townsend et al., 2011; Lehmann and Scherberger, 2013) studies. Furthermore, premovement activity in both parietal and frontal regions has been shown to encode different hand configurations (Murata et al., 2000; Raos et al., 2004, 2006; Begliomini et al., 2007; Tunik et al., 2007; Fluet et al., 2010; Gallivan et al., 2011a; Verhagen et al., 2013).

Movements can be planned either on the basis of external cues in our environment (externally driven) or in the absence of such cues (internally driven). Although it has been reported that the same parietofrontal areas involved during externally driven movements are recruited during internally driven movements in monkeys (Cisek and Kalaska, 2005, 2010; Cui and Andersen, 2007; Pesaran et al., 2008), no previous study directly compared the planning of internally and externally driven movements in humans. Studies that compared externally and internally driven movements did not intend to separate movement planning from execution (Oliveira et al., 2010; Zhang et al., 2012; Bode et al., 2013). By contrast, studies separating between planning and execution focused on externally driven movements and thus did not allow distinguishing between internally and externally driven movements (Beurze et al., 2009; Gallivan et al., 2011a, 2011b, 2013a; Pertzov et al., 2011; Bernier et al., 2012).

Here we aimed to distinguish between brain regions representing abstract movement plans that are neither tied to specific external cues nor to internally driven decisions, and brain regions representing movement plans specific for internally driven or externally driven movements (Fig. 1A). We asked participants to perform a delayed-movement task in which they had to plan and execute one of three different movements (i.e., reach to grasp with a precision grip, with a power grip, or reach to touch) toward a single centrally located object (Fig. 1B). On each trial, a visual cue either instructed to plan a specific movement as instructed by the cue (Instructed condition, i.e., externally driven), or it indicated to select and plan one of the three movements (Free-Choice condition, i.e., internally driven; Fig. 1C). We used support-vector-machine (SVM)-based multivariate pattern analysis (MVPA) of fMRI data to compare the decoding of upcoming externally and internally driven movements. To examine abstract representations of movement plans that generalize across the planning conditions, we used cross-condition classification (i.e., training a classifier to distinguish between upcoming movements on the basis of externally driven trials) and testing on internally driven trials, and vice versa.

Figure 1.

Experimental question, design, timing, and setup. A, Schematic representation of the research question: is it possible to distinguish between areas representing externally triggered (instructed) movement plans (red), internally triggered (freely chosen) movement plans (blue), and abstract movement plans that are invariant to the way these movement plans are generated (purple)? B, A 2 × 3 mixed factorial design: Planning condition (Instructed, Free-Choice), blocked, and Movement type (precision grip, PRG: two fingers only, index and thumb; power grip, PWG: whole hand open; touch, TCH: hand closed in a fist, without hand preshaping), randomized. C, Example trial with timing (Instructed block, PRG). Each trial began with participants fixating a dot (Baseline) for a variable amount of time randomly selected from a geometric distribution (p = 0.3, 2000–6000 ms). This interval was followed by a color fixation cross (500 ms) either instructing which movement to plan (Instructed blocks), or indicating to freely select one of the movements (Free-Choice blocks). The Planning phase consisted of a jittered interstimulus interval (independently chosen from the same geometric distribution). After this delay, an auditory cue (100 ms) provided the GO-signal to start the movement (Execution phase, 2500 ms). In the Instructed condition, the color of the fixation cross corresponded to one of the three movements. In the Free-Choice condition, the cue always had the same, noninformative, color (in this example, blue). D, Lateral view of a participant with the right hand at the home position. The central wooden target object on which the reach-to-grasp movements were performed was mounted on a Plexiglas workspace positioned above the waist of the participant. The size of the small and large wooden cuboids were 2 × 2 × 1 and 7 × 7 × 2 cm, respectively. Participants saw the screen through a mirror attached to the head coil (line of sight illustrated by black dashed line). This setup ensured that participants saw neither the target object nor their own movements.

We reasoned that areas containing abstract movement plans should show movement selectivity that generalize across the planning condition. By contrast, areas involved in action selection should show movement selectivity in the Free-Choice but not in the Instructed condition. Finally, areas involved in the processing of sensory cues and/or the mapping between such cues and the corresponding movements should show movement selectivity in the Instructed, but not in the Free-Choice condition.

Materials and Methods

Participants.

Twenty-five right-handed volunteers (12 males, 13 females; mean age: 27.2 years; age range: 21–54 years) took part in the study. All participants were neurologically intact and had either normal or corrected-to-normal vision. The experimental procedures were approved by the ethics committee at the University of Trento. Participants gave written informed consent and were paid for their participation. Seven participants were subsequently excluded from data analysis: one due to technical problems with video recordings (see Setup), one due to not completing the experimental session, and five due to severe head motion. Rapid (i.e., taking place within one volume) head motion was detected on the basis of the three translation and rotation parameters resulting from 3D motion correction (cutoff criterion: >1 mm for translation, >1 degree for rotation). Overall, 18 participants were included in the successive analyses.

Setup.

Visual stimuli (i.e., fixation cross and fixation dot) were back-projected onto a screen (frame rate: 60 Hz; screen resolution: 1024 × 768 pixels; mean luminance: 109 cd/m²) via a liquid crystal projector (OC EMP 7900, Epson Nagano). Participants viewed the screen binocularly through a mirror mounted on the head coil (Fig. 1D). The screen was visible as a rectangular aperture of 17.8° × 13.4°. The auditory go-signal was delivered via MR-compatible headphones.

Participants performed unimanual (right hand only) reach-to-grasp movements (Fig. 1B) toward a single, centrally located object (according to each participant's sagittal midline) mounted on top of a workspace that consisted of a transparent Plexiglas board attached to the scanner bed above the waist of the participant (Fig. 1D). The target object consisted of two custom-made square pieces of wood, glued on top of each other (Fig. 1D). To exclude uncontrolled visual stimulation by the sight of the own hands and the object, or systematic eye movements toward the object, participants were scanned in a conventional fMRI configuration (i.e., horizontally, without tilting the head toward the body; Fig. 1D) and were instructed to maintain fixation throughout the experiment. This precluded direct viewing of their own limbs, or the target object, while performing the task without visual feedback.

An MR-compatible response button (Lumina LP 400, Cambridge Research Systems), attached to a custom belt around the waist, was pressed by the participant with the knuckles when at rest (home position, Fig. 1D). A microcontroller board (Arduino Uno) connected to the Lumina Controller positioned outside the magnet room was used to signal the release of that button. This time stamp was used to measure movement onset time.

To enable movements as comfortable as possible, the position of the workspace and the response button were adjusted individually to match each participant's arm length (mean distance hand-object: 16.6 cm). Head and trunk movements were minimized by stabilizing the head and the upper right arm with foam blocks and cushions.

To monitor movement execution, we recorded each experimental session using an MR-compatible digital video camera (VP-D15i; Samsung Electronics) mounted on a tripod in a corner of the MR room (outside the 0.5 mT line). Stimulus presentation, response collection, and synchronization with the scanner were controlled using “ASF” (Schwarzbach, 2011), based on the MATLAB (The MathWorks) Psychtoolbox-3 for Windows (Brainard, 1997).

Design.

We used a mixed design with the factors planning condition (Instructed, Free-Choice) and movement type (precision grip, PRG; power grip, PWG; touch, TCH; Fig. 1B). Planning condition was blocked; movement type was randomized within blocks. In Instructed blocks, each movement type occurred equally often (3 times), and the color of the fixation cross indicated which movement to perform. In Free-Choice blocks, participants were instructed to choose one of the three movement types with no restrictions.

Procedure.

To temporally isolate the neural processes associated with movement planning from movement execution, we used a delayed-movement paradigm (Gallivan et al., 2011a, 2011b, 2013a; Andersen and Buneo, 2002; Beurze et al., 2009) (Fig. 1C). Each trial started with a gray fixation dot lasting for a variable amount of time that served to alert participants of the upcoming trial. The duration of the fixation dot was chosen from a geometric distribution (p = 0.3; 2000–6000 ms, in steps of 500 ms). The fixation dot was followed by a colored fixation cross for 500 ms, either instructing the type of movement to perform (Instructed condition), or indicating to select one of the movements (Free-Choice condition). The colored fixation cross was followed by a jittered interstimulus interval (Planning phase) independently chosen from a geometric distribution with the same parameters as described above. At the end of the delay period, an auditory signal (duration: 100 ms, frequency: 350 Hz, amplitude: 0.6) provided the GO-cue to start the movement (Execution phase, 2500 ms), and to return to the home position after completion of the movement. Participants were asked to keep the hand still and relaxed in the home position throughout all the phases of the trial apart from the Execution phase. Reaction times were defined as the time when the response button was released time-locked to the GO-cue.

While in the Instructed condition, different color cues corresponded to different movement types, the cue always had the same, noninformative, color in the Free-Choice condition. We used two sets of color-cue assignments that were balanced across participants. Each participant completed a single experimental session consisting of a practice session outside the scanner (∼20 min), the structural scan (∼5 min), and 10 functional runs (∼6 min each). Each functional run started and ended with 15 s rest and contained 4 blocks of trials (2 blocks per planning condition) separated by 15 s rest each. Between the second and the third block, a longer rest period (25 s) allowed participants to relax their right arm, wrist, and hand. The order of block types (I = Instructed; F = Free-Choice) was pseudo-randomized such that the first two (or second two) blocks could never be of the same type (i.e., IFIF, FIFI, IFFI, or FIIF). Each block (∼60 s) consisted of 9 trials, for a total of 360 trials per participant. For the Instructed condition, after excluding error trials, we had an average of 58.70 (range: 50–60) repetitions per movement type and planning condition per participant. For the Free-Choice condition, the number of trials per movement type depended on the choices of the participant, with an average of 59.68 (range: 35–81) repetitions per condition per participant (for further details, see Multivariate pattern classification analysis).

Data acquisition.

Functional and structural data were collected using a 4T Bruker MedSpec Biospin MR scanner and an 8 channel birdcage head coil. Functional images were acquired with a T2*-weighted gradient-recalled EPI sequence. Acquisition parameters were a TR of 2000 ms; voxel resolution, 3 × 3 × 3 mm; TE of 33 ms; flip angle (FA), 73°; FOV, 192 × 192 mm; gap size, 0.45 mm. We used 28 slices, acquired in ascending interleaved order, slightly tilted to run approximately parallel to the calcarine sulcus. The number of volumes acquired in the main experiment for each functional run varied according to the length of variable delay periods (range: 178–183 volumes). Before each functional run, we performed an additional scan to measure the point-spread function of the acquired sequence, which served for distortion correction, expected with high-field imaging (Zaitsev et al., 2004). To be able to coregister the low-resolution functional images to a high-resolution anatomical scan, we acquired a T1-weighted anatomical scan (magnetization-prepared rapid-acquisition gradient echo; TR: 2700 ms; voxel resolution: 1 × 1 × 1 mm; TE: 4.18 ms; FA: 7°; FOV: 256 × 224 mm; 176 slices; generalized autocalibrating partially parallel acquisition with an acceleration factor of 2; inversion time: 1020 ms).

Data analysis

Behavioral analyses.

We measured reaction time (RT) as the time to release the response button (see Procedure) with respect to the auditory GO-cue. Moreover, we analyzed video recordings of the experimental sessions to ensure that participants performed the movements correctly, and to know which movements were performed during the Free-Choice condition. Trials were considered errors either when performed incorrectly (i.e., incorrect hand preshaping; temporal anticipation: RT < 100 ms; reaction time timeout: RT > 1500 ms) or, in the Instructed condition only, when participants executed a movement that was different from the one instructed by the cue. Using the videos, we also counted the number of correct trials per movement type, of particular importance for the Free-Choice condition. Next, to potentially detect participants that showed stereotyped selections (i.e., cognitive strategies) or excessively frequent movement choices, we created a transition matrix that showed the number of times each movement followed any other (3 × 3 matrix, trial_n × trial_n + 1). This allowed us to calculate a measure of randomness (i.e., entropy) for movement selection in Free-Choice trials (separately per participant and run), the Shannon's Entropy (Uncertainty) Index (Shannon, 1948) as follows:

where X is a random variable with n outcomes {x₁, …, x_n}, and p(x_i) is the probability mass function of the outcome x_i. Shannon's Entropy index (H) ranges from 0 to log₂n, where n is the number of states or possible outcomes.

fMRI data analysis

Preprocessing.

Data were preprocessed and analyzed using BrainVoyager QX 2.8.0 (BrainInnovation) in combination with the BVQX Toolbox and custom software written in MATLAB R2012b (The MathWorks). To correct for distortions in geometry and intensity in the EPI images, we applied distortion correction on the basis of the point-spread function (see Data acquisition) (Zeng and Constable, 2002). To avoid T1 saturation, we discarded the first 4 volumes. The first volume of the first functional run of each participant was aligned to the high-resolution anatomy (6 rigid-body transformation parameters). Next, we performed 3D motion correction (trilinear interpolation for estimation and sinc interpolation for resampling) using the first volume of the first run of each participant as reference, followed by slice timing correction (ascending interleaved even-odd order) and high-pass temporal filtering (3 cycles per run). Spatial smoothing was applied with a Gaussian kernel of 8 mm FWHM for univariate analysis only. For successive group analysis, both functional and anatomical data were transformed into a common Talairach space, using trilinear interpolation.

Univariate analysis (GLM).

To localize brain areas preferentially involved in movement preparation, we computed a group random-effects (RFX) GLM analysis in the volume. To avoid making assumptions about the shape of the HRF during the Planning phase, we used a deconvolution analysis, estimating the amplitude of the BOLD signal separately for each predictor and time point (TR). We created six (2 planning conditions × 3 movement types) predictors both for the Planning and Execution phases, and 1 predictor modeling the baseline between the first and second half of each run, leading to 13 (predictors) × 8 (time points) = 104 predictors. This led to independent estimates of the BOLD amplitude for each condition and time point resulting from the deconvolution analysis. Parameters from 3D motion correction (translation and rotation) and regressors for error trials (modeled separately for each time point) were also included in the model as predictors of no interest. For each voxel, the average of the estimated β-value at the third and fourth time points (i.e., 4–8 s after the onset of the planning cue) was used both for univariate and multivariate analyses (for a similar procedure, see Eisenberg et al., 2010).

We aimed to identify ROIs commonly reported to be involved in the planning and execution of prehension movements (Beurze et al., 2009; Gallivan et al., 2011a, 2011b, 2013a; Fabbri et al., 2014; for a review see Turella and Lingnau, 2014). To do so, we contrasted the Planning phase against the Baseline [Planning > Baseline] (Fig. 2), collapsing across the two planning conditions. The resulting volumetric statistical map was corrected for multiple comparisons using a false discovery rate (FDR) < 0.05 and projected on the group-averaged surface mesh for visualization (Fig. 2A).

Figure 2.

Univariate RFX-GLM analysis. A, The univariate contrast [Planning > Baseline] (collapsing across planning conditions) was used to identify ROIs preferentially involved in movement planning. The resulting statistical RFX group map (N = 18) was corrected for multiple comparisons using a FDR q < 0.05 and projected on the group-averaged inflated surface mesh for visualization. Individual ROIs were defined as spheres (8 mm radius) around individual peak voxels resulting from single-subject statistical maps. Black circles represent an example of the individual spherical ROIs (for additional details, see Materials and Methods; Table 1). B, Univariate contrast [Planning > Baseline], separately for each Planning condition (red represents [Planning Instructed > Baseline]; blue represents [Planning Free-Choice > Baseline]), projected on the same group-averaged inflated surface mesh. Purple areas represent the overlap between the two statistical group maps.

ROI definition.

To identify individual ROIs objectively, we followed a similar procedure as recently used by Oosterhof et al. (2012a). In brief, we first manually outlined the activations individuated through the RFX-GLM contrast [Planning > Baseline] on the group-averaged surface mesh (for details on the creation of the group-averaged surface mesh, see Brain segmentation, mesh reconstruction, and cortex-based alignment), approximately circumscribing the ROIs around known anatomical landmarks (see also Gallivan et al., 2011a, 2011b, 2013a). Specifically, we used the following criteria:

• Primary motor cortex (M1)—around the hand-knob area in the anterior bank of the central sulcus;

• Dorsal premotor cortex (PMd)—at the junction of the superior frontal sulcus and the precentral sulcus;

• Ventral premotor cortex (PMv)—slightly inferior and posterior to the junction of the inferior frontal sulcus and the precentral sulcus;

• Anterior intraparietal sulcus (aIPS)—on the anterior segment of the intraparietal sulcus, at the junction with the postcentral sulcus;

• Middle intraparietal sulcus (mIPS)—on the middle segment of the intraparietal sulcus, not overlapping with aIPS;

• Posterior intraparietal sulcus (pIPS)—on the posterior segment of the intraparietal sulcus, not overlapping with mIPS;

• Superior parietal lobule (SPL)—the anterior portion of the superior parietal lobule, superior to the IPS and slightly posterior to the postcentral sulcus;

• Supramarginal gyrus (SMG)—the anterior portion of the supramarginal gyrus, slightly posterior to the postcentral sulcus and superior to the lateral sulcus;

• Dorsolateral prefrontal cortex (dlPFC)—on the anterior portion of the middle frontal gyrus, around Brodmann area 46 (Badre and D'Esposito, 2009);

• Supplementary motor area (SMA)—on the medial wall of the superior frontal gyrus, anterior to the medial end of the central sulcus, posterior to the vertical projection of the anterior commissure;

• Presupplementary motor area (preSMA)—on the anterior segment of the cingulate sulcus, slightly anterior to the vertical projection of the anterior commissure;

• Posterior superior temporal gyrus (pSTG)—the posterior portion of the superior temporal gyrus, inferior to the supramarginal gyrus;

• Posterior middle temporal gyrus (pMTG)—the posterior portion of the middle temporal gyrus.

Next, we projected these marked activation patches from the surface back to the volume. Within each of them, we looked for individual peak voxels coming from the single-subject GLM contrasts [Planning > Baseline], computed as described above. We defined individual ROIs, separately for each participant, as spheres (8 mm radius) centered around each individual peak voxel (for a summary of the Talairach coordinates of individual ROIs, see Table 1). To examine classification performance in regions that are not expected to show predictive power, we additionally included a non-brain control ROI outside the skull of the brain near the right frontal cortex (same size and shape as before, and identical location for all participants).

Table 1.

Talairach coordinates (x, y, z rounded mean and SD across participants) of individual peak voxels for the ROIs identified by the group contrast [Planning > Baseline]^a

Region	x	y	z	SD x	SD y	SD z
L-M1	−33	−25	50	2.7	2.7	2.4
L-PMd	−25	−11	48	3.1	3.3	4.0
L-PMv	−46	3	27	4.5	2.3	5.1
L-aIPS	−39	−34	39	3.5	3.6	2.2
L-mIPS	−35	−45	40	2.7	3.5	2.1
L-pIPS	−30	−57	42	2.5	2.8	2.8
L-SPL	−31	−51	54	2.9	5.5	2.9
L-SMG	−56	−28	29	2.3	5.0	4.8
L-dlPFC	−36	34	28	3.4	3.3	2.8
L-SMA	−7	−3	50	1.5	2.6	4.4
L-preSMA	−8	4	41	1.7	3.6	2.5
R-pIPS	30	−50	42	2.3	3.3	2.5
R-pSTG	53	−39	13	3.9	2.7	3.0
R-pMTG	51	−51	4	3.4	5.2	3.7
R-SMA	6	−4	51	2.3	3.1	2.8
R-preSMA	7	7	39	1.6	3.3	2.3
Out of brain	51	53	56	0	0	0

^aL-M1, Left primary motor cortex; L-PMd, left dorsal premotor cortex; L-PMv, left ventral premotor cortex; L-aIPS, left anterior intraparietal sulcus; L-mIPS, left middle intraparietal sulcus; L-pIPS, left posterior intraparietal sulcus; L-SPL, left superior parietal lobule; L-SMG, left supramarginal gyrus; L-dlPFC, left dorsolateral prefrontal cortex; L-SMA, left supplementary motor area; L-preSMA, left presupplementary motor area; R-pIPS, right posterior intraparietal sulcus; R-pSTG, right posterior superior temporal gyrus; R-pMTG, right posterior middle temporal gyrus; R-SMA, right supplementary motor area; R-preSMA, right presupplementary motor area.

Multivariate pattern classification analysis

We ran both ROI- and searchlight-based MVPA using SVMs as implemented in LIBSVM (Chang and Lin, 2011). The ROI analysis served to test whether we could decode planned movements in the regions identified individually by the functional contrast [Planning > Baseline] as described above. In addition, to rule out that we missed potentially important regions in the ROI analysis, we performed a whole-brain surface-based searchlight analysis (Oosterhof et al., 2011) (see also Further observations). For the MVPA, we estimated β weights using the same design matrices as in the univariate analysis, except for the following: because participants freely selected which movements to plan and execute in the Free-Choice condition, the number of trials per movement type in this condition was not fully balanced. To prevent classification on the basis of the number of trials instead of the spatial patterns of brain activity, we balanced the number of trials per movement type in the Free-Choice and the Instructed condition by leveling to the minimum number of repetitions in either condition within each run, and discarding the trials in excess (randomly selected among the total). Beta maps containing the mean of the β estimates of the third and fourth time point for each predictor of interest (13, see Univariate analysis), individual spherical ROI (133 voxels), and run (10) were created for each participant. These maps were then z-transformed and normalized into multivoxel patterns of t values (β estimates divided by their SE) that we used as input for the classifier. This procedure resulted in 10 multivoxel patterns of t values per planning condition (one per experimental run). Classification accuracies were computed using a leave-one-run-out cross-validation method (i.e., the classifier was trained using data from 9 patterns and tested on the data from the remaining pattern). Although for the within-condition decoding all 10 patterns came from the same condition, the classifier was trained with 9 patterns from one planning condition (e.g., Free-Choice) and tested on one pattern from the other planning condition (e.g., Instructed) for the cross-condition decoding. Training and testing was repeated for 10 iterations, using all possible combinations of train and test patterns. The average across these 10 iterations constituted the mean decoding accuracy per participant and ROI.

To decode upcoming hand movements from preparatory brain activity patterns, multiple binary classifiers were trained to discriminate between two movements within each of the three possible pairs of movements (i.e., precision grip vs power grip, precision grip vs touch, and power grip vs touch) during the Planning phase, separately for the Instructed and the Free-Choice condition. Classification accuracies from the three binary classifiers were successively combined to produce an average accuracy per ROI.

To test for representations of planned movement types independent of the planning condition, we performed cross-condition decoding, i.e., training the classifier on discriminating movement pairs in one condition (e.g., precision grip vs power grip in the Instructed condition) and testing the performance of the classifier to distinguish between the same pair of movements in the other planning condition (e.g., precision grip vs power grip in the Free-Choice condition), and vice versa. As before, the mean of the three binary classifiers was computed to produce one accuracy score per ROI. Results from the two cross-condition decoding analyses (i.e., train on Instructed condition, test on Free-Choice condition, and vice versa) were also averaged. Finally, we performed the same within-condition decoding analysis described above for the Execution phase, but, given that no differences were expected after the movement had started, we collapsed across planning conditions.

To assess statistical significance of the decoding accuracy, we entered the individual (N = 18) classification accuracies (averaged across the three binary classifiers) into two-tailed one-sample t tests across participants against chance decoding (50%), separately for each ROI. Furthermore, to directly compare our main conditions of interest, we performed post hoc two-tailed paired-samples t tests between planning conditions for each ROI. Statistical results were corrected for multiple comparisons (number of ROIs × number of tests) using the FDR method (Benjamini and Yekutieli, 2001).

Brain segmentation, mesh reconstruction, and cortex-based alignment.

To create high-quality 3D brain reconstructions, we gathered, when available, multiple anatomical scans from each participant collected in different experiments performed at the Center for Mind/Brain Sciences, which we aligned and averaged (range: 1–13 scans). Individual surface meshes for each hemisphere were reconstructed along the border between gray and white matter. Next, individual reconstructions of each hemisphere were used to generate individual spherical surfaces for each participant that were then morphed to a template surface (a standard sphere). A coarse-to-fine moving target approach with four coarse-to-fine levels of smoothing was then used to extract multiscale surface curvature maps that reflect the gyral and sulcal folding patterns (Fischl et al., 1999; Goebel et al., 2006). This information allowed us to align the individual standardized spherical surfaces of all participants to a group-averaged spherical surface. Transformation matrices resulting from the cortex-based alignment of individual spherical surfaces to the group-averaged spherical surface were then used to align individual functional maps before entering group statistics. Finally, using the curvature maps from cortex-based alignment, we combined (i.e., averaged) the individual reconstructions of folded surfaces of all participants (N = 18) to create one group mesh for each hemisphere. Group-averaged left and right hemisphere meshes were used to display statistical maps resulting from both univariate and multivariate group analyses.

Surface-based searchlight SVM-MVPA.

The spherical searchlight (8 mm radius) was restricted to the surface by only including voxels from −1 to 3 mm along the gray/white matter boundary. Decoding procedures were very similar to the ones used for the ROI-based MVPA. For each hemisphere, we first created mesh time courses from the volume time courses. Next, we used mesh time courses to generate whole-brain t maps (20 per participant: 2 hemispheres × 10 runs), and finally we ran pairwise classifications on the t maps as described above. Decoding results of the spherical searchlight were assigned to the central voxel. Individual surface accuracy maps were projected onto the group-averaged cortical surface mesh (see Brain segmentation, mesh reconstruction, and cortex-based alignment) and then anatomically aligned using the transformation parameters derived from cortex-based alignment. We successively performed a two-tailed one-sample t test across individual cortical maps to identify vertices where classification was significantly greater than chance (50%). Statistical t maps were thresholded at p < 0.01 and corrected for multiple comparisons (p < 0.05) using a cluster-size algorithm (Forman et al., 1995) based on Monte Carlo simulations (1000 iterations) as implemented in Brain Voyager 2.8.0. For each hemisphere, we generated t maps and decoding accuracy maps separately for the Instructed condition, the Free-Choice condition, and across planning conditions.

Results

Behavioral results

RTs

Participants responded slightly faster in the Instructed (602.12 ± 18.67 ms) compared with the Free-Choice condition (605.51 ± 18.65 ms; F_(1,17) = 8.37, p < 0.01). However, RTs did not differ between movement types (F_(2,34) = 0.42, p < 0.65), and the interaction between planning condition and movement type was not significant (F_(2,34) = 2.66, p < 0.08).

Error rates

Participants were generally accurate in performing the delayed-movement task. Overall error rates were very low: 2.15% of all the trials in the Instructed condition and 0.54% in the Free-Choice condition. The fact that error rates were lower in the Free-Choice compared with the Instructed condition was expected given that, whereas errors in the Free-Choice condition only concerned kinematics, timing, or hand preshaping of the movements, errors in the Instructed condition also included executing a movement that was different from the instructed movement type.

Shannon's entropy in Free-Choice trials

To examine whether the movements selected in successive trials followed a regular pattern, we calculated a measure of randomness for movement selection in Free-Choice trials, defined as Shannon's Entropy index (Shannon, 1948) (see Materials and Methods). A low entropy index (0 < H < 1) indicates that one of the outcomes was chosen more often than others, or that the participant used a stereotyped transition pattern (e.g., 1, 2, 3, 1, 2, 3, etc.). By contrast, a high entropy index (H > 1.5) indicates that it is very hard to predict the next outcome on the basis of the previous outcomes. In our study, the mean entropy index per participant was 1.53, which is close to the maximum entropy level for three alternatives (H = 1.584). This analysis indicates that participants did not choose movements in a systematic, predictable way. As an example, this is a sequence chosen in the two consecutive blocks of one run by a representative participant: 2, 1, 2, 3, 2, 2, 1, 1, 3 and 2, 1, 2, 3, 2, 3, 1, 2, 2 (1 = PRG; 2 = PWG; 3 = TCH).

Univariate RFX-GLM results

To identify brain regions preferentially recruited during movement planning, we performed a univariate RFX-GLM contrast [Planning > Baseline] (Fig. 2A). This contrast is unbiased with respect to comparisons between the Instructed and Free-Choice Planning condition, or between different movement types. The resulting statistical map was used to define 16 group-ROIs: left primary motor cortex (L-M1); left dorsal and ventral premotor cortex (L-PMd, and L-PMv, respectively); left anterior, middle, and posterior intraparietal sulcus (L-aIPS, L-mIPS, and L-pIPS, respectively); left superior parietal lobule (L-SPL); left supramarginal gyrus (L-SMG); left dorsolateral prefrontal cortex (L-dlPFC); left supplementary motor area (L-SMA); left presupplementary motor area (L-preSMA); right posterior intraparietal sulcus (R-pIPS); right posterior superior temporal gyrus (R-pSTG); right posterior middle temporal gyrus (R-pMTG); right supplementary motor area (R-SMA); and right presupplementary motor area (R-preSMA; for details on the definition of individual ROIs, see Univariate analysis (GLM) and ROI definition and Table 1). Additionally, we contrasted the Planning phase against the Baseline separately for the two planning conditions ([Planning Instructed > Baseline]; [Planning Free-Choice > Baseline], Fig. 2B). Overall, the statistical maps for the Instructed and Free-Choice planning condition looked very similar, in particular in the left hemisphere, and the direct comparison [Planning Instructed > Planning Free-Choice] did not reveal any significant univariate effects.

Multivariate results

ROI-based MVPA

In the ROI-based MVPA, we tested whether upcoming movements could be decoded on the basis of patterns of preparatory brain activity within regions recruited during movement planning. To this end, for each ROI and planning condition, we ran two-tailed one-sample t tests (FDR corrected for multiple comparisons) on the mean decoding accuracy across participants (N = 18) against chance (50%). Figure 3 shows the mean classification accuracy in each ROI for averaged pairwise comparisons of movement types in four types of ROIs: (1) During the Planning phase, i.e., before any movement occurred, we found significant decoding of movement type both within (red and blue bars) and across (yellow bars) planning conditions in L-mIPS, L-pIPS, L-PMd, L-SPL, L-aIPS, and L-M1, suggesting abstract representations of planned movements that generalize across planning condition (i.e., Instructed vs Free-Choice; Fig. 3A). (2) In R-pIPS, L-dlPFC, R-pSTG, L-PMv, and R-pMTG, we were able to predict upcoming movements for the Free-Choice planning condition, but not for the Instructed planning condition (Fig. 3B). In L-SMG, we found a similar trend (p = 0.044) that did not survive FDR correction for multiple comparisons. (3) In L-SMA, we obtained above chance decoding for the Instructed, but not for the Free-Choice planning condition (Fig. 3C). R-SMA (p = 0.018), L-preSMA (p = 0.033), and R-preSMA (p = 0.026) showed trends in the same direction that did not pass FDR correction. (4) As expected, decoding of movement type was not possible (i.e., chance performance for all experimental conditions) in the non-brain control region outside the brain (Fig. 3D).

Figure 3.

ROI-based MVPA. Mean percentage decoding accuracies for movement type resulting from multiple binary classifiers. SVM classification accuracies for the three possible discriminations between movement pairs were averaged to produce a unique score per ROI and planning condition. Red bars, Planning Instructed. Blue bars, Planning Free-Choice. Yellow bars, Planning cross-condition (see Materials and Methods). Green bars, Execution (collapsing across Planning conditions). Statistical significance was assessed via one-sample t tests (two-tailed) against 50% chance. Results were FDR-corrected for multiple comparisons (number of ROIs × number of tests). Significance levels: one black asterisk, uncorrected p < 0.05; two black asterisks, uncorrected p < 0.005; one red asterisk, FDR corrected q < 0.05. A, Regions where we found both significant within- and cross-condition decoding. B, Regions where we observed significant effects (or trends) for the Free-Choice, but not for the Instructed Planning task. C, Regions where we observed significant effects (or trends) for the Instructed, but not for the Free-Choice Planning task. D, Control non-brain region outside the brain.

To further examine the nature of our effects, we performed post hoc two-tailed paired-samples t tests on the mean decoding accuracy between the two planning conditions for each ROI. After FDR correction for multiple comparisons (q < 0.05), these tests revealed a significant effect in L-PMv (t₍₁₇₎ = −4.44, p = 0.0004), indicating that decoding was significantly higher for Free-Choice compared with Instructed planning in this region. Post hoc comparisons that did not survive FDR correction for multiple comparisons include R-pIPS (p = 0.016), L-dlPFC (p = 0.027), R-pSTG (p = 0.042), and R-pMTG (p = 0.045).

Finally, during the Execution phase (Fig. 3, green bars), we were able to decode upcoming movements in all the ROIs, with the exception of R-pSTG (trend at p = 0.043), R-preSMA (p = 0.063) and the non-brain control region. Not surprisingly, we observed the highest decoding accuracy during the execution phase in the left (contralateral) primary motor cortex (L-M1), followed by the left aIPS.

Searchlight-based MVPA

To identify additional regions beyond our ROIs that potentially represent information about upcoming movements, we conducted a whole-brain searchlight-based MVPA on the surface (Figs. 4, 5). Figure 4 shows the performance of the classifier across the two planning conditions superimposed on the group-averaged inflated surface mesh. The cross-condition decoding t map (Fig. 4A) revealed significant clusters in left orbitofrontal (L-OFC) and frontopolar cortex (L-FP), L-dlPFC, posterior dorsal L-SMA, L-PMd, left anterior superior temporal sulcus (L-aSTS), L-IPS, inferior L-SPL, L-pSTG, L-SMG, left angular gyrus (L-AnG), and the left precuneus (L-preCu). In the right hemisphere, this analysis revealed significant clusters in R-FP, R-PMd, R-SPL, right superior parieto-occipital cortex (R-SPOC), R-pSTG, R-MTG, and right lateral occipital gyrus (R-LOG).

Figure 4.

Searchlight SVM-MVPA: cross-condition decoding. The spherical searchlight (8 mm radius) was restricted to the surface (−1 to 3 mm). Decoding procedures were very similar to the ones used for the ROI-based MVPA (see Materials and Methods). A, Group t map (thresholded at p < 0.01 and then cluster-size corrected) for the cross-condition decoding projected on the group-averaged surface mesh. White dashed lines indicate the outlines of the statistical map revealed by the univariate contrast [Planning > Baseline]. B, Group accuracy map (%) for cross-condition decoding.

Figure 5.

Searchlight SVM-MVPA: within-condition decoding. A, Group t maps (thresholded at p < 0.01 and then cluster-size corrected), separately for each planning condition (red represents Instructed; blue represents Free-Choice), projected on the group-averaged surface mesh. B, Group decoding accuracy maps (%) separately for each planning condition (left, Planning Instructed; right, Planning Free-Choice). All other conventions are the same as in Figure 4.

Figure 5A shows the within-condition-decoding t maps with cluster-size correction (p = 0.05) for multiple comparisons (red, Instructed; blue, Free-Choice) and their overlap (purple). Overall, significant clusters for Instructed and Free-Choice Planning appeared in neighboring but mostly nonoverlapping locations (except for the left anterior frontomedian cortex, bilateral superior dlPFC, and pSPL), and generally more widespread for the Free-Choice compared with the Instructed condition, especially in frontal (FP, dlPFC, PMd) and parietal (IPS, pIPL, pSPL) areas. For the Free-Choice planning condition, we obtained significant clusters in the left hemisphere in the anterior frontomedian cortex and L-OFC, L-FP, L-dlPFC, L-PMv, L-PMd, L-aIPS, L-pSPL, L-SPOC, and L-AnG. In the right hemisphere, this analysis revealed significant clusters in R-FP, superior R-dlPFC, R-aIPS, R-SMG, R-pSTG, R-pIPS, the right posterior inferior parietal lobule (R-pIPL), R-pSPL, R-SPOC, and, medially, the right cuneus (R-Cu) and R-preCu. For the Instructed planning condition, we obtained significant clusters in the left hemisphere in the superior L-dlPFC, the anterior frontomedian cortex (slightly anterior to L-SMA and superior to L-preSMA), L-PMd, L-SMG, L-pSPL, and L-LOG. For the right hemisphere, we obtained significant clusters in the superior R-dlPFC, the anterior R-SPL (right above R-aIPS), R-MTG (extending to the superior temporal sulcus), R-pSPL, and R-SPOC. When using a more conservative threshold of p < 0.001 (data not shown here), only clusters in L-dlPFC, L-PMd, L-IPS for the cross-condition decoding, and in bilateral dlPFC, pSPL, L-aIPS, and R-pIPS for the Free-Choice planning condition survived (i.e., no clusters for Instructed planning condition).

Figures 4B and 5B illustrate mean decoding accuracies for the cross-condition (Fig. 4B) and within-condition (Fig. 5B) decoding. These figures show both significant and subthreshold clusters of decoding accuracy to complement the information present in the searchlight t maps. Although we observed slight discrepancies between the ROI-based and searchlight-based MVPA results in some regions (L-M1, L-aIPS, L-mIPS, L-SMG, R-pMTG, R-pSTG), overall searchlight results appear to be largely in line with ROI results in several frontal (L-dlPFC, L-PMd, L-PMv, bilateral SMA, and preSMA) and parietal (L-pIPS, R-pIPS, L-SPL) regions (for a comparison of the two MVPA approaches, see Further observations).

Discussion

Frontal and parietal regions recruited during movement planning encode information about upcoming movements (Andersen and Buneo, 2002; Cisek and Kalaska, 2005; Cui and Andersen, 2007). Here we aimed to distinguish between areas representing abstract movement plans, areas involved in movement selection, and areas involved in the mapping between arbitrary sensory cues and the corresponding responses. We obtained three key results (summarized in Fig. 6) as follows: (1) contralateral (i.e., left) SPL and IPS, PMd, and M1 discriminate between planned movements regardless of the planning condition (i.e., both within and across internally and externally driven movements); (2) contralateral (i.e., left) PMv, dlPFC, SMG, and ipsilateral (i.e., right) pIPS, pSTG, and pMTG encode internally driven but not externally driven movement plans; and (3) bilateral SMA, possibly supported by pre-SMA, encodes the processing of externally driven movement plans.

Figure 6.

Summary of decoding results for the Planning phase. Circles superimposed on the group-averaged surface mesh represent examples of individual spherical ROIs color-coded according to the results of the ROI MVPA: yellow represents significant cross-condition decoding; blue represents preferential decoding for Free-Choice planning; red represents preferential decoding for Instructed planning. White-shaded areas with dashed outlines represent the statistical map revealed by the univariate contrast [Planning > Baseline].

Areas representing abstract movement plans

We obtained significant within-condition decoding of movement plans for both planning conditions, as well as significant cross-condition decoding, in the left (i.e., contralateral to the moving limb) SPL, pIPS, mIPS, aIPS, PMd, and M1 (Figs. 3A, 6). Our results are in line with studies showing that premotor regions are sensitive to arbitrary instructing cues (i.e., which movement to perform, or which effector to use) (Hoshi and Tanji, 2000, 2006, 2007), while also participating in action selection, when movements are freely chosen (Cisek and Kalaska, 2005; Pesaran et al., 2008; Beudel and de Jong, 2009; Klaes et al., 2011). Our results thus show that contralateral parietofrontal regions represent abstract movement plans that are invariant to the way these are generated rather than being tied to simple stimulus–response mapping (Hartstra et al., 2011, 2012) or movement decisions.

Movement plans can be abstract in a number of different ways. For instance, Gallivan et al. (2013a, 2013b) observed that bilateral posterior parietal cortex, PMd, posterior fusiform sulcus, and fusiform body area contain representations of upcoming movements that generalize across the effector (left vs right hand). These studies provide further evidence for abstract representations of movement plans in frontal, parietal, and ventral stream areas.

During movement execution, aIPS and M1 have been shown to represent handwriting movements generalizing across letter scale (Kadmon Harpaz et al., 2014). During movement observation, a number of recent studies revealed abstract action representations that generalize across viewpoint and modalities (Oosterhof et al., 2012a), and the object on which these actions are performed (Wurm and Lingnau, 2015; Wurm et al., 2015), in aIPS and lateral occipitotemporal cortex (LOTC). Further research is required to determine to which degree abstract movement representations are shared across planning, observation, and execution.

Areas involved in action selection

We were able to decode upcoming movements in the Free-Choice, but not in the Instructed condition in contralateral (left) PMv, dlPFC, SMG, and ipsilateral (right) pIPS, pSTG, and pMTG (Figs. 3B, 6). The dorsolateral pathway has been historically associated with grasping movements (Jeannerod et al., 1995; Luppino et al., 1999; for a recent review, see Turella and Lingnau, 2014). Our results extend these findings by revealing areas preferentially representing the selection rather than the planning of movements.

In contrast to studies that found significant decoding for instructed movements in PMv (Gallivan et al., 2011a, 2013a), we were able to decode upcoming movements in PMv for internally driven but not for externally driven movements, suggesting a more prominent role in action selection (i.e., deciding which movement to perform). It is possible that these inconsistencies are due to methodological differences. As an example, in contrast to the studies by Gallivan et al. (2011a, 2013a), participants in the current study saw neither the object nor their own hand throughout the experiment. Likewise, our planning phase was substantially shorter than the planning phase used by Gallivan et al. (2011a, 2013a). It is therefore possible that PMv represents both internally and externally triggered movement plans, depending on the availability of sensory cues and/or time for movement planning.

We were able to decode internally triggered movement plans in pMTG, a portion of the LOTC. LOTC is recruited during the processing of a variety of visual stimuli (e.g., basic and biological motion, tools, body parts, and actions) but also has been implicated to host action concepts (for a recent review, see Lingnau and Downing, 2015). In addition, and perhaps more surprising, LOTC has been demonstrated to be recruited during the planning and control of actions (Astafiev et al., 2004; Johnson-Frey et al., 2005; Verhagen et al., 2008; Kühn et al., 2011; Gallivan et al., 2013b, 2015; Kilintari et al., 2014). Integrating various kinds of information from the dorsal (e.g., visuo-spatial, motoric) and the ventral stream (e.g., semantics), LOTC might be an optimal site of convergence to create a link between perceiving, understanding, and interacting with the environment (Lingnau and Downing, 2015). Moreover, LOTC and the dorsal stream might exchange information about upcoming movements and/or anticipated sensory consequences of selected actions (Verhagen et al., 2008; Kühn et al., 2011; Gallivan, 2014; Lingnau and Downing, 2015). Finally, some studies suggest that, in contexts that lack visual feedback, occipitotemporal regions could play a role in motor imagery, dynamically updating representations of the moving limbs (Astafiev et al., 2004; Kühn et al., 2011; but see Orlov et al., 2010).

Areas involved in stimulus–response associations

We were able to decode externally triggered movement plans in left SMA, with a similar trend in the right SMA and left preSMA (Figs. 3C, 5A), in agreement with previous studies (Hoshi and Tanji, 2004; Mars et al., 2008; Gallivan et al., 2011a, 2011b, 2013a; Hartstra et al., 2012). This suggests a role for the frontomedian cortex in stimulus–response mapping, possibly in a broader network that includes also posterior parietal and premotor regions (Fig. 5). However, other studies have also linked SMA activity to voluntary action selection (Lau et al., 2004; Zhang et al., 2012, 2013) or self-initiated movements (Cunnington et al., 2002, 2003; Fried et al., 2011). Further work will be required to define the specific role of the SMA and preSMA, and possibly also posterior parietal and premotor regions, in stimulus–response mapping and movement planning.

Further observations

The univariate contrast [Planning > Baseline] revealed a more widespread recruitment of the contralateral compared with the ipsilateral hemisphere (Fig. 2), whereas the searchlight MVPA revealed significant clusters in both hemispheres (Figs. 4, 5). It thus appears that, despite weak activation, the hemisphere ipsilateral to the moving limb (in our study: the right hemisphere) also contains information about planned movements (see also Gallivan et al., 2013a; Leoné et al., 2014). This apparent inconsistency is likely due to the fact that MVPA relies on differences between activation patterns that can occur in the absence of amplitude differences (e.g., Kriegeskorte et al., 2006; Haxby, 2012).

We found significant cross-condition decoding in regions that only show significant within-condition decoding for one of the two planning conditions (Free-Choice: R-pSTG; R-MTG; Instructed: L-SMA; Fig. 3). At first glance, this result might look implausible: if a region codes movement plans independent of the task, then it should also reveal decoding in both tasks alone. There are, however, theoretical reasons that can explain this pattern of results. If Condition A tends to evoke more consistent patterns compared with Condition B, Condition A might improve cross-condition decoding. If Condition A is used for the training dataset, the classifier can more easily learn to distinguish the patterns. Likewise, if Condition A is used for the testing dataset, even if the classifier was trained on Condition B, it is more likely to guess correctly. In other words, training on more consistent patterns and testing on less consistent patterns (or vice versa) would produce better results than just training and testing within the same inconsistent pattern (see also Oosterhof et al., 2012b).

While the ROI- and the searchlight-based MVPA overall reveal converging results, the ROI analysis tended to be more sensitive than the searchlight analysis, in line with previous studies (Oosterhof et al., 2012b; Wurm and Lingnau, 2015). This is likely due to methodological differences between the two approaches (see also Etzel et al., 2013). In particular, the use of individual ROIs is less affected by individual differences in functional brain topography. By contrast, the searchlight approach is not limited to ROIs defined a priori but requires stricter criteria to produce significant results: (1) the exact same voxels in group space have to show significant decoding in the majority of participants; and (2) given the number of voxels in the brain, correcting for multiple comparisons is a much harder problem for searchlight-based MVPA. Given the pros and cons of both approaches, we present both analyses to provide the reader with a more complete picture of the results.

In conclusion, our results extend the existing literature on movement planning, distinguishing between regions containing abstract movement plans that are invariant to the way these were generated (externally vs internally driven), areas involved in movement selection, and areas containing movement plans for instructed movements.