Brain areas selective for both observed and executed movements

Abstract

When observing a particular movement a subset of movement-selective visual and visuomotor neurons are active in the observer’s brain forming a representation of the observed movement. Similarly, when executing a movement a subset of movement-selective motor and visuomotor neurons are active forming a representation of the executed movement. In this study we used an fMRI-adaptation protocol to assess cortical response selectivity to observed and executed movements simultaneously. Subjects freely played the rock-paper-scissors game against a videotaped opponent, sometimes repeatedly observing or executing the same movement on subsequent trials. Numerous brain areas exhibited adaptation (repetition suppression) during either repeated observations or repeated executions of the same movement. A subset of areas exhibited an overlap of both effects, containing neurons with selective responses for both executed and observed movements. We describe the function of these unique movement representation areas in the context of the human mirror system, which is expected to respond selectively to both observed and executed movements.

Introduction

The cortical motor system is commonly described as a hierarchically organized set of cortical areas where the task of planning and executing a movement is broken down into different components, which may range from sets of abstract motor goals to specific movement kinematics and dynamics variables. Particular executed movements are represented by specific subsets of cortical neurons that encode the movement’s spatial goal (Graziano et al. 2002), directions in space (Georgopoulos et al. 1986), muscle forces (Kakei et al. 1999), velocities (Reina et al. 2001), and joint angles (Scott et al. 1997). Specific motor neurons, therefore, are active during the execution of some movements and not others in a movement-selective manner forming a unique neural representation of the executed movement.

In a somewhat analogous manner the cortical visual system is also composed of a set of hierarchically organized areas which respond to different components of visual stimuli ranging from simple edge orientations in early visual areas (Hubel and Wiesel 1968) to complex objects (Malach et al. 1995), body parts (Downing et al. 2001), biological motion (Grezes et al. 2001; Grossman et al. 2000), and observed movements (Hamilton and Grafton 2006; Kable and Chatterjee 2006; Shmuelof and Zohary 2005; 2006) in higher visual areas. Specific visual neurons are hypothesized to be active during the observation of some movements and not others in a movement-selective manner forming a unique neural representation of the observed movement.

Mirror neurons are a unique class of visuomotor neurons that are active both during the execution of a movement and during the observation of the same movement (Gallese et al. 1996). Individual mirror neurons respond selectively during the observation and execution of specific movements and are believed to act as integration mechanisms for visual and motor representations of these movements (Rizzolatti et al. 2001). We, therefore, expect mirror neurons to be localized only in cortical areas that exhibit selective responses to movements whether observed or executed.

Functional magnetic resonance imaging (fMRI) has been used in conjunction with adaptation to assess the selectivity of neural responses in the human brain. This methodology is based on single-unit electrophysiology studies with non-human primates, which have shown that sensory neurons selective for a particular attribute (e.g. movement direction, object identity) commonly reduce their firing rate when their preferred stimulus is presented repeatedly. fMRI adaptation identifies cortical areas that exhibit smaller responses during trials in which a certain stimulus attribute is repeated (‘repeats’) in comparison to trials in which that attribute is changed (‘non-repeats’). Areas exhibiting smaller responses to repeats than non-repeats are interpreted as containing neurons selective for the stimulus attribute being repeated (Avidan et al. 2002; Boynton and Finney 2003; Desimone 1996; Engel and Furmanski 2001; Fang et al. 2005; Gardner et al. 2005; Grill-Spector 2006; Grill-Spector and Malach 2001; Henson and Rugg 2003; Huk and Heeger 2002; Huk et al. 2001; Kohn and Movshon 2003; Kourtzi and Kanwisher 2001; Kourtzi et al. 2003; Krekelberg et al. 2006; Larsson et al. 2006; Miller and Desimone 1994; Movshon and Lennie 1979; Neri et al. 2004; Suzuki et al. 1997). While there are numerous reports of repetition suppression in studies of visual processing, only one previous study has reported it in the motor domain (Karni et al. 1995).

Here, we report results from an fMRI experiment using the rock-paper-scissors (RPS) game in which subjects observed and executed long sequences of movements, without being instructed which movements to perform. Observed movements (performed by a virtual opponent) and executed movements (performed by the subject) were sometimes repeated on consecutive trials. We compared the responses to trials in which a movement was repeated (e.g., rock preceded by rock) with trials in which it was not (e.g., rock preceded by paper). Numerous cortical areas responded less to repeats, either observed or executed, than non-repeats. Six cortical areas showed an overlap or very close proximity of both effects: anterior inferior frontal sulcus (aIFS), ventral premotor (vPM), anterior intraparietal sulcus (aIPS), superior intraparietal sulcus (sIPS), posterior intraparietal sulcus (pIPS), and lateral occipital (LO). We propose that all six areas contain neurons that respond selectively to observed and executed movements, and may contain mirror neurons. Furthermore, our results successfully demonstrate the efficacy of fMRI adaptation for assessing selectivity in motor and somatosensory cortex, which may prove useful for studying motor control in humans.

Methods

Subjects

Thirteen healthy subjects (6 males) between the ages of 21 and 35 participated in this study. All subjects had normal or corrected-to-normal vision, provided written informed consent, and were paid for their participation in the study. The Committee on Activities Involving Human Subjects at New York University approved the experimental procedures, which were in compliance with the safety guidelines for MRI research. Twelve subjects participated in the main experiment, which included a high-resolution anatomical volume, one imitation experiment, one movement observation experiment, and four runs of the RPS experiment. Eight of the original subjects and one new subject also participated in an additional experimental session, which included a high-resolution anatomical volume and four runs of the instructed-movement experiment.

Visual Stimuli & Motor Response

Stimuli were presented via an LCD projector and custom optics onto a rear-projection screen in the bore of the MRI scanner. Subjects were supine and viewed the projected stimuli through an angled mirror, which also prevented them from seeing their own hands. A foamed rectangular tray was positioned above each subject’s pelvis and attached to the patient bed. Subjects executed motor responses above the tray. The motor responses were videotaped through a window from the console room.

Rock Paper Scissors (RPS) Experiment

The RPS experiment was based on the popular game rock-paper-scissors (http://en.wikipedia.org/wiki/Rock,_Paper,_Scissors). Subjects played the game against a videotaped opponent whose arm and hand were visible in the frame. Subjects were asked to perform their own game movements simultaneously with the observed movements. Each game movement consisted of banging twice on the tray and then performing the rock, paper, or scissors action. Video clips of all three movements contained the same banging segments so that subjects would not be able to predict one movement from another. A trial consisted of 2 seconds of movement followed by a 1 second inter-trial interval (blank screen). Each experimental run/game included 120 trials, 90 randomized game movements and 30 randomly interleaved blank trials (Figure 1A). We generated a different movie for each of the games by concatenating the same 3 movement video clips in different randomized sequences. Subjects played four runs/games in a single scanning session and were videotaped throughout the experiment to extract their movement choices. Each trial was categorized according to the movements observed and executed on the current and the preceding trial (Figure 1B). Trials were categorized into eight trial type categories, which included four repeat categories and their complementary non-repeat categories: executed repeats (repeated motor execution of a movement), observed repeats (repeated visual observation of a movement), executed-then-observed cross-modal repeats (execution of a movement followed by observation of the same movement), observed-then-executed cross-modal repeats (observation of a movement followed by execution of the same movement).

Figure 1.

Experimental design. A. Example experiment timeline, showing the simultaneously observed (green) and executed (orange) movement type in each trial. B. Each trial was sorted according to the movement preceding it into 8 categories: observed, executed, observed-then-executed, and executed-then-observed repeats and nonrepeats. We expected to find fMRI adaptation in repeat trials as compared with nonrepeat trials.

Subjects were told in advance that they would receive extra monetary compensation if they won more trials than they lost over the whole scanning session; hence, they were motivated to attend equally to every trial outcome regardless of whether a movement had been repeated or not. This unique feature of the game addresses an important concern regarding the source of response decreases in adaptation experiments as it has been shown that attention, arousal, and novelty can strongly affect fMRI responses (Huk et al. 2001), and subjects may naturally attend more to novel (non-repeated) stimuli.

Imitation Experiment

We wanted to compare the results from the RPS experiment described above with our own version of the imitation experiments used by others to define the human mirror system (Aziz-Zadeh et al. 2006; Buccino et al. 2004b; Carr et al. 2003; Dapretto et al. 2006; Grezes et al. 2003; Iacoboni et al. 1999; Leslie et al. 2004; Tanaka and Inui 2002; Williams et al. 2006). Subjects were instructed to either passively observe a movie clip of a specific movement (rock, paper, or scissors) being repeated six times, perform a specific movement six times in the dark, or imitate an observed movement six times (simultaneous observation and execution). The block alternation protocol contained 27 epochs in randomly shuffled order: three tasks (imitation, execution, and observation), on each of the three movements (rock, paper, scissors), each three times. Each 21 second long epoch consisted of 12 seconds of stimuli and 9 seconds of blank screen. The final second of each epoch contained a written instruction informing the subject of the task in the following epoch.

Movement Observation Experiment

We also wanted to compare the RPS experiment with our own variant of movement observation experiments used by others to define the human mirror system (Blakemore et al. 2005; Buccino et al. 2001; Buccino et al. 2004a; Haslinger et al. 2005; Iacoboni et al. 2005). The same protocol has also been used to define categorical visual areas (Hasson et al. 2003). Subjects viewed 32 epochs of category-specific movie-clips (each 15 seconds long), which were shuffled randomly into an 8-minute movie. Three distinct categories of movie clips were presented: faces under various natural situations (e.g., walking, socializing, eating, etc.), navigation of the camera through different urban areas, and object manipulation (e.g., playing guitar, cooking, answering phone, etc.).

Instructed-movement Experiment

This experiment was intended to test whether motor adaptation (‘repetition suppression’) is a robust effect, not only in the complex situation of playing the RPS game, but also when movements were instructed. In this experiment subjects performed the RPS movements in the absence of visual stimulation according to auditory instructions, in a randomly shuffled order. Subjects were asked to close their eyes throughout the scan. An auditory sound track contained a rhythmic ping every second and a verbal instruction to perform a movement every other second. Subjects performed the rock, paper, or scissors movement on the ping following the auditory instruction. In this experiment RPS movements were not preceded by banging twice on the tray and, therefore, enabled shorter scans that lasted only 4.5 minutes each. Each scan contained 120 trials, which consisted of 90 RPS movements and 30 blanks. Subjects were videotaped throughout the experiment to ensure that they were executing the instructed movements.

MRI Acquisition

Functional and anatomical images of the brain were acquired with a Siemens (Erlangen, Germany) 3T Allegra MRI scanner equipped with a birdcage head coil used for RF transmit and a four-channel phased array of surface receive coils positioned to cover the sides and back of the subject’s head (NM-011 transmit and NMSC-021 receive, NOVA Medical, Wakefield, MA, USA). Blood oxygenation level-dependent (BOLD) contrast was obtained using a T2*-sensitive echo planar imaging (EPI) pulse sequence (TR = 1500 ms for all experiments but instructed-movement experiment where TR = 2000 ms, TE = 30 ms, flip angle = 75°, 24 slices, 3×3×3 mm³ voxels, FOV = 192 mm). High resolution anatomical volumes were acquired with a T1-weighted 3D-MPRAGE pulse sequence (1×1×1 mm³).

Preprocessing, segmentation, and flattening

fMRI data were processed with the Brain Voyager software package (R. Goebel, Brain Innovation, Masstricht, The Netherlands) and with custom software written in Matlab (Mathworks, Natick, MA, USA). The first two images of each functional scan were discarded. Preprocessing of functional scans included 3D motion correction and temporal high-pass filtering with a cutoff frequency of 6 cycles per scan as well as 4 mm Gaussian spatial smoothing. Functional images were aligned with the high resolution anatomical volume using trilinear interpolation. The anatomical and functional images were transformed to the Talairach coordinate system (Talairach and Tournoux 1988). The cortical surface was reconstructed from the high-resolution anatomical images, separately for each subject. The procedure included segmenting the gray and white matter, inflating the gray matter, cutting along several medial locations including the calcarine sulcus, unfolding the cortical surface, and flattening.

Statistical Parameter Mapping

We used the Brain Voyager software package to run a standard statistical parameter mapping (SPM) analysis (Friston et al. 1994). In short, we constructed a general linear model (GLM) for the underlying neural response to each experimental condition. For example, the model for our imitation experiment was a matrix that contained a row for each time point, where neural activity was modeled as either “on” = 1 or “off” = 0, and a column for each condition: observe, execute, and imitate. For the RPS experiment, the matrix consisted of 13 columns corresponding to observed repeats and non-repeats, executed repeats and non-repeats, observed-then-executed repeats and non-repeats, executed-then-observed repeats and non-repeats, win lose and draw trials, blank trials, and trials preceded by blanks. The expected neural activity model (each column of the model matrix) was convolved with a canonical HRF to create a model of the expected hemodynamic response (Boynton et al. 1996). We used linear regression to estimate response amplitudes (beta values) for each voxel and each condition, solving an equation of the form y = Ax, where the vector y was the measured time course during one scan, the vector x contained the estimated response amplitudes, and A was the linear model described above. A voxel-by-voxel t-test identified brain areas exhibiting significantly different responses to particular conditions. Data were combined across subjects using a random-effects analysis; response amplitudes were computed separately for each subject and then a paired t-test was used to determine significant voxel-by-voxel response differences across all subjects (Friston et al. 1999). Only voxel clusters exceeding 25 mm³ are displayed in the statistical maps. All cortical activation maps presented in this paper were generated by rendering the random-effects analysis results on a representative individual’s cortical surface, except for Figure 2 showing single subject activation to illustrate how the regions of interest (ROIs) were defined.

Figure 2.

ROI Selection in a typical subject. Top Row. Identification of the junction between the anterior intraparietal sulcus and the post central sulcus. Middle Row. Functional activity from the imitation experiment is overlaid (purple). Bottom Row. Selection of the ROI to include only active voxels within a maximum diameter of 13 mm³ surrounding the anatomical land-mark.

Region of Interest (ROI) definition and analysis

We wanted to directly assess whether cortical areas exhibiting imitation responses also exhibited movement-selectivity. We, therefore, used the imitation experiment results to define ten ROIs individually for each subject. ROIs were defined with the Brain Voyager software package using a combination of anatomical and functional criteria. First, we identified single voxels that exhibited significant imitation responses using a maximal False Discovery Rate (FDR) of 0.01. FDR is a method of correcting for multiple comparisons by controlling for the expected proportion of false positives among suprathreshold voxels (Genovese et al. 2002) rather than for the rate of false positives among all comparisons as done by the stricter Bonferroni method. Next, we applied a cluster threshold and further considered only voxels that were part of a cluster with a minimum volume of 25 mm³. Finally, we identified particular anatomical land-marks on each subject’s anatomy and chose only functionally “active” voxels surrounding these locations while restricting the ROI size to a maximum diameter of 13 mm (Figure 2).

Because many cortical areas responded during imitation (Figure 4) our choice of ten particular ROIs (aIFS, vPM, CMA, M+S, PSF, aIPS, sIPS, pIPS, and V) may seem arbitrary. This, however, was not the case. We wanted to relate our findings to the literature and, therefore, chose five of these ROIs (aIFS, vPM, aIPS, sIPS, and LO) because they have been commonly reported as areas responding during imitation in mirror system studies (Table 1) although only two of them have been commonly referred to as the human mirror system (vPM and aIPS). We chose four more ROIs (CMA, M+S, PSF, and V) because they exhibited the strongest imitation activity. The final ROI (pIPS) was chosen because we noted that this area exhibited both observed and executed adaptation (Figure 3). The definitions of all 10 ROIs, however, were statistically independent from the adaptation results (separate data sets). The importance of the ROI analysis was not only to show that some of the independently defined imitation areas exhibited movement selectivity, but also for systematically assessing and quantifying the physiological characteristics of the areas, not otherwise be evident in parameter mapping analyses. These characteristics included the amplitudes of visual and motor adaptation (Figure 5), the lack of cross-modal adaptation (Figure 5), the similarity of responses to all trial outcomes (Figure 7), the similarity of adaptation effects regardless of trial outcome (Figure 8), and the similarity of adaptation effects regardless of the hemodynamic response function used to model the data (Supplementary Figure 2).

Figure 4.

Imitation and movement observation. A&B. Maps of activity evoked by imitation, execution, and observation (same format as left hemisphere in Figure 3). Purple, brain areas with larger responses during imitation than observation and execution. Cyan, larger responses during observation than rest. Orange, larger responses during execution than rest. C. Maps of activity evoked by observation of faces, houses, and movement observation. Red, larger responses to faces than houses & object manipulation. Green, larger responses to houses than faces & object manipulation. Blue, larger responses to object manipulation than faces & houses. Black ellipses loosely outline areas of interest.

Table 1.

Comparison of cortical areas described in current and previous studies (Talairach coordinates). Studies labeled with (I) used an imitation protocol and studies labeled with (O) used a passive movement observation protocol. Visual and motor adaptation coordinates refer to regions that exhibited smaller responses to both observed and executed repeats than non-repeats (Figure 3). Current imitation and movement observation coordinates refer to multi-subject maps from the imitation and movement observation experiments (Figures 4A and 4C). Coordinates refer to the center of mass of multi-subject ROIs chosen using the same anatomical and spatial limitation criteria described for single subject ROIs (see Methods, Figure 2).

	aIFS			vPM			aIPS			sIPS			pips			LO
	X	Y	Z	X	Y	Z	X	Y	Z	X	Y	Z	X	Y	Z	X	Y	Z
Visual & Motor Adaptation	−39	23	21	−43	0	26	−36	−39	35	−24	−59	48	−26	−70	29	−43	−66	−4
Current (I)	−34	25	25	−39	−2	29	−38	−40	41	−25	−61	42	−28	−67	33	−40	−71	−3
Current (O)	−41	30	16	−45	−3	32	−42	−36	38	−26	−59	47	−25	−74	29	−44	−65	3
Dapretto 2005 (I)				−50	8	10										−44	−64	−4
Leslie 2004(I)				−59	7	19	−56	−25	26							−45	−67	−2
Carr 2003 (I)	−46	36	12	−40	2	32	−40	−46	50	−24	−60	40
Buccino 2004a (I)	−32	40	27	−51	9	31	−32	−56	54	−24	−70	44
Iacoboni 1999 (I)				−50	12	12
Grezes 2003 (I)				−44	2	26	−32	−48	52
Tanaka 2002 (I)				−51	5	29	−36	−46	48	−20	−66	44
Jackson 2005 (I)				−60	10	30										−50	−74	2
Buccino 2004b (O)				−59	5	29	−32	−45	43
Buccino 2001 (O)				−64	4	24	−36	−40	52
Shmuelof 2005 (O)							−36	−42	54
Shmuelof 2006 (O)							−35	−45	58
Hamilton 2006 (O)							−52	−32	44	−32	−56	46

Figure 3.

Movement-selective (visual and motor) adaptation. Statistical maps of brain areas responding less to repeated than non-repeated movement observation (green) and execution (orange) of the RPS movements. Six areas in the left hemisphere (movements were performed with the right hand) exhibited either an overlap (yellow) or close proximity of both visual and motor adaptation: anterior intraparietal sulcus (aIFS), ventral premotor (vPM) cortex, anterior intraparietal sulcus (aIPS), superior intraparietal sulcus (sIPS), posterior intraparietal sulcus (pIPS), and an area within lateral occipital (LO) cortex. Black ellipses loosely outline areas of interest.

Figure 5.

Region of interest (ROI) analysis. Top row. Visual and motor adaptation. Comparison of fMRI response amplitudes in four conditions: observed non-repeat (dark green), observed repeat (light green), executed non-repeat (dark orange), executed repeat (light orange). Bottom row. Cross-modal interactions. Comparison of fMRI response amplitudes in four conditions: observed-then-executed non-repeat (dark blue), observed-then-executed repeat (light blue), executed-then-observed non-repeat (dark purple), and executed-then-observed repeat (light purple). ROIs were defined in each subject individually (Figure 2). fMRI responses from the RPS experiment were averaged across voxels in each ROI, and across games and subjects. Error bars, SEM across subjects. Asterisks, statistically significant difference (p < 0.05, paired t-test, Bonferronni corrected).

Figure 7.

Region of interest (ROI) analysis of trial outcomes. Error bars, SEM across subjects. There were no statistically significant differences between different trial outcomes in any of the examined ROIs (p > 0.18, ANOVA run independently for each ROI).

Figure 8.

Region of interest (ROI) analysis split according to trial outcomes. Top row. Visual adaptation. Comparison of fMRI response amplitudes in six conditions: observed non-repeat win, lose, and draw (all in dark green) and observed repeat win, lose, and draw (all in light green). Bottom row. Motor adaptation. Comparison of fMRI response amplitudes in six conditions: executed non-repeat win, lose, and draw (all in dark orange) and executed repeat win, lose, and draw (all in light orange). Error bars, SEM across subjects. See Behavioral and Outcome Analyses in Results section for statistical details.

All 10 ROIs were defined in all subjects except for area aIFS, which was defined in only 9 of the 12 subjects due to a lack of imitation activity in this area in 3 of the subjects. Note that the precise boundaries of these ROIs were arbitrary and depended on the choice of anatomical land-marks, functional statistical threshold, and spatial restriction used to define them. We repeated the analysis with ROIs defined using the same anatomical land-marks, but with different statistical thresholds ranging from an FDR of 0.1 to 0.005, individually for each subject as well as in multi-subject averaged maps, and always obtained comparable results to those reported here.

ROI analyses were carried out by averaging across voxels so as to yield a single response time-course for each ROI in each individual subject. We used standard event-related analysis methods to estimate fMRI response amplitudes, as described above, separately for each ROI in each subject individually. We then performed paired t-tests to determine which ROIs showed significant response differences across subjects for selected pairs of conditions (e.g., observed repeat versus non-repeat) while correcting for multiple comparisons using the Bonferroni method.

When performing this ROI analysis, we assumed a canonical hemodynamic response function (HRF). Because individual subjects may exhibit differences in their HRFs, this approach could lead to a poor fit to the data such that response amplitudes would be misestimated. To evaluate whether this was the case, we estimated the HRF for each subject individually (Supplementary Figure 1) and recomputed the ROI analysis with these individual subject HRFs instead of the canonical HRF (Supplementary Figure 2). See supplementary materials online for details. We found that the individually-defined HRFs were very similar to the canonical HRF (mean r = 0.87 averaged across subjects and ROIs, SEM = 0.03, see Supplementary Figure 1). The results of the ROI analysis were almost identical using the canonical and individually-defined HRFs, demonstrating that the visual and motor adaptation effects were robust in all but one ROI. The only exception was LO where the visual adaptation effect was lost (Supplementary Figure 2).

Results

Rock Paper Scissors Adaptation Experiment

In the main experiment subjects played the rock-paper-scissors (RPS) game against a virtual opponent (Figure 1). Subjects observed a video of the opponent’s hand and performed their own movements simultaneously while lying in the scanner (Figure 1A). Because subjects performed the game movements with their right hand, we have focused our analysis on the left hemisphere. Each trial was categorized according to the executed and observed movement on that trial and the one preceding it (Figure 1B). There were four possible types of repeat trials (Figure 1B): executed (repeated motor execution of a movement), observed (repeated visual observation of a movement), observed-then-executed (observation of a movement followed by execution of the same movement), and executed-then-observed (execution of a movement followed by observation of the same movement). Every trial was labeled as either a repeat or non-repeat in each of the categories. The irregular ordering and large number of trials collected from each subject enabled us to tease apart the fMRI responses associated with each trial type.

Some cortical regions exhibited smaller responses to observed movement repeats than non-repeats, and other cortical areas exhibited smaller responses to executed movement repeats than non-repeats (Figure 3, green and orange, respectively). Five areas exhibited an overlap of both effects (Figure 3, yellow): they were located in anterior inferior frontal sulcus (aIFS), ventral premotor (vPM) cortex, anterior intraparietal sulcus (aIPS), superior intraparietal sulcus (sIPS), and posterior intraparietal sulcus (pIPS). In a sixth area, the lateral occipital (LO) cortex, the effects were adjacent but did not overlap. We did not find any cortical areas exhibiting smaller or larger responses to cross-modal repeats than non-repeats, neither for observed-then-executed nor executed-then-observed repeats even at very low statistical thresholds.

These findings of repetition suppression are consistent with three recent studies, which have used fMRI adaptation protocols to assess selective neural responses to repeatedly observed movements (Hamilton and Grafton 2006; Shmuelof and Zohary 2005; 2006). All of these reported movement-selective repetition suppression in aIPS while some also reported repetition suppression in other locations throughout the intraparietal sulcus (IPS) as well as weak (but not statistically significant) repetition suppression in vPM (Table 1). In this study areas aIPS, sIPS, pIPS and vPM exhibited repetition suppression not only to repeatedly observed movements, but also to repeatedly executed movements. These results are also consistent with a large body of literature elucidating the roles of these areas in movement representation and planning (including the movement’s goal, speed, and spatial end point), visuomotor transformations, and coordination of grasping movements (Castiello 2005; Culham et al. 2006; Grefkes and Fink 2005; Orban et al. 2005; Tunik et al. 2005).

Perhaps surprisingly, we did not find any repetition suppression to repeatedly observed movements in the superior temporal sulcus (STS), an area commonly considered as important for the perception of movement (Puce and Perrett 2003) and often considered the “visual component” of the human mirror system (Iacoboni 2005). In agreement with our results, three previous studies failed to find visual adaptation to repeated movement observation in the STS (Hamilton and Grafton 2006; Shmuelof and Zohary 2005; 2006).

The finding of movement-specific repetition suppression in lateral occipital cortex (LO) may also seem surprising since this is a cortical area not commonly thought of as representing either observed or executed movements. The area we labeled as LO in our figures may include the adjacent extrastriate body area (EBA) (Downing et al. 2001). Both LO and EBA respond robustly when observing body parts (Saxe et al. 2006), yet this is the first report of LO exhibiting observed movement selectivity (Figure 3, green). A closely adjacent region also exhibited selectivity for executed movements (Figure 3, orange) as might be expected from a somatosensory or motor area, but not from a purely visual area. This result, however, is consistent with several studies showing that similar lateral occipital areas respond during motor tasks (Table 2) (Astafiev et al. 2004; Peelen and Downing 2005) perhaps reflecting a mechanism of motor imagery (Astafiev et al. 2004). The LO area we identified also responded more to movement execution than rest in two further experiments (see below, Figures 4B & 9). It is difficult, because of intersubject anatomical variability, to be certain whether the visual and motor adaptation effects were truly segregated into two neighboring areas within LO/EBA or were somewhat overlapping. Segregated yet adjacent movement-selective responses may be interpreted as evidence supporting a recently proposed functional role of this area in separation of self from others (Jeannerod 2004). Overlapping movement selectivity may be interpreted as supporting evidence for the existence of mirror neurons in this area.

Table 2.

Comparison of LO defined in the current study with previously reported foci of motor activation in LO/EBA (Talairach coordinates). Current study coordinates were computed as in Table 1.

Lateral Occipital
	X	Y	Z
Imitation LO ROI (Multi Subject)	−40	−71	−3
Observation LO ROI (Multi Subject)	−44	−65	3
Motor & Visual Adaptation (Multi Subject)	−43	−66	−4
Astafiev 2004	−48	−69	6
Peelen 2005	−45	−74	−3

Figure 9.

Motor adaptation from a separate instructed-movement experiment A. Maps of motor activity and motor adaptation (same format as left hemisphere in Figure 3). Blue, areas that responded more during executed movement than baseline. Orange, areas that responded less during executed movement repeats than non-repeats. B. Region of interest (ROI) analysis of response amplitudes during instructed-movement. Error bars, SEM across subjects. Asterisks, statistically significant difference (p < 0.05, paired t-test, Bonferronni corrected).

Finally, area aIFS is located in lateral prefrontal cortex, which is commonly active in tasks involving working memory (Curtis 2006; Petrides 2005). One might, therefore, postulate that movement-selective activity in this area during our experiments was due to the working memory demands of the tasks (e.g., loading observed and executed movements into working memory for comparison during the RPS game).

Imitation and Movement Observation Experiments

To compare our results with those of previous human mirror system studies, we performed a variant of the imitation protocol commonly used to localize candidate human mirror areas. (Aziz-Zadeh et al. 2006; Buccino et al. 2004b; Carr et al. 2003; Dapretto et al. 2006; Grezes et al. 2003; Iacoboni et al. 1999; Jackson et al. 2006; Leslie et al. 2004; Tanaka and Inui 2002; Williams et al. 2006). The rationale of the imitation protocol is that mirror neurons respond more strongly during imitation (simultaneous observation and execution) than observation or execution alone. In this experiment subjects were instructed to observe, execute, or imitate (simultaneously observe and execute) the rock, paper, or scissors movements repeatedly. Many cortical areas responded more strongly during imitation blocks than during observation or execution blocks (Figure 4A) including all six of the movement-selective areas described above (aIFS, vPM, aIPS, sIPS, pIPS, and LO). Note, however, that other cortical areas including the cingulate motor area (CMA), early visual areas (V), primary motor and somatosensory areas (M&S), and posterior Sylvian fissure (pSF) also exhibited stronger responses during imitation blocks (Figure 4A).

Another commonly reported expectation of cortical areas containing mirror neurons is that they respond both when executing movements without any visual stimulation and when observing movements without executing any movements. Using the same imitation protocol we identified areas responding during movement execution (Figure 4B, orange) and movement observation (Figure 4B, cyan) as compared to blank (rest) blocks. Four of the six cortical areas described above (vPM, aIPS, sIPS, and LO) responded during both observation and execution.

We also performed a variant of the passive movement observation protocol used in previous studies with the rationale that mirror neurons respond more strongly to visual stimuli containing movements than to visual stimuli that do not (Blakemore et al. 2005; Buccino et al. 2001; Buccino et al. 2004a; Hamilton and Grafton 2006; Haslinger et al. 2005; Iacoboni et al. 2005; Shmuelof and Zohary 2006). In this experiment subjects passively viewed three types of movie clips containing: object manipulation movements (e.g., picking up a phone), houses (e.g., downtown urban areas), and faces (e.g., people talking). Many cortical areas exhibited stronger responses to the object manipulation movements than to the other two categories of movies including all six of the movement-selective areas described above (Figure 4C, blue).

Both the imitation and movement observation experiments yielded results consistent with previously published reports (Tables 1 and 2).

Region of Interest Analysis

We re-analyzed data from the RPS experiment in ten ROIs: aIFS, vPM, aIPS, sIPS, pIPS, LO, M+S, V, CMA, and pSF (Table 3), which were defined using the imitation experiment. Cortical activity during imitation was widespread, making it difficult for us to isolate ROIs even at high statistical thresholds (e.g., defining where aIPS activity ends and sIPS activity begins). This is a common problem in imitation experiments and we, therefore, selected ROIs individually for each subject using a combination of anatomical land-marks and volume constraints, which limited their location and size (see Methods, Figure 2). The ROIs defined in this way were similar in their location to previously reported foci of activation during imitation and movement observation (compare Table 3 with Tables 1 and 2). We averaged the RPS responses across voxels in each ROI, estimated the response amplitude to each type (observed, execute, cross-model) of repeat and non-repeat, and then computed the mean and SEM across subjects.

Table 3.

Talairach coordinates for each ROI, computed by averaging the center of mass of individual subject ROIs. Standard deviation across subjects is shown in parentheses.

ROI Name	Talairach Coordinates: X Y Z
aIFS	−45 (5) 20 (6) 22 (5)
vPM	−44 (7) −1 (4) 27 (4)
aIPS	−38 (4) −40 (5) 41 (7)
sIPS	−28 (6) −61 (7) 46 (7)
pIPS	−25 (3) −75 (7) 27 (8)
LO	−43 (2) −67 (4) 3 (6)
V	−14 (3) −94 (2) 8 (6)
M+S	−36 (3) −28 (4) 53 (4)
CMA	−5 (2) −20 (8) 48 (5)
pSF	−44 (4) −32 (5) 20 (4)

The RPS response amplitudes were significantly smaller during both observed repeats (p < 0.05, paired t-test, Bonferronni corrected) and executed repeats (p < 0.01) in five ROIs: aIFS, vPM, aIPS, sIPS, and pIPS (Figure 5, Supplementary Figure 2). LO also exhibited smaller response amplitudes during observed and executed repeats, both of which were significant when assessed using a canonical HRF (Figure 5) while only executed repeats were significant when assessed using individually defined HRFs (Supplementary Figure 2). Responses in areas M+S and CMA were significantly smaller only for executed repeats (p < 0.001), while responses in early visual cortex (V) were significantly smaller only for observed repeats (p < 0.005). Area pSF responses were not significantly smaller for either executed repeats (p > 0.2, uncorrected) or observed repeats (p > 0.6, uncorrected). We also compared responses in the same ROIs to cross-modal repeats and non-repeats for both observed-then-executed and executed-then-observed trials (Figure 5B). There were no significant differences between cross-modal repeats and non-repeats in any of the ROIs (p > 0.08, uncorrected).

In a similar manner we also re-analyzed data from the RPS experiment in six ROIs defined using the passive movement observation experiment. We used the same anatomical land-marks, but now used the functional activity during movement observation (as compared with observation of houses and faces) to define areas aIFS, vPM, aIPS, sIPS, pIPS, and LO. We found similarly smaller responses during executed and observed movement repeats than non-repeats in all six ROIs (p < 0.05, corrected).

Behavioral and Outcome Analyses

In the RPS game subjects freely chose which movement to perform on every trial and were motivated to attend the trial outcomes (subjects received extra monetary compensation if they won more trials than they lost). This was done to preserve the visuomotor context of the game and keep subjects engaged with the experiment. However, it raises the concern that subjects may have adopted a behavioral strategy while playing the game that might confound the interpretation of the results in either of two ways. First, subjects might have selected their movements based on the movements that they or their opponent performed on the previous trial (e.g., copying the movements performed by their opponent on the previous trial). If so, the resulting repetition suppression might have been generated by adaptation of neural populations involved in making strategic decisions (whether to repeat). Second, subjects may have selected their movements based on the outcome of the previous trial (win, lose, or draw). If so, the resulting repetition suppression might have been generated by neural populations involved in reward representation rather than by neural populations involved in the representation of observed and executed movements.

The distributions of subjects’ executed repeats and non-repeats, however, did not differ significantly from the distributions expected by random movement choices (Table 4). The same was true for executed-then-observed repeats and non-repeats. In the case of observed-then-executed repeats and non-repeats, however, we found a slight yet significant deviation from the expected distributions. Subjects tended to avoid executing the previously observed movement (avoided “copying” their opponent) in a way that slightly skewed the actual distributions (Table 4). While this behavioral strategy does not confound the interpretation of observed and executed repeats and non-repeats where we found repetition suppression effects, it may confound the interpretation of the cross-modal trials where we did not find any repetition effects. Since we focus our attention on the former rather than the later we will not discuss this apparent behavioral strategy further.

Table 4.

Frequency of repeat and non-repeat trials compared to distributions expected by random movement choices. On average each game contained 67 trials that were categorized as repeats or non-repeats (movement trials preceded by blank were excluded from analysis). The mean number of trials in each group was compared to the expected number of trials when movements are random (where one would expect 22 repeat and 45 non-repeat trials for each of the trial types). A t-test was performed to determine whether the actual number of repeats/non-repeats differed from that expected, separately for each trial type. A small yet significant deviation from randomness was found in the observed-then-executed trial type where there were fewer repeats and more non-repeats than expected (suggesting that subjects actively avoided executing previously observed movements).

Trial Type	Mean # of Trials (Standard Error)	T value	P value
Observed Repeats	22 (0)	−0.52	0.61
Observed Non-Repeats	45 (0)	0.12	0.9
Executed Repeats	21.4 (2.4)	−1	0.32
Executed Non-Repeats	45.3 (2.4)	0.7	0.48
Observed-then-Executed Repeats	18.3 (1.8)	−5.75 *	< 0.001 *
Observed-then-Executed Non-Repeats	48.4 (1.8)	5.34 *	< 0.001 *
Executed-thenObserved Repeats	22.4 (1)	0.13	0.89
Executed-then-Observed Non-Repeats	44.3 (1)	−0.51	0.61

Moreover, subjects did not significantly alter the frequency of executed movement repeats and non-repeats based on the outcome of the previous trials. The distributions of trials where movements were repeated or not following a win, lose, or draw trial did not differ significantly from chance (Table 5). Hence, there was no evidence that subjects chose their executed movements based on whether they won, lost, or drew on the previous trial (e.g., they did not systematically choose to repeat winning movements).

Table 5.

Behavioral choices did not depend on the outcome of the previous trial. On average there were 67 trials per game, which were equally likely to be a win, loss, or draw. By chance subjects should have switched to a different movement (non-repeat) two thirds of the time and stayed (repeated) one third of the time. Given 22 trials in each outcome category (win, lose, or draw) chance would dictate switching 15 trials and staying 7 trials. The mean number of trials in each group was compared to the expected number of trials. A t-test showed that the actual number of trials did not differ significantly from the expected number of trials in any of the categories.

Trial Type	Mean # of Trials (Standard Error)	T value	P value
Win stay repeats	7.7 (0.5)	0.51	0.61
Win switch non-repeats	14.5 (0.6)	−0.42	0.67
Lose stay repeats	7.1 (0.5)	−0.52	0.6
Lose switch non-repeats	14.9 (0.7)	0.37	0.7
Draw stay repeats	6.7 (0.5)	−1.79	0.08
Draw switch non-repeats	15.9 (0.6)	1.31	0.2

To determine if cortical responses to trial outcome (lose, draw, or win) confounded the interpretation of our results we compared responses to winning trials with losing trials. A number of cortical areas responded more to winning trials than to losing trials (none responded more to the opposite comparison), including motor, somatosensory, and early visual areas (Figure 6A). The largest effect of trial outcome, however, was in sub-cortical areas including the bilateral striatum (Figure 6B), which was evident at much higher statistical thresholds (Figure 6C) and is consistent with previous reports concerning neural representations of reward (O’Doherty 2004). We also performed an ROI analysis to compare responses of all three outcomes within each of the ten ROIs described above (Figure 7). The effect of outcome on response amplitude was not significant in any of the six movement-selective ROIs (F(2,35) < 1.8, p > 0.18, ANOVA run independently for each ROI). Finally, to make absolutely certain there was no influence of trial outcome on the response amplitude of repeat trials (preceded by the same movement) and non-repeat trials (preceded by a different movement) we split the trials according to outcome (win, lose, or draw) on the current trial and compared repeats and non-repeats within each of the ROIs for each of the three outcomes (Figure 8). We then performed an ANOVA, independently for each of the ROIs, and separately for observed and executed trials. Each ANOVA had two factors: outcome (win, lose, and draw) and repetition (repeat and non-repeat). Repetition suppression was statistically significant for both observed and executed repeats in all six of the movement-selective ROIs (F(3,144) > 4.8, p < 0.03), but not in any of the control ROIs. Outcome effects were statistically significant in sIPS (observed movements only, p < 0.04) and in pIPS (both observed, p < 0.01 and executed, p < 0.02), but note that there were also clear repetition suppression effects in these ROIs for all three outcomes. There were no significant interactions between repetition and outcome in any of the ROIs (p > 0.1).

Figure 6.

Responses to trial outcomes. Orange, areas that responded more during winning trials than losing trials. A. Several cortical and sub-cortical areas exhibited larger responses to winning than losing (p < 0.05). B. The same results displayed with a stricter statistical threshold (p < 0.003). Only the ventral striatum and a small area of visual cortex survived at this higher threshold. The opposite comparison (lose > win) did not yield any statistically significant activation.

Instructed-movement Experiment

To assess the robustness of motor adaptation we ran a fourth experiment in which subjects performed the RPS movements in randomly shuffled order, according to auditory instructions in the absence of any visual stimulus. Large regions including motor and somatosensory cortex responded during movement execution (Figure 9A, blue) some of which (Figure 9A, orange) exhibited motor adaptation (smaller responses to repeated than non-repeated movements). Only three ROIs (aIPS, M+S, and vPM) exhibited motor adaptation in both this experiment (Figure 9B) and the RPS experiment. In fact, in comparison to area M+S, areas aIFS, vPM, sIPS, pIPS, and LO exhibited rather weak responses during this experiment in contrast to the RPS experiment (compare Figures 5 and 9B). Note that in this experiment the subjects did not freely choose their movements, they did not compare their executed movements to observed movements, nor did they have any motivation to perform the movements (other than to cooperate with the experimenter’s instructions). We attribute the robust difference between the results of the RPS and instructed-movement experiments to these differences in behavioral context, although there were other differences between the two experimental protocols (e.g., number of subjects, timing of each trial).

Discussion

Adaptation and Movement Representations

During the RPS game fMRI responses were smaller in several cortical areas when subjects observed or executed the same movement repeatedly (Figure 3). These smaller responses provide evidence for underlying neural adaptation (repetition suppression) in neurons selective for aspects of the movements being observed or executed. Specifically, the results suggest that there are separate subpopulations of neurons that respond selectively to each of the RPS movements such that each subpopulation adapts when its preferred movement is observed or executed repeatedly. Six cortical areas exhibited overlapping or closely adjacent visual and motor adaptation: aIFS, vPM, aIPS, sIPS, pIPS, and LO (Figure 3). We propose that these areas contain visual, motor, and visuomotor neurons, which are involved in the neural representation of both observed and executed movements.

Adaptation and Mirror Neurons

Approximately 20–30% of the neurons in areas F5 and IPL of the monkey have been described as mirror neurons, which are selective for particular movements whether executed or observed (Rizzolatti and Craighero 2004). The human mirror system areas, however, have been defined using the imitation protocol (Aziz-Zadeh et al. 2006; Buccino et al. 2004b; Carr et al. 2003; Dapretto et al. 2006; Grezes et al. 2003; Iacoboni et al. 1999; Jackson et al. 2006; Leslie et al. 2004; Tanaka and Inui 2002; Williams et al. 2006) and the passive movement observation protocol (Blakemore et al. 2005; Buccino et al. 2001; Buccino et al. 2004a; Haslinger et al. 2005; Iacoboni et al. 2005), neither of which assess the same criteria used in the monkey.

There are two concerns with both of these experimental protocols. The first concern is that large swaths of the cortex including areas not expected to contain mirror neurons (e.g., early visual areas) exhibit strong fMRI responses during imitation and movement observation both in our results (Figure 4) and in previously published studies (Aziz-Zadeh et al. 2006; Buccino et al. 2001; Buccino et al. 2004a; Buccino et al. 2004b; Carr et al. 2003; Dapretto et al. 2006; Grezes et al. 2003; Iacoboni et al. 2005; Iacoboni et al. 1999; Leslie et al. 2004; Tanaka and Inui 2002; Williams et al. 2006). This means that the utilized tasks and comparisons are unable to selectively activate only mirror neurons, but rather activate a host of other neurons. In spite of this, previous work has focused on imitation and movement observation responses in only two of the many active areas (vPM and aIPS), because these are assumed by some to be homologous to monkey areas F5 and IPL, respectively.

The second concern is that neither of these protocols assess whether neurons within the identified cortical areas respond selectively to particular movements, a defining physiological signature of mirror neurons in the monkey (Gallese et al. 1996). Movement selectivity is a crucial feature for mirror neurons’ proposed role in action understanding (Rizzolatti et al. 2001) and imitation learning (Buccino et al. 2004b). Mirror neurons are believed to act as a visual to motor integration mechanism where an observed movement representation is mapped onto the motor neurons used by the observer to execute the same movement. Successful mapping must, therefore, be accomplished in a movement-selective manner.

Using the rock-paper-scissors adaptation protocol we were able to address both of these concerns. Our results clearly demonstrate that a small number of cortical areas (aIFS, vPM, aIPS, sIPS, pIPS, and LO) contain neurons selective for both observed and executed movements (Figure 3). These movement-selective areas overlap well with areas exhibiting increased activity during imitation and movement observation as can be seen in the multi-subject averaged maps (Figure 4), in the ROI analysis results (Figure 5), and in a comparison with the literature (Table 1). The overlapping adaptation effects were, however, isolated to these particular cortical areas unlike the widely distributed effects associated with imitation and movement observation. Furthermore, we propose that an overlap of observed and executed movement selectivity is a superior criterion for identifying candidate mirror system areas because it assesses a fundamental feature of mirror neuron physiology as originally described in the monkey: movement selectivity. In contrast, mirror neuron responses to imitation have never been assessed in the monkey. It is possible that mirror neurons do not respond more strongly during imitation than during observation or execution, and that the fMRI responses observed during imitation are due instead to the activity of neurons that are involved in other cognitive processes engaged during imitation (e.g., working memory or attention). Note also that we found some cortical areas (the control ROIs in Figure 5) that exhibited no movement-selectivity but responded strongly during imitation. Our results, therefore, are an important validation that movement-selective neurons are contained in some (but not all) of the cortical areas active during imitation and movement observation.

Despite our claim that the adaptation protocol is a superior way of identifying mirror system areas, we do not claim that it is an exclusive measure of mirror neuron activity. It is possible that the overlapping visual and motor adaptation effects were generated by two separate (possibly intermingled) subpopulations of visual and motor neurons that adapted independently during repeated observation and execution of the movements. Cross-modal adaptation, in trials where the same movement was observed and then executed or executed and then observed, would have provided strong evidence that the visual and motor adaptation was taking place in a single subpopulation of visuomotor mirror neurons. Unfortunately, we did not find any cross-modal adaptation using our protocol (Figure 5B, Supplemental Figure 2B). There are two possible reasons for the lack of cross-modal adaptation. First, it is possible that mirror neurons simply do not adapt to cross modal repeats. The term ‘neural adaptation’ is used very broadly to refer to numerous different physiological mechanisms including presynaptic neurotransmitter depletion, postsynaptic receptor trafficking, postsynaptic receptor desensitization, and hyperpolarization leading to spike frequency adaptation (Zucker and Regehr 2002). One can draw a distinction between adaptation mechanisms where the stimulated neuron becomes less excitable regardless of the source of input (e.g., due to hyperpolarization) and mechanisms where the adaptation results from a reduced efficacy of the synapse(s) relaying the stimulation. The difference between these two classes of mechanisms is that in the former case a neuron would show adaptation upon repeated stimulation regardless of the source, whereas in the latter case the neuron would show adaptation only when stimulated via the same set of synapses that have adapted. Thus, the involvement of different adaptation mechanisms would lead to different predictions about whether or not cross-modal adaptation would be expected and it is entirely plausible that mirror neurons exhibit within-modality adaptation, but not cross-modal adaptation. A second possibility is that our protocol is not suitable for capturing the cross-modal adaptation effect. Different forms of neural adaptation take place at different time-scales lasting anywhere from a few hundred milliseconds to hours and days. As described above within and across modality adaptation might be expected to rely on different cellular mechanisms, which may act at different time-scales. It is possible that while the rate of movement observation and execution in our RPS experiment was suitable for generating within modality adaptation, it could have been too rapid or too slow to generate cross-modal adaptation. In any case, since this is a negative result we would not want to base any strong conclusions regarding the physiology of mirror neurons on the lack of cross-modal adaptation.

While keeping this caveat in mind note that three independent protocols expected to elicit mirror neuron activity (imitation, movement observation, and selectivity for observed and executed movements) suggest that mirror neurons may be localized in three new candidate areas (aIFS, sIPS, pIPS) on top of the two cortical areas (vPM and aIPS) commonly proposed as the human mirror system. LO might also be considered a candidate mirror area although it exhibited closely adjacent rather than overlapping adaptation (Figure 3), which was somewhat weaker in comparison to the other movement-selective ROIs (Figures 5, 8, and Supplementary Figure 2). Regardless of whether any of these areas contain mirror neurons, we have shown that they contain movement-selective neurons, which is by itself a notable feature of their physiology.

Movement Representation and Context

A movement may be represented at different levels of “abstraction” ranging from the trajectories and velocities of participating effectors to the goal or the intention of the movement, which may contain a symbolic meaning or emotional information depending on its context. An identical movement may, therefore, be represented by different neural populations in different contexts (Fogassi et al. 2005; Iacoboni et al. 2005; Umilta et al. 2001).

The visuomotor context of the game and the symbolic meaning of the movements may have modulated the activity of movement-selective neurons, resulting in stronger responses than what would have been observed without the game. In support of this conjecture, we found that only three areas (aIPS, M+S, and vPM) exhibited motor adaptation in both the RPS experiment and the instructed-movement experiment (during which the same movements were executed in a different context).

Similar adaptation protocols may enable the dissociation of selectivity for different movement representation dimensions, in different contexts. For instance, comparing adaptation to symbolic and non-symbolic hand movements or to movements with identical kinematics and different dynamics.

Conclusions

Using the rock-paper-scissors fMRI adaptation protocol we were able to successfully assess movement selectivity in the motor and visual domain simultaneously and identify six cortical areas containing neurons selective for both observed and executed movements: aIFS, vPM, aIPS, sIPS, pIPS, and LO. The RPS adaptation protocol provides a means for assessing human mirror system activity by movement selectivity, which is closer in rational to the original description of mirror neuron responses in monkeys and offers better isolation of movement-selective cortical areas. We, therefore, propose that future assessment of the human mirror system, for example, in autistic or schizophrenic individuals (Arbib and Mundhenk 2005; Dapretto et al. 2006; Williams et al. 2006), would benefit by taking advantage of this protocol. Finally, leaving mirror neurons aside, we suggest that assessing movement selectivity using motor repetition suppression may prove useful for the study of motor control in humans.

Supplementary Material

suppl fig 1

Supplementary Figure 1: Comparison of canonical and individually defined hemodynamic response functions (HRFs). Each graph shows two HRFs for a single subject (red, canonical HRF; blue, individually defined HRF). Next to each panel is the Pearson’s correlation coefficient of the two responses. The two HRFs were highly correlated in all of the subjects (mean r = 0.87 averaged across subjects and ROIs, SEM = 0.03).

To assess whether the results of the ROI analysis were dependent on the choice of a canonical HRF, we estimated an HRF individually for each subject and tested how similar it was to the canonical one. We began by averaging data from all of the voxels in the ten ROIs (see Methods) so as to generate a single time-course for each subject in each of the RPS games (runs). The four measured time-courses corresponding to the four RPS games were concatenated into a single long time-course and a “deconvolution” analysis was performed to estimate an HRF for each subject. This analysis relied on linear regression, solving an equation of the form y = Ax, where vector y was the measured fMRI time-course and vector x was the estimated individual-subject HRF containing 13 values. The model matrix A had 13 columns (corresponding to the number of time points in the estimated HRF). The first column of A contained a value of 1 at indices corresponding to the onset of movement trials, the second column contained a 1 at indices corresponding to the second time point, and so on. Thus the model was made up of diagonals of 13 ones corresponding to every trial where a movement was observed/executed and zeros everywhere else. The result of this analysis yielded an individual HRF for each subject (Supplementary Figure 2, blue curves).

The individual-subject HRFs were then compared with the canonical HRF. To make a fair comparison, we fit the fMRI time-courses using a canonical HRF. The model contained a row for every time point (2 per trial) and a single column. Neural activity was modeled as either “on” = 1 for all trials where a movement was executed/observed or “off” = 0 for all the blank trials (rest). The modeled neural activity was convolved with a canonical HRF (Boynton et al. 1996) and an estimate of a single response amplitude (beta value) was calculated for each subject using linear regression (solving an equation of the form y = Ax, where vector y was the measured fMRI time-course, x was the single response amplitude value, and A was the model containing a single column). Finally, we multiplied the canonical HRF with the computed response amplitude for each of the subjects to generate the estimated canonical response (Supplementary Figure 1, red curves). Technically, rather than simply scaling the canonical HRF, we performed a deconvolution analysis, as detailed above, on the model fits, to take into account influences of trial sequencing on the shape of the resulting HRF. Note that for a large enough number of trials this deconvolution analysis yields a copy of the canonical HRF scaled by the estimated response amplitude.

suppl fig 2

Supplementary Figure 2: Region of interest (ROI) analysis using hemodynamic response functions (HRF) estimated for each subject individually. We reanalyzed the rock-paper-scissors data in the six ROIs that exhibited movement selectivity (Figure 5) using each subject’s HRF (see Supplementary Figure 1). Top row. Visual and motor adaptation. Comparison of fMRI response amplitudes in four conditions: observed non-repeat (dark green), observed repeat (light green), executed non-repeat (dark orange), executed repeat (light orange). Bottom row. Cross-modal interactions. Comparison of fMRI response amplitudes in four conditions: observed-then-executed non-repeat (dark blue), observed-then-executed repeat (light blue), executed-then-observed non-repeat (dark purple), and executed-then-observed repeat (light purple). Error bars, SEM across subjects. Asterisks, statistically significant difference (p < 0.05, paired t-test). The results of this analysis are almost identical to those computed with the canonical HRF (Figure 5) demonstrating that the visual and motor adaptation effects were robust in all but one ROI. The only exception was LO where the visual adaptation effect was lost.