Skip to main content
NCBI home page
UKPMC Funders Author Manuscripts logoLink to UKPMC Funders Author Manuscripts
. Author manuscript; available in PMC: 2023 Mar 27.
Published in final edited form as: J Cogn Neurosci. 2011 Nov 29;24(3):575–587. doi: 10.1162/jocn_a_00173

The role of the parieto-premotor network in the processing of auditory-visual biological motion

SM Wuerger 1,✉, G Meyer 2, L Parkes 3, PA Lewis 4, A Crocker-Buque 5, R Rutschmann 6
PMCID: PMC7614374  EMSID: EMS172346  PMID: 22126670

Abstract

Our aim was to identify the cortical network involved in the integration of auditory and visual biological motion signals. We first determined the cortical regions of auditory and visual co-activation (Experiment 1); the conjunction analysis based on unimodal brain activations identified four regions consistent with an extended mirror-neurone system (MNS): the Middle Temporal area, Inferior Parietal Lobule (IPL), ventral premotor cortex (vPM) and the cerebellum. The brain activations arising from bimodal motion stimuli (Experiment 2) were then analysed within these regions of sensory convergence. Auditory footsteps were presented concurrently with either an intact visual point-light-walker (biological motion) or a scrambled point-light-walker; auditory and visual motion could either be congruent, i.e. same motion direction (both receding), or incongruent. Our main finding is that motion incongruency affects the activity in the parieto-premotor network (IPL and vPM) only if the visual point-light-walker is intact. Our results extend our current knowledge by showing that this network is not only involved in recognizing meaningful actions within a single modality, but also assimilates the information across the auditory and visual modality by comparing the incoming sensory input to an amodal representation.

Keywords: fMRI, integration, multisensorial, mirror neurone system

When an object moves in the real world, its movement is usually associated with a sensory signal in both the auditory and the visual modality (Baumann and Greenlee, 2007). These signals are processed in the sensory cortices before the information from both modalities is merged to yield a unified percept of the object in motion. Real object motion is subject to numerous physical constraints. The auditory and the visual perception systems have developed special processing strategies for ecologically valid motion stimuli, utilising some of the statistical properties of the real world. However, it remains unclear whether the integration across the auditory and visual modality is also subject to particular constraints, such as common motion direction. To investigate whether different integration mechanisms operate on ecologically valid motion sequences (such as biological motion) is the purpose of this study; more specifically we ask where in the cerebral cortex the auditory and visual motion signals converge and which of these areas of co-activation are likely to encode information about the ecological validity of multisensorial motion signals.

The cortical mechanisms underlying the processing of visual biological motion signals (such as point-light-walkers) have received much attention and a network encompassing occipital, parietal and temporal areas has been implicated in the processing of visual biological motion, including the posterior superior temporal gyrus and superior temporal sulcus (Bonda et al., 1996; Grossman and Blake, 2001; Grossman and Blake, 2002; Grossman et al., 2000; Howard et al., 1996; Pelphrey et al., 2003; Pelphrey et al., 2005; Servos et al., 2002; Thompson et al., 2005), the lingual gyrus (Vaina et al., 2001), motion-sensitive areas MT and MT+ (Grezes, 2001; Vaina et al., 2001), parietal areas (Bonda et al., 1996; Grezes, 2001; Vaina et al., 2001), and other areas including the amygdala (Bonda et al., 1996). The involvement of the pSTS/STG in biological motion processing is the most robust finding and consistent with macaque physiology (for a review see Puce and Perrett, 2003). The pSTS is also differentially activated by auditory footsteps (Bidet-Caulet et al., 2005), hence suggesting that pSTS may be a supramodal integration area for human biological motion.

More recent experiments suggest that, in addition to the STS, frontal (premotor) areas play an important role in the processing of visual biological motion (Schubotz and von Cramon, 2004) and studies using a clinical population confirm that the premotor cortex is necessary for intact biological motion perception (Saygin, 2007). From macaque physiology, we know that premotor areas are involved in the observation and planning of action (the ‘mirror neurone system’) and imaging studies on humans have confirmed this by demonstrating that premotor cortex is activated during action observation (Calvo-Merino et al., 2005). More importantly, in macaques, mirror neurones are frequently bimodal and can be driven by meaningful auditory and visual actions (for a review see Rizzolatti and Craighero, 2004); imaging studies on humans demonstrate that auditory and visual motion signals converge in the premotor cortex (Bremmer et al., 2001). Taken together, these studies suggest that the human premotor cortex is a good candidate for the perceptual integration of auditory and visual actions, such as human body motions.

The idea that the integration of multisensorial signals is constrained by the ecological validity of the combined percept is supported by behavioural evidence. Audio-visual integration is enhanced when the auditory and visual stimuli exhibit ecological validity (Arrighi et al., 2006; Arrighi et al., 2009; Saygin et al., 2008; Wuerger et al., 2010). Reaction time studies with biological motion stimuli (point-light walkers) showed that the integration of biological motion stimuli is constrained by the direction of the auditory and visual motion signals, whereas the integration of random motion sequences is not affected by the inconsistency of the auditory-visual motion direction (Brooks et al., 2007; Meyer and Wuerger, 2001; Wuerger et al., 2003). In the present imaging study we looked for neural correlates of these differential auditory-visual integration mechanisms for biological and non-biological motion signals that have been demonstrated behaviourally. As visual biological motion stimuli we used point-light walkers (Johansson, 1973) since they give a compelling percept of a person walking and are yet highly controllable; a ‘scrambled’ walker was obtained by randomising the starting position of each limb hence keeping the local motion signals intact but destroying the percept; the auditory stimulus consisted of synchronised footsteps. We focussed on the question whether the inconsistency of motion direction in the auditory and visual modality has a differential effect on the brain activity arising from the integration of biological (point-light walker and synchronised footsteps) and non-biological motion signals (‘scrambled’ walker and synchronised footsteps). Our hypothesis was that inconsistent motion across the auditory and visual modality (auditory: looming motion; visual: receding motion) should have a greater effect when both modalities signal biological motion. We first localised areas of co-activation, that is, brain regions where auditory and visual sensory signals converge (Experiment 1: Localiser). We then tested whether such differential neural activities were found within these regions of interests for biological compared to scrambled motion sequences (Experiment 2). Such differential neural activity must then be related to the ecological validity of the bimodal motion stimuli.

Materials and Methods

Experimental design

The present study consisted of two experiments. In experiment 1 (Localiser), subjects were presented with visual (point-light walkers), auditory (footsteps) or bimodal motion sequences and their task was to detect motion-in-depth (looming or receding motion). fMRI scans were performed to reveal cortical activations common to the auditory and the visual modality. These modality-unspecific cortical areas were determined by performing a conjunction analysis (Friston et al., 2005) of the unimodal (auditory only, visual only) brain activations. In experiment 2 we tested our main hypothesis by asking whether auditory-visual motion congruency (same versus different directions of motion in the two modalities) yields a differential effect on neural responses to biological motion in comparison to meaningless motion sequences. fMRI was performed while subjects were presented with incongruent and congruent bimodal motion sequences. The statistical analysis of the effect of motion congruency on biological versus non-biological motion is then performed within the regions of interest defined by experiment 1 (Szycik et al., 2008). Behavioural performances for both experiments were obtained at least one day prior to the scanning sessions under closely matched experimental conditions.

Subjects

Eighteen (15 naive and three authors) healthy volunteers (eight females) with normal or corrected-to normal vision participated in the experiments (mean age: 24 ± 5 years). All subjects gave written consent and were screened for MRI contra-indications. The study was approved by the Sefton Liverpool Research Ethics Committee.

Apparatus

Auditory stimuli were generated using a real-time signal processor (Tucker-Davis-Technologies, RM1; USA) and presented via MRI-compatible MR Confon Optime 1 headphones (MR Confon, Madeburg, Germany). Visual stimuli were generated using a visual stimulus generator (ViSaGe; Cambridge Research Systems LTD, Kent, UK) which was controlled by a standard PC (DELL Precision 390). Stimuli were back projected with a LCD projector (PANASONIC PT-L785U) onto a translucent circular screen, placed inside the scanner bore at 70 cm from the observer. The projector ran at a refresh rate of 60Hz and a resolution of 800 x 600 pixels. The TDT system and the ViSaGe system were interfaced via triggers to ensure that the auditory and visual stimuli were synchronised. For stimulus generation (auditory and visual) MatLab 7 (Mathworks) was used. Responses were acquired using an MRI-compatible response box. Behavioural data were obtained at least one day prior to the scanning session using a similar experimental setup (ViSaGe interfaced with a TDT system). Subjects were seated in a sound-proof booth (IAC 404-A), at a distance of 100cm from a CRT monitor (Mitsubishi DiamondPro 2070SB), running at a refresh rate of 60 Hz. Auditory stimuli were presented via conventional headphones (Sennheiser HD25SP). Reaction times were acquired using an infrared response box (Cambridge Research Systems Ltd, UK).

Stimuli

The auditory stimuli consisted of footsteps recordings and lasted 1.8 secs (4 footsteps) (diotic presentation, Fs=44100Hz, 64 dB(A)). The visual stimuli were either ‘point light walkers’ (PLW; biological motion) or ‘scrambled point-light walkers’ (SCR), subtending a visual angle of 3.8 deg (width) x 10 deg (height). The mean luminance of the display was fixed at 50 cd/m2 ; the contrast of the PLWs was 100% (black on grey). The PLW was defined by 13 points (indicating the joints and the head) representing the motion of the particular position of the body over four steps. Each point had a size of 3x3 pixels (0.09 x 0.09 deg) and one stimulus trial lasted 1.8 secs. The control stimuli, the ‘scrambled walkers’, were generated by using the same local limb movements as present in the PLW, but the starting positions of the limb movements were randomised within a kernel defined by the extent of the original figures, e.g. the knee movement could start near the elbow and vice versa. The advantage of this control stimulus is that it contains the same local motion signals (and hence the same spatio-temporal profile) as the point-light walker but is not recognised as a walker (Grossman and Blake, 2002). Auditory and visual motion stimuli could either be looming, receding, or neither looming nor receding. In the latter case the point-light-walker is walking ‘on a treadmill’ (‘No Motion’). Receding visual motion was generated by contracting the visual stimuli by a factor of 0.25; receding auditory motion was generated by linearly decreasing the amplitude of the footsteps by the same factor. Looming motion was generated by linearly increasing the amplitude/size. These stimuli yield compelling motion percepts (Johansson, 1973). We added dynamic visual noise to the visual stimuli in an attempt to roughly equate the noise levels in both modalities, since the scanner noise was always present in the auditory modality. New dynamic visual noise was generated on each trial. To match the behavioural study (this was a separate experiment conducted prior to the brain scans) as closely as possible with the scanning conditions, we recorded the scanner noise using an optical microphone (MR Confon; Manufacturer: Sennheiser, Germany) and then replayed the scanner noise in the sound-proof booth using loud speakers throughout the experiment. The auditory stimulus (footsteps) was presented via headphones.

Task and Procedure

We performed two experiments: in experiment 1 we presented unimodal motion stimuli (auditory footsteps (A), visual biological motion (VB), visual scrambled motion (VS)) and congruent bimodal stimuli (CONG_BIO=A+VB, CONG_SCR=A+VS). All five experimental stimuli conveyed the same motion direction (receding) and each experimental condition was presented 12 times. We included a control condition of no interest, which consisted of ‘no motion’ (walking on a treadmill) stimuli, presented either bimodally or unimodally. Each of the five control stimuli was presented four times and the task of the participant was to press a button when no motion was present. In addition, we included 20 null events at random times. The stimuli (experimental, control, null) were presented in a randomized order; each stimulus was presented for 1.8 sec and the average times between stimuli was 3 sec with a randomized jitter between -0.5 and +0.5 sec. Altogether, each scan of experiment 1 consisted of 100 trials and lasted just under 7 min.

In the main experiment (experiment 2), we tested whether auditory-visual congruency has a differential effect for biological motion (BIO) compared to scrambled motion (SCR). In the four experimental conditions, auditory and visual motion could either move in the same direction (both receding: CONG_BIO, CONG_SRC) or in different directions (auditory looming and visual receding: INCONG_BIO, INCONG_SCR). Within a single scan, each of the experimental stimuli was presented 16 times. We included a control condition of no interest, consisting of bimodal ‘no motion’ stimuli and each of the two control stimuli (BIO, SCR) was presented 12 times. 22 null events were included and all stimuli were presented in a randomized order. Altogether, each scan of experiment 2 consisted of 110 trials and lasted slightly longer than 7 min.

Each subject was in the scanner for less than one hour. First the participant performed a short practice experiment (less than 5 min) to familiarize herself/himself with the task again; then two scan sessions of experiment 1 were run (each about 7 minutes) followed by a structural scan (12 min) and by two sessions of experiment 2 (each about 7 minutes). For half of the participants the order of experiments 1 and 2 was reversed. In the scanner, the observers’ task was to press a button (with the right index finger) when there was no motion present (control condition) in the stimulus. No reaction times for the experimental (motion) conditions were obtained in the scanner. This ensures that the brain activity in response to the motion conditions is not confounded with the button presses.

Behavioural reaction time data for the experimental conditions were obtained prior to the scanning session, in a sound proof booth. Apparatus, stimuli and procedure were the same as in the scanning session; the only difference was that observers were asked to press one button when the stimulus contained any motion and another button when no motion was present in order to match the motor activity between the conditions. Participants were instructed to respond as fast and as accurately as possible.

Data Acquisition

Imaging was performed using a 3-Tesla MR body scanner (Siemens Trio, Erlangen, Germany) located at MARIARC, University of Liverpool. In the functional scans, Blood oxygen level-dependent (BOLD) responses were measured using a T2*-weighted echo planar imaging (EPI) sequence (echo time (TE) = 30 ms; volume repetition time (TR) = 2.0 s; resolution = 3 × 3 mm; number of slices = 33, interleaved; slice thickness = 3 mm; gap between slices = 0.3 mm; flip angle = 80°). Between the functional scans, 3D structural images of the whole brain were acquired using a T1-weighted MDEFT Sequence of 1 mm isotropic resolution.

Data Analysis

Preprocessing and statistical data analysis was performed using SPM5 (Wellcome Department of Imaging Neuroscience, London, UK, http://www.fil.ion.ucl.ac.uk/spm/) running under Matlab 7 (Mathworks, Natick, MA). Functional images of each participant were corrected for residual head motions, realigned to the first image and corrected for slice timing. Subsequently, all functional images were co-registered and normalized to the MNI-152 template and re-sampled to 2 × 2 × 2 mm3 spatial resolution. Spatial smoothing was applied to the functional images using an isotropic Gaussian kernel with a full-width half-max (FWHM) of 8 mm. A general linear model (GLM) was constructed for each participant in order to analyze the hemodynamic responses captured by the functional images. In all functional scans, an event-related design was used; regressors were generated by convolving unit impulses with the canonical hemodynamic function and also with the temporal derivative of this function (e.g. Henson et al., 2001). Experiment 1 was used to localize modality-unspecific motion-sensitive areas. The design matrix consisted of 10 regressors, the five experimental stimulus conditions (A, VB, VS, A+VB, A+VS, all depicting receding motion) and the five control conditions (A, VB, VS, A+VB, A+VS, all depicting a stationary ‘treadmill’ walker). Following (Bremmer et al., 2001), a second-level global null conjunction analysis (Friston et al., 2005) was used to reveal areas that respond significantly (whole brain family-wise error <0.05) to motion in the auditory and the visual modality. These regions of interest (ROIs) were extracted using the MarsBaR 0.38 toolbox for SPM (Brett et al., 2002). In experiment 2, a flexible factorial design was used to test our main hypothesis, namely whether there is an interaction between auditory-visual congruency (CONG vs INCONG) and motion type (BIO vs SCR). The design matrix consisted of 6 regressors, the four experimental conditions (CONG_BIO, CONG_SCR, INCONG_BIO, INCONG_SCR) and the two control conditions. Individual contrast estimates entered a t-test on second level, within the ROIs defined by experiment 1.

For reporting, stereotaxic Montreal Neurological Institute (MNI) coordinates are used. For the parietal lobe activations, the centres of gravity of suprathreshold regions were localized using the Anatomy toolbox for SPM (Eickhoff et al., 2005). For cortical areas where no probability maps were available in the Anatomy toolbox, we used the WFU_PickAtlas toolbox for SPM (Maldjian et al., 2003). For visualisation, the SPM T maps were superimposed with the selected threshold (family-wise error < 0.05) onto the population average landmark and surface-based (PALS-B12) standard brain (Van Essen, 2005) using Caret 5.6 (Van Essen et al., 2001); http://brainmap.wustl.edu/caret)).

To compute the correlations between the behavioural data (reaction times) and the brain activations we use the mean reaction times for each individual observer for each of the four experimental conditions (CONG_BIO, CONG_SCR, INCONG_BIO, INCONG_SCR) and the individual contrast values associated with the four experimental conditions. These contrast values are proportional to signal change and were extracted with MarsBaR (Brett et al., 2002).

Results

Localiser experiment: Areas of auditory-visual co-activation

To extract areas common to the auditory and visual modality, two conjunction analyses (‘Global Null’; Friston et al., 2005) were performed on the unimodal brain activations (ie. the conjunction of A and VBIO and the conjunction of A and VSCR). Each conjunction analysis revealed four areas with significant co-activations (family-wise error <0.05; Table1) common to the auditory and visual modality: the right precentral area (ventral premotor cortex; BA 6, bordering on BA 44), the right inferior parietal lobule (BA 7) on the border to the superior parietal lobule (SPL), the right middle temporal area (BA 39, bordering on BA 22 and BA 37) and the left cerebellum. Similar areas of co-activations emerged irrespective of whether the conjunction was computed between auditory footsteps (A) and the point-light walker (VBIO) or between the audio signal and the scrambled point-light walker (VSCR). Table 1 depicts the label of the cortical location, the type of conjunction (A ∩ VBIO or A ∩ VSCR), the MNI coordinates, and the number of voxels of this particular area. Both T and Z values are given; all neural activations are significant at p<0.05 (corrected for multiple comparisons). Figure 1a shows the SPM T maps of the conjunction analysis (group results) superimposed on an inflated standard brain; Figure 1b shows the saggital and coronal views of the four modality-unspecific regions. The co-activity in the premotor cortex (Precentral), the Inferior Parietal Lobule and area MT is lateralised in the right hemisphere; common activity in the cerebellum is only present in the left hemisphere. Location (MNI) and number of the significant voxels are given in Table 1. The unimodal activations underlying the conjunction analysis are shown in Table S1 (Supporting material).

Table 1. Conjunction analysis revealing activations common to the auditory and the visual modality (Exp 1).

Location Localiser Position (MNI) Voxels T Z pFWE
Frontal Lobe
BA6 R Precentral (vPM) A ∩ VBIO 56 6 40 152 3.74 5.53 0.002
BA6 /44 R Precentral (vPM) A ∩ VSCR 48 4 32
48 0 42		 521 4.89
4.20		 7.0
6.12		 <0.001
<0.001
Parietal Lobe
BA7 R Inferior Parietal
Lobule
(hIP3: 40%;
SPL(7PC): 30%;
SPL (7A): 20%)		 A ∩ VBIO 32 -52 52
36 -44 54		 207 4.46
3.74		 6.46
5.53		 <0.001
0.002
BA7 R Inferior Parietal
Lobule
(hIP3: 30%; SPL
(7PC): 30%; hIP1: 10%)		 A ∩ VSCR 32 -50 50
40 -40 52		 282 4.66
3.68		 6.70
5.45		 <0.001
0.002
Temporal Lobe
BA39 R Top Temporal A ∩ VBIO 54 -54 6 10 3.33 5.00 0.020
BA39 R Top Temporal A ∩ VSCR 54 -54 6 7 3.27 4.92 0.029
L Cerebellum A ∩ VBIO -32 -70 -20 47 3.54 5.28 0.006
L Cerebellum A ∩ VSCR -30 -74 -20 12 3.21 4.85 0.040
Open in a new tab

Note: The conjunction analysis revealed four areas of auditory-visual co-activation (family-wise error < 0.05). ‘A ∩ VBIO’ refers to the conjunction between the brain activations in response to auditory footsteps (A) and the brain activations in response to the visual point-light-walker (VBIO); ‘A ∩ VSCR’ refers to the conjunction analysis based on auditory footsteps and the scrambled point-light walker (VSCR). The conjunction analysis was performed using SPM5. For anatomical labelling of premotor cortex, the border between dorsal and ventral premotor cortex was assumed at a Z level of 50 in Tailarach coordinates (Rizzolatti & Craighero, 2004); we converted the Tailarach coordinates into MNI coordinates for our analysis.

Figure 1.

Figure 1

Open in a new tab

Experiment 1. Conjunction analysis for auditory footsteps and biological visual motion (A ∩ VBIO; left column) and auditory footsteps and scrambled visual motion (A ∩ VSCR; right column). Four regions of neural activity common to the auditory and visual modality were revealed for pFWE < 0.05 (cf Table1). (a) SPM T maps are depicted on an inflated PALS-B12 standard brain (Caret 5.6; von Essen, 2001). Similar modality-unspecific areas are revealed for both conjunctions. (b) The SPM T maps are projected onto the average of the normalised brains of all 18 participants. The colour represents the T-values for each cortical location as indicated by the key on the right.

Bimodal activations

Differential effects of auditory-visual motion congruency on biological and scrambled motion

In the main experiment (Experiment 2) we measured activations for the four bimodal conditions: congruent biological motion (CON BIO), incongruent biological motion (INCON BIO), congruent scrambled motion (CON SCR) and incongruent scrambled motion (INCON SCR). We tested the hypothesis that incongruent auditory-visual motion has a differential effect on biological and scrambled motion, when compared to congruent motion. We therefore determined for each region of interest (determined in experiment 1) whether the activation (INCON-CON) is different from zero for both kinds of motion stimuli (BIO and SCR). The ROIs were defined by either the conjunction of auditory and visual biological motion (A ∩ VBIO) or by the conjunction of auditory and visual scrambled motion (A ∩ VSCR); statistical tests were performed for both sets of ROIs. Since the results were identical, we only show the graphs for A ∩ VBIO; the figures for the other conjunction analysis (A ∩ VSCR) are in the supporting material.

Our main finding is that motion incongruency yields a significant increase in BOLD activity in the precentral cortex (BA 6), and to a lesser extent, in IPL (BA 7) only if both modalities convey a biological motion signal. Figure 2 shows the ROIs defined either by biological motion (A ∩ VBIO; solid red colour) or by scrambled motion (A ∩ VSCR; transparent RED colour; not visible in hIP3 and MT since coincident with BIO localiser) superimposed onto an MNI normalized flat map template (van Essen et al. 2001). The MNI coordinates of the ROIS are shown in Figure 1 and Table 1. BOLD contrasts within each ROI were extracted and the contrast differences (INCONGRUENT – CONGRUENT) for biological (green) and scrambled (purple) motion are shown. In the precentral cortex, incongruent auditory-visual motion leads to a larger BOLD contrast increase when both modalities convey a biological motion signal (INCON BIO – CON BIO; p=0.04); a similar trend (p=0.066) was found in the inferior parietal lobule. The differential activation in MT and the cerebellum were not different from zero (at p<0.1); no significant differential activations were found for scrambled motion (INCON SCR – CON SCR) in any of the four ROIs. This differential effect of motion incongruency on biological motion can also be seen in the whole brain group analysis: incongruent motion is associated with an increased precentral (BA 6) activity for biological motion only (Supporting material; Figure S1), and also only in the right hemisphere (cf Figure S1a: RH with S1b: LH). The average BOLD contrasts for the four experimental conditions (CON BIO, INCON BIO, CON SCR and INCON SCR) extracted from the four ROIs are also shown in Figure S2a (Supporting Material). No activation differences were found in area MT (BA 39) between scrambled and biological motion (Figure S2a; Supporting material).

Figure 2.

Figure 2

Open in a new tab

The locations of the ROIs defined either by biological motion (A ∩ VBIO; solid red colour) or by scrambled motion (A ∩ VSCR; transparent RED colour; not visible in hIP3 and MTG since co-incident with BIO localiser) are superimposed onto MNI normalized flat map template (van Essen et al. 2001). The fourth region is located in the Cerebellum and is not shown here. The black lines represent the borders of the Brodmann Areas from the PALS-B12 atlas. The bar graphs show the contrast difference (INCONGRUENT – CONGRUENT) for biological (green) and scrambled (purple) motion. In the precentral area (vPM), incongruent auditory-visual motion leads to significant increase in the BOLD contrast when both modalities convey a biological motion signal as opposed to the scrambled condition (p=0.041; cf Table2); a similar trend is found IPL (p=0.066). None of the contrast differences (INCON-CON) are different from zero for scrambled motion (cf Table2). The BOLD contrast differences (INCON – CON) shown above were obtained using the ROIs defined by A ∩ VBIO. Almost identical results are obtained when the analysis is performed in the ROIs defined by A ∩ VSCR (cf. Table2), since the ROIs are largely overlapping.

These BOLD contrast differences (INCON – CON) shown in Figure 2 were obtained using the ROIs defined by A ∩ VBIO. Almost identical results are obtained when the analysis is performed in the ROIs defined by A ∩ VSCR (cf. Table2 and Figure S3 in the Supporting material), since the ROIs are largely overlapping. Table 2 shows the BOLD contrasts, the t-values and the corresponding p values for both localisers. No significant activation differences were found for scrambled motion, that is, the difference ‘INCON SCR – CON SCR’ does not reach significance in any of the four ROIs. The BOLD contrasts for the four experimental conditions (CON BIO, INCON BIO, CON SCR and INCON SCR) extracted from the four ROIs defined by A ∩ VSCR are also shown in Figure S2b (Supporting Material). No activation differences were found in area MT (BA 39) between scrambled and biological motion (Figure S2b; Supporting material).

Table 2. Differential Activations for biological and scrambled motion in ROIs.
Location Localiser INCON BIO – CON BIO INCON SCR – CON SCR
Contrast T p Contrast T p
Frontal
BA6 R / Precentral A ∩ VBIO 1.25 1.75 0.041 -0.48 -0.84 0.799
BA6/44 R / Precentral A ∩ VSCR 1.30 1.92 0.028 -0.16 -0.30 0.618
Parietal
BA7 R / IPL A ∩ VBIO 1.24 1.51 0.066 0.52 0.79 0.216
BA7 R / IPL A ∩ VSCR 1.14 1.45 0.075 0.47 0.74 0.229
Temporal
BA39 R /MT A ∩ VBIO 0.20 0.30 0.380 0.47 0.92 0.178
A ∩ VSCR 0.23 0.35 0.362 0.59 1.14 0.128
Cerebellum L A ∩ VBIO -0.67 -0.78 0.781 -0.31 -0.46 0.677
Cerebellum L A ∩ VSCR -1.06 -1.14 0.871 -0.04 -0.06 0.997
Open in a new tab

Note. No significant activation differences were found for scrambled motion, that is, the difference ‘INCON SCR – CON SCR’ does not reach significance in any of the four ROIs. Only when the both modalities signal biological motion, significant differential activations are found in the premotor cortex (BA 6) and to a lesser extent in IPL (BA 7).

In summary, our ROI analysis revealed significant interactions in the ventral premotor area (precentral; BA 6) and to a lesser extent in the parietal lobe (IPL; BA 7) in the right hemisphere only: incongruent motion in the auditory and visual modality leads to an increase in the activation in these areas only if the auditory and visual modality depict biological motion signals. No increase in activation is found for incongruent scrambled motion in comparison to congruent scrambled motion in any of the four regions of interest (cf Table2). These differential activations arise from the fact that the brain activity is increased when observers are presented with consistent scrambled motion, compared to consistent biological motion (cf TableS2; supporting material).

Reaction times and their neural correlates

Figure 3 shows the differences in reaction times (INCON – CON) obtained for each observer prior to the scanning session (see Methods for details). For biological motion, observers are slowed down when the auditory and the visual modality signal different directions of motion (looming /receding motion). This difference in reaction times (incongruent – congruent) is 74 msec (SEM=31 msec) for biological motion which differs significantly from zero (t statistic = 2.34513; df=17; p=0.031). On the other hand, when the visual point-light-walker was scrambled, there is no significant reaction time difference between incongruent and congruent motion sequences (RT difference = -32 msec; p=0.6). Observers are not slowed down by incongruent information from the auditory and visual modality if the visual motion sequences do not depict biological motion, which is consistent with Brooks et al (2007).

Figure 3.

Figure 3

Open in a new tab

Behavioural data. Reaction time differences (incongruent AV – congruent AV motion) are plotted for biological and scrambled motion signals. Incongruency of auditory and visual motion signals has an effect only when the audio-visual sequences depict biological motion (p=0.031); for scrambled motion no significant difference is observed (p=0.6) between the incongruent and congruent condition. Error bars indicate standard errors of the mean.

Comparison of the differential brain activations (Figure 2) with the differential reaction times (Figure 3) reveals that both the precentral cortex (BA 6) and, to a lesser extent, the inferior parietal lobule (BA 7) show similar patterns, i.e. an increase in reaction times due to inconsistent information from the auditory and the visual modality is associated with an increased activation in these areas. To further investigate the association between brain activations and reaction times, we calculated the correlation between the individual brain activations within the ROIs and the individual reaction times for all four experimental conditions. Table 3 shows the correlation coefficients for all ROIs; only the activation in the vPM (precentral; BA 6) is significantly correlated with reaction times (r ~0.3; p<0.05).

Table 3. Correlations between reaction times and brain activations.
Location Localiser Pearson Correlation
Corr coeff Prob
Frontal
BA6 R / Precentral A ∩ VBIO 0.29 0.015
BA6/44 R / Precentral A ∩ VSCR 0.32 0.006
Parietal
BA7 R / IPL A ∩ VBIO 0.17 0.163
BA7 R / IPL A ∩ VSCR 0.15 0.196
Temporal
BA39 R / MT A ∩ VBIO -0.13 0.259
A ∩ VSCR -0.16 0.185
Cerebellum L A ∩ VBIO 0.15 0.203
Cerebellum L A ∩ VSCR 0.13 0.259
Open in a new tab

Note. The correlation coefficients between contrast level (which is proportional to the BOLD signal) in the four ROIs and the mean reaction times are shown. Only the activation in the precentral area (BA 6) is significantly correlated with reaction times (r ~0.3; p<0.05). Importantly, note that reaction time data was acquired outside the scanner prior to the experiment.

Discussion

It is well known that simple observation of human body movements activates a parieto-premotor network (‘Mirror neurone system’) ; these areas are coding incoming auditory or visual sensory inputs by translating these inputs into an ‘action vocabulary’ (Rizzolatti et al., 2001). Here we suggest that this parieto-premotor network not only translates the sight (or sound) of an action into a corresponding motor representation (e.g. Keysers and Gazzola, 2006) but that this network is also sensitive to incongruencies between the auditory and visual sensory inputs if the auditory and visual signals depict biological movements (e.g. a human walker). The idea that the MNS contains an action vocabulary of visually presented actions has recently been confirmed using TMS (Schwarzbach et al., 2009). Our results suggest that this action vocabulary is amodal: the MNS integrates auditory and visual information by comparing the combined auditory-visual input to an existing motor vocabulary

Our aim was to identify the cortical network that differentiates between biologically plausible and implausible auditory-visual inputs. We first determined the cortical regions where auditory and visual motion signals converge by performing a conjunction analysis based on unimodal brain activations (Experiment 1: Localiser). The regions identified by this conjunction analysis are consistent with an extended mirror-neurone system (Pineda, 2008): MTG, Inferior Parietal Lobule (IPL), and the ventral premotor cortex (vPM). The brain activations arising from bimodal (auditory-visual) motion stimuli (Experiment 2) were then analysed within these regions of sensory convergence. We presented observers with four different auditory-visual motion sequences: a visual point-light walker coupled with auditory footsteps, both signalling receding motion (congruent motion direction); a receding point-light walker coupled with looming footsteps (incongruent motion direction), a scrambled point-light walker coupled with either congruent or incongruent footsteps. We report two findings: (1) Abolishing the visual walker by scrambling the local motion vectors increases activity in the right parieto-premotor MNS (IPL and vPM).(2) Our novel finding is that the incongruency in the auditory and visual motion direction of the walker only affects the activity in the parieto-premotor MNS (IPL and vPM) if the visual walker is intact. The increased activity in IPL and vPM is not simply due to incongruent motion directions in the auditory and visual modality (looming vs receding) but is linked to the ecological validity of the motion signals. We therefore conclude that the parieto-premotor network plays an important role not only in recognising biologically meaningful motion sequences in the visual and auditory modality in isolation, but also in assessing whether the combined auditory-visual input is likely to arise from biological motion.

Areas of auditory and visual convergence in the right hemisphere

In experiment 1 we determined the cortical regions that respond to both auditory and visual motion stimuli. Both conjunctions (A ∩ VBIO or A ∩ VSCR) revealed similar regions of co-activation: area MT (BA 39 bordering on BA 22 and BA 37), vPM (precentral, BA 6) and IPL (BA7; at the border to SPL) in the right hemisphere and the cerebellum in the left hemisphere (see Table1, also TableS2 in supporting material showing unimodal activations). The strong right-lateralisation of brain activity in response to auditory footsteps is consistent with the findings that auditory motion-in-depth (looming/receding) is encoded in the right hemisphere (Seifritz et al., 2002), in particular in the right premotor cortex (Schubotz and von Cramon, 2002). Brain activation for the (visual) point-light walker was also right-lateralised, in accordance with experiments by Pelphrey et al. (2005). Lateralisation of auditory-visual co-activation in the right ventral intraperietal cortex and premotor cortex has also been found for random visual and auditory motion stimuli (Bremmer et al., 2001); the right IPL has been identified as a region of higher-level visual motion processing (Claeys et al., 2003). In our experiments, the intact as well as scrambled point-light walkers were embedded in dynamic visual noise (to ensure comparable difficulty level to the auditory footsteps) which might also contribute to the lateralisation in the right hemisphere (Decety et al., 1997).

Sensory convergence in the parieto-premotor network

All three cortical ROIs identified as areas of auditory and visual convergence (Experiment 1; Table1; Figure 1) are known to be part of the ‘mirror neurone system’ (Rizzolatti and Craighero, 2004). vPM (Rizolatti et al., 1996; Decety et al., 1997; Iacobini et al., 1999) and IPL neurones (Buccino et al, 2001) are activated by the passive observation of actions. This core mirror neurone system (IPL, vPM) is thought to receive input from the MTG/pSTS and pSTS neurones share with traditional mirror neurones their selectivity for biological motion, such as body, hand and lip movements (Barraclough et al., 2005; Puce and Perrett, 2003) and are engaged in the perception of animacy (Schultz et al., 2005). The particular MT region identified by our conjunction analysis (B39/BA22/BA19) is close to areas engaged in the processing of body motions (Puce and Perrett, 2003) and is sometimes labelled as pSTS due to functional similarities with pSTS (Materna et al., 2008); in this study we will refer to it as MT region. While all three areas, MT, IPL and vPM play a significant role in passive observation, imitation, and motion imagery (Hamzei et al., 2002), their connectivity is still a matter of debate (Bien et al., 2009). A simple common framework for action observation and imitation (Stanley and Miall, 2007) starts with a visual representation of action in the STS, an area which is active during observation but not execution (Barraclough et al., 2005). Visual information is then passed on to the IPL which codes for the predicted outcome of the action and, subsequently, the intended action is translated into a motor programme in vPM; an efferent copy of the planned action then returns to STS where it is compared to the original visual representation. In addition, direct bi-directional connections exist between the MT/pSTS and both the vPM and IPL (for a review, see Pineda, 2008). Our localiser experiment confirms that MT, IPL and vPM are not only involved in visual action observation but also in integrating information about actions from the auditory and visual modality. In addition to the MT area and the parieto-premotor network, we identified the cerebellum as an area of sensory convergence. The cerebellum may play a role in converting the visual representation into a motor codes, the ‘inverse model’ (Miall, 2003; Stanley and Miall, 2007) by receiving information from the parietal lobe and forwarding it to the premotor cortex. The observed auditory-visual co-activation suggests that the involvement of the cerebellum in the inverse model may not be restricted to visual representations.

Increased activity for incongruent auditory-visual biological motion signals in ventral premotor cortex and IPL

In our main experiment (experiment 2) we compared the brain activation resulting from congruent (same motion direction in the auditory and visual modality) with the activation resulting from incongruent motion (different motion direction in the auditory and the visual modality) within the areas of auditory-visual co-activation (derived in experiment 1). Incongruent auditory-visual motion resulted in an increased brain activity only when both modalities signal biological motion; for scrambled visual motion, congruent and incongruent AV motion is associated with the same brain activations (Figure 2; supplementary material Figure S1a and S2). This differential effect of motion incongruency is found in two of the four ROIs: in the ventral premotor cortex (precentral; BA 6) and, to a lesser extent, in the inferior parietal lobule (IPL; BA 7), both being core components of the mirror neurone system (MNS). The vPM plays a role not only in visual action observation and action imagery (Schubotz and von Cramon, 2001) but also responds to auditory actions (Bidet-Caulet et al., 2005; Kaplan and Iacoboni, 2007; Schubotz and von Cramon, 2002). A common vPM region is activated by visual motion imagery (Grafton et al. (1996)), the observation of biologically meaningful actions (Bien et al., 2009) and the observation of meaningless (non-biological) sequences (Schubotz and von Cramon, 2004), consistent with our findings that both biological and scrambled motion leads to vPM activation (Figure S1 and Table S2, first row). Schubotz et al (2002, 2004) concluded that the vPM is able to generate short-term action templates and that the vocabularly of motor acts stored in vPM is flexible and not innate, consistent with the findings by Calvo-Merino (2005). In our experiment we find an increased premotor activity for incongruent biological motion in comparison to congruent biological motion (Figure 2; Figure S2); this increased premotor activity is associated with longer reaction times (Figure 3; Table3). Increased right PM activity and associated increased reaction times have also been reported for incongruent visuomotor conditions (Blakemore and Frith, 2005; Grezes et al., 2003) and for directionally imcompatible or antiphase limb movements (de Jong et al., 2002; Wenderoth et al., 2004). Increased right PM activity (Jeannerod, 2001) and IPL activity (Farrer et al., 2003) is therefore likely to reflect conflicting or incompatible signals within or across sensory modalities as well as incompatible motor patterns.

The traditional view of how the MNS understands observed actions is that the sensory input is compared with the observer’s motor repertoire (Rizzolatti and Craighero, 2004; Vogt et al., 2007): to recognise visual actions it is necessary to simulate the corresponding motor programs that would result in the production of the same actions. One possible explanation for the increased premotor activity for incongruent biological motion, is, in accordance with Schubotz et al. (2004), the generation of novel motor templates based on the (inconsistent) sensory inputs. Since in this experimental condition, the auditory system signals a looming walker and the visual system signals a receding walker, no stored amodal action template provides a match to the bimodal sensory inputs hence necessitating the need for the generation of novel motor patterns. Congruent biological motion, on the other hand, yields auditory and visual motion signals that are likely to be matched to a single existing amodal template in the observer’s motor repertoire, yielding less premotor activity and shorter reaction times (cf Figure2 and Figure 3). An alternative explanation is that the incongruent auditory-visual walker triggers two motor templates, one for a receding walker (based on the visual input) and one for a looming walker (based on the auditory input). Either explanation predicts increased activity in premotor cortex for incongruent biological motion only.

Activity in vPM and IPL is also increased when the visual point-light-walker is not intact (scrambled point-light-walker (SCR) versus intact point-light walker (BIO); Figure S1a; Table S2, upper row; (see also Thompson et al., 2005). While neurones in vPM are likely to respond to the components of the scrambled point-light-walker such as legs, arms etc, the overall configuration is unlikely to match an existing action template hence generating more activity in right vPM. The involvement of the vPM in human body processing has been shown using TMS: the body inversion effect is absent when TMS is applied in this area, hence suggesting that the vPM is involved in configural processing of human body shapes (Urgesi et al., 2007). In line with our findings, increased right-lateralised vPM and IPL activity has been reported during the observation of meaningless hand sequences (Decety et al., 1997; Decety and Grezes, 2006; Grezes et al., 1999) ; parietal areas (BA 7) may have a role in selecting and monitoring motion sequences with online reference to a working memory in the right premotor cortex (Sadato et al., 1996). The increased joint activation of the right IPL and vPM in response to scrambled point-light walkers is consistent with the role of the right MNS in the processing of novel and complex visual stimuli (Schubotz and von Cramon, 2002). Such an increase in stimulus complexity and novelty can be brought about by conflicting information within or across modalities. This suggests that the right parietopremotor MNS is not only involved in recognizing meaningful actions within a single modality, but assimilates the information across the auditory and visual modality by comparing it with an amodal motor termplate, possible residing in the premotor area (Sadato et al., 1996 ; Schwarzbach et al., 2009).

Specialised neural machinery for biological motion?

Numerous studies have shown an increased activity for visual biological motion in pSTS (for a review see Puce and Perrett, 2003) and also identified pSTS as an area for the integration of auditory and visual biological motion signals. Our conjunction analysis (Figure 1) did not identify pSTS as an area of sensory convergence, but area MT (BA 39, bordering on BA22 and BA 37), IPL (BA 7) and vPM (BA 6). Within these areas of auditory-visual co-activation, activity for the intact point-light-walker was less (vPM, IPL) or equal (MT) to the activity in response to the scrambled walker (Figure S2a,b). One significant methodological difference between our study and previous studies using PLW was that we used looming and receding PLWs (instead of a PLW walking on a ‘treadmill’) hence signalling motion-in-depth which is not a stimulus feature STS is very sensitive to (Perrett et al., 1990). The task of our observers was to judge whether there was any motion-in-depth present as opposed to categorising or identifying the biological motion; our task therefore also favours the involvement of the vPM (Kakei et al., 2001; Ochiai et al., 2005; Schubotz and von Cramon, 2002). Finally, to equate the auditory and visual PLWs in difficulty, we added dynamic noise to the visual PLWs which might also bias the activation towards area MT and the right parieto-premotor network (Bremmer et al., 2001; Pelphrey et al., 2005).

The increased activity in the right MNS for scrambled compared to intact point-light walkers is in line with more recent imaging studies showing increased right-lateralised activity for incoherent vs coherent action sequences in the right vPM (Bien et al., 2009). A right-lateralised decrease in neural activity when novel stimuli become more familiar via training or prolonged observation (Downar et al., 2002; Vogt et al., 2007) is consistent with the idea that learned meaningless movements generate less cortical activity than unlearned meaningless sequences since the neural population that represents the familiar stimuli have become more selective during learning. Biological motion stimuli are special configurations of highly familiar local limb movements; while numerous neurones are likely to respond to individual limb movements (such as contained in a scrambled PLW), a small population of neurones is likely to respond to the particular configuration of limb movements depicted in an intact PLW.

The extent to which novel sensorimotor representations can be incorporated into an existing action vocabulary is yet unknown but there is convincing evidence that the action vocabulary can be changed by experience (Calvo-Merino et al., 2005). Based on our findings we predict a decrease in right MNS activity when novel movement patterns become as familiar as intact biological motion stimuli are by training or prolonged exposure. Our current findings are consistent with the idea that the right parieto-premotor network (IPL, vPM) is involved in the processing of body movements by comparing sensorimotor representations of familiar body movements with incoming sensory input. Our results extend our current knowledge by showing that the right parieto-premotor network is not only involved in recognizing meaningful actions within a single modality, but compares the information across the auditory and visual modality with an amodal motor template, possible residing in the premotor area (Sadato et al., 1996; Schwarzbach et al., 2009).

Supplementary Material

Supporting Material

Acknowledgements

This work was supported by the Wellcome Trust (grant numbers 082831; 080205). Part of this work has been presented at the European Conference on Visual Perception, held in Regensburg, August 2009. We thank Mark Greenlee for his support in the data analysis.

Contributor Information

SM Wuerger, Email: s.m.wuerger@liverpool.ac.uk.

G Meyer, Email: georg@liv.ac.uk.

L. Parkes, Email: laura.parkes@manchester.ac.uk.

PA Lewis, Email: P.Lewis@manchester.ac.uk.

A. Crocker-Buque, Email: A.Crocker-Buque@sms.ed.ac.uk.

R. Rutschmann, Email: roland.rutschmann@psychologie.uni-regensburg.de.

References

  1. Arrighi R, Alais D, Burr D. Perceptual synchrony of audiovisual streams for natural and artificial motion sequences. Journal of Vision. 2006;6:260–268. doi: 10.1167/6.3.6. [DOI] [PubMed] [Google Scholar]
  2. Arrighi R, Marini F, Burr D. Meaningful auditory information enhances perception of visual biological motion. Journal of Vision. 2009;9:1–7. doi: 10.1167/9.4.25. [DOI] [PubMed] [Google Scholar]
  3. Barraclough NE, Xiao D, Baker CI, Oram MW, Perrett DI. Integration of Visual and Auditory Information by Superior Temporal Sulcus Neurons Responsive to the Sight of Actions. Journal of Cognitive Neuroscience. 2005;17:377–391. doi: 10.1162/0898929053279586. [DOI] [PubMed] [Google Scholar]
  4. Baumann O, Greenlee MW. Neural Correlates of Coherent Audiovisual Motion Perception. Cereb Cortex. 2007;17:1433–1443. doi: 10.1093/cercor/bhl055. [DOI] [PubMed] [Google Scholar]
  5. Bidet-Caulet A, Voisin J, Bertrand O, Fonlupt P. Listening to a walking human activates the temporal biological motion area. NeuroImage. 2005;28:132. doi: 10.1016/j.neuroimage.2005.06.018. [DOI] [PubMed] [Google Scholar]
  6. Bien N, Roebroeck A, Goebel R, Sack AT. The Brain’s Intention to Imitate: The Neurobiology of Intentional versus Automatic Imitation. Cereb Cortex. 2009:bhn251. doi: 10.1093/cercor/bhn251. [DOI] [PubMed] [Google Scholar]
  7. Blakemore SJ, Frith C. The role of motor contagion in the prediction of action. Neuropsychologia. 2005;43:260–267. doi: 10.1016/j.neuropsychologia.2004.11.012. [DOI] [PubMed] [Google Scholar]
  8. Bonda E, Petrides M, Ostry D, Evans A. Specific involvement of human parietal systems and the amygdala in the perception of biological motion. Journal of Neuroscience. 1996;16:3737. doi: 10.1523/JNEUROSCI.16-11-03737.1996. [DOI] [PMC free article] [PubMed] [Google Scholar]
  9. Bremmer F, Schlack A, Shah NJ, Zafiris O, Kubischik M, Hoffmann K, Zilles K, Fink GR. Polymodal motion processing in posterior parietal and premotor cortex: a human fMRI study strongly implies equivalencies between humans and monkeys. Neuron. 2001;29:287–296. doi: 10.1016/s0896-6273(01)00198-2. [DOI] [PubMed] [Google Scholar]
  10. Brett M, Anton J-L, Valabregue R, Poline JB. Region of interest analysis using an SPM toolbox; 8th International Conference on Functional Mapping of the Human Brain; Sendai, Japan. 2002. [Google Scholar]
  11. Brooks A, van der Zwan R, Billard A, Petreska B, Clarke S, Blanke O. Auditory motion affects visual biological motion processing. Neuropsychologia. 2007;45:523–530. doi: 10.1016/j.neuropsychologia.2005.12.012. [DOI] [PubMed] [Google Scholar]
  12. Calvo-Merino B, Glaser DE, Grezes J, Passingham RE, Haggard P. Action observation and acquired motor skills: an FMRI study with expert dancers. Cereb Cortex. 2005;15:1243–1249. doi: 10.1093/cercor/bhi007. [DOI] [PubMed] [Google Scholar]
  13. Claeys KG, Lindsey DT, De Schutter E, Orban GA. A higher order motion region in human inferior parietal lobule: Evidence from fMRI. Neuron. 2003;40:631–642. doi: 10.1016/s0896-6273(03)00590-7. [DOI] [PubMed] [Google Scholar]
  14. de Jong BM, Leenders KL, Paans AMJ. Right Parieto-premotor Activation Related to Limb-independent Antiphase Movement. Cereb Cortex. 2002;12:1213–1217. doi: 10.1093/cercor/12.11.1213. [DOI] [PubMed] [Google Scholar]
  15. Decety J, Grezes J, Costes N, Perani D, Jeannerod M, Procyk E, Grassi F, Fazio F. Brain activity during observation of actions. Influence of action content and subject’s strategy. Brain. 1997;120:1763–1777. doi: 10.1093/brain/120.10.1763. [DOI] [PubMed] [Google Scholar]
  16. Decety J, Grezes J. The power of simulation: imagining one’s own and other’s behavior. Brain Res. 2006;1079:4–14. doi: 10.1016/j.brainres.2005.12.115. [DOI] [PubMed] [Google Scholar]
  17. Downar J, Crawley AP, Mikulis DJ, Davis KD. A Cortical Network Sensitive to Stimulus Salience in a Neutral Behavioral Context Across Multiple Sensory Modalities. J Neurophysiol. 2002;87:615–620. doi: 10.1152/jn.00636.2001. [DOI] [PubMed] [Google Scholar]
  18. Eickhoff SB, Stephan KE, Mohlberg H, Grefkes C, Fink GR, Amunts K, Zilles K. A new SPM toolbox for combining probabilistic cytoarchitectonic maps and functional imaging data. NeuroImage. 2005;25:1325–1335. doi: 10.1016/j.neuroimage.2004.12.034. [DOI] [PubMed] [Google Scholar]
  19. Farrer C, Franck N, Georgieff N, Frith CD, Decety J, Jeannerod M. Modulating the experience of agency: A positron emission tomography study. NeuroImage. 2003;18:324–333. doi: 10.1016/s1053-8119(02)00041-1. [DOI] [PubMed] [Google Scholar]
  20. Friston KJ, Penny WD, Glaser DE. Conjunction revisited. NeuroImage. 2005;25:661–667. doi: 10.1016/j.neuroimage.2005.01.013. [DOI] [PubMed] [Google Scholar]
  21. Grafton ST, Arbib MA, Fadiga L, Rizzolatti G. Localization of grasp representations in humans by positron emission tomography. Experimental Brain Research. 1996;112:103–111. doi: 10.1007/BF00227183. [DOI] [PubMed] [Google Scholar]
  22. Grezes J. Does perception of biological motion rely on specific brain regions? NeuroImage. 2001;13:775–785. doi: 10.1006/nimg.2000.0740. [DOI] [PubMed] [Google Scholar]
  23. Grezes J, Armony JL, Rowe J, Passingham RE. Activations related to “mirror” and “canonical” neurones in the human brain: an fMRI study. NeuroImage. 2003;18:928–937. doi: 10.1016/s1053-8119(03)00042-9. [DOI] [PubMed] [Google Scholar]
  24. Grezes J, Costes N, Decety J. The effects of learning and intention on the neural network involved in the perception of meaningless actions. Brain. 1999;122:1875–1887. doi: 10.1093/brain/122.10.1875. [DOI] [PubMed] [Google Scholar]
  25. Grossman ED, Blake R. Brain activity evoked by inverted and imagined biological motion. Vision Research. 2001;41:1475. doi: 10.1016/s0042-6989(00)00317-5. [DOI] [PubMed] [Google Scholar]
  26. Grossman ED, Blake R. Brain Areas Active during Visual Perception of Biological Motion. Neuron. 2002;35:1167. doi: 10.1016/s0896-6273(02)00897-8. [DOI] [PubMed] [Google Scholar]
  27. Grossman ED, Donnelly M, Price R, Pickens D, Morgna V, Neighbour G, Blake R. Brain Areas involved in the perception of biological motion. Journal of Cognitive Neuroscience. 2000;12:711–720. doi: 10.1162/089892900562417. [DOI] [PubMed] [Google Scholar]
  28. Hamzei F, Dettmers C, Rijntjes M, Glauche V, Kiebel S, Weber B, Weiller C. Visuomotor control within a distributed parieto-frontal network. Experimental Brain Research. 2002;146:273–281. doi: 10.1007/s00221-002-1139-0. [DOI] [PubMed] [Google Scholar]
  29. Howard RJ, Brammer M, Wright I, Woodruff PW, Bullmore ET, Zeki S. A direct demonstration of functional specialization within motion-related visual and auditory cortex of the human brain. Current Biology. 1996;6:1015. doi: 10.1016/s0960-9822(02)00646-2. [DOI] [PubMed] [Google Scholar]
  30. Jeannerod M. Neural Simulation of Action: A Unifying Mechanism for Motor Cognition. NeuroImage. 2001;14:S103–S109. doi: 10.1006/nimg.2001.0832. [DOI] [PubMed] [Google Scholar]
  31. Johansson G. Visual perception of biological motion and a model for its analysis. Perception and Psychophysics. 1973;14:201–211. [Google Scholar]
  32. Kakei S, Hoffman DS, Strick PL. Direction of action is represented in the ventral premotor cortex. Nat Neurosci. 2001;4:1020–1025. doi: 10.1038/nn726. [DOI] [PubMed] [Google Scholar]
  33. Kaplan J, Iacoboni M. Multimodal action representation in human left ventral premotor cortex. Cognitive Processing. 2007;8:103–113. doi: 10.1007/s10339-007-0165-z. [DOI] [PubMed] [Google Scholar]
  34. Keysers C, Gazzola V. Towards a unifying neural theory of social cognition. Prog Brain Res. 2006;156:379–401. doi: 10.1016/S0079-6123(06)56021-2. [DOI] [PubMed] [Google Scholar]
  35. Maldjian JA, Laurienti PJ, Kraft RA, Burdette JH. An automated method for neuroanatomic and cytoarchitectonic atlas-based interrogation of fMRI data sets. NeuroImage. 2003;19:1233–1239. doi: 10.1016/s1053-8119(03)00169-1. [DOI] [PubMed] [Google Scholar]
  36. Materna S, Dicke PW, Thier P. Dissociable roles of the superior temporal sulcus and the intraparietal sulcus in joint attention: A functional magnetic resonance imaging study. J Cognitive Neuroscience. 2008;20:108–119. doi: 10.1162/jocn.2008.20.1.108. [DOI] [PubMed] [Google Scholar]
  37. Meyer G, Wuerger S. Cross-modal integration of auditory and visual motion signals. NeuroReport. 2001;12:2557–2600. doi: 10.1097/00001756-200108080-00053. [DOI] [PubMed] [Google Scholar]
  38. Miall RC. Connecting mirror neurons and forward models. Neuroreport. 2003;14:2135–2137. doi: 10.1097/00001756-200312020-00001. [DOI] [PubMed] [Google Scholar]
  39. Ochiai T, Mushiake H, Tanji J. Involvement of the Ventral Premotor Cortex in Controlling Image Motion of the Hand During Performance of a Target-capturing Task. Cereb Cortex. 2005;15:929–937. doi: 10.1093/cercor/bhh193. [DOI] [PubMed] [Google Scholar]
  40. Pelphrey KA, Mitchell TV, McKeown MJ, Goldstein J, Allison T, McCarthy G. Brain Activity Evoked by the Perception of Human Walking: Controlling for Meaningful Coherent Motion. J Neurosci. 2003;23:6819–6825. doi: 10.1523/JNEUROSCI.23-17-06819.2003. [DOI] [PMC free article] [PubMed] [Google Scholar]
  41. Pelphrey KA, Morris JP, Michelich CR, Allison T, McCarthy G. Functional anatomy of biological motion perception in posterior temporal cortex: An fMRI study of eye, mouth and hand movements. Cerebral Cortex. 2005;15:1866. doi: 10.1093/cercor/bhi064. [DOI] [PubMed] [Google Scholar]
  42. Perrett DI, Harries MH, Benson PJ, Chitty AJ, Mistlin AJ. In: AI and the Eye. Blake A, Troscianko T, editors. John Wiley & Sons Ltd; 1990. Retrieval of structure from rigid and biological motion: an analysis of the visual responses of neurones in the macaque temporal cortex; pp. 181–200. [Google Scholar]
  43. Pineda J. Sensorimotor cortex as a critical component of an ‘extended’ mirror neuron system: Does it solve the development, correspondence, and control problems in mirroring? Behavioral and Brain Functions. 2008;4:47. doi: 10.1186/1744-9081-4-47. [DOI] [PMC free article] [PubMed] [Google Scholar]
  44. Puce A, Perrett D. Electrophysiology and brain imaging of biological motion. Philos Trans R Soc Lond B Biol Sci. 2003;358:435–445. doi: 10.1098/rstb.2002.1221. [DOI] [PMC free article] [PubMed] [Google Scholar]
  45. Rizzolatti G, Craighero L. The mirror-neuron system. Annual Review of Neuroscience. 2004;27:169–192. doi: 10.1146/annurev.neuro.27.070203.144230. [DOI] [PubMed] [Google Scholar]
  46. Rizzolatti G, Fogassi L, Gallese V. Neurophysiological mechanisms underlying the understanding and imitation of action. Nat Rev Neurosci. 2001;2:661–670. doi: 10.1038/35090060. [DOI] [PubMed] [Google Scholar]
  47. Sadato N, Campbell G, Ibanez V, Deiber M, Hallett M. Complexity affects regional cerebral blood flow change during sequential finger movements. J Neurosci. 1996;16:2691–2700. doi: 10.1523/JNEUROSCI.16-08-02691.1996. [DOI] [PMC free article] [PubMed] [Google Scholar]
  48. Saygin AP. Superior temporal and premotor brain areas necessary for biological motion perception. Brain. 2007;130:2452–2461. doi: 10.1093/brain/awm162. [DOI] [PubMed] [Google Scholar]
  49. Saygin AP, Driver J, de Sa VR. In the Footsteps of Biological Motion and Multisensory Perception: Judgments of Audiovisual Temporal Relations Are Enhanced for Upright Walkers. Psychological Science. 2008;19:469–475. doi: 10.1111/j.1467-9280.2008.02111.x. [DOI] [PubMed] [Google Scholar]
  50. Schubotz RI, von Cramon DY. Functional organization of the lateral premotor cortex: fMRI reveals different regions activated by anticipation of object properties, location and speed. Cognitive Brain Research. 2001;11:97–112. doi: 10.1016/s0926-6410(00)00069-0. [DOI] [PubMed] [Google Scholar]
  51. Schubotz RI, von Cramon DY. Predicting Perceptual Events Activates Corresponding Motor Schemes in Lateral Premotor Cortex: An fMRI Study. NeuroImage. 2002;15:787–796. doi: 10.1006/nimg.2001.1043. [DOI] [PubMed] [Google Scholar]
  52. Schubotz RI, von Cramon DY. Sequences of Abstract Nonbiological Stimuli Share Ventral Premotor Cortex with Action Observation and Imagery. J Neurosci. 2004;24:5467–5474. doi: 10.1523/JNEUROSCI.1169-04.2004. [DOI] [PMC free article] [PubMed] [Google Scholar]
  53. Schultz J, Friston KJ, O’Doherty J, Wolpert DM, Frith CD. Activation in Posterior Superior Temporal Sulcus Parallels Parameter Inducing the Percept of Animacy. Neuron. 2005;45:625–635. doi: 10.1016/j.neuron.2004.12.052. [DOI] [PubMed] [Google Scholar]
  54. Schwarzbach JV, Sandrini M, L C. Neural populations in the parietal and premotor cortices of humans perform abstract coding of motor acts: a TMS-adaptation study. Perception Abstract Supplement. 2009;38:158. [Google Scholar]
  55. Seifritz E, Neuhoff JG, Bilecen D, Scheffler K, Mustovic H, Schächinger H, Elefante R, Di Salle F. Neural Processing of Auditory Looming in the Human Brain. Current Biology. 2002;12:2147–2151. doi: 10.1016/s0960-9822(02)01356-8. [DOI] [PubMed] [Google Scholar]
  56. Servos P, Osu R, Santi A, Kawato M. The neural substrates of biological motion perception: An fMRI study. Cerebral Cortex. 2002;12:772. doi: 10.1093/cercor/12.7.772. [DOI] [PubMed] [Google Scholar]
  57. Stanley J, Miall RC. Functional activation in parieto-premotor and visual areas dependent on congruency between hand movement and visual stimuli during motor-visual priming. NeuroImage. 2007;34:290–299. doi: 10.1016/j.neuroimage.2006.08.043. [DOI] [PubMed] [Google Scholar]
  58. Szycik GR, Tausche P, Münte TF. A novel approach to study audiovisual integration in speech perception: Localizer fMRI and sparse sampling. Brain Research. 2008;1220:142–149. doi: 10.1016/j.brainres.2007.08.027. [DOI] [PubMed] [Google Scholar]
  59. Thompson JC, Clarke M, Stewart T, Puce A. Configural Processing of Biological Motion in Human Superior Temporal Sulcus. J Neurosci. 2005;25:9059–9066. doi: 10.1523/JNEUROSCI.2129-05.2005. [DOI] [PMC free article] [PubMed] [Google Scholar]
  60. Urgesi C, Calvo-Merino B, Haggard P, Aglioti SM. Transcranial Magnetic Stimulation Reveals Two Cortical Pathways for Visual Body Processing. J Neurosci. 2007;27:8023–8030. doi: 10.1523/JNEUROSCI.0789-07.2007. [DOI] [PMC free article] [PubMed] [Google Scholar]
  61. Vaina LM, Solomon J, Chowdhury S, Sinha P, Belliveau JW. Functional neuroanatomy of biological motion perception in humans. Proceedings of the National Academy of Sciences of the United States of America. 2001;98:11656–11661. doi: 10.1073/pnas.191374198. [DOI] [PMC free article] [PubMed] [Google Scholar]
  62. Van Essen DC. A Population-Average, Landmark- and Surface-based (PALS) atlas of human cerebral cortex. NeuroImage. 2005;28:635–662. doi: 10.1016/j.neuroimage.2005.06.058. [DOI] [PubMed] [Google Scholar]
  63. Van Essen DC, Drury HA, Dickson J, Harwell J, Hanlon D, Anderson CH. An Integrated Software Suite for Surface-based Analyses of Cerebral Cortex. J Am Med Inform Assoc. 2001;8:443–459. doi: 10.1136/jamia.2001.0080443. [DOI] [PMC free article] [PubMed] [Google Scholar]
  64. Vogt S, Buccino G, Wohlschläger AM, Canessa N, Shah NJ, Zilles K, Eickhoff SB, Freund H-J, Rizzolatti G, Fink GR. Prefrontal involvement in imitation learning of hand actions: Effects of practice and expertise. NeuroImage. 2007;37:1371–1383. doi: 10.1016/j.neuroimage.2007.07.005. [DOI] [PubMed] [Google Scholar]
  65. Wenderoth N, Debaere F, Sunaert S, Hecke Pv, Swinnen SP. Parieto-premotor Areas Mediate Directional Interference During Bimanual Movements. Cereb Cortex. 2004;14:1153–1163. doi: 10.1093/cercor/bhh075. [DOI] [PubMed] [Google Scholar]
  66. Wuerger SM, Hofbauer M, Meyer GF. The integration of auditory and visual motion signals at threshold. Perception & Psychophysics. 2003;65:1188–1196. doi: 10.3758/bf03194844. [DOI] [PubMed] [Google Scholar]
  67. Wuerger SM, Meyer G, Hofbauer M, Zetzsche C, Schill K. Motion extrapolation of auditory-visual targets. Information Fusion. 2010;11:45–50. [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

Supporting Material
EMS172346-supplement-Supporting_Material.pdf (426.8KB, pdf)

RESOURCES