Skip to main content
NCBI home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2013 Oct 15.
Published in final edited form as: Neuroimage. 2012 Jun 27;63(1):111–118. doi: 10.1016/j.neuroimage.2012.06.029

Motion-sensitive cortex and motion semantics in American Sign Language

Stephen McCullough a, Ayse Pinar Saygin b, Franco Korpics a, Karen Emmorey a
PMCID: PMC3429697  NIHMSID: NIHMS390844  PMID: 22750056

Abstract

Previous research indicates that motion-sensitive brain regions are engaged when comprehending motion semantics expressed by words or sentences. Using fMRI, we investigated whether such neural modulation can occur when the linguistic signal itself is visually dynamic and motion semantics is expressed by movements of the hands. Deaf and hearing users of American Sign Language (ASL) were presented with signed sentences that conveyed motion semantics (“The deer walked along the hillside.”) or were static, conveying little or no motion (“The deer slept along the hillside.”); sentences were matched for amount of visual motion. Motion-sensitive visual areas (MT+) were localized individually in each participant. As a control, the Fusiform Face Area (FFA) was also localized for the deaf participants. The whole-brain analysis revealed static (locative) sentences engaged regions in left parietal cortex more than motion sentences, replicating previous results implicating these regions in comprehending spatial language for sign languages. Greater activation was observed in the functionally defined MT+ ROI for motion than static sentences for both deaf and hearing signers. No modulation of neural activity by sentence type was observed in the FFA. Deafness did not affect modulation of MT+ by motion semantics, but hearing signers exhibited stronger neural activity in MT+ for both sentence types, perhaps due to differences in exposure and/or use of ASL. We conclude that top down modulation of motion-sensitive cortex by linguistic semantics is not disrupted by the visual motion that is present in sign language sentences.

Introduction

Neuroimaging studies investigating interactions between language and other cognitive systems have revealed that language processing is not limited to classic language areas but can also involve brain regions known to be responsible for low-level sensorimotor processes (e.g., Aziz-Zadeh, Wilson, Rizzolatti, & Iacoboni, 2006; Hauk, Johnsrude, & Pulvermüller, 2004; Kemmerer, Castillo, Talavage, Patterson, & Wiley, 2008). For example, comprehending action verbs appears to selectively engage premotor and motor cortices that are relevant to the particular action semantics being computed (Pulvermüller et al. 2001; Hauk et al. 2004; Aziz-Zadeh et al. 2006; Kemmerer et al., 2008). Specifically, reading verbs that describe lip/tongue actions (e.g., bite, smile), arm/hand actions (e.g., stir, grab), or leg/foot actions (e.g., kick, run) somatopically activates neural representations of the corresponding body parts in motor cortex (e.g., Pulvermüller, Härle, Hummel, 2001; Hauk et al., 2004). These studies suggest that linguistic processes that involve the retrieval of semantic representations may activate the same sensorimotor neural networks that were initially involved in the formation of such concepts (Damasio, 1989; Barsalou, 1999).

Furthermore, behavioral studies have found that linguistic processing of action semantics can be affected by performing a related motor action. For example, Glenberg and colleagues (Glenberg & Kaschak, 2002; Borghi, Glenberg, & Kaschak, 2004) discovered the Action-sentence Compatibility Effect (ACE) in which judgments of sentence plausibility are negatively affected if the action semantics encoded by the sentence are incongruent with a motor response (e.g., pressing a response button that requires participants to move their hand away from the body when reading the sentence "Open the drawer"). Similarly, with respect to perceptual systems, there appears to be a two-way interaction between linguistic processing and motion perception. Listening to verbs denoting downward motion (e.g., fall, drop, plummet) impairs the ability to detect upward motion in random dot kinematograms (Meteyard, Bahrami, & Vigliocco, 2007). Likewise, perceiving language-incongruent visual motion negatively affects the comprehension of motion words and sentences (Kaschak et al., 2005; Meteyard, Zokaei, Bahrami, & Vigliocco, 2008).

Consistent with these findings, Saygin, McCullough, Alac, and Emmorey (2010) showed that listening to audiovisually presented English sentences that expressed motion (e.g., “The deer jumped over the brook”) activated the lateral temporal motion sensitive area known as MT+ (localized individually in each participant) significantly more than the matched static sentences that expressed no motion (e.g., “The deer slept next to the brook”). This effect was not found in a control region of interest in ventral temporal cortex (Fusiform Face Area, FFA). Several studies have now reported activation in lateral temporal cortex during the comprehension of verbs or sentences that contain action or motion-related semantics (Kable, Lease-Spellmeyer, & Chatterjee, 2002; Revill, Aslin, Tanenhaus, & Bavelier, 2008; Rueschemeyer, Glenberg, Kaschak, Mueller, & Friederici, 2010). Thus, the linguistic semantics of motion appears to modulate neural responses in brain regions engaged in early visual processing of perceptual motion information.

The behavioral and neuroimaging evidence for a two-way interaction between linguistic and perceptual systems raises an important and interesting question: what is the nature of this interaction for signed languages? For these languages, visual-spatial information must be encoded within two distinct processing streams: top-down language comprehension and bottom-up visual perception. Movement is one of the phonological parameters that make up a sign, along with hand configuration, palm orientation, and location (Stokoe, 1960; Battison, 1978), and signs are considered ill-formed if they do not contain movement (Brentari, 1998). Crucially, with respect to motion semantics, signed languages have verbs that represent motion semantics isomorphically and iconically. For example, in American Sign Language (ASL), the verb OPEN-DRAWER is produced with two fists (palms up) that move toward the body, as if opening a drawer, and the verb PUSH is produced with an outward motion of the arms, with flat palms facing outward (as if pushing a large object). Furthermore, the speed of the movement can also depict the speed of the motion (e.g., quickly opening a drawer or slowing pushing an object). However, the phonological movement of a sign may also bear no relation to the meaning of the sign. For example, the ASL sign LIKE is made with an outward motion, while the sign LOVE is made with movement toward the body. Nonetheless, there appear to be no “anti-iconic” signs that move in a direction that is contrary to their meanings, such as a sign specified for upward phonological movement but with semantics involving downward motion (e.g., fall, drop, dive, etc.).

The fact that perception of ASL signs and sentences entails visual motion processing introduces a unique challenge to the language processing system with respect to motion semantics. It is possible that the neural response to physical motion of the hands overwhelms top-down modulation by linguistic semantics of motion-sensitive brain areas. Specifically, one mechanism that has been proposed to account for the involvement of sensory and motor systems in language comprehension is mental simulation (e.g., Gallese & Lakoff, 2005; Zwaan & Taylor, 2006). Under this proposal, linguistic descriptions of motion are understood in part by subconsciously simulating the described motion event, which activates the same neural circuitry engaged when actually observing motion. For signed languages, it is possible that motion simulation is disrupted because the neural regions involved in motion processing are actively engaged in perceptual processing of the signed signal.

On the other hand, the fact that the physical movement of a verb carrying motion semantics isomorphically depicts aspects of the motion of a referent may allow for synergistic top-down modulation by the linguistic system. That is, the mapping between motion semantics and the physical motion of the hands may lead to increased activation within motion-sensitive cortical regions compared to when the motion of the hands does not depict motion information. Supporting this hypothesis,MacSweeney et al. (2002) reported that comprehension of “topographic” sentences (sentences that express spatial relationships) in British Sign Language (BSL) generated greater activation in MT+ bilaterally, compared to “non-topographic” sentences. Although the BSL sentences were not selected to contrast motion vs. no-motion semantic content, many of the topographic sentences expressed the movement of a referent from one location to another.MacSweeney et al. (2002) speculated that enhanced MT+ activation reflected top-down modulation by the meaning of physical movement in the topographic sentences.

To investigate MacSweeney et al.’s (2002) hypothesis, we conducted an fMRI study with both deaf and hearing signers, using a paradigm similar to Saygin et al. (2010). Signers were asked to make plausibility judgments to ASL sentences with motion semantics (i.e., “The deer walked along the hillside”) or static (no-motion) semantics (i.e., “The deer slept along the hillside”), as illustrated in Figure 1. Crucially, the video clips were matched for amount of physical motion (see Methods). In addition, motion-sensitive cortical regions were functionally mapped for each individual.

Figure 1.

Figure 1

Open in a new tab

Illustration of matched ASL motion and static (no motion) sentences.

Both the deaf and hearing signers in this study acquired ASL as a first language from birth. We included hearing signers because deafness may have a unique effect on visual motion processing. Specifically, deaf signers exhibit enhanced attention to motion in the periphery of vision, while hearing signers perform on a par with their non-signing peers (e.g., Neville & Lawson, 1987; Bavelier et al. 2000; Bosworth & Dobkins, 2002). Further, although recruitment of MT+ is similar for deaf and hearing signers under passive viewing conditions, group differences are observed when motion must be selectively attended (either centrally or peripherally) (Bavelier et al., 2001; Fine et al., 2005). Thus, the inclusion of hearing signers allows us to investigate whether deafness has a unique effect on the modulation of MT+ by top-down linguistic processing of motion semantics. For example, it is possible that deaf signers might exhibit a larger MT+ response to motion semantics due to an enhanced ability to selectively attend to motion within the sign stream.

For the deaf signers, we also functionally mapped the fusiform face area (FFA), a region in ventral temporal cortex that is activated during face processing. Due to time constraints for the hearing signers, we were only able to collect MT+ localizer data. FollowingSaygin et al. (2010), FFA was chosen as a control region because like MT+, it is a localizable feature-specific visual region, but the FFA is not expected to respond differentially to motion semantics encoded by language. Area MT+ was localized by contrasting random dot motion flow fields with static dot displays, and the FFA was localized by contrasting unknown faces with houses. The MT+ and FFA cluster maps within each hemisphere were subsequently used as Regions of Interest (ROIs) for statistical analysis of BOLD responses to the video clips of ASL motion and static sentences. If the ASL linguistic system interacts with a low-level motion sensitive visual area (MT+), we predicted there would be an increase in neural activity for visually presented sentences containing motion semantics in MT+ (but not in the FFA) in contrast to ASL sentences with static (locative) semantics.

Materials and Methods

Participants

12 deaf signers (6 females; mean age = 27. 6 years; SD = 6.7 years) and 13 hearing signers (9 females; mean age = 26. 4 years; SD = 4.7 years) participated in the study. All participants were born into deaf signing families, were right handed, and had normal or corrected-to-normal vision by self-report. Of the hearing signers, seven worked as professional ASL interpreters. Participants were recruited primarily from metropolitan areas of San Francisco, San Diego, Los Angeles, and Washington D.C. Informed consent was obtained according to procedures approved by the UCSD and SDSU Human Research Protection Programs.

Materials

Two types of ASL sentences were presented: motion sentences contained a motion verb and described an event with movement and static sentences described an event with little or no movement. To create the stimuli, three deaf native signers first created 80 pairs of matched motion and static ASL sentences. Each sentence pair was required to have a similar number of signs, similar grammatical structure, and similar semantic content (except for the depiction of movement). An example sentence pair is shown in Figure 1, and video examples are also provided in supplementary materials.

The initial set of 160 sentences were filmed and presented in random order to 15 deaf ASL signers who did not subsequently participate in the fMRI study. These signers were asked to rate the vividness of the movement described in each sentence on a 5 point scale (0 = no motion, 4 = a lot of motion). As a guide, participants were shown two example sentences depicting little or no motion (e.g., English translation: ”The lion slept in his enclosure at the zoo”) and two example sentences depicting a high amount of movement (e.g., English translation: “Many dogs were running loose around the farmyard”). Participants were told that we were specifically interested in the strength of the movement imagery (the vividness of the movement) expressed in the sentence, not in the strength of the visual imagery or in how much the hands moved when signing. There were six practice items. All instructions were presented in ASL. Mean motion ratings were calculated for each sentence.

For the neuroimaging study, we selected only sentences that were consistently given high motion imagery ratings (N = 32 motion sentences; mean rating = 2.85, SD = .56) or low motion imagery ratings (N = 32 static sentences; mean rating = 0.65, SD = .35). Sentences with middle ratings or with a large variance in the motion ratings across participants were excluded. The final set of motion sentences had significantly higher movement imagery ratings than the static sentences, t(60) = 17.55, p < .0001. The final sentences did not differ significantly in duration (mean duration for motion sentences and static sentences was 7.57 sec and 6.87 sec, respectively, t(60) = 1.6, p = .10), and were matched for number of signs, grammatical structure, and semantic content.

We also created signed sentences that resembled the experimental sentences in structure and content but contained a semantic anomaly (e.g., English translation: ”A tired tourist dragged his ocean”). Anomalous sentences were similar semantically and grammatically to the experimental sentences and were rendered implausible through lexical substitution. These sentences constituted the target sentences for the experimental task (i.e., detecting an implausible sentence). All stimuli were produced by a deaf male native ASL signer (FK) who was seated, looking directly into the camera.

Because we were concerned with the top-down modulation of motion sensitive areas by semantics, and not by the physical motion present in the ASL stimuli, it was critical that the motion and static sentences contain the same amount of physical motion. To assess the amount of physical motion present the ASL sentences, each video clip was measured for amount of movement shown on the screen using previously established motion energy detection algorithms (Grammer, Filova, Fieder, 1997; Grammer, Fink, & Renniger, 2002). This measurement is based on automatic analysis of body contour changes recorded on digitized video (Automatic Movie Analysis – AMA). AMA subtracts successive video frames from each other and determines the total amount of change that occurs within a given time span (Motion Energy Detection – MED). Thus, AMA cumulates all movements in the video into one measure of total image change. This analysis was conducted over the entire video clip for each sentence. Based on this measure, the final set of motion and static ASL sentences did not differ in the amount of physical motion (the total motion energy) shown on the screen, t(60) = .084, p = .40.

Procedure

A MacBook Pro (Apple, Cupertino, CA) computer was used to present the localizer and experimental stimuli. Videos were presented using QuickTime (Apple, Cupertino, CA), and participants’ key press responses were recorded using MATLAB with the Psychophysics Toolbox extensions (psychtoolbox.org) running on a separate MacBook Pro computer. All stimuli were projected onto a screen that was set up at the foot of the scanner bed, using an XGA video projector. Participants viewed the stimuli through a mirror that was placed inside the head coil, and they responded using an MR-safe Lumitouch button box (Photon Control, Burnaby, Canada). All participants completed short practice runs of the experimental and localizer tasks outside the scanner so that they were familiarized with the stimuli and the tasks.

A block design was chosen for the experimental presentation of ASL sentences to maximize statistical power. Specifically, 30-second linguistic blocks alternated with 30- second baseline blocks. The linguistic blocks alternated between motion and static sentence blocks, with the baseline blocks evenly interspersed between them. In each run, there were 8 linguistic and 8 baseline blocks. During the linguistic blocks, participants were instructed to pay close attention to the ASL sentences and press a button if the sentence was semantically anomalous. Anomalous sentences occurred 0–1 times per linguistic block and were never the first or last sentence in a block. On average, each linguistic block contained four sentences (range 4 – 6), presented with 1000 ms ISI between sentences, and an additional 1000 ms visual cue indicating the change in task at the beginning of the linguistic and baseline blocks. In the baseline blocks, participants viewed video clips of the same sign model seated in the same position but not signing. Participants were asked to decide whether the color of a black dot superimposed on the model’s chin changed to white during the block (cf. Saygin et al., 2010; MacSweeney et al., 2002). We chose to intersperse baseline blocks in between each experimental block so that our comparisons of interest (the different sentence types) were always presented after the same baseline block. Thus, the change in hemodynamic response was as uniform as possible for each linguistic block.

In addition, functional localizer scans were conducted to define precise regions of interest for the main analysis contrasting the comprehension of ASL motion versus static sentences. The order of task presentation was as follows: MT+ localizer, FFA localizer, and ASL experimental task blocks.

MT+ localizer

To map MT+ regions in each individual in each hemisphere, we presented random dot flow motion stimuli using MATLAB software (Mathworks, Natick, MA). Moving and stationary dots alternated for 32-second long blocks within a 4 min and 16 sec run. During the Motion block, white dots were randomly generated to travel inwardly or outwardly within the 15° circle aperture field (4.5 deg/sec). The identical random dots were completely stationary within the aperture field for the duration of the Stationary block. Participants were asked to fixate on the center of the screen and did not perform a task.

Fusiform Face Area (FFA) localizer

Scrambled and normal grayscale images of houses and faces were presented in 5° viewing angle using Psyscope X software (http://psy.ck.sissa.it). Participants viewed 20-sec blocks of pictures of faces, scrambled faces, houses, and scrambled houses, for a total scan duration of 5 min, 20 sec. Participants performed a one-back working memory task (i.e., detecting a repeated item). Face localizer data were collected only for the deaf participants, and we were unable to obtain FFA localizer data for one participant.

MRI Data Acquisition

MRI and fMRI data were collected using a 3-Tesla GE Signa Excite scanner equipped with an eight-element phased-array head coil at the Center for fMRI at the University of California, San Diego. For anatomical reference and spatial normalization, high resolution images of the brain structure of each participant were acquired using T1-weighted Fast Spoiled Gradient-Recalled Echo (FSPGR) in the middle of the scanning session (FOV 256mm, 1 mm×1mm in-plane, 1.3 mm thick slices, 106 axial slices, 256×256 matrix, flip angle = 8°, inversion time 600 ms). For functional images, 38 T2*-weighted, gradient-echo echo-planar (EPI) axial slices were acquired interleaved from inferior to superior, covering the whole brain, with a repetition time (TR) of 2000 msec, an echo time (TE) of 30 ms, flip angle = 90°, FOV 224 mm, 64×64 matrix, 3.5×3.5 mm in-plane resolution, and 4 mm slice thickness (no gap).

We collected data from each participant in two runs of 240 EPI volumes each (8 minutes) for the ASL sentence condition, a single run of 128 EPI volumes for the MT+ localizer condition (4 min 16), and a single run of EPI 160 volumes for the FFA localizer condition (5 min 20 sec). Three “dummy” volumes were added to the beginning of all functional runs to allow the magnetization to reach steady state before stimulus presentation. These “dummy” volumes were removed during the data pre-processing and were not included in subsequent data analyses. In addition, we acquired B0 field maps from each participant to correct for distortions due to field inhomogeneities, using protocols from the UCSD Center for fMRI.

Data Preprocessing and Analysis

All functional scans were preprocessed with in-house software and the AFNI software package (version AFNI_2010_10_19_1028; Cox, 1996), using the following steps. Each participant’s functional scans were unwarped and corrected for geometric distortions in the echo planar images (caused by static magnetic field inhomogeneities) using B0 field maps and UNIX-based in-house software developed at the UCSD Center for fMRI. All EPIs were corrected for slice timing using AFNI 3dTshift. Participants’ head movements during each functional run were estimated and corrected with AFNI 3dvolreg (using fourier interpolation with a middle volume as the reference point). Estimates of the three translation and three rotation parameters were computed during this motion correction and saved. Prior to running AFNI 3dDeconvole, all EPIs were normalized by the mean signal at each voxel. The impulse response function (IRF) to each non-baseline stimulus type (motion and static ASL sentences) was computed and estimated using AFNI 3dDeconvolve. The model contained a second order polynomial for each run to account for slow drifts and the six motion regressors obtained during the 3dvolreg motion correction. Percent signal change values for the ROIs were obtained by averaging time points in the range where the hemodynamic response stabilized.

ROIs were individually defined for each participant using the localizer scans plus anatomical constraints. For MT+ ROI localization, the most reliable anatomical landmark for area MT+ is near the junction of the inferior temporal sulcus and the ascending limb of the inferior temporal sulcus. In this region, we selected voxels showing a strong (p <10−10, corrected for multiple comparisons) response to dot flow motion > static contrast. For FFA ROI localization, face-sensitive voxels in the fusiform gyrus were defined by using a faces > house contrast at (p < 10−6, corrected for multiple comparisons).

We also conducted a whole-brain voxel-wise analysis of the MT+ localizer and the ASL sentence conditions. For these analyses, anatomical images of each participant were spatially normalized to the standard Talairach-Tournoux space using the AFNI TT_N27 template (Colin27) and the AFNI @auto_tlrc program. Statistical maps (beta coefficients computed by AFNI 3dDeconvolve) from each participant were aligned to each participant’s anatomical structure using AFNI @auto_tlrc and spatially blurred using a Gaussian filter with a full-width half maximum of 6 mm. The AFNI programs, 3dANOVA and 3dANOVA3 (type 5), were chosen for whole-brain voxel-wise analyses of the MT+ localizer task and the ASL sentence condition, respectively. The statistical results were also corrected with AFNI 3dFDR (a false-discovery rate algorithm), and the threshold value for voxel-wise statistics and report statistics was set at p = .001 (corrected). AFNI’s implementation of the Brett transform (a two-step affine transformation procedure; Brett, Johnsrude, & Owen, 2002) was used to convert Talairach coordinates into MNI coordinates for presenting results in Tables 1 and 2.

Table 1.

Mean MNI coordinates and mean voxel sizes for MT+ and FFA regions

Brain regions MNI coordinates (x,y,z) vol (mm3)
Hearing signers
  MT+
    Left −38 −73   0 855
    Right   42 −71 −1 585
Deaf signers
  MT+
    Left −40 −72   1 432
    Right   45 −67   1 603
  FFA
    Left −38 −43 −17 972
    Right   39 −44 −16 999
Open in a new tab

Table 2.

Results of the whole brain analysis for the contrast between ASL sentence types (p = .001; corrected). Damasio (2005) was used as a guide for anatomical labels.

Brain regions MNI coordinates (x,y,z) vol (mm3) t
Static Sentences
Left Hemisphere
  Inferior Parietal Lobule −34 −79 +19 1031 −7.5
  Superior Parietal Lobule −24 −56 +48 144 −7.0
Right Hemisphere
  Parahippocampal Gyrus +28 −38 −13  153 −7.4
Motion Sentences
  No voxels survived
Open in a new tab

Results

Behavioral results

Signal detection statistics were used to analyze participants’ task performance for detecting semantic anomalies in the ASL sentences and for detecting repeated faces or houses for the FFA localizer task. For the ASL sentence task, the mean d’ was 3.94 (SD = .37), and there was no difference in performance between the deaf and hearing signers, t(21.5)= –.25, p = .802 (Welch two sample t test). For the FFA localizer task (deaf only), the mean d’ was 3.74 (SD = .36). These d’ values indicate strong performance for the ASL comprehension task and for the FFA localizer task.

Neuroimaging results

Table 1 shows the mean MNI coordinates and mean volumes for the MT+ ROI (for both deaf and hearing signers) and for the FFA ROI (deaf signers). The mean MNI coordinates for the MT+ ROI are consistent with previous studies (e.g., Tootell et al., 1995; Watson et al., 1993) and consistent with the location of the cytoarchitectonic correlate of human MT+ proposed byMalikovic et al. (2007). The mean MNI coordinates for FFA are also quite consistent with previous studies (e.g., Grill-Spector, Knouf, & Kanwisher, 2004).

For MT+, we conducted a whole-brain voxel-wise group contrast using the beta coefficients from the MT+ localizer task as the dependent measure. This analysis revealed no significant difference in MT+ responses to motion stimuli between deaf and hearing signers (i.e., no voxels survived the correction for multiple comparisons). We also conducted a mixed-design 2 (group: deaf, hearing) X 2 (hemisphere: left, right) ANOVA, with percent signal change in the MT+ ROI as the dependent measure. This analysis also revealed no significant group difference, and no significant difference between hemispheres or interaction between group and hemisphere (all Fs < 1).

For the ASL sentence condition, we conducted a mixed-design 2 (group: deaf, hearing) X 2 (sentence type: motion, static) X 2 (hemisphere: left, right) ANOVA, with percent signal change from MT+ ROIs as the dependent measure. As illustrated in Figure 2, both groups exhibited a larger percent signal change in MT+ for motion sentences than for static sentences, F(1,21) = 10.15, p = .004, for the main effect of sentence type. The results also revealed a significant main effect of group, F(1,21) = 10.40, p = .004. Compared to the deaf signers, hearing signers exhibited a greater percent signal change within MT+ for both sentence types. However, there was no interaction between sentence type and participant group, F < 1, p = .72. Overall, there was no difference in percent signal change between the two hemispheres, F < 1, p = .43, and hemisphere did not interact with either participant group, F < 1, p = .83, or sentence type, F (1,21) = 1.81, p= .18. The three-way interaction between sentence type, group, and hemisphere was also not significant, F < 1, p = .95.

Figure 2.

Figure 2

Open in a new tab

An axial MRI image of the averaged brain showing the approximate locations of the MT+ ROI in each hemisphere. Each graph shows the BOLD responses averaged across hemispheres for motion (black) and static (grey) sentences. Error bars represent standard deviation. The x axis is time, and the y axis is percent signal change relative to baseline.

For the FFA ROI, we conducted a similar ANOVA, but without participant group as a factor. The results revealed no significant difference in percent signal change for motion and static sentences, F < 1, p = .87, and no difference in neural response between the hemispheres, F < 1, p = .43. There was no interaction between sentence type and hemisphere, F < 1, p = .41.

Finally, we also performed a whole-brain voxel-wise 2 (group: deaf, hearing) X 2 (sentence type: motion, static) ANOVA with the beta coefficients from the ASL sentence condition as the dependent measure. The results revealed no significant difference between deaf and hearing signers and no interaction between participant group and sentence type. However, a significant difference between sentence types was observed (see Table 2). As shown in Figure 3, the direct contrast revealed greater activation in left parietal cortex and in the right parahippocampal gyrus for static than for motion sentences. There were no voxels that responded more for motion sentences than for static sentences after correction for multiple comparisons.

Figure 3.

Figure 3

Open in a new tab

The clusters in orange are shown on a high resolution TT_N27 human brain atlas and represent brain regions activated for static, locative sentences in contrast to motion sentences, p = .001 (corrected). Activation up to 25 mm beneath the surface of the cortex is displayed.

Discussion

Our results indicate that the neural response in motion-sensitive area MT+ to physical motion of the hands during sign language comprehension does not disrupt topdown modulation of this area by motion semantics. Comprehension of ASL sentences conveying motion information led to greater neural activity within MT+ (functionally identified in each individual) compared to ASL sentences conveying static, non-motion related information (see Figure 2). Further, neural activity within a control region of interest for the deaf signers, the Fusiform Face Area, was unaffected by motion semantics, suggesting that the neural response to motion semantics was specific to this motionsensitive brain region. These results confirm MacSweeney et al.’s (2002) post-hoc interpretation of MT+ activation by topographic sentences in BSL, replicate and extend the results ofSaygin et al. (2010), and establish the cross-linguistic and cross-modal robustness of MT+ activation by motion semantics.

If visual motion simulation within MT+ is involved in comprehending motion semantics for ASL signers, our results indicate that such neural simulation is not disrupted because this region is simultaneously engaged in the perceptual processing of a dynamic visual linguistic signal. One reason that physical movement of the hands may not disrupt semantic-based motion simulation (or motion imagery) is that the physical movement of the hands is congruent with the motion semantics expressed by the sentence. That is, ASL verbs expressing upward motion physically move upward (e.g., CLIMB), verbs expressing downward motion move downward (e.g., FALL), verbs expressing motion away from the body move away from the body (e.g., PUSH), etc. Compatible physical and semantic motion may thus enable top-down modulation of sensorimotor systems by higher order language regions. Further, the requirement to simultaneously process visual motion and motion semantics does not appear to result in a competition for neural resources. Our results suggest that the interaction between language and motion perception is similar for spoken and signed languages, despite the fact that visual motion is an essential part of the linguistic signal for signed but not for spoken languages.

The results of the whole brain analysis revealed greater activation in left parietal cortex for static sentences than for motion sentences (see Table 2; Figure 3). This finding replicates MacSweeney et al. (2002) who reported greater activation in left parietal cortex for BSL sentences expressing spatial information (“topographic” sentences) compared to non-topographic sentences. As illustrated in Figure 1, the static ASL sentences in our study primarily conveyed information about the spatial location of a non-moving referent. Further, MacSweeney reported no difference in parietal activation when hearing participants watched audio-visual English translations of the BSL topographic and non-topographic sentences. Similarly, when hearing individuals watched audio-visual English sentences that were parallel to the ASL sentences presented here (although not exact translations), no difference between sentence types was observed in the whole brain analysis (Saygin et al., 2010, Supplementary Materials).MacSweeney et al. (2002) argued that the left parietal lobe is specifically involved in processing the precise configuration of the hands in space to represent objects, agents, and actions. Our findings refine this interpretation and suggest that left parietal cortex may be particularly engaged during sign language comprehension when static spatial configurations are conveyed by the location and orientation of the signer’s hands. These regions may be preferentially engaged for comprehending location expressions for sign languages because the precise configuration of the hands in space, rather than a preposition or other closed-class morpheme, must be mapped to a conceptual representation of the spatial relationship between entities (i.e., the spatial configuration of figure and ground referents).

Hearing status did not affect modulation of MT+ activation by motion semantics – comprehending ASL motion sentences increased neural activity within MT+ for both hearing and deaf native signers. Although deafness impacts the extent of activation within motion-sensitive brain regions when attending to non-linguistic movement (e.g., Bavelier et al., 2001), we did not observe differential activation within MT+ for motion semantics for deaf compared to hearing signers. In addition, the analysis of the MT+ localizer task revealed no difference between deaf and hearing signers, replicatingBavelier et al. (2000) who found that effects of deafness on the neural response to motion flow fields are not observed under passive viewing conditions.

However, greater activation was observed within MT+ for hearing signers compared to deaf signers for both ASL sentence types. This result does not appear to be due to generally higher BOLD signal for the hearing signer group because no group effects were found for the MT+ localizer analysis or for the whole brain analysis for the ASL sentences. We suggest that this difference in overall activation for the hearing signers may reflect the fact that ASL is the less dominant language (see Emmorey, Borinstein, Thompson, & Gollan, 2008; Emmorey, Petrich, and Gollan, in press). Although the hearing signers in this study may have been ASL dominant as children, English is likely to have rapidly become the dominant language. This switch in dominance occurs for many bilinguals living in the US because they are immersed in an English-dominant environment outside the home (Kohnert, Hernandez, & Bates, 1999). In contrast, for the deaf signers in this study, ASL is their dominant language – they sign more often than they speak, and they frequently rely on ASL-English interpreters when interacting with non-signers.

Comprehension of a less dominant language may require additional neural resources. For example, Perani et al. (2003) found that a lower level of language use/exposure was associated with greater neural activity in left inferior frontal cortex for highly proficient early bilinguals performing a word generation task (see also Indefrey, 2006). Further,MacSweeney et al. (2008) reported increased activation in the left inferior frontal gyrus (IFG) for non-native deaf signers compared to native deaf signers when performing a sign form judgment task in BSL. If increased activation in MT+ were due to less ASL exposure or less use by the hearing signers, then we would predict increased activation for the hearing signers would also be observed in classic language regions, such as the left inferior frontal cortex. Therefore, we compared the percent signal change for hearing and deaf signers within left IFG (this ROI encompassed pars opercularis, pars orbitalis, and pars trangularis regions). As predicted, greater left IFG activation was found for the hearing compared to deaf signers, t(46.35) = 3.13, p = .003 (Welch two sample t test). This finding supports the hypothesis that increased activation within MT+ for hearing signers may be related to language dominance. Hearing signers may exhibit greater neural activity during ASL comprehension compared to deaf signers because, although highly proficient and native-learners, they have less daily exposure to ASL and may require more neural resources for equally accurate performance.

In sum, we found that linguistic semantics related to motion in ASL sentences modulates motion-sensitive visual area MT+, but not face-sensitive visual area FFA. Overall neural activity for ASL sentences in MT+ and in left IFG was greater for the hearing signers, likely due to the fact that ASL is the dominant language for deaf, but not for hearing signers. However, the modulation of MT+ by semantics was similar for deaf and hearing signers, indicating that deafness did not alter how this region was recruited for linguistic processing. In conclusion, we have shown that a dynamic visual linguistic signal does not block the modulation of early visual areas that subserve motion processing when comprehending language about motion.

Supplementary Material

01
02
Download video file (1MB, mov)
03
Download video file (1MB, mov)

Figure 4.

Figure 4

Open in a new tab

Figure 5.

Figure 5

Open in a new tab

Acknowledgments

This work was supported by a grant from the National Institute on Deafness and other Communication Disorders (R01 DC010997) to Karen Emmorey and San Diego State University. We thank all of the staff at the UCSD Center for fMRI and the SDSU Laboratory for Language and Cognitive Neuroscience for their support and assistance with the study. We also thank Karl Grammer for conducting the motion image analyses, Heather Larrabee for help with the ASL stimuli rating, and Mairéad MacSweeney and an anonymous reviewer for very helpful comments on the manuscript. Finally, we thank all of the deaf and hearing signers for their participation in the study.

Footnotes

Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final citable form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.

References

  1. Aziz-Zadeh L, Wilson SM, Rizzolatti G, Iacoboni M. Congruent embodied representations for visually presented actions and linguistic phrases describing actions. Curr. Biol. 2006;16(18):1818–1823. doi: 10.1016/j.cub.2006.07.060. [DOI] [PubMed] [Google Scholar]
  2. Barsalou LW. Perceptual symbol systems. Beh. Brain Sci. 1999;22:577–660. doi: 10.1017/s0140525x99002149. [DOI] [PubMed] [Google Scholar]
  3. Battison R. Lexical borrowing in American Sign Language. Silver Spring, MD: Linstok Press; 1978. [Google Scholar]
  4. Bavelier D, Tomann A, Hutton C, Mitchell T, Cornia D, Liu G, Neville H. Visual attention to the periphery is enhanced in congenitally deaf individuals. J. Cog. Neurosci. 2000;20:1–6. doi: 10.1523/JNEUROSCI.20-17-j0001.2000. [DOI] [PMC free article] [PubMed] [Google Scholar]
  5. Bavelier D, Brozinsky C, Tomann A, Mitchell T, Neville H, Liu G. Impact of early deafness and early exposure to sign language on the cerebral organization for motion processing. J. Cog. Neurosci. 2001;21(22):8931–8942. doi: 10.1523/JNEUROSCI.21-22-08931.2001. [DOI] [PMC free article] [PubMed] [Google Scholar]
  6. Borghi AM, Glenberg AM, Kaschak MP. Putting words in perspective. Memory and Cognition. 2004;32(6):863–873. doi: 10.3758/bf03196865. [DOI] [PubMed] [Google Scholar]
  7. Bosworth RG, Dobkins KR. Visual field asymmetries for motion processing in deaf and hearing signers. Brain and Cog. 2002;49:173–181. doi: 10.1006/brcg.2001.1498. [DOI] [PubMed] [Google Scholar]
  8. Brentari D. A prodsodic model of sign language phonology. Cambridge, MA: MIT Press; 1998. [Google Scholar]
  9. Brett M, Johnsrude LS, Owen AM. The problem of functional localization in the human brain. Nat Rev Neurosci. 2002;3:243–249. doi: 10.1038/nrn756. [DOI] [PubMed] [Google Scholar]
  10. Cox RW. AFNI: software for analysis and visualization of functional magnetic resonance neuroimages. Comput Biomed Res. 1996;29(3):162–173. doi: 10.1006/cbmr.1996.0014. [DOI] [PubMed] [Google Scholar]
  11. Damasio AR. Time-locked multiregional retroactivation: a systems-level proposal for the neural substrates of recall and recognition. Cognition. 1989;33(1–2):25–62. doi: 10.1016/0010-0277(89)90005-x. [DOI] [PubMed] [Google Scholar]
  12. Damasio H. Human brain anatomy in computerized images. 2nd Edition. Oxford: Oxford University Press; 2005. [Google Scholar]
  13. Emmorey K, Borinstein HB, Thompson T, Gollan TH. Bimodal bilingualism. Bilingualism: Language and Cognition. 2008;11(1):43–31. doi: 10.1017/S1366728907003203. [DOI] [PMC free article] [PubMed] [Google Scholar]
  14. Emmorey K, Petrich J, Gollan TH. Bilingual processing of ASL-English code-blends: The consequences of accessing two lexical representations simultaneously. Journal of Memory and Language. doi: 10.1016/j.jml.2012.04.005. in press. http://dx.doi.org/10.1016/j.jml.2012.04.005. [DOI] [PMC free article] [PubMed] [Google Scholar]
  15. Fine I, Finney EM, Boynton GM, Dobkins KR. Comparing the effects of auditory deprivation and sign language within the auditory and visual cortex. J. Cog. Neurosci. 2005;17(10):1621–1637. doi: 10.1162/089892905774597173. [DOI] [PubMed] [Google Scholar]
  16. Gallese V, Lakoff G. The brain’s concepts: the role of the sensory-motor system in conceptual knowledge. Cognitive Neuropsych. 2005;22(3/4):455–479. doi: 10.1080/02643290442000310. [DOI] [PubMed] [Google Scholar]
  17. Glenberg AM, Kaschak MP. Grounding language in action. Psychon Bull & Rev. 2002;9:558–565. doi: 10.3758/bf03196313. [DOI] [PubMed] [Google Scholar]
  18. Grammer K, Filova V, Fieder M. The communication paradox and possible solutions. In: Schmitt A, Atzwanger K, Grammer K, Schaefer K, editors. New Aspects of Human Ethology. New York: Plenum Press; 1997. pp. 90–120. [Google Scholar]
  19. Grammer K, Fink B, Renniger L. Dynamic systems and inferential information processing in human communication. Neuroendocrinol. Lett. 2002;23(Suppl. 4):103–110. [PubMed] [Google Scholar]
  20. Grill-Spector K, Knouf N, Kanwisher N. The fusiform area subserves face perception, not generic within category identification. Nat. Neuro. 2004;7(5):555–562. doi: 10.1038/nn1224. [DOI] [PubMed] [Google Scholar]
  21. Grossman ED, Blake R. Brain areas active during visual perception of biological motion. Neuron. 2002;35:1167–1175. doi: 10.1016/s0896-6273(02)00897-8. [DOI] [PubMed] [Google Scholar]
  22. Hauk O, Johnsrude I, Pulvermuller F. Somatotopic representation of action words in human motor and premotor cortex. Neuron. 2004;41(2):301–307. doi: 10.1016/s0896-6273(03)00838-9. [DOI] [PubMed] [Google Scholar]
  23. Indefrey P. A meta-analysis of hemodynamic studies on first and second language processing: Which suggested differences can we trust and what do they mean. Language Learning. 2006;56:279–304. [Google Scholar]
  24. Kable JW, Lease-Spellmeyer J, Chatterjee A. Neural substrates of action event knowledge. J. Cog. Neurosci. 2002;14(5):795–805. doi: 10.1162/08989290260138681. [DOI] [PubMed] [Google Scholar]
  25. Kaschak MP, Madden CJ, Therriault DJ, Yaxley RH, Aveyard M, Blanchard AA, Zwann RA. Perception of motion affects language processing. Cognition. 2005;94(3):B79–B89. doi: 10.1016/j.cognition.2004.06.005. [DOI] [PubMed] [Google Scholar]
  26. Kemmerer D, Castillo JG, Talavage T, Patterson S, Wiley C. Neuroanatomical distribution of five semantic components of verbs: Evidence from fMRI. Brain Lang. 2008;107:16–43. doi: 10.1016/j.bandl.2007.09.003. [DOI] [PubMed] [Google Scholar]
  27. Kohnert K, Bates E, Hernandez AE. Balancing bilinguals: Lexical-semantic production and cognitive processing in children learning Spanish and English. JSLHR. 1999;42:1400–1413. doi: 10.1044/jslhr.4206.1400. [DOI] [PubMed] [Google Scholar]
  28. Lin L, Lu X, Fang F, Han Z, Bi Y. Is the category semantic effect in lateral temporal cortex due to motion property differences? NeuroImage. 2011;55:1853–1864. doi: 10.1016/j.neuroimage.2011.01.039. [DOI] [PubMed] [Google Scholar]
  29. MacSweeney M, Woll B, Campbell R, Calvert GA, McGuire P, David A, Simmons A, Brammer MJ. Neural correlates of British Sign Language comprehension: spatial processing demands of topographic language. J. Cog. Neurosci. 2002;14(7):1064–1075. doi: 10.1162/089892902320474517. [DOI] [PubMed] [Google Scholar]
  30. Malikovic A, Amunts K, Schleicher A, Mohlberg H, Eickhoff SB, Wilms M, et al. Cytoarchitectonic analysis of the human extrastriate cortex in the region of V5/MT+: A probabilistic, stereotaxic map of area hOc5. Cerebral Cortex. 2007;17:562–574. doi: 10.1093/cercor/bhj181. [DOI] [PubMed] [Google Scholar]
  31. Meteyard L, Bahrami B, Vigliocco G. Motion detection and motion verbs: Language affects low-level visual perception. Psych Sci. 2007;18(11):1007–1013. doi: 10.1111/j.1467-9280.2007.02016.x. [DOI] [PubMed] [Google Scholar]
  32. Meteyard L, Zokaei N, Bahrami B, Vigliocco G. Visual motion interferes with lexical decision on motion words. Curr. Biol. 2008;18(17):R732–R733. doi: 10.1016/j.cub.2008.07.016. [DOI] [PubMed] [Google Scholar]
  33. Neville HJ, Lawson D. Attention to central and peripheral visual space in a movement detection task. III-Separate effects of auditory deprivation and acquisition of a visual language. Brain Res. 1987;405:284–294. doi: 10.1016/0006-8993(87)90297-6. [DOI] [PubMed] [Google Scholar]
  34. Perani D, Abutalebi J, Paulesu E, Brambati S, Scifo P, Cappa SF, et al. The role of age of acquisition and language usage in early, high-proficient bilinguals: An fMRI study during verbal fluency. Human Brain Map. 2003;19(170):170–182. doi: 10.1002/hbm.10110. [DOI] [PMC free article] [PubMed] [Google Scholar]
  35. Pulvermüller F, Härle M, Hummel F. Walking or talking?: Behavioral and neurophysiological correlates of action verb processing. Brain Lang. 2001;78:143–168. doi: 10.1006/brln.2000.2390. [DOI] [PubMed] [Google Scholar]
  36. Revill KP, Aslin RN, Tanenhaus MK, Bavelier D. Neural correlates of partial lexical activation. PNAS. 2008;105(35):13111–13115. doi: 10.1073/pnas.0807054105. [DOI] [PMC free article] [PubMed] [Google Scholar]
  37. Rueschemeyer SA, Glenberg AM, Kaschak MP, Mueller K, Friederici AD. Top-down and bottom-up contributions to understanding sentences describing objects in motion. Front. Psychology. 2010;1(183) doi: 10.3389/fpsyg.2010.00183. [DOI] [PMC free article] [PubMed] [Google Scholar]
  38. Saygin AP, Wilson SM, Dronkers NF, Bates E. Action comprehension in aphasia: linguistic and non-linguistic deficits and their lesion correlates. Neuropsychologia. 2004;42(13):1788–1804. doi: 10.1016/j.neuropsychologia.2004.04.016. [DOI] [PubMed] [Google Scholar]
  39. Saygin A, McCullough S, Alac M, Emmorey K. Modulation of BOLD response in motion sensitive lateral temporal cortex by real and fictive motion sentences. J. Cog. Neuro. 2010;22(11):2480–2490. doi: 10.1162/jocn.2009.21388. [DOI] [PMC free article] [PubMed] [Google Scholar]
  40. Stokoe W. Studies in linguistics, occasional papers 8. Silver Spring, MD: Linstok Press; 1960. Sign language structure: An outline of the visual communication systems of the American Deaf. [DOI] [PubMed] [Google Scholar]
  41. Tootell RB, Reppas JB, Kwong KK, Malach R, Born RT, Brady TJ, et al. Functional analysis of human MT and related visual cortical areas using magnetic resonance imaging. J Neurosci. 1995;15:3215–3230. doi: 10.1523/JNEUROSCI.15-04-03215.1995. [DOI] [PMC free article] [PubMed] [Google Scholar]
  42. Watson JD, Myers R, Frackowiak RS, Hajnal JV, Woods RP, Mazziotta JC, et al. Area V5 of the human brain: Evidence from a combined study using positron emission tomography and magnetic resonance imaging. Cerebral Cortex. 1993;3:79–94. doi: 10.1093/cercor/3.2.79. [DOI] [PubMed] [Google Scholar]
  43. Zwaan RA, Taylor LJ. Seeing, acting, understanding: motor resonance in language comprehension. J Exp Psychol [Gen] 2006;135(12):1–11. doi: 10.1037/0096-3445.135.1.1. [DOI] [PubMed] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

01
NIHMS390844-supplement-01.pdf (32.3KB, pdf)
02
Download video file (1MB, mov)
03
Download video file (1MB, mov)

RESOURCES