Fronto-temporal regions encode the manner of motion in spatial language

Abstract

When describing spatial events, dynamic actions can be decomposed into the path of motion (where the object moves), and the manner of motion (how the object moves). These components may be instantiated in two processing streams in the human brain, wherein dorsal parietal areas process path-related information, while ventral temporal regions process manner information. Previous research showed this pattern during the observation of videos showing animate characters in motion [15] It is unknown whether reading language describing path and manner information—a level of abstraction beyond the perception of visual motion—relies on similar mechanisms. Here, we use functional neuroimaging to show that the left pMTG processes the manner of motion during reading. We also demonstrate the involvement of other ventral fronto-temporal regions in the understanding of manner of motion in spatial language.

Introduction

Language and action are closely intertwined. Much research suggests that overlaps in the language and action processing systems of the brain support our ability to extract event-related information from language [1]. These overlaps may reflect some degree of common neural coding between action language processing and action perception. While the precise nature of this overlap is debated, ample evidence demonstrates that brain activity during reading of action language is, to a degree, evocative of the production or observation of corresponding actions [2-7].

One open question concerns how we process linguistic information describing paths and manners of motion. Path and manner are two important components of how entities move in space: path refers to the trajectory of a moving figure with respect to its background, while manner refers to the way a figure moves regardless of its trajectory [8]. For example, in the phrase The girl skips down the hill, ‘skips’ refers to the manner of motion, and ‘down the hill’ refers to the path of motion. These components can be thought of as distinguishing intrinsic or extrinsic motion [9]. Intrinsic manners of motion refer to the motion of local parts within a coherent whole object, whereas extrinsic paths of motion refer to larger-scale trajectories of motion in which figures move against a ground.

Manner and path are generally linguistically segregated in English [10, 11], with manner expressed by action verbs (e.g. “skips”), and path expressed by locative prepositions (e.g. “through”). In other languages, like Turkish and Mandarin Chinese, however, manner and path are expressed differently. In Turkish, manner and path are expressed in two independent clauses (e.g., “The ball descended the hill while rolling”) [12]. In Mandarin Chinese, two consecutive verbs can be used to describe a spatial event (e.g., “Run exit kitchen” rather than “run out of the kitchen”) [13].

In normal development, even infants notice changes to manner and path of motion events, suggesting that they comprise two separable components for understanding events [14]. More recent neuroimaging work with adults has shed light on the differential categorization of path and manner during the perception of dynamically moving stimuli. In a study from our laboratory, when participants viewed videos depicting moving characters, paths of motion were encoded in bilateral middle frontal gyrus (MFG), the right superior parietal lobule, and the left inferior parietal lobule (IPL), while manners of motion were encoded in bilateral postero-lateral inferior and middle temporal regions [15]. Here, we ask whether the same regions are involved in processing path and manner during action language processing. Such a finding would provide evidence for commonalities between verbal, conceptual, and perceptual representations of action [9, 16]. Additionally, we used a functionally-masked analysis to examine what additional brain regions might be involved in processing path and manner in a language-based task.

Path and manner information is generally conveyed in language by prepositions and verbs, respectively. While extensive research has examined the neural correlates of noun and verb processing [17, 18], less is known about how prepositional phrases are processed. Some prior work shows that the neural instantiations of action verbs and locative prepositions are largely independent [19]. Specifically, prepositions are associated with activity in the supramarginal gyrus of the parietal lobe [19, 20], while verbs generally involve the inferior and middle frontal gyrus [18, 21-23] and the posterior middle temporal gyrus (pMTG) [24-26]. Importantly, the pMTG is also related to conceptual action representation, beyond simply processing verb-related stimuli [27-29]. We hypothesize that the cortical regions involved in processing visual motion will be recruited similarly during reading about spatial motion. Additionally, we hypothesized that the grammatical structure of path and manner trials (i.e. prepositions and verbs) would be reflected in the neuroimaging results.

We designed a functional neuroimaging (fMRI) study to investigate these questions. Participants performed a matching task for three different trial types (Path, Manner, and False). We examined whether the regions that are involved in path and manner processing during the perception of moving figures would show the same pattern during a reading task. We predicted that the MFG and the IPL would be more active while reading about the path of motion, while the pMTG would be more active while reading phrases about manners of motion. We also predicted that within the language-processing network, verb- and preposition-processing regions of the brain would be active for Manner and Path, respectively.

Materials and methods

Participants

Sixteen right-handed participants (9 females; mean age = 25.75, SD = 3.80) participated in exchange for monetary compensation. All participants gave their informed consent, and none reported history of neurological abnormality.

Stimuli

Stimuli were 72 triads describing either the path or manner of a movement (“Path” and “Manner” conditions). Each triad consisted of a phrase-base (either a preposition + the in the Path condition or a verb + the in the Manner condition) and two possible completions, one plausible and one implausible. Path triads used one of six prepositions (across, along, around, through, into, up) for the phrase-base, and Manner triads used one of six verbs (jump, hug, kick, pull, punch, push) for the phrase-base. For each preposition, six plausible and six implausible completions were generated. For a Path trial, “along the” might be paired with “road” (plausible) and “dog” (implausible). The same 60 nouns were then re-paired with each verb in order to generate plausible and implausible completions for Manner triads. For a Manner trial, “jump the” might be paired with “stream” (plausible) and “mist” (implausible). See Supplementary Materials for complete list of stimuli.

All phrases were rated on a 5-point plausibility scale (1 = not at all plausible, 5 = very plausible) by a group of independent raters (n = 8). All plausible phrases received a rating of 3.5 or higher (Path, M = 4.7, SD = 0.35; Manner = M = 4.6, SD = 0.36) and all implausible phrases received ratings of 3 or lower (Path, M = 1.9, SD = 0.47; Manner, M = 1.7, SD = 0.47). Independent t-tests confirmed that Path and Manner phrases did not differ significantly on plausibility. Additionally, the words that made up the correct and incorrect answers for each triad were matched on frequency (Kucera-Francis word frequency) and concreteness (MRC Psycholinguistic Database).

A false-font matching condition (“False”) consisting of word-length displays of the Symbol font served as a control condition.

Procedure

Stimuli were visually presented in a triad format, with the base phrase presented at the top of the screen, and the two targets below. On Path and Manner trials, participants were instructed to select the word on the bottom of the triad that went best with the phrase at the top. On False trials, participants were instructed to select the symbol string at the bottom of the triad that matched the string at the top. Participants were instructed to select the correct target as quickly as possible. Each condition was presented in 6 blocks, with block order counterbalanced across participants. Each block consisted of 6 experimental trials. Each trial lasted 3 seconds, consisting of a 500 ms blank screen followed by 2500 ms presentation of the triad.

Data acquisition and processing

Functional and structural MRI data was collected on a Siemens Trio 3T scanner (Seimens AG, Munich, Germany) using an eight-channel head coil. Functional images were acquired using echo-planar T2*-weighted scans. We collected 50 transversal slices acquired in interleaved order (TR = 3000 ms; TE = 30 ms; flip angle = 90°; FOV = 220 mm; matrix size = 64×64; voxel size = 3.4 × 3.4 × 3 mm). Functional images were collected in one run consisting of 111 volumes. Structural images were acquired for each participant with T1-weighted MP-RAGE scans (transversal slice orientation; TR = 1630 ms; TE = 3.87 ms; flip angle 15°; FOV = 240 mm; matrix size = 256 × 256; voxel size = 1×1×1 mm).

Data were pre-processed using the afni_proc.py script in the AFNI software package (afni.nimh.nih.gov [30]). Specific AFNI commands are given parenthetically below. After discarding the first 2 volumes, data were checked for outliers in each volume (3dToutcount). Volumes containing outliers were not removed, but were used to screen for data quality. Data then underwent slice-timing correction (3dTshift), alignment to skull-stripped anatomical images in Talaraich-Tornaux space, and warping to Talaraich-Tournoux space (3dvolreg and 3dAllineate). A 6.0 mm full-width/half-maximum smoothing kernel was applied to each functional volume (3dmerge). A regression analysis (3dDeconvolve) modeled the time series against the stimuli, and demeaned motion parameters were included as regressors of no interest. TRs during which the participant moved more than .3 mm were excluded.

Coordinate-based ROIs

Using AFNI's 3dcalc command, we created three coordinate-based regions of interest (ROIs) based on our prior work [15]. Each ROI (radius = 6 mm) was centered on the voxel of peak activation for a significant cluster from [15], and since the current study was verbal in nature, only those activations in the left hemisphere were included. The ROIs were located in the IPL, caudal MFG, and pMTG (see Figure 1A).

Figure 1.

A) Three regions of interest in the left hemisphere, created by forming 6-mm radius spheres around the peak voxels for Path-Manner differences in work by [15]. B) Functional mask showing regions in the left hemisphere where activation was greater for Path + Manner compared to False, p < .1 uncorrected, k > 20. Color gradient represents t values.

Functional mask

We created a mask of regions where activation in Path and Manner conditions was significantly greater than in the False condition (Figure 1B). We used a liberal threshold (p < .1 uncorrected, k > 20) since this analysis was not designed to test any hypothesis, but rather to constrain the subsequent analysis of Path compared to Manner. The regions included in this mask are typical of cortical areas activated during reading tasks, including much of the left frontal and temporal lobes [31].

Data analysis

First, we performed a t-test on the contrast comparing Path and Manner across all subjects, within the functional mask. Results were considered significant below a threshold of p < .01, cluster-corrected (k > 17). Second, in each of the ROIs, we used the AFNI command 3dROIstats to average the t-values (from the Path/Manner contrast) at each voxel across all subjects and test this value against zero.

Results

Behavioral results

Participants performed well on the task (see Table 1). We compared all conditions on reaction time (RT) and accuracy. There was no significant difference between Path and Manner for either variable (RT: t(15) = 1.48, p > .1; Accuracy: t(15) = 1.94, p > .05). The False condition had faster RTs and but similar accuracy compared to Path (False vs. Path RT: t(15) = 10.43, p < .001; False vs. Path accuracy: t(15) = 2.08, p > .05), and faster RTs and higher accuracy compared to Manner (False vs. Manner RT: t(15) = 9.03, p < .001; False vs. Manner accuracy: t(15) = 2.48, p = .025).

Table 1.

Accuracy and reaction time (RT) for Path, Manner, and False conditions.

	Path	Manner	False
Mean Accuracy (SD)	.97 (.02)	.95 (.05)	.99 (.02)
Mean RT (SD)	1366 (212)	1408 (246)	953 (213)

Manner-preferring ROIs

Activation was greater for Manner trials than for Path trials in the pMTG (t(15) = 2.15; p = .024; see Table 2). No significant difference in activation between Path and Manner was seen in the IPL or caudal MFG.

Table 2.

Regions of Interest.

	Center coordinates
Region	X	Y	Z	t(15)
Posterior middle temporal gyrus	−47	−69	4	2.15*
Inferior parietal lobule	−37	−44	55	0.28
Caudal middle frontal gyrus	−25	0	55	0.50

All regions are in the left hemisphere, and coordinates are given in standard Talairach space. The regions were created by forming spheres with a 6 mm radius centered at the voxels that showed the greatest differentiation between path and manner conditions in a prior study [15].

^*p = .024.

Masked analysis

Within the functional mask, three regions showed stronger activity during Manner than Path conditions: the left medial fusiform gyrus, the left dorsolateral prefrontal cortex (DLPFC), and BA47 of the inferior frontal gyrus (IFG; ps < .01, cluster corrected; see Figure 3, Table 3). No areas were significantly more active for Path compared to Manner.

Table 3.

Regions significantly more active during Manner than Path conditions, within the functional mask where activity was greater for Manner and Path than for the False condition.

		Peak coordinates
Effect	Region	X	Y	Z	t(15)	k
M>P	L medial fusiform gyrus	−31	−34	−16	4.61*	27
	L inferior frontal gyrus (BA 47)	−37	29	−10	5.10*	22
	L dorsolateral prefrontal cortex	−49	35	20	3.56*	18

^*p < .01, cluster corrected (k > 17).

Coordinates are given in standard Talairach space.

Discussion

The region of interest analysis showed that during a reading task, as when viewing dynamic moving stimuli, the pMTG is more active when processing words describing manners of motion than paths of motion. This finding is consistent with the idea that the pMTG is involved in storing semantic information about actions [25, 29], is involved in processing motion-related action verbs [2], and processes abstract representations of actions [24]. Additionally, this finding is in agreement with prior work from our lab [15], which demonstrated the role of the pMTG in processing manner information while viewing objects in motion. In contrast to this prior work, we did not find greater activity in frontal and parietal regions during processing of path information. This may be due to the abstract nature of spatial language, in comparison to dynamic visual representations of spatial events. Viewing moving objects might more strongly engage path-sensitive regions in fronto-parietal cortices due to the overt nature of the spatial events.

We also demonstrate that BA47 of the IFG, DLPFC, and the medial fusiform were more active during Manner trials. The IFG has been associated with the generation and inflection of action verbs, so its activation during Manner trials may reflect the presence of verbs in these trials. Manner trials likely contained richer semantic information, so the increased activity in BA47 may reflect greater semantic processing [32, 33]. The DLPFC is involved in top-down selection of conceptual information stored in the temporal lobe [34]. Thus, the recruitment of the DLPFC during Manner trials may indicate the controlled recruitment of temporal regions while processing manner information.

The greater activation for Manner than Path in the left medial fusiform gyrus echoes past work showing that this region is responsive to conceptual representations of action [29]. This region may be involved in event processing, as seen in prior work showing increased activation during perception of manipulable tools [35]. Alternatively, these effects may be explained by the greater semantic content of Manner trials [36]. Since all the Manner trials involved bodily actions, these results might reflect the involvement of the fusiform body area (FBA) [37]. The linguistic nature of our stimuli may have caused these effects to be seen in the left FBA, rather than the more typical right FBA.

Based on the current study and our prior work [15], we suggest that the pMTG might have common properties that serve both visual motion and spatial language. Given the ventral stream's association with object identity, we propose that this region computes local motion in the context of a coherent grouped object perceptually (e.g., biomechanical constraints of how limbs move) and conceptually (e.g., the actions in which a dog might participate— running but not flying). This ventral processing may be bound together through inferior and middle longitudinal fasciculus connections along with ventral premotor areas [38].

There were no regions that were significantly more active during Path trials. Since participants were performing an easy task, stimuli may have been processed on a semantically shallow level. Future work in this area should consider the difficulty of the behavioral task. Future work should also incorporate “verbs of inherently directed motion” [39], to further distinguish between path and manner processing, in the absence of differences in grammatical class.

Conclusion

These findings suggest that the manner of motion is encoded in fronto-temporal regions of the brain during reading. This finding reinforces the notion of pMTG involvement in processing dynamic actions across presentation formats and show that action language relies on some of the same neural mechanisms involved in action perception. In a functionally masked analysis, we found that reading about manner activated left DLPFC, IFG/BA47, and the medial fusiform. This shows that in a linguistic task, the grammatical distinctions between typical expressions of path and manner drive further differentiation between the two components.

Highlights.

• Dynamic actions can be described by path (location) or manner (type) of motion.

• We investigated neural encoding of path and manner during reading.

• Reading phrases about the manner of motion involved fronto-temporal regions.

• Posterior middle temporal gyrus involvement may reflect action-related processing.

Figure 2.

Surface renderings (top) and axial slices (bottom) showing areas where Manner > Path, within the functionally-defined mask of the language processing network (see Figure 1B).