The grounding of temporal metaphors

Abstract

Grounded cognition suggests that the processing of conceptual knowledge cued by language relies on the sensory-motor regions. Does temporal language similarly engage brain areas involved in time perception? Participants read sentences that describe the temporal extent of events with motion verbs (Her seminar stretches across the afternoon) and their static controls. Comparison conditions were fictive motion (Her backyard stretches across the desert) and literal motion (Her arm stretches across the table), along with their static controls. Several time sensitive locations, identified using a meta-analysis, showed activation specific to temporal metaphors, including in the left insula, right claustrum, and bilateral posterior superior temporal sulci. Fictive and literal motion contrasts did not show this difference. Fictive motion contrast showed activation in a conceptual motion sensitive area of the left posterior inferior temporal sulcus. These data suggest that language of time is at least partially grounded in experiential time. In addition, motion semantics has different consequences for events and objects: temporal events become animate, while static entities become motional.

1. Introduction

We experience time continuously, and we talk about time from time to time. The semantic processing of temporal concepts, while very common in language, is a mystery. According to grounded cognition theories, the processing of conceptual knowledge cued by language relies on the sensory-motor regions of the brain (Barsalou, 2008, Gallese & Lakoff, 2005). This has been supported by a large body of evidence, in the domains of action, vision, audition, and emotion (Binder & Desai, 2011). Does temporal language similarly engage brain areas involved in the processing of experiential time? We can typically comprehend concepts describing temporal intervals from seconds to millennia in a few hundred milliseconds. Nonetheless, it is possible that some level of grounding is maintained in the regions sensitive to temporal processing, especially when time is described metaphorically. Here, we ask whether metaphorical temporal language processing maintains grounding in brain regions implicated in the perception of time.

Neuroimaging work has implicated an extensive, distributed network for time perception, estimation, and production (Meck et al., 2008; Penney & Vaitilingam, 2008; Wiener et al., 2010, Merchant et al., 2013). While there is no evidence for dedicated timing machinery, several regions have been consistently associated with temporal processing, as identified in a meta-analysis (Wiener et al. 2010). In studies of supra-second timing capabilities, most studies have employed motor timing tasks (e.g., finger tapping) or perceptual timing tasks (e.g., comparing two time intervals). Partially overlapping sets of regions in both hemispheres have been identified for both of these tasks (see Methods).

Instead of investigating whether temporal language is related to temporal perception, numerous studies have examined whether temporal language is related to spatial perception (Boroditsky & Ramscar, 2002; Bottini et al., 2015) or to spatial language (Boroditsky, 2000; Kemmerer, 2005; Teuscher et al., 2008; Kranjec, Cardillo et al., 2010; Lai & Boroditsky, 2013). The majority of the findings support the idea that temporal language is tied to spatial perception, cognition, and language (Lakoff & Johnson, 1999). For instance, Matlock, Ramscar, & Boroditsky (2005) tested whether motion semantics was recruited in descriptions of non-motion entities and scenes, known as fictive motion, e.g., “The tattoo runs along his spine” and ambiguous temporal statements “Next Wednesday’s meeting has been moved forward two days”. They found that reading fictive motion sentences affected subsequent interpretation of the ambiguous temporal statements (i.e., whether the meeting has been moved earlier or later).

Not all studies support the association between space and time. Kemmerer (2005) examined spatial-temporal prepositions (e.g., “at”), which can be read temporally (e.g., “at noon”) and spatially (e.g., “at home”), in four patients with left perisylvian lesions. Two patients performed significantly better on understanding spatial than temporal meanings, and the other two showed the reverse pattern. Lower performance on spatial meaning was associated with damage to the left supramarginal gyrus. Localization of temporal processing was unclear, beyond the suggestion that it was likely perisylvian. Kranjec et al. (2012) examined space, time, and causality using short phrases such as “The dog barked” and “The rain poured”. Participants judged the space/time/causal relations between the short phrases. They found that space trials, compared against the other two, activated visual areas including bilateral frontal and occipitoparietal networks. The time contrast showed no significant activation. The causality contrast, however, showed activations in time processing areas such as supplementary motor area (SMA), caudate, and cerebellum. Reflecting on these findings, Kranjec and Chatterjee (2010) noted that the brain regions involved in temporal perception as at least partially known, and it should be possible to examine representations of temporal concepts and their grounding. They questioned whether temporal concepts are grounded in spatial perception areas, time perception areas, or not grounded at all, and pointed out that there is little neural evidence for grounding of temporal concepts. (We add that on the flip side, there is no evidence, either neural or behavioral, of a circular amodal symbol system representing, or having the capability to represent, concepts.)

The present study investigated the extent to which temporal language activates areas implicated in temporal processing. We used fictive motion sentences about time (FM-time). These sentences use motion verbs to describe the temporal extent of events, e.g., “The graduation party stretches through the night”. As comparisons, two other types of sentences were used: fictive motion sentences for space (FM-space), e.g., “Her backyard stretches across the desert”, and literal motion (LM), e.g., “Her arm stretches across the table”. For each sentence type, corresponding static versions were used as controls (Table 1). Activation in temporal processing regions by FM-time sentences, relative to their static controls, addresses the question whether temporal language leads to activations in time processing regions. The FM-space and LM sentences allow us to test whether this activation is specific to time, or is more general and also elicited by other types of figurative motion or by literal motion sentences.

Table 1.

Example stimuli.

Conditions	Sentences
FM-time	Her seminar stretches across the afternoon.
FM-time-control	Her seminar is in the afternoon.
FM-space	Her backyard stretches across the desert.
FM-space-control	Her backyard is by the desert.
Literal Motion	Her arm stretches across the table.
LM-control	Her arm is on the table.

2. Methods

2.1. Participants

Twenty-three healthy, right-handed, native English speakers participated in the experiment for payment. One participant was removed due to high degree of movement. The remaining twenty-two (11 female) included in the analysis have a mean age of 21.9 (SD=3.13, range 19–34). All participants gave written informed consent prior to participation.

2.2 Materials

The stimuli were sentences, divided into six conditions (Table 1). The temporal fictive motion sentences (FM-time) each described the temporal extent of an event using a motion verb. The spatial fictive motion (FM-space) sentences each described the spatial spread of a concrete entity using the same verb. The literal motion (LM) sentences each described motion of an animate agent using the same verb. These conditions necessarily differed in terms of nouns. Hence in a direct comparison between them, it would not be clear whether the differences were only due to the nouns or due to sentence-level meaning. Therefore we also included three control conditions corresponding to each of the sentence types. These conditions used the same nouns, but described static scenarios using locative prepositions or static verbs. Critical comparisons constituted the contrasts between each motion sentence and its static control.

62 sets of six conditions each were selected (from a larger set of 98 sets created initially) such that each motion condition was matched to its control on a number of variables: number of words, phonemes, syllables, and letters, content words’ log frequencies (Balota et al., 2007), content words’ concreteness ratings (Brysbaert et al., 2014), and comprehension difficulty ratings (Table 2). Comprehension difficulty ratings were collected using Amazon Mechanical Turk. 16 native English speakers whose IP addresses were located in the United States participated. The original 588 sentences, and an additional 216 nonsense or difficult-to-understand sentences (e.g., A rocking horse cooked vacuum cleaner pies.) included as a quality check measure, were rated on a scale from 1 (very easy to understand) to 7 (very difficult to understand).

Table 2.

Stimulus characteristics. The mean values of each variable, and the p value from a t-test between each motion condition and its control, are shown.

Conditions	# words	# phonemes	# syllables	# letters	Log Freq.	Concrete ness	Comprehension difficulty
FM-time	8.06	28.36	10.81	37.08	3.29	3.52	2.26
FM-time-control	8.44	29.28	11.42	37.61	3.43	3.58	1.94
FM-space	8.28	28.61	10.78	36.44	3.07	4.28	2.31
FM-space-control	8.72	28.75	11.39	36.03	3.14	4.39	2.07
LM	8.14	27.83	10.42	34.75	3.33	4.18	2.04
LM-control	8.64	28.83	11.64	34.78	3.48	4.31	1.98
FM-time vs control (p)	0.388	0.585	0.426	0.797	0.158	0.423	0.091
FM-space vs control (p)	0.330	0.925	0.335	0.827	0.490	0.226	0.132
LM vs control (p)	0.221	0.536	0.128	0.989	0.100	0.136	0.717

Fillers of 36 grammatical and meaningful sentences such as “The ice cream shop features a huckleberry milkshake” were included in the fMRI experiment to conceal the set-wise construction of the stimuli, and provide variability in the materials.

2.3 Procedure

A rapid event-related design was used, with stimuli divided into six runs, six minutes each. In each run, the orders and durations of the trials were pseudo-randomized via the optseq2 software (Dale, 1999). Each sentence was displayed for 2.3 seconds, and a fixation cross was displayed in the inter-trial interval whose duration was determined by optseq2. Participants were instructed to read attentively. A probe word appeared after 25% of the sentences, randomly selected for each participant. The participants were to determine whether the word was related to the sentence, by pressing a yes/no button with their index/middle fingers. Half of the participants used their right hand to respond, and the other half, left. There was a practice with 8 trials.

2.4. MRI data acquisition

MR data were collected on a Siemens Medical Systems 3T Trio. The multi-echo whole brain T1 scans had 1 mm isotropic voxel size with the following parameters: TR=2250 ms, TE=4.15 ms, T1=925 ms, flip angle=9°, FoV=256 mm. Functional images were acquired using a multiband echo, echo-planar images with the following parameters: TR=1200 ms, TE=34 ms, flip angle=65°. Volumes consisted of 36 3.3 mm slices in axial orientation. Each volume covered the whole brain with FoV=212mm and voxel size =3.3 × 3.3 × 3.0 mm³.

2.5. MRI analysis

The AFNI software (Cox, 1996) was used for analysis, using the afni_proc.py processing pipeline. Within-subject analysis involved spatial co-registration (Cox and Jesmanowicz, 1999), registration of functional images to the anatomy (Saad et al., 2009), and projection of images to standard stereotaxic space (Talairach and Tournoux, 1988), and smoothing with a 6 mm FWHM Gaussian kernel.

The preprocessed fMRI time series were analyzed with condition regressors corresponding to the six conditions along with the fillers and the probe words. Additionally, reference functions representing the six motion parameters, and the average signals extracted from white matter and cerebrospinal fluid were also included as covariates of no interest. Voxel-wise multiple linear regression was performed with the program 3dDeconvolve, using these reference functions representing each condition convolved with a standard hemodynamic response function.

2.6. Regions of Interest

We performed a hypothesis-driven analysis to investigate whether regions associated with time perception were activated for temporal or motion sentences relative to their static controls.

We used time sensitive locations reported in a meta-analysis by Wiener et al. (2010). Across perceptual and motor timing tasks,Wiener et al. (2010) reported 27 coordinates (Tables 5 and 6 in Wiener et al. 2010). In order to select the most consistent areas as part of our a priori hypothesis, we used the coordinates in the areas reported in both perceptual and motor timing tasks. This led to 14 coordinates in the right cingulate gyrus, claustrum, middle frontal gyrus (MFG), and SMA, and the left insula and precentral gyrus (each area containing 2 coordinates, one from perceptual and the other from motor timing). The remaining 13 coordinates in the following areas were also examined: bilateral IFG and supramarginal gyrus, left cingulate gyurs, putamen, and superior frontal gyrus, and right middle temporal gyrus (MTG), inferior parietal lobule, and precentral gyrus. False Discovery Rate was used for correction for multiple comparisons (Benjamini and Hochberg, 1995).

We also defined ROIs to examine activation related to motion, based on the hypothesis that both FM-space and FM-time sentences may induce motion-related processing. Two main areas are associated with motion – the MT+ complex and posterior superior temporal sulcus (pSTS), the latter associated with biological motion. We used the average MT+ coordinates reported in Saygin et al. (2010), bilaterally. In addition to the MT+ proper, in studies of motion/action semantics, activation was frequently found anterior to the left area MT+, interpreted as a partially abstracted motion processing (Binder and Desai, 2011). Thus we also defined an ROI in this anterior area, based on a study that examined motion-related concepts using linguistic stimuli (Kable et al., 2005), at [−52, −57, −7] in the left posterior inferior temporal sulcus (ITS). For pSTS, we used the bilateral masks from the maximum probability map (TT_desai_dd_mpm probabilistic atlas) created with the Destrieux et al. (2010) parcellation in AFNI.

The right pSTS was also found to be sensitive to perceptual temporal processing in Wiener et al. (2010). Thus, the right pSTS served as a ROI for both motion and time.

We placed spheres of 10 mm diameters centered at each coordinate and examined activations for the contrasts FM-space>FM-space-control, FM-time>FM-time-control, and LM>LM-control in these ROIs.

Whole brain analyses tend to have low sensitivity to activations in small, targeted regions, due to the need for corrections for multiple comparisons. Nonetheless, we also conducted whole brain analyses comparing the three motion conditions with their respective controls, and also for the interaction (FM-time > FM-time-control) > (FM-space > FM-space-control), to examine differences between temporal and spatial metaphors anywhere in the brain, although these analyses were not likely to reveal all temporally sensitive regions. Group maps were thresholded at uncorrected p<0.01, and corrected using Monte Carlo simulations to a mapwise α<0.05.

3. Results

The average percentage of participants responding to the probe questions was 89% (SD=9%), with averge accuracy of 72 % (SD=12).

In the temporal ROIs, greater activation for FM-time relative to FM-time control was found in the left insula (t(21)=2.342, p<0.05, Cohen's d=0.35), right claustrum (t(21)=2.338, p<0.05, Cohen's d=0.42) (Figure 1), and bilateral pSTS (left: t(21)=2.830, p<0.01, Cohen's d=0.28, right t(21)=3.487, p<0.005, Cohen's d=0.39) (Figure 2). Marginal differences between the two were found in the left precentral gyrus (t(21)=1.678, p=0.05, Cohen's d=0.16) and the right SMA (t(21)=1.438, p=0.08, Cohen's d=0.19). No difference was found for FM-space>FM-space-control or LM>LM-control, in any of the temporal ROIs.

Figure 1.

Activation in temporal ROIs. Y axis represents percent signal change. Error bars indicate standard error of the mean. Dark bars indicate motion sentences, and light bars indicate corresponding static control sentences. 6 ROIs in the bilateral cingulate gyri, right claustrum, right middle frontal gyrus, left precentral gyrus, right supplementary motor area did not show a difference between in any of the contrasts, and are not shown. * < .05, ** < .01, *** < .005

Figure 2.

Activation in ROIs associated with motion processing. Other details are the same as Fig 1.

Among the post-hoc time-sensitive ROIs, greater activation for FM-time relative to FM-time-control was found in the left IFG (t(21)=2.680, p<0.01, Cohen's d=0.48).

In the motion-related ROIs, FM-space > FM-space-control activation was found in the left posterior ITS (pITS) (t(21)=1.846, p<0.05, Cohen's d=0.20) (Figure 2). No difference was found in the MT+. No differences were found for LM>LM-control in any of the motion areas.

In a direct comparison, the difference between FM-time and FM-time control was greater than the difference between FM-space and FM-space control in the right claustrum (t(21)=1.739, p<0.05, Cohen's d=0.53), left pSTS (t(21)=2.057, p<0.05, Cohen's d=0.55), and right pSTS (t(21)=1.821, p<0.05, Cohen's d=0.60).

In the whole brain analysis, FM-time > FM-time-control contrast revealed activations in the left posterior MTG and IFG, the right anterior insula, anterior superior temporal gyrus (STG), as well as the posteiror and middle STS. No regions survied in the FM-space and LM contrasts.

A direct comparions between the FM-time and FM-space contrasts revealed greater activation for the time contrast in the left posteiror STS, MTG, IFG, posterior insula, and the right posterior insula and claustrum. (See Suppplementary Material for coordinates and maps.)

4. Discussion

Processing temporal semantics, at least when temporal events are described in a fictive manner using action/motion verbs, activates some of the areas that keep track of time intervals. This is the first evidence of the involvement of time processing areas in temporal language. This activation is not simply due to the use of motion verbs or the metaphoric nature of the sentences, because the fictive motion and literal motion sentences do not show this difference relative to their controls. Thus, the temporal nature of the sentences appears to be the critical factor for eliciting activation in temporal regions.

The temporal processing regions found to be sensitive to time metaphors – the left insula, right clustrum, and bilateral pSTS – were all perisylvian regions. This is consisent with the lesions in the patients with temporal processing deficit in Kemmerer (2005). Perisylvian regions are commonly associated with language processing, but appear to serve as an interface between language and temporal processing.

It is noteworthy that Kemmerer (2005) tested temporal locations, whereas here temporal locations were used in the control condition (e.g., “in the afternoon”). The condition of interest here was temproal extent (e.g., “across the afternoon”). One possiblity is that regions sensitive to temporal semantics found here respond to both temporal locations and extent, but to a greater degree to the latter. Alternatively, temporal location and extent may be subserved by partially or completely different neural substrates. These possiblities await further investigation. ¹

The posterior insula is thought to function as an accumulator, while the anteiror insula is associated with comparing two time intervals through meta-representations leading to awareness (Craig, 2009). In two studies (Wittmann et al., 2010, Wittmann, Simmons et al., 2011), participants encoded durations of 9 or 18 seconds and later reproduced them. During the encoding phase, bilateral posterior insula and STS showed increasing activation with increasing duration. Accumulating activation was seen in the reproduction phase in bilateral anterior insula, adjacent IFG, and SMA. Similarly, we found that both anterior and posterior insula, STS, and SMA were sensitive to temporal sentences (or showed trends). No explicit comparison of temporal intervals was required in sentence reading, yet both time accumulation and comparison-related processes appeared to be activated. This suggests that comprehending temporal concepts engages regions sensitive to temporal processing, and additionally, may engage automatic comparisons between the described event duration and a typical duration of such event. For example, “A meeting stretching through the day” may be compared with common durations of meetings, leading to the understanding that the sentence describes an unusually long meeting.

The claustrum is also consistently found to be active in temporal tasks. It is often viewed as a multisensory integration center, integrating percepts from different parts of the brain (Crick and Koch, 2005; Goll et al., 2015). High degree of convergence in claustrum allows it to serve as a “coincidence detector”, detecting synchrony of oscillations in brain areas, which enables its role in interval timing (Smythies et al., 2012, 2014). Here, the claustrum may integrate the temporal features emphasized by the time metaphors.

The pSTS is a highly multi-functional area (Liebenthal et al., 2014), associated with both time and motion perception. One function relevant here is biological motion and animacy (Grosbras et al., 2012; Grossman and Blake, 2002). The animacy interpretation has implications for cognitive theories of time metaphors. It has been proposed that one of the primary time-is-motion metaphors is a time-moving metaphor (Lakoff & Johnson, 1980, Gentner, 2001), e.g., “Christmas is approaching”, where time is conceived as an entity moving toward the speaker. All sentences here were time-moving metaphors. The pSTS activation, if interpreted as representing animacy, suggests that time-space mapping includes personification process. The temporal event “seminar” in “seminar stretches across the afternoon” is likely personified as an animate being that physically stretches.

The SMA and the left precentral gyrus were marginally activated in the temporal contrast. SMA was activated more in the sub-second than the supra-second tasks (Wiener et al., 2010), and more in the automatic than cognitively controlled timing tasks (Lewis and Miall, 2003). Weaker activation of SMA here was perhaps due to the fact that the durations depicted in the temporal sentences were much longer (>1 second) and likely required more cognitive control.

Among our post-hoc time regions, the perceptual timing region of left IFG was significant in the temporal contrast. Like the pSTS, IFG is also a famously multi-functional area. The specific coordinates found in time perception studies fall in the anterior portion of the IFG (Brodmann areas 47 and 45). The left anterior IFG is also activated for semantic processing, and commonly interpreted as involving executvie processing specific to semantics (Binder et al. 2009). Here, the activation can be interpreted as a specific type of temporal semantics, leveraging the same type of executive processing that occurs in perceiving or comparing time intervals. It is unlikely that this activation was due to general comprehension difficulty, as comprehension difficulty was matched between conditions.

Fictive motion sentences for space showed activation in the ROIs in the left pITS, anterior to the motion processing complex MT+, relative to its static control. This region is commonly activated in studies of action and motion semantics (e.g., Noppeney and Price 2004; Kable et al., 2005; Desai et al., 2013). Its likely role is in processing a partially abstracted version of motion, engaged in language (Chen et al., 2008). Its activation for FM-space contrast is consistent with this proposal, and suggests that motion features are attributed to static spatial entities (e.g., “backyard”) when described using fictive motion. This does not extend to temporal entities (e.g., “seminar”). Thus, the findings present a double dissociation between FM-time and FM-space sentences. The pSTS, implicated in animacy and biological motion, was activated by FM-time but not FM-space. The pITS, implicated in inanimate, spatial motion, was activated by FM-space but not FM-time. This suggests that events, when described fictively, are processed as animate or personified entities. Static entities are not personified, but are imbued with features representing physical motion across a landscape.

FM-space or LM sentences did not show significant activation in the MT+ relative to their controls, inconsistent with Saygin et al. (2010). This is potentially due to the fact that Saygin et al. (2010) localized MT+ individually using an independent motion localizer task, whereas we used averaged coordinates. The use of averaged coordinates likely reduced sensitivity greatly, given that the location of MT+ is highly variable between individuals. In Saygin et al. (2010), the peak locations of MT+ varied up to 17 mm in x, 16 mm in y, and 15 mm in z directions.

The LM > LM-control contrast did not show a difference in either pSTS or pITS. While this finding was unexpected, it is consistent with Chen et al. (2008), who also found no activation in their literal motion vs. no-motion contrast. This may be because in both studies, both motion and control sentences used animate agents that evoke motion semantics. Animate agents are likely sufficient to elicit motion processing and imagery.

We caution that the functions of temporal perception regions of the brain are not fully understood. While regions such as posterior and anterior insula, claustrum, and the posterior STS are frequently interpreted as temporal accumulators and comparators, and there is evidence from multiple experiments supporting these interpretations, we cannot yet be sure of the exact computations in these regions. We obtained moderate to small effect sizes, but several regions of interest also survived a whole-brain analysis, increasing our confidence. The temporal regions are involved in numerous other functions, leading to the obvious concern as to how specific the activations are to temporal semantics. However, we used specific coordinates found to be time-sensitive across a multitude of studies, rather than searching within large anatomical regions such as the insula or STS. Combined with the use of the carefully matched control conditions, we are optimistic that our results reflect temporal semantics and their grounding in time perception circuitry, rather than general cognitive processes related to semantics or executive functioning. Further careful experimentation with a verity of stimuli and tasks would be of obvious interest.

Summarizing, processing temporal metaphors activates some of the areas involved in time perception, suggesting that temporal language processing is partially grounded in time perception circuitry. Furthermore, both static entities and events, when described with fictive motion, evoke motion semantics, but in different ways. Events evoke animate motion and state entities evoke inanimate motion.

Highlights.

• -Time-is-motion metaphors showed activation in time sensitive areas.
• -Controls showed that this is not simply due to metaphor or motion verb use.
• -An area associated with biological motion also activated from temporal language.
• -The language of time is at least partially grounded in the perception of time.