Processing Time Shifts Affects the Execution of Motor Responses

Abstract

We explore whether time shifts in text comprehension are represented spatially. Participants read sentences involving past or future events and made sensibility judgment responses in one of two ways: (1) moving toward or away from their body and (2) pressing the toward or away buttons without moving. Previous work suggests that spatial compatibility effects should be observed, where the future is mapped onto responses away from the body, and the past is mapped onto responses toward the body. These effects were observed, but only when participants were moving to make their responses, and only for larger time shifts (e.g., a month).

How are concepts understood? Embodied approaches to cognition suggest that concepts are understood via sensorimotor simulations in which the neural systems that are involved in understanding real objects, actions, and events in the world are used to internally simulate those objects, actions, and events at later points in time (e.g., Barsalou, 2008; Kan, Barsalou, Solomon, Minor, & Thompson-Schill, 2003). Concrete entities, such carrots, are understood by activating the same perceptual and action codes acquired through experience: What do carrots look and feel like? How do they taste? and so on. Thus, representing a carrot calls for the visual system to simulate its shape and color, the auditory system to simulate the sound of a bite, and the motor system for information about the heft of the carrot and the force required by the jaw to bite into the carrot. Although embodied accounts of the sensorimotor grounding of concrete concepts are reasonably straightforward, the simulation of less tangible, abstract concepts is typically considered to be less straightforward (e.g., Arbib, 2008). How might these be understood? The embodied approach proposes that the understanding of such concepts is similarly grounded in domains of concrete experience via our bodies’ systems of perception and action planning (e.g., Arbib, 2008; Barsalou, 1999, 2008; Borghi & Cimatti, 2009, 2010; Gallese & Lakoff, 2005; Glenberg & Kaschak, 2002).

Although we are far from a general embodied approach to the understanding of abstractions (but see Barsalou, 2008; Barsalou, Santos, Kyle Simmons, & Wilson, 2008; Barsalou & Wiemer-Hastings, 2005, for outlines of what form an embodied approach might take; and Andrews, Vigliocco, & Vinson, 2009; Borghi & Cimatti, 2009; Dove, 2009, for accounts that integrate sensorimotor information with either distributional information from language use or different types of non-sensorimotor representations), reports in the literature do suggest that sensorimotor simulations can play a role in the comprehension of such concepts. Glenberg and Kaschak (2002) and Glenberg et al. (2008) argued that the understanding of abstract transfer situations (e.g., transfer of information between individuals) is grounded in the motor system in a manner similar to the understanding of concrete transfer situations (e.g., the transfer of tangible objects between people). Boot and Pecher (in press) found that the understanding of the concept of “categories” is grounded in the concrete representation of a container (i.e., a category is seen as a container in which some items are inside, and some are outside). Similarly, Richardson, Spivey, Barsalou, and McRae (2003) found that understanding abstract verbs such as “respect” involves activation of a spatial image-schema. The work reported in this paper is aimed at exploring the ways that understanding the abstract concept of time is grounded in the concrete understanding of the space around our bodies, being organized along the front–back axis (with future events represented as being in front of the body, and past events represented as being behind the body; Boroditsky & Ramscar, 2002; Lakoff & Johnson, 1980).

The idea that time should be understood through spatial representations (particularly front–back representations) has received support from several sources (Boroditsky, 2000; Boroditsky & Ramscar, 2002; Casasanto & Boroditsky, 2008; Genter, Imai, & Boroditsky, 2002; Lakoff & Johnson, 1980; Torralbo, Santiago, & Lupianez, 2006). Lakoff and Johnson’s (1980) study of linguistic metaphors includes an analysis of temporal metaphors, a number of which employ the front–back axis. Many expressions are based on the metaphor that “life is a journey,” suggesting that future events are in front of us on the path, and past events are behind us on the path. Expressions such as, “I am looking forward to my vacation,” and, “Let’s put the past behind us” similarly suggest a “future in front, past in back” spatial representation of time. Experimental work by Boroditsky and Ramscar (2002) provides further evidence for this claim. They found that thinking about moving changes the perspective that one uses to understand time. Thinking about moving through space primes individuals to think about moving through time, as in the expression, “I am almost to the weekend.” On the contrary, thinking about remaining stationary, or about something moving in one’s direction, primes individuals to think about time moving toward them, as in the expression, “The weekend is fast approaching.” Note that in both expressions, a future event is seen as occupying (or moving in) the space in front of the observer. Finally, Torralbo et al. (2006) and Santiago, Lupianez, Perez, and Fuenes (2007) have reported experimental studies showing that the processing of verbs marked with the past or future tense affects spatial responses on the right–left axis (a point to which we return in the discussion).

Boroditsky and Ramscar’s (2002) results suggest that the use of the front–back axis to represent future and past events may involve not only a spatial component, but also an action component – specifically the notion of moving through space. They examined the movement component of the representation of time by asking participants to imagine someone moving an object in space, or by asking participants to think about time when either moving in a cafeteria line or moving when riding on a train. The purpose of the present study is to ask whether the motion component of spatial representations of time is manifested in the execution of motor responses in the space around one’s body.

Although the execution of motor responses, such as reaches in peripersonal space, have not received much attention in the linguistic or experimental literature on time, there is evidence from neuroscience suggesting that thinking about temporal events should affect the execution of such actions. Walsh (2003) notes that cortical areas known to be involved in the perception of space, and the control of action in space (particularly areas in the inferior parietal lobe, near and slightly ventral to the intraparietal sulcus) are also associated with the understanding of time, temporal concepts, and quantity. Critchley’s (1953) classic account of parietal lobe function includes the observation that lesions resulting in deficits of spatial processing are almost always accompanied by deficits in the understanding of temporal and quantity-related concepts. Walsh (2003) argues that space, time and quantity are linked because of the need for coordinating movement: when planning an action, it is necessary to know, “how much, how long, how fast, and where” (p. 486). It is important to note that the parietal regions identified by Walsh as being important for the processing of space, time, and quantity feed into regions of the premotor cortex (e.g. areas F4 and F5 in non-human primates) known to be involved in motor planning, particularly the planning of reaches and grasps in peripersonal space (see Rizzolati, Sinigaglia, & Anderson, 2008, for an extensive discussion).

Single-cell recording from non-human primates suggest that intraparietal and inferior parietal regions feed into motor planning regions (e.g., Rizzolati et al., 2008), and both case studies from clinical neurology and fMRI studies have shown that these parietal regions are important sites for the representation of time and magnitude (Dehaene, Piazza, Pinel, & Cohen, 2003; Fias, Lammertyn, Reynvoet, Dupont, & Orban, 2003; Simon, Mangin, Cohen, Bihan, & Dehaene, 2002). These lines of evidence form the basis for our prediction that comprehending language about temporal concepts should affect the execution of motor responses. The processing of temporal concepts should activate spatial representations (e.g., locations on the front–back axis) in the parietal regions that play a role both in representing space, time, and quantity, and in preparing motor responses. The activation of these spatial representations will serve to prime motor responses to specific spatial locations: for example, thinking about the future will prime spatial locations in front of the body on the front–back axis, and therefore facilitate the execution of a motor response outward in front of the body. Whereas our hypothesis involves a degree of conjecture, it is worthwhile to note that effects of the sort that we predict have been observed in tasks involving quantity. Processing quantity information has been shown both to activate intraparietal and inferior parietal regions (Chochon, Cohen, Moortele, & Dehaene, 1999; Dehaene, Dehaene-Lambertz, & Cohen, 1998; Dehaene et al., 2003; Fias et al., 2003; Simon et al., 2002) and to affect motor responses in a range of behavioral tasks (e.g., Badets, Andres, Di Luca, & Pesenti, 2007; Badets & Pesenti, 2010; Chiou, Chang, Tzeng, & Wu, 2009; Dehaene, Bossini, & Giraux, 1993; Gevers & Lammerty, 2005; Prado, Henst, & Noveck, 2008; For review of both behavioral and neuroimaging studies, see Hubbard, Piazza, Pinel, & Dehaene, 2005).

Thus, evidence from linguistics, experimental cognitive science, and neuroscience lead to the as-yet-untested prediction that thinking about events in the future or past should affect one’s ability to execute arm movements toward or away from the body. Specifically, it is expected that the understanding of events occurring in the future should facilitate the execution of arm movements away from the body (i.e., moving toward the “future is in front” location) and that the understanding of events occurring in the past should facilitate the execution of arm movements toward the body (i.e., moving toward the “past is behind” location). Confirmation of this prediction is of much theoretical import. Whereas several recent articles have suggested that motor representations play a key role in the understanding of abstractions (e.g., Arbib, 2008; Barsalou, 2008; Gallese & Lakoff, 2005), empirical demonstrations of such effects have lagged behind the theoretical proposals (but see above for some examples). Our goal in this paper is to provide an experimental demonstration that the understanding of time has an influence on the execution of motor responses.

Participants were asked to read a series of texts sentence-bysentence. The display of each sentence was initiated by pressing a “START” button (an elevated “S” key on a standard QWERTY keyboard; see Fig. 1). After reading each sentence, participants were required to make a sensibility judgment before moving onto the next sentence in the text. They made this response by releasing the START button and pressing a button located either close to their body (i.e., a response toward the body) or far from their body (i.e., a response away from the body; see Fig. 1). The critical sentence in each passage indicated a time shift to the future or past. Our prediction was that participants would be faster to indicate that sentences involving a future time shift were sensible when making a response away from their body (i.e., reaching out to a location farther from their body), and that participants would be faster to respond to sentences involving a time shift to the past when making a response closer to their body (i.e., reaching to a location close to the body). This prediction is consistent with the “future = front, past = back” mapping seen in previous work (e.g. Boroditsky & Ramscar, 2002).

Fig. 1.

(A) Profile view of subject position and general apparatus configuration. (B) Top view of keyboard configuration for “away” response condition. (C) Top view of keyboard configuration for “towards” response condition.

Although our general concern was with demonstrating a motor compatibility effect for the comprehension of sentences involving time shifts, we examined two additional issues. First, in order to demonstrate that the use of the front–back axis to represent time is based on movement in space (rather than the location of individual points in space), we manipulated whether participants needed to move in order to execute their sensibility judgment responses. Participants in the Movement experiment (Experiment 1) made their sensibility judgments by moving their hand from the START button to the appropriate response button (as discussed above), but participants in the No Movement experiment (Experiment 2) did not move to make their judgments – their left hand pressed the START button, and their right hand was already positioned on the appropriate response button. Our expectation that sentences involving time shifts should affect sensibility judgment responses in the Movement experiment, but not the No Movement experiment, is consistent with Walsh’s (2003) proposal that the execution of action within reachable space is the “linking function” of the parietal cortex which drives the need for a common processing mechanism between space and time. It is also consistent with Boroditsky and Ramscar’s (2002) data suggesting that movement may a key part of the representation of time on the front–back axis (recall that movement toward a goal state/location was an important driver of participants’ use of the front–back axis to represent time in their studies).

The second additional issue explored in this work concerned the magnitude of the time shift indicated by the critical sentences in each text. Zwaan (1996) and Speer and Zacks (2005) demonstrated that time Shift Magnitude has implications for language processing, with larger time shifts incurring larger processing costs. Within the context of the present study, we can ask whether the presence and magnitude of the motor compatibility effects that are observed depends on the magnitude of the time shift that is described. Because effects of time Shift Magnitude have not been examined in the context of motor compatibility effects, we did not make a strong prediction as to whether the motor effects would be affected by the size of the time shift.

RESULTS

The results for the Movement and No Movement experiments are presented in Figs. 2 and 3, respectively. Analyses were conducted across participants (denoted F1), and across items (denoted F2).

Fig. 2.

Residual response times (with standard deviations) from the movement condition.

Fig. 3.

Residual response times (with standard deviations) from the non-movement condition.

Experiment 1: Movement

The response times for the Movement condition are presented in Fig. 2. Analysis revealed a three-way interaction of Shift Magnitude, Response location, and Shift direction [F1(1, 77) = 6.13, p = .015; F2(1, 21) = 12.11, p = .002] and a main effect for Shift Magnitude in the analysis by participants [F1(1, 74) = 6.79, p = .010; F2(1, 21) = 1.26, p = .275]. Follow-up analyses showed an interaction of Response location and Shift direction when participants responded to the sentences describing time shifts of a month [F1(1, 77) = 9.69, p = .003; F2(1, 11) = 33.93, p = .001]. Consistent with our predictions, participants were faster to respond to future time shifts when moving away from their body, and faster to respond to past time shifts when moving toward their body. The Response location × Time Shift Direction interaction was not significant when participants were responding to sentences involving time shifts of a day [F1 and F2 < 1]. There were no other significant effects.

Experiment 2: No Movement

The response times for the No Movement condition are presented in Fig. 3. There were no statistically reliable interactions or main effects, although the Response location by Shift Magnitude interaction [F1(1, 74) = 3.89, p = .052, F2(1, 21) = 2.84, p = .107] and the main effect of Shift Magnitude [F1(1, 74) = 3.89, p = .073, F2(1, 21) = 1.26, p = .275] approached significance. All other F-values were < 3.08, p > .107]. Thus, there was no evidence of a “future = front, past = back” compatibility effect in the No Movement experiment.

To further confirm that the pattern of data observed in the Movement experiment was different from that observed in the No Movement experiment, we conducted a cross-experiment analysis. This analysis revealed a four-way interaction of Shift Magnitude, Response location, Shift Direction and Experiment [F1(1, 151) = 4.78, p = .03; F2(1, 21) = 18.58, p = .001]. Thus, the three-way interaction pattern observed in the Movement experiment is statistically different from the pattern observed in the No Movement condition, strengthening the conclusion that the predicted compatibility effect is observed only when participants need to move to execute their responses.

DISCUSSION

The goal of this study was to test the prediction that thinking about time (specifically, thinking about events in the future or past) would affect participants’ ability to execute motor responses along the front–back axis. With respect to this goal, we can report three main findings. First, we observed a motor compatibility effect in our Movement experiment – participants were faster to produce responses away from their body when processing sentences about future events, and faster to produce responses toward their body when processing sentences about past events. Second, consistent with Boroditsky and Ramscar (2002), we found that movement was important to the representation of time on the front–back axis. Although participants in the No Movement experiment responded to the same physical locations as participants in the Movement experiment, they showed no sign of a spatial compatibility effect. Third, our data show that the motor compatibility effect seen in the Movement experiment was modulated by the magnitude of the time shift to the future or past. There was a motor compatibility effect for large shifts (a month), but not for smaller shifts (a day).

We predicted that the processing of sentences about future or past events would affect the execution of motor responses along the front–back axis based on linguistic evidence derived from analysis of cultural metaphors (Lakoff & Johnson, 1980), experimental evidence showing that moving through space can affect one’s perspective on time (Boroditsky & Ramscar, 2002), and neuroscientific evidence showing that intraparietal and inferior parietal regions implicated in the understanding of space, quantity, and time are also involved in the preparation of actions in peripersonal space (Rizzolati et al., 2008; Walsh, 2003). The finding that the comprehension of large time shifts to the future or past affects motor responses joins with other observations of motor compatibility effects involving sentences with abstract concepts (e.g., Borreggine & Kaschak, 2006; Glenberg & Kaschak, 2002; Glenberg et al., 2008) to support the claim that the mechanisms that are responsible for preparing and executing bodily action play a role in grounding the comprehension of language about abstract situations (at least under some circumstances; see Arbib, 2008; Gallese & Lakoff, 2005). In addition, our data make an important qualification to claims that the processing of space, time, and quantity are closely related: unlike spatial effects involving quantity (e.g., Dehaene et al., 1993; Sell & Kaschak, submitted for publication), compatibility effects involving the understanding of time in a language comprehension task seem to require movement in order to be observed. Thus, consistent with reports from Walsh (2003) and Boroditsky and Ramscar (2002), and with Lakoff and Johnson’s (1980) linguistic analysis it appears that movement may be an important component to the representation of temporal concepts.

Whereas our finding that the effects of time shifts on the execution of responses in different spatial locations are observed only when participants move to make their responses is consistent with the main hypothesis of this paper, it is seemingly at odds with other recent work showing that thinking about temporal concepts such as future and past produce spatial effects on response times in the absence of movement (e.g., Santiago et al., 2007; Torralbo et al., 2006). These spatial effects have largely been observed along the left–right axis (past is left, future is right), although in some cases they have been observed on the front–back axis (e.g., Torralbo et al., 2006). In our view, these results are not contradictory, but rather reflect a distinction in the ways that temporal concepts are handled across tasks. Torralbo et al. (2006) and Santiago et al. (2007) asked participants to make categorical judgments involving the temporal domain that drew explicit attention to that domain (e.g., asking participants to indicate whether a stimulus refers to a past or future event). Under such circumstances, it is likely that the participant uses whatever information is available (e.g., the spatial distinction between the left and right hand) in order to organize their responding to the task. This would be similar to the observation that spatial dimensions can be used in different ways to organize responding in categorical judgment tasks such as those employed in studies of the SNARC paradigm (e.g., Fischer, 2006). In contrast, our experiment did not require participants to explicitly attend to distinctions between past and future events, or to respond specifically to the temporal domain; the time shifts in our sentences were encountered as participants naturally read through the short texts. We suggest that this task context led to time being represented in a less task-dependent manner, with the representation reflecting the widespread cultural convention of mapping the understanding of moving through time onto the understanding of moving through space on the front–back axis (e.g., Lakoff & Johnson, 1980). Thus, we speculate that when temporal concepts are encountered in contexts that do not explicitly call attention to making distinctions in the temporal domain, movement-based effects on the front–back axis should be observed (at least when the time shift is large). In cases where temporal concepts are the subject of explicit categorical judgment, we expect that spatial and movement effects of different sorts may be observed along the front-back and right–left axis. Testing this speculation will be an important agenda for future research in this domain.

We did not directly assess brain activity during our experiments, but our results may nonetheless have implications for our understanding of the neural circuitry involved in the processing of space, time, quantity, and movement. Although very similar neural regions have been identified as being involved in the processing of space, time, and quantity in different tasks, our data suggest that these regions may have different sorts of interplay with the motor system – regions involved in processing quantity and space may not interact with the motor system in the same way as regions involved in processing time. The use of neuroimaging methods to identify the neural circuitry involved in the execution of the range of linguistic and nonlinguistic tasks discussed throughout this paper may provide a fruitful look at the way that intraparietal and inferior parietal regions underlie the comprehension of space, time, and quantity.

Although we did not have strong expectations as to whether the magnitude of time shift depicted in our critical sentences would modulate the size of our motor compatibility effect, we observed that the motor compatibility effect was only present for time shifts of a month. This result is broadly consistent with previous research showing that larger time shifts incur greater processing effort than shorter ones (e.g., Speer & Zacks, 2005; Zwaan, 1996). One intriguing possibility raised by our results is that the use of the spatial domain to represent time shifts may depend on the extent to which the time shift is seen as a break in the ongoing events. As discussed by Zwaan, Langston, and Graesser (1995), Speer and Zacks (2005) and Zwaan and Radvansky (1998), the greater the shift between the content of one sentence and the next in a passage, the more likely it will be that a major update of the comprehension model needs to occur. In our case, it may be that time shifts of a day are not a large enough break in the narrative timeline to require a major update of the comprehension model, and thus may not require the use of the spatial domain to represent the movement through time. Time shifts of a month represent a much larger break in the narrative timeline, and thus participants are more likely to use spatial representations to ground their understanding of how the events are changing through time.

In conclusion, our data are consistent with other reports showing both that the space around our bodies can be used to represent abstract concepts (e.g., Dehaene et al., 1993; Fischer, 2006), and that the motor system plays an important role in the understanding of language about abstract and novel situations (e.g., Glenberg & Kaschak, 2002; Kaschak & Glenberg, 2000; Masson, Bub, & Warren, 2008). Whereas there is clearly much work to be done in order to flesh out our understanding of the ways that the body can ground cognition, we hope that the success of projects such as these demonstrates the utility of such approaches to cognitive science.

METHOD

Participants

One hundred and seventy-seven undergraduate students participated in these experiments. One hundred and fifty-five participants were (self-reported) right-handed, 21 were left-handed, and the handedness of one participant was undetermined. In the final analysis we excluded all but right-handed participants, leaving 79 participants in the Movement experiment and 76 in the No Movement experiment. [Note: the results of the data analysis were essentially identical with left-handers included in the sample]. They received course credit in exchange for participating.

Materials

Sixty-four three-sentence texts were created for these experiments. Forty of these texts were filler items that contained no time shifts, and 28 of these filler texts contained one sentence that was not sensible (e.g. Prank they blanketed to meet dogs.) for the purpose of eliciting “no” responses in the sensibility judgment task. The remaining 24 texts were the critical elements of the experiment (e.g., Jackie is taking a painting class; Tomorrow, she will learn about paintbrushes; It is important to learn paintbrush techniques). The second sentence in each text contained a time shift. Each text had a past and future version (e.g., “Yesterday/Tomorrow she learned/will learn about paint brushes”). Twelve of the critical items had time shifts of a day, and 12 had time shifts of a month. We created two counterbalanced lists of stimuli, such that an item appeared in one version (past or future) on one list, and appeared in the opposite version on the other list. The experiment was counterbalanced so that across participants, each text appeared equally often in its past and future version in both the away-response and toward-response conditions.

Critical sentences did not differ significantly in length between the month and day conditions F(1, 22) = 1.67, p = .210. However, sentences did differ significantly in length between the future and past conditions F(1, 21) = 176, p = .001. There was no interaction between Shift Direction (future/past) and Shift Magnitude (month/ day); F(1, 21) = 2.34, p = .141.

Procedure

Participants in each experiment were randomly assigned to make sensibility judgments by pressing the response button closer to their body, or farther from their body (i.e., toward and away responses, respectively). Texts were presented to the participants sentence-by-sentence. Participants were asked to determine whether each sentence was sensible or not. They were to press a particular response button on a computer keyboard if the sentence was sensible, and were to press another button if the sentence was not. Speed and accuracy were emphasized.

Apparatus

Participants responded via a standard QWERTY keyboard with modified keys. The “START” button was made by elevating the “s” key by adhering a small plastic block to the key. We labeled the block “START” using paper adhered to the block. The “P” button was elevated and labeled in the same way. This button was affixed to the “4” key of the number pad, about 12 in. away from the “START” key. The “X” button was made by elevating the “X” key next to the “START” button. The overall location of the “START” and “P” buttons was determined by the orientation of the keyboard (see Fig. 1). The keyboard was oriented such that its longest dimension stretched outward from the participant. In both cases, the locations of the “START” and “P” buttons were changed by flipping the keyboard around 180 (see Fig. 1).

For the Movement experiment, participants were told to hold down the “START” button while reading the sentence. If the sentence was sensible they were release the start button, and move their hand to press the “P” button. Participants pressed all buttons with the index finger of their right hand. If the sentence was not sensible, they were to press the “X” button. For the No Movement experiment, participants used both hands to respond. They kept their left index finger over the “START” button, and their right index finger over the “P” button. If the sentence was sensible, they released the “START” button and pressed the “P” button with their right hand. A response indicating a non-sensible statement required moving the left hand to the “X” button.

Design and analysis

In both the Movement and No Movement experiments, the primary dependent variable was the time between the participants’ press down of the “START” button to when they lifted off the “START” button to initiate the sensibility judgment response. Each experiment had a 2 (Shift Magnitude: day vs. month) 2 (Response location: towards vs. away) 2 (Shift Direction: future vs. past) design, with Response location manipulated between participants. We also conducted a cross-experiment analysis adding Experiment (Movement vs. No Movement) into the design. We also conducted an analysis across items. In the analysis by items, Shift Magnitude was a between-items factor, and all other factors were within-items.

The data were screened as follows. First, we eliminated any trials on which the participant pressed the incorrect response key (this occurred on approximately 4% of the trials). Second, we trimmed the response times for outliers. To do this, we first controlled for variability in response times produced by differences in sentence length by using the regression procedure described by Ferreira and Clifton (1986). The data from each participant was entered into a regression analysis with reading time as the dependent measure and sentence length (in characters) as the predictor. The residuals from these regressions were used in subsequent analyses. To screen for outliers, we first eliminated any residual response times that were less than 1500 ms and greater than 1500 ms (two standard deviations above and below the mean residual times). This eliminated 4.7% of the data. We then eliminated any remaining response times that were more than two standard deviations from each participants’ mean response time in each cell of the design (This eliminated .1% of the data). Additionally, one sentence had very high error rates (14%) and was excluded from all analyses. The remaining response times were entered into a mixed-factor ANOVA, as described above.

In addition to the dependent variable described above, we also recorded the time between participants’ release of the ‘START” button and press of the “P” key in both experiments. Consistent with previous explorations of motor compatibility effects in language comprehension (e.g., Glenberg & Kaschak, 2002), no effects were found in this measure (all F’s < 1.9). Thus, we do not discuss this variable further in the paper.