Skip to main content
PMC home page
Proceedings of the Royal Society B: Biological Sciences logoLink to Proceedings of the Royal Society B: Biological Sciences
. 2015 Apr 22;282(1805):20150381. doi: 10.1098/rspb.2015.0381

Binding space and time through action

N Binetti 1,3,✉, N Hagura 1, C Fadipe 1, A Tomassini 2, V Walsh 1, S Bestmann 2
PMCID: PMC4389624  PMID: 25808892

Abstract

Space and time are intimately coupled dimensions in the human brain. Several lines of evidence suggest that space and time are processed by a shared analogue magnitude system. It has been proposed that actions are instrumental in establishing this shared magnitude system. Here we provide evidence in support of this hypothesis, by showing that the interaction between space and time is enhanced when magnitude information is acquired through action. Participants observed increases or decreases in the height of a visual bar (spatial magnitude) while judging whether a simultaneously presented sequence of acoustic tones had accelerated or decelerated (temporal magnitude). In one condition (Action), participants directly controlled the changes in bar height with a hand grip device, whereas in the other (No Action), changes in bar height were externally controlled but matched the spatial/temporal profile of the Action condition. The sign of changes in bar height biased the perceived rate of the tone sequences, where increases in bar height produced apparent increases in tone rate. This effect was amplified when the visual bar was actively controlled in the Action condition, and the strength of the interaction was scaled by the magnitude of the action. Subsequent experiments ruled out that this was simply explained by attentional factors, and additionally showed that a monotonic mapping is also required between grip force and bar height in order to bias the perception of the tones. These data provide support for an instrumental role of action in interfacing spatial and temporal quantities in the brain.

Keywords: ATOM, magnitudes, temporal estimation, crossmodal correspondence

1. Introduction

Space and time are tightly interwoven dimensions in the brain, as evidenced by psychophysical [1–4], neuropsychological [5–8] and neuroimaging accounts [9]. Spatial and temporal information are known to interact in a variety of different contexts [10–14]. For example, size information biases the perception of the velocity of moving stimuli [15] as well as the duration of stationary stimuli [16]. The spatial separation between sequentially presented stimuli is known to affect judgements of the temporal interval separating them (Kappa effect). Conversely, the temporal interval between sequential stimuli affects the perception of their spatial separation (Tau effect) [12,17,18].

These interactions between spatial and temporal information have led to the hypothesis that space and time are represented in a shared magnitude format (ATOM—A Theory Of Magnitude; [19]; see also [15]). The ATOM model suggests that the brain is equipped with a shared analogue magnitude system used to process quantities of space, time and number, based on a common neural metric [19–21]. A shared system for quantities of space and time explains monotonic magnitude compatibility effects where more of one quantity ‘A’ determines the perception of more of another quantity ‘B’ (e.g. a stimulus of a larger size appears to last longer; [16]; see also [10]).

The ATOM model proposes that we learn the concepts of ‘how far’, ‘how long’ and ‘how many’ by acting upon our environment. An indication of this is provided by the behavioural interactions between action and time [3,22–27], actions and numbers [28,29], and actions and space [30–32]. A pivotal role of action in establishing representations of time, space and number is also supported by the fact that the processing of such quantities overlaps in parietal brain regions concerned with action control [11,33–35]. The ATOM model also hypothesizes that actions are instrumental in establishing a common system for representing space, time and number in a shared magnitude format. Through actions we can learn associations that occur across different magnitude domains, such, for example, that larger objects tend to be generally heavier [36] or that the time it takes us to cover a certain distance by foot will be proportional to the number of steps we take. This seems also intuitive given how in the context of action, space and time are rarely segregated [29]: actions are constrained in time and space and are characterized by precisely coordinated spatio-temporal neuromuscular events. Moreover, for actions such as pointing, reaching, walking or catching, magnitudes of space and time frequently covary in that actions that cover larger distances often take more time to unfold. However, aside these hints, there is still no experimental evidence in support of an instrumental role of action in binding spatial and temporal information in a shared representational format.

Here we sought to directly assess whether actions can enhance the interaction between space and time. If actions are indeed instrumental in interfacing spatial and temporal magnitudes in a shared magnitude format, then the interaction between these magnitudes should be modulated by actions.

2. Experiment 1: action enhances the interaction between space and time

(a). Methods and stimuli

In Experiment 1, participants (N = 12, eight females, 24.4 ± 3.5 years) evaluated linearly accelerating/decelerating sequences of brief acoustic tones (temporal magnitude), while viewing linear increases or decreases in height of a vertical red bar (spatial magnitude) (figure 1a,d). We tested how actively controlling changes in the spatial magnitude (Action condition) affects the processing of the temporal magnitude, when compared to a control condition in which equivalent changes in spatial magnitude were externally controlled (No-Action condition) (figure 1d). On each trial, an empty rectangular frame was presented at the centre of the display, covering approximately 10° of visual angle at a 57 cm viewing distance. A text presented above the frame indicated whether on the current trial the participant had to act upon the force gripper with their dominant hand (‘Action’ or ‘No Action’ trial). In the Action trials, pressure exerted on the force gripper controlled the filling or emptying of the frame with a red bar. In order to start each trial, participants had to preliminarily match the bar height to a set of stationary green trackers positioned on the left and right sides of the rectangular frame. In the Increasing force trials, the green trackers were near the bottom of the frame, while in the Decreasing force trials the green trackers were near the top of the frame. Two events ensued as soon as this threshold was reached: (i) the green markers started moving at a fixed rate upwards or downwards along the frame, requiring the participant to either progressively apply more (Increasing force trials = increase bar height) or less force (Decreasing force trials = decrease bar height) in order to match their motion and (ii) an auditory tone sequence was presented (figure 1b). In the No Action condition, participants passively viewed increases/decreases in the visual bar's height without any active force production (i.e. the changes in bar height mirrored the upward or downward motion of the tracker). We tested seven different tones sequences (8 tones, 50 ms each). The inter-onset intervals that separated the tones could linearly increase or decrease, determining sequences with seven possible degrees of acceleration/deceleration (figure 1c; see [37]). On each trial, one type of sequence was randomly selected and presented to the participant, at the end of which participants indicated whether the sequence appeared to accelerate or decelerate (perceptual task) on a mouse button with their non-dominant hand. We collected 20 observations per trial type, for a total of 560 trials per participant (20 trials × 2 Task conditions × 2 Spatial Magnitude conditions × 7 tone sequence Degrees of acceleration). Trial order was randomized.

Figure 1.

Figure 1.

Open in a new tab

(a) Dual task set-up: participants perform gripping actions with increasing/decreasing force (all experiments), or with sustained force (Experiment 5) while simultaneously evaluating whether a tone sequence accelerated or decelerated. A red bar provided a visual indicator of grip force. The upwards or downwards motion of two green triangular trackers provided the required rate of force increase or decrease (or indication of force error in Experiments 4 and 5). Participants had to adjust force throughout the trial to match the bar height to the tracker's motion (Increase/Decrease trials). (b) Both the action and the tone sequence took place in 3900 ms, after which participants responded whether the tone sequence appeared to accelerate or decelerate in a 2AFC scenario. (c) Inter-onset interval of each tone sequence (8 tones, 7 intervals). Δt ms indicates the difference in ms between the first and seventh interval. (d) Experiment 1. In the Action trials, grip force was directly mapped to the changes in bar height (Spatial Magnitude), i.e. increases (or decreases) in force were translated online into increases (or decreases) in bar height. In the No Action trials, the changes in bar height were externally controlled and perfectly matched the motion of the green trackers. (e) Experiment 1—PSI (points of subjective isochrony) in Δt ms for Increase/Decrease trials of the Action and No Action condition (error bars depict standard error). (Online version in colour.)

(b). Analysis

We fit the proportion of ‘tones accelerated’ responses with a cumulative logistic function [38,39] to obtain a Point of Subjective Isochrony (PSI): degree of acceleration in tone sequence, in Δt ms, required to produce a 50% proportion of ‘tones accelerated’ responses (psychometric fits provided in electronic supplementary material). This value represented how much the perception of the tone sequence was affected by the change in height of the visual bar. PSIs were compared within a 2 × 2 repeated measures ANOVA with factors Task (Action/No Action) and Spatial Magnitude (Increase/Decrease).

(c). Results

The analysis revealed no significant main effect of Task (F1,11 = 2.13, p = 0.173, Inline graphic Inline graphic) and a significant Task × Spatial Magnitude interaction (F1,11 = 16.58, p < 0.01, Inline graphic). We explored the significant interaction with Bonferroni-corrected t-tests, which revealed a significant difference between increases and decreases in Spatial magnitude in the Action condition (t11 = 4.19, p < 0.01, d = 1.21), and a marginally significant difference in the No Action condition (t11 = 2.61, p = 0.047, d = 0.75; figure 1e). The significant Task × Spatial Magnitude interaction resulted from a disparity in the difference in PSI between the Increasing and Decreasing trials of the Action and No Action conditions. The difference in PSI between Increasing and Decreasing trials was far more pronounced in the Action condition with respect to the No Action condition (302% larger PSI difference). While space–time interactions have been extensively reported independently of action [20,40], the strength of this interaction was far more pronounced when participants actively produced the changes in the spatial magnitude.

3. Experiment 2: differences in attentional demands do not account for the effect of action on the interaction of space and time

(a). Methods and stimuli

Experiment 1 showed that the magnitude compatibility between the spatial and temporal magnitudes was enhanced when changes in bar height were actively controlled (Action condition). However, differences in attentional load between the Action and No Action conditions may have contributed to this result, as the interaction between different sensory events can be modulated by attention [41–44]. The smaller magnitude compatibility effect observed in the No Action condition could have resulted from reduced attentional commitment to the changes in bar height, which in turn might affect its interaction with the auditory stimuli. In order to constrain attention to changes in bar height throughout the trial, in Experiment 2 participants performed an attentionally demanding No Action-jitter task condition, along with an Action condition (identical to that in Experiment 1; N = 12, eight females, 23.7 ± 3.9 years). Within the No Action-jitter task condition, the changes in bar height mirrored the motion of the tracker, but did so with the addition of a random jitter (figure 2a). The jitter was normally distributed with a mean either centred on the tracker's position (providing a baseline), or positioned slightly above or below the tracker (mean = 0, +3 or −3 pixels with respect to the tracker, s.d. = 15 pixels). In the No Action-jitter task trials, additionally to paying attention to the tone sequence, participants also had to report at the end of the trial with a mouse button press whether the bar had been on average more time above or below the trackers.

Figure 2.

Figure 2.

Open in a new tab

(a) Experiment 2. Action trials were identical to those in Experiment 1. In the No Action trials a random jitter was added to the externally controlled increases or decreases in bar height. Simultaneously to monitoring the tone sequence, participants had to indicate if the jittering bar had been on average more time above or below the trackers throughout the trial. (b) Experiment 2—Performance in the No Action-jitter detection task. Proportion of responses ‘bar was more time above the trackers' as a function of trials where the mean bar position was below, centred or above the tracker position. (c) Experiment 2—PSI (points of subjective isochrony) for Increase/Decrease trials of the Action and No Action-jitter task condition. (d) Experiment 3. Participants performed trials involving Large or Small changes in grip force and bar height (Δ G/B). In the Large Δ G/B trials, participants had to match the bar height to the trackers, which moved upwards or downwards across the whole extent of the rectangular frame. In the Small Δ G/B trials, the tracker's motion was confined to a smaller region around the centre of the rectangular frame. (e) Experiment 3—PSI (points of subjective isochrony) for Increase/Decrease trials in the Large and Small Action conditions. (Online version in colour.)

(b). Results

We measured participant's performance in the jitter detection trials to determine whether the task was difficult enough to engage participants' attention throughout the whole extent of the trial. Participants classified the bar as ‘above the tracker’ in 45% of trials when mean jitter centred below the tracker, in 59% of trials when centred on the tracker position (baseline), and in 69% of trials when positioned above the tracker (figure 2b). Performance was significantly different to the baseline, both when the mean jitter was positioned above and below the tracker (bar below tracker versus baseline: t11 = −3.67, p = 0.004, d = −1.06; bar above tracker versus baseline: t11 = 2.42, p = 0.03, d = 0.7). Therefore, it is likely that the jitter task strongly engaged participant's attention.

We ran a 2 × 2 repeated measures ANOVA on PSIs with factors Task (Action/No Action-dual task) and Spatial Magnitude (Increase/Decrease). The analysis revealed no significant effect of Task (F1,11 = 0.73, p = 0.41, Inline graphic), a significant effect of Spatial Magnitude (F1,11 = 14.62, p < 0.01, Inline graphic and a significant Task × Spatial Magnitude interaction (F1,11 = 6.62, p < 0.05, Inline graphic). Bonferroni-corrected t-tests revealed that, although a significant difference in PSI between Increasing and Decreasing trials was observed in the No Action condition (t11 = 2.95, p < 0.05, d = 0.85), the degree of the effect was stronger in the Action condition (t11 = 3.85, p < 0.01, d = 1.11), which resulted in the significant Task × Spatial Magnitude interaction (figure 2c). The difference in PSI was larger by a factor of 200% in the Action condition with respect to the No Action condition. However, this result does not necessarily entail that the level of attention required in the No Action-task was completely matched to the attention required to minimize tracking error in the Action condition. In order to more directly assess the impact of attentional load on the biased perception of the tone sequence, we compared performance in the Experiment 1 No Action condition to that in the Experiment 2 No Action-jitter condition. We ran a 2 × 2 Mixed Factorial ANOVA with Spatial Magnitude (Increase/Decrease) as within factor and Experiment (Experiment 1 No Action/Experiment 2 No Action-jitter) as categorical predictor. The analysis revealed a significant effect of Spatial Magnitude (F1,22 = 14.69, p < 0.01, Inline graphic), no effect of Experiment (F1,22 = 1.53, p = 0.23, Inline graphic) and no significant Spatial Magnitude × Experiment interaction (F1,22 = 1.77, p = 0.197, Inline graphic). The lack of an Experiment main effect and of a Spatial Magnitude × Experiment interaction shows that differences in attentional engagement (which is necessarily higher in the No Action-jitter condition) have no significant impact on the biased perception of the tone sequence.

Also, given that participants produced categorically similar responses for the tone sequence (‘accelerating/decelerating’) and for the jittered bar (‘above/below’), we tested whether PSIs in the No Action-jitter condition might reflect a semantic compatibility confound. We separately fit ‘semantically compatible’ (e.g. ‘tones increasing’/‘jitter above’) and ‘semantically incompatible’ (e.g. ‘tones increasing’/‘jitter below’) trials based on the compatibility of the responses to the tone sequence and jitter task. A comparison of the resulting PSIs revealed no significant difference as a function of Semantic Compatibility (t1,11 = 1.25, p = 0.24, d = 0.36), therefore ruling out that shifts in PSI reflected an interaction of responses to the tone sequence and to the jittered bar position.

4. Experiment 3: action magnitude scales the interaction between space and time

(a). Methods and stimuli

Experiments 1 and 2 showed that active control of changes in bar height (opposed to externally controlled changes in bar height) resulted in a stronger space–time association, thus showing that active control was driving the biased perception of the tone sequence. In Experiment 3, we wanted to explore whether the strength of the space–time interaction was scalable as a function of the amount of change in grip force and bar height [3,45]. This was important in order to establish whether active control simply determined a ‘stepwise’ increase in the space–time interaction, or if the strength of this interaction was modulated by the amount of change in grip force and/or amount of change in bar height controlled by grip force.

Participants (N = 12, five females, 23.9 ± 5.1 years; new sample) carried out a similar task to the Action condition of Experiment 1, where we compared actions that entailed large changes in grip force and bar height (‘Large Δ G/B’) to actions that entailed small changes in grip force and bar height (‘Small Δ G/B’). These different actions were determined by the extent of space covered by the green trackers throughout the trial (figure 2d). In order to eliminate any confound of differences in maximum force output, we matched both conditions for average force. In both ‘Large Change’ and ‘Small Change’ trials, the motion of the trackers was centred on the mid-section of the bar, thus equating all trials for average force.

(b). Results and discussion

A 2 × 2 repeated measures ANOVA on PSIs revealed a non-significant effect of Δ G/B (F1,11 = 3.56, p = 0.09, Inline graphic), a significant effect of Spatial Magnitude (F1,11 = 24.56, p < 0.001, Inline graphic) and a significant Δ G/B × Spatial Magnitude interaction (F1,11 = 6.96, p < 0.05, Inline graphic). Bonferroni corrected t-tests revealed a significant difference in PSI between Increasing and Decreasing trials in the Large Δ G/B condition (t11 = 4.69, p < 0.01, d = 1.35), and a marginally significant difference in PSI between Increasing and Decreasing trials in the Small Δ G/B condition (t11 = 2.59, p = 0.05, d = 0.74; figure 2e). The difference in PSI between Increasing and Decreasing trials was larger by a factor of 267% in the Large Δ G/B condition with respect to the Small Δ G/B condition (Large Δ G/B PSI mean difference = 91 ms versus Small Δ G/B PSI mean difference = 34 ms), thus explaining the significant Δ G/B × Spatial Magnitude interaction.

5. Experiment 4: opposite actions that produce no changes in spatial magnitude do not bias the perception of the temporal magnitude

(a). Methods and stimuli

Experiments 1–3 showed that actively controlled changes in bar height biased the perception of the tone sequence. But what specifically biased the perception of the tones: the changes in grip force, the changes in bar height or both? In Experiment 4, we isolated the contribution of changes in grip force from the changes in bar height. We did this by having participants (N = 12; eight females, 26.6 ± 4 years; new sample) carry out gripping actions with increasing or decreasing force that were translated into a visual representation of force error (i.e. error in rate of change of grip force; figure 3a). On each trial, a text instructed the participant what type of action to produce (Force Magnitude: Increase or Decrease grip force). As in Experiment 1, participants had to slowly apply force until they reached a predetermined threshold level. Once the threshold was reached, the participant would have to increase or decrease grip force at an appropriate rate (identical to that of Experiment 1). Stationary green trackers positioned at the centre of the bar indicated the desired rate, while the bar height depicted the error between required and produced rate. The bar would be beneath the markers when the rate was too slow (produced < required), while it would be above the markers when it was too fast (produced > required). The greater the displacement of the bar form the trackers, the greater the error. Participants thus learned to keep the bar continuously positioned in proximity to the green markers throughout the whole extent of the trial.

Figure 3.

Figure 3.

Open in a new tab

(a) Experiment 4. Grip force was mapped onto a visual depiction of force error. Participants had to increase or decrease force at an appropriate rate. If the bar was below the trackers, it meant the rate was too slow, while above the trackers it meant the rate was too fast. Ideally participants had to increase or decrease force at a rate that maintained bar height constant. (b) Experiment 4—PSI (points of subjective isochrony) for Increase/Decrease force trials. (c) Experiment 5. Sustained grip force was mapped onto a visual depiction of force error (Sustained Mapping). A constant grip force determined the velocity with which bar height would increase or decrease. Participants had to maintain a target force level to produce increases/decreases in bar height that matched the tracker's motion. If the bar was below the trackers, it meant the force was too low, and vice versa. The Sustained Mapping condition was compared to a Direct Mapping condition (identical to Experiment 1). (d) Experiment 5—PSI (points of subjective isochrony) for Increase/Decrease bar height trials, in the Sustained and Direct Mapping conditions. (e) Experiment 6. Grip force was inversely mapped to bar height where increases (or decreases) in force translated to decreases (or increases) in bar height. The Inverse mapping condition was compared to a Direct mapping condition. (f) PSI (points of subjective isochrony) for Increase/Decrease bar height trials in the Inverse and Direct mapping conditions. (Online version in colour.)

(b). Results

We carried out a paired sample t-test comparing PSIs between the trials requiring increasing versus decreasing grip force (Force Magnitude: increase versus decrease). The perception of the tone sequences did not significantly differ between increasing and decreasing gripping actions that entailed no changes in bar height (t11 = 0.78, p = 0.45, d = 0.22; figure 3b).

6. Experiment 5: identical actions that produce opposite changes in spatial magnitude bias the perception of the temporal magnitude

(a). Methods and stimuli

In Experiment 5, we isolated the contribution of changes in bar height by eliminating changes in grip force. In this case, a new sample of participants (N = 11, eight females, 26.6 ± 4 years) performed gripping actions with constant force, which translated into either a filling or emptying of the bar (constant force/varying consequence mapping). In this case, grip force was mapped to the velocity with which the bar would either fill or empty: a greater force produced faster filling or emptying of the bar. The goal was that of filling or emptying the bar at the correct rate, matching the motion of the green trackers (figure 3c). At the beginning of each trial, participants had to reach a fixed force threshold to trigger the motion of the green trackers and the tone sequence onset (equivalent to the mean force of all previous experiments). Once the threshold was met, participants had to sustain this force level throughout the trial. A text at the beginning of the trial informed the participant whether their action would translate into a filling or emptying of the bar (Spatial Magnitude, increase or decrease). Participants also carried out in a separate block a direct grip force/bar height mapping task (as Experiment 1) to provide a comparison measure for the effect (Mapping, constant or direct).

(b). Results and discussion

A 2 × 2 repeated measures ANOVA, with factors Mapping (Sustained/Direct), and Spatial Magnitude (Increase/Decrease) showed no main effect of Mapping (F1,11 = 0.15, p = 0.71, Inline graphic), a significant effect of Spatial magnitude (F1,11 = 23.03, p < 0.01, Inline graphic) and no significant Mapping × Spatial Magnitude interaction (F1,11 = 0.04, p = 0.85, Inline graphic). This indicated that sustained force actions determined significant differences in the perception of the tone sequences depending on whether they entailed an increase or decrease in visual bar height (figure 3d). Participants were more likely to report the tone sequence as accelerating when the sustained grip entailed an increase in bar height. This effect was equivalent in sign and magnitude to the direct Mapping condition. This indicated that changes in grip force are not necessary to bias the perception of the tone sequence as long as the action entails changes in bar height.

7. Experiment 6: conflicting changes in force and spatial magnitude do not bias the perception of the temporal magnitude

(a). Methods and stimuli

In Experiments 4 and 5, we found that, when changes in grip force and bar height were assessed individually, the key element biasing the perception of the tone sequence were the changes in bar height. In Experiment 6, we tested whether grip force and bar height interact when both are subject to changes in magnitude, but these changes are inversely mapped (i.e. increase in force = decrease in spatial magnitude; figure 3e). If changes in action magnitude do not interact with changes in spatial magnitude, then the biased perception of the tone sequence should be exclusively explained by the sign of changes in the spatial magnitude. If, on the other hand, changes in action magnitude interact with changes in spatial magnitude, then we should expect a weakening or cancelling of their combined effect on the tone sequence.

A new sample of participants (N = 12, six females, 27.9 ± 7.3 years) performed gripping actions with increasing or decreasing force: in one block grip force was inversely mapped to bar height (‘Inverse’), whereas in another block, it was directly mapped to bar height (‘Direct’; as Experiment 1) to provide a comparison measure for the effect.

(b). Results and discussion

We ran a 2 × 2 repeated measures ANOVA, with Mapping (Inverse/Direct) and Spatial Magnitude (Increase/Decrease) as factors. The analysis revealed no main effect of Mapping (F1,11 = 0.98, p = 0.34, Inline graphic), a main effect of Spatial Magnitude (F1,11 = 13.5, p < 0.01, Inline graphic) and a borderline non-significant Mapping × Spatial Magnitude interaction (F1,11 = 3.68, p = 0.08, Inline graphic). Despite the non-significant interaction, we ran a post-hoc analysis since we were specifically interested in assessing if there were differences between Increases and Decreases in Spatial Magnitude across the two Mapping conditions. Bonferroni corrected t-tests revealed that increases and decreases in Spatial Magnitude significantly differed in the Direct Mapping condition (t11 = 3.21, p < 0.01, d = 0.92, figure 3f), while they did not differ in the Inverse Mapping condition (t11 = 2.27, p = 0.08, d = 0.66). When both grip force and bar height changed, but did so with opposite signs, the magnitude compatibility effect was cancelled. Therefore, the way grip force is mapped to the changes in the spatial quantity is important in mediating the space–time interaction.

Alternatively to a space–time magnitude compatibility being enhanced by action, we could also hypothesize that the action directly biases the perception of the bar height, and this in turn affects the perception of the tone sequence. We tested this possibility by comparing tracking errors in the Direct versus Inverse Mapping conditions of Experiment 6. If actions distort the perception of the changes in bar height, then opposite actions (increase versus decrease in force) should determine differences in tracking overshoot/undershoot when controlling the same changes in bar height (increase OR decrease in height). In other words, filling the bar by increasing force (Direct mapping) should result in undershooting the trackers (which are not controlled through action) opposed to when the bar is filled by decreasing force (Inverse mapping). We calculated for each participant the average tracking overshoot/undershoot for the Increase and Decrease Spatial magnitude trials in both the Direct and Inverse mapping conditions. We entered average overshoot/undershoot errors into a repeated measures ANOVA with factors Mapping (Direct/Inverse) and Spatial Magnitude (Increase/Decrease). The analysis revealed no main effect of Mapping (F1,11 = 2.34, p = 0.15, Inline graphic), no main effect of Spatial Magnitude (F1,11 = 1.44, p = 0.25, Inline graphic) and, crucially, no Mapping × Spatial Magnitude interaction (F1,11 = 1.88, p = 0.2, Inline graphic). The lack of a significant interaction shows that in the present set-up it is unlikely that the perception of changes in bar height were distorted by action.

8. Discussion

In this study, we provide the first experimental evidence that the association between visuo-spatial and auditory-temporal quantities is enhanced in the context of action. This provides behavioural evidence in support of an instrumental role of action in binding spatial and temporal sensory magnitudes in the brain.

The interaction we observed probably represents an example of cross-modal correspondence, based on the coupling of stimulus features across the visual and auditory domain [46,47]. Cross-modal correspondences can be classified under different categories based on the mechanism mediating the interaction: semantically mediated, i.e. determined by overlaps in linguistic tags used to classify stimuli; structural, i.e. emerging from an intrinsic overlap in the processing of a set of stimulus features; and statistically mediated, i.e. through stable correspondences in the environment [47].

Given the nature of our task, the visual–auditory interaction could be supported by a compatibility between linguistic labels associated with the changes in grip force (‘increasing/decreasing’), bar height (‘increasing/decreasing’) and tone rate (‘accelerating/decelerating’). However, a series of elements in this dataset suggest a significant involvement of sensory magnitude factors in promoting the interaction. In Experiment 3, we manipulated visual information along two dimensions: the direction of change in bar height (increasing/decreasing) and the amount of change in bar height (high/low). We found that the strength of this cross-modal correspondence was scaled by the amount of visual bar increase/decrease. If the effects were exclusively explained as an overlap in the linguistic terms used to describe the stimuli [47], we could expect the directional information to be the critical dimension that interacts with the binary ‘accelerating/decelerating’ responses to the tone sequences, whereas the amount of change could be redundant information in this respect. This modulation could on the other hand suggest an interaction of sensory quantities, where larger increases in one dimension (visual bar) determine the misperception of larger increases in a second dimension (auditory tones).

In Experiment 4, we observed that increasing/decreasing grip force alone does not interfere with the accelerating/decelerating judgements of the auditory tones. If we were exclusively dealing with the interaction of linguistic tags, we should probably expect equivalent interactions between grip force and auditory stimuli to those observed between visual and auditory stimuli, based on equivalent semantic overlaps. In addition, despite action having no direct significant effect on the auditory information (Experiment 4), and no direct significant effect on the visual information (see Experiment 6 discussion, §7(b)), action clearly modulated the strength of the visual–auditory interaction (Experiments 1 and 2). Collectively, this suggests a more complex relationship linking action, visual and auditory stimuli than a straightforward overlap in signs of linguistic tags.

The visual–auditory interaction could therefore involve an intrinsic overlap in the processing of visual and auditory stimulus features (structural correspondence). The features in question could be represented by overlapping abstract ‘prothetic’ (magnitude-related, see [47]) spatial and temporal representations, or overlaps occurring at earlier processing stages, prior to the extraction of abstract magnitude information (e.g. [48]). Action is known to promote cross-modal integration [49–52], and here, action might provide a bridge that favours the interaction of visual and auditory signals as well as the quantitative information evaluated within these signals. While ATOM proposes that different magnitude processes overlap in parietal areas, the mechanism that promotes the integration of these sensory signals must not necessarily be hardwired into the brain's architecture. The modulatory effect of action could also emerge as a function of a priori joint distributions of the visual and auditory signals (‘coupling prior’; see [53]), as predicted by Bayesian integration theory. The fact that the combination of the visual–auditory signals is enhanced by actions could depend on an expectation that these signals covary when action is involved.

The modulatory effect of action on the strength of the space–time interaction could also be potentially explained in terms of sense of agency: i.e. the fact of feeling that one is controlling the changes in bar height might amplify its impact on the tone sequence. As a matter of fact, Experiment 5 showed that actions involving different sensorimotor associations (Direct and Constant mapping) resulted in equivalent effects on the tone sequence. Experiment 6, however, suggests that being in control of the changes in bar height alone cannot fully account for this magnitude compatibility effect. In the Inverse Mapping scenario, subjects control the bar by releasing grip force to increase bar height. In this condition, however, no influence was observed on the auditory sequences despite involving an active control over bar height.

In conclusion, we show that the interaction between visuo-spatial and auditory-temporal signals is enhanced in the context of action. We learn the statistical relationship between different sensory channels—as well as the relationship between different magnitudes carried by these channels—through action.

Supplementary Material

Psychometric fits Experiments 1-6
rspb20150381supp1.tif (244.8KB, tif)

Supplementary Material

Grip force plots
rspb20150381supp2.tif (835.2KB, tif)

Data accessibility

Raw behavioural and grip force data can be found here: http://dx.doi.org/10.5061/dryad.73fq4.

References

  • 1.Binetti N, Lecce F, Doricchi F. 2012. Time-dilation and time-contraction in an anisochronous and anisometric visual scenery. J. Vis. 12, 8 ( 10.1167/12.7.8) [DOI] [PubMed] [Google Scholar]
  • 2.Casasanto D, Boroditsky L. 2008. Time in the mind: using space to think about time. Cognition 106, 579–593. ( 10.1016/j.cognition.2007.03.004) [DOI] [PubMed] [Google Scholar]
  • 3.Morrone MC, Ross J, Burr D. 2005. Saccadic eye movements cause compression of time as well as space. Nat. Neurosci. 8, 950–954. ( 10.1038/nn1488) [DOI] [PubMed] [Google Scholar]
  • 4.Frassinetti F, Magnani B, Oliveri M. 2009. Prismatic lenses shift time perception. Psychol. Sci. 20, 949–954. ( 10.1111/j.1467-9280.2009.02390.x) [DOI] [PubMed] [Google Scholar]
  • 5.Calabria M, Jacquin-Courtois S, Miozzo A, Rossetti Y, Padovani A, Cotelli M, Miniussi C. 2011. Time perception in spatial neglect: a distorted representation? Neuropsychology 25, 193–200. ( 10.1037/a0021304) [DOI] [PubMed] [Google Scholar]
  • 6.Cappelletti M, Freeman ED, Cipolotti L. 2009. Dissociations and interactions between time, numerosity and space processing. Neuropsychologia 47, 2732–2748. ( 10.1016/j.neuropsychologia.2009.05.024) [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 7.Magnani B, Oliveri M, Mancuso G, Galante E, Frassinetti F. 2011. Time and spatial attention: effects of prism adaptation on temporal deficits in brain damaged patients. Neuropsychologia 49, 1016–1023. ( 10.1016/j.neuropsychologia.2010.12.014) [DOI] [PubMed] [Google Scholar]
  • 8.Oliveri M, Magnani B, Filipelli A, Avanzi S, Frassinetti F. 2013. Prismatic adaptation effects on spatial representation of time in neglect patients. Cortex 49, 120–130. ( 10.1016/j.cortex.2011.11.010) [DOI] [PubMed] [Google Scholar]
  • 9.Dormal V, Dormal G, Joassin F, Pesenti M. 2012. A common right fronto-parietal network for numerosity and duration processing: an fMRI study. Hum. Brain Mapp. 33, 1490–1501. ( 10.1002/hbm.21300) [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 10.Bonato M, Zorzi M, Umiltà C. 2012. When time is space: evidence for a mental time line. Neurosci. Biobehav. Rev. 36, 2257–2273. ( 10.1016/j.neubiorev.2012.08.007) [DOI] [PubMed] [Google Scholar]
  • 11.Morrone MC, Cicchini M, Burr D. 2010. Spatial maps for time and motion. Exp. Brain Res. 206, 121–128. ( 10.1007/s00221-010-2334-z) [DOI] [PubMed] [Google Scholar]
  • 12.Oliveri M, Koch G, Caltagirone C. 2009. Spatial–temporal interactions in the human brain. Exp. Brain Res. 195, 489–497. ( 10.1007/s00221-009-1834-1) [DOI] [PubMed] [Google Scholar]
  • 13.Kanai R, Paffen CL, Hogendoorn H, Verstraten FA. 2006. Time dilation in dynamic visual display. J. Vis. 6, 1421–1430. ( 10.1167/6.12.8) [DOI] [PubMed] [Google Scholar]
  • 14.Kaneko S, Murakami I. 2009. Perceived duration of visual motion increases with speed. J. Vis. 9, 14 ( 10.1167/9.7.14) [DOI] [PubMed] [Google Scholar]
  • 15.Stavy R, Tirosh D. 2000. How students (mis-)understand science and mathematics: intuitive rules. Columbia University, NY: Teachers College Press. [Google Scholar]
  • 16.Xuan B, Zhang D, He S, Chen X. 2007. Larger stimuli are judged to last longer. J. Vis. 7, 2 ( 10.1167/7.10.2) [DOI] [PubMed] [Google Scholar]
  • 17.Bill JC, Teft LW. 1969. Space–time relations: effects of time on perceived visual extent. J. Exp. Psychol. 81, 196–199. ( 10.1037/h0027425) [DOI] [PubMed] [Google Scholar]
  • 18.Sarrazin J-C, Giraudo M-D, Pailhous J, Bootsma RJ. 2004. Dynamics of balancing space and time in memory: tau and kappa effects revisited. J. Exp. Psychol. Hum. Percept. Perform. 30, 411–430. ( 10.1037/0096-1523.30.3.411) [DOI] [PubMed] [Google Scholar]
  • 19.Walsh V. 2003. A theory of magnitude: common cortical metrics of time, space and quantity. Trends Cogn. Sci. 7, 483–488. ( 10.1016/j.tics.2003.09.002) [DOI] [PubMed] [Google Scholar]
  • 20.Dormal V, Pesenti M. 2012. Processing magnitudes within the Parietal Cortex. Horizons Neurosci. Res. 8, 107–140. [Google Scholar]
  • 21.Hayashi MJ, Kanai R, Tanabe HC, Yoshida Y, Carlson S, Walsh V, Sadato N. 2013. Interaction of numerosity and time in prefrontal and parietal cortex. J. Neurosci. 33, 883–893. ( 10.1523/JNEUROSCI.6257-11.2013) [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 22.Gavazzi G, Bisio A, Pozzo T. 2013. Time perception of visual motion is tuned by the motor representation of human actions. Sci. Rep. 3, 1168 ( 10.1038/srep01168) [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 23.Hodinott-Hill I, Thilo KV, Cowey A, Walsh V. 2002. Auditory chronostasis: hanging on the telephone. Curr. Biol. 12, 1779–1781. ( 10.1016/S0960-9822(02)01219-8) [DOI] [PubMed] [Google Scholar]
  • 24.Haggard P, Clark S, Kalogeras J. 2002. Voluntary action and conscious awareness. Nat. Neurosci. 5, 382–385. ( 10.1038/nn827) [DOI] [PubMed] [Google Scholar]
  • 25.Hagura N, Kanai R, Orgs G, Haggard P. 2012. Ready steady slow: action preparation slows the subjective passage of time. Proc. R. Soc. B 279, 4399–4406. ( 10.1098/rspb.2012.1339) [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 26.Moore J, Haggard P. 2008. Awareness of action: inference and prediction. Conscious Cogn. 17, 136–144. ( 10.1016/j.concog.2006.12.004) [DOI] [PubMed] [Google Scholar]
  • 27.Wenke D, Haggard P. 2009. How voluntary actions modulate time perception. Exp. Brain Res. 196, 311–318. ( 10.1007/s00221-009-1848-8) [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 28.Andres M, Davare M, Pesenti M, Olivier E, Seron X. 2004. Number magnitude and grip aperture interaction. Neuroreport 15, 2773–2777. [PubMed] [Google Scholar]
  • 29.Lindemann O, Abolafia JM, Girardi G, Bekkering H. 2007. Getting a grip on numbers: numerical magnitude priming in object grasping. J. Exp. Psychol. Hum. Percept. Perform. 33, 1400–1409. ( 10.1037/0096-1523.33.6.1400) [DOI] [PubMed] [Google Scholar]
  • 30.Stefanucci JK, Geuss MN. 2009. Big people, little world: the body influences size perception. Perception 38, 1782–1795. ( 10.1068/p6437) [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 31.Witt JK, Proffitt DR, Epstein W. 2004. Perceiving distance: a role of effort and intent. Perception 33, 577–590. ( 10.1068/p5090) [DOI] [PubMed] [Google Scholar]
  • 32.Witt JK, Proffitt DR, Epstein W. 2005. Tool use affects perceived distance, but only when you intend to use it. J. Exp. Psychol. Hum. Percept. Perform. 31, 880–888. ( 10.1037/0096-1523.31.5.880) [DOI] [PubMed] [Google Scholar]
  • 33.Bueti D, Walsh V. 2009. The parietal cortex and the representation of time, space, number and other magnitudes. Phil. Trans. R. Soc. B 364, 1831–1840. ( 10.1098/rstb.2009.0028) [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 34.Vallesi A, Binns MA, Shallice T. 2008. An effect of spatial–temporal association of response codes: understanding the cognitive representations of time. Cognition 107, 501–527. ( 10.1016/j.cognition.2007.10.011) [DOI] [PubMed] [Google Scholar]
  • 35.Magnani B, Oliveri M, Mangano GR, Frassinetti F. 2010. The role of posterior parietal cortex in spatial representation of time: a TMS study. Behav. Neurol. 23, 213–215. ( 10.1155/2010/282950) [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 36.Brenner E, Smeets JB. 1996. Size illusion influences how we lift but not how we grasp an object. Exp. Brain Res. 111, 473–476. ( 10.1007/BF00228737) [DOI] [PubMed] [Google Scholar]
  • 37.Binetti N, Siegler IA, Bueti D, Doricchi F. 2013. Adaptive tuning of perceptual timing to whole body motion. Neuropsychologia 51, 197–210. ( 10.1016/j.neuropsychologia.2012.10.029) [DOI] [PubMed] [Google Scholar]
  • 38.Wichmann FA, Hill NJ. 2001. The psychometric function: I. Fitting, sampling, and goodness of fit. Attent. Percept. Psychophys. 63, 1293–1313. ( 10.3758/BF03194544) [DOI] [PubMed] [Google Scholar]
  • 39.Wichmann FA, Hill NJ. 2001. The psychometric function: II. Bootstrap-based confidence intervals and sampling. Attent. Percept. Psychophys. 63, 1314–1329. ( 10.3758/BF03194545) [DOI] [PubMed] [Google Scholar]
  • 40.Grondin S. 2010. Timing and time perception: a review of recent behavioral and neuroscience findings and theoretical directions. Attent. Percept. Psychophys. 72, 561–582. ( 10.3758/APP.72.3.561) [DOI] [PubMed] [Google Scholar]
  • 41.Corbetta M, Shulman GL. 2002. Control of goal-directed and stimulus-driven attention in the brain. Nat. Rev. Neurosci. 3, 215–229. ( 10.1038/nrn755) [DOI] [PubMed] [Google Scholar]
  • 42.Hopfinger J, Buonocore M, Mangun G. 2000. The neural mechanisms of top–down attentional control. Nat. Neurosci. 3, 284–291. ( 10.1038/72999) [DOI] [PubMed] [Google Scholar]
  • 43.Desimone R, Duncan J. 1995. Neural mechanisms of selective visual attention. Annu. Rev. Neurosci. 18, 193–222. ( 10.1146/annurev.ne.18.030195.001205) [DOI] [PubMed] [Google Scholar]
  • 44.Lavie N. 2005. Distracted and confused?: selective attention under load. Trends Cogn. Sci. 9, 75–82. ( 10.1016/j.tics.2004.12.004) [DOI] [PubMed] [Google Scholar]
  • 45.Yarrow K, Haggard P, Heal R, Brown P, Rothwell JC. 2001. Illusory perceptions of space and time preserve cross-saccadic perceptual continuity. Nature 414, 302–305. ( 10.1038/35104551) [DOI] [PubMed] [Google Scholar]
  • 46.Evans KK, Treisman A. 2010. Natural cross-modal mappings between visual and auditory features. J. Vis. 10, 6 ( 10.1167/10.1.6) [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 47.Spence C. 2011. Crossmodal correspondences: a tutorial review. Attent. Percept. Psychophys. 73, 971–995. ( 10.3758/s13414-010-0073-7) [DOI] [PubMed] [Google Scholar]
  • 48.Kelly D. 1966. Frequency doubling in visual responses. JOSA 56, 1628–1632. ( 10.1364/JOSA.56.001628) [DOI] [Google Scholar]
  • 49.Abrams RA, Pratt J. 1993. Rapid aimed limb movements: differential effects of practice on component submovements. J. Motor Behav. 25, 288–298. ( 10.1080/00222895.1993.9941650) [DOI] [PubMed] [Google Scholar]
  • 50.Hu B, Knill DC. 2010. Kinesthetic information disambiguates visual motion signals. Curr. Biol. 20, R436–R437. ( 10.1016/j.cub.2010.03.053) [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 51.Stein BE. 1998. Neural mechanisms for synthesizing sensory information and producing adaptive behaviors. Exp. Brain Res. 123, 124–135. ( 10.1007/s002210050553) [DOI] [PubMed] [Google Scholar]
  • 52.Vercher J-L, Gauthier G. 1992. Oculo-manual coordination control: ocular and manual tracking of visual targets with delayed visual feedback of the hand motion. Exp. Brain Res. 90, 599–609. ( 10.1007/BF00230944) [DOI] [PubMed] [Google Scholar]
  • 53.Roach NW, Heron J, McGraw PV. 2006. Resolving multisensory conflict: a strategy for balancing the costs and benefits of audio-visual integration. Proc. R. Soc. B 273, 2159–2168. ( 10.1098/rspb.2006.3578) [DOI] [PMC free article] [PubMed] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

Psychometric fits Experiments 1-6
rspb20150381supp1.tif (244.8KB, tif)
Grip force plots
rspb20150381supp2.tif (835.2KB, tif)

Data Availability Statement

Raw behavioural and grip force data can be found here: http://dx.doi.org/10.5061/dryad.73fq4.

Articles from Proceedings of the Royal Society B: Biological Sciences are provided here courtesy of The Royal Society

RESOURCES