Time Order as Psychological Bias

Abstract

Incorrectly perceiving the chronology of events can fundamentally alter our understanding of the causal structure of the world, as historically exemplified by astronomers who, using the “eye and ear” method, systematically showed inter-individual errors in locating stars (Sanford, 1888). Here, we show that temporal order perception is a psychological bias that attention can modulate but not fully eradicate. According to Titchener’s law of prior-entry, attention prioritizes the perception of an event, and could thus help compensate for possible inter-individual differences by normalizing perception in time. In a longitudinal study, we tested the stability of participants’ temporal order perception across (and within) sensory modalities, together with their magnitude of prior-entry. In all measurements, we show the persistence of stable inter-individual variability. Crucially, the magnitude of prior-entry was insufficient to realign individuals on a unique time base: conscious time order is subjective, but systematically so, hence tractable on a per individual basis.

Introduction

The perception of temporal order is seldom veridical (Sanford, 1888) and can be modulated by attention (Spence, Shore, & Klein, 2001; Shore, Spence, & Klein, 2001). According to Titchener’s law (1908) of prior-entry, a stimulus being attended to will be systematically perceived before a stimulus being ignored. Prior-entry predicts that attention prioritizes the arrival time of sensory stimuli, hence may govern our subjective awareness of timing and be instrumental in adjusting neural latencies across senses (Spence & Squire, 2003). Attention, but also temporal integration (Colonius & Diederich, 2004; van Wassenhove, Grant, & Poeppel, 2007) and active compensation (Sugita & Suzuki, 2003), can all contribute to even out the timing of sensory information in the brain, enabling multisensory integration. Yet, if integration yields to the compression of information in time causing an a priori loss of temporal resolution, the serial order of multisensory events in the few tens of milliseconds does not fully escape our consciousness (Efron, 1973; van Eijk, Kohlrausch, Juola, & van de Par, 2008). High inter-individual variability in temporal order perception has been observed (Sanford, 1888; Stone, et al., 2001; Vroomen, Keetels, Gelder, & Bertelson, 2004; Boenke, Deliano, & Ohl, 2009; Freeman, et al., 2013; Kösem, Gramfort, & van Wassenhove, 2014) suggesting that attention may not be fully sufficient to compensate for possible individual biases in order perception but to date, no studies have tackled this question.

Here, we asked whether inter-individual variability in temporal order perception could be fully accounted for by participants’ attentional state, by using audiovisual temporal order judgments, which allowed manipulating participants’ attention towards one (e.g. auditory) or the other (e.g. visual) sensory modality. The literature on audiovisual temporal order has either not explicitly controlled for cross-modal attention (Zampini, Shore, & Spence, 2003; Vroomen, Keetels, Gelder, & Bertelson, 2004; Boenke, Deliano, & Ohl, 2009) or considered prior-entry as an index of temporal order perception (Spence, Shore, & Klein, 2001; Weiss, Hilkenmeier, & Scharlau, 2013). We combined both approaches and assessed audiovisual temporal order thresholds or Point of Subjective Simultaneity (PSS) with two experimentally independent measures: a first measure was derived from a split-attention condition between sensory modalities (PSS_AV) and the second measure (PSS_free), free of any attentional bias, was computed using two experimental conditions manipulating attention towards one or the other sensory modality. To test the stability of an individual’s temporal order perception over time, we used a longitudinal approach and systematically measured both PSSs over a four month period. The use of longitudinal approaches to establish the robustness of multisensory timing during the life-span was recently advocated (Noel, Niear, Burg, & Wallace, 2016) and, to the best of our knowledge, only one study has reported strong within-participant correlations of audiovisual simultaneity over a one week period (but with a change of viewing distance in between, see Stone, et al., 2001). The stability of PSSs and prior-entry could thus be tested, along with the relationship between an individual’s prior-entry effect (i.e. attention) and PSS (i.e. temporal order perception). The unisensory (auditory and visual) PSS and spatial biases were also assessed thanks to the spatialized nature of the task that was used.

Methods

Participants

24 right-handed naïve participants with normal or corrected-to-normal vision, and normal audition took part in the study (9 males; 25 +/- 4 years old). Each provided a written informed consent in accordance with the Declaration of Helsinki (2008) and the Ethics Committee of Human Research at NeuroSpin, Gif-sur-Yvette, France. 5 participants were excluded as their PSS were located outside the range of tested SOAs (following the criteria used in Spence, Shore & Klein, 2001). 19 participants were thus considered in the final analysis (7 males; 25 +/- 4 years old), well above the requirement of N=13 suggested by a sample size analysis (R, ICC.Sample.Size with alpha =.05; p = .8; p0 = .4; power = .9). The post-hoc power analysis revealed that the smallest ICC in our study (.46) had a power of .97 (with alpha = .05 and a null hypothesis of ICC = 0 for no intraclass reliability).

Stimuli

The experiment was designed using Psychtoolbox (v.3.0.11) with Matlab 2014a (The MathWorks, Inc., Natick, Massachusetts, USA). Visual stimuli were white dots (4 lux of luminance, 1.8° of diameter) presented for 35 ms (3 frames) on a black screen (85 Hz refresh rate, 1024 x 728 pixels, CRT) with 14° of visual angle on the horizontal meridian. Auditory stimuli were 2 kHz sinewave tones (incl. 5 ms fade-in and fade-out, 63 dB) presented for 35 ms with speakers placed on each side of the monitor screen at 19° of eccentricity from the center of the screen on the horizontal meridian. Stimuli were separated by variable Stimulus Onset Asynchronies (SOAs) with thirteen possible values: 0, +/-34, +/-54, +/-68, +/-98, +/-145 and +/-237 milliseconds (ms). A negative (positive) delay indicates that the left (right) stimulus was leading. Synchronization and the stability of stimulus timing was verified with an oscilloscope using a photodiode and a microphone: each SOA was measured 18 times for each pair of stimuli (visual, auditory and audiovisual). A mean standard deviation of 5 ms was observed across all conditions (s.d. of 3, 4, 7, 5, 6, 7, 5 ms with increasing SOA values described above, respectively). To insure that this variability was not detrimental, we simulated 1000 repetitions of PSS measures with variable SOAs picked from a Gaussian distribution centered on the mean SOAs with a standard deviation of 7 ms (i.e. the highest possible s.d. observed in the measured SOAs). The mean difference between the surrogate distribution and the veridical PSS computed with no-delay SOAs was 0.2 ms (s.d. = 2.4; CI_95% = [-3.7, 4.1]). All PSS estimations reported in the Results section are thus robust, and estimation errors are largely below the observed and reported individual differences in the study.

Procedure

The longitudinal study was composed of four sessions taking place 7 +/-2 days, 1 month +/-3 days and 4 months +/-8 days after the first session. An individual’s perception of temporal order was assessed using a spatialized Temporal Order Judgement (TOJ) task (Shore, Spence, & Klein, 2001). Participants sat in a darkened room and maintained their head on a chin-rest located 65 cm away from a computer screen. On any given trial, they were presented with a pair of stimuli composed of two sounds (AA), two flashes (VV) or a sound and a flash (AV) (Fig. 1a). The pair of stimuli was spatially segregated - one on the left side, one on the right side - independently of their sensory modality. Participants reported the side of the stimulus that they perceived first in a 2-AFC (‘left’ or ‘right’) and asked to favor accuracy over speed. Before the experiment, participants were trained with 14 trials at maximal SOAs.

Fig. 1. Experimental design and prior-entry effect.

A spatial temporal order judgement (TOJ) task was used in which participants were presented with (a) a pair of spatialized stimuli which could be auditory, visual or audiovisual. Participants reported whether the first stimulus occurred on their left or on their right side, irrespective of its sensory modality. Three main experimental conditions were tested (b) to control for attentional biases in participants’ Point of Subjective Simultaneity (PSS): in visual attention blocks (red), participants attended to visual events while trials could be audiovisual (AV) or visual only (VV). In the split-attention blocks (orange), participants paid equal attention to the auditory and to the visual events and trials could be audiovisual (AV) or unisensory (AA or VV). In the auditory attention blocks (blue), participants attended to auditory events, and trials could be audiovisual (AV) or auditory only (AA). In split-attention conditions, responses were sorted as a function of which sensory modality was perceived first, yielding (c) an estimate of the probability to report seeing the flash first (% ‘flash first’) as a function of Stimulus Onset Asynchrony (SOA). Results for one representative participant are illustrated in the top panel. PSSs were estimated on a per individual basis, and averaged across participants as a function of attentional condition. Grand-average PSSs per attentional condition are reported in the bottom panel. Significant differences across attentional conditions were found, replicating the prior-entry effect (gray shaded area): specifically, attending the visual modality (red) shifted the PSS so that the sound had to be presented earlier to be perceived as simultaneous as compared to when attending the auditory modality (blue). * p < .05 ; ** p< .01 ; Bars are 2 s.e.m.

Additionally, participants’ attention to sensory modalities was manipulated in a manner orthogonal to the requirements of the TOJ task (Spence & Parise, 2010) using three attentional conditions (Fig. 1b): in the auditory attention blocks (blue), participants were asked to pay attention to the sounds only; in the visual attention blocks (red), to flashes only; in the AV blocks (orange), to split attention between auditory and visual sensory modalities. At the beginning of each block, a letter centered on the screen informed participants which sensory modality they should attend to (A for audition, V for vision and AV for split-attention). In the split-attention blocks, visual and auditory stimuli were equally likely so that 6 AV trials (3 with the sound on the left, 3 with the sound on the right), 3 auditory (AA) and 3 visual (VV) trials were presented for each SOA. In the auditory (visual) attention blocks, sounds (flashes) were presented in 66% of the trials so that 8 AV and 4 AA (VV) trials were presented for each SOA (Table S1 in the Supplemental Material available online). No AA (VV) trials were presented in the visual attention (auditory attention) blocks. There were a total of 10 blocks per experimental session (3 auditory attention blocks, 3 visual attention blocks, and 4 split-attention blocks) presented in random order and separated by rest periods. One experimental session lasted ~75 min in total.

Point of Subjective Simultaneity (PSS) and IntraClass Correlation (ICC)

For each of the four experimental sessions, the percent of responses ‘right stimulus first’ was plotted as a function of the SOA in the AA and VV conditions; in the AV conditions, responses were first converted to ‘flash first’. To establish the individual’s psychometric curve per condition and per session, data were fitted to binomial distributions using a probit link function estimated with a generalized linear model in Matlab. Goodness-of-fit measures (R²) were .90 on average, and above .73 in 95% of the fits. The temporal order thresholds or Point of Subjective Simultaneity (PSS) were evaluated as being the SOA for which participants were at chance level i.e. at 50% of ‘flash first’ for the PSS estimating audiovisual temporal order (Fig 1c, top), and of ‘right first’ for the PSS estimating auditory (PSS_A) and visual (PSS_V) temporal order.

Three different audiovisual PSSs could be estimated in this experiment (Fig. 1b): in split-attention (orange), when attending audition (blue) and when attending vision (red). To know whether the audiovisual PSS computed in split-attention condition (thereafter PSS_AV) was subject to an individual’s attentional bias towards one or the other sensory modality, we computed an additional PSS measure (thereafter, PSS_free) as the average of audiovisual PSSs in visual and auditory attention conditions:

$$PS{S}_{free}=\frac{PSS(auditoryattention)+PSS(visualattention)}{2}$$

The rationale for PSS_free was that biases towards one sensory modality should be maximal in visual and auditory attentional conditions, and should thus cancel out when averaged. Here, PSS_free was considered a measure of audiovisual PSS which would be free of any attentional biases. The existence of an individual bias independent of top-down attentional strategy in temporal order estimation should lead to a stable PSS_free. If PSS_AV and PSS_free significantly and systematically differed, the need to control participants’ attentional focus during task would be substantiated.

To assess the stability and the reliability of an individual’s PSS across sessions, we used IntraClass Correlation (“irr” package in R) which is a statistical measure establishing whether the variance in the data is best accounted for by intra- or inter-class variability. Here, the class was the individual, the intra-class variability was the measure of variance of PSS across sessions within an individual, and the inter-class variability was a measure of variance observed across individuals and sessions. Thus, the ICC corresponds to the ratio of the inter-individual variability with the sum of the intra-individual, inter-individual, and noise variabilities. A one-way model quantification of the consistency of measures based on single ratings was used (Hallgren, 2012).

Results

Prior-entry

Based on previous literature (Spence, Shore, & Klein, 2001), the audiovisual PSSs observed during the different attentional conditions were expected to shift towards the non-attended sensory modality. For instance, when attention was focused on audition, we expected that a flash would have to be presented earlier to be perceived as simultaneous with the sound, yielding a more negative PSS; conversely, a more positive PSS was expected when participants were paying attention to vision. The audiovisual PSSs obtained in each attentional condition were submitted to a two-way analysis of variance (ANOVA) with factors of attention (3: auditory, visual, split-attention), sessions (4: first session, one week, two weeks and four months apart) and nested random effects modeling the across-subjects variability. As predicted, a main effect of attention was found (F(2,18) = 6.12; p = .006, η²_p = .290) supporting the observation that the PSS observed in split-attention shifted towards more negative and more positive values under auditory and visual attention, respectively (Fig. 1c, bottom). A post-hoc Bonferroni-corrected paired t-test indicated that the audiovisual PSS under visual attention was significantly smaller (-14 +/- 3 ms) than under auditory attention (28 +/- 4 ms, t(15) = -2.72, p = .047, d = .75) and also smaller than under split-attention (5+/- 3 ms, t(15) = -4.05, p = .003, d = .34). However, the audiovisual PSS under auditory and split-attention did not differ statistically (t(15) = -1.77, p > .250, d = .45). No effects of sessions (F(3,18) = 0.70, p > .250, η²_p = .044) or interactions between attention and sessions (F(6,18) = 1.27; p > .250, η²_p = .078) were found, suggesting that the modulations of PSS under various attentional conditions remained stable across sessions, hence over time. Although participants were asked to favor accuracy over speed, we performed an analysis of reaction times (RT) as a function of attention and sensory modalities: results showed that prior-entry did not predict the differences of RT in TOJ, and that RT in audiovisual conditions were overall faster when the sound was presented first (Fig. S1a in the Supplemental Material available online). These analyses also suggested that RT were largely decorrelated from PSS in line with previous studies (Jaskowsi, 1999).

Stability of the PSS across sessions

The stability of the PSS across experimental sessions was assessed using ICC. PSS_AV in split-attention condition showed an ICC = .69 (CI_95% = [.50, .85], F(18,57) = 10.10, p < .001), meaning that 69% of the observed variance was due to inter-subject variability and 31% to intra-subject variability. This result suggested that PSS_AV was reliable across the four experimental sessions (Fig. 2a). Similarly, PSS_free, with an ICC of .77 (CI_95% = [.61, .89], F(18,57) = 14.70, p < .001) remained stable over time (Fig. 2b). We thus questioned the link between the two estimates of audiovisual PSS considering that sorting the individuals’ PSS_AV (orange) and PSS_free (purple) showed very similar ranking (Fig. 2c). The correlation between PSS_AV and PSS_free was highly significant (R = .92, CI_95% = [.81, .97], t(17) = 9.99, p < .001). The coefficients of the linear regression were 1.02 for the slope, -2.10 for the intercept, indicating that both measures were near equality (panel inset Fig. 2c). These results strongly suggested that PSS_AV was a reliable measure of an individual’s bias in a TOJ task that seemed immune to participants’ attentional strategy. The measures of the unisensory (auditory and visual) PSSs showed stability over time as well, along with a spatial bias of participants in these tasks (Fig. S2, in the Supplemental Material available online). Interestingly, participants’ introspective reports on task difficulty rated audiovisual TOJ to be the most difficult conditions – as compared to all other unisensory TOJ – and attentional conditions were rated as being equally difficult (Fig. S3, in the Supplemental Material available online).

Fig. 2. Stability of individuals’ audiovisual PSS.

Distribution of (a) individual audiovisual PSS in split-attention condition (PSS_AV, orange) in the four experimental sessions (shades of grey). The within-individual variance across all sessions is reported with box plots with the bar as the median PSS value. Participants were sorted as a function of increasing PSS i.e. from requiring the sound to be presented first (negative PSS) to requiring the visual event to be presented first (positive PSS) to perceive audiovisual simultaneity. A significant Intra-Class Correlation (ICC) of .69 signified that, over the four experimental sessions, the within-individual PSS variance was smaller than the inter-individual variance. Distribution of (b) individual audiovisual PSS free of attentional bias (PSS_free, purple) in the four experimental sessions (shades of grey). Box plots of individual PSS_free were sorted as a function of PSS_AV. As previously, the ICC of .77 signified that, over the four experimental sessions, the within-individual PSS_free variance was smaller than the inter-individual variance. The sorting of PSS_AV (orange) and PSS_free (purple) showed nearly (c) the same individual ranking to the exception of individual 7. Bottom right inset: PSS_AV and PSS_free were significantly correlated (R= .92, p <.001); shaded area is 95% CI *** p< .001; bars are 2 s.e.m.

Linking prior-entry and PSS

To further investigate the relationship between PSS and prior-entry, we quantified the magnitude of the prior-entry effect [PSS(auditory attention) − PSS(visual attention)] and tracked its stability over time. A significant ICC of .46 across sessions (CI_95% = [.23, .70], F(18,57) = 4.41, p < .001) suggested that the magnitude of the prior-entry effect was also stable within individuals over the four-month period (Fig. 3a). Hence, prior-entry showed a large inter-individual variability that remained stable over time. In particular, 3 (out of 19) participants showed a negative prior-entry effects in more than 3 out of 4 sessions, and six additional values spread over the remaining participants also showed a negative prior-entry effect. This suggested that either non-attended events were prioritized, or that participants may not have correctly followed task instructions. The inclusion or exclusion of these data did not affect the main results of our previous or, of our subsequent analyses; hence, these data points were preserved.

Fig. 3. Stability of individuals’ prior-entry effect.

The magnitude of prior-entry showed (a) a clear inter-individual variability with a significant ICC of .46, indicating that the magnitude of prior-entry was stable over 4 months. These results suggest that (b) an individual’s PSS fluctuated around a fixed bias (PSS_free, purple) as a function of whether the individual’s attention was oriented towards vision (red) or audition (blue). One dot is one individual’s average PSS over the four experimental sessions per experimental condition. The significant inter-individual variability of the magnitude of prior-entry indicates that the effect of attention on time order is specific to the individual. Additionally, the magnitude of prior-entry significantly (c) correlated with the absolute value of the PSS_AV, suggesting that attention could partially (but not fully due to (b)) compensate for an individual’s intrinsic temporal delays. Each dot corresponds to an individual and a session. ** p < .01

The extent of the prior-entry effect can be seen as the range of possible values an individual PSS can take as a function of the individual’s attentional focus (Fig. 3b). Considering that one functional role of prior-entry could be the compensation of temporal delays to achieve veridical simultaneity (or simultaneity constancy, e.g. Kopinska & Harris, 2004), individuals with a PSS far from veridical simultaneity were predicted to display larger magnitudes of prior-entry. To test this, the magnitude of the prior-entry (Fig. 3b, shaded area) was correlated with the absolute value of the PSS_AV (Fig. 2c, orange dots) for each individual and for each session: the two were significantly correlated (R = .34, CI_95% = [.13, .53], t(74) = 3.16, p = .002), suggesting that an individual with a large PSS magnitude tended to exhibit a larger prior-entry effect (Fig. 3c).

Discussion

With a longitudinal study, we showed the existence of a robust and stable inter-individual variability in audiovisual (as well as auditory and visual) temporal order perception. Our results suggest that temporal order is a psychological bias unique to each individual, which may result from structural constraints such as the intrinsic neural delays of an individual’s brain (Freeman, et al., 2013), or from functional constraints akin to hidden state variables (Wexler, Duyck, & Mamassian, 2015).

Inter-individual variability in audiovisual temporal order perception was observed irrespective of whether participants split attention across senses (PSS_AV) or not (PSS_free): the two indices were significantly correlated, indicating that an individual’s temporal order bias persisted through attentional manipulation. When participants attended one or the other sensory modality, shifts in the audiovisual PSS predictably followed the law of prior-entry, but interestingly fluctuated around the individual’s PSS_free. These shifts were consistent over time and opposite to the direction of the attended sensory modality. We also found that the magnitude of the prior-entry effect was significantly correlated with the absolute value of PSS_AV so that the further away an individual’s bias was from physical simultaneity, the larger the magnitude of the prior-entry. One possible functional interpretation for this finding is that prior-entry may reflect brain mechanisms that compensate for an individual’s temporal order bias or prior. In other words, our results suggest that attention, mediating the prior-entry effect, may enable an individual’s brain to compensate, but only to some extent, for its intrinsic temporal biases.

The existence of a psychological bias in temporal order is consistent with the observations that PSSs measured for various stimulus classes (beeps/flashes, audiovisual speech) are correlated within individuals (Love, Petrini, Cheng, & Pollick, 2013) despite differences in integration time (Vatakis & Spence, 2010). Thus, intrinsic biases may not depend on low-level stimulus features but on high-order brain computations, consistent with temporal order judgments typically activating parietal cortices (Battelli, Pascual-Leone, & Cavanagh, 2007; Woo, Kim, & Lee, 2009; Davis, Christie, & Rorden, 2009; Adhikari, Goshorn, Lamichhane, & Dhamala, 2013). To which extent such temporal order bias plays a role in the integration of information is however unclear: if an individual’s subjective simultaneity has been shown to correlate with the size of the temporal integration windows in audiovisual speech (Freeman, et al., 2013), the PSSs obtained in simultaneity and temporal order judgement tasks do not correlate (van Eijk, Kohlrausch, Juola, & van de Par, 2008; Love, Petrini, Cheng, & Pollick, 2013). These two measures likely capture different processes and temporal integration windows computed across individuals may be confounded by inter-individual variability in temporal order.

Altogether, these results emphasize the importance of quantifying inter-individual differences in psychological and in neuroimaging studies (Kanai & Rees, 2011) in order to shed light on possibly unnoticed neural mechanisms underlying temporal cognition.