Skip to main content
PMC home page
eLife logoLink to eLife
. 2020 Aug 25;9:e61238. doi: 10.7554/eLife.61238

Sensory experience during early sensitive periods shapes cross-modal temporal biases

Stephanie Badde 1,2,✉, Pia Ley 1, Siddhart S Rajendran 1,3, Idris Shareef 1,3, Ramesh Kekunnaya 1,3, Brigitte Röder 1
Editors: Tamar R Makin4, Barbara G Shinn-Cunningham5
PMCID: PMC7476755  PMID: 32840213

Abstract

Typical human perception features stable biases such as perceiving visual events as later than synchronous auditory events. The origin of such perceptual biases is unknown. To investigate the role of early sensory experience, we tested whether a congenital, transient loss of pattern vision, caused by bilateral dense cataracts, has sustained effects on audio-visual and tactile-visual temporal biases and resolution. Participants judged the temporal order of successively presented, spatially separated events within and across modalities. Individuals with reversed congenital cataracts showed a bias towards perceiving visual stimuli as occurring earlier than auditory (Expt. 1) and tactile (Expt. 2) stimuli. This finding stood in stark contrast to normally sighted controls and sight-recovery individuals who had developed cataracts later in childhood: both groups exhibited the typical bias of perceiving vision as delayed compared to audition. These findings provide strong evidence that cross-modal temporal biases depend on sensory experience during an early sensitive period.

Research organism: Human

Introduction

In every moment, a multitude of information reaches our brain through the different senses. These sensory inputs need to be separated, ordered in space and time to derive a coherent representation of the environment. Yet, the perception of temporal order is seldom veridical. Reports illustrating the subjectivity of cross-modal temporal perception date back to 18th and 19th century astronomers; small but stable individual biases in the perceived timing of visual and auditory events caused significant differences in the measurements of stellar transit times which in turn resulted in bitter scientific disputes. These early reports inspired the pioneering work of Wilhelm Wundt and colleagues on cross-modal temporal perception (Sanford, 1888; Canales, 2010) and, hence, are seen as the origin of experimental psychology (Mollon and Perkins, 1996). Early psychologists like William James transferred the personal equation, developed by Bessel to solve the astronomers’ problem of stable cross-modal temporal biases by adding a fixed amount to an observer’s measurements, to cognitive processes. Yet, even though the relative timing of events is crucial for the perception of time and causality (Hume, 2012; Pöppel, 1988; Pöppel, 1997; Spence and Squire, 2003; Aghdaee et al., 2014), it has yet remained unsolved why humans consistently misjudge temporal order across different sensory modalities.

Determining the spatio-temporal order of events across sensory modalities poses an especially difficult challenge as information arriving through different senses travels at different speeds – in the environment and within the nervous system (Hirsh and Sherrick, 1961). Light travels faster than sound, but within the nervous system, auditory information reaches the brain faster than visual information (Fain, 2019). At a distance of approximately 10 m from the observer, the environmental and the physiological differences presumably cancel each other out (Pöppel, 1988). However, both the relative time of arrival at the sensors and neural transmission times vary with the physical properties of the stimulus (King, 2005). Curiously, the brain seems to be able to adjust the perception of temporal order for variations in signal strength (Kopinska and Harris, 2004) and physical distance (Engel and Dougherty, 1971; Alais and Carlile, 2005), yet, within their peripersonal space humans typically perceive visual events as delayed compared to sounds (Keetels and Vroomen, 2012; Zampini et al., 2003; Grabot and van Wassenhove, 2017).

Cross-modal temporal biases exhibit a paradoxical characteristic, they are remarkably stable across longer periods of time (Sanford, 1888; Grabot and van Wassenhove, 2017) while being malleable within short time periods. After humans are exposed to a series of asynchronous stimulus pairs with a constant lag between vision and audition (Fujisaki et al., 2004; Vroomen et al., 2004) (or vision and touch; Keetels and Vroomen, 2008), they perceive the temporal order of subsequent visual-auditory events as shifted in the lag-reversed direction. Such temporal recalibration effects typically last for several minutes. Moreover, on a moment-by-moment basis, the perceived relative timing of events in different modalities changes with the focus of attention. Titchener’s law of prior entry (Titchener, 1908) states that attention towards one sensory input channel is capable of speeding up perceptual processing of that sensory information resulting in a demonstrable shift of temporal order perception towards events in the attended sensory modality (Spence et al., 2001; Zampini et al., 2005; Vibell et al., 2007; Spence, 2010). The co-existence of short-term plasticity and long-term stability re-emphasizes the question that has troubled scientists for 250 years: why do cross-modal temporal biases persist despite the brain’s capability for recalibration?

A potential reason for the high stability of cross-modal temporal biases is that they are shaped during a sensitive period of development. Sensitive periods are limited periods during brain development in which the influence of sensory experience on the brain is particularly strong (Knudsen, 2004). During a sensitive period, the architecture of a neural circuit is shaped to meet an individual’s environment. The long-term result is a preference of the neural circuits for certain states of activity even if the environment dramatically changes (Röder et al., 2007; Röder et al., 2004; Sourav et al., 2019), without prohibiting short-term changes in the neural circuits’ activity patterns (Knudsen, 2004; Knudsen, 2002). The role of early sensory experience for the genesis of perceptual biases is extremely difficult to address in humans; only individuals whose early sensory environments were atypical due to natural causes open a window into the role of experience in perceptual development. We tested the hypothesis of a sensitive period for cross-modal perceptual biases by asking individuals born with dense, bilateral cataracts whose sight was restored 6 to 168 months after birth (Table 1) to temporally order spatially separated events across vision, audition and touch.

Table 1. Sample characteristics.

N Sex Handedness Visual acuity of
the better eye		 Age at testing Age at surgery Time period
between surgery
and testing
CC – Expt. 1
visual-auditory		 10 8 males 10 right-handed 0.16–1.3 logMAR,
mean 0.84 logMAR		 9–46 years,
mean 30 years		 6–168 months,
mean 60 months		 24–528 months,
mean 296 months
DC – Expt. 1
visual-auditory		 9 7 males 9 right-handed −0.5–0.8 logMAR,
mean 0.25 logMAR		 8–19 years,
mean 13 years		 74–183 months,
mean 120 months		 12–66 months,
mean 30 months
CC – Expt. 2
visual-tactile		 10 10 males 10 right-handed 0.16–1.3 logMAR,
mean 0.75 logMAR		 11–45 years,
mean 30 years		 5–216 months,
mean 57 months		 93–516 months,
mean 296 months
DC – Expt. 2
visual-tactile		 9 6 males 9 right-handed 0–0.5 logMAR,
mean 0.22 logMAR		 9–19 years,
mean 14 years		 30–183 months,
mean 102 months		 13–174 months,
mean 63 months
Open in a new tab

Results

Thirteen individuals with a history of congenital bilateral, dense cataracts which had been surgically removed (CC-group) participated in two spatial temporal order judgement tasks, ten in a visual-auditory task (Expt. 1) and ten in a visual-tactile task (Expt. 2). To test for the role of vision during infancy as well as to control for effects related to having had eye surgery and persisting visual impairments, sixteen individuals, nine per experiment, who had undergone surgery for cataracts which had developed during childhood served as controls (DC-group). As both cataract-reversal groups differed in age (Table 1) and age is known to influence cross-modal temporal perception (Röder et al., 2013; Noel et al., 2016), their performance was not directly compared but each group was contrasted separately against age-matched typically sighted individuals (MCC- and MDC-groups). In every trial, two successive stimuli were presented, one in each hemifield (Figure 1A). Stimulus order and the time interval between the two stimuli (stimulus onset asynchrony; SOA) varied randomly across trials. Visual-auditory and visual-tactile stimulus pairs were randomly interleaved with unimodal stimulus pairs. Participants reported the side of the first stimulus irrespective of its modality (Zampini et al., 2003). For the CC-group, we predicted that due to their history of visual loss and persisting visual impairments participants would give auditory and tactile stimuli a higher preference than visual stimuli. Such a preference would result in a shift of the point of perceived simultaneity (PSS) –the SOA at which auditory-visual or tactile-visual pairs are perceived as simultaneous– towards SOAs with greater auditory or tactile lags, and an overall lower proportion of ‘visual first’-responses compared to their controls. Additionally, we predicted a lower visual spatio-temporal resolution resulting in a lower proportion of correct temporal order judgments for the CC-group.

Figure 1. Effects of early visual experience on spatio-temporal order biases and resolution within and across vision, audition, and touch.

Thirteen individuals with surgically removed congenital, bilateral dense cataracts (CC) and sixteen individuals whose reversed cataracts had developed later in life (DC) as well as normally sighted individuals matched for age, gender, and handedness (MCC, MDC) took part in the study (see Table 1 for details about the samples). (A) Participants judged the spatio-temporal order of two successive stimuli –one presented in each hemifield– by indicating the location of the first stimulus. In Expt. 1, visual-visual, auditory-auditory, and visual-auditory stimulus pairs were presented, in Expt. 2, visual-visual, tactile-tactile, and visual-tactile stimulus pairs. (B) Psychometric curves: group average probabilities of a ‘right side first’-response as a function of SOA (negative values indicate ‘left side first’-stimulation). Data are split by the modality presented at the right side (visual = grey, auditory = dark blue, tactile = dark red) and the modality at the other side (rows, bimodal = different modality, unimodal = same modality). Curves display cumulative Gaussian distributions fitted to the group data for illustrative purposes. Error bars indicate standard errors of the mean in all panels. (C) Probit analysis: Group mean probits (CC = filled circle, MCC = open circle, DC = filled square, MDC = open square) of the probability for a ‘visual first’-response as a function of SOA (positive values indicate ‘visual first’-stimulation) in bimodal trials (visual-auditory = light blue, visual-tactile = light red). Lines show group averages of linear regression lines fitted to individual participant’s data. The longest SOA was excluded from the linear regression, if a participant’s performance had reached an asymptote. (D) Point of cross-modal subjective simultaneity (PSS): Box-and-whisker plots show the distribution (median, quartiles, minimum and maximum bounded at 1.5 x interquartile range, outliers) of individual PSS values; big circles indicate group mean values. Positive PSS values indicate that stimulus pairs in which the visual stimulus was presented first were perceived as simultaneous. (E) ‘Visual first’-bias: Box-and-whisker plots show the distribution of the proportion of ‘visual first’-responses per participant; big circles indicate group mean values. All SOAs were presented equally often, thus, a PSS of zero and a proportion of ‘visual first’-responses equal to 0.5 would have been correct. A positive PSS corresponds to a proportion of ‘visual first’-responses below 0.5 and vice versa. (F) Temporal order judgment accuracy: Distribution and group average probabilities of correct responses separately for each group and modality condition for each of three different modality combinations per experiment (light blue: visual-auditory, light red: visual-tactile, grey: visual-visual, dark blue: auditory-auditory, dark red: tactile-tactile). (G) Bias-accuracy relation: Single participants’ proportion of correct responses (x-axis) plotted against the absolute value of their ‘visual first’-bias (y-axis; markers as in (C)). Lines show linear regressions and 95% confidence intervals. (H) Reaction time (RT) distributions: Individually normalized RT in bimodal trials as a function of SOA (positive values indicate ‘visual first’-stimulation) per group. Markers and lines show group mean values (markers and line styles as in (C)).

Figure 1.

Figure 1—figure supplement 1. Excluded data.

Figure 1—figure supplement 1.

Participant-level mean proportions of ‘right side first’-responses are shown as a function of the stimulus onset asynchrony (SOA) of the two stimuli, with negative values indicating ‘left side first’-stimulation. The data are split into responses to bimodal (top row) and unimodal (bottom row) stimulus pairs and according to the modality presented at the right side (visual = grey, auditory = dark blue, tactile = dark red). Sigmoid curves fitted to the group mean data reported in Figure 1B are shown as a reference for participants for whom the onset of the cataract could be identified.

Figure 1—figure supplement 2. Visual-auditory temporal order judgments in typically sighted individuals in an additional experiment controlling for the effects of visual stimulus brightness.

Figure 1—figure supplement 2.

Five additional, typically sighted participants (all female and right-handed, 23–50 years old, mean age 34 years) completed the visual-auditory experiment. Here, a visual stimulus consisted of three activated LEDs whereas in the main experiment for typically sighted controls only one LED had been activated. Group mean proportions of ‘right side first’-responses are shown as a function of the stimulus onset asynchrony (SOA) of the two stimuli, with negative values indicating ‘left side first’-stimulation. The data are split into responses to bimodal (top row) and unimodal (bottom row) stimulus pairs and according to the modality presented at the right side (visual = grey, auditory = dark blue). Participants showed a typical bias of perceiving visual stimuli as delayed compared to auditory stimuli.

Figure 1—figure supplement 3. Just noticeable differences (JND).

Figure 1—figure supplement 3.

Box-and-whisker plots show the distribution (median, quartiles, minimum and maximum bounded at 1.5 x interquartile range, outliers) of individual JND values; big circles indicate group mean values.
Open in a new tab

Most strikingly and contrary to our predictions, CC-individuals showed a bias toward perceiving visual stimuli as occurring earlier than auditory (Figure 1D,E; CC-group, Expt. 1, PSS, t(9) = −1.51, p=0.033, requiv = 0.57; ‘visual first’-response probability, χ2 (1)=4.31, p=0.038, requiv = 0.55; see Supplementary file 1 for full statistical models) and tactile stimuli (CC-group, Expt. 2, PSS, t(9) = −3.04, p=0.009, requiv = 0.69; ‘visual first’-response probability, χ2(1)=12.46, p<0.001, requiv = 0.85), respectively. CC-individuals’ bias toward perceiving visual stimuli as earlier stood in contrast to the bias observed for their matched controls (CC- vs. MCC-group, Expt. 1, visual-auditory, PSS, t(18) = −2.34, p=0.005, requiv = 0.56; ‘visual first’-response probability, χ2 (1)=8.48, p=0.004, requiv = 0.57; Expt. 2, visual-tactile, PSS, t(18) = −2.47, p=0.008, requiv = 0.53; ‘visual first’-response probability, χ2 (1)=5.31, p=0.021, requiv = 0.45) who perceived auditory stimuli as occurring earlier than visual stimuli (MCC-group, Expt. 1, visual-auditory, PSS, t(9) = 2.30, p=0.009, requiv = 0.69; ‘visual first’-response probability, χ2 (1)=4.18, p=0.041, requiv = 0.55) and did not show a significant bias for visual-tactile comparisons (MCC-group, Expt. 2, visual-tactile, PSS, t(9) = −0.48, p=0.321, requiv = 0.16; ‘visual first’-response probability, χ2 (1)=0.15, p=0.701, requiv = 0.17). In contrast to the CC-group, DC-individuals showed the typical bias toward perceiving visual stimuli as occurring later than auditory and tactile stimuli (DC-group, Expt. 1, visual-auditory, PSS, t(8) = 1.42, p=0.017, requiv = 0.67; ‘visual first’-response probability, χ2 (1)=7.14, p=0.008, requiv = 0.74; Expt. 2, visual-tactile, PSS, t(8) = 2.31, p=0.017, requiv = 0.67; ‘visual first’-response probability, χ2 (1)=8.86, p=0.003, requiv = 0.79). In fact, their spatio-temporal bias toward perceiving visual stimuli as later than tactile stimuli was stronger than the bias of their matched controls (DC- vs. MDC-group, Expt. 1, visual-auditory, PSS, t(18) = 1.40, p=0.123, requiv = 0.29; ‘visual first’-response probability, χ2 (1)=1.84, p=0.174, requiv = 0.23; Expt. 2, visual-tactile, PSS, t(18) = 2.17, p=0.019, requiv = 0.49; ‘visual first’-response probability, χ2 (1)=8.08, p=0.004, requiv = 0.60; MDC-group, Expt. 1, visual-auditory, PSS, t(8) = 1.32, p=0.152, requiv = 0.36; ‘visual first’-response probability, χ2 (1)=0.83, p=0.362, requiv = 0.13; Expt. 2, visual-tactile, PSS, t(8) = −0.36, p=0.361, requiv = 0.13; ‘visual first’-response probability, χ2=0.85, p=0.356, requiv = 0.15).

CC-individuals’ proportion of correct temporal order judgments for visual and cross-modal stimulus pairs (Figure 1F) was reduced compared to their controls’ temporal resolution (CC-group vs. MCC-group, Expt. 1, visual, χ2 (1)=9.68, p=0.002, requiv = 0.62; Expt. 1, visual-auditory, χ2 (1)=17.88, p<0.001, requiv = 0.78; Expt. 2, visual, χ2 (1)=4.72, p=0.030, requiv = 0.42; Expt. 2, visual-tactile, χ2 (1)=4.03, p=0.045, requiv = 0.39; see Supplementary file 1 for full statistical models), but no significant difference between the CC-group and their controls emerged for auditory and tactile stimulus pairs, (CC-group vs. MCC-group, Expt. 1, auditory, χ2 (1)=1.42, p=0.233, requiv = 0.17; Expt. 2, tactile, χ2 (1)=0.11, p=0.742, requiv = 0.15). DC-individuals’ spatio-temporal resolution was lower than the resolution of their controls independent of modality condition (DC-group vs. MDC-group, Expt. 1, all modalities, χ2 (1)=4.98, p=0.026, requiv = 0.46; Expt. 2, all modalities, χ2 (1)=11.99, p=0.001, requiv = 0.68).

Across participants, the size of the bias in the proportion of ‘visual first’-responses in bimodal trials correlated negatively with the proportion of correct temporal order judgements in these trials (Figure 1G; Expt. 1, visual-auditory, r = −0.39, p=0.018; Expt. 2, visual-tactile, r = −0.49, p=0.002). The distribution of reaction times in bimodal trials peaked at different SOAs across groups (Figure 1H; CC- vs. MCC-group, Expt. 1, visual-auditory, t(16) = 2.11, p=0.041, requiv = 0.40; Expt. 2, visual-tactile, t(18) = 1.56, p=0.067, requiv = 0.35) with peaks being shifted in the same direction as the PSS.

Discussion

Here, we investigated whether biases in cross-modal temporal perception are shaped during a sensitive period in early development by testing sight-recovery individuals with a history of congenital loss of pattern vision. In two spatio-temporal order judgement tasks, individuals with reversed congenital cataracts disproportionally often reported visual stimuli as occurring earlier than auditory and tactile stimuli (Figure 1B–E), exhibiting a reversed cross-modal bias compared to typically sighted controls and sight-recovery individuals whose cataracts had developed during childhood. These results, for the first time, demonstrate a sensitive period for cross-modal temporal biases. Moreover, the ability to correctly determine the spatio-temporal order of separate events across different sensory modalities and within vision was reduced after a transient phase of visual loss both after birth and later in childhood (Figure 1F) suggesting a persistent impairment of visual and cross-modal temporal resolution following transient visual deprivation.

At first glance, CC-individuals’ bias to perceive visual events as earlier than auditory and tactile events seems to indicate that visual stimuli were processed faster than auditory and tactile stimuli following transient, congenital visual deprivation. However, CC-individuals’ response accuracy (Figure 1F) indicated temporal processing impairments for vision but not for audition and touch. Consistently, studies measuring event-related potentials have provided no evidence for a shorter latency of the first visual cortical response in CC-individuals (Sourav et al., 2018; Mondloch et al., 2013; McCulloch and Skarf, 1994) and behavioral studies have demonstrated no advantage in reaction times to simple visual stimuli (Putzar et al., 2012; de Heering et al., 2016). Moreover, reduced visual contrast – typical for cataract-reversal individuals – is associated with delayed responses of the visual system and lower visual temporal sensitivity (Watson, 1986; Stromeyer and Martini, 2003). In sum, there is currently no evidence indicating accelerated processing of visual information following congenital visual deprivation.

During sensitive periods, experience shapes neural circuits customizing them to an individual’s body and environment (Knudsen, 2004). CC-individuals’ reversed cross-modal bias towards perceiving visual stimuli as occurring earlier than auditory and tactile stimuli might be rooted in consistent exposure to lagging visual input during an early sensitive period. Residual light perception, which exists even in the presence of dense cataracts, is typically sluggish and retinal transduction rates have been reported to be reduced in patients with dense, untreated cataracts (Yamauchi et al., 2016). Moreover, a suppression of visual cortex activity has recently been observed in CC-individuals in the context of cross-modal stimulation (Guerreiro et al., 2016). Thus, CC-individuals likely were exposed to consistently delayed visual signals (with respect to auditory and tactile signals) before the cataracts were removed. Recalibration studies have shown that exposure to a consistent visual delay results in a bias towards perceiving visual input as earlier (Fujisaki et al., 2004; Vroomen et al., 2004), consistent with the present results in the CC-group. Thus, we suggest that the reversed bias exhibited by CC-individuals reflects adaptation to their atypical sensory environment during infancy, which results in structural differences in the neural circuits processing temporal order across modalities.

Individual cross-modal temporal biases in typically sighted humans might reflect the relative timing of sensory information across modalities during infancy, too. One-month old infants have been found to already show rudimentary cross-modal temporal biases for audition and vision (Lewkowicz, 1996). Slight differences in the relative rate of brain development across individuals, for example, in the myelination of neural circuits in different sensory areas, could give rise to the characteristic but stable (Sanford, 1888; Grabot and van Wassenhove, 2017) inter-individual differences in cross-modal temporal biases in humans. In turn, the variation of individual cross-modal temporal biases as a function of task (Freeman et al., 2013; Ipser et al., 2018; Ipser et al., 2017) might arise from differences in the relative rate of development across cortical areas. In sum, we suggest that individual cross-modal temporal biases (Zampini et al., 2003; Grabot and van Wassenhove, 2017) show a high stability because the brain had optimized cross-modal temporal perception (Roseboom et al., 2015) during a sensitive period which constitutes a setpoint for all future recalibration.

Additionally, the congenital absence of pattern vision might have increased CC-individuals’ attention towards this modality after sight restoration, resulting in a permanent prior entry effect for vision (Titchener, 1908; Spence et al., 2001; Vibell et al., 2007; Shore et al., 2001). However, a previous study found longer switch times from audition to vision but not from vision to audition in CC-individuals whose cataracts had been removed in the first months of life than in controls, which was interpreted as indicating an attentional bias towards audition over vision (de Heering et al., 2016). Moreover, an attentional bias towards vision would have been expected to result in relatively shorter visual reaction times and early cortical potentials, which, as argued at the beginning of the Discussion is incompatible with the literature (Sourav et al., 2018; Mondloch et al., 2013; McCulloch and Skarf, 1994; Putzar et al., 2012; de Heering et al., 2016). Thus, while scarce existing data points towards a reduced dominance of vision in CC-individuals, the consequences of congenital loss of pattern vision for modality-specific attention are not yet understood.

The finding, that CC-individuals but not DC-individuals showed a reversed visual-auditory and visual-tactile temporal bias strongly suggests that perceptual biases are shaped by sensory experience during the initial period of life. In contrast to CC-individuals, DC-individuals had encountered unimpaired cross-modal stimulation after birth, which we suggest had enabled them to establish a typical bias. An early sensitive period would have prevented DC-individuals –who lost their vision later– from adapting this typical bias to their changed sensory environment. As a consequence, their partially severe, persisting visual impairments might have caused an increase of the typical bias of perceiving vision as delayed. However, the bias of some DC-individuals seems too large to be accounted for by delays in visual processing alone. The very high temporal uncertainty of some DC-individuals might have resulted in an overestimation of their cross-modal biases (Figure 1G): In agreement with Bayes’ law, temporal perceptual judgments tend to rely more on prior information such as existing biases when the sensory information is uncertain (Shi and Burr, 2016; Vercillo et al., 2015). Consistent with this idea, the current data indicated a correlation between the size of the bias and the degree of sensory uncertainty. The sensitive period for cross-modal temporal perception might culminate during the first six months of life, the minimal duration of deprived vision in our CC-group. A previous study tested CC-individuals whose vision was restored within early infancy (4 months of age on average) in an audio-visual simultaneity judgment task with spatially aligned visual-auditory and visual-tactile stimulus pairs and did not report a reversed cross-modal bias (Chen et al., 2017). However, simultaneity judgments of spatially aligned cross-modal stimuli are simpler and less sensitive to biases than the spatial temporal order judgements we used (van Eijk et al., 2008; Linares and Holcombe, 2014). Moreover, both tasks rely on different neuronal processes (Basharat et al., 2018) and even one-month old infants seem to be able to evaluate simultaneity (Lewkowicz, 1996). Thus, it is questionable whether the previous and our experimental design addressed the same multisensory processes. In sum, our results provide strong evidence for a sensitive period for cross-modal temporal perception, which leads to the consolidation of cross-modal spatio-temporal biases during early human ontogeny.

Typically, reaction times increase with increasing perceptual uncertainty. In temporal order judgements, uncertainty is highest for simultaneously perceived cross-modal stimuli. Thus, reaction times are expected to be longest at the PSS, if the PSS reflects biases in the perceived relative timing of the modalities, but not if the PSS reflects a response bias towards one modality (Spence, 2010; Shore et al., 2001). Here, CC-individuals’ reaction times and thus sensory uncertainty peaked when the visual stimulus slightly lagged behind the auditory or the tactile stimulus (Figure 1H), whereas all other participants responded slowest when the visual stimulus led by a short amount of time. Thus, even though response speed was not stressed in the instructions, reaction times of the present study suggest that the observed biases reflect perceptual shifts in the relative timing of cross-modal sensory information.

Both cataract-reversal groups exhibited lower spatio-temporal accuracy (Figure 1F) –indicative of a decreased temporal resolution and an increased lapse rate (Figure 1B)– than their controls in the visual and the cross-modal tasks. Persisting visual impairments might have contributed to the reduced visual spatio-temporal resolution of both cataract-reversal groups given that visual contrast affects visual temporal perception (Watson, 1986; Stromeyer and Martini, 2003). The observation that a lower temporal resolution was found not only for visual but additionally for cross-modal temporal order judgments suggests a strong dependence of cross-modal temporal ordering on the visual sense. This might be related to the spatial nature of our task; training of non-spatial, purely visual temporal order judgments did not lead to an improved visual-auditory temporal resolution (Alais and Cass, 2010). The finding that both cataract groups exhibited a lower temporal resolution but only the CC-group an altered cross-modal bias strongly suggests that cross-modal spatio-temporal biases and resolution are dissociable processes. The conjunction of CC- and DC-individuals’ reduced resolution might point towards a long sensitive period for the development of spatio-temporal sensory resolution which would be compatible with the protracted developmental time course of cross-modal temporal ordering of spatially separate events (Röder et al., 2013).

Furthermore, the present finding of increased cross-modal temporal uncertainty could explain why recent studies have found altered multisensory integration for some but not all functions following congenital, transient periods of visual deprivation (Chen et al., 2017; Putzar et al., 2007; Guerreiro et al., 2015). An increased spatio-temporal uncertainty hinders the detection of temporal correlations (Parise and Ernst, 2016) between complex signals such as speech stimuli (Putzar et al., 2007; Guerreiro et al., 2015). At the same time, higher temporal uncertainty predicts wider temporal integration windows for simple, spatially-aligned stimuli (Chen et al., 2017), which in turn might have enabled typical multisensory redundancy gains for such cross-modal stimuli (Putzar et al., 2012; de Heering et al., 2016). It is interesting to note, that this pattern mirrors the typical development of multisensory temporal perception. Very young infants integrate simple visual and auditory stimuli across a wide window of asynchronies but do not integrate more complex stimuli based on other features than stimulus onset (Lewkowicz, 1996; Lewkowicz, 2012). The multisensory integration window narrows during childhood (Noel et al., 2016; Lewkowicz and Flom, 2014; Hillock et al., 2011), in parallel to the improvement of unisensory temporal perception (Brannon et al., 2007). Some developmental theories suggest that more complex temporal multisensory functions build on previously acquired temporal multisensory skills (Lewkowicz, 2000) predicting that the impact of transient loss of pattern vision on multisensory perception should as suggested for the visual system (Hyvärinen et al., 1981; Maurer et al., 2005), increase with increasing task difficulty.

In conclusion, congenital but not late transient visual deprivation was associated with a bias towards perceiving visual events as earlier than auditory or tactile events in sight-recovery individuals, suggesting that cross-modal temporal biases depend on sensory experiences during an early sensitive period.

Materials and methods

Participants

The sample of the visual-auditory experiment (Expt. 1) comprised ten individuals who were born with bilateral dense cataracts (congenital cataracts, CC) and whose vision was restored later in life (for details see Table 1) and nine individuals with transient, bilateral cataracts which had developed during childhood (developmental cataracts, DC). The sample tested in the visual-tactile experiment (Expt. 2) comprised ten CC- and nine DC-individuals. For each CC- and DC-participant an age-, gender- and handedness-matched control participant was recruited. Seven CC- and two DC-individuals as well as 13 control participants took part in both experiments. The majority of participants with a history of cataracts were recruited and tested at the LV Prasad Eye Institute in Hyderabad, India. Three CC-individuals and all control participants were recruited and tested at the University of Hamburg, Germany. The presence of congenital cataracts was affirmed through an analysis of the medical records by the participant’s optometrist and ophthalmologists. Since cataracts were sometimes diagnosed at a progressed age, additional criteria such as presence of nystagmus, strabismus, the density of the lenticular opacity, the lack of fundus visibility prior to surgery, a family history of congenital cataracts, and parents’ reports were employed to confirm the onset of the cataract. Data of five additional participants were excluded from all analyses because (a) the onset of the cataract remained unclear (two participants, one took part in the visual-auditory experiment and one in the visual-tactile experiment), (b) the time period between surgery and testing was shorter than 12 months (one participant from the CC-group who took part in both experiments and one participant from the DC-group who took part in the visual-auditory experiment), or because (c) additional neurological problems were suggested by medical records (one participant from the CC-group who took part in both experiments). Data of two further participants (one CC- and one DC-individual) were excluded because they performed even in the easiest 400-ms-long SOA condition at chance level or below in the visual-auditory experiment. The 400-ms-long SOA was included to check whether participants had understood the task, which was explained to them by an interpreter. All excluded data are shown in the supplementary information (Figure 1—figure supplement 1) and are in accordance with the results presented in the main text. Adult participants and legal guardians of minors were reimbursed for travel expenses, accommodation, and absence from work, if applicable; adult participants tested in Hamburg received a small monetary compensation or course credit. Children received a small present. All participants or, if applicable, their legal guardian, provided written informed consent before beginning the experiment. The study was conducted in accordance with the ethical guidelines of the Declaration of Helsinki and was approved by the ethical board of the German Psychological Society as well as the local ethical committee of the LV Prasad Eye Institute.

Apparatus and stimuli

Participants sat at a table, facing two speakers, positioned at 14° visual angle (15 cm at 60 cm distance) to the left and to the right of the participant’s midline (Figure 1A). Three LEDs were mounted on top of each speaker. In the visual-tactile experiment, custom-made, noise-attenuated tactile stimulators were attached to the dorsal sides of both index fingers. A stimulus lasted 15 ms, independent of modality. During a stimulus, the LEDs emitted red light, the speakers played white noise, and the tactile stimulators vibrated at a frequency of 100 Hz. All three LEDs were used for cataract-reversal participants, but only one LED for typically sighted participants, to roughly compensate for persistent visual impairments in cataract-reversal participants. To rule out that typically sighted participants perceived vision as delayed due to the lower number of LEDs, we tested five additional typically sighted participants (all female and right-handed, 23–50 years old, mean age 34 years) in the visual-auditory experiment while using all three LEDs. These participants showed a significant typical bias towards perceiving vision as delayed (t(4)=6.36, p=0.003, Figure 1—figure supplement 2). Constant white noise was presented from a centrally located speaker, to mask residual noise produced by the tactile stimulators. During the experiment, participants fixated a mark placed centrally between the loudspeakers and rested both hands on buttons aligned with the loudspeakers (visual-auditory experiment) or both feet on foot pedals (visual-tactile experiment). Younger participants sometimes experienced problems activating the response devices in a controlled manner. These participants (visual-auditory experiment: one CC- individual; visual-tactile experiment: two DC-individuals) and their controls responded by waving one hand and the experimenter entered the response. The experiment was controlled by Presentation (Version 17.1.05, Neurobehavioral Systems, Inc, Berkeley, CA, www.neurobs.com), which recorded responses and interfaced with custom-built hardware to drive the stimulators.

Task, procedure, and design

In each trial, two stimuli were presented in close succession; one stimulus in each hemifield. Participants indicated at which side they perceived the first stimulus. Responses had to be withheld until the second stimulus had been presented. Response times were not restricted, and the next trial started 2 s after the response had been registered.

The modality of the stimulus presented at either side (visual or auditory, Expt. 1; visual or tactile, Expt. 2) and the stimulus onset asynchrony (SOA;±30,±90,±135,±400 ms, with negative SOAs indicating ‘left side first’-stimulus pairs) of the two stimuli varied pseudo-randomly across trials. Each of the 32 stimulus conditions (2 modalities x 2 sides x 8 SOAs) was repeated 10 times; the 320 trials were divided into 10 blocks. Participants additionally completed ten practice trials with an SOA of ±400 ms at the beginning of the experiment. If necessary, the practice trials were repeated until participants felt confident about the task. In the visual-tactile experiment, a subsample of participants was additionally tested while holding the hands crossed (data not reported here). Participants were encouraged to take breaks in between blocks. Some of the cataract-reversal participants did not complete the full experiment, mostly due to time constraints. Except for practice trials, participants did not receive feedback.

Data analysis

Data and analysis scripts are made available online (Badde et al., 2019). Trials with reaction times shorter than 100 ms and more than 2.5 standard deviations above the participant’s mean reaction time (RT) were excluded from the analysis (2.1% of trials; responses entered by the experimenter were not filtered).

Each participants’ data were split according to the modality of the stimulus pair (visual-visual, auditory-auditory, or visual-auditory for Expt. 1; visual-visual, tactile-tactile, or visual-tactile for Expt. 2). Participants’ left-right responses were transformed (a) into binary ‘right first’ – values (1 = ‘right first’-response, 0 = ‘left first’-response; Figure 1B), and (b) for bimodal trials only, into binary ‘visual first’ – values (1 = ‘visual first’-response, 0 = ‘auditory/tactile first’-response).

To test for temporal order biases toward one modality, two complementary analyses of ‘visual first’-values were conducted. (1) We estimated each participant’s point of subjective simultaneity (PSS), the theoretical SOA at which a visual and a non-visual stimulus are perceived as simultaneous. To this aim, we linearized the proportion of ‘visual first’-responses as a function of SOA (with positive values indicating ‘visual first’-stimulation) using a probit transformation and fitted a linear regression line (Figure 1C). If a participant’s performance had reached an asymptote, that is if performance at the longest SOA was equal to that at the second longest SOA, the longer SOA was dropped to allow for a better fit. The PSS equals the zero point of the linear function. PSS estimates were bounded at ±150 ms; the results pattern was not influenced by the value of the boundary (Figure 1D). To test for group effects, we conducted a one-way ANOVA and followed-up by planned unpaired t-tests comparing the CC- and the DC-group each with their respective control groups. Moreover, each groups’ PSS values were tested against zero with a t-test. As the normality assumption was violated, permutation tests were used to derive a null-distribution and thus non-parametric p-values for all tests. (2) We additionally analyzed participants’ probability of a ‘visual first’-response independent of stimulation, that is across SOAs. This measure corresponds to the criterion in signal detection theory and was added because it is robust against outliers along the psychometric functions. Single trial ‘visual first’-values were fitted with a generalized linear mixed model with a binomial distribution family and a log link function. Group was included as fixed effect and participants were treated as random effects. As planned comparisons, we first calculated pairwise contrasts comparing each cataract-reversal group with its matched control group and second estimated fixed contrasts separately for each group to evaluate whether the probability to perceive the visual stimulus before the auditory or tactile stimulus significantly differed from chance level.

To analyze the spatio-temporal resolution across groups and modality conditions, single trial accuracy values (1 = correct, 0 = incorrect) were fitted with a generalized linear mixed model with a binomial distribution family and a log link function. Group and stimulus modality were included as fixed effects and participants were again included as random effects. To resolve interactions between both predictors, we first conducted pairwise contrasts on both predictors comparing group differences between each cataract-reversal group and its matched control group across modalities and second pairwise contrasts testing for group differences separately for each modality condition. Additionally, we derived and report another measure of temporal resolution, just noticeable difference (JND) values, that is the SOA at which participants perceive the correct temporal order with 75% probability (Figure 1—figure supplement 3). However, for participants whose performance had not reached an asymptote at an SOA of ±135 ms these estimates are less reliable than for participants for whom this was the case. The first situation was more frequent in both the CC- and the DC-group, while the second situation was more frequent in the control groups. Therefore, we refrained from statistical group comparisons for JNDs.

To explore a possible relation between participant’s cross-modal temporal bias and temporal resolution, a Pearson correlation was calculated across participants from all groups. The size of the bias was defined as the distance between the proportion of ‘visual first’-responses and 0.5, which is equivalent to no bias. The resolution was defined as the proportion of correct temporal order judgments. To avoid leverage effects, data points exceeding 2.5 standard deviations distance from the group mean in any measure were excluded.

Finally, reaction times (RT) in bimodal trials were analyzed as a function of SOA (with positive values indicating ‘visual first’-stimulation). We normalized each participant’s RT (responses entered by the experimenter were excluded) across SOAs and compared maximum likelihood estimates of the SOA at which the RT distribution peaked across groups. Permutation tests of the unpaired t-statistic were employed for the comparisons, because the normality assumption was violated.

We report requivalent (Rosenthal and Rubin, 2003) as effect size estimate, since no generally accepted standardized effect size measure exists for mixed models and permutation analyses.

Acknowledgements

Our work was supported by the European Research Council (ERC-2009-AdG 249425 CriticalBrainChanges) to BR and the German Research Foundation (DFG) with a research fellowship grant to SB (BA 5600/1–1) and DFG Ro 2625/10–1 to BR as well as by the University of Hamburg with a post-doctoral fellowship to SB. We thank Marlene Hense, Nicola Kaczmareck, Carla Petroll, Lea Hornung, Deniz Froemke, and Rakesh Balachandar for help with data acquisition as well as Kabilan Pitchaimuthu and Prativa Regmi for help with data curation. We thank D Balasubramanian of the L V Prasad Eye Institute for supporting this study.

Funding Statement

The funders had no role in study design, data collection and interpretation, or the decision to submit the work for publication.

Contributor Information

Stephanie Badde, Email: stephanie.badde@nyu.edu.

Tamar R Makin, University College London, United Kingdom.

Barbara G Shinn-Cunningham, Carnegie Mellon University, United States.

Funding Information

This paper was supported by the following grants:

  • Deutsche Forschungsgemeinschaft BA 5600/1-1 to Stephanie Badde.

  • H2020 European Research Council ERC-2009-AdG 249425 CriticalBrainChanges to Brigitte Röder.

  • Deutsche Forschungsgemeinschaft DFG Ro 2625/10-1 to Brigitte Röder.

  • University of Hamburg postdoctoral fellowship to Stephanie Badde.

Additional information

Competing interests

No competing interests declared.

Author contributions

Data curation, Software, Formal analysis, Supervision, Visualization, Methodology, Writing - original draft, Writing - review and editing.

Software, Investigation, Methodology, Writing - review and editing.

Investigation, Writing - review and editing.

Data curation, Writing - review and editing.

Resources, Investigation, Writing - review and editing.

Conceptualization, Resources, Supervision, Funding acquisition, Investigation, Methodology, Writing - original draft, Writing - review and editing.

Ethics

Human subjects: All participants or, if applicable, their legal guardian, provided written informed consent before beginning the experiment. The study was conducted in accordance with the ethical guidelines of the Declaration of Helsinki and was approved by the ethical board of the German Psychological Society as well as the local ethical committee of the LV Prasad Eye Institute.

Additional files

Supplementary file 1. Supplementary statistical information.
elife-61238-supp1.docx (15.8KB, docx)
Transparent reporting form

Data availability

All data have been deposited on Open Science Framework under CQN48.

The following dataset was generated:

Badde S, Ley P, Rajendran SS, Kekunnaya R, Shareef I, Röder B. 2019. Cross-modal temporal biases emerge during early sensitive periods. Open Science Framework.

References

  1. Aghdaee SM, Battelli L, Assad JA. Relative timing: from behaviour to neurons. Philosophical Transactions of the Royal Society B: Biological Sciences. 2014;369:20120472. doi: 10.1098/rstb.2012.0472. [DOI] [PMC free article] [PubMed] [Google Scholar]
  2. Alais D, Carlile S. Synchronizing to real events: subjective audiovisual alignment scales with perceived auditory depth and speed of sound. PNAS. 2005;102:2244–2247. doi: 10.1073/pnas.0407034102. [DOI] [PMC free article] [PubMed] [Google Scholar]
  3. Alais D, Cass J. Multisensory perceptual learning of temporal order: audiovisual learning transfers to vision but not audition. PLOS ONE. 2010;5:e11283. doi: 10.1371/journal.pone.0011283. [DOI] [PMC free article] [PubMed] [Google Scholar]
  4. Badde S, Ley P, Rajendran SS, Shareef I, Kekunnaya R, Röder B. Resources of sensory experience during early sensitive periods shapes cross-modal temporal biases. Open Science Framework. 2019;1:CQN48. doi: 10.17605/OSF.IO/CQN48. [DOI] [PMC free article] [PubMed] [Google Scholar]
  5. Basharat A, Adams MS, Staines WR, Barnett-Cowan M. Simultaneity and temporal order judgments are coded differently and change with age: an event-related potential study. Frontiers in Integrative Neuroscience. 2018;12:15. doi: 10.3389/fnint.2018.00015. [DOI] [PMC free article] [PubMed] [Google Scholar]
  6. Brannon EM, Suanda S, Libertus K. Temporal discrimination increases in precision over development and parallels the development of numerosity discrimination. Developmental Science. 2007;10:770–777. doi: 10.1111/j.1467-7687.2007.00635.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
  7. Canales J. A Tenth of a Second: A History. University of Chicago Press; 2010. [DOI] [Google Scholar]
  8. Chen YC, Lewis TL, Shore DI, Maurer D. Early binocular input is critical for development of audiovisual but not visuotactile simultaneity perception. Current Biology. 2017;27:583–589. doi: 10.1016/j.cub.2017.01.009. [DOI] [PubMed] [Google Scholar]
  9. de Heering A, Dormal G, Pelland M, Lewis T, Maurer D, Collignon O. A brief period of postnatal visual deprivation alters the balance between auditory and visual attention. Current Biology. 2016;26:3101–3105. doi: 10.1016/j.cub.2016.10.014. [DOI] [PubMed] [Google Scholar]
  10. Engel GR, Dougherty WG. Visual-auditory distance constancy. Nature. 1971;234:308. doi: 10.1038/234308a0. [DOI] [PubMed] [Google Scholar]
  11. Fain GL. Sensory Transduction. Oxford University Press; 2019. [DOI] [Google Scholar]
  12. Freeman ED, Ipser A, Palmbaha A, Paunoiu D, Brown P, Lambert C, Leff A, Driver J. Sight and sound out of synch: fragmentation and renormalisation of audiovisual integration and subjective timing. Cortex. 2013;49:2875–2887. doi: 10.1016/j.cortex.2013.03.006. [DOI] [PMC free article] [PubMed] [Google Scholar]
  13. Fujisaki W, Shimojo S, Kashino M, Nishida S. Recalibration of audiovisual simultaneity. Nature Neuroscience. 2004;7:773–778. doi: 10.1038/nn1268. [DOI] [PubMed] [Google Scholar]
  14. Grabot L, van Wassenhove V. Time order as psychological bias. Psychological Science. 2017;28:670–678. doi: 10.1177/0956797616689369. [DOI] [PMC free article] [PubMed] [Google Scholar]
  15. Guerreiro MJ, Putzar L, Röder B. The effect of early visual deprivation on the neural bases of multisensory processing. Brain. 2015;138:1499–1504. doi: 10.1093/brain/awv076. [DOI] [PMC free article] [PubMed] [Google Scholar]
  16. Guerreiro MJ, Putzar L, Röder B. The effect of early visual deprivation on the neural bases of auditory processing. The Journal of Neuroscience. 2016;36:1620–1630. doi: 10.1523/JNEUROSCI.2559-15.2016. [DOI] [PMC free article] [PubMed] [Google Scholar]
  17. Hillock AR, Powers AR, Wallace MT. Binding of sights and sounds: age-related changes in multisensory temporal processing. Neuropsychologia. 2011;49:461–467. doi: 10.1016/j.neuropsychologia.2010.11.041. [DOI] [PMC free article] [PubMed] [Google Scholar]
  18. Hirsh IJ, Sherrick CE. Perceived order in different sense modalities. Journal of Experimental Psychology. 1961;62:423–432. doi: 10.1037/h0045283. [DOI] [PubMed] [Google Scholar]
  19. Hume D. A Treatise of Human Nature. Courier Corporation; 2012. [Google Scholar]
  20. Hyvärinen J, Carlson S, Hyvärinen L. Early visual deprivation alters modality of neuronal responses in area 19 of monkey cortex. Neuroscience Letters. 1981;26:239–243. doi: 10.1016/0304-3940(81)90139-7. [DOI] [PubMed] [Google Scholar]
  21. Ipser A, Agolli V, Bajraktari A, Al-Alawi F, Djaafara N, Freeman ED. Sight and sound persistently out of synch: stable individual differences in audiovisual synchronisation revealed by implicit measures of lip-voice integration. Scientific Reports. 2017;7:46413. doi: 10.1038/srep46413. [DOI] [PMC free article] [PubMed] [Google Scholar]
  22. Ipser A, Karlinski M, Freeman ED. Correlation of individual differences in audiovisual asynchrony across stimuli and tasks: new constraints on temporal renormalization theory. Journal of Experimental Psychology: Human Perception and Performance. 2018;44:1283–1293. doi: 10.1037/xhp0000535. [DOI] [PubMed] [Google Scholar]
  23. Keetels M, Vroomen J. Temporal recalibration to tactile-visual asynchronous stimuli. Neuroscience Letters. 2008;430:130–134. doi: 10.1016/j.neulet.2007.10.044. [DOI] [PubMed] [Google Scholar]
  24. Keetels M, Vroomen J. Perception of synchrony between the senses. In: Murray MM, Wallace MT, editors. The Neural Bases of Multisensory Processes. CRC Press/Taylor & Francis; 2012. pp. 147–177. [Google Scholar]
  25. King AJ. Multisensory integration: strategies for synchronization. Current Biology. 2005;15:R339–R341. doi: 10.1016/j.cub.2005.04.022. [DOI] [PubMed] [Google Scholar]
  26. Knudsen EI. Instructed learning in the auditory localization pathway of the barn owl. Nature. 2002;417:322–328. doi: 10.1038/417322a. [DOI] [PubMed] [Google Scholar]
  27. Knudsen EI. Sensitive periods in the development of the brain and behavior. Journal of Cognitive Neuroscience. 2004;16:1412–1425. doi: 10.1162/0898929042304796. [DOI] [PubMed] [Google Scholar]
  28. Kopinska A, Harris LR. Simultaneity constancy. Perception. 2004;33:1049–1060. doi: 10.1068/p5169. [DOI] [PubMed] [Google Scholar]
  29. Lewkowicz DJ. Perception of auditory-visual temporal synchrony in human infants. Journal of Experimental Psychology: Human Perception and Performance. 1996;22:1094–1106. doi: 10.1037/0096-1523.22.5.1094. [DOI] [PubMed] [Google Scholar]
  30. Lewkowicz DJ. The development of intersensory temporal perception: an epigenetic systems/limitations view. Psychological Bulletin. 2000;126:281–308. doi: 10.1037/0033-2909.126.2.281. [DOI] [PubMed] [Google Scholar]
  31. Lewkowicz DJ. Development of Multisensory Temporal Perception. In: Murray MM, Wallace MT, editors. The Neural Bases of Multisensory Processes. CRC Press/Taylor & Francis; 2012. pp. 325–343. [Google Scholar]
  32. Lewkowicz DJ, Flom R. The audiovisual temporal binding window narrows in early childhood. Child Development. 2014;85:685–694. doi: 10.1111/cdev.12142. [DOI] [PMC free article] [PubMed] [Google Scholar]
  33. Linares D, Holcombe AO. Differences in perceptual latency estimated from judgments of temporal order, simultaneity and duration are inconsistent. I-Perception. 2014;5:559–571. doi: 10.1068/i0675. [DOI] [PMC free article] [PubMed] [Google Scholar]
  34. Maurer D, Lewis TL, Mondloch CJ. Missing sights: consequences for visual cognitive development. Trends in Cognitive Sciences. 2005;9:144–151. doi: 10.1016/j.tics.2005.01.006. [DOI] [PubMed] [Google Scholar]
  35. McCulloch DL, Skarf B. Pattern reversal visual evoked potentials following early treatment of unilateral, congenital cataract. Archives of Ophthalmology. 1994;112:510–518. doi: 10.1001/archopht.1994.01090160086026. [DOI] [PubMed] [Google Scholar]
  36. Mollon JD, Perkins AJ. Errors of judgement at Greenwich in 1796. Nature. 1996;380:101–102. doi: 10.1038/380101a0. [DOI] [PubMed] [Google Scholar]
  37. Mondloch CJ, Segalowitz SJ, Lewis TL, Dywan J, Le Grand R, Maurer D. The effect of early visual deprivation on the development of face detection. Developmental Science. 2013;16:728–742. doi: 10.1111/desc.12065. [DOI] [PubMed] [Google Scholar]
  38. Noel JP, De Niear M, Van der Burg E, Wallace MT. Audiovisual simultaneity judgment and rapid recalibration throughout the lifespan. PLOS ONE. 2016;11:e0161698. doi: 10.1371/journal.pone.0161698. [DOI] [PMC free article] [PubMed] [Google Scholar]
  39. Parise CV, Ernst MO. Correlation detection as a general mechanism for multisensory integration. Nature Communications. 2016;7:11543. doi: 10.1038/ncomms11543. [DOI] [PMC free article] [PubMed] [Google Scholar]
  40. Pöppel E. Mindworks: Time and Conscious Experience. Harcourt Brace Jovanovich; 1988. [Google Scholar]
  41. Pöppel E. A hierarchical model of temporal perception. Trends in Cognitive Sciences. 1997;1:56–61. doi: 10.1016/S1364-6613(97)01008-5. [DOI] [PubMed] [Google Scholar]
  42. Putzar L, Goerendt I, Lange K, Rösler F, Röder B. Early visual deprivation impairs multisensory interactions in humans. Nature Neuroscience. 2007;10:1243–1245. doi: 10.1038/nn1978. [DOI] [PubMed] [Google Scholar]
  43. Putzar L, Gondan M, Röder B. Basic multisensory functions can be acquired after congenital visual pattern deprivation in humans. Developmental Neuropsychology. 2012;37:697–711. doi: 10.1080/87565641.2012.696756. [DOI] [PubMed] [Google Scholar]
  44. Röder B, Rösler F, Spence C. Early vision impairs tactile perception in the blind. Current Biology. 2004;14:121–124. doi: 10.1016/j.cub.2003.12.054. [DOI] [PubMed] [Google Scholar]
  45. Röder B, Kusmierek A, Spence C, Schicke T. Developmental vision determines the reference frame for the multisensory control of action. PNAS. 2007;104:4753–4758. doi: 10.1073/pnas.0607158104. [DOI] [PMC free article] [PubMed] [Google Scholar]
  46. Röder B, Pagel B, Heed T. The implicit use of spatial information develops later for crossmodal than for intramodal temporal processing. Cognition. 2013;126:301–306. doi: 10.1016/j.cognition.2012.09.009. [DOI] [PubMed] [Google Scholar]
  47. Roseboom W, Linares D, Nishida S. Sensory adaptation for timing perception. Proceedings of the Royal Society B: Biological Sciences. 2015;282:20142833. doi: 10.1098/rspb.2014.2833. [DOI] [PMC free article] [PubMed] [Google Scholar]
  48. Rosenthal R, Rubin DB. R equivalent: a simple effect size Indicator. Psychological Methods. 2003;8:492–496. doi: 10.1037/1082-989X.8.4.492. [DOI] [PubMed] [Google Scholar]
  49. Sanford EC. Personal equation. The American Journal of Psychology. 1888;2:3–38. doi: 10.2307/1411405. [DOI] [Google Scholar]
  50. Shi Z, Burr D. Predictive coding of multisensory timing. Current Opinion in Behavioral Sciences. 2016;8:200–206. doi: 10.1016/j.cobeha.2016.02.014. [DOI] [PMC free article] [PubMed] [Google Scholar]
  51. Shore DI, Spence C, Klein RM. Visual prior entry. Psychological Science. 2001;12:205–212. doi: 10.1111/1467-9280.00337. [DOI] [PubMed] [Google Scholar]
  52. Sourav S, Bottari D, Kekunnaya R, Röder B. Evidence of a retinotopic organization of early visual cortex but impaired extrastriate processing in sight recovery individuals. Journal of Vision. 2018;18:22. doi: 10.1167/18.3.22. [DOI] [PubMed] [Google Scholar]
  53. Sourav S, Kekunnaya R, Shareef I, Banerjee S, Bottari D, Röder B. A protracted sensitive period regulates the development of cross-modal sound-shape associations in humans. Psychological Science. 2019;30:1473–1482. doi: 10.1177/0956797619866625. [DOI] [PMC free article] [PubMed] [Google Scholar]
  54. Spence C, Shore DI, Klein RM. Multisensory prior entry. Journal of Experimental Psychology: General. 2001;130:799–832. doi: 10.1037/0096-3445.130.4.799. [DOI] [PubMed] [Google Scholar]
  55. Spence C. Prior entry: attention and temporal perception. In: Nobre A. C, Coull J. T, editors. Attention and Time. Oxford University Press; 2010. pp. 89–104. [Google Scholar]
  56. Spence C, Squire S. Multisensory integration: maintaining the perception of synchrony. Current Biology. 2003;13:R519–R521. doi: 10.1016/S0960-9822(03)00445-7. [DOI] [PubMed] [Google Scholar]
  57. Stromeyer CF, Martini P. Human temporal impulse response speeds up with increased stimulus contrast. Vision Research. 2003;43:285–298. doi: 10.1016/S0042-6989(02)00412-1. [DOI] [PubMed] [Google Scholar]
  58. Titchener EB. Lectures on the elementary psychology of feeling and attention. Macmillan; 1908. [Google Scholar]
  59. van Eijk RL, Kohlrausch A, Juola JF, van de Par S. Audiovisual synchrony and temporal order judgments: effects of experimental method and stimulus type. Perception & Psychophysics. 2008;70:955–968. doi: 10.3758/PP.70.6.955. [DOI] [PubMed] [Google Scholar]
  60. Vercillo T, Burr D, Sandini G, Gori M. Children do not recalibrate motor-sensory temporal order after exposure to delayed sensory feedback. Developmental Science. 2015;18:703–712. doi: 10.1111/desc.12247. [DOI] [PMC free article] [PubMed] [Google Scholar]
  61. Vibell J, Klinge C, Zampini M, Spence C, Nobre AC. Temporal order is coded temporally in the brain: early event-related potential latency shifts underlying prior entry in a cross-modal temporal order judgment task. Journal of Cognitive Neuroscience. 2007;19:109–120. doi: 10.1162/jocn.2007.19.1.109. [DOI] [PubMed] [Google Scholar]
  62. Vroomen J, Keetels M, de Gelder B, Bertelson P. Recalibration of temporal order perception by exposure to audio-visual asynchrony. Cognitive Brain Research. 2004;22:32–35. doi: 10.1016/j.cogbrainres.2004.07.003. [DOI] [PubMed] [Google Scholar]
  63. Watson AB. Temporal sensitivity. In: Boff K, Thomas J, editors. Handbook of Perception and Human Performance. Wiley; 1986. pp. 6.1–6.6. [Google Scholar]
  64. Yamauchi Y, Mochizuki JI, Hirakata A, Uda S. Single flash electroretinograms of mature cataractous and fellow eyes. Clinical Ophthalmology. 2016;10:2031–2034. doi: 10.2147/OPTH.S118677. [DOI] [PMC free article] [PubMed] [Google Scholar]
  65. Zampini M, Shore DI, Spence C. Audiovisual temporal order judgments. Experimental Brain Research. 2003;152:198–210. doi: 10.1007/s00221-003-1536-z. [DOI] [PubMed] [Google Scholar]
  66. Zampini M, Guest S, Shore DI, Spence C. Audio-visual simultaneity judgments. Perception & Psychophysics. 2005;67:531–544. doi: 10.3758/BF03193329. [DOI] [PubMed] [Google Scholar]

Decision letter

Editor: Tamar R Makin1
Reviewed by: Tamar R Makin2

In the interests of transparency, eLife publishes the most substantive revision requests and the accompanying author responses.

Acceptance summary:

In this study the authors compare congenital and developmental cataract survivors to typically developed individuals, to study the influence of early life experience on multisensory processing. It is this double comparison that allows them to suggest that perceptual biases develop in early life. We were most excited about the inclusion of the late-cataract control group, as well as the rich dataset, which includes replication of the findings during visuo-tactile and visuo-auditory multisensory processing. We believe this unique dataset will set a new bar for retrospective behavioural research of sensitive periods.

Decision letter after peer review:

[Editors’ note: the authors submitted for reconsideration following the decision after peer review. What follows is the decision letter after the first round of review.]

Thank you for submitting your work entitled "Cross-modal temporal biases emerge during early sensitive periods" for consideration by eLife. Your article has been reviewed by three peer reviewers, including Tamar R Makin as the Reviewing Editor and Reviewer #1, and the evaluation has been overseen by a Senior Editor.

Our decision has been reached after consultation between the reviewers. Based on these discussions and the individual reviews below, we regret to inform you that your work, as currently presented, will not be considered further for publication in eLife.

All reviewers were very excited about the general experimental approach involving comparisons of congenital versus developmental cataract survivors for studying the influence of early life experience on multisensory processing. The reviewers agreed that the presented findings were intriguing. However, even when considering the unique challenges of data collection with these precious populations, it felt that the interpretation of the findings was overly simplistic. In particular, they felt that several alternative accounts need to be better considered in order to interpret these findings more mechanistically. It was agreed that further experimental evidence will need to be presented in order to tease apart these alternative accounts, and in particular the role of attention. If a more comprehensive account for this data could be offered, then we will be very happy to consider a resubmission of the manuscript. Please note that the reviewers expressed an interest in the alternatives provided for interpreting the results, so they do not necessarily expect that these will be discounted as confounds. Instead, they are looking for more conclusive evidence for interpreting the mechanisms driving the findings. We leave it to the authors to decide what type of evidence could be raised to best address the reviewers issues, be it new experiments acquired from control participants, more detailed analysis of the existing data or presentation of additional data collected with the cataract groups. If you wish to resubmit your work after substantial revisions, please provide a point-by-point response to address each of the reviewers major comments. Of course, we understand should you wish to take this present version elsewhere at this time.

Reviewer #1:

I think it’s a very exciting dataset and the results are quite interesting. I also liked the main interpretation of the results. But at present, this all feels a little too speculative/tentative. I feel like important opportunities in analysis of this precious data have been missed, and that at present we are only provided a very partial glims at the results.

1) I'm not entirely clear about the choices for the statistical analysis, particularly with respect to the “bias” parameter. To begin with, why did they chose a hierarchical logistic regression? Why is it advantageous to, say, signal detection analysis which is often used to study biases? It is also not clear how the ISI parameter is handled in the model

2) considering the relative richness of the data collected, the analysis is quite simplistic and leaves something to be desired. Why did they not follow standard analysis procedures based on the ISI in their design (i.e. psychometric function, calculation of the slope/JND and point of subjective equality)? Considering the tendency to report “visual first” could be modulated by several dissociated factors (as elaborated in the Discussion), I feel like the traditional parameters could help interpret the described bias better. For example, attentional effects could be hypothesised to be impacting all ISI similarly (a shift of the curve), whereas the experience-based structural differences should be more time-dependent, based on biological onset asynchrony (i.e. shifting the slope)?

3) Perhaps the most interesting comparison is between the DC and CC groups, especially considering methodological and technical differences between the cataract groups and the controls. But this wasn't reported in the results? From the figures it appears that biases were indeed different (?) but it also seems that they might sustain different visual and tactile abilities (Figure 1D)? How would that potentially impact the main results?

4) The MDC group didn’t show the typical visual bias effect, this is a risk when using small sample sizes, and thus requires some consideration. Was this group not a representative control group? Again, based on Figure 1C it indeed appears that their performance at the bimanual condition was low. I appreciate that the interpretation of the DC group stands alone, by showing a bias effect in the vision direction (and opposite from the CC group). But still, this should be considered. Please don't address this question by showing no significant differences between the MDC and MCC groups, unless there are clear evidence to support this as lack of a difference (e.g. bayesian statistics)

Reviewer #2:

Badde and colleagues presented a cross-modal temporal task (which stimulus was presented first) to controls, individuals born with dense, bilateral cataracts whose sight was restored after birth (CC), and those who developed cataracts after childbirth (DC). They found that those born with cataracts that were subsequently removed (the CC group) were more likely to respond visual 1st (vs. auditory or tactile) compared to the DC and matched control groups. The results are intriguing. However, there are major concerns with regards to the interpretation that diminish enthusiasm for these findings.

The authors claim that "cross-modal perceptual biases are not innate but rather acquired during a sensitive period." My primary question is whether the observed effects are truly "cross-modal perceptual biases". An alternative hypothesis is based on the "law of prior entry", in which "the object of attention comes to consciousness more quickly than the objects which we are not attending to" (Titchener, 1908). The authors are aware of this, citing articles that show biases in temporal order when attending unequally across modalities (Titchener, 1908; Shore, Spence and Klein, 2001; Spence, Shore and Klein, 2001; Vibell et al., 2007). They then argue that (a) "attention has been ruled out as a cause of persistent biases in cross-modal spatiotemporal processing" and (b) "it is not obvious why CC- and DC-individuals would entertain opposing attentional foci given that both groups experienced transient severe visual impairment and suffer from remaining visual acuity impairments." For (a), I am not sure how their citation has ruled out attention as a cause of these biases from their Abstract: "we showed that temporal-order perception may be considered a psychological bias that attention can modulate but not fully eradicate". For (b), the argument would be that those w/o high-quality visual information at birth (CC) would need to attend more to the visual stimulus due to deficits/differences in their visual systems at birth, whereas those in DC group wouldn't need to, given that they were born with intact visual input.

This is not to say that the author's hypothesis or this alternative hypothesis aren't interesting, but more data would be needed to tease these apart. This leads to my second issue, the paper is lacking in mechanism. The Introduction (especially for the lay reader) provides minimal information about why this bias occurs, and why this matters. The paper could be improved substantially with a more mechanistic rationale, especially in the Introduction.

Finally, the authors note that the visual first bias in the CC group did not significantly correlate with duration of visual deprivation in CC group. Is this meaningful given the small number of subjects tested? It seems like there isn't enough evidence, as presented, to make this claim.

Reviewer #3:

The authors investigate cross-modal temporal perceptual biases in participants born with congenital bilateral cataract and treated on different ages. They find that in this population, the biases are opposite to healthy and developmental cataract populations. They conclude such biases develop in early sensitive period. The paper is intriguing and the results are important, however, it is to my view that the interpretation of the results and the main conclusion might be problematic.

1) The authors main point is trying to rule out the possibility of innate cross-modal perceptual biases. The option of those biases being innate might still be possible, but biases might be reversed due to lack of visual input at early age while the brain is still flexible enough. This possibility should be mentioned. So, throughout the paper, I believe the claim of lack of innateness should be transform to that of sensitive period and innateness should only be discussed as a possibility in the Discussion. If there are any developmental studies showing the emergence of the biases during the first few months after birth, they should be mentioned since they can support the authors' claim. The word "emerge" at the heading should also be change to something more accurate.

2) In the Introduction: "Moreover, it is not obvious why CC- and DC-individuals would entertain opposing attentional foci given that both groups experienced transient severe visual impairment and suffer from remaining visual acuity impairments." The authors cannot use this claim to rule out attentional foci since it is also not obvious why CC and DC individuals will have an opposing cross-modal effect as well (this study demonstrates this exact result). Attention could be opposite for the same unknown reason. I can think of one reason why CC and DC groups will entertain opposing attentional foci. Developmental individuals were used to having high resolution input, now resolution decreased to medium so they are less attentive to vision. CCs were used to having low resolution input, now resolution increased to medium, so they are more attentive. This could be one possible reason why both groups will have opposing results in visual tests.

3) DC group differed from the control group in the visuo-tactile experiment (with a trend in the visuo-auditory). Doesn't it mean that the sensitive period for the cross-modal perceptual biases is very long? Since those biases were altered (increased, not reversed) at a relatively late age? At least, this difference should be acknowledged in the Discussion acknowledging that there is no theory justifying this effect.

4) Overall, following my previous comments, I think one can only conclude that cross-modal biases are sensitive to visual experience. If a stronger claim is needed, it should take into account both CC and DC groups' results.

eLife. 2020 Aug 25;9:e61238. doi: 10.7554/eLife.61238.sa2

Author response

[Editors’ note: the authors resubmitted a revised version of the paper for consideration. What follows is the authors’ response to the first round of review.]

Reviewer #1:

I think it’s a very exciting dataset and the results are quite interesting. I also liked the main interpretation of the results. But at present, this all feels a little too speculative/tentative. I feel like important opportunities in analysis of this precious data have been missed, and that at present we are only provided a very partial glims at the results.

1) I'm not entirely clear about the choices for the statistical analysis, particularly with respect to the “bias” parameter. To begin with, why did they chose a hierarchical logistic regression? Why is it advantageous to, say, signal detection analysis which is often used to study biases? It is also not clear how the ISI parameter is handled in the model

Our choice for an analysis with generalized linear mixed models was based on the sample size and composition. Even though large for this patient group, the sample size is small when compared to studies in healthy volunteers. And, though the participants were highly homogenous with respect to their visual history, they considerably varied in other parameters that are known to influence temporal order perception such as age. Mixed models are especially suited for such a situation because participants are included as random effects in the model. We have added this in the revised manuscript (subsection “Data Analysis”).

The bias parameter indicates the probability of a “visual first”-response across stimulus conditions. It is identical to the criterion in signal detection theory (c = (hit rate + false alarm rate)/2). In the revised manuscript, we now point this relation out (subsection “Data Analysis”).

The time interval between the stimuli was not included in this analysis because generalized mixed models require a considerable number of trials and this demand increases non-linearly with each additional factor. We now report an additional analysis based on the psychometric function (see below). Please note, an additional advantage of the original analysis is its robustness against outliers in the psychometric functions, which arise due to constraints in the number of trials that are inevitable in this type of research. (Several of the patients recruited in India were children. They simply cannot complete the same number of trials and with the same degree of concentration as university students who are used to computers or even psychological experiments). Thus, in the revised manuscript we report both analyses.

2) Considering the relative richness of the data collected, the analysis is quite simplistic and leaves something to be desired. Why did they not follow standard analysis procedures based on the ISI in their design (i.e. psychometric function, calculation of the slope/JND and point of subjective equality)? Considering the tendency to report “visual first” could be modulated by several dissociated factors (as elaborated in the Discussion), I feel like the traditional parameters could help interpret the described bias better. For example, attentional effects could be hypothesised to be impacting all ISI similarly (a shift of the curve), whereas the experience-based structural differences should be more time-dependent, based on biological onset asynchrony (i.e. shifting the slope)?

We have added a classical analysis of the point of subjective simultaneity; the results match those derived with the original bias measure, thus validating the latter.

The psychometric curves were shifted (Figure 1B); thus, the effect is of similar size across all short SOAs. This pattern is consistent with an auditory/tactile delay built into the neural circuits underlying temporal order judgments as well as with an attention-mediated visual processing gain following congenital loss of vision.

We analyzed both measures, the PSS and the original bias parameter, as the former is more precise but the latter more robust. In the revised manuscript, we moreover investigated the proportion correct and additionally report JNDs but only as a descriptive measure. Our JND estimates suffer from the fact that we sampled many short but only one long SOA, a design which was tailored to gather best-possible insights into the bias (subsection “Data Analysis”).

3) Perhaps the most interesting comparison is between the DC and CC groups, especially considering methodological and technical differences between the cataract groups and the controls. But this wasn't reported in the results? From the figures it appears that biases were indeed different (?) but it also seems that they might sustain different visual and tactile abilities (Figure 1D)? How would that potentially impact the main results?

We compared CC- and DC-groups each to their controls but not to each other because of the large age difference between groups (Table 1), a variable known to influence cross-modal temporal perception. The only way to investigate whether a difference between these groups is due to their different history of visual deprivation or their age is the comparison to an age matched healthy control group. The finding that both group comparisons revealed opposing effects, provides strong evidence for the role of vision following birth and against a role of unspecific factors such as the location of testing (the equipment was identical) for the reported group differences. We now describe this reasoning in further detail (Results paragraph one).

Our data indicate that DC-individuals show an impaired auditory and tactile resolution compared to their controls whereas these effects did not reach significance for the comparisons between CC-individuals and their normally sighted controls. Both groups differed from their controls with respect to bimodal and visual resolution (Results paragraph two).

We analyzed the correlation between resolution and bias. Participants with a lower temporal resolution relied more on temporal biases (Results paragraph three). This is reminiscent of the Bayesian framework in which the influence of the prior increases with decreasing reliability of the likelihood and makes intuitive sense: the less reliable the information extracted from the stimulus is, the more the brain relies on the readily available bias. This mechanism provides a potential explanation for the larger biases in DC-individuals compared to their controls (Discussion paragraph six). However, it cannot explain the main effect, the reversed direction of the bias in CC-individuals.

4) The MDC group didnt show the typical visual bias effect, this is a risk when using small sample sizes, and thus requires some consideration. Was this group not a representative control group? Again, based on Figure 1C it indeed appears that their performance at the bimanual condition was low. I appreciate that the interpretation of the DC group stands alone, by showing a bias effect in the vision direction (and opposite from the CC group). But still, this should be considered. Please don't address this question by showing no significant differences between the MDC and MCC groups, unless there are clear evidence to support this as lack of a difference (e.g. bayesian statistics)

The MDC-group did not show a significant bias in the proportion of “visual first” responses, yet, this does not allow us to conclude that they did not have a bias. Indeed, a closer look at the data suggests that the available power resulting from small sample sizes might play a role here. The apparent group difference is driven by one respectively two participants, which in a larger sample would not influence the statistical outcomes.

The sample sizes are too small to calculate reliable Bayes factors and collecting more participants for the control groups renders the comparison to the cataract-reversal groups problematic as the signal-to-noise ratio becomes skewed with unequal group sizes. We now show the distribution of biases across participants of each group using box-and-whisker plots (Figure 1D,E). This allows readers to convince themselves visually that both control samples show similar biases.

Finally, as stated in the comment, the key point for our study is the relation between the bias in the cataract-reversal groups compared to their matched controls.

Reviewer #2:

Badde and colleagues presented a cross-modal temporal task (which stimulus was presented first) to controls, individuals born with dense, bilateral cataracts whose sight was restored after birth (CC), and those who developed cataracts after childbirth (DC). They found that those born with cataracts that were subsequently removed (the CC group) were more likely to respond visual 1st (vs. auditory or tactile) compared to the DC and matched control groups. The results are intriguing. However, there are major concerns with regards to the interpretation that diminish enthusiasm for these findings.

The authors claim that "cross-modal perceptual biases are not innate but rather acquired during a sensitive period." My primary question is whether the observed effects are truly "cross-modal perceptual biases". An alternative hypothesis is based on the "law of prior entry", in which "the object of attention comes to consciousness more quickly than the objects which we are not attending to" (Titchener, 1908). The authors are aware of this, citing articles that show biases in temporal order when attending unequally across modalities (Titchener, 1908; Shore, Spence and Klein, 2001; Spence, Shore and Klein, 2001; Vibell et al., 2007). They then argue that (a) "attention has been ruled out as a cause of persistent biases in cross-modal spatiotemporal processing" and (b) "it is not obvious why CC- and DC-individuals would entertain opposing attentional foci given that both groups experienced transient severe visual impairment and suffer from remaining visual acuity impairments." For (a), I am not sure how their citation has ruled out attention as a cause of these biases from their Abstract: "we showed that temporal-order perception may be considered a psychological bias that attention can modulate but not fully eradicate". For (b), the argument would be that those w/o high-quality visual information at birth (CC) would need to attend more to the visual stimulus due to deficits/differences in their visual systems at birth, whereas those in DC group wouldn't need to, given that they were born with intact visual input.

We now discuss the possibility that congenital transient loss of pattern vision could lead to attentional biases that differ those of from normally sighted individuals and individuals with a later period of transient visual deprivation (Discussion paragraph five). In our (revised) view, the two accounts of the effect are not necessarily mutually exclusive: the pattern of sensory experience available at birth might calibrate modality-specific attentional biases as well as the underlying neural circuits recruited for temporal order perception.

To conclusively determine the role of attention for the observed reversal of cross-modal temporal biases, extensive new studies at the LV Prasad Eye Institute would be necessary. In order to collect data from a reasonable number of CC-individuals we would need 1-2 years under typical conditions. Participants for our studies are extremely difficult to recruit because we use strict selection criteria to maximize the likelihood that the tested CC-individuals were indeed born without any pattern vision. Currently under the condition of the pandemic it is unknown when we will be able to return to a typical testing schedule. Most of our participants come from rural areas and travel several hours in a bus to Hyderabad.

However, we think that our results are interesting irrespective of the role of attention for the effect. We can say for sure that whichever mechanism accounts for the reversal of cross-modal biases in CC-individuals, the perceived relative timing across modalities depends on early sensory (visual) experience during a sensitive period rather than, for instance, on visual experience later in life or on the current visual abilities. Thus, in the revised manuscript we relate our results to previously published studies, which seem to rather point against a shift of modality-specific attention towards vision in CC-individuals, but in the end due to the scarcity of studies directly targeting attention leave the possibility for such a mechanism open.

This is not to say that the author's hypothesis or this alternative hypothesis aren't interesting, but more data would be needed to tease these apart. This leads to my second issue, the paper is lacking in mechanism. The Introduction (especially for the lay reader) provides minimal information about why this bias occurs, and why this matters. The paper could be improved substantially with a more mechanistic rationale, especially in the Introduction.

We have rewritten large parts of the manuscript. The revised Introduction provides additional information about cross-modal temporal perception and describes mechanistic accounts of cross-modal temporal biases. The revised Discussion is more explicit about the mechanism which we consider being responsible for the reversed cross-modal bias in CC-individuals, and we now relate this “compensatory mechanism” to additional factors such as attention and response biases.

Finally, the authors note that the visual first bias in the CC group did not significantly correlate with duration of visual deprivation in CC group. Is this meaningful given the small number of subjects tested? It seems like there isn't enough evidence, as presented, to make this claim.

We have removed the correlational analysis.

Reviewer #3:

The authors investigate cross-modal temporal perceptual biases in participants born with congenital bilateral cataract and treated on different ages. They find that in this population, the biases are opposite to healthy and developmental cataract populations. They conclude such biases develop in early sensitive period. The paper is intriguing and the results are important, however, it is to my view that the interpretation of the results and the main conclusion might be problematic.

1) The authors main point is trying to rule out the possibility of innate cross-modal perceptual biases. The option of those biases being innate might still be possible, but biases might be reversed due to lack of visual input at early age while the brain is still flexible enough. This possibility should be mentioned. So, throughout the paper, I believe the claim of lack of innateness should be transform to that of sensitive period and innateness should only be discussed as a possibility in the discussion. If there are any developmental studies showing the emergence of the biases during the first few months after birth, they should be mentioned since they can support the authors' claim. The word "emerge" at the heading should also be change to something more accurate.

A stated in this comment, theoretically innate biases could be reversed during a sensitive period. Since we agree with Lewkowicz (2011, Infancy) that research on developmental principles should not be narrowed down to the simplistic dichotomy of nature – nurture, but rather investigate how experience dynamically contributes to brain development, we more precisely argue along these lines throughout the revised manuscript.

We now report findings which suggest that very young infants already show biases in the perception of cross-modal synchrony (Discussion paragraph three) and discuss parallels between the typical development of multisensory temporal perception and multisensory temporal perception in individuals with reversed congenital cataracts (paragraph nine).

We have changed the title of our manuscript.

2) In the Introduction: "Moreover, it is not obvious why CC- and DC-individuals would entertain opposing attentional foci given that both groups experienced transient severe visual impairment and suffer from remaining visual acuity impairments." The authors cannot use this claim to rule out attentional foci since it is also not obvious why CC and DC individuals will have an opposing cross-modal effect as well (this study demonstrates this exact result). Attention could be opposite for the same unknown reason. I can think of one reason why CC and DC groups will entertain opposing attentional foci. Developmental individuals were used to having high resolution input, now resolution decreased to medium so they are less attentive to vision. CCs were used to having low resolution input, now resolution increased to medium, so they are more attentive. This could be one possible reason why both groups will have opposing results in visual tests.

We now discuss the possibility that congenital loss of pattern vision leads to increased attention towards that modality (Discussion paragraph four).

3) DC group differed from the control group in the visuo-tactile experiment (with a trend in the visuo-auditory). Doesn't it mean that the sensitive period for the cross-modal perceptual biases is very long? Since those biases were altered (increased, not reversed) at a relatively late age? At least, this difference should be acknowledged in the Discussion acknowledging that there is no theory justifying this effect.

An increased bias towards perceiving vision as delayed in DC-individuals is incompatible with the idea of an extended sensitive period: if an extended sensitive period existed, the group difference between the DC- and the MDC-group should go in the same direction as that between the CC- and the MCC-group (since the DC- and CC-group share similarly altered experiences). Therefore, we hypothesize that the opposite cross-modal bias in the DC- compared to the CC-group fits well with the idea of a short sensitive period.

Unlike the normally sighted control group, DC-individuals might experience residual delays of visual processing for which the brain never learned to compensate because the sensitive period had closed early. An additional explanation for their increased cross-modal bias in the typical direction might be that DC-individuals rely stronger on typical biases because they experience greater temporal uncertainty. This idea is compatible with a common finding in individuals with restored hearing: Late deaf cochlear implant recipients show enhanced use of multisensory cues (Rouger et al., 2007, PNAS) while congenitally deaf cochlear implant recipients show no or impaired use of multisensory cues if the implants were received late (Schorr et al., 2005, PNAS). We now discuss this account in the manuscript (Discussion paragraph six).

4) Overall, following my previous comments, I think one can only conclude that cross-modal biases are sensitive to visual experience. If a stronger claim is needed, it should take into account both CC and DC groups' results.

We agree and now interpret the results as indicator for a sensitive period.

Associated Data

    This section collects any data citations, data availability statements, or supplementary materials included in this article.

    Data Citations

    1. Badde S, Ley P, Rajendran SS, Kekunnaya R, Shareef I, Röder B. 2019. Cross-modal temporal biases emerge during early sensitive periods. Open Science Framework. [DOI] [PMC free article] [PubMed]

    Supplementary Materials

    Supplementary file 1. Supplementary statistical information.
    elife-61238-supp1.docx (15.8KB, docx)
    Transparent reporting form
    elife-61238-transrepform.docx (246.6KB, docx)

    Data Availability Statement

    All data have been deposited on Open Science Framework under CQN48.

    The following dataset was generated:

    Badde S, Ley P, Rajendran SS, Kekunnaya R, Shareef I, Röder B. 2019. Cross-modal temporal biases emerge during early sensitive periods. Open Science Framework.

    Articles from eLife are provided here courtesy of eLife Sciences Publications, Ltd

    RESOURCES