Task-dependent audiovisual temporal sensitivity is not affected by stimulus intensity levels

Abstract

Flexibility and robustness of multisensory temporal recalibration is paramount for maintaining perceptual constancy of the surrounding natural world. Different environments impart various impediments, distances and routes that alter the propagation times of sight and sound cues comprising a multimodal event. One’s ability to rapidly calibrate and account for these external variations allows for maintained perception of synchrony which is crucial for coherent and consistent perception. The two common paradigms used to compare precision of temporal processing between experimental and control groups, the simultaneity judgment (SJ) and temporal order judgment (TOJ) tasks, often use supra-threshold stimuli. However, few studies have specifically examined the effects of normalizing stimulus intensities to participant’s unisensory detection thresholds. The current project presented multiple combinations of auditory and visual stimulus intensity levels, based on individual detection thresholds, during a TOJ and a SJ task. While no effect of stimulus intensity was found on temporal sensitivity or perceived temporal synchrony, there was a significant difference in point of subjective simultaneity (PSS) measures between tasks. In addition, PSS estimates were audio-leading, rather than visual-leading as previously reported, suggesting that exposure to the particular combinations of stimulus intensity levels used influenced temporal synchrony perception. Overall, these results support the use of supra-threshold stimuli in TOJ and SJ tasks as a way of minimizing the confound from differences in unisensory processing.

1. Introduction

Our ability to react and adapt to the external environment is critical for precise perception and subsequent responses and actions. A classic example is when we attend a baseball game and notice the extreme delay between our visual perception of the bat making contact with the ball and our auditory perception of the sound generated by that contact. However, after just a few pitches, our conscious awareness of this temporal delay is eliminated, and we perceive the visual and auditory features of this event as simultaneous. This type of short-term temporal recalibration is inherent to multisensory processing and allows for precision and flexibility in navigating the environment. Further, it enables individuals to maintain consistency in their perception of synchrony, a sensation that is built from their prior experience with and exposure to natural asynchronies of multisensory stimuli (Aschersleben & Prinz, 1995). For instance, observers generally display a bias toward visual-leading stimuli due to the naturally faster propagation times of light compared to sound (Bedard & Barnett-Cowan, 2016; Keetels & Vroomen, 2012). In addition, the temporal discrepancy required between two signals to perceive synchrony are not correlated between modalities, presumably due to the contextual factors dictating simultaneous perception (Scurry et al., 2019).

The point of subjective simultaneity (PSS) is the standard estimate used to reflect perceptual synchrony and is defined as the temporal delay required between two cues (i.e. a sound and a flash) so that the temporal order is unknown (Stone et al., 2001). Two common paradigms used to assess the PSS are the temporal order judgment (TOJ) and simultaneity judgement (SJ) tasks in which a participant is asked to determine the sequential order or the temporal simultaneity, respectively, of asynchronous as well as synchronous auditory and visual cues. As the TOJ and SJ tasks tap into different perceptual processes (Love et al., 2013), it is not surprising that PSS estimates are not correlated between these two tasks (Bedard & Barnett-Cowan, 2016; van Eijk et al., 2010). Indeed, while TOJ performance is heavily dependent on sequential timing information and is often classified as a more difficult temporal task, the SJ task appears to measure perceptual integration and is influenced by an asymmetric criterion bias at narrow SOAs (Linares & Holcombe, 2014).

PSS measures also reliably estimate the magnitude and effect of temporal recalibration, an innate strategy to perceptually realign one’s temporal bias of audiovisual stimuli in order to retain coherent multisensory percepts when audiovisual signaled are misaligned due to spatial and contextual factors. Classic adaptation paradigms are traditionally used to investigate temporal recalibration aftereffects, measured as the magnitude and direction of the PSS shift following prolonged exposure (i.e. 3 minutes) to a constant temporal asynchrony between two cues (Fujisaki et al., 2004; Vroomen et al., 2004). In conjunction with this prolonged adaptation effect, there is also evidence for a much more rapid, low-level process that can produce immediate perceptual realignment on the current trial (t) toward the temporal delay (a negative aftereffect) presented in the previous trial (t-1) for both simple and complex audiovisual stimuli (Van der Burg et al., 2013; Van Der Burg et al., 2015; Van der Burg & Goodbourn, 2015). This more immediate strategy of temporal realignment makes sense from an ethological perspective where delays between naturally occurring audiovisual signals are vast and highly influenced by environmental obstacles and features. Indeed, as light is most often perceived prior to sound, it is not surprising that the extent of this rapid recalibration is asynchronous. For instance, when an auditory pure tone led a visual flash, the effect of recalibration was significant only for the smallest delay tested (64 ms) while this effect was significant over a larger range of SOAs for visual-leading t-1 trials (Van der Burg et al., 2013).

The previously described studies depicting negative aftereffects induced after short-term exposure to an asynchronous audiovisual stimulus only used the SJ task. However, a more recent study that compared rapid recalibration between an SJ and a TOJ task found opposing directions of temporal recalibration (Roseboom, 2019). Specifically, while modality order of the preceding trial (t-1) elicited an immediate shift in PSS on the current trial (t) toward that delay (negative aftereffect) in the SJ task, in line with prior findings (Van der Burg et al., 2013; Van der Burg & Goodbourn, 2015), the PSS shifted away from the preceding (t-1) temporal delay (positive aftereffect) in the TOJ task (Roseboom, 2019). This positive aftereffect reported in the TOJ task is likely a result from a choice-repetition bias which leads a participant to repeat their temporal order decision from the preceding trial on the current trial. Indeed, choice repetition occurred in both SJ and TOJ tasks however, while this bias affected the amplitude of the SJ response distribution (i.e. shifts in the proportion of synchronous reports) and therefore didn’t affect the PSS, the shift in the TOJ response distribution had a direct effect on PSS estimates (Keane et al., 2020).

Interestingly, the effect of adaptation on temporal sensitivity thresholds, a measure of the smallest temporal difference between the two signals that can be discerned by the observer, is dependent on the temporal scale of the adaptation. For instance, the more classical adaptation designs in which participants are continuously exposed to a constant audiovisual delay result in larger thresholds for visual-leading stimuli (Y. M. Chan et al., 2014a; Fujisaki et al., 2004), likely enabling more flexibility for integrating audiovisual stimuli characterized by this temporal relationship. However, in studies examining rapid realignment of audiovisual simultaneity, the modality order of the preceding trial had no significant effect on temporal thresholds (Van der Burg et al., 2013; Van Der Burg et al., 2015; Van der Burg & Goodbourn, 2015). Although, worse thresholds were associated with increased magnitudes of the rapid recalibration effect (Van der Burg et al., 2013), again likely due to greater flexibility toward perceiving asynchronous audiovisual stimuli.

Temporal sensitivity appears to be driven by separate cortical processes relative to perceptual synchrony. For instance, older adults that demonstrated impaired temporal sensitivity across audiovisual, visual-motor and visual only temporal tasks did not show any differences in perceptual synchrony estimates relative to young adult controls (Scurry et al., 2019). Similar findings from SJ tasks have been reported in individuals with high levels of attention deficit hyperactivity disorder (ADHD) traits (Panagiotidi et al., 2017) and those with autism spectrum disorder (ASD) (Noel et al., 2017) suggesting that in those neurological disorders that normally affect temporal acuity, perceptual synchrony remains stable.

In majority of studies that investigate multisensory temporal processes, the auditory and visual stimuli are presented at a presumed supra-threshold level. However, unisensory detection sensitivities are not necessarily equivalent across control and experimental groups. This is an especially valid consideration in aging studies as the natural aging process generally results in impairments to the separate sensory systems. Indeed, the sensory receptor organs and the primary regions within the central nervous system that are required for sensory perception deteriorate with age (Celesia et al., 1987; Goodin et al., 1978; Kraus & Anderson, 2013; Lemaître et al., 2005; Ostroff et al., 2003; Werner et al., 2010). These common age-related modifications to sensory processing result in poorer detection abilities of unisensory stimuli. For example, hearing ability and visual acuity both decline with age and subsequent auditory and visual detection thresholds worsen (Humes et al., 2009). Therefore, use of a supra-threshold stimulus may actually be closer to detection thresholds for the experimental groups rather than for the control groups and resultant group effects in multisensory processing could be partially due to their differences in unisensory processing impairments. Without specifically equating intensity levels relative to each individual’s threshold, this confound is difficult to rule out.

Stimulus intensity can also significantly affect synchrony perception. For instance, in a rat model, lowered auditory intensity led to asynchronous perception of simultaneous audiovisual stimuli during a SJ task and to a bias toward visual-first perception in a TOJ task (Schormans & Allman, 2018). Stimuli of higher intensity were reported as being perceived first in a TOJ task and the extent of this bias was correlated to the difference in stimulus intensities between the two cues being compared (Neumann & Niepel, 2004). In a separate study where stimuli were presented in the periphery rather than centrally, increasing the intensity of the visual stimulus led to a shift toward an auditory-leading PSS in a TOJ task (Boenke et al., 2009). Despite these differing directions of a PSS shift, together these studies serve as evidence of stimulus intensity contributing to perceptual bias over simultaneity.

As stimulus intensity has an evident impact on temporal processing and integrating multisensory stimuli, it follows that an individual’s ability to detect stimuli has a similar influence and participant’s thresholds should be controlled for during experimentation. A recent study examined how stimulus effectiveness influenced synchrony perception in young adults using an SJ task. Higher rates of synchrony perception were found when the auditory and visual cue were both presented at low intensity compared to when they were both presented at high intensity levels (Fister et al., 2016). However, this finding was specific for trials in which the auditory and visual stimulus were separated by 200 ms suggesting increased perceptual flexibility for less effective stimuli at larger delays (Fister et al., 2016). Another study compared the temporal binding windows, or the length of temporal delay that an individual is likely to perceptually bind asynchronous stimuli, of older and younger adults after performing a version of the SJ task when the stimulus intensity was based on the individual’s threshold and found that the effect of age was not due to reduced unisensory discrimination sensitivity (Y. M. Chan et al., 2014b). However, since visual acuity is positively correlated to the temporal binding window (the temporal limits of integrating audiovisual stimuli) measured by the TOJ task but not the SJ task (Begum & Barnett-Cowan, 2018), stimulus intensity may contribute differently dependent on the mechanism the temporal task requires.

Therefore, the current study sought to investigate how stimulus intensities based on individual detection thresholds affect temporal sensitivity and temporal synchrony in two separate temporal processing tasks, the TOJ and SJ. We expected that in incongruent conditions, where one stimulus is presented at supra-threshold (i.e. visual) and the other is presented at near-threshold (i.e. auditory), temporal sensitivity would be biased toward the supra-threshold stimulus (i.e. visual). In addition, we hypothesized an effect of stimulus intensity on temporal thresholds estimated during a TOJ, not an SJ task (Begum & Barnett-Cowan, 2018). As the brain demonstrates an innate process to maintain perceptual constancy despite external changes, we did not expect an effect of stimulus intensity on perceptual synchrony in either task. However, we did expect an interaction between stimulus intensity and modality order of trial t-1 on PSS estimates. Specifically, when the leading modality (i.e. visual) of trial t-1 was at supra-threshold, we expected a shift in PSS toward that temporal delay (i.e. negative aftereffect) similar to the bias toward supra-threshold vs near-threshold stimuli previously reported (Neumann & Niepel, 2004; Schormans & Allman, 2018).

2. Methods

2.1. Participants

19 young adults (M = 20.42 ± 0.25 years; 9 females) were recruited from the University of Nevada, Reno and the surrounding community. All participants reported no history of psychiatric or neurological disorders, normal cognitive function, and normal or corrected to normal vision. They were additionally tested for normal hearing using AudioScope 3, a screening audiometer (Welch Allyn, Skaneateles Falls, NY, USA), defined as pure tone thresholds ≤ 25 dB for 1000 and 2000 Hz in both ears. All participants provided signed informed consent before any experimentation and were financially compensated for their time. The experimental protocol was reviewed and approved by the Institutional Review Board at the University of Nevada, Reno.

2.2. Apparatus and Stimuli

The visual stimulus was an isotropic two-dimensional Gaussian blob subtending 3.5˚ visual angle presented on a uniform grey background for 30 ms. The contrast of the visual stimulus was set to either near- or supra-threshold levels based on the participant’s detection threshold measured in during the estimation of unisensory detection thresholds (see below). The auditory stimulus was a 30 ms 1000 Hz tone presented at a dB level set at either near- or supra- individual detection threshold levels. All stimuli were generated using MATLAB (Mathworks, Natick, MA, USA) and Psychtoolbox extensions (Brainard, 1997; Pelli, 1997). Auditory and visual stimuli were presented via an AudioFile Stimulus Processor and a Display++ system with a refresh rate of 120 Hz, respectively (Cambridge Research Systems, Rochester, UK) in order to preserve precise timing. Auditory stimuli were presented via a speaker (Fantech HellScream GS 201, Nepal) that was centered underneath the display to align with presentation of the visual stimuli.

2.3. Estimation of Unisensory Detection Thresholds

A 2-down-1-up staircase was used to quantify individual detection thresholds. Participants were instructed to respond if they detected the stimulus or not for a total of 80 trials. Trials were separated by a variable intertrial interval between 1200 and 1400 ms. The stimulus intensities of the final 10 trials were extracted and averaged to determine the 70.7% detection threshold. For each individual, the near-threshold value was defined as 3 SDs above the mean while the supra-threshold value was defined as 10 SDs above the mean (Y. M. Chan et al., 2014b). The near-threshold level should elicit detection of a stimulus 99.7% of the time (Y. Chan et al., 2017) and piloting procedures with 2 participants demonstrated they were able to detect the near-threshold stimuli on 100% of the 15 test trials. The experimental procedure used to determine detection thresholds was carried out separately for the auditory and visual stimuli.

2.3.1. Visual Detection Threshold

The initial contrast, measured as a Weber fraction, between the visual stimulus (white circle) and the grey background was 1%. The step size used was also 1% so that on trials where participants were unable to detect the white circle, the contrast increased by 1% and when participants did detect the white circle for two consecutive trials, the contrast decreased by 2%. In the case of ceiling performance, the minimum contrast allowed was 1%. Following the onset of the visual stimulus, a brief 750 Hz beep played in order to alert the participant to the stimulus presentation and encourage a response. If a participant detected the Gaussian blob, they pressed the ‘y’ key on the keyboard and if they did not detect the stimulus, they pressed the ‘n’ key. Participants began the visual detection staircase with a brief practice session so that on 3 of the 6 practice trials, the white circle was presented at .56 Weber fraction while no circle was presented on the other 3 to ensure participants understood the task.

2.3.2. Auditory Detection Threshold

The 1000 Hz tone was initially presented at .5 dB and varied by 2 dB dependent on the participant’s response. When the participant was able to detect the tone in 2 consecutive trials, the sound level decreased by 2 dB (with a minimum value of .5 dB) and when the participant could not detect the tone, the sound level increased by 2 dB. 200 ms following the tone’s onset, the fixation cross changed to a blue color during the response window so that individuals knew a stimulus had been presented regardless of whether or not they perceived the sound. If a participant detected the sound, they pressed the ‘y’ key on the keyboard and if they did not detect the sound, they pressed the ‘n’ key. Participants began this task with 6 practice trials with the tone presented at 55 dB on 3 of the trials and no tone on the other 3 trials to ensure they understood the task.

2.4. Temporal Processing Tasks

All participants completed two separate temporal processing tasks, a simultaneity judgment (SJ) and a temporal order judgement (TOJ) task. For each task, 4 experimental blocks were completed where each block contained one of the 4 possible combinations of visual and auditory stimulus intensity levels – auditory near-threshold, visual near-threshold (ANVN); auditory near-threshold, visual supra-threshold (ANVS); auditory supra-threshold, visual near-threshold (ASVN); auditory supra-threshold, visual supra-threshold (ASVS). The order of the 4 stimulus intensity conditions within each task, as well as the order of task (SJ vs TOJ), was randomized. However, all 4 stimulus conditions were completed within a single task (i.e. TOJ) prior to performing the second task (i.e. SJ).

2.4.1. Simultaneity Judgment (SJ) Task

On each trial of a single block, the visual and auditory stimuli were presented at variable stimulus onset asynchronies (SOAs) ranging from −350 to +350 ms in 50 ms steps where negative SOAs represent auditory leading visual trials and positive SOAs represent visual leading auditory trials. Each SOA was repeated 5 times for a total of 75 trials/block (5 repetitions x 15 SOAs). Participants were instructed to respond via a keyboard press as to whether or not they perceived the two stimuli as simultaneous.

Prior to performing any of the 4 SJ blocks, participants completed a short practice session to ensure they understood the task. The contrast of the visual stimulus was .56 Weber fraction and the auditory stimulus was presented at 55 dB. Half of the trials were synchronous (SOA of 0 ms) and the other half were asynchronous presented at larger and different delays than the experimental version of the task (420, 460 or 510 ms).

2.4.2. Temporal Order Judgment (TOJ) Task

The same parameters were used for the TOJ task so that 4 blocks were performed by each participant and each AV stimulus intensity combination was tested separately in the 4 blocks. Again, 15 SOAs (−350 to + 350 ms in 50 ms steps) were each repeated 5 times for a total of 75 trials per/block. At the end of each trial, participants were asked to judge the temporal order of the stimuli so that a 1 on the keyboard indicated they perceived the auditory stimulus as preceding the visual and a 2 indicated the perceived the visual preceding the auditory stimulus.

Prior to running any of the TOJ blocks, participants did complete a short practice session of 10 trials to ensure they understood the task and the correct response. Equivalent to the SJ practice session, the visual stimulus was presented at .56 Weber fraction and the auditory stimulus at 55 dB. SOA values were also different from the experimental TOJ blocks (360, 420, 460 or 510 ms).

2.5. Behavioral and Statistical Analysis

In order to understand how stimulus intensity and task affected rapid recalibration and temporal sensitivity, trials were categorized based on the leading modality in the preceding trial (t-1) so that auditory-leading t-1 trials and visual leading t-1 trials data subsets were created. This categorization occurred for each of the stimulus intensity conditions in both the SJ and TOJ task prior to the fitting procedures described below. Modality order is therefore implicated in generating any difference in the fits between the auditory-leading t-1 trials and visual leading t-1 trials (Alais et al., 2015; Van der Burg et al., 2013). Specifically, the difference in PSS estimates (ΔPSS = PSS_{visual-leading t-1 trials} – PSS_{audio-leading t-1 trials}) captured how modality order could rapidly recalibrate the system and shift perceptual bias of temporal simultaneity.

For the SJ task, the average response of ‘simultaneous’ was plotted as a function of SOA separately for each participant. The data were then fit with a gaussian function that used three free parameters: the mean, the standard deviation, and the amplitude. The mean of the fitted function reflected the PSS, the standard deviation reflected the temporal window of integration and the amplitude reflected the proportion of simultaneous responses.

Similarly, for the TOJ task, the proportion of ‘flash first’ response was plotted as a function of SOA for each individual and then fit with a normal cumulative distribution function. The mean was estimated from this cumulative distribution and reflected the PSS (Alais et al., 2015; Fechner, 1860; Scurry et al., 2019; van Eijk et al., 2008; Weber, 1834). Temporal order sensitivity thresholds were estimated using the standard deviation of the fitted cumulative distribution function (Alais et al., 2015; Scurry et al., 2019).

4 participants reported simultaneous audiovisual stimuli in the SJ task over 50% of the time when the pair was not veridically synchronous and thus were excluded from further analysis.

To determine statistically significant effects and interactions that affected PSS and threshold estimates, a repeated measures ANOVA was performed using task (SJ vs TOJ), condition (ANVN; ANVS; ASVN; ASVS), and modality order in t-1 trial (auditory vs visual) as factors. A repeated measures ANOVA was also used to understand how task, auditory stimulus intensity (near vs supra) and visual stimulus intensity (near vs supra) affected ΔPSS. To understand and interpret any non-statistically significant results (Dienes, 2014; van Doorn et al., 2020), Bayes factors were calculated using the BayesFactor package in R with default priors (Morey, 2018; Rouder et al., 2012). All statistical analysis was performed in R statistical software.

3. Results

Detection thresholds are shown in Figure 1 for both auditory (left panel) and visual (right panel) stimuli. The average auditory detection threshold was 6.10 ± 1.03 dB while the average visual detection threshold was .094 ± .010 contrast (Weber fraction).

Figure 1.

Unisensory detection thresholds. Group average and individual data are shown for auditory (left panel) and visual (right panel) detection thresholds. Error bars represent SEM.

Initially, we categorized trials based on the leading modality of the previous trial (t – 1) and fit a normal cumulative distribution and gaussian function to both data subsets for the TOJ and SJ task, respectively. This procedure was done for all 4 stimulus intensity conditions and temporal sensitivity thresholds and PSS estimates were quantified. The individual along with the group fits for both auditory leading (blue) and visual leading (orange) t-1 trial orders are displayed in Figure 2 for the TOJ task and in Figure 3 for the SJ task.

Figure 2.

Group and individual fits of the cumulative distribution function for TOJ responses. The group-averaged proportion of flash first responses in the TOJ task are plotted as a function of stimulus onset asynchrony (SOA) for both auditory-leading t-1 trials (blue) and visual-leading t-1 trials (orange). The SOAs shown on the x-axis depict the asynchrony level of the current trial (t). Group (bold lines) and individual fits (thin lines) of a normal cumulative distribution function are shown for both subsets of trials. Error bars represent SEM.

Figure 3.

Group and individual fits of a Gaussian function for SJ responses. The group-averaged proportion of simultaneous responses in the SJ task are plotted as a function of stimulus onset asynchrony (SOA) for both auditory-leading t-1 trials (blue) and visual-leading t-1 trials (orange). The SOAs shown on the x-axis depict the asynchrony level of the current trial (t). Group (bold lines) and individual fits (thin lines) of a Gaussian function are shown for both subsets of trials. Error bars represent SEM.

A 2×4×2 repeated measures ANOVA was conducted to determine the effect of the leading modality in trial t-1, stimulus intensity condition, and task on PSS estimates. Across all conditions and tasks, group-averaged PSS values were negative indicating an auditory-leading bias. PSS values were significantly smaller (a shift toward visual-leading) during the SJ compared to the TOJ task (F(1,190) = 7.61 p < .01) while the leading modality in trial t-1 (F(1,187) = 3.26, p = .07) and stimulus intensity condition (F(3, 187) = 1.35, p = .26) had no significant influence. Similarly, none of the 2-way or the 3-way interactions were significant (Fs(3,187) ≤ 1.49, ps ≥ .22). Group averaged PSS values with individual PSS estimates are plotted in Figure 4 for both the SJ (left panel) and TOJ (right panel) tasks.

Figure 4.

Point of subjective simultaneity (PSS) estimates. Group averaged and individual PSS estimates for audio-leading t-1 trials (dark grey bars w/black circles) and visual-leading t −1 trials (light grey bars w/light grey circles) are displayed for the SJ (left panel) and TOJ (right panel) tasks for each of the 4 conditions (ANVN; ANVS; ASVN; ASVS). Error bars represent SEM.

This data was further examined using Bayes Factors to determine the likelihood of the null hypotheses compared to other possible models. As expected from the results derived from the 2×4×2 ANOVA, the model with Task as a main effect was preferred (BF = 4.94), followed by a model including both task and the modality order of trial t-1 (BF = 3.61). However, modality order of t-1 trial alone had insufficient evidence (BF = 0.68) to support the null hypothesis that it does not influence the PSS. The null hypothesis that the main effect of interest, stimulus intensity condition, does not significantly influence PSS measures is favored in a model that only contained condition as a main effect (BF = 0.11) and in models containing stimulus intensity condition in the interaction term(s) (BFs ≤ 0.16).

The same statistical procedures were conducted to determine effects on temporal sensitivity measures (Figure 5). Results from the 2×4×2 repeated measures ANOVA showed that none of the main effects (leading modality in trial t-1, stimulus intensity condition, and task) were significant (Fs(3,188) ≤ 0.99, ps ≥ .32). Further, the 2-way and 3-way interactions did not significantly influence temporal thresholds (Fs(3,188) ≤ 2.01, ps ≥ .16). All possible models were again compared to the null model and subsequent Bayes Factors displayed moderate support for the null hypothesis that Task (BF = 0.24) and leading modality in trial t-1 (BF = 0.22) did not influence thresholds. Further, there was strong support for the null hypothesis that stimulus intensity condition did not significantly affect temporal thresholds (BF = 0.05) and for models containing any combination of these main effects or their interactions (BFs ≤ 0.05).

Figure 5.

Temporal thresholds. Group averaged and individual temporal sensitivity thresholds for audio-leading t-1 trials (dark grey bars w/black circles) and visual-leading t −1 trials (light grey bars w/light grey circles) are displayed for the SJ (left panel) and TOJ (right panel) tasks for each of the 4 conditions (ANVN; ANVS; ASVN; ASVS). Error bars represent SEM.

In order to examine how stimulus intensity and task affected rapid recalibration, a ΔPSS was calculated that compared the PSS in visual leading t-1 trials to the PSS from auditory leading t-1 trials (see Figure 6). A 2×2×2 repeated measures ANOVA that was conducted to examine the effects of task, auditory stimulus intensity and visual stimulus intensity on ΔPSS did not reveal any significant main effects (Fs(1, 104) ≤ 1.34, ps ≥ .25) or interactions (Fs(1, 104) ≤ 3.16, ps ≥ .08). The null model was again compared to all possible models and Bayes Factors were extracted to estimate the likelihood of the null. All possible models with main effects and their interactions demonstrate moderate to strong support for the null hypothesis of no significant influence on ΔPSS (BFs ≤ 0.37).

Figure 6.

ΔPSS measures to examine the effect of modality order in preceding trial (t-1). Group-averaged and individual differences in PSS between visual- and auditory-leading t-1 trials (ΔPSS) are displayed for the SJ (dark grey) and TOJ (light grey) tasks for all 4 stimulus intensity conditions (ANVN; ANVS; ASVN; ASVS). Error bars represent SEM.

To investigate if the modality order of trial t-1 affected PSS estimates, separate t-tests were used to investigate if ΔPSS values were significantly different from 0. Across conditions in the TOJ task, no significant influence of the modality order of trial t-1 occurred (ts(14) ≤ 1.14, ps ≥ .28). Bayes Factor t-tests were also conducted and revealed only anecdotal support for the null hypothesis that the ΔPSS = 0 (BFs were between 0.32 and 0.47). In the SJ task, only the ANVN condition revealed any significant recalibration (t(14) = 2.84, p < .05, BF = 4.50) while the remaining 3 conditions did not (ts(14) ≤ 1.34, ps ≥ .20). Again, Bayes Factor t-tests were between 0.34 and 0.56 suggesting that there was not sufficient evidence to support that stimulus intensity condition had no influence on the PSS. This coincides with the observed difference in the group-averaged simultaneous response proportion in A-lead versus V-lead conditions in Figure 3 (i.e., A-lead (blue) show higher proportions than V-lead (orange) on left hand side of curve in ASVN and ASVS condition).

4. Discussion

Flexibility in temporal processing of multisensory cues is imperative for proper recalibration dependent on changing environmental contexts. The strength and features driving multisensory recalibration is often measured by exposing participants to a constant temporal offset between signals and then measuring how our system has adapted to this offset by our shift in perceived temporal synchrony. In some experimental groups, such as older adults that present with reduced temporal sensitivity for discriminating temporal delays between multisensory cues, a consequent reduction in the strength of audiovisual recalibration is also reported (Y. M. Chan et al., 2014a). However, without equating stimulus amplitudes across participants, it is difficult to declare our basic sensory processing deficits as contributing to an experimental subject’s recalibration deficit. As the two common paradigms used to assess temporal recalibration, the SJ and TOJ tasks, often rely on supra-threshold stimuli as negating this potential confound, our study sought to explicitly examine the contribution of individualized stimulus intensity levels to temporal sensitivity and synchrony measures estimated from both tasks. The effect of stimulus intensity level did not significantly impact temporal sensitivity or synchrony estimates, regardless of modality order in trial t-1. However, PSS estimates were significantly smaller in the SJ compared to the TOJ task while no difference in thresholds were found between tasks.

Initially, we predicted that stimulus intensity levels would affect temporal sensitivity estimates, particularly when one stimulus was at near-threshold while the other was presented at supra-threshold. For instance, when the auditory cue was presented at near-threshold and the visual at supra-threshold, we would expect improved temporal order detection and thus sensitivity for visual leading stimuli. This prediction was in line with prior reports of temporal order bias toward a stimulus presented at a higher level than the stimulus from the opposing modality (Boenke et al., 2009; Neumann & Niepel, 2004; Schormans & Allman, 2018). Fister et al. (2016), also found reduced temporal sensitivity (reported as increased perceived synchrony) for asynchronous stimuli when both auditory and visual stimuli were at a low intensity compared to when they were both at a high intensity level.

Interestingly, we did not find any significant effects of stimulus intensity on temporal thresholds. While the present study used a total of 15 SOAs, a narrower range of SOAs have previously been used (4 SOAs in Fister et al., 2016; 7 SOAs in Schormans & Allman, 2018; 10 SOAs in Boenke et al., 2009). The extended range of SOAs currently used likely provided participants with additional information to improve their decision strategies and accurately perform the temporal tasks leading to equivalent temporal thresholds. This explanation is supported by the recent finding of reduced proportions of temporally dependent, sound-induced flash illusions when individuals were presented with 10 vs 6 SOAs (J. S. Chan et al., 2018).

Prior studies also manipulated the intensity of stimuli from only 1 modality while keeping the intensity of the other stimulus constant (Boenke et al., 2009; Schormans & Allman, 2018) or had congruent levels of stimulus intensity (both low or both high intensity) (Fister et al., 2016). The increased number of stimulus intensity combinations used in the present study may have negated any potential effects due to complex interactions. In addition, the more salient stimulus may have helped drive the perception of sequential order so the particular combination wouldn’t exert any effect on performance. For instance, when needing to determine temporal order, a participant may be judging the timing of the most salient stimulus relative to the overall trial length. Therefore, even when that salient stimulus lagged the weaker stimulus, a participant could still correctly discern the order in that trial. In addition, similar thresholds across conditions may have been a consequence of a practice effect as participants completed all 4 stimulus intensity blocks for one task (i.e. TOJ) prior to completing the second task (i.e. SJ) providing ample time to develop effective criterions for judging temporal relationships.

Differences in the PSS values derived from visual-leading compared to auditory-leading t-1 trials also did not show any significant effect of stimulus intensity. Prior studies examining rapid recalibration of temporal synchrony perception found that the physical temporal properties between audiovisual cues, and not the perceived temporal relationship, affected temporal realignment (Van der Burg et al., 2013, 2018). Therefore, it is likely that while stimulus intensity may affect perceived temporal order or simultaneity (Boenke et al., 2009; Schormans & Allman, 2018), it cannot alter the veridical temporal relationship and thus does not influence any temporal realignment that occurs between trials. One limitation of the present study was the sparse number of trials per SOA during each experimental condition (5 trials/SOA). This prevented analysis of how the ΔPSS was influenced by the degree of temporal asynchrony from the previous trial, as shown in prior studies (Roseboom, 2019; Van der Burg et al., 2013).

At the group level, the PSS was negative (audio-leading) across all tasks and stimulus intensity conditions. This is contrary to prior studies finding a visual-leading PSS for the SJ (Bedard & Barnett-Cowan, 2016; Fister et al., 2016; van Eijk et al., 2008) and TOJ (Bedard & Barnett-Cowan, 2016; Scurry et al., 2019) tasks indicating that auditory signals are processed at a slightly faster rate than visual signals. Indeed, a PSS shift in the audio-leading direction for higher intensity visual stimuli was reported from a TOJ task suggesting that visual cues were processed faster (Boenke et al., 2009). However, this explanation doesn’t hold when the two cues were presented at a similar intensity (ANVN and ASVS) or when the visual stimulus was presented at a higher intensity relative to auditory (ANVS). Perhaps, the different exposures to stimulus intensities throughout the experiment adjusted the overall temporal sensitivity of participants leading to a shift toward audio-leading PSS values. In prior studies using complex stimuli (i.e. a bouncing ball (van Eijk et al., 2008) or a Newton’s cradle toy with manipulated predictive and postdictive information in the visual signal (van Eijk et al., 2010)) an audio-leading PSS was explained by participant’s enhanced temporal sensitivity for audio-leading trials. Regardless, the negative PSS values do indicate a shift in natural perception as visual information naturally precedes auditory. Further, the individual variability in PSS values across all conditions suggest that inherent bias exists, and additional individual factors likely contribute to the temporal synchrony perception.

The overall auditory-leading PSS was significantly larger in the TOJ task relative to the SJ task indicating that auditory stimuli needed to lead visual by a greater extent to elicit a simultaneous percept. As discussed above, this may have reflected improved discrimination for audio-leading trials specifically during the TOJ task. It has been previously reported that the SJ and TOJ tasks probe different functions involved in temporal processing. SJ tasks are subject to an asymmetric criterion bias that increases the likelihood of perceiving simultaneity in asynchronous cues while TOJ tasks are associated with a difficulty in ordering successive events that impairs performance (Linares & Holcombe, 2014). Therefore, it is probable that participants became more sensitive toward temporal order cues during the TOJ compared to the SJ thus inducing the auditory-lead shift in their PSS.

The absence of any effect of stimulus intensity on temporal sensitivity or synchrony supports abundant prior findings that rely on supra-threshold stimulus presentation levels. Other effects, such as stimulus type, task, or between subject factors (i.e. age) are less likely confounded by any differences at the initial steps of sensory processing that are influenced by detection sensitivity and resultant speed of processing. While the reported null effects certainly substantiate prior findings, replications of the current design should be carried out in experimental groups rather than just young adults. In addition, replicable findings using different stimulus types, especially more ethologically relevant stimuli, would further strengthen the current findings and help support the practice of using a standard supra-threshold intensity level. Finally, extensions of this project that use additional definitions for near- and supra-threshold stimuli may help to parse out the point at which unisensory detection levels, and the specific combination of detection levels between stimulus modalities, affect multisensory temporal processing.

5. Conclusions

This project sought to investigate how individualized stimulus intensity levels influenced temporal precision and perceived temporal synchrony. While stimulus intensity did not have any significant effect on temporal precision or synchrony, the type of temporal processing task did affect perceived temporal synchrony. This finding is likely due to the separate functions required by temporal order vs simultaneity judgments. Determining sequential order decodes asynchronous cues in a less ambiguous way than determining synchrony and likely allows participants’ judgement strategies to improve for more difficult audio-leading trials over experimentation shifting their PSS toward audio-leading stimuli. Future studies that probe experimental groups and expand the definition of near- and supra-threshold stimuli will increase the validity of these findings and further promote the use of supra-threshold stimuli when investigating aspects of temporal processing as a way to reduce the confounding effect of unisensory detection ability.