Skip to main content
NCBI home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2017 Nov 1.
Published in final edited form as: Exp Brain Res. 2016 Jul 11;234(11):3269–3277. doi: 10.1007/s00221-016-4724-3

Multisensory Perceptual Learning is Dependent Upon Task Difficulty

Matthew De Niear 1,2,3, Bonhwang Koo 4, Mark T Wallace 2,5,6
PMCID: PMC5073017  NIHMSID: NIHMS802268  PMID: 27401473

Abstract

There has been a growing interest in developing behavioral tasks to enhance temporal acuity as recent findings have demonstrated changes in temporal processing in a number of clinical conditions. Prior research has demonstrated that perceptual training can enhance temporal acuity both within and across different sensory modalities. Although certain forms of unisensory perceptual learning have been shown to be dependent upon task difficulty, this relationship has not been explored for multisensory learning. The present study sought to determine the effects of task difficulty on multisensory perceptual learning. Prior to and following a single training session, participants completed a simultaneity judgment (SJ) task, which required them to judge whether a visual stimulus (flash) and auditory stimulus (beep) presented in synchrony or at various stimulus onset asynchronies (SOAs) occurred synchronously or asynchronously. During the training session, participants completed the same SJ task but received feedback regarding the accuracy of their responses. Participants were randomly assigned to one of three levels of difficulty during training: easy, moderate and hard, which were distinguished based on the SOAs used during training. We report that only the most difficult (i.e., hard) training protocol enhanced temporal acuity. We conclude that perceptual training protocols for enhancing multisensory temporal acuity may be optimized by employing audiovisual stimuli for which it is difficult to discriminate temporal synchrony from asynchrony.

Introduction

Accurate perception of the world requires synthesizing information across multiple sensory modalities (Stein et al. 2014). The temporal proximity of these sensory signals is an important property for determining if these signals arise from a common event or distinct events (Vroomen and Keetels 2010). Accordingly, the capacity to accurately utilize temporal cues to integrate sensory signals from different modalities is essential for developing perceptual coherence (Stein and Stanford 2008; Wallace and Stevenson 2014) and for driving the well-established behavioral benefits associated with multisensory integration (Sumby and Pollack 1954; Diederich and Colonius 2004). Numerous studies have demonstrated that the integration of multisensory stimuli occurs over a range of time in which the separate sensory signals may be proximate, but asynchronous (Dixon and Spitz 1980; Meredith et al. 1987; Vroomen and Keetels 2010). This epoch, which typically spans several hundred milliseconds, has been termed the temporal binding window (TBW) (Wallace and Stevenson 2014).

The TBW has emerged as a useful construct for indexing temporal acuity for multisensory stimuli, particularly audiovisual stimuli (Wallace and Stevenson 2014). As the TBW is probabilistic in nature, it flexibly resolves differences in the neural and environmental propagation times of sensory signals, allowing for appropriate sensory integration (Vroomen and Keetels 2010). Furthermore, the TBW appears to be flexibly specified depending upon the nature of the stimuli to be integrated, such that narrower TBWs are reported for simple audiovisual stimuli (e.g. flashes and beeps) and wider TBWs are reported for more complex stimuli (e.g. audiovisual speech) (Dixon and Spitz 1980; Stevenson and Wallace 2013). It has been suggested that the TBW reflects experience with the statistical relationships of sensory stimuli as over the course of development, the TBW appears to narrow to more accurately reflect the statistics of the environment (Hillock et al. 2011; Hillock - Dunn and Wallace 2012; Chen et al. 2016).

Although the TBW makes a great deal of ecological sense in regards to tolerating the natural asynchronies for stimuli in our world, if the TBW is too large, as is observed for those with autism spectrum disorder (ASD), stimuli that should be segregated will instead be integrated . A large TBW (i.e. decreased temporal acuity) has also been associated with decreased strength of overall multisensory integration (Stevenson et al. 2012; Stevenson et al. 2014). For example, in individuals with ASD, the size of the TBW has been reported to be negatively correlated with perceptual fusion of audiovisual speech as measured by the McGurk illusion (i.e. larger TBWs are associated with lower rates of perceptual fusion) (Stevenson et al. 2014). As multisensory temporal acuity appears to be impaired for those with ASD, there has been a growing interest in identifying ways to improve temporal performance, with the goal that these improvements in (multi)sensory function will generalize to improvements in domains of weakness such as social communication (Wallace and Stevenson 2014).

Recent work has demonstrated that perceptual training with feedback can produce significant and lasting changes in temporal acuity (i.e. narrowing of the TBW) for audiovisual stimuli (Powers et al. 2009; Stevenson et al. 2013; Setti et al. 2014). In addition, subsequent work has found that this improvement in audiovisual temporal acuity generalizes to other measures that are dependent upon temporal processing (Powers III et al. 2016). Although multisensory perceptual learning appears to exhibit similarities to modality-specific (i.e., unisensory) perceptual learning, much remains unknown as to how perceptual training enhances multisensory acuity. One aspect of perceptual training that might influence the extent of multisensory perceptual learning is task difficulty. For example, in the visual domain, the difficulty of the task appears to be a strong determinant of whether learning will or will not occur (Ahissar and Hochstein 1997; Seitz et al. 2006; Wang et al. 2010; DeLoss et al. 2014). Furthermore, based on evidence from other studies of perceptual training, it has been suggested that the greatest changes in behavioral performance are observed when the stimuli presented are challenging while also eliciting sufficiently rewarding feedback (Mishra and Gazzaley 2014). Often, adaptive procedures (Deveau et al. 2014; Mishra et al. 2014) or baseline performance measurements (Anguera et al. 2013) are employed to control for task difficulty prior to or during the course of a training session as individual differences in perceptual abilities are likely to be present prior to training. Such individualized perceptual training protocols that equate for differences in perceptual ability allow for a clearer interpretation of the actual effects of training (Anguera et al. 2013). Furthermore, perceptual training protocols that account for an individual’s baseline performance (in this case temporal acuity) and subsequently individualize a training regimen based on these data are most likely to elicit the greatest changes in perception following training (Mishra et al. 2016). As perceptual training has been suggested as a potential therapeutic approach in circumstances in which multisensory temporal acuity is poor, it is crucial to understand how training protocols may be optimized to enhance perceptual learning.

The present study sought to address if task difficulty affects the capacity for multisensory perceptual training to enhance temporal acuity. We employed a perceptual training paradigm similar to prior studies in which a simultaneity judgment (SJ) task was employed to measure the TBW prior to and following training. These prior studies have employed a single, fixed set of stimuli that remained constant during training sessions and that were independent of an individual participant’s temporal acuity (Powers et al. 2009; Powers et al. 2012; Powers III et al. 2016). In contrast to this approach, in the current study we individually tailored the stimuli presented during the training sessions based on a participant’s pre-training temporal acuity. Individually tailoring the training stimuli allowed us to develop three training protocols of varying difficulty based on an individual participant’s ability to detect asynchrony. Importantly, by presenting SOAs that were equated based on each individual’s likeliness to perceive audiovisual asynchrony, we were able to control for individual differences in temporal acuity prior to training (in contrast, changes in temporal acuity following perceptual training employing a single set fixed SOAs may be influenced by pre-training temporal acuity). We hypothesized that increasing task difficulty during training sessions would potentiate the effects of perceptual training.

Methods

Subjects

All participants received informed consent prior to participating in this study. A total of N = 51 typically-developing adults were included in the analyses for this study (Age, M =20.21 years; Gender, 28 female). An additional 6 individuals were enrolled in the study but were excluded from final analyses as they did not meet the minimal performance criteria during the SJ task. Participants were excluded from final analyses if they did not perceive at least ≥ 80% of trials to be synchronous for at least one SOA across all SOAs measured. All participants had self-reported normal hearing and normal or corrected to normal vision. All recruitment and experimental procedures for this study involving human participants were approved by the Vanderbilt University Institutional Review Board and in accordance with the ethical standards of the 1964 Helsinki declaration and its later amendments or comparable ethical standards.

Stimuli

All stimuli were presented in a light and sound attenuating WhisperRoom™ (SE 2000 Series, Whisper Room Inc.) room in which participants were seated approximately 60 cm from a computer monitor. The visual stimulus consisted of a white ring on a black background that subtended 7.2° of visual space with an outer diameter of 6.0 cm and an inner diameter of 3.0 cm. Visual stimuli were presented for 8.3 ms (the duration of a single screen refresh cycle) on a monitor (Samsung syncmaster 22 inch 2233 RZ LCD) with a refresh-rate of 120 Hz. A visual fixation marker (1cm × 1cm) on a black background was present on the screen both during the intertrial interval (ITI) and throughout the duration of a trial, which included presentation of the visual stimulus (See Fig. 1A). Participants were asked to maintain fixation on the fixation marker throughout the experiment. The auditory stimulus consisted of a 10 ms, 1800 Hz tone that was presented binaurally via headphones (Sennehiser HD 558) with no interaural time or level differences. Auditory stimuli were presented at 83 dB and were calibrated using a sound level meter (Larson Davis SoundTrack® LxT2).

Fig. 1.

Fig. 1

Open in a new tab

a, A simultaneity judgment (SJ) task was utilized to measure the TBW prior to and following perceptual training. b, Illustration of correct feedback given for a response of synchronous after an objectively synchronous trial followed by subsequent asynchronous trial for which incorrect feedback is presented for a response of synchronous. c, Perceptual training sessions were individualized by determining training SOAs at three different levels of difficulty. Pre-training SJ assessment data from a representative participant (circles, report of synchrony at each SOA; solid lines, individual sigmoid curve fit to data) illustrates how training SOAs would have been selected if the participant was randomly assigned to the either the easy- (left), medium- (center), or hard- (right) difficulty condition. The value of the asynchronous training SOAs corresponds to the point at which the vertical dashed lines intersect the x-axis.

Experimental Procedure

Participants completed a simultaneity (SJ) judgment task (See Fig. 1A) during pre-training, training, and post-training trial blocks similar to SJ tasks utilized by previous studies of multisensory perceptual learning (Powers et al. 2009; Powers et al. 2012). For each trial of the SJ task, the visual and auditory stimuli were presented either synchronously or asynchronously at various stimulus onset asynchronies (SOAs). Following presentation of visual and auditory stimuli, a response screen was presented at which time participants could make a response. Participants were instructed to judge whether the visual stimulus and auditory stimulus “were synchronous, at the same time” or “were asynchronous, at different times” by pressing either 1 or 2 respectively using a keyboard. The ITI between trials was randomly jittered from 500 to 1500 ms. MATLAB (The MathWorks, Inc.) with Psychophysics Toolbox extensions (Brainard 1997; Pelli 1997) was used to present stimuli and record participant responses during the SJ task. To ensure accurate timing of the presentation of auditory and visual stimuli, SOAs were verified externally using an oscilloscope.

Experimental Protocol

Participants completed all tasks over the course of a single 1.5–2.0 hour session. The SJ task without feedback was utilized to assess temporal acuity (i.e. derive the TBW) prior to and following perceptual training. For pre-training and post-training SJ assessments, each SOA (±400, 300, 250, 200, 150, 100, 50, and 0 ms) was randomly presented 20 times (total of 300 trials) for each assessment. Following the completion of the pre-training SJ assessment, participants completed six perceptual training blocks consisting of 120 trials per block (720 total trials across all training blocks). SOAs presented during training blocks were determined on a subject-by-subject basis dependent upon the initial estimate of the TBW derived from the pre-training SJ assessment and the training group to which the participant was randomly assigned.

Perceptual Training

The structure and stimuli of the SJ task during training blocks was identical to the pre-training and post-training tasks, however, training tasks differed from the SJ assessment in that participants received trial-by-trial visual feedback following responses. Visual feedback was presented for 500 ms immediately following the participant’s response. Visual feedback consisted of a blue-green check mark or red X following correct and incorrect responses respectively (Fig. 1B).

Participants were randomly assigned to one of three training groups for which the difficulty of the task varied based on the subject’s individual capacity to discriminate asynchronous from synchronous stimuli. We defined these three difficulty levels as the SOAs at which the subjects easily detected asynchrony (easy-difficulty), were able to detect asynchrony at approximately chance (medium-difficulty), and rarely were able to detect asynchrony (hard-difficulty). To vary the task difficulty, we individually determined the training SOAs based on the participants’ performance for the pre-training SJ assessment. Correspondingly, there existed for each subject a potential set of training SOAs that would have been either been easy-, medium-, or hard-difficulty (See Fig. 1C). We employed a fitting method that has been applied to derive estimates of temporal acuity in numerous prior studies (Powers et al. 2009; Stevenson and Wallace 2013; Noel et al. 2016). Individual participants’ data on pre-training SJ assessment was fit with two sigmoid curves using the glmfit function in MATLAB to separately fit the data for the auditory-leading stimuli (utilized to estimate the auditory-leading TBW) and visual-leading stimuli (utilized to estimate visual-leading TBW). Using the estimates of the fitted data, the training SOAs were defined by the SOA (i.e. x-axis value along the distribution) at which the best-fit sigmoids y-axis value equaled a value of 10, 20, and 30% report of synchrony for the easy-difficulty group; 40, 50, and 60% report of synchrony for the medium-difficulty group; and 70, 80, and 90% report of synchrony for the hard-difficulty group (Fig. 1C). Thus, we determined six asynchronous training SOAs (3 auditory-leading; 3 visual-leading) from the individuals’ perceptual report while also presenting the truly synchronous condition.

The number of trials per SOA was not equally distributed during the training sessions. The veridical simultaneous condition had a 6:1 ratio to the asynchronous conditions such that the total number of simultaneous trials presented was equal to the total number of asynchronous trials presented. We presented an equal number of synchronous and asynchronous trials to mitigate concerns about creating a response bias.

Grand Mean SOA analysis

We first characterized the effect of perceptual training on the perceptual report of synchrony in the SJ task by measuring how the overall group probability of report of synchrony changed as a function of training status and SOA. We measured individual means at each SOA that were then averaged to produce grand averages. A repeated-measures ANOVA was conducted for each group to determine if a significant training status × SOA interaction existed. An analysis of simple effects was conducted if a significant training status × SOA interaction existed. Post hoc pairwise comparisons of simple effects were conducted with Bonferroni correction to determine if the training session for each group significantly altered the report of synchrony at each SOA. Degrees of freedom were adjusted for all interactions where sphericity was violated using a Greenhouse-Geisser correction.

TBW Analysis

Estimates of the whole, auditory-leading, and visual-leading TBWs were calculated for pre- and post-training SJ assessments using the same fitting method as described for determining the individualized training SOAs. A criterion at which the best-fit sigmoids’ y value equaled 75% report of synchrony was utilized to estimate the TBW. The auditory-leading TBW was defined as the time epoch for which the auditory-leading sigmoid fit’s y value equaled 75% report of synchrony to the point of subjective simultaneity (PSS, i.e. the SOA at which the participant is most likely to report the stimuli as synchronous); the visual-leading TBW was defined as the time epoch for which the visual-leading sigmoid fit’s y value equaled 75% report of synchrony to the PSS. The whole TBW was defined as the sum of the auditory-leading and visual-leading TBWs. Group-level analysis of the TBW was taken by calculating the mean auditory-leading, visual-leading, and whole TBW individually fit for each participant. To determine within each group if there was an effect of perceptual training, the pre-training and post-training TBWs were compared using a paired samples t-test, consistent with prior analyses of perceptual training. In order to compare the training effect across each group, we first determined the mean change in the TBW. We then conducted a one-way between subjects ANOVA to assess whether the change in the TBW differed across training difficulty levels.

Results

Perceptual Training with Easy Training Stimuli Increases Report of Simultaneity while Difficult Training Stimuli Decrease Report of Simultaneity

To determine if perceptual training altered simultaneity judgments differentially dependent upon the difficulty of the training, we conducted separate SOA × pre-/post-training status within-subjects repeated measures ANOVAs for each group (i.e. easy-, medium-, and hard-difficulty training protocols). Thus, the measure of interest was whether a significant SOA × training status interaction was present. A significant interaction was observed for the easy-difficulty group (F6.750, 107.993 = 3.692, p = .001) and hard-difficulty group (F5.832, 93.318 = 2.507, p = .028). A significant interaction was not observed for the medium-difficulty group (F4.706, 75.298 = 1.175, p = .330). Post hoc analysis of simple main effects was conducted with Bonferroni correction for the simple effect of training status at each SOA. For the easy-difficulty group (Fig. 2A), an analysis of simple main effects showed that perceptual training increased the probability of reporting simultaneity at SOAs of −400 ms (p = .017) and −300 ms (p = .024) for which the auditory lead was the greatest. In contrast, for the hard-difficulty group (Fig. 2C), perceptual training decreased the probability of reporting simultaneity at SOAs of −150 ms (p = .012), −100 ms (p < .001), 50 ms (p = .003), 100ms (p = .001), 150 ms (p < .001), 200 ms (p = .049), 250 ms (p = .014), and 300 ms (p = .026). Thus, these results illustrate that training difficulty had opposing effects on audiovisual simultaneity judgments, and suggest that training on more difficult judgments may improve temporal acuity and narrow the TBW.

Fig. 2.

Fig. 2

Open in a new tab

a, An increase in the report of synchrony at the most auditory-leading SOAs was reported for the easy-difficulty group. b, No change in the report of synchrony was observed for the medium difficulty group. c, A decrease in the report of synchrony at both auditory-leading and visual-leading SOAs was observed. Circles indicate pre-training and squares indicate post-training mean report of synchrony. Error bars, SEM. (*, p < .05, **, p < .01, ***, p < .001).

Effect of Training Difficulty on TBW

We next sought to determine the effect of each training protocol on the TBW within each group by conducting paired samples t-tests for the auditory-leading, visual-leading, and whole (i.e., left and right sides) TBWs (Fig. 3). Analysis of the whole TBW revealed that only for those individuals in the hard-difficulty group did narrowing following perceptual training (from 440 ms to 329 ms; t(16) = 2.669, p = .017). In contrast, those in the medium-difficulty (from 386 ms to 393 ms; t(16) = -.165, p = .871) and easy-difficulty (from 461 ms to 569 ms; t(16) = −2.044, p = .058) groups were not significantly affected by perceptual training (although the easy-difficulty group approached a significant increase in the TBW).

Fig. 3.

Fig. 3

Open in a new tab

a, For the easy-difficulty group, an increase in the auditory-leading TBW was observed post-training. b, No change in the TBW was observed for the medium-difficulty group. c, A decrease in the whole and auditory-leading TBW was observed post-training for the hard-difficulty group. d, A comparison of the ΔTBW between groups determined that relative ΔTBW differed between easy-difficulty and hard-difficulty groups. In contrast the ΔTBW for the medium-difficulty group was not significantly different from either the easy-difficulty or hard-difficulty groups. Error bars, SEM. (*, p < .05, **, p < .01).

We next determined if the auditory-leading and visual-leading TBWs were differentially altered by perceptual training difficulty. For the easy-difficulty group, the auditory-leading TBW significantly increased following perceptual training (from 178 ms to 258 ms; t(16) = −2.553, p = .021) while the visual-leading TBW was not altered (from 283 ms to 310 ms; t(16) = -.722, p = .481). For the medium-difficulty group, neither the auditory-leading TBW (from 171 to 161; t(16) = 1.249, p = .230) nor the visual-leading TBW (from 214 ms to 232 ms; t(16) = −.451, p = .658) were altered by perceptual training. For the hard-difficulty group, the auditory-leading TBW significantly decreased following perceptual training (from 188 ms to 153 ms; t(16) = 2.991, p = .009) while the visual-leading TBW was not altered (from 212 ms to 176 ms; t(16) = 1.797, p = .091).

Greater Enhancement of Temporal Acuity with More Difficult Training Protocols

Although we previously determined if each group experienced a change in temporal acuity following training (i.e the ΔTBW was significantly different from 0) by individual paired samples t-tests, we sought to compared if the relative change in the TBW differed when compared across groups. To determine if the change in the TBW (ΔTBW = post-training - pre-training) differed by difficulty of the training protocol (Fig. 3D) we conducted a one-way between-subjects ANOVA. This analysis demonstrated a significant effect of training protocol difficulty (F2,48 = 4.498, p = .016). Post hoc comparisons with Bonferroni correction revealed that ΔTBW for the easy-difficulty group (107 ms) was significantly different from the hard-difficulty group (−71 ms), (p = .013). Thus, the decrease in the TBW for individuals completing the hard-difficulty protocol resulted in relatively greater enhancement of temporal acuity in comparison to both within-group pre-training levels as well as relative to those individuals completing perceptual training with the easy difficulty protocol. In contrast, the ΔTBW for the medium-difficulty group (7 ms) did not differ relative to either the easy-difficulty group (p = .298) or hard-difficulty group (p = .587).

Discussion

In the current study, we observe that task difficulty during perceptual training is a critical element for eliciting multisensory perceptual learning. We find that only the perceptual training protocol that employed asynchronous audiovisual stimuli for which individuals had the greatest difficulty detecting asynchrony resulted in an enhancement of temporal acuity. Surprisingly, we also find that the training protocol for which participants were able to most easily detect the asynchrony actually impaired temporal acuity following perceptual training (i.e., resulted in an enlargement in the TBW). Our results suggest that perceptual training protocols to enhance multisensory temporal acuity can be optimized by employing only stimuli during training for which it is difficult for individuals to discriminate synchronous and asynchronous stimuli.

Numerous factors, including stimulus properties and reinforcement signals, interact to promote perceptual learning (Seitz and Dinse 2007). While passive exposure to asynchronous stimuli has been previously observed to alter the perceptual representation of perceived synchrony, both over the course of many trials (Fujisaki et al. 2004; Vroomen et al. 2004; Van der Burg et al. 2015a) and trial-to-trial (Van der Burg et al. 2013; Van der Burg et al. 2015b), passive exposure to asynchronous does not seem to enhance temporal acuity. Although these prior studies, along with those studies that observe enhancement of temporal acuity with feedback (Powers et al. 2009), suggest that a reinforcement signal was needed to enhance multisensory temporal acuity, the properties of the stimuli employed during perceptual training have not been systematically explored (Powers et al. 2009). Our findings suggest that while feedback may be an important element for enhancing multisensory temporal acuity, there is an interaction between the feedback signal and difficulty of the perceptual training task. Powers et al. (2009) first demonstrated that perceptual training enhances temporal acuity by using a fixed set of SOAs for asynchronous stimuli during training sessions that were independent of an individual’s temporal acuity. In this study, they reported that perceptual training did not enhance temporal acuity for a subset of the participants whose TBWs tended to be narrower prior to perceptual training relative to other participants for which training enhanced temporal acuity. Based on the results of the current study, we hypothesize that the SOAs employed by Powers et al. during perceptual training sessions were of insufficient difficulty to elicit perceptual learning in this subset of participants who failed to demonstrate training-mediated changes. This, along with our findings, suggests that individual differences in temporal acuity prior to perceptual training must be accounted for to optimize the effects of perceptual learning.

Although it remains possible that the effects of perceptual training are due to changes in an internal criterion (as the SJ task, like most 2-alternative tasks, is subject to some degree of decisional bias), prior evidence suggests that the changes in temporal acuity we observe are not simply the result of a change in response criterion. Earlier reports of multisensory perceptual training observe similar changes in temporal acuity following training on various tasks susceptible to greater or lesser bias, suggesting that the observed changes in temporal acuity cannot be accounted for by changes in response criterion (Powers et al. 2009). Similarly, the changes in temporal acuity elicited by perceptual training using a SJ task appear to be durable (Powers et al. 2009) and generalize to improvements in visual temporal acuity without evidence of a response bias (Powers III et al. 2016). As the SJ task and other commonly used tasks to measure temporal perception (e.g., temporal order judgments [TOJ]) are invariably subject to some degree of decisional bias (Van Eijk et al. 2008), we can not rule out the contribution of some criterion changes to our observed training related effects. Indeed, we argue that the most parsimonious explanation likely involves changes at both the low-level stages of sensory processing as well as at the higher-order stages of perception. While it is difficult to determine that no change in criterion occurs following perceptual training, recently, it has been suggested that prolonged changes in the calibration of multisensory temporal representations may be elicited primarily from changes in criterion at the higher-order decisional level whereas as transient trial-to-trial changes in multisensory temporal representations (i.e. rapid recalibration) result from changes at low-level sensory processes (Van der Burg et al. 2015a). Furthermore, changes at decisional stages of perception have been recognized to contribute to other forms of perceptual learning (Law and Gold 2008; Law and Gold 2010). Future studies will be necessary to determine the contribution of changes in top-down and bottom-up processing to the changes in temporal acuity we observe following perceptual training.

We were surprised to observe that for those in the easy-difficulty training protocol, presenting SOAs during training for which asynchrony was easily detected increased the report of synchrony at the SOAs with the greatest auditory-lead, thus resulting in a widening of the auditory-leading TBW. While this finding initially seemed surprising as we expected that the presence of a feedback signal during the training session would either maintain or enhance temporal acuity, it supports the earlier report by Powers et. al (2009) who observed that some individuals with narrow TBWs prior to training experienced an increase in their TBW following training in their protocol (Powers et al. 2009). One possible explanation for this paradoxical effect is that the feedback this group received was not actively informative in shaping their perceptual decisions, but rather served to simply passively confirm these decisions. The lack of an informative feedback signal may be similar to the task without feedback, making these individuals more susceptible to adaptation effects. Prior studies shown that extensive, passive exposure (i.e., in the absence of feedback) to asynchronous stimuli can widen the TBW (Fujisaki et al. 2004; Vroomen et al. 2004; Navarra et al. 2005; Navarra et al. 2007). Furthermore,Powers et al. (2009) report an increase in the TBW for individuals who were passively exposed to the same stimuli as those who received feedback during the training session (Powers et al. 2009). An alternative explanation is that the shift in the auditory-leading TBW for the easy-difficulty group resulted from an adaptation effect produced by presenting particularly large SOAs. Interestingly, passive exposure to audiovisual stimuli with auditory-leading SOAs has been shown to elicit the greatest change in temporal representations (Fujisaki et al. 2004; Vroomen et al. 2004). It is possible that relative to the other conditions, the large SOAs at which participants could easily detect asynchrony were sufficient enough to widen the TBW. Although the distribution appears to be shifted towards an auditory-leading stimulus for this portion of the TBW, the majority of the effect is observed at only the largest auditory-leading SOAs, suggesting the change we observe is not a true adaptation effect as all auditory-leading SOAs are not uniformly affected following training.

Our findings collectively suggest that in order to optimize perceptual training protocols for enhancing temporal acuity, the difficulty of the perceptual training task as well as pre-training differences in temporal acuity must be considered. Based on our findings, one area of future research to optimize perceptual training would be to determine if increasing the task difficulty following an initial improvement would further enhance temporal acuity. Progressively adaptive training methods might be the ideal method for improving temporal acuity. Another issue for future research to address is possibility that increasing task difficulty impairs generalization of the changes in temporal acuity to stimuli with different properties as has been observed for visual perceptual learning. Although the reverse hierarchy theory of visual perceptual learning would suggest that if multisensory perceptual learning occurs at lower-levels of sensory processing, difficult tasks would not generalize (Ahissar and Hochstein 2004), evidence suggests that multisensory perceptual learning engages higher order cortical regions, such as the posterior superior temporal sulcus (pSTS), and thus may not exhibit entirely similar properties as unisensory perceptual learning (Powers et al. 2012). Determining how to improve perceptual training by understanding these additional contributions of task difficulty will be important in tailoring future training regimens, particularly in efforts to use such training to improve perceptual function in those with disorders such as autism and dyslexia.

Acknowledgments

The project was supported by NIH grants CA183492 and HD083211, the Simons Foundation Autism Research Initiative and the Wallace Foundation.

Footnotes

The authors have no conflicts of interest to declare.

References

  1. Ahissar M, Hochstein S. Task difficulty and the specificity of perceptual learning. Nature. 1997;387:401–406. doi: 10.1038/387401a0. [DOI] [PubMed] [Google Scholar]
  2. Ahissar M, Hochstein S. The reverse hierarchy theory of visual perceptual learning. Trends in cognitive sciences. 2004;8:457–464. doi: 10.1016/j.tics.2004.08.011. [DOI] [PubMed] [Google Scholar]
  3. Anguera JA, Boccanfuso J, Rintoul JL, et al. Video game training enhances cognitive control in older adults. Nature. 2013;501:97–101. doi: 10.1038/nature12486. [DOI] [PMC free article] [PubMed] [Google Scholar]
  4. Brainard DH. The psychophysics toolbox. Spatial vision. 1997;10:433–436. [PubMed] [Google Scholar]
  5. Chen Y-C, Shore DI, Lewis TL, Maurer D. The development of the perception of audiovisual simultaneity. Journal of experimental child psychology. 2016;146:17–33. doi: 10.1016/j.jecp.2016.01.010. [DOI] [PubMed] [Google Scholar]
  6. DeLoss DJ, Watanabe T, Andersen GJ. Optimization of perceptual learning: Effects of task difficulty and external noise in older adults. Vision research. 2014;99:37–45. doi: 10.1016/j.visres.2013.11.003. [DOI] [PMC free article] [PubMed] [Google Scholar]
  7. Deveau J, Ozer DJ, Seitz AR. Improved vision and on-field performance in baseball through perceptual learning. Current Biology. 2014;24:R146–R147. doi: 10.1016/j.cub.2014.01.004. [DOI] [PMC free article] [PubMed] [Google Scholar]
  8. Diederich A, Colonius H. Bimodal and trimodal multisensory enhancement: effects of stimulus onset and intensity on reaction time. Perception & psychophysics. 2004;66:1388–1404. doi: 10.3758/bf03195006. [DOI] [PubMed] [Google Scholar]
  9. Dixon NF, Spitz L. The detection of auditory visual desynchrony. Perception. 1980;9:719–721. doi: 10.1068/p090719. [DOI] [PubMed] [Google Scholar]
  10. Fujisaki W, Shimojo S, Kashino M, Nishida Sy. Recalibration of audiovisual simultaneity. Nature neuroscience. 2004;7:773–778. doi: 10.1038/nn1268. [DOI] [PubMed] [Google Scholar]
  11. Hillock AR, Powers AR, Wallace MT. Binding of sights and sounds: Age-related changes in multisensory temporal processing. Neuropsychologia. 2011;49:461–467. doi: 10.1016/j.neuropsychologia.2010.11.041. [DOI] [PMC free article] [PubMed] [Google Scholar]
  12. Hillock-Dunn A, Wallace MT. Developmental changes in the multisensory temporal binding window persist into adolescence. Developmental Science. 2012;15:688–696. doi: 10.1111/j.1467-7687.2012.01171.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
  13. Law C-T, Gold JI. Neural correlates of perceptual learning in a sensory-motor, but not a sensory, cortical area. Nat Neurosci. 2008;11:505–513. doi: 10.1038/nn2070. doi: http://www.nature.com/neuro/journal/v11/n4/suppinfo/nn2070_S1.html. [DOI] [PMC free article] [PubMed] [Google Scholar]
  14. Law CT, Gold JI. Shared mechanisms of perceptual learning and decision making. Topics in cognitive science. 2010;2:226–238. doi: 10.1111/j.1756-8765.2009.01044.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
  15. Meredith MA, Nemitz JW, Stein BE. Determinants of multisensory integration in superior colliculus neurons I. Temporal factors. The Journal of neuroscience. 1987;7:3215–3229. doi: 10.1523/JNEUROSCI.07-10-03215.1987. [DOI] [PMC free article] [PubMed] [Google Scholar]
  16. Mishra J, Anguera JA, Gazzaley A. Video Games for Neuro-Cognitive Optimization. Neuron. 2016;90:214–218. doi: 10.1016/j.neuron.2016.04.010. [DOI] [PubMed] [Google Scholar]
  17. Mishra J, de Villers-Sidani E, Merzenich M, Gazzaley A. Adaptive training diminishes distractibility in aging across species. Neuron. 2014;84:1091–1103. doi: 10.1016/j.neuron.2014.10.034. [DOI] [PMC free article] [PubMed] [Google Scholar]
  18. Mishra J, Gazzaley A. Harnessing the neuroplastic potential of the human brain & the future of cognitive rehabilitation. Frontiers in human neuroscience. 2014;8:218. doi: 10.3389/fnhum.2014.00218. [DOI] [PMC free article] [PubMed] [Google Scholar]
  19. Navarra J, Soto-Faraco S, Spence C. Adaptation to audiotactile asynchrony. Neuroscience letters. 2007;413:72–76. doi: 10.1016/j.neulet.2006.11.027. [DOI] [PubMed] [Google Scholar]
  20. Navarra J, Vatakis A, Zampini M, Soto-Faraco S, Humphreys W, Spence C. Exposure to asynchronous audiovisual speech extends the temporal window for audiovisual integration. Cognitive Brain Research. 2005;25:499–507. doi: 10.1016/j.cogbrainres.2005.07.009. [DOI] [PubMed] [Google Scholar]
  21. Noel J-P, Łukowska M, Wallace M, Serino A. Multisensory simultaneity judgment and proximity to the body. Journal of Vision. 2016;16:21–21. doi: 10.1167/16.3.21. [DOI] [PMC free article] [PubMed] [Google Scholar]
  22. Pelli DG. The VideoToolbox software for visual psychophysics: Transforming numbers into movies. Spatial vision. 1997;10:437–442. [PubMed] [Google Scholar]
  23. Powers AR, Hevey MA, Wallace MT. Neural correlates of multisensory perceptual learning. The Journal of Neuroscience. 2012;32:6263–6274. doi: 10.1523/JNEUROSCI.6138-11.2012. [DOI] [PMC free article] [PubMed] [Google Scholar]
  24. Powers AR, Hillock AR, Wallace MT. Perceptual training narrows the temporal window of multisensory binding. The Journal of Neuroscience. 2009;29:12265–12274. doi: 10.1523/JNEUROSCI.3501-09.2009. [DOI] [PMC free article] [PubMed] [Google Scholar]
  25. Powers AR, III, Hillock-Dunn A, Wallace MT. Generalization of multisensory perceptual learning. Scientific Reports. 2016;6:23374. doi: 10.1038/srep23374. [DOI] [PMC free article] [PubMed] [Google Scholar]
  26. Seitz AR, Dinse HR. A common framework for perceptual learning. Current opinion in neurobiology. 2007;17:148–153. doi: 10.1016/j.conb.2007.02.004. [DOI] [PubMed] [Google Scholar]
  27. Seitz AR, Nanez JE, Sr, Holloway S, Tsushima Y, Watanabe T. Two cases requiring external reinforcement in perceptual learning. Journal of Vision. 2006;6:966–973. doi: 10.1167/6.9.9. [DOI] [PubMed] [Google Scholar]
  28. Setti A, Stapleton J, Leahy D, Walsh C, Kenny RA, Newell FN. Improving the efficiency of multisensory integration in older adults: Audio-visual temporal discrimination training reduces susceptibility to the sound-induced flash illusion. Neuropsychologia. 2014;61:259–268. doi: 10.1016/j.neuropsychologia.2014.06.027. [DOI] [PubMed] [Google Scholar]
  29. Stein BE, Stanford TR. Multisensory integration: current issues from the perspective of the single neuron. Nature Reviews Neuroscience. 2008;9:255–266. doi: 10.1038/nrn2331. [DOI] [PubMed] [Google Scholar]
  30. Stein BE, Stanford TR, Rowland BA. Development of multisensory integration from the perspective of the individual neuron. Nature Reviews Neuroscience. 2014;15:520–535. doi: 10.1038/nrn3742. [DOI] [PMC free article] [PubMed] [Google Scholar]
  31. Stevenson RA, Siemann JK, Schneider BC, Eberly HE, Woynaroski TG, Camarata SM, Wallace MT. Multisensory temporal integration in autism spectrum disorders. The Journal of Neuroscience. 2014;34:691–697. doi: 10.1523/JNEUROSCI.3615-13.2014. [DOI] [PMC free article] [PubMed] [Google Scholar]
  32. Stevenson RA, Wallace MT. Multisensory temporal integration: task and stimulus dependencies. Experimental Brain Research. 2013;227:249–261. doi: 10.1007/s00221-013-3507-3. [DOI] [PMC free article] [PubMed] [Google Scholar]
  33. Stevenson RA, Wilson MM, Powers AR, Wallace MT. The effects of visual training on multisensory temporal processing. Experimental Brain Research. 2013;225:479–489. doi: 10.1007/s00221-012-3387-y. [DOI] [PMC free article] [PubMed] [Google Scholar]
  34. Stevenson RA, Zemtsov RK, Wallace MT. Individual Differences in the Multisensory Temporal Binding Window Predict Susceptibility to Audiovisual Illusions. Journal of Experimental Psychology-Human Perception and Performance. 2012;38:1517–1529. doi: 10.1037/a0027339. [DOI] [PMC free article] [PubMed] [Google Scholar]
  35. Sumby WH, Pollack I. Visual contribution to speech intelligibility in noise. The journal of the acoustical society of america. 1954;26:212–215. [Google Scholar]
  36. Van der Burg E, Alais D, Cass J. Rapid Recalibration to Audiovisual Asynchrony. Journal of Neuroscience. 2013;33:14633–14637. doi: 10.1523/JNEUROSCI.1182-13.2013. [DOI] [PMC free article] [PubMed] [Google Scholar]
  37. Van der Burg E, Alais D, Cass J. Audiovisual temporal recalibration occurs independently at two different time scales. Scientific reports. 2015a;5 doi: 10.1038/srep14526. [DOI] [PMC free article] [PubMed] [Google Scholar]
  38. Van der Burg E, Orchard-Mills E, Alais D. Rapid temporal recalibration is unique to audiovisual stimuli. Experimental brain research. 2015b;233:53–59. doi: 10.1007/s00221-014-4085-8. [DOI] [PubMed] [Google Scholar]
  39. Van Eijk RL, Kohlrausch A, Juola JF, van de Par S. Audiovisual synchrony and temporal order judgments: Effects of experimental method and stimulus type. Perception & psychophysics. 2008;70:955–968. doi: 10.3758/pp.70.6.955. [DOI] [PubMed] [Google Scholar]
  40. Vroomen J, Keetels M. Perception of intersensory synchrony: a tutorial review. Attention, Perception, & Psychophysics. 2010;72:871–884. doi: 10.3758/APP.72.4.871. [DOI] [PubMed] [Google Scholar]
  41. Vroomen J, Keetels M, De Gelder B, Bertelson P. Recalibration of temporal order perception by exposure to audio-visual asynchrony. Cognitive brain research. 2004;22:32–35. doi: 10.1016/j.cogbrainres.2004.07.003. [DOI] [PubMed] [Google Scholar]
  42. Wallace MT, Stevenson RA. The construct of the multisensory temporal binding window and its dysregulation in developmental disabilities. Neuropsychologia. 2014;64:105–123. doi: 10.1016/j.neuropsychologia.2014.08.005. [DOI] [PMC free article] [PubMed] [Google Scholar]
  43. Wang Y, Song Y, Qu Z, Ding Y. Task difficulty modulates electrophysiological correlates of perceptual learning. International Journal of Psychophysiology. 2010;75:234–240. doi: 10.1016/j.ijpsycho.2009.11.006. [DOI] [PubMed] [Google Scholar]

RESOURCES