Skip to main content
PMC home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2018 Jan 1.
Published in final edited form as: Schizophr Res. 2016 Oct 13;179:97–103. doi: 10.1016/j.schres.2016.09.035

The associations between multisensory temporal processing on symptoms of schizophrenia

Ryan A Stevenson 1,2, Sohee Park 3, Channing Cochran 3, Lindsey G McIntosh 3, Jean-Paul Noel 4, Morgan D Barense 5,6, Susanne Ferber 5,6, Mark T Wallace 4,7,8,9,10
PMCID: PMC5463449  NIHMSID: NIHMS823253  PMID: 27746052

Abstract

Recent neurobiological accounts of schizophrenia have included an emphasis on changes in sensory processing. These sensory and perceptual deficits can have a cascading effect onto higher-level cognitive processes and clinical symptoms. One form of sensory dysfunction that has been consistently observed in schizophrenia is altered temporal processing. In this study, we investigated temporal processing within and across the auditory and visual modalities in individuals with schizophrenia (SCZ) and age-matched healthy controls. Individuals with SCZ showed auditory and visual temporal processing abnormalities, as well as multisensory temporal processing dysfunction that extended beyond that attributable to unisensory processing dysfunction. Most importantly, these multisensory temporal deficits were associated with the severity of hallucinations. This link between atypical multisensory temporal perception and clinical symptomatology suggest that clinical symptoms of schizophrenia may be at least partly a result of cascading effects from (multi)sensory disturbances. These results are discussed in terms of underlying neural bases and the possible implications for remediation.

Keywords: Multisensory Integration, Schizophrenia, Temporal Processing, Speech Perception, Hallucinations, Audiovisual

1.0 Introduction

Hallucinations are a positive symptom in schizophrenia (SCZ) that can present as false perceptions in any sensory modality, but commonly take the form of perceived auditory voices. They are often conceptualized as false attribution of internal voices to an external source. As such, hallucinations in SCZ are frequently linked to the audiovisual speech-perception network, including areas of superior temporal and inferior frontal (i.e., Broca’s) cortex (Jardri et al., 2011). One cognitive operation of this network is the integration of information across the auditory and visual systems, forming coherent percepts that comprise our conscious experience. Speech is a powerful example of audiovisual, though integration extends to all manner of sensory inputs: we seamlessly bind together audible speech signals with their associated visual cues, affording substantial behavioral and perceptual benefits, ranging from faster response times (Raab, 1962) to improved speech perception (Sumby and Pollack, 1954) in healthy participants but not as much in SCZ patients. For example, seeing a speaker’s visual articulation enhances speech perception under noisy conditions in healthy participants but less so in SCZ patients (Ross et al., 2007). Similarly, SCZ patients are less susceptible to the McGurk effect (Pearl et al., 2009), where the mouth movements an individual sees can alter what they believe to “hear” a speaker to be saying (McGurk and MacDonald, 1976), despite preserved unisensory abilities (Ross et al., 2007).

Impaired sensory integration is a hallmark neurological “soft sign” of SCZ (Heinrichs and Buchanan, 1988) that is often noted at the time of an individual’s first psychotic episode and is correlated with SCZ symptomatology (Williams et al., 2010). Most germane to this report is the possible link between alterations in sensory integration and positive symptoms in SCZ, most notably hallucinations (Postmes et al., 2014). Exploring an integration-hallucination link is motivated by the overlap in the neural substrates for audiovisual integration and hallucinations, specifically in regions of the audiovisual speech perception network. For example, SCZ is associated with structural (Kim et al., 2003) and functional changes within the superior temporal cortex (Surguladze et al., 2001; Szycik et al., 2009). This same area of cortex is heavily implicated in multisensory temporal processing (Stevenson et al., 2010). Furthermore, individuals with SCZ exhibit alterations in temporal processing (Carroll et al., 2008; Davalos et al., 2002; Elvevag et al., 2003; Foucher et al., 2007; Freedman, 1974; Giersch et al., 2009; Lalanne et al., 2012; Tysk, 1983a, b; Volz et al., 2001), and impaired audiovisual temporal precision in SCZ has been linked to inaccurately attributing auditory components of speech to temporally disparate visual speech signals (Martin et al., 2013).

Given the relationship between temporal processing and sensory integration, and links between sensory integration and hallucination in SCZ, we hypothesize that impaired temporal perception in SCZ may be associated with hallucinations in SCZ. To investigate this, we first measured auditory, visual, and multisensory temporal perception in SCZ patients and a group of matched controls, verifying the presence of temporal dysfunction in SCZ and assessing if temporal-perception deficits were uniquely multisensory. Second, and of paramount importance, we measured the severity of hallucinations in SCZ participants with the a priori prediction that changes in multisensory temporal processing would be predictive of hallucinations. This finding would point to shared mechanistic substrates for changes in audiovisual temporal integration and the presence and severity of hallucinations.

2.0 Methods and Materials

2.1 Overview

Participants completed four behavioral tasks: two unisensory timing tasks in which participants performed temporal order judgments (TOJ; “Which came first?”) with either auditory or visual stimuli, and two audiovisual timing tasks in which participants performed audiovisual simultaneity judgments (SJ; “Same time or different time?”), one with speech stimuli and one with simple flash-beep stimuli. Finally, participants completed standard metrics assessing SCZ symptomatology. Protocols were approved by Vanderbilt University Institutional Review Board and participants gave written informed consent to participate in the study.

2.2 Participants

Thirty-two participants competed the study, half who met the DSM-IV criteria for schizophrenia (SCZ; mean age=42.3±8.9 years, 8 female), and half healthy controls (HC; mean age=41.9±9.3 years, 10 female) matched for age (t(30)=0.12, p=0.91) and gender (χ2=0.51, p=0.48). SCZ symptoms were rated using the Brief Psychiatric Rating Scale (BPRS; mean=15.4±7.9), the Scale for Assessment of Positive Symptoms (SAPS; mean=13.7±11.7), and the Scale for Assessment of Negative Symptoms (SANS; mean=32.2±15.9), with hallucination severity derived from the SAPS global rating of hallucination scores (mean=1.6±1.6).

2.3 Stimuli and Procedures

For all tasks, participants were asked to fixate towards a cross, and were actively monitored for compliance. Visual stimuli were presented on a screen approximately 60 cm from the participants. Auditory stimuli were presented through centrally aligned speakers. Tasks and trials were randomized in all cases. All responses were made via button press.

2.3.1 Unisensory timing tasks

For the unisensory auditory timing task, participants were presented with a pair of auditory beeps consisting of one high- and one low-pitch (1000 and 500 Hz) beep each (duration=7ms), and performed a temporal order judgment task (TOJ; “which came first?”). Individual unisensory-auditory beeps within each pair were separated by SOAs of 10, 20, 35, 50, 75, 100, 150, 200, and 250ms. Twenty trials at each SOA were presented.

For the unisensory visual timing task, participants were presented with two white circles on a black background, one above and one below a fixation cross (duration=10ms) and performed a TOJ task. Individual unisensory-visual flashes within each pair were separated by SOAs of 10, 20, 30, 40, 60, 80, 100, and 150ms. Twenty trials at each SOA were presented.

Temporal order judgment tasks were used with unisensory tasks as opposed to the SJ tasks used with multisensory stimuli based on previously collected data. When an SJ tasks is used with unisensory stimuli, most participants were near ceiling performance at detecting asynchronies even at the shortest SOAs.

2.3.2 Audiovisual timing tasks

In the audiovisual tasks, participants were presented with an auditory and a visual stimulus, and performed a simultaneity judgment task (SJ; “Were the auditory and visual stimuli presented at the same time?”). Two types of audiovisual stimuli were presented, each in a separate run. One set of stimuli were simple flash-beeps pairs. The visual flashes consisted of a white ring circumscribing the visual fixation cross on a black background presented for 10ms. Auditory beep stimuli consisted of a 3500 Hz pure tone with a duration of 7ms. For simple flash-beeps, SOAs included 0, ±10, ±20, ±50, ±80, and ±100 to 300ms in 50ms intervals. Twenty trials at each SOA were presented.

The second type of audiovisual stimuli were single syllable utterances, which were selected from a stimulus set that has been previously used successfully in studies of multisensory integration (Baum et al., 2015; Quinto et al., 2010; Stevenson et al., 2014b; Stevenson and Wallace, 2013). Stimuli consisted of two audiovisual clips of a female speaker uttering single instances of the syllables “ga” and “ba”. Visual stimuli were grayscale, and spanned 18.25cm per side, and two seconds in duration, with each presentation containing the entire articulation of the syllable, including pre-articulatory gestures. For speech stimuli, SOAs included 0 to ±300ms in 50ms intervals and ±400ms.

2.4 Analysis of behavioral tasks

In both auditory and visual unisensory TOJ, individuals’ mean responses were calculated at each SOA (Figure 1A–B). A general linear model (GLM) was used to predict responses based on the categorical factor of diagnosis and the continuous factor of SOA. Additionally, each individual’s mean responses were fit with a sigmoid curve, and the 75% threshold was extracted from this function (Figure 1C) for both the visual and auditory tasks. Thresholds were then compared across groups, and subsequently used to predict multisensory temporal processing abilities. Twenty trials at each SOA were presented.

Figure 1. Temporal Perception.

Figure 1

Open in a new tab

Individuals with (red) and without (black) schizophrenia completed a series of temporal tasks. In unisensory tasks, individuals with SCZ showed less temporal precision during auditory (Panel A) and visual (Panel B) perception. Thresholds for both can be seen in Panel C. Audiovisual temporal perception was also less temporally precise in schizophrenia with both simple flash-beep stimuli (Panel D) and speech stimuli (Panel E). Thresholds for both can be seen in Panel F. Error bars represent standard error.

In both audiovisual SJ tasks, individuals’ mean responses were calculated at each SOA (Figure 1D–E). Individuals’ mean responses from SJ tasks were used to calculate a temporal binding window (TBW) using a well-established method (Fister et al., 2016; Noel et al., 2016; Schlesinger et al., 2014; Stevenson et al., 2012a; Stevenson et al., 2014a; Stevenson et al., 2014b; Stevenson and Wallace, 2013; Stevenson et al., 2013; Stevenson et al., 2012b). Two psychometric sigmoid functions were fit to rates of perceived synchrony across SOAs; one to the audio-first (left) presentations and a second to the visual-first presentations (right). To account for non-zero points of subjective simultaneity (PSS), the SOA at which these two sigmoid functions crossed was extracted. If this point was greater or less than the next closest data point, two new sigmoid functions were fit splitting the data at the SOA at which the original sigmoid functions crossed. This process was continued in an iterative manner until the SOA at which best-fit sigmoid functions crossed fell between the two data points at which the data were split. Based off these final curves, the time interval between the 75% threshold of their left, auditory-leading curve and their right, visual-leading curve was calculated as the individual’s TBW (Figure 1F).

All measures were compared between groups. Consecutive multiple hierarchical regressions were then conducted to investigate (a) the ability of unisensory temporal processing to account for impaired multisensory processing in SCZ, and (b) the ability of audiovisual temporal processing to predict hallucination symptomatology in SCZ.

3.0 Results

3.1 Unisensory temporal perception

Auditory temporal perception was indexed via an auditory TOJ task. Responses were averaged for each SOA for each individual. A mixed linear model (MLM) was then used to measure the impact of SOA and diagnosis on response accuracy (Figure 1A). In this 2-factor MLM, both factors of diagnosis (F(1, 228.7)=84.57, p<0.001) and SOA (F(1, 26.2)=58.84, p<0.001) significantly contributed, but the two did not interact (F(1, 56.0)=0.06, p<0.81). To quantify this between-group difference, thresholds (75%-correct performance) for each individual were compared between diagnostic groups, with SCZ patients showing poorer auditory temporal acuity. (Figure 1C; meanSCZ=140ms, meanHC=53ms, t(30)=2.70, p=0.01, d=0.99).

Visual temporal perception was indexed via a visual temporal order judgment (TOJ) task. Responses were averaged across SOAs for each individual. An MLM was then used to measure the impact of SOA and diagnosis on response accuracy (Figure 1B). In this 2-factor MLM, both factors of diagnosis (F(1, 57.7)=28.51, p<0.001) and SOA (F(1, 214.5)=234.70, p<0.001) significantly contributed, but the two did not interact (F(1, 214.5)=0.08, p<0.78). To quantify this between-group difference, 75% thresholds were compared between diagnostic groups, with SCZ patients showing poorer visual acuity (Figure 1C; meanSCZ=45ms, meanHC=26ms, t(30)=2.22, p=0.03, d=0.81).

3.2 Multisensory temporal perception

Audiovisual temporal acuity was tested using SJ. Data from the SJ task using simple flash-beep stimuli were analyzed with a mixed-model, two-way, repeated-measures ANOVA (diagnosis x SOA). This analysis revealed significant effects of diagnosis (F(1,18)=13.18, p=0.001 , partial-η2=0.31), SOA (F(1,18)=68.33, p<0.001 , partial-η2=0.70), and an interaction between the two (F(1,18)=7.94, p<0.001 , partial-η2=0.21) (Figure 1D). Follow-up t-tests were conducted at each SOA (see Figure 1D and Supplementary Table 1 for detailed statistics). Individuals’ temporal binding windows (TBW) were calculated and groups were compared (Figure 1D; t(30)=4.61, p=6.94e−5, d=1.68). Controls showed a mean TBW of 240ms±114ms, and SCZs exhibited a significantly enlarged mean TBW of 550ms±243ms. These results suggest that multisensory temporal acuity is less precise in individuals with SCZ (Figure 1F).

SJ tasks using more complex audiovisual stimuli specifically speech stimuli were analyzed in an identical manner. A mixed-model, two-way, repeated-measures ANOVA showed significant effects of diagnosis (F(1,18)=12.65, p=0.001 , partial-η2=0.30), SOA (F(1,18)=69.66, p<0.001 , partial-η2=0.70), and an interaction between the two (F(1,18)=4.54, p<0.001 , partial-η2=0.13). Follow-up t-tests were conducted for each SOA (see Figure 1E and Supplementary Table 2 for detailed statistics). Controls showed a mean TBW of 432ms±181ms, and the SCZ group exhibited a significantly wider mean TBW of 682ms ±182ms (t(30)=3.90, p=0.0005, d=1.42). These results suggest that multisensory temporal acuity for audiovisual speech stimuli is less precise in individuals with SCZ (Figure 1F).

3.3 Predicting multisensory temporal perception

An outstanding question in the prior analyses is whether the unisensory (i.e., auditory and visual) temporal processing changes in SCZ account for the changes in audiovisual temporal perception. Hierarchical multiple regressions were used to identify which factors predicted multisensory temporal acuity as indexed via the TBW. For data derived from the SJ task using both simple flash-beep and speech stimuli, a three-model hierarchical regression was run. Factors in Model 1 included the demographic variables of age and gender, Model 2 added unisensory auditory and visual temporal acuity (i.e., TOJ performance), and diagnosis was added as a predictor in Model 3. Detailed statistical results can be seen in Table 1. In a synopsis of these data, unisensory auditory, but not visual, TOJ performance was predictive of TBW width. In model 3, SCZ diagnosis also predicted a wider TBW beyond what was accounted for by unisensory deficits.

Table 1.

Hierarchical multiple regression predicting temporal binding windows

Flash-beep Stimuli Speech Stimuli
Predictor Partial correlation (pr) p-Value Predictor Partial Correlation (pr) p-value
Step 1: R = 0.06, F-change(2,29) = 0.05, p-change = 0.95 Step 1: R = 0.29, F-change(2,29) = 1.33, p-change = 0.28
Gender 0.02 0.93 Gender −0.27 0.62
Age 0.06 0.78 Age −0.06 0.27
Step 2: R = 0.61, F-change (2,27) = 7.93, p-change = 0.002 Step 2: R = 0.64, F-change(2,27) = 7.57, p-change = 0.002
Gender −0.01 0.96 Gender −0.40 0.03
Age −0.16 0.40 Age −0.29 0.13
Auditory threshold 0.52 0.004 Auditory threshold 0.38 0.044
Visual threshold −0.02 0.94 Visual threshold 0.22 0.25
Step 3: R = 0.74, F-change (1,26) = 10.60, p-change = 0.003 Step 3: R = 0.71, F-change(1,26) = 4.90, p-change = 0.036
Gender 0.13 0.50 Gender −0.26 0.08
Age −0.11 0.58 Age −0.34 0.18
Auditory threshold 0.46 0.015 Auditory threshold 0.30 0.13
Visual threshold −0.15 0.45 Visual threshold 0.15 0.45
Diagnosis 0.54 0.003 Diagnosis 0.40 0.036
Open in a new tab

Significant results of added predictors are shown in bold.

3.4 Multisensory temporal precision and hallucinations

Global hallucination severity measures in participants with SCZ were extracted from the SAPS. Initial correlations were conducted between hallucination severity and participants’ audiovisual temporal precision (i.e., TBW width). The widths of the TBW measured using both flash-beep stimuli (R=−0.52, p=0.038) and speech stimuli (R=−0.65, p=0.006) were significantly correlated with hallucination severity, in that the wider the TBW, the less severe their hallucinations were (Figure 2). To control for the effects of age and gender, a hierarchical regression was conducted, revealing that (a) neither gender nor age accounted for a significant portion of variance (Ps>0.37), and (b) audiovisual temporal precision was predictive of hallucination severity even when controlling for demographic variables (R=0.76, F-change(2,11)=5.76, p =0.019).

Figure 2. Audiovisual temporal precision predicts severity of hallucinations in SCZ.

Figure 2

Open in a new tab

Temporal binding windows with both simple flashes and beeps (Panel A) and speech (panel B) were significantly related to hallucination severity as measured by the Scale for Assessment of Positive Symptoms.

Finally, an exploratory correlation analysis was conducted to detect any possible relationships between audiovisual temporal precision and overall positive and negative symptomatology. No relationships were observed for either flash-beep (Rs=0.23 and 0.20, respectively) or speech (Rs=0.08 and 0.12, respectively) stimuli.

4.0 Discussion

This study provides a novel view into the relationships between impaired temporal processing, multisensory integration, and hallucinations in SCZ. Three main findings are evident in the data. First, this study confirms that individuals with SCZ show decreased temporal acuity in both auditory and visual perception, as well as in audiovisual temporal perception. Second, SCZ participants exhibit impairments in multisensory temporal acuity that extend beyond these unisensory changes, suggesting a level of specificity for these multisensory changes. Finally, and perhaps most importantly, multisensory temporal perception predicted one aspect of SCZ symptomatology, specifically the severity of hallucinations.

The ability to integrate sensory inputs into a unified perceptual whole is an essential cognitive process. The ability to link what is seen and heard, such as linking a voice one hears to the sight of the speaker’s mouth movements, creates a coherent representation of the external world. Changes in sensory integration are clinically significant features in SCZ, as impaired integration is a common neurological soft sign of SCZ (Heinrichs and Buchanan, 1988). Here, we provide novel evidence suggesting that aberrant audiovisual temporal perception in SCZ is significantly predictive of hallucinations, a clinical symptom typically associated with SCZ. Dysfunction in binding information in SCZ was described as far back as 1911 (Bleuler, 1911), and persists in modern theories of SCZ, specifically for patients experiencing auditory hallucinations (Behrendt and Young, 2004). These observations, taken together with the increasing evidence for sensory integration deficits in SCZ, and the accumulating recognition that this associative binding process is foundational in the scaffolding of a sense of ‘self’ drew researchers to propose that perceptual incoherence may manifest in many of the clinical symptoms of SCZ (Postmes et al., 2014).

Neuroanatomical evidence also provides strong links between hallucinations in SCZ and audiovisual temporal processing, particularly in the audiovisual speech perception network, including superior temporal regions and inferior frontal regions. Hallucinations in SCZ have commonly been linked to atypical anatomy and function in the posterior temporal cortex, including regions of the middle temporal gyrus (MTG), superior temporal sulcus (STS), and superior temporal gyrus (STG). Direct stimulation of STS induces auditory hallucinations (Postmes et al., 2014), and during hallucinations in individuals with SCZ, there are changes in the neural activation patterns in STS (Jardri et al., 2011). Additionally, both anatomical (Aguayo, 1990; Cachia et al., 2008; Levitan et al., 1999) and functional (Kim et al., 2003) differences in STS have been directly linked to SCZ symptomatology including, but not limited to, increases in frequency and severity of hallucinations. Similarly, activation increases in IFG during hallucination in SCZ (Jardri et al., 2011), and anatomical irregularities in this region of IFG are a common feature in SCZ (Wisco et al., 2007). Furthermore, functional connectivity between superior temporal and inferior frontal regions is reduced in individuals exhibiting hallucinations (Vercammen et al., 2010). Collectively, these results point to striking changes in the network subserving speech processing associated with hallucinations in SCZ.

While STS and IFG are clearly involved in hallucinations in SCZ, they are also central nodes for audiovisual processing. Both STS and IFG have been shown to respond differentially based on the timing and/or perceptual fusion of audiovisual inputs, and have been implicated in various facets of audiovisual integration in humans (Miller and D'Esposito, 2005). Both are also central nodes in the speech perception network, with direct connections existing between STS, IFG, and auditory cortex (Sommers et al., 2005). Given these remarkable overlaps, disruptions within STS/IFG network may be the mechanistic feature that links audiovisual temporal impairments and the symptomatic hallucinations they predict.

These findings of multisensory temporal processing impairments also point to a potential strategy for SCZ remediation. Numerous studies show that temporal precision, and notably visual (Stevenson et al., 2013) and audiovisual (Powers et al., 2012) temporal acuity, is highly plastic and modifiable through perceptual training. Given the strong associative links between audiovisual temporal processing impairments and clinical symptomatology, it is conceivable that training focused on temporal processing may provide clinical benefits by increasing the level of perceptual coherence an individual experience. This hope is bolstered by findings of neural plasticity in STS in SCZ patients. While left STS progressively reduces in volume (correlating with severity of hallucinations) following the first episode of SCZ (Song et al., 2015), this trend is reversible following a year of neuroleptic treatment, left STS volume increased in SCZ patients (Song et al., 2015). Remarkably, neural activity and functional connectivity in STS have also been shown to be strongly modulated by multisensory perceptual learning (Powers et al., 2012). The ability to induce changes in STS through simple perceptual training, as well as the ability of remediation in SCZ to influence structure (and presumably function) in STS, suggests that this line of work is promising. Indeed, in future work it will be necessary to further relate multisensory processing deficits with fine-grain characterizations of psychopathology with SCZ. Specifically, the idea that individuals with SCZ exhibit wider TBWs overall, yet narrower windows were associated with more sever hallucinations is intriguing. One possible explanation is that wider TBWs in SCZ accommodates less reliable unisensory temporal processing, and thus is an adaptive feature of SCZ thus SCZ individuals with narrow windows would represent a failure of multisensory adaptation to imprecise unisensory inputs. Such characterization and its association with SCZ-related psychopathology will be informative from a mechanistic point of view and also from a clinical perspective as an area of possible remediation.

5.0 Conclusions

Our results support the hypothesis that sensory disturbances, specifically those in the temporal processing realm, contribute to hallucinations in SCZ. SCZ is associated with auditory and visual temporal dysfunction, with additional multisensory temporal dysfunction beyond that predicted by these unisensory deficits. These audiovisual temporal perceptual disturbances are also significantly predictive of clinical measures of hallucination severity, supporting the hypothesis that hallucinations may result from aberrant attribution, or integration, of internal auditory speech to an external speaker. These data are, to our knowledge, the first to demonstrate the cascading impact of sensory disturbances on higher-level, clinical symptomatology in SCZ. The highly overlapping neural architecture underlying temporal processing, multisensory integration, and speech perception, and that associated with hallucinations in SCZ, further support these findings. These findings also offer hope for the use of temporally-based sensory training methods as possible remediation tools in SCZ.

Supplementary Material

Table 2.

Audiovisual temporal precision predicts hallucination severity

Predictor Partial correlation (pr) p-Value
Step 1: R = 0.38, F-change(2,13) = 1.06, p-change = 0.37
Gender 0.25 0.38
Age 0.20 0.48
Step 2: R = 0.76, F-change(2,11) = 5.76, p-change = 0.019
Gender 0.54 0.06
Age 0.05 0.87
TBW Flashbeep −0.57 0.04
TBW Speech −0.43 0.14
Open in a new tab

Significant results of added predictors are shown

Footnotes

Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final citable form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.

References

  1. Aguayo J. Auditory hallucinations and smaller superior temporal gyral volume in schizophrenia. Am J Psychiatry. 1990;147:1457–1462. doi: 10.1176/ajp.147.11.1457. [DOI] [PubMed] [Google Scholar]
  2. Baum SH, Stevenson RA, Wallace MT. Behavioral, perceptual, and neural alterations in sensory and multisensory function in autism spectrum disorder. Progress in neurobiology. 2015;134:140–160. doi: 10.1016/j.pneurobio.2015.09.007. [DOI] [PMC free article] [PubMed] [Google Scholar]
  3. Behrendt RP, Young C. Hallucinations in schizophrenia, sensory impairment, and brain disease: a unifying model. Behav Brain Sci. 2004;27(6):771–787. doi: 10.1017/s0140525x04000184. discussion 787–830. [DOI] [PubMed] [Google Scholar]
  4. Bleuler E. Dementia praecox oder Gruppe der Schizophrenien. Handbuch der psychiatrie 1911 [Google Scholar]
  5. Cachia A, Paillère-Martinot ML, Galinowski A, Januel D, de Beaurepaire R, Bellivier F, Artiges E, Andoh J, Bartrés-Faz D, Duchesnay E. Cortical folding abnormalities in schizophrenia patients with resistant auditory hallucinations. NeuroImage. 2008;39(3):927–935. doi: 10.1016/j.neuroimage.2007.08.049. [DOI] [PubMed] [Google Scholar]
  6. Carroll CA, Boggs J, O'Donnell BF, Shekhar A, Hetrick WP. Temporal processing dysfunction in schizophrenia. Brain and cognition. 2008;67(2):150–161. doi: 10.1016/j.bandc.2007.12.005. [DOI] [PMC free article] [PubMed] [Google Scholar]
  7. Davalos DB, Kisley MA, Ross RG. Deficits in auditory and visual temporal perception in schizophrenia. Cogn Neuropsychiatry. 2002;7(4):273–282. doi: 10.1080/13546800143000230. [DOI] [PubMed] [Google Scholar]
  8. Elvevag B, McCormack T, Gilbert A, Brown GD, Weinberger DR, Goldberg TE. Duration judgements in patients with schizophrenia. Psychological medicine. 2003;33(7):1249–1261. doi: 10.1017/s0033291703008122. [DOI] [PubMed] [Google Scholar]
  9. Fister JK, Stevenson RA, Nidiffer AR, Barnett ZP, Wallace MT. Stimulus intensity modulates multisensory temporal processing. Neuropsychologia. 2016 doi: 10.1016/j.neuropsychologia.2016.02.016. [DOI] [PMC free article] [PubMed] [Google Scholar]
  10. Foucher JR, Lacambre M, Pham BT, Giersch A, Elliott MA. Low time resolution in schizophrenia Lengthened windows of simultaneity for visual, auditory and bimodal stimuli. Schizophrenia research. 2007;97(1–3):118–127. doi: 10.1016/j.schres.2007.08.013. [DOI] [PubMed] [Google Scholar]
  11. Freedman BJ. The subjective experience of perceptual and cognitive disturbances in schizophrenia. A review of autobiographical accounts. Archives of general psychiatry. 1974;30(3):333–340. doi: 10.1001/archpsyc.1974.01760090047008. [DOI] [PubMed] [Google Scholar]
  12. Giersch A, Lalanne L, Corves C, Seubert J, Shi Z, Foucher J, Elliott MA. Extended visual simultaneity thresholds in patients with schizophrenia. Schizophr Bull. 2009;35(4):816–825. doi: 10.1093/schbul/sbn016. [DOI] [PMC free article] [PubMed] [Google Scholar]
  13. Heinrichs DW, Buchanan RW. Significance and meaning of neurological signs in schizophrenia. The American journal of psychiatry. 1988;145(1):11–18. doi: 10.1176/ajp.145.1.11. [DOI] [PubMed] [Google Scholar]
  14. Jardri R, Pouchet A, Pins D, Thomas P. Cortical activations during auditory verbal hallucinations in schizophrenia: a coordinate-based meta-analysis. American Journal of Psychiatry. 2011 doi: 10.1176/appi.ajp.2010.09101522. [DOI] [PubMed] [Google Scholar]
  15. Kim JJ, Crespo-Facorro B, Andreasen NC, O'Leary DS, Magnotta V, Nopoulos P. Morphology of the lateral superior temporal gyrus in neuroleptic na ve patients with schizophrenia: relationship to symptoms. Schizophrenia research. 2003;60(2):173–181. doi: 10.1016/s0920-9964(02)00299-2. [DOI] [PubMed] [Google Scholar]
  16. Lalanne L, van Assche M, Giersch A. When predictive mechanisms go wrong: disordered visual synchrony thresholds in schizophrenia. Schizophr Bull. 2012;38(3):506–513. doi: 10.1093/schbul/sbq107. [DOI] [PMC free article] [PubMed] [Google Scholar]
  17. Levitan C, Ward PB, Catts SV. Superior temporal gyral volumes and laterality correlates of auditory hallucinations in schizophrenia. Biological psychiatry. 1999;46(7):955–962. doi: 10.1016/s0006-3223(98)00373-4. [DOI] [PubMed] [Google Scholar]
  18. Martin B, Giersch A, Huron C, van Wassenhove V. Temporal event structure and timing in schizophrenia: preserved binding in a longer “now”. Neuropsychologia. 2013;51(2):358–371. doi: 10.1016/j.neuropsychologia.2012.07.002. [DOI] [PubMed] [Google Scholar]
  19. McGurk H, MacDonald J. Hearing lips and seeing voices. Nature. 1976;264(5588):746–748. doi: 10.1038/264746a0. [DOI] [PubMed] [Google Scholar]
  20. Miller LM, D'Esposito M. Perceptual fusion and stimulus coincidence in the cross-modal integration of speech. J Neurosci. 2005;25(25):5884–5893. doi: 10.1523/JNEUROSCI.0896-05.2005. [DOI] [PMC free article] [PubMed] [Google Scholar]
  21. Noel JP, De Niear MA, Stevenson R, Alais D, Wallace MT. Atypical rapid audio-visual temporal recalibration in autism spectrum disorders. Autism Research. 2016 doi: 10.1002/aur.1633. [DOI] [PMC free article] [PubMed] [Google Scholar]
  22. Pearl D, Yodashkin-Porat D, Katz N, Valevski A, Aizenberg D, Sigler M, Weizman A, Kikinzon L. Differences in audiovisual integration, as measured by McGurk phenomenon, among adult and adolescent patients with schizophrenia and age-matched healthy control groups. Compr Psychiatry. 2009;50(2):186–192. doi: 10.1016/j.comppsych.2008.06.004. [DOI] [PubMed] [Google Scholar]
  23. Postmes L, Sno H, Goedhart S, van der Stel J, Heering H, de Haan L. Schizophrenia as a self-disorder due to perceptual incoherence. Schizophrenia research. 2014;152(1):41–50. doi: 10.1016/j.schres.2013.07.027. [DOI] [PubMed] [Google Scholar]
  24. Powers AR, 3rd, Hevey MA, Wallace MT. Neural correlates of multisensory perceptual learning. J Neurosci. 2012;32(18):6263–6274. doi: 10.1523/JNEUROSCI.6138-11.2012. [DOI] [PMC free article] [PubMed] [Google Scholar]
  25. Quinto L, Thompson WF, Russo FA, Trehub SE. A comparison of the McGurk effect for spoken and sung syllables. Atten Percept Psychophys. 2010;72(6):1450–1454. doi: 10.3758/APP.72.6.1450. [DOI] [PubMed] [Google Scholar]
  26. Raab DH. Statistical facilitation of simple reaction times. Transactions of the New York Academy of Sciences. 1962;24:574–598. doi: 10.1111/j.2164-0947.1962.tb01433.x. [DOI] [PubMed] [Google Scholar]
  27. Ross LA, Saint-Amour D, Leavitt VM, Molholm S, Javitt DC, Foxe JJ. Impaired multisensory processing in schizophrenia: deficits in the visual enhancement of speech comprehension under noisy environmental conditions. Schizophrenia research. 2007;97(1–3):173–183. doi: 10.1016/j.schres.2007.08.008. [DOI] [PubMed] [Google Scholar]
  28. Schlesinger JJ, Stevenson RA, Shotwell MS, Wallace MT. Improving pulse oximetry pitch perception with multisensory perceptual training. Anesth Analg. 2014;118(6):1249–1253. doi: 10.1213/ANE.0000000000000222. [DOI] [PubMed] [Google Scholar]
  29. Shergill SS, Brammer MJ, Williams SC, Murray RM, McGuire PK. Mapping auditory hallucinations in schizophrenia using functional magnetic resonance imaging. Archives of general psychiatry. 2000;57(11):1033–1038. doi: 10.1001/archpsyc.57.11.1033. [DOI] [PubMed] [Google Scholar]
  30. Sommers MS, Tye-Murray N, Spehar B. Auditory-visual speech perception and auditory-visual enhancement in normal-hearing younger and older adults. Ear and hearing. 2005;26(3):263–275. doi: 10.1097/00003446-200506000-00003. [DOI] [PubMed] [Google Scholar]
  31. Song JJ, Lee HJ, Kang H, Lee DS, Chang SO, Oh SH. Effects of congruent and incongruent visual cues on speech perception and brain activity in cochlear implant users. Brain Structure and Function. 2015;220(2):1109–1125. doi: 10.1007/s00429-013-0704-6. [DOI] [PubMed] [Google Scholar]
  32. Stevenson RA, Altieri NA, Kim S, Pisoni DB, James TW. Neural processing of asynchronous audiovisual speech perception. NeuroImage. 2010;49(4):3308–3318. doi: 10.1016/j.neuroimage.2009.12.001. [DOI] [PMC free article] [PubMed] [Google Scholar]
  33. Stevenson RA, Fister JK, Barnett ZP, Nidiffer AR, Wallace MT. Interactions between the spatial and temporal stimulus factors that influence multisensory integration in human performance. Experimental brain research. Experimentelle Hirnforschung. 2012a doi: 10.1007/s00221-012-3072-1. [DOI] [PMC free article] [PubMed] [Google Scholar]
  34. Stevenson RA, Ghose D, Fister JK, Sarko DK, Altieri NA, Nidiffer AR, Kurela LR, Siemann JK, James TW, Wallace MT. Identifying and Quantifying Multisensory Integration: A Tutorial Review. Brain Topogr. 2014a doi: 10.1007/s10548-014-0365-7. [DOI] [PubMed] [Google Scholar]
  35. Stevenson RA, Siemann JK, Schneider BC, Eberly HE, Woynaroski TG, Camarata SM, Wallace MT. Multisensory temporal integration in autism spectrum disorders. J Neurosci. 2014b;34(3):691–697. doi: 10.1523/JNEUROSCI.3615-13.2014. [DOI] [PMC free article] [PubMed] [Google Scholar]
  36. Stevenson RA, Wallace MT. Multisensory temporal integration: task and stimulus dependencies. Experimental brain research Experimentelle Hirnforschung. 2013;227(2):249–261. doi: 10.1007/s00221-013-3507-3. [DOI] [PMC free article] [PubMed] [Google Scholar]
  37. Stevenson RA, Wilson MM, Powers AR, Wallace MT. The effects of visual training on multisensory temporal processing. Experimental brain research Experimentelle Hirnforschung. 2013;225(4):479–489. doi: 10.1007/s00221-012-3387-y. [DOI] [PMC free article] [PubMed] [Google Scholar]
  38. Stevenson RA, Zemtsov RK, Wallace MT. Individual differences in the multisensory temporal binding window predict susceptibility to audiovisual illusions. J Exp Psychol Hum Percept Perform. 2012b;38(6):1517–1529. doi: 10.1037/a0027339. [DOI] [PMC free article] [PubMed] [Google Scholar]
  39. Sumby WH, Pollack I. Visual contribution to speech intelligibility in noise. Journal of the Acoustical Society of America. 1954;26:212–215. [Google Scholar]
  40. Surguladze SA, Calvert GA, Brammer MJ, Campbell R, Bullmore ET, Giampietro V, David AS. Audio-visual speech perception in schizophrenia: an fMRI study. Psychiatry research. 2001;106(1):1–14. doi: 10.1016/s0925-4927(00)00081-0. [DOI] [PubMed] [Google Scholar]
  41. Szycik GR, Munte TF, Dillo W, Mohammadi B, Samii A, Emrich HM, Dietrich DE. Audiovisual integration of speech is disturbed in schizophrenia: an fMRI study. Schizophrenia research. 2009;110(1–3):111–118. doi: 10.1016/j.schres.2009.03.003. [DOI] [PubMed] [Google Scholar]
  42. Tysk L. Estimation of time and the subclassification of schizophrenic disorders. Percept Mot Skills. 1983a;57(3 Pt 1):911–918. doi: 10.2466/pms.1983.57.3.911. [DOI] [PubMed] [Google Scholar]
  43. Tysk L. Time estimation by healthy subjects and schizophrenic patients: a methodological study. Percept Mot Skills. 1983b;56(3):983–988. doi: 10.2466/pms.1983.56.3.983. [DOI] [PubMed] [Google Scholar]
  44. Vercammen A, Knegtering H, den Boer JA, Liemburg EJ, Aleman A. Auditory hallucinations in schizophrenia are associated with reduced functional connectivity of the temporo-parietal area. Biological psychiatry. 2010;67(10):912–918. doi: 10.1016/j.biopsych.2009.11.017. [DOI] [PubMed] [Google Scholar]
  45. Volz HP, Nenadic I, Gaser C, Rammsayer T, Hager F, Sauer H. Time estimation in schizophrenia: an fMRI study at adjusted levels of difficulty. Neuroreport. 2001;12(2):313–316. doi: 10.1097/00001756-200102120-00026. [DOI] [PubMed] [Google Scholar]
  46. Williams LE, Light GA, Braff DL, Ramachandran VS. Reduced multisensory integration in patients with schizophrenia on a target detection task. Neuropsychologia. 2010;48(10):3128–3136. doi: 10.1016/j.neuropsychologia.2010.06.028. [DOI] [PMC free article] [PubMed] [Google Scholar]
  47. Wisco JJ, Kuperberg G, Manoach D, Quinn BT, Busa E, Fischl B, Heckers S, Sorensen AG. Abnormal cortical folding patterns within Broca's area in schizophrenia: evidence from structural MRI. Schizophrenia research. 2007;94(1):317–327. doi: 10.1016/j.schres.2007.03.031. [DOI] [PMC free article] [PubMed] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

NIHMS823253-supplement-supplement_1.pdf (296KB, pdf)

RESOURCES