Reduced Multisensory Integration in Patients with Schizophrenia on a Target Detection Task

Abstract

A growing body of literature demonstrates impaired multisensory integration (MSI) in patients with schizophrenia compared to non-psychiatric individuals. One of the most basic measures of MSI is intersensory facilitation of reaction times (RTs), in which bimodal targets, with cues from two sensory modalities, are detected faster than unimodal targets. This RT speeding is generally attributed to super-additive processing of multisensory targets. In order to test whether patients with schizophrenia are impaired on this basic measure of MSI, we assessed the degree of intersensory facilitation for a sample of 20 patients compared to 20 non-psychiatric individuals using a very simple target detection task. RTs were recorded for participants to detect targets that were either unimodal (auditory alone, A, visual alone, V) or bimodal (auditory + visual, AV). RT distributions to detect bimodal targets were compared with predicted RT distributions based on the summed probability distribution of each participant’s RTs to visual alone and auditory alone targets. Patients with schizophrenia showed less RT facilitation when detecting bimodal targets relative to non-psychiatric individuals, even when groups were matched for unimodal RTs. Within the schizophrenia group, RT benefit was correlated with negative symptoms, such that patients with greater negative symptoms showed the least RT facilitation (r² = .20, p < .05). Additionally, schizophrenia patients who experienced both auditory and visual hallucinations showed less multisensory benefit compared to patients who experienced only auditory hallucinations, indicating that the presence of hallucinations in two modalities may more strongly impair MSI compared to hallucinations in only one modality.

Introduction

Schizophrenia is a debilitating psychiatric disorder with a global prevalence rate of approximately 1.0% (American Psychiatric Association, 2000). It is well known that individuals with schizophrenia, their unaffected first degree relatives, and individuals with schizotypal personality disorder exhibit processing abnormalities at both the behavioral and neural level in a number of sensory modalities (Adler et al., 1982; Braff, 1989; Ford et al., 2004; Light et al., 2006; Turetsky, Colbath, & Gur, 1998). Although these sensory and “gating” deficits, such as reduced prepulse inhibition, reduced P50 suppression, and reduced mismatch negativity to auditory stimuli, are well characterized within individual modalities (Braff & Light, 2005; Grillon, Ameli, Charney, Krystal, & Braff, 1992; Light & Braff, 2005a; Light & Braff, 2005b; Umbricht & Krljes, 2005), the integration of information from multiple sensory modalities has only recently emerged as a key topic in the study of schizophrenia.

Historically, perceptual research has focused on sensory processing within individual modalities. However, in real-world situations people are constantly bombarded with input from multiple sensory channels, and must be able to detect, integrate and filter this information to form an accurate perception of their environment. Within the last 10 −15 years there has been a dramatic growth in research on how information from different sensory modalities is combined, both for cues that are congruent along a task-relevant dimension, such as time, space, and/or identity, as well as the perception of events that are defined by conflicting cues. Previous research on multisensory integration (MSI) in non-psychiatric individuals has shown that the presentation of congruent information from multiple modalities confers an advantage for speed and accuracy of processing (for reviews see Frassinetti, Bolognini, & Ladavas, 2002; Loveless, Brebner, & Hamilton, 1970; Stein & Meredith, 1993; Welch & Warren, 1986). For example, reaction times (RTs) in a target detection task are faster when auditory and visual targets are presented simultaneously, compared to RTs for targets presented in either modality alone (Hershenson, 1962). Similarly, Frens, Van Opstal, and Van der Willigen (1995) showed that participants saccade more quickly to a visual target when an irrelevant auditory tone is spatially and temporally aligned with the target. In contrast, when individuals are presented with conflicting cues from multiple sensory modalities, perception tends to be dominated by the most reliable source of input (Welch & Warren, 1980; Welch & Warren, 1986). In many cases the presentation of incongruent information in different modalities can lead to perceptual illusions, such as the McGurk effect (McGurk & MacDonald, 1976), in which conflicting visual speech influences auditory speech perception, or the ventriloquist effect (Bertelson & Radeau, 1981; Choe, Welch, Gilford, & Juola, 1975), in which misaligned visual cues disrupt auditory localization.

Only a handful of previous studies have directly assessed MSI in patients with schizophrenia. All these investigations have focused on the integration of auditory and visual cues, and results have been inconsistent across studies and stimulus type. Within this small body of existing literature, studies can be classified along two dimensions: 1) does the task utilize complex cognitive stimuli (e.g. speech, emotional stimuli) or simple sensory stimuli (e.g. light, shapes, sounds) and 2) are participants presented with cues from different modalities that are congruent along the task-relevant judgment dimension (e.g. spatial, temporal, and/or semantic properties), which typically facilitates processing, or cues that are incongruent along the dimension of judgment (in conflict in one of these domains), which typically interferes with processing, and may result in perceptual illusions. Almost all previous studies with patients with schizophrenia have tested participants using cues that are incongruent along the task-relevant dimension, a common experimental approach in MSI research that allows for insight into which modality has the strongest influence on the final perceptual judgment. Of these, most have found that relative to healthy participants, schizophrenia patients show reduced MSI of complex cognitive stimuli, such as audio-visual speech (de Gelder, Vroomen, Annen, Masthof, & Hodiamont, 2003, Experiment 2) and emotional faces and voices (de Gelder et al., 2005; de Jong, Hodiamont, Van den Stock, & de Gelder, 2009); though also see Surguladze et al. (2001) for a study that finds no difference between patients and healthy controls for the integration of audio-visual speech cues. A single previous study has evaluated the integration of simple sensory cues that are incongruent along the task-relevant dimension, and found that schizophrenia patients and healthy control participants showed a similar degree of mislocalization of auditory cues (pure tones) when presented with spatially incongruent visual shapes (de Gelder et al., 2003, Experiment 1).

In contrast to these sensory conflict paradigms that often induce perceptual illusions, enhanced processing of events defined by congruent multisensory cues is advantageous for detecting biologically relevant stimuli, and the super-additive activation these stimuli typically induce causes objects and events defined by multiple sensory modalities to have increased neural salience (Stein & Stanford, 2008). Therefore, it is equally if not more important to understand how patients with schizophrenia utilize congruent information from multiple sensory channels. However, to date only a single study has evaluated the degree of processing benefit schizophrenia patients gain from congruent, task-relevant multisensory stimulation, finding evidence for reduced integration of congruent audio-visual speech relative to control participants (Ross et al., 2007).

The current study adds to this existing literature by testing a previously unexplored question – do patients with schizophrenia and healthy control participants show similar multisensory gain for the detection of simple, temporally congruent sensory cues from two different modalities? To evaluate this question, we tested group of schizophrenia patients and matched control participants on a target detection task to assess intersensory facilitation of RT (Hershenson, 1962) using very simple auditory and visual cues. Participants were asked to respond with a button press as quickly as possible when they detected a target, which could be either unimodal (auditory alone, A; visual alone, V) or bimodal (auditory + visual, AV). Previous studies utilizing this type of paradigm have found that RTs to detect bimodal targets are faster not only than RTs to either single modality alone, but are also faster than would be predicted due to redundant stimulation (e.g. the presence of two targets rather than one), suggesting that bimodal targets lead to super-additive processing (e.g. Forster, Cavina-Pratesi, Aglioti, & Berlucchi, 2002; Hughes, Reuter-Lorenz, Nozawa, & Fendrich, 1994; Miller, 1982; Miller, 1986). We used this paradigm to quantify and compare the amount of multisensory gain exhibited by a sample of chronic schizophrenia patients and a sample of matched non-psychiatric participants.

Methods

Participants

Participants were 20 patients with schizophrenia and 20 non-psychiatric participants recruited via the UCSD Schizophrenia Research Program. All patients with schizophrenia had confirmed diagnoses based on the Structured Clinical Interview for DSM-IV, with no other Axis 1 diagnoses or history of neurologic insult. Current clinical symptoms were assessed using the Scale for the Assessment of Negative Symptoms (SANS; Andreasen, 1984a) and the Scale for the Assessment of Positive Symptoms (SAPS; Andreasen, 1984b). Non-psychiatric participants were recruited through newspaper advertisements and flyers posted at the UCSD medical center, and were screened to rule out past or present Axis I or II diagnoses and drug abuse. Participants were assessed on their capacity to provide informed consent, and given a detailed description of their participation in the study. Written consent was obtained via a consent form approved by the University of California, San Diego institutional review board (Protocol # 070052).

All patients in the schizophrenia group were clinically stable, and Tables 1 and 2 contain demographic, clinical and medication information. Groups did not differ in age (p =.95) or years of education (p =.11). Handedness was assessed with Edinburgh Handedness Inventory (Oldfield, 1971).

Table 1.

Participant Characteristics

Characteristic	NCP	SZ Patients
Age, mean (SD), years	50.90 (9.33)	51.75 (8.77)
Male/Female, #	9/11	8/12
Education, mean (SD), years	14.70 (2.96)	13.45 (2.35)
Handedness, Right/Left/Ambidextrous, #	19/1/0	18/1/1

Abbreviations: NCP, Non-psychiatric comparison participants; SZ, schizophrenia.

Table 2.

Characteristics of Patients with Schizophrenia

Characteristic	Value
Duration of Illness, mean (SD), years	29.90 (10.65)
Hospitalizations, mean (SD), #	7.40 (8.44)
Diagnostic Subtype, #
Paranoid	11
Undifferentiated	6
Residual	3
Medication, #
Atypical antipsychotic	19
Unmedicated	1
Living situation, #
Independently or with family	14
Board and Care facility	6
SAPS Score, mean (SD)	8.6 (4.94)
SANS Score, mean (SD)	13.65 (4.70)

Reaction time task

Participants were seated in front of an 18” ViewSonic Graphics Series monitor (1024 × 768, 60 Hz) with their eyes 57 cm from the center of the screen. The experiment was programmed in E-Prime (Psychology Software Tools, Inc.). Participants pressed the space bar on a standard computer keyboard to start each trial. Trials began with a fixation cross presented for 1.5 seconds, followed after a variable random delay (500, 750, 1000, 1250 or 1500 ms) by either a visual stimulus (red letter X printed in 12 pt Times New Roman font, subtending 0.7 degrees of visual angle, presented for 100 ms), an auditory stimulus (100 ms tone presented binaurally via headphones) or both cues simultaneously in the bimodal condition, making them temporally congruent. Blank trials were also included to reduce anticipatory responses. Participants were instructed to press the “K” key of a standard PC keyboard with the index finger of their dominant hand as quickly as possible when they detected either a visual or an auditory target, and the RT to press the button was recorded. Participants completed four blocks of 74 trials; each block began with 4 randomly selected trials that were treated as practice trials and excluded from analysis, followed by 20 trials in each condition (A, V, AV) plus 10 blank catch trials, all presented in random order.

As the bimodal AV targets provide two cues for detection rather than one, comparing the raw RTs for unimodal versus bimodal stimuli cannot disentangle the effects of redundant stimulation (e.g. the presentation of two cues) from true multisensory benefit driven by congruent stimulation in two separate sensory modalities. Therefore, to quantify specific multisensory gain, RTs in the AV condition were compared to predicted RTs generated by the “independent race model” (Miller, 1982; Miller, 1986), which calculates the summed probabilities of the A and V conditions (Pr Auditory + Pr Visual) for each individual participant. When RTs in the bimodal condition are significantly faster than this summed race model probability, which is the expected distribution of RTs for presenting two cues as opposed to one, this is considered a violation of the race model and thought to reflect RT facilitation that is specifically attributable to multisensory integration (Miller, 1982; Miller, 1986).

Prior to analysis, individual responses more than 3 standard deviations above or below the mean of a particular condition were excluded. This outlier rejection process did not differentially affect the patient group (no significant difference in number of trials excluded per participants in each group; patients M = 1.37, SD = 0.94, non-psychiatric M = 1.27, SD = 0.90, t (118) = .59, ns). Data analysis was performed with Matlab (The Mathworks, Natick, MA) using a program published by Ulrich, Miller, and Schroter (2007). Briefly, for each participant, raw RTs in each experimental condition (A, V, AV) were used to generate a cumulative density function (CDF). The CDFs for the auditory and visual targets were summed to calculate the race model prediction curve for each individual participant (see Ulrich et al., 2007 for more details of analysis). These curves were evaluated to determine percentile values at 20 different points (.05 – 1.0) and these percentile values were averaged across individuals for each group (Schizophrenia versus Control). Any given percentile value determines the upper limit of RTs associated with that percentile. For example, an RT value of 300 ms for the .05 percentile indicates that 5% of trials in this condition have an RT of 300 ms or faster. Violations of the race model at individual percentiles were evaluated using paired t-tests. As these multiple t-tests inflate the family-wise alpha level, we also report Bonferroni-corrected violations. This is considered a somewhat conservative correction (Ulrich, Miller, & Schroter, 2007), and the general pattern of results is consistent for both non-corrected and corrected comparisons.

To ensure that our results are robust even when RT distributions are not normalized by this percentile method, we also tested an alternate version of the race model in addition to our main analysis. In this variation, raw RTs are divided into 10 ms bins and used to create each participant’s CDFs for all conditions (Laurienti, Burdette, Maldjian, & Wallace, 2006). RT benefit was quantified at each time bin by subtracting the race model prediction from the average AV RT, and these difference curves for individual subjects were averaged for each group. Within each diagnostic group, race model violations were tested at each time bin with a one sample t-test (difference score compared to 0). As this alternative method is based on difference scores between the multisensory RTs and race model prediction RTs, it may also be more robust to differences in variability between diagnostic groups.

Results

False alarm rates were extremely low and did not differ significantly between groups (0.11% of trials for patients, 0.23% for non-psychiatric; t = −0.50, p = 0.62). In an attempt to sample the distribution of RTs for each condition equivalently for individual participants, all missed trials were rerun once at the end of each block, and RTs for the rerun trials replaced misses during analysis. Overall miss rates were calculated based on the number of rerun trials per condition for each participant plus any trials missed again when rerun. These miss rates were relatively low and differed between groups for the Auditory condition (2.57% patients vs. 0.24% non-psychiatric, t = 2.63, p = 0.01) but not for Visual condition (1.85% patients vs. 0.97% non-psychiatric, t = 1.73, p = 0.09), or Audio-Visual condition (0.41% patients vs. 0.30% non-psychiatric, t = 0.44, p = 0.66). Misses that remained after rerun were extremely rare, and did not differ significantly between groups in any condition (Auditory 0.16% patients, 0% non-psychiatric, t = 1.38, p = 0.17; Visual 0.11% patients, 0% non-psychiatric, t = 1, p = 0.32; Audio-Visual 0.11% patients, 0.06% non-psychiatric, t = 0.52 p = 0.60). The significant difference in overall misses was primarily driven by one participant in the schizophrenia group. To investigate whether this participant’s data had a large effect on the group averages, these analyses were run with that participant excluded and results were identical; therefore reported results include all participants.

Average cumulative probability RT curves for patients with schizophrenia and non-psychiatric participants are presented in Figures 1A and 1B. The data for non-psychiatric individuals (Figure 1A) replicates many previous findings, in which RTs for the bimodal AV condition are faster than either of the unimodal (A or V) conditions. In addition, for almost the entire curve, observed RTs in the AV condition are faster than the race model predictions, indicating that non-psychiatric subjects show significant multisensory benefit through the 80^th percentile of RTs (paired t-tests, all p’s <.05; Bonferroni corrected violations also up through 80^th percentile). In contrast, patients with schizophrenia (Figure 1B) only show violations of the race model up through the 50^th percentile (Bonferroni corrected violations up through the 30^th percentile). Therefore, although schizophrenia patients do show some of the expected multisensory benefit on this task, it is not to the same extent as non-psychiatric individuals.

Figure 1.

Cumulative density functions of reaction times (RTs) for target detection with three cue types; unimodal Auditory (A), unimodal Visual (V) and bimodal Auditory + Visual (AV) as well as Race Model predictions (RACE) based on sum of unimodal RT curves. Panel A shows data from the full sample of non-psychiatric comparison participants and Panel B shows data from the full sample of patients with schizophrenia. The percentiles for which the AV curve violates the race model for each group using a paired t-test (p <. 05) are marked with a star, Bonferroni corrected violation percentiles are also marked with an X. Schizophrenia patients show significant intersensory facilitation of reaction time over less of the curve, up to only the 50^th percentile, compared to non-psychiatric individuals who show race model violations up to the 80^th percentile. This reduced facilitation in the schizophrenia group persists when participants are matched for overall unisensory RTs, as well as intersubject variability, with a subset of ten relatively slow non-psychiatric participants, in Panel C, and ten relatively fast patients with schizophrenia, in Panel D.

To further evaluate these between-group differences statistically, we counted the number of percentiles for which the observed AV RT was faster than the race model prediction for each individual participant. This RT-benefit metric has a maximum score of 20 (AV faster than race model prediction for all percentiles, multisensory benefit across entire curve) and a minimum score of 0 (AV not faster than race model prediction for any percentiles, no multisensory benefit), with higher scores indicative of multisensory benefit across a greater proportion of the RT distribution. A two-sample t-test of the RT-benefit scores between groups confirmed that patients with schizophrenia (M = 10.9, SD = 5.94) had significantly lower scores compared to non-psychiatric individuals (M = 16.7, SD = 3.85; t (38) = 3.66, p<.001, 2-tailed).

Results of the alternative race model variation are consistent with this pattern, with schizophrenia patients showing multisensory benefit over a smaller range of the RT curve (Figure 2A). Patients with schizophrenia showed significant violations of the race model predictions over a smaller time window, only 160 ms (270 – 430 ms bins; all p’s < .05), compared to non-psychiatric participants, who showed significant multisensory gain over a 310 ms window (270 – 580 ms bins; all p’s < .05). This difference between diagnostic groups is further illustrated in Figure 2B, in which the RT benefit difference curve for schizophrenia patients is subtracted from the curve for non-psychiatric participants. Healthy controls show significantly greater RT benefit than schizophrenia patients over the 270 – 380 ms range (2 tailed t-test, all p’s < 0.05). These findings confirm that the between-group difference in RT facilitation found in our main analysis is not driven by a statistical artifact due to the normalization of RT curves into percentiles.

Figure 2.

A) Cumulative probability difference curves, in which race model predictions are subtracted from the observed bimodal Audio-Visual reaction times (RTs), for non-psychiatric comparison participants (filled squares) and patients with schizophrenia (open diamonds); error bars represent standard error of the mean. Curves were generated using an alternative race-model approach, in which raw RTs are organized into 10 ms bins rather than normalized to percentiles as in the main analysis. Positive values indicate faster RTs for the AV stimuli than the race model prediction. Starred bins are those for which both patients and non-psychiatric participants show violations of the race model; bins marked with an X are those with significant violations for non-psychiatric participants only. Similar to the main analysis, patients with schizophrenia showed significant intersensory facilitation over a smaller time range, 160 ms total from 270 to 430 ms, relative to non-psychiatric participants, 310 ms total from 270 to 580 ms. This difference between diagnostic groups is further highlighted in panel B, which represents a subtraction of the difference curve for schizophrenia patients from the difference curve for the non-psychiatric group. Bins marked with a plus sign indicate significantly greater intersensory facilitation in non-psychiatric participants compared to schizophrenia patients, over a 110 ms range (270 – 380 ms).

Interestingly, a simple ANOVA on the average RTs for each target condition yields a significant effect of group (control faster than schizophrenia, F (1,38) = 12.7, p < .001), condition (AV faster than V faster than A, F (2,37) = 242.5, p < .001), but no group by condition interaction (F (2,37) = .71, p = .50). This pattern of results is consistent with a selective reduction in multisensory gain for the schizophrenia group, as both groups show similar speeding of responses in AV condition due to the presence of two cues (redundancy gain), but the more sensitive race-model analysis isolates a difference in the RT facilitation that can be specifically attributed to the bimodal nature of the targets.

Matching for Unisensory RTs

Although the race model prediction values are calculated relative to each participant’s individual RT distributions, one possible concern is that these differences in MSI could be driven by overall slower RTs in the patient group (compare Figure 1A and 1B), or by significant differences in variability between participants in the two groups. We tested this alternative explanation by running the same analyses described above on ten participants from each group, matched for unimodal RTs by including relatively fast patients with schizophrenia and relatively slow healthy comparison participants. A comparison of intersensory facilitation for this subset of the dataset yields a similar pattern of results to the full sample analyses (Figures 1C and 1D), with race model violations through the 75^th percentile for slow non-psychiatric individuals (Figure 1C; Bonferroni corrected violations up through the 60^th percentile), and violations through only the 35^th percentile for fast schizophrenia patients (Figure 1D; no significant violations of the race model with Bonferroni correction). A t-test of the RT-benefit scores between these matched groups is also significant, with schizophrenia patients (M = 10.4, SD = 6.20) exhibiting significantly fewer percentiles of multisensory gain on average compared to non-psychiatric individuals (M = 17.2, SD = 1.14; t (18) = 3.41, p < .001, 2-tailed). Comparison of the unimodal RT curves for patients with and non-psychiatric individuals at each of the twenty percentiles found no significant differences between groups (2 tailed t-tests, all p’s >.34), indicating successful matching for unimodal RTs. These groups were also matched on intersubject variability (two sample t-test on standard deviation values for each condition between groups, all p’s > 0.64). As such, it is unlikely that the observed difference in multisensory benefit between groups can be attributed to slower overall RTs, or increased variability between participants in the schizophrenia group relative to the non-psychiatric group.

Visual inspection of individual participant data indicates that these group average graphs are an accurate representation of the two diagnostic groups. Although there is variability in the extent to which the AV curve is shifted relative to the race model prediction curve, there were no participants in either group whose data greatly deviated from the general pattern of the group data; all participants had Auditory and Visual curves that were slower than both the AV and race model prediction curves. To further illustrate this point, data from four individual participants are presented in Figure 3 (3A and 3B are non-psychiatric participants; 3C and 3D are patients with schizophrenia). The data in 3A and 3C are examples of participants who show greater RT benefit, compared to data in 3B and 3D, which show relatively weaker integration effects.

Figure 3.

Cumulative density functions of reaction times (RTs) for target detection with three cue types; unimodal Auditory (A), unimodal Visual (V) and bimodal Auditory + Visual (AV) as well as Race Model predictions (RACE) based on sum of unimodal RT curves. Panels A and B show data from two individual non-psychiatric participants, and panels C and D show data from two individual patients with schizophrenia. All participants showed similar patterns of RTs, with Auditory and Visual curves that were slower than both the AV and race model prediction curves, though individual participants varied in the extent to which the AV curve is shifted in the faster direction relative to the race model prediction.

Intersensory Facilitation by Modality of Hallucinations

In order to test whether the pattern of MSI is affected by presence of unusual sensory experiences in two sensory modalities as opposed to only one, the patient group was divided into individuals who have experienced only auditory hallucinations (n = 10; Figure 4A), and those who have experienced both visual and auditory hallucinations (n = 10; Figure 4B). Patients who have experienced only auditory hallucinations show violations of the race model up through the 55^th percentile (no significant violations with Bonferroni correction), compared to violations through only the 10^th percentile for patients who have experienced both auditory and visual hallucinations (no significant violations with Bonferroni correction). In addition, patients who experienced only auditory hallucinations had significantly higher RT-benefit scores (M = 13.7, SD = 4.85) compared to patients who had experienced both auditory and visual hallucinations (M – 8.1, SD = 5.80). This pattern of results indicates that patients with schizophrenia who have experienced both visual and auditory hallucinations may be especially impaired with respect to audio-visual MSI.

Figure 4.

Cumulative density functions of reaction times (RTs) for target detection with three cue types; unimodal Auditory (A), unimodal Visual (V) and bimodal Auditory + Visual (AV) as well as Race Model predictions (RACE) based on unimodal RT curves. Panel A shows the data for patients with schizophrenia who have experienced only auditory hallucinations, and panel B shows the data for patients who have experienced both visual and auditory hallucinations. The percentiles for which the AV curve violates the race model for each group using a paired t-test (p <. 05) are marked with a star, Bonferroni corrected violation percentiles are also marked with an X. Schizophrenia patients who have experienced both auditory and visual hallucinations show intersensory facilitation of reaction time over less of the curve, up to only the 10^th percentile, compared to non-psychiatric individuals who show race model violations up to the 55^th percentile.

This split based on modality of hallucinations was partially confounded with current symptom severity, as the auditory and visual hallucination group also had higher positive symptom scores than the auditory-only group (global Scale for the Assessment of Positive Symptoms (Andreasen, 1984b) [SAPS] score of 10.8 for the auditory and visual group, versus 6.4 for the auditory-only group, t = −2.18, p < .05, 2 tailed). The positive symptoms of schizophrenia include hallucinations, delusions, bizarre behavior, and formal thought disorder (Andreasen, 1984b). To test whether this finding was driven by modality of hallucination or overall positive symptoms, we also examined integration patterns for the patient group with a median split based on global SAPS score. Both the high and low SAPS groups showed similar patterns of MSI (race model violations through the 35^th percentile for high SAPS, and up through the 40^th percentile for low SAPS; RT-benefit score t-test ns, p = .83), indicating that the integration differences observed between the different hallucination groups cannot solely be explained by positive symptom severity.

We were also interested to explore whether these MSI deficits observed at the group level were related to current schizophrenia symptom severity. To this end, correlations were run between the RT-benefit scores and overall symptom ratings for individual participants with schizophrenia. Consistent with the median split analysis described above, no significant correlation was found between RT-benefit scores and positive symptoms (global SAPS score). However, RT-benefit scores showed a significant negative correlation with negative symptoms (i.e., affective flattening, reduced speech, avolition, antisociality, and attentional problems, as assessed using the Scale for the Assessment of Negative Symptoms (Andreasen, 1984a; r² = 0.20, p < .05, Figure 5), such that participants who showed multisensory benefit over less of the curve also had higher current negative symptom ratings.

Figure 5.

Correlation between RT-benefit metric (maximum score of 20 with higher scores indicating more multisensory benefit) and negative symptoms (global Scale for the Assessment of Negative Symptoms [SANS] scores) for individual patients with schizophrenia. There is a significant negative correlation between these factors, such that individuals who exhibited less multisensory benefit had higher negative symptom ratings.

Discussion

Intersensory facilitation of reaction time (RT) is a well-documented behavioral effect first reported by Hershenson (1962), in which bimodal stimuli (e.g. auditory and visual, or visual and tactile cues presented simultaneously) are detected more quickly than either single stimulus alone. Multiple studies have found that this speeding of RTs for bimodal cues is above and beyond what would be expected by statistical summation of the unimodal RTs (e.g. Forster et al., 2002; Hughes et al., 1994; Miller, 1982; Miller, 1986), and is thought to reflect genuine facilitation of processing due to the presentation of congruent cues from multiple sensory modalities.

These data indicate that patients with schizophrenia do not integrate simple, temporally congruent auditory and visual cues as strongly as non-psychiatric individuals. More importantly, we find for the first time a relationship between the degree of MSI impairment and the clinical symptoms of schizophrenia. The experimental task employed was extremely simple and largely self-paced in administration, minimizing the possibility that performance in the schizophrenia group was impaired due to problems understanding the experiment, or heavy dependence on working memory or sustained attention processes known to be impaired in this population (e.g. Barch, 2005; Braff, 1993; Green, 2006; Lee & Park, 2005; Nuechterlein & Dawson, 1984). Additionally, because the race model predictions are derived from individual participants’ data for each modality, the MSI measurement is normalized to any individual differences in the unisensory RTs. This analysis method, as well as confirmation of the main between-group difference with a subset of participants matched for overall RT and between-subjects variability, is especially important given the overall variability between participants and diagnostic groups in this sample.

Previous research on MSI in non-psychiatric individuals has demonstrated that the presentation of congruent information from multiple modalities confers advantages in speed and accuracy of processing (for reviews see Frassinetti et al., 2002; Loveless et al., 1970; Stein & Meredith, 1993; Welch & Warren, 1986). The fact that patients with schizophrenia are impaired on this very basic sensory process likely reduces their ability to detect and respond to relevant stimuli in real-world environments. Given that patients with schizophrenia are also known to have problems filtering out redundant sensory information at the neural level (e.g. Braff & Light, 2005), a reduced multisensory processing benefit may add to these issues. These early sensory processes also directly interact with other cognitive domains, such as working memory, an additional domain of impairment for patients with schizophrenia (Lee & Park, 2005). Working memory studies have shown increased encoding of bimodal stimuli (Mastroberardino, Santangelo, Botta, Marucci, & Olivetti Belardinelli, 2008), as well as items that have higher salience (Fine & Minnery, 2009). Thus, the early MSI deficits observed here may feed into other cognitive systems that directly affect these patients’ ability to successfully navigate their environments.

The fact that patients with the most severe negative symptoms also exhibit the most impaired MSI indicates these deficits are related to characteristics of the disorder itself, as opposed to representing an overall deficit in this population. The finding that patients who have experienced both auditory and visual hallucinations are more impaired on this MSI task relative to patients who have experienced only auditory hallucinations is also very intriguing. Although this will require further dedicated studies to test directly, it is possible that having two disordered sensory channels in the auditory and visual hallucination group, as opposed to only one in the auditory hallucination group, leads to increased variability in the timing of processing signals that enter the brain via those channels. Studies of MSI in non-psychiatric individuals indicate that the timing of input from multiple sensory channels is crucial to elicit the enhanced neural activity associated with multisensory cues (Senkowski, Talsma, Grigutsch, Herrmann, & Woldorff, 2007). As such, small differences in the timing of and synchrony between both the auditory and visual channels could lead to larger deficits compared with having only one disordered channel in the auditory hallucination group.

Functional neuroimaging studies find a distributed network of cortical areas involved in MSI including the superior temporal sulcus, the superior temporal gyrus, the medial temporal gyrus, and parietal regions (Amedi, von Kriegstein, van Atteveldt, Beauchamp, & Naumer, 2005; Beauchamp, Lee, Argall, & Martin, 2004; Calvert, Campbell, & Brammer, 2000; Calvert, Hansen, Iversen, & Brammer, 2001; Driver & Noesselt, 2008; Stevenson & James, 2009). In addition, there is emerging evidence that the presentation of multiple sensory cues also affects activation of and processing in areas traditionally considered to be “unisensory” regions (for review see Driver & Noesselt, 2008). Previous EEG studies of RT paradigms similar to the current study with non-psychiatric individuals find evidence for both early and late AV interaction effects (Giard & Peronnet, 1999; Molholm et al., 2002; but also see Teder-Salejarvi, McDonald, Di Russo, & Hillyard, 2002) as well as enhanced neural synchrony associated with multisensory processing (Senkowski, Molholm, Gomez-Ramirez, & Foxe, 2006), an aspect of neural processing which is also disrupted in patients with schizophrenia (Uhlhaas, Haenschel, Nikolic, & Singer, 2008). This extensive literature on the basic mechanisms and neural bases of MSI may therefore provide interesting new sites of investigation for schizophrenia research, based on the deficits observed in the current study.

Future studies should explore the neural basis of these between group differences using neuroimaging and electrophysiological methods to try and establish where and when processing differences may arise between patients with schizophrenia and healthy comparison participants. In addition to providing insight into the neural correlates of the behavioral differences observed in the current study, neuroimaging investigations may also help disentangle whether the decreased benefit seen in the average data for the patient group is due to a general reduction in multisensory benefit across all trials, or whether differences are driven by super-additive multisensory processing on fewer individual trials compared to non-psychiatric participants. It will also be informative to further explore the clinical correlates of these between group differences, such as whether integration deficits are state or trait dependent, and whether they are present early in the course of illness. Finally, testing these integration effects in other clinical populations, such as patients with bipolar disorder, would help determine whether these deficits are unique to schizophrenia, and whether they are affected by antipsychotic medications. The simple nature of the RT task used in the current study is well suited for further testing with clinical populations, as it is easy to administer, and may correlate with psychiatric symptoms in other samples as well.

The current findings add to a small body of existing literature on MSI in patients with schizophrenia. Though the majority of previous studies have found that patients exhibit reduced integration of complex cognitive stimuli like speech (de Gelder et al., 2003, Experiment 2; Ross et al., 2007) and emotional faces and voices (de Gelder et al., 2005; de Jong et al., 2009), findings are not entirely consistent (Surguladze et al., 2001). Only one previous study has quantified MSI of relatively simple sensory stimuli, and found no differences between patients and controls for the integration of auditory tones and geometric shapes on a spatial localization task (de Gelder et al., 2003, Experiment 1). As there are a number of differences between the current study and de Gelder et al. (2003, Experiment 1), including the nature of the perceptual judgment (target detection versus spatial localization), and the congruence of the stimuli along the dimension of perceptual judgment, it is unclear what may be driving the different patterns of results between these two studies. However, the results of the current study do not support the idea that schizophrenia patients exhibit MSI deficits for only complex cognitive cues, as we find evidence for reduced integration of very simple sensory stimuli.

It is clear that further research is needed to understand the integrity of MSI in patients with schizophrenia, and there are several relevant dimensions to consider for future studies. For one, studies that evaluate multisensory gain driven by the presence of congruent task-relevant bimodal cues are underrepresented; as these paradigms may provide valuable insight into the behavioral consequences of reduced MSI, further investigations should be a high priority. Additionally, the types of stimuli employed are also a relevant factor to consider in future investigations, as existing research indicates that for individuals on the autism spectrum, MSI impairments may be greatest for more complex speech stimuli (Massaro & Bosseler, 2003; Mongillo et al., 2008; Smith & Bennetto, 2007; Williams, Massaro, Peel, Bosseler, & Suddendorf, 2004) in the context of relatively intact integration of simple sensory (Mongillo et al., 2008; van der Smagt, van Engeland, & Kemner, 2007), and normal early integration effects at the neural level (Magnee, de Gelder, van Engeland, & Kemner, 2008). Though a similar pattern has not emerged to date in the schizophrenia MSI literature, a further exploration of these parameters may be useful for understanding the extent and impact of these impairments, as well as the associated neural correlates, and possible points of convergence and divergence of these deficits between clinical populations. Another important dimension that requires further systematic exploration is the nature of the perceptual judgment required, for example categorizing the identity of a stimulus, the spatial location of a stimulus, the relative timing of stimuli, or merely detection of the stimuli as in the current task. As these different types of perceptual decisions rely on distinct neural substrates, a deeper understanding of any differential impairment on a particular type of task may be helpful in understanding the nature of this deficit. Finally, investigation of MSI between modalities other than vision and audition (e.g. vision and touch, or touch and audition) would also provide insight into whether these impairments are modality specific, or a more general phenomenon.

In conclusion, we find patients with schizophrenia to be impaired on one of the simplest indices of MSI, intersensory facilitation of RT for the detection of simple, temporally-congruent audio-visual targets. Within our sample, we are able to rule out a number of possible alternative explanations for the observed between group differences by matching a subset of our sample on overall unisensory RTs, as well as inter-subject variability. We also report, for the first time, a relationship between an individuals’ degree of MSI impairment and their current psychotic symptoms. This finding indicates this very basic sensory processing deficit may be related to the core features of schizophrenia, and should be further explored in future dedicated studies.