Neural Basis of Speech-Gesture Mismatch Detection in Schizophrenia Spectrum Disorders

Abstract

Patients with schizophrenia spectrum disorders (SSD) exhibit an aberrant perception and comprehension of abstract speech-gesture combinations associated with dysfunctional activation of the left inferior frontal gyrus (IFG). Recently, a significant deficit of speech-gesture mismatch detection was identified in SSD, but the underlying neural mechanisms have not yet been examined. A novel mismatch-detection fMRI paradigm was implemented manipulating speech-gesture abstractness (abstract/concrete) and relatedness (related/unrelated). During fMRI data acquisition, 42 SSD patients (schizophrenia, schizoaffective disorder, or other non-organic psychotic disorder [ICD-10: F20, F25, F28; DSM-IV: 295.X]) and 36 healthy controls were presented with short video clips of an actor reciting abstract or concrete sentences accompanied by either a semantically related or unrelated gesture. Participants indicated via button press whether they perceived each gesture as matching the speech content or not. Speech-gesture mismatch detection performance was significantly impaired in patients compared to controls. fMRI data analysis revealed that patients showed lower activation in bilateral frontal areas, including the IFG for all abstract > concrete speech-gesture pairs. In addition, they exhibited reduced engagement of the right supplementary motor area (SMA) and bilateral anterior cingulate cortices (ACC) for unrelated > related stimuli. We provide first evidence that impaired speech-gesture mismatch detection in SSD could be the result of dysfunctional activation of the SMA and ACC. Failure to activate the left IFG disrupts the integration of abstract speech-gesture combinations in particular. Future investigations should focus on brain stimulation of the SMA, ACC, and the IFG to improve communication and social functioning in SSD.

Introduction

In schizophrenia spectrum disorders (SSD), gesture interpretation and production impairment negatively affects social and community functioning, as meaningful manual movements are integral for successful communication.^1,2 There is converging evidence for disturbances of imitation, pantomime,^3–7 recognition,^8,9 and interpretation of gestures in schizophrenia.^10,11 Furthermore, gesture deficits are related to symptom severity, eg, negative symptoms, hallucinations, and formal thought disorder.^3,10–13 Negative symptom progression and social functioning can even be predicted by patients’ gesture performance.¹⁴ The perception of metaphoric gestures and detection of speech-gesture mismatches seem to be particularly aberrant in SSD.^13,15,16

Metaphoric gestures complement abstract speech by figuratively illustrating the semantic meaning.¹⁷ For instance, interlocking 2 fingers depicts a close friendship in the statement “The friends are inseparable.” However, if the same gesture accompanies a concrete phrase, eg, “The chains are firmly connected,” it is classified as an iconic gesture. In SSD patients, compared to controls, the integration of metaphoric gestures and abstract speech is impaired, reflected in reduced neural responses in the left middle temporal gyrus (MTG) and IFG.¹⁵ Additionally, neural connectivity from the left superior temporal sulcus (STS) to the bilateral IFG is disrupted for the perception of metaphoric gestures.¹⁶ IFG dysfunctions in schizophrenia are also associated with comprehension deficits for abstract language.^18,19 Disturbed bilateral IFG connectivity is a correlate of defective gesturing, being a part of the praxis network.^20,21 More generally, the IFG is responsible for semantic unification of multimodal complex stimuli.²²

With regard to speech-gesture relatedness, Nagels and colleagues¹³ demonstrated that SSD patients perform worse than controls in a mismatch-detection task, during which the relatedness between gesture and speech was judged as either related or unrelated. Likewise, patients’ relatedness evaluations were less accurate than controls’ in a transcranial direct current stimulation (tDCS) study where speech-gesture relatedness was evaluated on a Likert-like scale.²³ Left frontal inhibitory tDCS improved patients’ relatedness rating accuracy of speech-gesture pairs, suggesting frontal cortex involvement.²³ More general studies of audio-visual mismatch perception in schizophrenia further revealed increased frontal and insular signals²⁴ and an aberrant engagement of the right motor-speech area, including the pars opercularis of the IFG, the middle frontal sulcus, and the STG. This possibly reflects an increased processing demand for mismatched stimuli.^25–27 But so far, no study has examined the neural signature of speech-gesture mismatch perception in SSD.

The current study aimed to clarify SSD patients’ perception of and behavioral response to related and unrelated gestures. For this purpose, concrete and abstract sentence contexts were presented and subjects performed a mismatch-detection task while fMRI data were acquired for neural responses. This novel approach allows for a task-related examination of abstractness- and relatedness-processing in SSD. It might pave the way towards treatments that incorporate gesture-therapy²⁸ and neurostimulation to ameliorate communicative abilities and curb symptom progression.

Based on previous findings,¹³ we hypothesized that SSD patients would have difficulties in judging the relatedness of speech and gesture.

On a neural level, we expected that both groups would engage left temporal areas (STG, MTG) for abstract vs concrete stimuli. However, we hypothesized that SSD patients, compared to controls, would show reduced frontotemporal activation in response to abstract stimuli, particularly in the left IFG, as a sign of disturbed abstractness processing and multimodal integration.^15,16

We hypothesized that in both groups, perception of unrelated speech and gesture (mismatches) would result in higher frontal activation than the perception of semantically related pairs, due to an increased processing demand.^26,27

Yet, we expected the processing of mismatches to elicit less activation in the frontal cortex in patients compared to healthy controls, reflecting impaired mismatch-detection ability.

Methods

Patients

Forty-two patients (9 female, mean age = 34.3, SD = 11.1, range = 19–57) diagnosed with schizophrenia (n = 30), schizoaffective disorder (n = 11), and other non-organic psychotic disorder (n = 1) according to ICD-10 (F20, F25, F28) or DSM-IV (295.X) criteria, recruited from 2012 to 2019, were included in the final analysis. Out of an initial sample of 57 patients recruited and assessed by psychiatrists and psychologists of the Department of Psychiatry and Psychotherapy (Philipps-University Marburg), 15 patients were excluded from the final analysis (exclusion criteria: missing responses [≥ 10%] in behavioral task [n = 2], signal dropouts in functional images [n = 3], excessive movement during fMRI data acquisition [defined as > 1.5 mm relative movement or > 3 mm absolute movement, n = 10]^29–32). All 42 patients were German native speakers (3 bilinguals), 4 were left-handed, 24 were high school graduates. In the Multiple-Choice Word Test B (MWT-B),³³ patients achieved a mean score of 27.7 (SD = 7.5). Symptoms were assessed according to the Scales for Assessment of Positive Symptoms (SAPS, n = 39, sum mean = 19.74, SD = 19.1) and Negative Symptoms (SANS, n = 41, sum mean = 23.05, SD = 21.2).^34,35 All patients were clinically stable under antipsychotic medication (mean dose in chlorpromazine equivalents = 735.65 mg/day, SD = 1355.17).³⁶ Patients with a self-reported or documented history of psychiatric disorders were included if SSD was their main diagnosis.

All subjects were free of visual, auditory, and additional neurological deficits. Cerebral integrity was assessed by a T1-weighted MRI sequence; the T1 was missing for four patients due to technical issues.

Healthy Controls

Thirty-six out of 50 healthy control subjects (11 female, mean age = 36.8, SD = 11.2, range = 20–56) were included in the final analysis. Fourteen subjects were excluded (exclusion criteria: missing responses [≥10%] in behavioral task [n = 1], incidental abnormalities in T1-weighted MRI [n = 1], signal dropouts in functional images [n = 6], excessive movement during the fMRI paradigm [n = 6]).

All subjects were German native speakers (2 bilinguals), one was left-handed and 23 were high school graduates. Patients and controls were matched for education (supplementary table 1). They achieved a mean score of 30.9 (SD = 3.4) in the MWT-B, which was not significantly different from the patient sample (supplementary table 2 for participants’ neuropsychological test performance). Participants did not have any visual or auditory deficits, neurological or psychiatric disorders. T1-images of 6 subjects were missing due to technical issues.

This study was approved by the local Ethics Committee and written informed consent was obtained from all participants. Subjects received 50 Euro for participation.

Stimuli

The stimuli consisted of 160 video clips depicting an actor articulating a German sentence while performing a manual gesture. The material has been described and successfully used in previous behavioral¹³ and tDCS^23,37 studies. Part of it has been used in fMRI studies using implicit tasks.^{15,26,38–40} The sentence content of each video was either abstract (abs) or concrete (con) and the co-verbal gesture was either semantically related (rel) or unrelated (unr) to the speech, resulting in four video conditions (figure 1):

Fig. 1.

Video conditions. Four types of speech-gesture combinations. AR, abstract related; AU, abstract unrelated; CR, concrete related; CU, concrete unrelated.

• (1) Abstract speech and related metaphoric gesture (AR, abs/rel)

• (2) Abstract speech and unrelated gesture (AU, abs/unr)

• (3) Concrete speech and related iconic gesture (CR, con/rel)

• (4)  Concrete speech and unrelated gesture (CU, con/unr)

To countervail possible sequence effects, 2 stimulus sets of 80 videos each (20 videos per condition) were generated. Videos were presented in a pseudorandomized and counterbalanced order. Each sentence appeared only once per set, either with a related or unrelated gesture. Each subject saw only one set.

Experimental Design

For the fMRI experiment, participants were provided with earplugs and headphones. Videos were displayed on an MRI-compatible screen using Presentation software (Version 18.3, Neurobehavioral Systems, Inc.).

To best detect changes in BOLD-response across conditions, an event-related design was chosen. During data acquisition, subjects were presented with 20 stimuli per condition. Every video with a duration of 5 seconds was followed by a gray screen (low-level implicit baseline) for 5000 ms on average (variable between 3750 and 6750 ms), resulting in a total duration of 14 minutes for the experiment.

For each video, subjects were asked to determine whether the presented gesture and spoken sentence were semantically matching or not. Responses were given via button press on an MR-compatible answering device attached to the left thigh.

fMRI Data Acquisition

MRI data were collected using a Siemens 3 Tesla MR Magnetom Trio Trim scanner. Functional data were obtained applying a T2-weighted echo-planar imaging (EPI) sequence (repetition time [TR] = 2000 ms; echo time [TE] = 30 ms; flip angle = 90°). The volume included 33 transversal slices (slice thickness = 3.6 mm; interslice gap = 0.36 mm; field of view [FoV] = 230 mm, voxel resolution = 3.6 mm²). Four hundred twenty volumes were acquired for each subject. Subsequently, T1-weighted anatomical images were obtained.

Behavioral Data Analysis

Following signal detection theory,⁴¹ detection rates d′abs and d′con were calculated to determine each subject’s sensitivity for differentiating between related and unrelated stimuli in abstract and concrete contexts, respectively:

$${\displaystyle \begin{array}{l}{\displaystyle d\prime =z(hitrate)-z(falsealarmrate)}\end{array}}$$

Hits were defined as the number of correctly identified related items (AR and CR) and false alarms as the number of unrelated items (AU and CU) incorrectly identified as related. A repeated-measures ANOVA of d′abs and d′con values was performed in a 2 × 2 design with abstractness as a within-subject factor and group as a between-subject factor.

fMRI Data Analysis

For data quality control, all structural and functional files were visually inspected for artifacts, neuropathology, or abnormalities by authors M.C. and M.S. Functional MRI data were then analyzed using the Statistical Parametric Mapping software (SPM12, v6685, https://www.fil.ion.ucl.ac.uk/spm/software/spm12/) implemented in MATLAB 7.9.0 (release 2009b, The MathWorks, Inc.). To avoid saturation effects, the first 5 images of the measurement were discarded from the analysis.

First, all functional data were realigned to the mean image of the run. Next, images were normalized to the Montreal Neurological Institute (MNI) space (defined by tissue probability maps), resulting in a resliced voxel size of 2 mm³. Lastly, smoothing was performed with an 8 mm³ Gaussian kernel to adjust for anatomical variance between subjects. After preprocessing, realignment parameters were checked for excessive movement.^29–31

On a single-subject level, the onset of each event was defined as the integration point (the time when the stroke of the gesture coincides with the keyword of the sentence²⁶). All 80 events were modeled with a duration of 1 second and assigned to one of the four conditions (AR, AU, CR, and CU). Movement parameters were included as multiple regressors to correct for movement artifacts during data acquisition. The time between 2 videos was not modeled, thus serving as an implicit low-level baseline. This approach has been successfully implemented in previous experiments.^26,39,42

Group Analysis

Contrast images (baseline contrasts) for the four conditions were entered into a flexible-factorial analysis, considering group (patients, controls) as a between-subject factor and conditions (abstractness × relatedness: AR, AU, CR, and CU) as within-subject factors (2 × 2 × 2 design). Age was included as a covariate of no interest.

A Monte-Carlo-Simulation was performed (acquisition matrix: x = 64, y = 64; slices: 33; DIM: xy = 3.58 mm, z = 3.96 mm; FWHM = 13.4 mm; DIM resampled = 2 mm; no mask; iterations: 1000) to calculate the minimum voxel contiguity threshold needed to correct for multiple comparisons at P < .05, assuming an individual voxel type I error of P < .05.^43,44 A cluster extent threshold of 1308 contiguous resampled voxels at P < .05 (whole-brain analysis) was used for all contrasts of interest.

Voxel coordinates reported are located in MNI space. For anatomical location, functional data were referenced to the Automated Anatomical Labeling toolbox in SPM12.^45,46 For further statistical analyses of neural and behavioral data, SPSS (version 24.0) for Linux was utilized.

Contrasts of Interest

For main effects, interaction effects (F-tests), and within-group effects (T-tests), see the supplementary material.

For the contrast of abstract > concrete conditions, conjunctions were calculated to examine group similarities, and interaction T-tests were performed to clarify group differences.

• (1) C(abs > con) ∩ P(abs > con)

• (2)  C(abs > con) > P(abs > con)

• (3) P(abs > con) > C(abs > con)

For the contrast of unrelated > related conditions, conjunction and interaction T-tests were calculated likewise.

• (4) C(unr > rel) ∩ P(unr > rel)

• (5) C(unr > rel) > P(unr > rel)

• (6) P(unr > rel) > C(unr > rel)

Results

Behavioral Analysis

Patients exhibited significantly lower detection rates d′ compared to healthy controls (rm-ANOVA, between-subjects effect: F(1, 76) = 16.31, P < .001; post-hoc-tests, d′abs: t(76) = 3.975, P < .001; d′con: t(76) = 3.292, P = .002; 2-way). No difference between abstract and concrete conditions was found.

fMRI Data

Effects of Abstractness (Abstract > Concrete)

• (1) Controls and patients: C(abs > con) ∩ P(abs > con)

  • Conjunction analysis showed that both groups exhibited common activation of left middle temporal areas, the right STG, and bilateral superior frontal gyri (SFG) for abstract > concrete stimuli (figure 2a, red, table 1).

• (2) Controls > Patients: C(abs > con) > P(abs > con)

  • Interaction analysis revealed higher activation in bilateral frontal areas including the precentral gyri and IFG in healthy subjects > patients for abstract > concrete stimuli (figure 2b, blue, table 1).

• (3) Patients > Controls: P(abs > con) > C(abs > con)

  • The reverse interaction showed patients additionally engaging cerebellar structures for abstract conditions (figure 2c, yellow, table 1).

Fig. 2.

Left: Activation patterns for the contrast abs > con (abstract stimuli [AR and AU] > concrete stimuli [CR and CU]) in controls (C) and patients (P) (a, red); controls > patients (b, blue); patients >controls (c, yellow). Right: Contrast estimates of the significantly activated regions of each contrast, based on the extracted eigenvariate of activated clusters in respectively masked analyses (masks from WFU PickAtlas) using the VOI function in SPM12. Blue bars: control group. Yellow bars: patient group (for color figure refer online version). AR = abstract related, AU = abstract unrelated, CR = concrete-related, CU = concrete unrelated. Error bars: 95% CI of the mean.

Table 1.

fMRI Clusters Resulting From the Within-Group Conjunction and Between-Group Interactions of Abstract > Concrete Speech-Gesture Pairs, Corrected at P < .05 (Whole-Brain Analysis)

Abstract > Concrete Stimulus Processing
				MNI Coordinates
Contrast	Hemisphere	Cluster Extent	Anatomical Region of Peak	x	y	z	t-Value	No. of Voxels
Conjunction
C(abs > con) ∩ P(abs > con)	L	TMP, TSP, IFG p. opercularis, ROL, IFG p. orbitalis, MTG	Left MTG	−50	−36	−2	7.55	5981
			Left MTG	−56	−10	−12	7.15
			Left MTG	−54	4	−16	6.41
	R	TSP, TMP, IFG p. opercularis, MTG, STG, hippocampus	Right STG	52	−6	−12	6.35	4596
			Right STG	60	−10	0	5.55
			Right TSP	50	12	−22	5.25
	L + R	Right MSFG, right SMA, right SFG, left MSFG	Left SFG	−12	56	28	4.62	1986
			Left SMA	−6	10	70	4.22
			Left SFG	−12	32	56	3.13
	L + R	Left cuneus, right cuneus, vermis, left lingual gyrus, right precuneus	Left PCC	−4	−44	28	4.18	6378
			Right calcarine	14	−78	12	3.49
			Left precuneus	−6	−72	34	3.3
Interaction
C(abs > con) > P(abs > con)	L	PrG, IFG pars orbitalis, postcentral gyrus, insula, IFG pars triangularis	Left insula	−36	−8	18	3.99	2918
			Left IFG p. triangularis	−52	20	14	3.10
			Left IFG p. triangularis	−36	26	12	3.00
	R	PrG, insula, IFG pars orbitalis, Heschl gyrus, MFG	Right PrG	48	0	20	3.74	3733
			Right IFG p. triangularis	44	28	6	3.55
			Right ROL	42	−6	24	3.39
P(abs > con) > C(abs > con)	L + R	Cerebellum, vermis, fusiform gyrus, parahippocampal gyrus	Vermis	4	−54	−16	3.10	1844
			Right cerebellum	20	−46	−18	2.98
			Right cerebellum	24	−38	−24	2.74

Note: For each cluster, MNI coordinates and t-values of the first 3 peak voxels are listed. Anatomical regions refer to peak voxel localization based on the AAL toolbox (local maxima labeling) and cluster extent to the cluster labeling. C, control group; P, patient group; abs, abstract conditions; con, concrete conditions; L, left; R, right; TMP, temporal pole of middle temporal gyrus; TSP, temporal pole of superior temporal gyrus; IFG, inferior frontal gyrus; ROL, rolandic operculum; MTG, middle temporal gyrus; STG, superior temporal gyrus; MSFG, medial segment of superior frontal gyrus; SMA, supplementary motor area; SFG, superior frontal gyrus; SOG, superior occipital gyrus; MCC, middle cingulate cortex; PCC, posterior cingulate cortex; PrG, precentral gyrus; MFG, middle frontal gyrus.

Effects of Relatedness (Unrelated > Related)

• (4) Controls and patients: C(unr > rel) ∩ P(unr > rel)

  • Conjunction analysis revealed common activation in both groups for unrelated > related stimuli in bilateral medial segments of the superior frontal gyri (MSFG) and rostral supplementary motor areas (SMA) (figure 3a, red, table 2).

• (5) Controls > patients: C(unr > rel) > P(unr > rel)

  • In the interaction analysis, controls exhibited increased activation in the right SMA, bilateral ACC, and left precentral gyrus for unrelated > related stimuli compared to patients (figure 3b, blue, table 2).

• (6) Patients > controls: P(unr > rel) > C(unr > rel)

  • The reverse interaction revealed activations in right hippocampal, superior temporal, left frontal and bilateral cerebellar regions in patients > controls (figure 3c, yellow, table 2).

Fig. 3.

Left: Activation patterns for the contrast unr > rel (unrelated stimuli [AU and CU] > related stimuli [AR and CR]) in controls and patients (a, red), controls > patients (b, blue), and patients > controls (c, yellow). Right: Contrast estimates of the peak activated regions of each contrast, based on the extracted eigenvariate of activated clusters in respectively masked analyses (masks from WFU PickAtlas) using the VOI function in SPM12. Blue bars: control group. Yellow bars: patient group (for color figure refer online version). AR = abstract related, AU = abstract unrelated, CR = concrete related, CU = concrete unrelated. Error bars: 95% CI of the mean.

Table 2.

fMRI Clusters Resulting From the Within-Group Conjunction and Between-Group Interactions of Unrelated > Related Speech-Gesture Pairs, Corrected at P < .05 (Whole-Brain Analysis)

Unrelated vs Related Stimulus Processing
				MNI Coordinates
Contrast	Hemisphere	Cluster Extent	Anatomical Region of Peak	x	y	z	t-Value	No. of Voxels
Conjunction
C(unr > rel) ∩ P(unr > rel)	L + R	Right MSFG, right SMA, left ACC, left MSFG	Right MSFG	8	26	42	4.26	1976
			Left SMA	−6	18	50	3.73
			Right SMA	8	24	56	3.26
Interaction
C(unr > rel) > P(unr > rel)	L + R	Right SMA, left ACC, left precentral gyrus, left paracentral lobule, right MCC	Left SMA	−6	6	62	3.48	3100
			Right ACC	2	16	24	3.35
			Left MCC	−10	2	42	3.04
P(unr > rel) > C(unr > rel)	R	Hippocampus, lingual gyrus, thalamus, parahippocampal gyrus, STG	Right STG	40	−40	4	3.97	1471
			Right putamen	28	−24	2	3.17
			Right thalamus	22	−14	−2	3.00
	L + R	Left SFG, left MFG, left insula, right SFG, right MFG, right MSFG	Left IFG pars orbitalis	−24	36	−4	3.12	1339
			Right MSFG	12	54	10	2.99
			Left SFG	−24	52	0	2.95
	L + R	Vermis, right cerebellum, left cerebellum, right ITG	Right cerebellum	8	−80	−20	3.05	2331
			Left cerebellum	−28	−76	−34	2.93
			Right cerebellum	20	−66	−20	2.93
	L + R	Left angular gyrus, right precuneus, right SPG, left precuneus, left MOG	Left precuneus	0	−66	60	2.96	1613
			Left IPL	−32	−72	46	2.87
			Right precuneus	6	−58	32	2.79

Note: For each cluster, MNI coordinates and t-values of the first peak voxel are listed. Anatomical regions refer to peak voxel localization based on the AAL toolbox (local maxima labeling) and cluster extent to the cluster labeling. C, control group; P, patient group; unr, unrelated condition; rel, related condition; MSFG, medial segment of superior frontal gyrus; SMA, supplementary motor area; ACC, anterior cingulate cortex; MCC, middle cingulate cortex; STG, superior temporal gyrus; SFG, superior frontal gyrus; MFG, middle frontal gyrus; IFG, inferior frontal gyrus; ITG, inferior temporal gyrus; SPG, superior parietal gyrus; MOG, middle occipital gyrus; IPL, inferior parietal lobule

Discussion

In this fMRI study, we examined the neural processes underlying speech-gesture mismatch detection for abstract and concrete semantic contexts in SSD patients and healthy controls. Interaction analysis revealed a bilateral IFG dysfunction for abstract speech-gesture conditions in patients with SSD. However, both patients and controls showed increased temporal cortex activation for the processing of abstract in contrast to concrete stimuli. While superior frontal cortex activation during mismatch perception was found in both groups, patients still exhibited reduced activity of the SMA and ACC and frontotemporal hyperactivation. These neural aberrations may contribute to impaired mismatch-detection performance in SSD.

In line with our hypothesis, the IFG showed reduced activation in the patient group for abstract stimuli compared to controls. This region plays a key role in the processing of abstract speech-gesture pairs^15,16 and seems to be disrupted in SSD patients. This is supported by studies suggesting that SSD patients’ diminished ability to distinguish between abstract and concrete stimuli is related to IFG dysfunctions.^15,19 The IFG’s role as an integration site for complex multimodal stimuli,^22,27 such as metaphoric gestures, may further explain how disturbances of this region lead to gesture comprehension deficits in SSD.^10,11,13 Different neural network connectivity from STS to bilateral IFG might underlie the impaired processing of metaphoric gestures in these patients.¹⁶

Still, our data suggest that the temporal lobe is similarly engaged in patients and controls for the perception of metaphoric gestures. This common activation suggests that at least some neural mechanisms relevant for abstractness processing are unimpaired in patients with SSD, providing the basis for successful interventions, such as gesture training,²⁸ transcranial magnetic stimulation,⁴⁷ or tDCS.²³

The IFG and MTG are part of the left-hemispheric “praxis network” associated with gesture planning.²¹ A recent study showed a significant correlation of gesture performance and functional connectivity between the bilateral STG in healthy subjects, whereas patients displayed reduced connectivity.⁴⁸ Furthermore, gesture deficits were predicted by reduced connectivity between the bilateral IFG in the schizophrenia group.²⁰ These functional alterations in the frontotemporal network may hinder efficient gesture planning and processing in SSD.

In accordance with earlier behavioral studies,^13,23 we found reduced speech-gesture mismatch detection performance in SSD patients compared to healthy subjects, but no differences between abstract and concrete conditions. Furthermore, during processing of mismatches, patients exhibited decreased activation in the right SMA and the left ACC. In contrast, the right STG and left frontal cortex showed hyperactivation in patients. This network may be associated with impaired speech-gesture mismatch perception and detection performance.

The SMA is involved in movement control, speech production,^49–51 and gesture perception.⁵² Increased SMA gray matter volume was found in schizophrenia patients with strong motor deficits.⁵³ Motor dysfunctions are a common phenomenon in SSD patients with gesture impairment^7,54 and may therefore contribute to decreased recognition of mismatching speech-gesture pairs. The common activation of the rostral SMA and SFG in both groups shows that mismatch processing in this area is partially unaffected in SSD patients.

ACC activity has been elicited by unrelated iconic gestures in healthy subjects,⁵⁵ reflecting a surprise reaction to an unexpected speech-gesture combination. Decreased ACC response could result from patients’ reduced ability to differentiate the relatedness of co-verbal gestures. Patients might be less surprised by a mismatching speech-gesture combination, since they tend to evaluate them as related. Deficits in conflict monitoring and error-processing have also been attributed to ACC dysfunctions in schizophrenia.^56,57

The inadequate engagement of superior temporal and frontal cortices in SSD patients for mismatches might reflect an increased effort to disambiguate stimuli, as previously observed in frontal areas during an audio-visual mismatch trial.²⁴

Profound knowledge of dysfunctional neural networks can promote new therapeutic approaches to improve social functioning, which is especially hard to address in SSD patients. Given that gesture deficits outlast improving symptom severity, alternative therapy methods are needed.^4,28 Existing evidence shows the positive effect of brain stimulation on relatedness assessment accuracy^23,37 and gesture performance.⁴⁸ The current study results suggest that the IFG, SMA, and ACC could be new targets for stimulation interventions in combination with speech- and gesture-therapies.^28,48

Some limitations have to be considered for the interpretation of our findings. Patients were moderately ill and received individual medication; thus, medication effects cannot be ruled out (supplementary material). Patient samples displaying stronger symptoms in certain categories (eg, formal thought disorder, negative symptoms) could yield different effects. Also, patients were not specifically tested for psychiatric diagnoses other than SSD.

Furthermore, intelligence and semantic processing are tightly intertwined, so that differences caused by cognition are not precluded, although patients and controls were matched for education.

Age was included as a covariate of no interest and no statistical correlation was found with our main results. Since the groups were matched, effects evoked by the age range are unlikely (supplementary material).⁵⁸

In both groups, more male than female participants were included. Future investigations should aim for an equal representation of the sexes. However, groups were matched for sex, so that no relevant effects on our results are expected.

Although the number of left-handed subjects was not balanced across groups, no difference in activation was found whether they were included in the analysis or not (supplementary material).

Also, fMRI-compatibility was required for participation. Because equal numbers of patients and controls had to be excluded from the analysis based on our data quality criteria,^29–32 we do not expect the selection to have influenced our results.

While our paradigm was optimized to detect speech-gesture matching impairments, effects provoked by speech, vs by gesture alone, cannot be differentiated.

Additionally, it is not possible to distinguish task effects from purely perceptive effects in the neural data since we did not implement a non-task control condition for task practicability.

Our results suggest that aberrations of the IFG in SSD contribute to defective processing of abstract gesture and speech. Furthermore, dysfunctions of the SMA and ACC may affect the processing of mismatched speech-gesture information in SSD, contributing to impaired semantic processing. These regions may be appropriate targets for brain-stimulative interventions²³ and speech-therapeutic methods,^28,47 which could improve social interaction and prevent symptom progression in SSD patients.

Funding

This research project is supported by a grant from the “Deutsche Forschungsgemeinschaft” (project no. DFG: Ki 588/6-2; STR 1146/11–2; B.S. is funded by STR 1146/15-1) and the Rhön Clinic (project no. FI_3).