Aberrant activity in conceptual networks underlies N400 deficits and unusual thoughts in schizophrenia

Abstract

Background

The N400 event-related potential (ERP) is triggered by meaningful stimuli that are incongruous, or unmatched, with their semantic context. Functional magnetic resonance imaging (fMRI) studies have identified brain regions activated by semantic incongruity, but their precise links to the N400 ERP are unclear. In schizophrenia (SZ), N400 amplitude reduction is thought to reflect overly broad associations in semantic networks, but the abnormalities in brain networks underlying deficient N400 remain unknown. We utilized joint independent component analysis (JICA) to link temporal patterns in ERPs to neuroanatomical patterns from fMRI and investigate relationships between N400 amplitude and neuroanatomical activation in SZ patients and healthy controls (HC).

Methods

SZ patients (n = 24) and HC participants (n = 25) performed a picture-word matching task, in which words were either matched (APPLE→apple) by preceding pictures, or were unmatched by semantically related (in-category; IC, APPLE→lemon) or unrelated (out of category; OC, APPLE→cow) pictures, in separate ERP and fMRI sessions. A JICA “data fusion” analysis was conducted to identify the fMRI brain regions specifically associated with the ERP N400 component. SZ and HC loading weights were compared and correlations with clinical symptoms were assessed.

Results

JICA identified an ERP-fMRI “fused” component that captured the N400, with loading weights that were reduced in SZ. The JICA map for the IC condition showed peaks of activation in the cingulate, precuneus, bilateral temporal poles and cerebellum, whereas the JICA map from the OC condition was linked primarily to visual cortical activation and the left temporal pole. Among SZ patients, fMRI activity from the IC condition was inversely correlated with unusual thought content.

Conclusions

The neural networks associated with the N400 ERP response to semantic violations depends on conceptual relatedness. These findings are consistent with a distributed network underlying neural responses to semantic incongruity including unimodal visual areas as well as integrative, transmodal areas. Unusual thoughts in SZ may reflect impaired processing in transmodal hub regions such as the precuneus, leading to overly broad semantic associations.

Highlights

• •Hemodynamic activity linked to N400 responses is distributed across the brain.
• •N400 linked networks depend on conceptual relatedness.
• •Magnitude of the N400 ERP-fMRI fused component is reduced in schizophrenia.
• •N400 linked network activity inversely correlates with unusual thought content.

1. Introduction

1.1. he N400 as a tool to study semantic processing

The human capacity to map semantic meaning onto objects has long been thought to result from an interaction between perceptual and conceptual functions (Peirce, 1868). Semantic processing refers to the generation and mapping of those meanings in symbolic language (Deacon, 1998). The brain is exquisitely tuned to deviations in semantic context, revealed electrophysiologically as a negative deflection known as the N400 ERP (Kutas and Federmeier, 2011), peaking 400 ms after the presentation of a stimulus that is not semantically related to the context. Although initially discovered from semantic violations within a sentence (Kutas and Hillyard, 1980), the N400 has been reported for a variety of paradigms, including incongruous placement (Ganis and Kutas, 2003) or utilization of objects (Balconi and Caldiroli, 2011). Such evidence suggests a range of cortical networks might subserve the N400 response, however, the anatomical generators have been studied primarily in language tasks, where most stimuli are words, spoken or written (Lau et al., 2008). Knowledge of the anatomical generators is also limited by the use of scalp recorded ERPs, which have excellent temporal, but poor anatomical precision (Nunez and Srinivasan, 2006).

1.2. Relevance to schizophrenia (SZ)

Impaired semantic networks are seen in SZ (Kreher et al., 2008; Spitzer, 1997) and might underlie aspects of formal thought disorder or unusual thought content (Covington et al., 2005). We have previously found evidence for diminished N400 responses in SZ for cross-modal picture-word matching tasks, which have been interpreted as an overly broad spread of semantic spreading activation (Mathalon et al., 2002, Mathalon et al., 2010). N400 amplitudes in SZ are associated with real-world functional impairment (Jackson et al., 2014), psychotic symptoms such as delusions (Kiang et al., 2008), formal thought disorder (Kostova et al., 2005) and responses to pharmacotherapy (Condray et al., 2003). The contribution of visual processing deficits and/or the brain networks subserving semantic impairment remains unknown.

1.3. Goals of this experiment

To understand the neural basis of N400 and how it is affected by SZ, we integrated ERP and fMRI datasets collected from independent recording sessions during a picture-word semantic priming task. Results from the ERP portion of this dataset have already been reported (Mathalon et al., 2010). We utilized joint independent component analysis (JICA) to determine the patterns of spatial (from fMRI) and temporal (from ERP) covariance associated with processing words preceded by pictures that were matches, related semantically, or unrelated to the pictures.

JICA is a data-driven, blind source separation technique which can explain the underlying structure of multimodal data. We predicted that an N400 joint component would reflect covariation of N400 amplitude, revealing reduced loading weights of the identified joint component in SZ patients relative to healthy control (HC) subjects. Furthermore, we hypothesized that the joint component would be linked to activation in a distributed language network including inferior frontal and temporal regions as reported previously (Lau et al., 2008). To address these hypotheses, we utilized JICA (Calhoun et al., 2009; Sui et al., 2012) to identify brain regions associated with the ERP N400 component. We used the results of the JICA to probe for correlations in clinical symptoms.

2. Materials and methods

2.1. Participants

Patients with schizophrenia (n = 26) and age-matched healthy controls (n = 27) participated in both EEG and fMRI recording sessions (five participants were excluded for motion artifact, 3 HCs and 2 SZs, see below). ERP and fMRI recording session order was counterbalanced across groups and performed on average 16.1 days apart (range: same day-93 days, see Supplementary Table 1). Demographic data (including age, sex, parental SES, handedness, years of education and PANSS symptom ratings are included in Supplementary Table 1.) Experimental procedures were fully described to all participants after which they provided written informed consent. This study was approved by the Human Subjects Committees at the Connecticut VA Healthcare System and Yale University. Patients were recruited from community mental health centers, outpatient services of the Veterans Affairs Connecticut Healthcare System in West Haven, and outpatient services of the Connecticut Mental Health Center in New Haven. All patients were interviewed by a clinical psychologist or a clinically trained research assistant using the Structured Clinical Interview for DSM-IV (SCID; (First et al., 1995). Patients who met DSM-IV criteria for schizophrenia were included in this analysis. Patients were excluded if they met DSM-IV criteria for alcohol or drug abuse within 30 days prior to the study. Patients and control participants were excluded for significant head injury, neurological disorders, or other medical illnesses compromising the central nervous system. Symptoms were rated using the positive and negative symptom scale (PANSS; (Kay et al., 1987). Control participants were recruited by newspaper advertisements and word-of-mouth, screened by telephone using questions from the SCID non-patient screening module, and excluded for any history of Axis I psychiatric illness.

2.2. Task and behavior

We used the same picture–word verification task (Ford et al., 2001) utilized in the prior publication of ERP data pertaining to this dataset (Mathalon et al., 2010) for both ERP and fMRI recording sessions (Fig. 1A). In this task, a simple line drawing of an object is presented (250 ms), and following a short delay (75 ms), a word appears. Subjects are instructed to respond by indicating whether the word was a semantic match (50% of trials) or non-match (unmatched, 50% of trials) to the preceding picture. Pictures consisted of 102 line drawings selected from (Snodgrass and Vanderwart, 1980) and classified into 10 natural categories (clothing, animal, bird, appliance, tool, vehicle, vegetable, fruit, toy, and musical instrument). The full set of pictures comprised a block of 102 trials which was repeated across a total of four blocks, with the order of pictures varying across blocks. Half of the unmatched trials comprised picture-word pairs from a related semantic category (related co-hyponym, e.g., APPLE →lemon) and the other half comprised picture-word pairs from unrelated semantic categories (unrelated, e.g., APPLE →cow). Subjects were not asked to distinguish between related and unrelated unmatched target words. Occasional null events were inserted between trials, resulting in trial-to-trial intervals that ranged from 3 to 11 s. Participants responded by button press with right or left index fingers to indicate if the word matched or did not match the picture. Left / right button presses were counterbalanced across participants. All participants performed several practice trials prior to ERP or fMRI recordings. No feedback was given to signal performance accuracy and participants were told to respond as quickly as possible without sacrificing accuracy. All subjects performed the task at >90% accuracy (See Supplementary Table 1. HC = 95.0% and SZ = 91.8%, F(1,45) = 5.7295, p = 0.02). We found no effect of practice across the two sessions, performance accuracy was similar for the first and second session across groups (paired t-test, t-stat = −0.70, p = 0.49) and remained insignificant after controlling for the time difference between sessions (p = 0.49). Moreover, there was no effect of session order or time between sessions on the loading weight of the joint component reported below (ANCOVA model with covariates of group, session order and times between sessions). Reaction latency data were collected, and the median reaction times per subject, per condition (matched, unmatched) was determined (see Supplementary Methods and Results). All participants showed slower reaction times during the fMRI session than EEG session (see Supplementary Fig. 1) without higher level interaction effects between Group (HC or SZ) or Session (EEG or fMRI).

Fig. 1.

A. The visual stimuli and behavioral paradigm used in these studies. In the priming task, participants indicated whether a word was a semantic match or non-match (unmatched) relative to the preceding picture. In this task, a simple line drawing of an object is presented (250 ms), and following a short delay (75 ms), a word appears. Half of the non-match trials comprised picture-word pairs from a related semantic category (e.g., banana-lemon) and the other half comprising picture-word pairs from OC semantic categories (e.g., banana-cow). Subjects performed a 2-alternative forced choice (2-AFC) to indicate whether the word was a semantic match (50% of trials) or non-match (50% of trials) to the preceding picture. B. Average N400 ERP difference waves (unmatched minus matched) for IC, left, and OC, right, shown as a grand average across groups (solid black) and as back-projected waveforms for each component (component #1 solid blue; component #2 dashed blue). (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.)

2.3. EEG acquisition

Data acquisition is as described in (Mathalon et al., 2010), to summarize briefly: EEG data were digitized using a 1000 Hz sampling rate, low pass filtered at 100 Hz, and high pass filtered at 0.05 Hz. Data were acquired from F7, F3, Fz, F4, F8, T3, C3, Cz, C4, T4, T5, P3, Pz, P4, T6, O1, Oz, and O2. Data were referenced to linked ears. Additional electrodes were place on the outer canthi of both eyes and above and below the left eye to record eye movements and blinks (vertical and horizontal electro-oculogram (EOG); VEOG, HEOG). All impedances were maintained at or below 10 kΩ throughout the recording session with most EEG sites around 5 kΩ.

2.4. EEG processing

Continuous EEG Data were bandpass filtered between 1 and 30 Hz, epoched and stimulus-locked to picture onset. Individual trials were baseline corrected using the 100 ms period preceding picture onset after correcting for eye movements and blinks using EOG data (Gratton et al., 1983). Trials containing artifacts (voltages exceeding ±100 μV) were rejected. Only data from correct trials were included in the analyses. N400 difference waveforms were derived on a subject-by-subject basis by subtracting matched trial averages from the two unmatched conditions (related and unrelated). The resulting difference waveforms from electrode Cz (electrode with maximal N400 magnitude) were used for Joint Independent Component Analysis (see details below).

2.5. MRI data acquisition, preprocessing and analysis

Structural and functional MRI data were collected using a 3T Siemens Trio scanner. The structural imaging protocol was a magnetization-prepared rapid gradient-echo (MPRAGE) T1-weighted high-resolution image (2530 ms TR, 3.66 ms TE, 1 mm slice thickness, 256 mm field of view, 1.0 × 1.0 × 1.0 voxel size, flip angle 7°, sagittal orientation, 8:07 min). The fMRI protocol was an AC-PC aligned echo planar imaging (EPI) sequence (2000 ms TR, 28 ms TE, flip angle 60°, 33 slices collected sequentially in ascending order with a 12% (0.5 mm) gap, 4.0 × 4.0 × 4.0 mm voxel size, 257 frames, 8:34 min per run).

Preprocessing of fMRI images was done using Statistical Parametric Mapping 8 (SPM8; http://www.fil.ion.ucl.ac.uk/spm/software/spm8/). Motion correction was performed via affine registration, where the first image of each run was realigned to the first image of the first run, followed by re-alignment within each run. Next, Images were slice-time corrected with respect to the middle slice. We implemented aCompCor (anatomic component-based noise correction, Behzadi et al., 2007), a principal components-based approach for noise reduction of fMRI time-series data. Given that aCompCor uses white matter and CSF regions identified from FreeSurfer anatomical segmentation of T1 images (Fischl, 2012) to identify noise components, five participants were excluded (3 HCs and 2 SZs) due to excessive head motion or scanning artifacts as determined by manual examination of FreeSurfer reconstructions.

First-level modeling of the fMRI data was performed using an event-related analysis aligned to the onset of the picture. SPM's canonical hemodynamic response function (a double gamma function) was convolved with task event vectors to create a first-level task regressor representing the predicted response of a voxel activated by visual onset of the picture. A high pass temporal filter (128 s cut-off) was applied to remove low-frequency noise and a general linear model was implemented voxel-wise, regressing each voxel's time series on the task regressor, as well as on nuisance regressors consisting of the aCompCor noise components. The time of correct or incorrect button presses were modeled and included as nuisance regressors during the denoising step. Beta coefficients representing the fit of the task regressor to a voxel's time series were estimated, to generate beta images for the 3 conditions of interest (matched, OC unmatched and IC unmatched) versus implicit baseline (null time). Mean beta images across runs were normalized by applying the spatial transformation matrix derived from normalization of the mean functional image (generated during motion correction) to the Montreal Neurological Institute's EPI template (http://www.bic.mni.mcgill.ca) (Calhoun et al., 2017). Normalized beta images were resliced to 3 mm³ isotropic voxel dimensions using a fourth degree B-spline and were then spatially smoothed with an 8 mm full-width-half-maximum Gaussian kernel. Subsequently, each subject's first-level analysis brain mask image was normalized and applied to the smoothed beta image to mask out regions with insufficient signal intensity, as determined by SPM during first level modeling. The resulting first-level model contrast images were utilized in the JICA, see below. We report the results of a second-level general linear model (GLM) in the supplementary material.

2.6. Joint independent component analysis of EEG and fMRI data

To identify anatomical regions linked to the N400 component, we conducted a joint independent component analysis (JICA) using the Matlab-based Fusion ICA Toolbox or “FIT” (Rachakonda, Liu, and Calhoun 2008; http://mialab.mrn.org/software/fit). We entered each subject's N400 difference waves and first-level model contrast images for the two conditions of interest: IC versus matched and OC versus matched. These two conditions formed two “features” within the JICA analysis, so were concatenated side-by-side for a total of four “features:” (1) OC versus matched ERPs, (2) OC versus matched fMRI, (3) IC versus matched ERPs, (4) OC versus matched fMRI. The ERP waveforms are upsampled by a factor of 121 so that the number of samples in time match the number of fMRI samples in space. The use of difference waves, instead of condition grand averages as reported previously (Mathalon et al., 2010) resulted in similar statistical effects: N400 difference waves which are larger for OC and reduced in SZ (Supplementary Fig. 2; main effect of group, F(1,47) = 15.36, p = 0.0003; main effect of condition, F(1,47) = 217.22, p < 0.00001; group x condition, F(1,47) = 14.22, p = 0.0004).

Briefly, the JICA algorithm utilizes temporal information in the ERPs and spatial information in the fMRI to look for inter-subject covariation and identify “fused” ERP-fMRI components which maximize the joint likelihood function. This approach identifies “features” which covary the same way across participants. The algorithm outputs loading weights for each subject, which reflect the magnitude of the joint component for that particular subject (the coefficients of the unmixing matrix). Loading scores were compared between groups using independent samples t-tests. Loading weights can be used to “back-project” temporal and spatial maps for each feature: in our case, one spatial map and time course for the IC vs. match condition and one spatial map and time courses for the OC vs. match condition. A priori, we planned to investigate components with time courses that correlated with the N400 and regressed those time courses onto the N400 grand average ERP to assist with component selection. We estimated the number of components to calculate based on minimum descriptive length criteria for fMRI (Rissanen, 1983) and a parallel test for ERP (Parmet et al., 2010) for each feature. Given the primacy of our interest in identifying a joint component that clearly captured the N400 difference wave in the ERP waveforms, we considered ICA solutions involving extractions of between 2 and 7 components. After reviewing the components resulting from these solutions, it was clear that the 2-component solution yielded a joint component that clearly captured the N400 difference waveform for both IC and OC conditions (IC condition: R² = 0.924, p < 0.0001, OC condition: R² = 0.944, p < 0.0001). Accordingly, this joint component was extracted and used in our subsequent analyses.

Given our interest in identifying regions correlated with the N400, we focused our follow-up analyses on voxels with a positive correlation with the N400. Spatial maps were subject to thresholding at Z > 4.42 (p < 0.000005) and convolved with the FSL-Harvard Standard Atlas (Desikan et al., 2006). All voxels in the brain contribute to the loading weight; empirical thresholding identifies regions that are most strongly linked to the N400 magnitude. In our analyses, the maps revealed a max z-score of ~8 for the OC map and max z-score of ~10 for the IC map, so we selected a z-score cutoff that approximated half of the height of the average maximum for the two maps (Z > 4.42). Threshold free maps are also available on neurovault (https://neurovault.org/). All clusters larger than 18 voxels (the minimum ROI size in the FSL-Harvard Standard Atlas) were included and labeled according to the regional area that composed the majority of the voxels for that cluster.

2.7. Clinical correlations

We conducted correlations between activity (contrast estimates) extracted from z-thresholded IC and OC JICA spatial maps with clinical measures. We predicted that clinical deficits in language function (as reflected in scores on thought process and content) in SZ patients would be related to reductions of activity in conceptual networks (inverse correlation between activity and impairment). We extracted conceptual disorganization (P2) and unusual thought content (G9) from the PANSS ratings. Given the correlation between disorganization and content (r = 0.69, p = 0.0002) we conducted multivariate regression analysis to look for independent effects of process and content (Andreasen, 1982). To examine the distribution of symptom correlations within the thresholded JICA maps, conceptual disorganization and unusual thought content PANSS ratings were entered as predictors in a GLM analysis in SPM separately for the IC minus matched and OC minus matched contrasts of interest. We used inclusive masks from the corresponding JICA z-thresholded spatial maps (one for each condition) to restrict whole brain T-maps for visualizing the distributions of relationships between contrast images and PANSS ratings.

3. Results

3.1. Joint independent component analysis (JICA)

Subject level data from fMRI first-level contrasts and ERP N400 difference waves (OC minus matched and IC minus matched) were utilized for JICA (see methods for details) to examine the two priming conditions of interest, OC and IC. A single JICA component (#1) captured the N400 difference waveform for both conditions (IC and OC, Fig. 1B). This component also revealed a pronounced reduction in loading weights for SZ participants (t-stat = 3.3047, p = 0.0018). In an ANCOVA model, controlling for age, there was no effect of age on the component loading weights (t-stat = −1.29, p = 0.21) and the group effect remained significant (t-stat = −2.803, p = 0.007).

Two fMRI whole brain maps show the ICA spatial weights (expressed as z-scores) for the IC and OC condition contrasts (each relative to the matched condition) associated with this JICA component (see Fig. 2). These maps represent a wide range of modeled hemodynamic activations evoked during IC and OC task conditions and show the degree to which they are linked to the N400 difference wave (max z-score of ~8 for the OC map, and max z-score of ~10 for the IC map). In order to isolate the brain regions with the strongest links to N400 for additional analyses, the ICA spatial maps were thresholded (Z > 4.42, see methods). The resulting brain regions, with their peaks, corresponding to the voxels with the strongest contribution to the N400 joint component, are shown in Table 1. The thresholded IC and OC spatial maps appeared to be quite distinct, overlapping by only 3 voxels in the L anterior temporal pole: The IC map included midline cingulate areas, cerebellum, the precuneus and posterior cingulate as well as the bilateral anterior temporal pole, while the OC map included occipital visual areas near the lingual gyrus, the right superior parietal lobule and the left anterior temporal pole.

Fig. 2.

Z-scored spatial maps for the joint component during the unmatched, IC (left) and OC (right) conditions. Voxels with a negative Z score or those that did not meet threshold (Z < 4.42) are mostly transparent. Top row: medial view of right hemisphere (slice coordinate x = 1); Middle row: posterior view; Bottom row: anterior view with coronal cut (slice coordinate y = 8) for visualization of the temporal poles.

Table 1.

Regions of peak activation from IC and OC joint component maps.

Region	Brodmann area(s)	MNI coordinates	Peak Z
IN Category Map
R Cerebellum	Vermis 1,2	3–43 -26	7.4055
R Cerebellum	3 R	18–28 -23	6.9175
L Cerebellum	6 L	−18 -70 -11	5.6867
L Temporal Pole	38	−33 8–23	7.1122
R Temporal Pole	38	39 5–20	6.2919
R Middle Temporal Gyrus	39	51–64 13	6.1017
Precuneus R/L	7	15–82 46	10.0686
Cingulate R/L	31	3–34 43	8.52
Mid Cingulate R/L	24	3 11 37	6.3788
OUT of Category Map
L Temporal Pole	38	−42 5–14	6.5322
Calcarine L/R	17, 18	15–100 -5	6.2919
R Sup Parietal Lobule	7	18–67 55	6.0185

3.2. Relation to clinical measures

Among SZ patients, loading weights for the joint component were not correlated with conceptual disorganization (r = −0.17, p = 0.47) or unusual thought content (r = −0.16, p = 0.46). Unusual thought content (while controlling for conceptual disorganization) was inversely correlated with extracted contrast values from the IC map (Fig. 3A, r = −0.54, p = 0.006) and the OC map (r = −0.42, p = 0.047). This correlation held across a range of Z-thresholds (z = 2 to z = 6) for the component maps (test for homogeneity among correlated correlation coefficients (Raghunathan, 2003) indicated no correlation coefficient was significantly different from the others; all p's > 0.57). This correlation appeared to reflect distributed activity from across the spatial maps (Fig. 3B). Conceptual disorganization (controlling for unusual thought content) did not correlate with contrast values from either map (both p's > 0.2). We did not find a correlation between unusual thought content or disorganization with N400 amplitude in the IC or OC conditions (all p's > 0.500). In addition, there was no significant correlation between unusual thought content and age(r = 0.06, p = 0.78), education (r = −0.21, p = 0.34) or parental SES (r = −0.33, p = 0.11). Thus, the hemodynamic response from voxels linked to the N400, but not ERP amplitudes themselves, inversely correlates with unusual thought content.

Fig. 3.

A. Scatter plots of PANSS unusual thought content (controlling for disorganization, top row) with contrast estimates extracted from the IC (left) and OC (right) joint component spatial maps. B. T-maps of the relationship between the OC vs matched (left) and IC vs matched (right) and unusual thought content (controlling for disorganization). Views of the brain are the same as in Fig. 2.

4. Discussion

The principle finding of this study is that JICA captured the N400 waveform with a component linked to activity in a distributed conceptual network which includes unimodal visual as well as transmodal areas (Mesulam, 1994). Loading weights for this ERP-fMRI “fused” component were reduced in SZ. Hemodynamic activity in these networks was negatively correlated with unusual thought content in SZ when controlling for conceptual disorganization.

4.1. Neuroanatomy associated with N400

We hypothesized that a distributed network of language areas, rather than a focal brain region, might underlie the N400 ERP. The JICA approach measures covariation, so we cannot infer causality regarding underlying ERP generators or dipole localization. It is possible that the N400 generators (which are not captured by fMRI) are in fact associated with the component map. Additional generators may mediate the relationship between the N400 response and the areas captured by the JICA. Neural generators might be associated with a distributed network even if source localization procedures map the ERP to a focal site, as is suggested by dynamic causal modeling of ERP data (Daunizeau et al., 2009). Alternatively, the generators themselves may be distributed intracranial sources (Halgren, 2004; McDonald et al., 2010), given that not all cortical currents are revealed in EEG and MEG, potentially owing to effects of cancellation (Halgren, 2004; Irimia et al., 2012). Lastly, the JICA approach employs neural data to improve the identification of hemodynamic spatial information, which are likely to reflect different neuronal computations than ERPs (Logothetis, 2008; Logothetis and Wandell, 2004). We agree with Kutas and Federmeier (2011), that regarding the functional localization of the N400: “...the question becomes not where is the N400 generator localized, or whether there are multiple N400s, but rather what are the functions of the dynamic neural system of which scalp N400s are reflections?” Our results suggest that the N400 reflects dynamic networks which are flexibly engaged depending on context.

4.2. IN category versus OUT of category

We find that the anatomy of hemodynamic activity linked to the N400 depends on conceptual relatedness. For related stimuli (IC), peak activity is seen in the posterior cingulate, precuneus, the bilateral temporal poles, as well as cerebellar regions. Whereas for unrelated stimuli (OC), peak activity is linked primarily to visual cortex and the left temporal pole. There are some similarities between IC and OC spatial maps: N400 responses are linked to activity in the temporal poles, although with some differences in magnitude and laterality. Activity in the temporal poles corresponds to a meta-analysis of fMRI studies of conceptual knowledge (Rice et al., 2015) and atrophy linked to impaired object recognition in progressive primary aphasia (Mesulam et al., 2013).

IC and OC spatial maps are most notable for their differences. For IC trials, pronounced peaks of activity are seen in midline cortical structures, including the cingulate cortex, posterior cingulate cortex (PCC) and the precuneus. The posterior cingulate and precuneus are highly metabolically active regions which are principle hubs for the default mode network (Whitfield-Gabrieli and Ford, 2012), but also a semantic network (Binder et al., 2009). We interpret activity within these areas as mediating the task demands of distinguishing between conceptually related objects, a process similar to semantic interference (de Zubicaray et al., 2001). Our findings are in apparent contrast with prior studies which found suppression of ACC or PCC activity for IC versus OC words (Matsumoto et al., 2005; Kircher et al., 2009; Abel et al., 2012). Notably, these studies employed naming or lexical decision tasks, rather than a matching task, as we did. This distinction may be crucial as there is evidence for differential loci of semantic interference for word-picture naming tasks relative to word-picture matching tasks (Harvey and Schnur, 2016). Furthermore, matching tasks may show more robust priming effects than lexical decision tasks (Renoult et al., 2012) perhaps owing to effects of prediction (Lau et al., 2013). Lastly, our finding linking activity in cerebellum to IC N400 responses was somewhat surprising. The role of the cerebellum in mediating semantic association has been probed with TMS, but effects were not specific to categorization (Argyropoulos and Muggleton, 2013). The specificity of cerebellar activity linked to the IC condition warrants further investigation.

With the possible exception of the temporal pole, regions linked to IC and OC N400 responses are somewhat at odds with our initial hypothesis of an N400 language network identified by (Lau et al., 2008). As noted above, this may reflect our use of a matching task, rather than a naming task. Moreover, our use of picture primes, rather than words, recruits distinct superior occipital networks (Vandenberghe et al., 1996). The OC spatial map reveals activity in visual occipital areas, in addition to the temporal pole. Our use of contrast estimates in the JICA suggests that activity in visual areas is related to the violation of semantic expectations, rather than purely reflecting the visual nature of the task. Visual cortex activation is consistent with semantic models of “embodied abstraction” in which perceptual networks are recruited in language tasks during novel or unexpected contexts (Binder and Desai, 2011). This is similar to models of predictive coding, where deviations in expectancy lead to increased responses from unimodal perceptual areas (Friston, 2005). Our results are consistent with other findings of flexible recruitment of visual areas within conceptual networks (Hoenig et al., 2008).

4.3. Diminished activation in schizophrenia linked to unusual thought content

In SZ we find that hemodynamic activity from the IC network (and less so, the OC network) is inversely correlated with unusual thought content (while controlling for disorganization). This finding is distinct from other studies which related larger N400 amplitude to delusions (Kiang et al., 2008) and reduced N400 amplitude to formal thought disorder (Kostova et al., 2005). Notably, these studies report associations with N400 amplitude, which we did not find. This may relate to differences in the priming task (matching, or lexical decision) and stimulus type (words versus pictures). The posterior cingulate / precuneus is hyperconnected in schizophrenia, with hyperconnectivity correlating with composite measures of psychopathology (Whitfield-Gabrieli et al., 2009). Hyperconnectivity in the PCC might undermine attentional resources during cognition, destabilizing the balance between internal (often semantic) and externally directed processing (Leech and Sharp, 2014). Deficient activity in the PCC during externally triggered semantic violations, as we observe, may reflect an inability in SZ to balance external attention to the semantic violation with internally oriented semantic memory. This imbalance might attribute excessive internal (self-referential or idiosyncratic) meaning to external objects, contributing to unusual thought content. Impairments in a distributed network, that includes visual and transmodal integrative centers, point toward semantic impairment in SZ at the level of semiosis: a sign process mediating internal interpretation of external objects (Wróbel, 1990). Not merely being a disorder of language per se, unusually broad semantic networks in SZ may fundamentally reflect disordered relations between external objects and inner representations, leading to unusual linguistic content.

4.4. Limitations and future directions

A primary limitation of this study was that ERP and fMRI data were acquired from separate recording sessions. Repetition of the same paradigm could have introduced practice effects into the N400 response, which is otherwise highly reliable (Kiang et al., 2013). We suspect that repetition had a minimal effect on our findings, given that there was no effect of session order or time between sessions on the loading weight of the joint component. Furthermore, we found that performance accuracy was similar between EEG and fMRI recording sessions, the first and second session, and was not associated with the time between sessions. Future studies might benefit from concurrent recordings to rule-out practice or repetition effects, but with caution given the recording and analytic challenges introduced by simultaneous EEG-fMRI methods (Abreu et al., 2018; Fellner et al., 2016).

An additional limitation of this study was that the JICA approach explicitly excludes spatial information from the ERP data and temporal information from the fMRI data. Instead, the JICA approach leverages the temporal precision of ERPs and the spatial precision of fMRI. There is pronounced similarity between IC and OC ERP waveforms and striking dissimilarity between IC and OC spatial maps. This finding raises the question of how similar voltage fluctuations are associated with distinct hemodynamic maps. We note that despite similar waveform morphology, N400 amplitudes do distinguish IN and OUT category stimuli (see Methods and Supplemental Fig. 2) as we reported previously (Mathalon et al., 2010). In addition, the N400 shows consistency in timing across a wide range of stimulus modalities and features (Kutas and Federmeier, 2011). Our findings suggest that the voltage time course of activation linked to these networks is remarkably similar, despite differences in spatial anatomy derived from a sluggish hemodynamic response. Although hemodynamic signals are typically considered as a surrogate for neural activity, they directly reflect vascular and metabolic activity that may be dissociated or partially uncoupled from neural activity (Ekstrom, 2010; Winder et al., 2017). Cardiovascular signals interact with ongoing neural activity to influence semantic representations (Babo-Rebelo et al., 2016), so we cannot exclude the possibility that hemodynamic activity reflects embodied signals relevant to semantic processing that are distinct from neural signals. Future studies might employ concurrent EEG-fMRI or EEG source localization techniques to compare anatomy linked to voltage fluctuations with anatomy linked to hemodynamic changes.

Appendix A. Supplementary data

Supplementary material