Differential patterns of visual sensory alteration underlying face emotion recognition impairment and motion perception deficits in schizophrenia and autism spectrum disorders

Abstract

Background:

Impaired face-emotion recognition (FER) and abnormal motion processing are core features in schizophrenia (SZ) and autism spectrum disorder (ASD) that have been linked to atypical activity within visual cortex. Despite overlaps, only a few studies have directly explored convergent versus divergent neural mechanisms of altered visual processing in ASD and SZ. We employed a multimodal imaging approach to evaluate FER and motion perception in relation to functioning of subcortical and cortical visual regions.

Methods:

Subjects were 20 high-functioning adults with ASD, 19 schizophrenia patients and 17 control participants. Behavioral measures of coherent motion-sensitivity and FER along with electrophysiological and functional MRI (fMRI) measures of visual pattern and motion processing were obtained. Resting-state fMRI was used to assess the relationship between cortico-cortical and thalamo-cortical connectivity and atypical visual processing.

Results:

SZ and ASD participants had intercorrelated deficits in FER and motion-sensitivity. In both groups, reduced motion-sensitivity was associated with reduced fMRI activation in occipitotemporal cortex and lower delta-band EEG power. In ASD, FER deficits correlated with hyperactivation of dorsal stream regions and increased evoked theta power. Activation of the pulvinar correlated with abnormal alpha-band modulation in SZ and ASD with under-and over-modulation, respectively, predicting increased clinical symptoms in both groups.

Conclusion:

SZ and ASD participants showed equivalent deficits in FER and motion-sensitivity but markedly different profiles of physiological dysfunction. The specific pattern of deficits observed in each group may help guide development of treatments designed to down-vs. up-regulate visual processing within the respective clinical groups.

Introduction

The ability to rapidly and accurately perceive and respond to facial expressions is an important component of social functioning. Impaired face-emotion recognition (FER) in SZ (1-3) and ASD (4-8) correlates significantly with symptoms (9) and functional outcome (10, 11). Visual motion perception facilitates the detection of facial and bodily cues important for socio-emotional communication (12). Motion discrimination is impaired in both SZ (reviewed in 13) and ASD (14-16). Despite their shared characteristics, the neural mechanisms underlying FER and motion deficits in SZ and ASD are poorly understood.

Processing complex visual stimuli depends upon coordinated functioning of cortical/subcortical components of the visual system. These include early, ventral, and dorsal regions (reviewed in 17, 18) as well as subcortical structures such as the pulvinar nucleus of the thalamus (19, 20). Together, these regions contribute to analysis of features of the visual environment and coordination of activity across visuo-cortical areas. Abnormal neural responses of these cortical regions and the pulvinar have been reported in SZ (21-24) and ASD (25-28) and may contribute to impairments in both FER and motion perception.

Here, we used electrophysiological and functional neuroimaging approaches to investigate convergent and divergent patterns of visual processing in SZ and ASD in relation to FER. Electrophysiological responses have high temporal but low spatial resolution and are well-suited to analysis of brain activity at the ensemble level (29). By contrast, neuroimaging methods such as task-based and resting-state fMRI (rsFMRI) have high spatial resolution and are particularly suitable for evaluating subcortical-cortical and cortico-cortical connectivity patterns.

Electrophysiological responses to stimulus-onset, motion-onset and oscillatory-entrainment were obtained and analyzed in the frequency domain to differentiate processing-specific patterns of neural activity. Activity associated with stimulus processing occurs primarily in the theta (4-7Hz) frequency band (30); whereas activity associated with processing stimulus motion occurs primarily at delta (1-4Hz) frequencies (31). In both cases, these responses are tightly phase-locked to stimulus- and motion-onset, respectively, thus leading to increases in evoked-power. The appearance of visual stimuli additionally induces a reduction of ongoing alpha (7-14Hz) activity ( “event-related desynchronization,” ERD) over occipital cortex, believed to occur when cortical regions are brought “on-line” for information processing (32, 33). The alpha ERD response is not phase-locked to the eliciting stimulus, therefore changes in oscillatory power are analyzed using single-trial analysis.

Functional MRI (fMRI) was used to localize disturbances within subcortical and cortical visual regions during motion processing, and resting-state fMRI to evaluate connectivity between these regions. Aberrant connectivity, including thalamo-cortical connectivity has been reported previously in both SZ (34, 35) and ASD (36, 37) but the association between these disrupted circuits and impaired visual processing has not been investigated.

Based upon prior studies, we hypothesized that SZ and ASD participants would have convergent deficits in FER and motion processing, consistent with convergent disturbances in social cognition. We also hypothesized that participants would show intercorrelated disturbances in electrophysiological and fMRI-based measures related to behavioral performance. The present study evaluates the degree to which neural mechanisms underlying these disturbances are convergent or divergent across disorders.

Methods:

Participants

Participants were 19 SZ, 20 ASD and 17 controls (CN), recruited from the central database at the Nathan Kline Institute. All were without significant medical/neurological disorders. SZ diagnoses (determined <1 year from participation) used the Structured Clinical Interview for DSM-IV (SCID) (38). Symptoms were evaluated using the Positive and Negative Syndrome Scale (PANSS) (39) (<3 months from participation). All SZ patients were on a stable dose of antipsychotic medication. ASD diagnoses were determined by DSM criteria and confirmed by the Autism Diagnostic Observation Schedule, Second Edition (ADOS) (<3 months from participation). Any participant meeting diagnostic criteria for both SZ and ASD or with IQ<70 was excluded. IQ (Ammons Quick Test, (40)) and education levels were lower in SZ patients compared to CN’s (Supplementary Table 1). All participants had minimum 20/22 corrected visual acuity on a Logarithmic Visual Acuity Chart. The investigation was approved by the NKI institutional review board. Informed consent was obtained after all procedures had been fully explained. The study was carried out in two experimental sessions (scheduled <4 weeks apart).

Behavioral measures

Coherent motion detection:

Coherent motion thresholds were determined using random-dot kinematograms (Supplementary Figure 1A). Coherence thresholds at 82% correct were determined by the QUEST (Quick Estimation) procedure (41) from 100 trials. Motion-sensitivity scores were defined as 1/coherence threshold. Data from one ASD participant were unavailable.

Face-emotion recognition:

Face-emotion recognition was evaluated using the Penn Emotion Recognition Task (ER-40) (42, 43). Scores were unavailable from two CN’s and one ASD participant.

Electrophysiology

Stimuli and task:

The present study utilizes the “JH-FLKR” paradigm described previously (31) (Supplementary Figure 1B). Stimuli were grayscale sinusoidal gratings of low spatial frequency (LSF, 0.8cpd) at high (75%) or low (8%) luminance contrast or high spatial frequency (HSF, 5cpd) at high luminance contrast. On each trial, stimuli appeared (400ms), drifted rightwards (200ms) then counterphase-reversed at 10Hz (3000ms), yielding a steady-state visual evoked potential (ssVEP). Participants responded by button-press to occasional dimming of a central fixation cross.

Recordings and data analysis:

The electroencephalogram (EEG) was recorded using a Waveguard cap (ANT, Enschede, Netherlands) containing 64 equally-spaced electrodes (44) with a sampling rate of 512Hz and re-referenced off-line to the average of all electrodes. Blink-related artifacts were removed by independent component analysis. Epochs with amplitudes exceeding ±100μV at any electrode were excluded. On average, 7.2% (CN), 12.8% (SZ) and 9.1% (ASD) of trials were excluded. Analyses were performed using MATLAB (Mathworks, Natick, MA) with the EEGLAB and ERPLAB toolboxes (45).

Evoked-power analyses:

Event-related activity was analyzed separately for each stimulus type. Evoked-power measures were obtained by convolving the time-domain averaged event-related potentials (time-locked to stimulus onset) with a 3-cycle Morlet wavelet over a 3s window, beginning 1s before onset, as described previously (31). Evoked-power was extracted at each time point over 74 frequency scales (0.48-27.6Hz), incremented logarithmically. Statistical analyses were carried out in theta (4-7Hz) and delta (1-4Hz) frequency bands, for stimulus-onset and motion-onset activity, respectively. Measurement latency windows were centered around the peak amplitude based on combined data from all participants, yielding a theta window of 150-250ms post-stimulus onset and a delta-window of 50-250ms post-motion onset.

ssVEP analyses:

ssVEP data were analyzed at the driving stimulation frequency (10Hz) using fast Fourier transform (FFT) and tested between 500-3000ms following the onset of counterphase stimulation.

Single-trial power analyses:

The single-trial EEG signal from each channel was convolved with a 3-cycle Morlet wavelet and total power was extracted as above. The alpha ERD was measured as the reduction in total power within the alpha, 7-14Hz, frequency, from the pre-(−150 to 0ms) to post-stimulus latency interval and tested between 250-400ms post-stimulus onset.

Functional MRI

Stimuli and task:

Motion-sensitive visual areas were identified using low-contrast concentric rings (15° diameter) extending throughout a circular region (31, 46). Participants monitored a central cross and responded via button-press to occasional dimmings. The rings expanded/contracted for 20-seconds followed by 20-seconds during which the stimuli were static (Supplementary Figure 1C).

Acquisition and data analysis:

Functional images were acquired on a Siemens 3T TIM-Trio scanner. Pre-processing included despiking, slice-time and motion correction. Anatomical surfaces were generated from high-resolution images using FreeSurfer (47), registered to the 141-fsaverage standard mesh. Post-processing and statistical analyses were carried out on gray-matter ordinates of the surface using a combination of AFNI (48) and SUMA (https://afni.nimh.nih.gov/Suma) software. Single-participant analyses of the contrast of moving versus stationary stimuli used general linear model procedures (GLM) incorporated in AFNI. The HCP-MMP1.0 cortical parcellation (18), registered to the Freesurfer standard mesh, was used to extract beta parameters from V1, Early visual, Dorsal, Ventral, and MT-complex (MTC) regions (18) of the left (LH) and right (RH) hemispheres. Beta values were also extracted from the pulvinar nucleus, (individually parcelled using FreeSurfer (49)), following parallel processing/GLM procedures of volumetric data. Group-wise analyses were carried out on mean beta values from these six regions. Significance levels were set to a (corrected) p<.01.

Resting-state functional connectivity:

Resting-state images (180 acquisitions) were acquired during one six-minute scan and pre-processed as above. Additional pre-processing on the surface was performed to remove physiological confounds and 24 motion parameters. Average signal time-courses were extracted from each cortical/subcortical region and pairwise-correlated.

Random Forest

The Random Forest (RF) machine learning method (50) was used to test the ability of electrophysiological and fMRI measures of brain function for discriminating 1) between ASD and SZ, and 2) between all three groups, at an individual level. In both analyses, the number of trees was set to 5000 and 3 variables were tried at each tree node split. The R-package randomForest (51) was used for analyses. Generalization error was based on classification accuracy of out-of-bag (OOB) samples.

Statistics

Between-group comparisons used one-way or repeated-measures ANOVA. For EEG analyses, factors were stimulus-type (low-contrast LSF, high-contrast LSF, high-contrast HSF) and (where appropriate) hemisphere (left, right). Tukey-HSD tests were used for post-hoc comparisons.

ANCOVA’s assessed the interrelationship between physiological (EEG/fMRI, covariates) and behavioral (FER/motion-sensitivity) measures with group-membership as a categorical predictor. The Group×Covariate interaction assessed homogeneity of slopes. ANCOVA’s were followed by within-group Pearson correlation or stepwise regression. Correlations were considered significant after Bonferroni-correcting p-values for the number of tests.

Results:

The effects of gender and age were analyzed by multivariate ANOVA using the means from all physiological/behavioral variables. The main effect of gender (F(1,47)=.207,p=.651), the Group×Gender (F(2,47)=1.72,p=.191) and the Variable×Gender (F(13,35)=1.82,p=.085) interaction were non-significant, thus, subsequent analyses collapsed across gender.

Across groups, there was a main effect of age (F(2,53)=3.96,p=.025), with no age differences between SZ (F(1,34)=.947,p=.337) nor ASD (F(1,35)=3.02,p=.091) participants compared to CN’s. When regressed against age, none of the physiological measures were significantly associated with age (p<.250, all). Unless noted, subsequent analyses did not include age as a factor.

Motion and face-emotion processing, behavioral measures:

FER (ER-40 scores) (F(2,48)=6.99,p=.002) and motion-sensitivity (F(2,50)=9.37,p<.001) differed significantly across groups even following control for age and IQ. This difference also remained significant after excluding participants (n=10) with motion-sensitivity scores <4 (F(2,39)=5.54,p=007).

Compared to CN’s, FER was significantly reduced in SZ (F(1,32)=23.61,p<001) and ASD (F(1,32)=11.20,p=002) participants, as was motion-sensitivity (SZ: F(1,34)=19.47,p<001; ASD: F(1,34)=9.07,p=005) (Supplementary Figure 2A).]

Motion-sensitivity predicted FER across groups (F(1,47)=9.47,p=.003) but the relationship differed significantly, as reflected in a Group×Motion-sensitivity interaction (F(2,47)=7.73,p=.001). Within-group, FER correlated with motion-sensitivity in CN (r=.542,p=.037) and ASD participants (r=.570,p=.011) as well as across both groups (r=.661,p<.001). In SZ the correlation was not significant (r=−.357,p=.133) (Supplementary Figure 2B).

Electrophysiological results

Behaviorally, correct target detections (F(2,53)= 1.80,p=.176) and false-alarm rates (F(2,53)=2.65,p=.080) did not differ significantly across all groups but were lower in SZ compared to HC (Supplementary Table 1).

Stimulus Onset Response (Theta):

Stimulus-onset elicited an increase in evoked theta power which differed significantly across groups (F(2,53)=10.27,p<.001) (Table 1). Relative to CN’s mean theta power was reduced in SZ (p=.009) but was increased in ASD participants (p=.038) (Figure 1B).

Table 1:

For each subject group, mean amplitude (in microvolts) and standard deviations (in parentheses) of all EEG measures are given for the left (LH) and right (RH) measures (ssVEP measures were taken from midline electrode sites). Main effects of Group (G.), Hemisphere (Hemi., H.), Stimulus (S.) and interactions with Group are given. Associated p values are shown on right for each measure (degrees of freedom in parentheses).

EEG Measure		CN	SZ	ASD	Group (2,53)	Hemi. (1,53)	Stimulus (2,52))	G. × H. (2,53)	G. × S. (4,104)
Theta		0.21 (.07)	0.16 (.04)	0.28 (.11)	10.27 p=.0002	n/a	36.39 p<.0001	n/a	3.28 p=.014
Delta	LH	0.22 (.09)	0.19 (.05)	0.16 (.06)	6.20 p=.004	3.23 p=.078	17.76 p<.0001	.754 p=.475	1.94 p=.108
	RH	0.26 (.09)	0.19 (.04)	0.20 (.08)
Alpha ERD	LH	−0.12 (.13)	−0.04 (.09)	−0.27 (.26)	8.91 p=.0005	4.33 p=.042	4.69 p=.013	3.52 p=.036	1.77 p=.140
	RH	−0.15 (.22)	−0.03 (.11)	−0.45 (.53)
ssVEP		0.13 (.09)	0.03 (.03)	0.24 (.16)	18.36 p<.0001	n/a	.291 p=.748	n/a	.200 p=.937

Figure 1.

A. Time-frequency and scalp topography maps of evoked-power to stimulus and motion onset. Time-frequency plots and scalp topography of mean evoked-power (average of all stimuli) for the control (CN), schizophrenia (SZ) and autism spectrum disorder (ASD) groups. For each stimulus-type, theta power (4-7Hz) was tested during the latency window 150-250ms (solid rectangle) following stimulus onset at time 0 and measured across 4 mid-occipital electrode sites from the specially developed ‘Duke’ system featuring equidistant spacing between electrodes (https://www.ant-neuro.com) (7Z, 8Z, 8L, 8R, green circles). Delta (1-4Hz) activity was tested between 50-250ms (dashed rectangle) interval following the onset of motion (at time 400ms), across 4 bilateral lateral-occipital sites (8L/8R, 5LB/5RB, 9L/9R, 5LC/5RC, red circles).

B. Theta evoked-power, group differences. Bar plots of mean theta power (collapsed across stimulus type) for each participant group. Theta activity was significantly lower in SZ patients compared to controls. In contrast, theta was significantly elevated in ASD participants. For this and all figures, asterisks denote statistical significance as follows: p<.001 (***),p<.01 (**),p<.05 (*).

C. Correlation of ER-40 scores and theta evoked-power. In ASD participants, enhanced theta activity significantly correlated with impaired face-emotion recognition (FER). This correlation was not significant either in SZ (r=−.36,p=.135) or CN groups (r=−.32,p=.240).

D. Delta evoked-power, group differences. Bar plots of mean delta power (collapsed across stimulus type) for each participant group. In contrast to theta, delta power was significantly lower in both SZ and ASD groups, compared to controls.

E. Correlation between motion-sensitivity and delta evoked-power. In both clinical groups, reduced delta power correlated with behavioral measures of impaired motion-sensitivity. This correlation was not significant in CN’s (r=.38,p=.131).

Across groups, mean theta power significantly predicted FER (F(1,47)=7.38,p=.009). The Group×Theta interaction (F(2,47)=7.97,p=.001) was also significant. In follow-up correlations, enhanced theta activity in ASD participants correlated with reduced FER (r=−.722,p<.001) (Figure 1C). This correlation was not significant within the SZ or CN groups (p>.500). Finally, theta power was not a significant predictor of motion-sensitivity (F(1,49)=1.86,p=.179).

Motion Onset Response (Delta):

Motion-onset elicited an increase in delta (1-4Hz) evoked-power which differed across groups (F(2,53)=6.20,p=.004) with significant reductions in both SZ (p=.020) and ASD participants (p=.007) compared to CN’s (Figure 1D).

Mean delta power did not predict FER scores (F(1,47)=.333,p=.566) but did predict motion-sensitivity (F(1,49)=5.96,p=.018) across groups, with a significant difference in the slopes of the effect (Group×Delta: F(2,49)=4.51,p=.007). Reduced delta power in both SZ (r=.461,p=.047) and ASD (r=.598,p=.007) participants, correlated with lower motion-sensitivity (Figure 1E).

Alpha measures:

ERD: Alpha ERD amplitude differed significantly across groups (F(2,53)=8.91,p=.0005) (Figure 2A,B, Table 1) and was reduced in SZ patients compared to both CN (p=.017) and ASD participants (p<.001) (Figure 2B). In ASD participants, ERD amplitude was larger (p=.039) and temporally prolonged (p=.021) relative to CN’s. Across groups, alpha ERD did not predict FER (F(1,47)=.002,p=.962) nor motion-sensitivity (F(1,49)=.196,p=.747).

Figure 2.

A. Single-trial time-frequency histograms and scalp topography maps of event-related desynchronization (ERD) of alpha activity. Time-frequency plots and scalp topography of mean alpha ERD single-trial power for each participant group. Alpha ERD was measured between 7-14Hz over the latency window of 250-400ms post-stimulus onset, (white rectangles) across 4 bilateral lateral-occipital electrodes (8L/8R, 5LB/5RB, 9L/9R, 5LC/5RC, red circles).

B. Time-course and amplitude of alpha ERD. In the test interval of 250-400ms (dashed rectangle) ERD amplitude was significantly reduced in SZ compared to CN participants. In contrast, the ERD was enhanced in ASD compared to both SZ and CN groups. ERD enhancement persisted in ASD participants relative to CN and SZ groups.

C. Counterphase reversals at 10Hz. Tracings are of group-averaged ssVEP power collapsed across stimulus types and plotted by spectral frequency. Relative to CN’s, ssVEP amplitude was reduced in SZ and enhanced in ASD participants. Asterisks denote significance of the difference between SZ patients and ASD participants compared to CN’s. The ssVEP was tested across 3 mid-occipital electrodes (6Z, 7Z, 8Z).

ssVEP: ssVEP power at 10Hz differed significantly across groups (F(2,53)=18.36,p<.0001), with a reduction in SZ (p=.028) but an increase in ASD (p=.007) relative to CN’s (Figure 2C, Table 1). Across groups, ssVEP power predicted FER (F(1,47)=4.66,p=.016) with no significant difference in the slopes (F(2,47)=1.83,p=.172). Motion-sensitivity was not predicted by ssVEP (F(1,49)=.77,p=.782).

Functional MRI

During scanning, target detection (F(2,53)=.08,p=.924) and false-alarm rates (F(2,53)=.49,p=.614) were equivalent across groups (Supplementary Table 1).

Across hemispheres and cortical/subcortical regions, there were no significant effects of group membership (F(2,53)=2.82,p=.068). However, there was a significant main effect of region (F(5,49)=43.62,p<.001) and a Group×Region interaction (F(10,98)=2.86,p<.004) (Figure 3A). The main effect of hemisphere was significant (F(1,53)=6.03,p=.017) but did not interact with group (F(2,53)=1.21,p=.303) or region (F(5,49)=1.66,p=.163), therefore within-region analyses were collapsed across hemispheres.

Figure 3.

A. Group-averaged fMRI activation for the contrast of moving versus static stimuli. For each participant group, mean activation is shown on the semi-inflated fsaverage brain. Colored outlines are boundaries of the five cortical regions included in analyses (V1: dark blue; E.V. (Early Visual): blue; Dorsal: purple; Ventral: violet, MTC (MT Complex): green). Each region consists of between 1 and 9 individual parcels (total of 26; see (18) supplement and Supplementary Table 1). Inset shows all parcels (drawn in black) of the HCP-MMP atlas, including the parcels comprising the five regions (colored) from which data was extracted and averaged. Single- participant statistical analyses followed GLM procedures incorporated in AFNI (afni_proc.py). Group differences in activation were evaluated by ANOVA using AFNI’s 3dMVM program. Corrections for multiple comparisons were carried out at the cluster level using Monte Carlo simulation (AFNI’s slow_surf_clustsim.py,p< 0.01, corrected). Arrows (on CN maps) point to the MTC region where activation was significantly higher in CN’s compared to both SZ patients and ASD participants. Green arrows (on ASD maps) point to the dorsal region which showed enhanced activation in ASD participants.

B. Mean beta contrast parameter estimates within each region. Asterisks denote regions where activations in SZ or ASD participants differed significantly from that of CN subjects. Activation of the MTC region was reduced in both SZ and ASD participants. These reductions were localized within the MST, MT, LO1 and LO3 parcels of the HCP-MMP atlas. In ASD participants, activation within both early visual and dorsal stream regions was significantly greater compared to both SZ patients and CN participants. The enhanced early visual activations were located within the V2 and V3 parcels and the dorsal stream region with increased activation localized to the V3A parcel.

C. Mean activation within the pulvinar nucleus of the thalamus. Subcortically, pulvinar activation was reduced in SZ patients relative to CN’s (dashed rectangle on sagittal slice indicates magnified region shown to the right).

D. Correlations with EEG variables. In all participant groups, greater activation within the MTC regions was significantly associated with increased delta evoked-power elicited by the motion onset of all stimuli (left). In ASD participants, enhanced activation of the dorsal region correlated with participants’ abnormally high theta power following stimulus onset (right).

E. Correlation of pulvinar activation with alpha ERD. In both SZ patients and ASD participants, the magnitude of pulvinar activation correlated significantly with ERD amplitude. Note vertical scale is reversed to show that such that larger (more negative) alpha ERD is associated with greater pulvinar activation.

In the MTC region, activation was reduced in SZ (F(1,34)=4.84,p=.034) and ASD (F(1,35)=4.00,p=.043) participants compared to CN’s (Figure 3B, Supplementary Table 2). ASD participants, however, had significantly greater activation in the early-visual and dorsal regions compared to SZ patients (early visual: F(1,37)=4.63,p=.039; dorsal: (1,37)=4.25,p=.047) and CN’s (early visual: F(1,35)=5.79,p=.022; dorsal: (1,35)=4.23,p=.048). Activation of V1 was equivalent in ASD and CN (F(1,35)=.19,p=.663) groups, but was reduced in SZ relative to both CN (F(1,34)=6.68,p=.014) and ASD participants (F(1,37)=7.19,p=.011). There were no significant group differences within the ventral region.

Subcortically, activation of the pulvinar nucleus was significantly reduced in SZ (F(1,34)=6.29,p=.017), but not ASD participants (F(1,35)=.13,p=.715) compared to CN’s (Figure 3C,D).

Correlations with EEG:

Based on our previous study (31), activation within dorsal, MTC and pulvinar regions were correlated with EEG measures (theta, delta, ERD, ssVEP). In ASD participants, enhanced theta-band activity, correlated with increased dorsal activation (r=.61,p=.004) (Figure 3E). In parallel, delta activity to stimulus motion, correlated with activation within the MTC (CN: r=.66,p=.004; SZ: r=.48,p=.038; ASD: r=.52,p=.019). Finally, reduction in alpha ERD in SZ (r=−.48,p=.038) as well as its increase in ASD (r=−.61,p=.004), correlated with the magnitude of pulvinar activation (Figure 3F). Exploratory correlations (n=9) between EEG measures and activation within V1, early visual and ventral regions were non-significant within or across subject groups (all p>.250).

Resting State Functional Connectivity:

Mean resting-state functional connectivity (rsFC) across all regions differed significantly across groups (F(2,53)=11.81,p<.001) (Figure 4A). The Group×Region interaction was also significant (F(10,98)=2.48,p=.011).

Figure 4.

A. Resting state functional connectivity. Bar graphs are of mean connectivity between each region and the average of all others. Compared to CN participants, mean rsFC was reduced in SZ patients in the ventral and MTC regions. Compared to ASD participants, SZ patients had reduced rsFC in the V1, dorsal and ventral regions. In contrast, mean rsFC of all cortical regions was equivalent in ASD participants compared to CN’s. Mean rsFC of the pulvinar, however, was significantly greater in ASD subjects relative to both CN’s and SZ patients.

B. Group differences in pairwise rsFC. For each participant, rsFC between regions was calculated in a pairwise fashion and entered into between-group (two-tailed) t-tests. Resulting T values are plotted in heatmaps comparing CN’s to SZ patients (left) and CN’s to ASD participants (right). Blue scale denotes greater rsFC in CN’s versus SZ and lower rsFC in CN compared to ASD participants (red scale, thus, denotes greater rsFC in ASD relative to CN’s).

Compared to CN’s, mean rsFC of V1 (F(1,34)=7.27,p=.011), early visual (F(1,34)=8.26,p=.007), dorsal (F(1,34)=7.86,p=.008), ventral (F(1,34)=23.90,p<.001) and MTC (F(1,34)=7.69,p=.009) regions (to all other regions) was reduced in SZ patients, whereas, mean rsFC of pulvinar to cortex (F(1,34)=.64,p=.802) was equivalent.

In ASD, mean cortical rsFC was similar to that of CN’s (p>.6 for all). Subcortically, however, rsFC between pulvinar and the mean of all cortical regions was significantly greater in ASD compared to CN participants (F(1,35)=6.21,p=.017) and SZ patients (F(1,37)=8.03,p=.007).

In pairwise comparisons (Figure 4B), reduced rsFC in SZ was observed primarily in ventral and MTC regions. By contrast, in ASD, increased rsFC was observed especially between pulvinar and the dorsal region.

Finally, the relationship between pulvinar rsFC and EEG alpha measures was tested in pre-planned correlations (31). In ASD, but not CN (r=−.06,p=.832) or SZ (r=−.38,p=.113) participants, the amplitude of alpha ERD correlated with mean connectivity between pulvinar and cortex (r=−.53,p=.015) (Figure 4C).

Classification analyses:

The discriminative power of physiological variables (4 EEG, 6 fMRI, 2 rsFC) on classification of ASD and SZ participants was assessed using RF. The out-of-bag classification accuracy was 97.4%, with correct classification of all SZ patients and 19/20 ASD participants. Four variables (theta, ssVEP, ERD and mean cortical rsFC) were significantly (p<.01) important for classification (Supplementary Table 3). When all three groups were entered into the RF analysis, classification accuracy was 82.1%, however, SZ and ASD participants were classified with 100% and 94.12%, accuracy respectively.

To compare the discriminative efficacy of multimodal versus unimodal variables, RF was conducted with only EEG or fMRI variables. Using EEG measures alone, classification accuracy between SZ and ASD dropped to 74.4%. With only fMRI variables, accuracy was 69.2%.

Correlations with behavior and symptoms :

Step-wise regression was used to evaluate the relationship between all measures of visual processing, together, and FER. Across groups, the variables significantly predicted FER (F(1,51)=12.12,p=.001). Given the observed heterogeneity of slopes between key physiological measures and FER, within-group analyses covarying against FER were conducted.

In CN’s, there was a strong association between sensory-processing measures and FER (Adj R²=.653) with both motion-sensitivity (r_p =.78, p<.001) and mean cortical rsFC (r_p=.64, p=.002) as significant independent predictors (F(2,12)=14.19,p<.001).

In ASD participants, the combined measures accounted for ~64% of the variance in FER (Adj R²=.635). Significant predictors of FER impairment were theta activity (r_p=−.59, p<.001) and motion-sensitivity (r_p =.39, p=.014) (F(2,16)=16.64,p<.001).

In SZ patients, ~45% of the variance in FER (Adj R²=.445) was accounted for by the combined measures. Alpha activity (r_p=.54, p=.008), activation of V1 (r_p=−.53, p=.008) and mean rsFC of pulvinar (r_p=−.39, p=.043) were significant independent predictors (F(3,15)=5.80,p=.007).

Exploratory correlations (n=18) evaluated the relationship between physiological variables and clinical symptoms in SZ and ASD. In both groups, abnormal alpha ERD was associated with greater clinical symptoms. In SZ, alpha ERD correlated inversely with PANSS (negative) scale (r=.51,p=.027) and in ASD ERD amplitude correlated directly with the ADOS social interaction rating (r=−.62,p=.003) (Supplementary Figure 3). Similarly, ADOS correlated with ssVEP power (r=.45,p=.048).

No significant relationship was observed between medication dose (chlorpromazine equivalents) and any behavioral/physiological measure in SZ patients (all p >.25).

Discussion:

The ability to recognize intended emotions based upon facial expression is a critical component of human social interaction and is impaired in both SZ and ASD. In SZ, we have previously observed that deficits in early visual processing contribute to impaired FER (31). In the present study, ASD and SZ groups showed equivalent FER impairments, however the underlying patterns of visual processing dysfunction contributing to these deficits differed substantially, permitting 97% discrimination between groups. In SZ, deficits were attributable to reduced sensory activation, as reported previously (31, 52). By contrast, in ASD, increased sensory-driven responses predicted impaired FER. In addition to re-affirming the importance of visual sensory deficits to higher order cognition in SZ and ASD, these findings highlight the ability of convergent EEG and fMRI-based measures to distinguish between disorders.

Electrophysiology:

Electrophysiological measures included theta-frequency responses to stimulus onset, delta-frequency responses to motion onset, and alpha ERD/ssVEP. Consistent with prior findings by our group (31, 53) and others (54, 55), significant reductions across all measures were observed in SZ. In ASD, delta responses were similarly reduced and correlated with impaired motion-sensitivity. However, theta and alpha responses were markedly enhanced, consistent with previous findings (56, 57). The alpha ERD, in particular, was elevated and prolonged and may represent a physiological substrate for the clinical observation that ASD individuals struggle to disengage from visual stimuli (58, 59). Finally, both the reductions in alpha amplitude in SZ and the excesses in ASD, correlated with symptom severity, suggesting that either hypo- or hyperengagement could undermine face-emotion recognition and, by extension, social function.

Neuroimaging:

In addition to EEG, fMRI measures were obtained during resting-state and visual stimulation. As with electrophysiology, convergent and divergent patterns of dysfunction were observed across groups. Specifically, both clinical groups showed convergent deficits in MTC activation that correlated with the convergent deficit in delta evoked-power. By contrast, divergent patterns in other visual regions were observed, such that SZ patients showed a decrease in activation in V1, whereas ASD participants showed increased activation across early-visual and dorsal regions, which correlated selectively to the increased theta activity observed electrophysiologically.

Differential patterns were also observed in rsFMRI analyses, with SZ patients showing reduced connectivity between visual regions and ASD participants showing normal cortical connectivity but markedly enhanced connectivity between pulvinar and visual cortex, which, in turn, significantly predicted enhanced alpha ERD. Increased pulvinar-visual cortex connectivity has been reported in ASD (36) however, to our knowledge, pulvinar activation or pulvinar-cortical connectivity relative to alpha modulation has not been previously investigated.

Implications for pathophysiology:

Deficits in visual sensory function in SZ and ASD are proposed to reflect altered excitation/inhibition (E/I) balance (reviewed in 60, 61). In SZ, excess developmental pruning (62) may lead to a hypoglutamatergic state particularly involving impaired N-methyl-D-aspartate (NMDA) function (63). Visual deficits, which preferentially involve magnocellular-system dysfunction (64), are consistent with patterns induced by NMDA receptor antagonists such as ketamine (65). In ASD, both “hypopruning” (66) of glutamatergic systems and underfunction of gamma-Aminobutyric acid (GABA)-mediating signaling (67, 68) have been proposed and may interrelate with deficits in visual perceptual processes (69). The patterns observed here of reduced power and activation across all measures in SZ, are consistent with reduced excitation throughout the visual system. Such deficits would lead to impaired sensory processing and are consistent with prevailing NMDA receptor hypofunction models (reviewed in 29, 70) and concepts of underdevelopment (71) and hyperpruning of cortical glutamatergic terminals (62).

By contrast, patterns in ASD were of both hypo-and hyper-responsivity in EEG and fMRI measures, as well as increased pulvinar-cortex connectivity and increased/prolonged stimulus-related alpha suppression. This pattern is consistent with either disrupted cortical inhibition (72, 73) or cortical overgrowth/reduced synaptic pruning (66, 71). We have recently observed that the amplitude of the visual theta response declines progressively during childhood/late adolescence (74), mirroring the pruning-induced synapse reduction in visual cortex during that time period (75). Thus, in ASD the present findings are compatible with underpruning and persistence of early childhood patterns into adolescence and adulthood whereas findings of reduced theta amplitude in SZ (below those of age-matched controls) are consistent with hyperpruning (31, 74, 76).

The present study also highlights the potential involvement of the pulvinar nucleus in sensory-level impairments in ASD. The pulvinar is thought to play a critical role in cortical integration of visual information via modulation of alpha rhythms (77). Abnormalities of the pulvinar have been observed previously in ASD (28, 78), but have not been a primary research focus. The present findings of enhanced and prolonged ERD, along with increased alpha ssVEP suggest the need for increased use of single-trial, rather than average, EEG approaches for evaluating neurophysiological deficits in both SZ and ASD, and for greater focus on the investigation of potential pulvinar pathology.

Implications for diagnosis and management:

Across neuropsychiatric disorders, similar deficits at the behavioral level may result from differential underlying pathological mechanisms. To date, physiological measures are incorporated into diagnostic/management algorithms to only a limited degree. In the present study, leveraging of neurophysiological and neuroimaging-based measures through a machine-learning approach obtained close to 100% separation between SZ and ASD participants, encouraging further use of sensory-level multimodal approaches for patient classification. Sensory-level measures are well-suited for diagnostic use given they are highly consistent across control individuals and require only limited participant engagement to collect, in some cases, using routine clinical equipment. If the present results are confirmed in larger samples, measures of this type might be useful in guiding treatment, cognitive remediation, or non-invasive brain stimulation-type interventions.

Limitations:

Although differences between groups were statistically robust, they require replication in larger samples. In particular, both SZ and ASD are characterized by substantial heterogeneity that is difficult to parse in samples of this size. Further, the ASD group was limited to adults with IQ>70 and future studies should evaluate neurophysiological measures across the full spectrum of ASD and across the developmental lifespan.

While deficits did not correlate with medication dose, SZ patients were receiving antipsychotics, whereas ASD participants were medication-free; thus, a medication effect cannot be excluded. Finally, eye-tracking was not obtained in this study and deficits in SZ could be related to lack of sustained fixation. Given the interleaved nature of the task, however, poor fixation cannot explain the discrepancy between increased stimulus-onset but reduced motion-onset responses in ASD.

Lastly, while the Random Forest analysis uses a cross-validation method the sample size did not permit the use of separate training and test datasets.

Conclusions:

Both SZ and ASD are increasingly associated with perturbed sensory function that contributes to impaired social functioning but may stem from differential underlying pathophysiological mechanisms. Using a multimodal imaging approach, we demonstrate a primarily hypoactive state in SZ and a mixed pattern of hyper-and hypoactivation in ASD. These findings highlight the importance of physiologically-based measures in guiding etiological and interventional studies in neuropsychiatry.

KEY RESOURCES TABLE

Resource Type	Specific Reagent or Resource	Source or Reference	Identifiers	Additional Information
Add additional rows as needed for each resource type	Include species and sex when applicable.	Include name of manufacturer, company, repository, individual, or research lab. Include PMID or DOI for references; use “this paper” if new.	Include catalog numbers, stock numbers, database IDs or accession numbers, and/or RRIDs. RRIDs are highly encouraged; search for RRIDs at https://scicrunch.org/resources.	Include any additional information or notes if necessary.
Antibody	N/A
Bacterial or Viral Strain	N/A
Biological Sample	N/A
Cell Line	N/A
Chemical Compound or Drug	N/A
Commercial Assay Or Kit	N/A
Deposited Data; Public Database	N/A
Genetic Reagent	N/A
Organism/Strain	N/A
Peptide, Recombinant Protein	N/A
Recombinant DNA	N/A
Sequence-Based Reagent	N/A
Software; Algorithm	N/A
Transfected Construct	N/A
Other	N/A