Multisensory Temporal Processing in Early Deaf

Abstract

Navigating the world relies on understanding progressive sequences of multisensory events across time. Early deaf (ED) individuals are more precise in visual detection of space and motion than their normal hearing (NH) counterparts. However, whether ED individuals show altered multisensory temporal processing abilities is less clear. According to the connectome model, brain development depends on experience, and therefore the lack of audition may affect how the brain responds to remaining senses and how they are functionally connected. We used a temporal order judgment (TOJ) task to examine multisensory (visuotactile) temporal processing in ED and NH groups. We quantified BOLD responses and functional connectivity (FC) in both groups. ED and NH groups performed similarly for the visuotactile TOJ task. Bilateral posterior superior temporal sulcus (pSTS) BOLD responses during the TOJ task were significantly larger in the ED group than in NH. Using anatomically defined pSTS seeds, our FC analysis revealed stronger somatomotor and weaker visual regional connections in the ED group than in NH during the TOJ task. These results suggest that a lack of auditory input might alter the balance of tactile and visual area FC with pSTS when a multisensory temporal task is involved.

1. Introduction

Plasticity endows neural circuits with the capacity to reorganize and receive input from spared modalities when sensory deprivation occurs (Milshtein-Parush et al., 2017). The fundamental role of plasticity for the reorganization of the deprived auditory cortex (and other structures) in early deaf (ED) individuals presents the opportunity to learn more about altered abilities in a specific population (see Pavani & Bottari, 2012 for a review). The sensory compensation hypothesis outlines that auditory deprivation causes the remaining visual and tactile senses to become more physiologically sensitive (Heimler et al., 2014; Renier et al., 2014). Furthermore, the functional constancy hypothesis posits that some cortical structures maintain their computational specialization, but switch to processing of a different input modality (Bock & Fine, 2014). In reorganizing, the cortex also avoids the physiological implications of sensory deafferentation (such as paroxysmal activity or cortical atrophy). Singh et al. (2018) discussed an inverse link between compensatory neural reorganization and the age of hearing loss onset (i.e., the earlier in life one becomes deaf, the greater the potential for compensatory neural reorganization). Hence, research has suggested that early deafness carries greater potential for neural reorganization than losing audition later in life. Opposingly, the perceptual deficit hypothesis suggests that sensory deprivation weakens the remaining senses (Pavani & Bottari, 2012). If deafness fundamentally alters auditory regions and their interactions with primary visual regions (Shiell et al., 2014), neural reorganization would negatively impact visual processing.

Previous research on ED individuals has focused primarily on unisensory visual-spatial processing. Some behavioral studies have reported visual advantages in ED individuals for stimuli presented in the visual periphery (for reviews see Alencar et al., 2019; Pavani & Bottari, 2012). For example, ED are superior to normal hearing (NH) individuals at detecting directional movement (Hauthal et al., 2013). Further investigation of motion processing showed that ED signers displayed greater sensitivity in the inferior than the superior, visual field. Learning sign language has induced redistribution of attentional resources toward the inferior peripheral visual field (Stoll & Dye, 2019) and enlargement of the inferior peripheral visual field (Buckley et al., 2010). These changes reflect functional adaptation in ED signers: gaze is maintained on the face of their interlocutor, while peripherally monitoring hand movements. Since ED individuals rely more heavily on vision than NH, the prevalence of reported visual behavioral differences reflects the role of plasticity for compensation. While NH controls show a left-visual-field dominance for motion detection, ED individuals show opposite right-visual-field dominance (Strong et al., 2019; Bosworth & Dobkins, 2002). This reorganization is believed to occur for greater dependence on visual language processing in the contralateral left hemisphere of the brain (Bosworth & Dobkins, 1999). This is known as the language capture hypothesis, rooted in the fact that motion detection is important for ASL comprehension (Mitchell et al., 2013).

While visual processing has been a primary focus of ED research, unisensory tactile spatial processing is less studied in ED individuals, with fewer conclusive findings. Superior haptic orientation processing has been reported in ED (van Dijk et al., 2013), but touch is rarely prioritized over vision, audition, or olfaction. ED individuals displayed enhanced tactile performance on a spatial length discrimination task (Papagno et al., 2016), outperforming NH controls. The findings of this tactile spatial task revealed a difference in perceptual sensitivity between ED and NH groups, rather than a difference in post-perceptual mechanisms. In contrast, ED individuals displayed impaired tactile acuity on an orientation task using gratings (grooves and ridges) cut into plastic objects (Pellegrino et al., 2020). This paradigm tested tactile spatial resolution of pressure mechanoreceptors, which seemed to be dampened with deafness. This study did not test vibrotactile response that depends on rapidly adapting mechanoreceptors (RA). RA sensitivity was also found to be dramatically reduced in ED than in NH adolescents (Moshourab et al., 2017). Studies reporting enhanced tactile sensitivities linked them to sign language comprehension and relocation of 3D objects based on their spatial characteristics (van Dijk et al., 2013). The sensory apparatus and post-perceptual mechanisms for spatial tasks are distinctly different between visual and tactile modalities, making direct comparisons of the effects of auditory deprivation challenging (Heimler et al., 2017). In addition, after hearing loss, visual perception is perhaps more practiced and relied upon than tactile for unisensory spatial processing.

Multisensory spatial processing provides location and body boundary awareness, and each sense provides relevant information that allows human beings to socially and physically interact with the world. Weaker visuotactile (VT) interaction was found in ED individuals, compared to NH, on a speeded detection task (Hauthal et al., 2015). While both groups showed a redundant signals effect on reaction times, attributed to coactivated VT processing, the redundancy gain was less in the ED group. Heimler et al. (2017) reported enhanced visual influence for ED vs NH on a spatial multisensory (VT) conflict task (to focus attention on vibrotactile targets). The ED group was worse at ignoring visual distractors than NH. However, the increased visual interference in the ED group was more evident for ipsilateral peripheral distractors (same side as the tactile target) than contralateral ones, suggesting that visual stimuli were attended to with greater resource in the ED group when visual and tactile modalities were competing for spatial attention.

Orientation in the world and human communication rely on interpreting progressive sequences of events/utterances across time. According to the auditory scaffolding hypothesis, the rhythmic patterns in speech and music require refined abilities to rapidly analyze temporal information; deaf individuals do not develop such abilities (Conway et al., 2009; Conway et al., 2011). In keeping with this hypothesis, studies in NH have shown that audition dominates temporal perception over the other senses (Burr et al., 2009; Jones et al., 2009; Repp & Penel, 2002). For example, Jones et al. (2009) examined the saliency of each modality for interval timing and reported the lowest thresholds for auditory stimuli, followed by vibrotactile, and finally, visual. Kowalska & Szelag (2006) found impaired performance for visual duration judgment in ED individuals than in NH, providing support for the auditory scaffolding hypothesis. Their results of both reproduction (production of a time interval to match the one previously presented) and production (producing a time interval from memory) tasks indicated that congenital deafness impeded accuracy, with overestimations on shorter (1 s) and underestimations on longer (>3 s) duration judgments. Impaired performance in ED was also revealed using a simultaneity judgment task, where ED individuals showed larger visual thresholds than NH regardless of stimulus location (central/peripheral) (Heming & Brown, 2005). In contrast, ED individuals displayed an enhanced ability over NH to synchronize with temporally discrete visual stimuli (flashes and bouncing balls) for sensorimotor synchronization, suggesting that ED individuals have access to metrical timing abilities previously thought only accessible via auditory processing (Iversen et al., 2015). An earlier study that used a visual temporal order judgment (TOJ) task reported no change in temporal sensitivity in ED signers (Poizner & Tallal, 1987). Nava et al. (2008) reported similar visual TOJ task results, with the only significant difference between ED and NH groups being faster discrimination responses in ED, particularly when the first stimulus was presented in the periphery. Findings of impaired, enhanced, and comparable performance in ED individuals for visual temporal tasks warrant further investigation.

Tactile temporal processing was reportedly impaired in congenitally deaf individuals, with worse performance than in NH for discriminating the longer of two subtly different (25 ms vs. 15 ms) vibrations to the finger (Bolognini et al., 2012). A link between lack of speech experience and impaired temporal processing ability was given as a possible explanation. Heming & Brown (2005) also tested tactile simultaneity judgment, and found larger tactile thresholds in ED individuals than in NH, regardless of unimanual/bimanual stimulus location. Bimanual presentation was utilized to require an interhemispheric transfer of information along with expected higher temporal thresholds when timing onset judgments between hands were made. Although ED individuals displayed worse performance overall, they did not show the same increase in the threshold for bimanual trials as NH. The authors’ interpretation was that ED individuals utilized interhemispheric interaction for unimanual judgments, with more regions recruited to compensate for higher task demand than with NH (though no imaging was used to verify this).

The speed and efficiency of neural transmission differ across modalities, contributing to relative differences in temporal saliency (Nobre & van Ede, 2018; King, 2005). Processing of a stimulus from a particular modality might be vulnerable to subsequent cross-modal influence. Potential cross-modal neuroplasticity in ED individuals was investigated by Karns et al. (2012) using a double-flash VT illusion (where a single flash presented with two sequential air puffs to the face is perceived as two flashes). ED individuals, but not NH, displayed a susceptibility to the illusion that only occurs when visual and tactile stimuli are delivered in close temporal proximity. The heightened susceptibility suggested reduced sensitivity to the temporal relationship between modalities, and a larger influence of tactile stimulation on visual perception. Using a VT TOJ task, a recent study in our lab did not reveal a difference in temporal precision between ED and NH groups (Scurry et al., 2020a). Both TOJ and double-flash VT illusion tasks involve manipulation of temporal asynchrony, however, the induction of the illusion is an implicit perceptual phenomenon, whereas the TOJ task is more explicit and may involve post-perceptual processing (Hendrich et al., 2012).

Overall, inconclusive behavioral reports in ED illuminate the influence of task and domain (temporal vs. spatial). Altered behavioral functions observed in ED individuals are usually accompanied by neural reorganization. Activation of the auditory cortex (AC) has been found for cross-modal processing of visuospatial moving patterns in the periphery (Finney et al., 2001; Bavelier et al., 2006). Visual activation patterns of the AC were similar to NH individuals performing a matched rhythm task in the auditory modality (Bola et al., 2017). Furthermore, the location of a visual stimulus can be decoded from the patterns of neural activity in AC of congenitally deaf, but not NH individuals (Almeida et al., 2015). Several neuroimaging studies report larger patterns of activation for somatosensory and visuotactile processing in the AC of ED individuals compared to NH (Karns et al., 2012; Levänen et al., 1998; Auer et al., 2007). For example, this occurred in ED individuals during the VT double-flash illusion (Karns et al., 2012). In this case, the ED group’s susceptibility to the illusion was attributed to the cross-modal reorganization of the AC, and its altered processing of vision and touch. These results might further suggest more reorganization in the AC for touch than for vision.

The AC clearly shows reorganization to process both visual and tactile stimulation and is not the only structure that does so. By receiving converging auditory, visual, and tactile inputs, the posterior superior temporal sulcus (pSTS) plays a key role in integrating unisensory and multisensory input (Beauchamp et al., 2008). Enhanced peripheral visual responses were reported in the pSTS of congenitally deaf individuals (Scott et al., 2014). The right pSTS processes sensory input in ED individuals with increased activation of neural populations than for NH, e.g. for dot motion (Bavelier et al., 2001). Specifically, for direction-selective responses to translational visual motion stimuli, the extent of activation in right pSTS was over five times larger in the ED group than in NH (Retter et al., 2019). In contrast to right pSTS, activation of left pSTS is often associated with language-related visual tasks. Twomey et al. (2017) investigated the laterality of pSTS activations during different visual tasks in ED individuals, using object pictures with sign language embedded in them. Based on their British sign language (BSL) phonological content, detected matches between pairs of objects recruited left posterior superior temporal gyrus (pSTG) and pSTS. Furthermore, right pSTS was recruited consistently for low-level visual processing in semantic, BSL phonological, and simple (same or different object) visual judgment tasks (Twomey et al., 2017). These results from deaf, and lack thereof in hearing signers, suggested that the cortical function of bilateral pSTS regions is differentially altered for visual processing in deafness.

Reorganization of the pSTS to process tactile stimulation is also evident in ED individuals, with an increased extent of regional activation than in NH (Auer et al., 2007). Right pSTS showed increased recruitment for tactile motion in ED individuals (Scurry, Huber, et al., 2020). Overlap of visual and somatosensory responses in right pSTS of ED individuals suggested altered neural architecture for multisensory or high-order multimodal function, e.g., attentional processing (Cardin et al., 2020). Multisensory temporal processing in the pSTS has not been specifically investigated in ED. We hypothesized that VT temporal processing would recruit pSTS to a larger extent in our ED group than in NH. fMRI studies have presented a direct link between pSTS and multisensory (audiovisual) temporal processing in NH individuals (Powers et al., 2012; Noesselt et al., 2012). After initial scanning, perceptual training devised by Powers et al. (2009) resulted in a narrowing of the temporal binding window, decreased BOLD activation within pSTS, AC, visual cortex, and was followed by increased coupling between these regions.

In addition to altered responses of cortical structures, the communication between adjacent and distant structures is also reportedly affected by deafness. A recent review of neurosensory restoration in ED individuals indicated that auditory deprivation has profound implications on brain development in widely distributed networks (Kral et al., 2016). According to the connectome model, brain development depends on experience, lacking audition may affect how the brain responds to remaining senses, and how they are functionally connected. The connectome model also implies that sequential/temporal processing is particularly ‘at risk’ in ED individuals. Altered functional connectivity (FC) between sensory systems and networks of high-order cognitive functions has been reported in ED individuals, mostly based on resting-state data. For example, Ding et al. (2016) found enhanced spontaneous FC in ED individuals between the STG and bilateral insula and dorsal anterior cingulate cortices. These regions are primarily associated with the salience network. Furthermore, increased FC was predictive of better visuospatial working memory task performance in the ED group compared to NH (Ding et al., 2016). Abnormal FC patterns during task-based temporal processing have been reported in ED individuals (Kral et al., 2016). These included weakened connectivity between auditory and somatomotor regions, enhanced connections within the frontoparietal network and its coupling with other large-scale networks, and increased connectivity of the default mode network (Bonna et al., 2019). Compared to resting-state, task-based FC analyses may reveal altered networks involved in these cognitive functions as a result of early deafness (Nobre & van Ede, 2018).

In the current study, we aimed to investigate the effects of early deafness on multisensory temporal processing. Using a multisensory (visuotactile) TOJ task, we compared behavioral performance, BOLD responses, and functional connectivity (both during resting-state and TOJ task) between ED and NH groups. We hypothesized that ED individuals would perform worse than NH on the multisensory TOJ task, if the auditory scaffolding hypothesis holds. Also, we hypothesized that there would be larger recruitment of pSTS for multisensory temporal processing in ED individuals. We further hypothesized that FC between pSTS and unisensory areas would be altered in ED individuals, and this alteration might be more readily revealed in the presence of a multisensory temporal task.

2. Experimental Methods

2.1. Participants

10 early deaf individuals and 10 gender/age-matched hearing controls (matched on a one-to-one basis) were tested. Our research was approved by the Institution Review Board of the University of Nevada, Reno, and conducted in accordance with the Code of Ethics of the World Medical Association (Declaration of Helsinki). Our deaf participants included those who suffered from severe to profound early sensorineural hearing loss (see table 1). They could not understand auditory speech and were proficient in sign language. All participants were right-handed (one deaf individual writes with the right hand, and signs with the left hand). All participants reported having normal or corrected-to-normal visual acuity.

Table 1.

Demographic Details of Early Deaf Participants.

Subject	Sex	Age (years)	Handedness	Deafness Onset (months)	Degree of Deprivation
1	F	42	R	Birth	Both 90dB
2	M	37	R	Birth	L-Total loss R-90dB
3	F	45	R	9	Both 90dB
4	F	56	R	12	L-85dB R-90dB
5	M	50	R	26	L-Total R-120dB
6	F	48	R	Birth	L-90dB R-85dB
7	M	48	R	Birth	Both 90dB
8	M	34	R	15	L-100dB R-100dB
9	F	54	R	Birth	L-110dB R-105dB
10	F	53	R	Birth	L-100dB R-90dB

2.2. Stimuli

Visual and tactile stimuli were generated in MATLAB and PsychToolbox (Brainard, 1997; Pelli, 1997). Visual stimuli were 30 ms of a white circle (3° diameter) on a grey background, presented on a monitor, refreshing at a rate of 60 Hz. In fMRI scans, participants viewed the monitor with a mirror attached to the MR head coil. The tactile stimulus was a 30 ms vibration at 50 Hz delivered to the participant’s index fingers, generated with PiezeTac tactor (Engineering Acoustic), an MRI-compatible vibrotactile transducer.

2.3. fMRI Data Acquisition

Scanning was performed at the Imaging Facility of Renown Health Hospital in Reno, NV on a 3T Philips Ingenia scanner using a 32-channel digital SENSE head coil (Philips Medical Systems, Best, Netherlands). Three-dimensional (3D) anatomical images were acquired at 1 × 1 × 1 mm resolution using a T1-weighted magnetization-prepared rapid gradient echo (MPRAGE) sequence. Functional images were obtained using a standard echo-planar imaging sequence (EPI) with 2.75 × 2.75 × 3 mm voxels. A repetition time (TR) of 2 seconds was used to acquire 40 transverse slices in ascending order, with an echo time (TE) of 25 ms, flip angle 76°, and 220 × 220 mm² field of view.

fMRI data were collected in one or two sessions with the order of experiments (passive, active, resting-state) varied across participants. Resting-state data were typically gathered after the passive experiment and before the active experiment.

2.4. fMRI Visuo-Tactile Passive Experiment

A block design was used to identify regions that responded to visual and tactile stimuli with the hypothesis that pSTS would be recruited in multisensory conditions, and more evidently in ED individuals. For this experiment participants were instructed to pay attention to the stimuli without performing a task while remaining as still as possible. 4 conditions were presented in blocks in a fixed order: visual only (V), tactile only (T), visuotactile asynchronous (VT_A), and visuotactile synchronous (VT_S). Each stimulus block lasted 10 s, followed by a 10 s baseline block with only a fixation cross presented. The total scan lasted for 8 mins, consisting of 24 stimulus blocks (6 repetitions per condition) and 24 baseline blocks. Each participant performed 3–4 scans.

In each stimulus block, a total of 10 trials were presented. In the V blocks, each trial consisted of one visual stimulus, presented in either right or left visual field (9.3° from central fixation) in a randomized order. In the T blocks, each trial consisted of one tactile stimulus, presented either to the right or left index finger in a randomized order. In the VT_S blocks, each trial consisted of a pair of stimuli (one visual and one tactile) presented simultaneously in the right visual field and on the right index finger, or in the left visual field and on the left index finger, in a randomized order. In VT_A blocks, the visual and tactile stimuli were presented in the same spatial arrangement as in the VT_S blocks, but with a 300 ms SOA.

2.5. fMRI Visuo-Tactile Active Experiment

While participants performed a visuotactile TOJ task, a 12-minute scan examined activation and functional connectivity patterns. The TOJ procedure was identical to that described in Scurry et al. (2020a), and based on their results we selected two SOA levels (‘Easy vs. ‘Difficult’). We used a block design comprised of alternating ‘Easy’ (150 ms SOA) and ‘Difficult’ (50 ms SOA) blocks. A total of 30 blocks were presented, separated by a fixation cross interval of 12 seconds. Each 12 s block (6 TRs) contained 4 trials, each lasting 3 seconds including response time, with the order of ‘tactile first’ and ‘visual first’ randomized across trials. Participants completed a total of 120 trials, 60 per SOA level. The tactile device was placed in the participants’ lap and tactile stimulation was delivered to participants’ right index finger. The visual stimulus was presented in the center of the screen. Due to logistical restrictions in the scanner and supine position of the participant, there was no spatial congruency between the visual and tactile stimuli. Participants were asked to judge the temporal order of the stimuli and respond using a two-button response box with their left hand: one button for ‘visual first’ and the other for ‘tactile first’.

For behavioral responses collected during fMRI scanning, a mixed ANOVA was used with between factor of group (ED vs NH) and within factor of SOA level (Easy vs Difficult) on percentage correct judgments of temporal order. All data were analyzed using R statistical software.

2.6. Resting-State Procedure

Eyes-open resting-state data were collected while the participant fixated on the central cross for 10 minutes. The participant was instructed to think about something neutral and remain still. Resting-state data were acquired with parameters specified in section 2.3. fMRI Data Acquisition.

2.7. fMRI Data Pre-Processing

Data were analyzed using Brain Voyager QX (Version 2.8, Brain Innovation, Maastricht, the Netherlands). Initially, functional data underwent pre-processing steps that included 3D motion correction (trilinear/sinc interpolation), high-pass filtering including linear trend removal using GLM approach with a design matrix containing a Fourier basis set (sines and cosines for 2 cycles), and slice scan time correction (cubic spine). No spatial smoothing was applied to functional data. For each participant, pre-processed functional data were co-registered to their corresponding anatomical data. The initial alignment was based on header information from functional and anatomical sessions and fine-tuning alignment was gradient-based. Anatomical and functional data were then transformed into Talairach space (Talairach & Tournoux, 1988). All participants had a root-mean-squared motion of less than one functional voxel size in both resting-state scan (0.60±0.19 mm) and TOJ task scan (0.71±0.30 mm).

2.8. fMRI Regions of Interest (ROIs)

Left and right pSTS regions of interest (ROIs) were anatomically defined based on the Atlas of Intrinsic Connectivity of Homotopic Areas (AICHA, area label 88; Joliot et al., 2015). Additionally, we constrained the anatomically defined areas based on the overlap with functional activity obtained from our passive experiment. This activity was based on the ED group’s significant activation for VT_S and VT_A relative to baseline (q(FDR) < 0.001). This was done due to minimal activation near pSTS in the NH group. We used these (left and right) pSTS overlap ROIs for further analysis, in addition to anatomically defined regions. Using the Julich probabilistic atlas (Eickhoff et al., 2005), we also anatomically defined bilateral primary auditory cortex (PAC) ROIs, based on maximum probability maps (Eickhoff et al., 2006). Anatomical ROIs were transformed to Talairach space (Talairach & Tournoux, 1988). For connectivity analyses, all ROIs were transformed to Montreal Neurological Institute (MNI) space.

2.9. fMRI General Linear Model (GLM) Analysis

For GLM analysis, the BOLD responses during easy and difficult TOJ trials were modeled using a general linear model. Significant responses were quantified using a threshold of q(FDR) < 0.001 for both groups. For bilateral pSTS ROIs (both anatomically defined and the overlap of functional and anatomical definitions), beta weights were extracted for each participant and a mixed ANOVA was calculated for each ROI using a between factor of group (ED vs NH) and a within factor of SOA level (‘Easy’ vs. ‘Difficult’). We also computed whole-brain contrasts of ‘Difficult’ vs. ‘Easy’ conditions to identify regions modulated by task difficulty, and ‘Difficult’ + ‘Easy’ vs. baseline to identify any additional regions implicated during the TOJ task.

2.10. Seed-Based Functional Connectivity Analysis

Pre-Processing.

For each subject, the same preprocessing steps were applied to both resting-state and TOJ task fMRI time series. The first five volumes (10 s) of fMRI data were removed to allow the MR signal to achieve T1 equilibrium. The remaining volumes were slice-timing corrected and realigned to the mean echo-planar image in SPM12 (http://www.fil.ion.ucl.ac.uk/spm/); co-registered to the subject T1 space and then normalized to the standard MNI-152 2 mm-template using Advanced Normalization Tools software (ANTs, http://stnava.github.io/ANTs/). Time series were finally spatially smoothed using a 3D Gaussian filter with a full-width-half-maximum (FWHM) of 6 mm. The root-mean-square (RMS) motion of the fMRI time series was computed for each subject. Specifically, rotational displacements were converted to translational displacements by projection to a surface of a 50 mm radius sphere and RMS head motion was then computed from both the original translational displacements and the converted rotational displacements. The T1 image for each subject was segmented into grey matter (GM), white matter (WM), and cerebrospinal fluid (CSF) to generate subject-specific WM and CSF masks. These masks were then normalized to the standard MNI-152 2 mm space using ANTs. Average time series from subject WM and CSF masks, as well as six head motion parameters, were regressed out from each dataset. All voxel time courses were further bandpass filtered (0.008 Hz < f < 0.1 Hz) and variance normalized.

Functional Connectivity Map.

The six ROIs are described in section 2.8. fMRI Regions of Interest (ROIs) were used as seeds, including both anatomically defined bilateral pSTS, its overlap with activation found in the passive experiment for bilateral pSTS, and anatomically defined bilateral PAC. Each seed ROI was also normalized to the standard MNI-152 2 mm space using ANTs. For every subject, the average time series of each ROI was extracted. A whole-brain voxel-wise correlation map with each average seed signal was computed, and correlation values were finally converted to Fisher’s z scores to ensure a Gaussian-like distribution.

Statistical Analysis.

The same statistical analyses were applied to both resting-state and TOJ task fMRI series. For each seed, a one-sample t-test was used to compute t-statistics of the group-average seed-based functional connectivity map for the ED and NH group, respectively. Cluster-wise correction method was used to correct for multiple comparisons of whole-brain voxels. Specifically, the t-statistics map was first held at a threshold of |t|≥3.25 (corresponding to uncorrected p_uncorrected ≤ 0.001, with 9 degrees of freedom). A cluster size of 245, computed using the 3dClustSim program in AFNI (https://afni.nimh.nih.gov/), was then used to generate the significant t-statistics map at the level of corrected p_corrected = 0.05.

In addition, a two-sample t-test was used to compute the difference map between ED and NH groups for each seed. We used the same cluster-wise correction method to correct for multiple comparisons, with the only difference being between-group difference map thresholding of |t| ≥ 2.86, corresponding to p_uncorrected ≤ 0.001 with 19 degrees of freedom. Overall, these seed-based FC analyses followed a bi-regional (seed vs. the whole brain) method of simple correlation. This allowed us to investigate group differences of stronger or weaker connectivity between our seeds and all other brain regions (Wang et al., 2019).

3. Results

3.1. fMRI Visuo-Tactile Passive Activation Experiment

Illustrated below (Figure 1) was significant activation in the ED group for VT processing (VT_S + VT_A vs. baseline) near bilateral pSTS regions (q(FDR) < 0.001). The lack of significant activity in the NH group led us to define functional pSTS ROIs exclusively from ED group data. We also ran contrasts VT_A vs. VT_S and VT_S vs. V + T to establish the region(s) involved in temporal synchrony and multisensory integration, respectively. We found significant activation (q(FDR) < 0.05) in the ED group for temporal asynchrony (VT_A vs. VT_S) within the left pSTS (peak activation Talairach coordinates: x: = −54, y: = −46, z: = 10, and 20 surrounding voxels). The contrast testing multisensory integration (VT_S vs. V + T) did not reveal any significant activation.

Figure 1. Visuo-Tactile Processing Recruits Bilateral pSTS to a Larger Extent in ED Than in NH.

Whole-brain GLM Analysis revealed significant activity near bilateral pSTS (q(FDR) < 0.001) for VT synchrony and asynchrony conditions in ED individuals, but not in NH. The cross is centered on Talairach coordinates: x: = 49, y: = −38, z: = 6. The color bar represents t-values.

Overall, these four stimulus conditions (V, T, VT_A, VT_S) drove similar activation within areas of bilateral pSTS. To confirm this, we further extracted beta weights for each condition in the anatomically defined pSTS (left and right, separately). ANOVAs did not reveal significant differences across conditions.

Anatomically defined ROIs are illustrated in Figure 2 (a-b, e-f). The central coordinates and sizes of left pSTS and right pSTS were x: = −60, y: = −44, z: = 12, and 291 voxels and x: = 50, y: = −44, z: = 14 and 323 voxels, respectively. For anatomically defined left PAC and right PAC these were x: = −44, y: = −16, z: = 7 (214 voxels); and x: = 45, y: = −19, z: = 9 (223 voxels), respectively. Functionally defined left and right pSTS are also illustrated in Figure 2 (c-d) with central Talairach coordinates x: = −60, y: = −35, z: = 10 and 95 voxels at functional resolution (3 × 3 × 3 mm) and x: = 51, y: = −39, z: = 6, and 73 voxels, respectively. The overlap of our functionally and anatomically defined pSTS (left and right) was 14.09% and 15.79%, respectively. The central coordinates and sizes for left and right overlap pSTS were x: = −59, y: = −41, z: = 8 and 40 voxels, and x: = 52, y: = −41, z: = 8 and 49 voxels, respectively.

Figure 2. Functionally and Anatomically Defined ROIs on a Representative Brain in Talairach Space.

Sagittal view of anatomically defined left pSTS (a), right pSTS (b), functionally defined left pSTS (c), right pSTS (d), anatomically defined left PAC (e), and right PAC (f).

3.2. Larger Activation in Bilateral pSTS in ED Than in NH During Active Experiment

ED and NH individuals performed similarly on the TOJ task during fMRI scanning. There was an effect of SOA level on task performance (F(1,32) = 229.6, p < .0001), with both groups showing higher accuracy (% correct) for ‘Easy’ trials (ED: mean 95%, SEM 1.2%; NH: mean 94.6%, SEM 1.1%) than for ‘Difficult’ trials (ED: mean 68.5%, SEM 2%; NH: mean 68.3%, SEM 2.3%). There was no effect of group and no interaction between group and SOA level (ps > 0.87).

Beta weights extracted for anatomically defined left and right pSTS ROIs during the task confirmed an effect of group (left pSTS: F(1,36) = 15.402, p < .0001, right pSTS: F(1,36) = 16.845, p < .0001). The effect of group was also confirmed using beta weights extracted for the overlap of functionally and anatomically defined left and right pSTS ROIs (left pSTS: F(1,36) = 25.335, p < .0001, right pSTS: F(1,36) = 18.022, p < .0001). Overall, ED individuals showed higher activation than NH in left anatomically defined pSTS (ED mean beta = 0.517, SEM = 0.127, NH mean beta = −0.005, SEM = 0.135) and right anatomically defined pSTS (ED mean beta = 0.626, SEM = 0.158, NH mean beta = −0.033, SEM = 0.121). This was also found in the overlap of anatomically and functionally defined bilateral pSTS areas (left: ED mean beta = 0.630, SEM = 0.150, NH mean beta = 0.024, SEM = 0.071; right: ED mean beta = 0.545, SEM = 0.119, NH mean beta = −0.011, SEM = 0.136). There was no effect of SOA level (‘Easy’ vs. ‘Difficult’) and no interaction between group and SOA level (ps > 0.378), indicating higher activation of bilateral pSTS in deaf individuals regardless of trial difficulty.

The whole-brain contrast of ‘Difficult’ vs. ‘Easy’ revealed no significant differences in activation for either group. The whole-brain contrast of ‘Difficult’ + ‘Easy’ vs. baseline revealed significant activation within bilateral pSTS for ED individuals, but not for NH. In addition, there were regions only activated in ED individuals, including bilateral pSTG, right inferior parietal lobule, and bilateral insula cortex. The only common region activated in both groups was the left supplementary motor area.

3.3. Between-Group FC Differences with Anatomically Defined pSTS Seed in Resting-State and During Active Experiment

Connectivity patterns for each group are illustrated side-by-side in both Figure 3 (left pSTS seed) and Figure 4 (right pSTS seed). Also included in these figures are between-group difference maps (panels e & f in both figures). Tables 2 -7 list labels of significant regions corresponding to the Automated Anatomical Labelling atlas (AAL, Rolls et al., 2020) based on cluster peak coordinates. Each entry contains the cluster extent, statistical value, and MNI coordinates. Tables 2 & 3 provide details of FC for anatomically defined left and right pSTS seeds, respectively. For left pSTS seed during resting-state (Table 2), the stronger connected regions for ED than in NH individuals were left precuneus and vermis. One weaker connected region was the left gyrus rectus. There were no stronger connected regions in resting-state for the ED group than in NH with anatomically defined right pSTS seed (Table 3). The only weaker connected region was the right Rolandic operculum.

Figure 3. Left pSTS Seed-Based Voxel-Wise Functional Connectivity Maps for Active Experiment.

Anatomically defined seed results are shown for ED (a) and NH groups (b). Overlap of anatomically and functionally defined seed results are shown for ED (c) and NH (d), and the difference between ED and NH groups for anatomically defined seed (e) and overlap seed (f). Color bars represent t-values. For (e) and (f) positive (negative) t-values represent stronger (weaker) connectivity in ED individuals.

Figure 4. Right pSTS Seed-Based Voxel-Wise Functional Connectivity Maps for Active Experiment.

Anatomically defined seed results are shown for ED (a) and NH groups (b). Overlap of anatomically and functionally defined seed results are shown for ED (c) and NH (d), and the difference between ED and NH groups for anatomically defined seed (e) and overlap seed (f). Color bars represent t-values. For (e) and (f) positive (negative) t-values represent stronger (weaker) connectivity in ED individuals.

Table 2.

Anatomically defined left pSTS seed to whole-brain FC Differences between ED & NH in Resting-State and During Active Experiment.

Resting-State			MNI Coordinates
Anatomical left pSTS with region	Extent	t-value	x	y	z
Vermis_1_2	69	6.021	4	−36	−20
Precuneus_L	54	4.542	−14	−64	32
Rectus_L	89	−4.792	0	30	−22
During Active Experiment			MNI Coordinates
Anatomical left pSTS with region	Extent	t-value	x	y	z
Insula_R	420	6.738	36	18	0
Frontal Inf_Oper_R	420	5.096	58	16	6
Cerebelum_8_L	95	5.384	−40	−44	−50
ParaHippocampal_L	243	−7.371	−30	−2	−30
Occipital_Sup_L	47	−7.256	−18	−82	44
Calcarine_L	148	−6.116	−18	−70	14
Lingual_R	63	−4.751	14	−60	6
Cerebelum_9_L	54	−4.457	−6	−52	−32

Displayed are region labels corresponding to peak coordinate of cluster based on AAL, size of cluster, t-statistic, and MNI coordinates. All entries represent stronger (grey) or weaker (white) connectivity in ED than in NH. Analyses represent between-group differences at uncorrected p<0.001.

Table 7.

Overlap of Anatomically and Functionally defined right pSTS seed to whole-brain FC Differences between ED & NH in TOJ task and Resting-State.

Resting-State			MNI Coordinates
Right pSTS with region	Extent	t-value	x	y	z
Postcentral_R	57	−5.400	56	−2	20
Supp_Motor_Area_R	113	−5.233	10	−16	74
Rolandic_Oper_L	46	−4.869	−54	0	8
During Active Experiment			MNI Coordinates
Right pSTS with region	Extent	t-value	x	y	z
Supp_Motor_Area_R	550	9.854	4	22	54
Insula_L	192	7.740	−36	16	−4
Temporal_Sup_L	165	6.324	−60	−34	16
Insula_R	146	5.380	38	20	0
Temporal_Sup_R	73	4.937	56	−22	0
Frontal_Mid_2_R	54	4.578	44	50	20
Occipital_Mid_L	72	−4.332	−32	−94	−6

Displayed are region labels corresponding to peak coordinate of cluster based on AAL, size of cluster, t-statistic, and MNI coordinates. All entries represent stronger (grey) or weaker (white) connectivity in ED than in NH. Analyses represent between-group differences at uncorrected p<0.001.

Table 3.

Anatomically defined right pSTS seed to whole-brain FC Differences between ED & NH in Resting-State and During Active Experiment.

Resting-State			MNI Coordinates
Anatomical right pSTS with region	Extent	t-value	x	y	z
Rolandic_Oper_L	47	−5.199	−52	2	2
During Active Experiment			MNI Coordinates
Anatomical right pSTS with region	Extent	t-value	x	y	z
Supp_Motor_Area_R	338	6.944	4	22	58
Temporal_Sup_R	151	6.603	60	−18	0
Insula_L	105	5.668	−34	18	−4
Frontal_Inf_Orb_2_R	85	5.488	48	36	−10
Calcarine_R	52	−5.596	18	−62	10

Displayed are region labels corresponding to peak coordinate of cluster based on AAL, size of cluster, t-statistic, and MNI coordinates. All entries represent stronger (grey) or weaker (white) connectivity in ED than in NH. Analyses represent between-group differences at uncorrected p<0.001.

During the TOJ task, the left anatomically defined pSTS seed (Figure 3a-b & e) showed stronger connections in the ED group than in NH to regions including left cerebellum, right inferior frontal gyrus (opercular), and right insula. Weaker-connected regions found for the ED group than in NH included the left hippocampal gyrus and numerous visual areas (left calcarine, left superior occipital gyrus, and right lingual gyrus). For the right anatomically defined pSTS seed during the active experiment (Figure 4a-b & e), stronger connected regions in the ED group than in NH were the right inferior frontal gyrus (pars orbitalis and opercular), left insula, right superior temporal gyrus, and right supplementary area. The only weaker connected region to right anatomically defined pSTS seed in the ED group than in NH during the task was the right calcarine.

3.4. Between-Group FC Differences with Overlap of Anatomically and Functionally Defined pSTS Seeds in Resting-State and During Active Experiment

Differences between groups in resting-state and during the active experiment are detailed in Tables 6 & 7. For resting-state, a stronger connected region in the ED group than in NH with the overlap left pSTS as seed (Table 6) was the right Rolandic operculum. Weaker-connected regions were the left gyrus rectus, medial superior frontal gyrus, precentral gyrus, and right angular gyrus. With the overlapped right pSTS as seed (Table 7), there were no stronger connections in the ED group than in NH. Weaker connections were right supplementary motor area and left Rolandic operculum.

Table 6.

Overlap of Anatomically and Functionally defined left pSTS Seed to Whole-Brain FC Differences Between ED & NH in Resting-State and During Active Experiment.

Resting-State			MNI Coordinates
Left pSTS with region	Extent	t-value	x	y	z
Rolandic_Oper_R	52	5.690	62	12	2
Frontal_Sup_Medial_R	58	−5.545	12	62	10
Angular_R	331	−5.281	52	−62	28
Frontal_Sup_Medial_L	68	−4.959	−14	58	10
Precentral_R	132	−4.804	16	−26	70
Rectus_L	75	−4.484	−4	38	−16
During Active Experiment			MNI Coordinates
Left pSTS with region	Extent	t-value	x	y	z
Temporal_Mid_R	65	6.292	56	−58	10
Temporal_Pole_Sup_R	81	5.657	60	10	−2
Insula_R	82	5.256	38	16	−6
Vermis_4_5	158	5.140	−2	−56	−22
Occipital_Sup_L	243	−5.980	−20	−82	44
Frontal_Sup_2_L	218	−5.832	−12	42	44
Frontal_Sup_Medial_L	218	−4.744	−6	58	28
Frontal_Sup_Medial_L	156	−5.305	−14	54	0
Precuneus_L	45	−4.850	−12	−58	16

Displayed are region labels corresponding to peak coordinate of cluster based on AAL, size of cluster, t-statistic, and MNI coordinates. All entries represent stronger (grey) or weaker (white) connectivity in ED than in NH. Analyses represent between-group differences at uncorrected p<0.001.

During our active experiment, left and right overlap pSTS seeds revealed a similar list of stronger and weaker connected regions to those revealed using anatomically defined pSTS seeds. For the overlap left pSTS seed (Figure 3c-d & f), stronger connected regions in the ED group than in NH included the middle temporal gyrus, right superior temporal pole, right insula, and areas of the vermis. Weaker connected regions included dorsolateral and medial superior frontal gyrus, left precuneus, and left superior occipital gyrus. For the overlap right pSTS seed (Figure 4c-d & f), stronger connected regions in the ED group than in NH were the right supplementary motor area, bilateral insula, bilateral superior temporal gyrus, and right middle frontal gyrus. The only weaker connected region was the left middle occipital gyrus.

3.5. Between-Group FC Differences with Anatomically Defined PAC Seeds in Resting-State and During Active Experiment

Differences between groups in resting-state and during the active experiment are detailed in Tables 8 & 9 and illustrated in Supplementary Figure 1a-f. Resting-state FC analysis using left PAC as seed (Table 8) revealed no instances of stronger connectivity in the ED group than in NH. Weaker connected regions were left opercular of inferior frontal gyrus, bilateral postcentral gyrus, right precentral gyrus, right middle temporal gyrus, and the left paracentral lobule. For right PAC as seed in resting-state, there were no stronger connected regions in the ED group than in NH, and the only region to show reduced connectivity was the left postcentral gyrus.

Table 8.

Anatomically defined left PAC Seed to Whole-Brain FC Differences Between ED & NH in TOJ Task and Resting-State.

Resting-State			MNI Coordinates
Left PAC with Region	Extent	t-value	x	y	z
Frontal_Inf_Oper_L	185	−5.761	−52	8	18
Postcentral_R	189	−5.607	62	−6	22
Temporal_Mid_R	53	−5.312	52	−44	−4
Paracentral_Lobule_L	52	−4.925	−10	−24	52
Precentral_R	111	−4.872	32	−14	62
Postcentral_L	179	−4.847	−50	−14	50
During Active Experiment			MNI Coordinates
lPAC with Region	Extent	t-value	x	y	z
Putamen_R	592	7.032	20	12	−6
Frontal_Sup_Medial_R	65	4.883	6	20	42

Displayed are region labels corresponding to peak coordinate of cluster based on AAL, size of cluster, t-statistic, and MNI coordinates. All entries represent stronger (grey) or weaker (white) connectivity in ED than in NH. Analyses represent between-group differences at uncorrected p<0.001.

Table 9.

Anatomically Defined Right PAC Seed to Whole-Brain FC Differences Between ED & NH in Resting-State and Active Experiment.

Resting-State			MNI Coordinates
Right PAC with Region	Extent		x	y	z
Postcentral_L	106	4.580	−56	−2	22
During Active Experiment			MNI Coordinates
Right PAC with Region	Extent		x	y	z
Insula_R	493	6.161	32	18	−6
Pallidum_R	493	4.534	20	0	−4
Temporal_Inf_R	44	5.949	64	−34	−18
Frontal_Sup_Medial_R	185	5.662	4	24	42

Displayed are region labels corresponding to peak coordinate of cluster based on AAL, size of cluster, t-statistic, and MNI coordinates. All entries represent stronger (grey) or weaker (white) connectivity in ED than in NH. Analyses represent between-group differences at uncorrected p<0.001.

In contrast to resting-state FC, bilateral PAC (left and right separately) as seed during the active experiment (Table 9) displayed predominantly stronger connectivity in ED than in NH. For the left PAC seed (Supplementary Figure 1e), these were right putamen and superior medial frontal gyrus. No regions showing weaker connectivity with left PAC during the active experiment. For the right PAC as seed (Supplementary Figure 1f), stronger connected regions for the ED group than in NH were the right insula, right pallidum, right inferior temporal gyrus, and right medial superior frontal gyrus. There were no weaker connections.

4. Discussion

The main aim of the current study was to examine the effects of early deafness on temporal processing. Using a visuotactile TOJ task (referred to as our active task), we compared the precision of multisensory temporal processing between early deaf individuals and normal hearing controls. We quantified BOLD responses to visuotactile stimuli in both groups. In addition, we conducted functional connectivity analysis to identify changes in connectivity patterns as a result of early deafness during both resting-state and the active task.

4.1. Similar Behavioral Performance in Active Experiment Between Groups

We found that deafness does not significantly impair multisensory temporal processing on a VT TOJ task. This finding is consistent with a recent study from our lab where more SOA levels (15 total) were used in the same task while EEG signals were recorded (Scurry et al., 2020a). Therefore, despite auditory deprivation, VT temporal processing ability was largely intact, providing no evidence for the auditory scaffolding hypothesis that temporal sequencing abilities are disturbed in ED individuals due to the lack of auditory input (Conway et al., 2009). The explicit VT TOJ task might not be sensitive enough to reveal the impact of deafness on multisensory temporal processing. In addition, our study only included 10 participants in each group. The relatively small sample size might have limited our ability to find a significant between-group difference.

Differences between ED and NH groups in multisensory temporal processing have been revealed using the touch-induced double-flash illusion paradigm. The illusion of a double-flash can be induced by two touches to the face when visual and tactile stimuli are delivered in close temporal proximity (Karns et al., 2012). ED individuals showed greater susceptibility to this illusion than NH, implying greater tactile influence on visual perception when temporal processing was involved. This double-flash illusion is more implicit and might be a more sensitive paradigm to detect the effects of deafness on multisensory temporal processing.

4.2. BOLD Activation in Bilateral pSTS in ED During Passive Experiment

We observed activation in bilateral pSTS of ED individuals for both unisensory and multisensory conditions (V, T, VT_S, VT_A). When comparing multisensory asynchronous and synchronous conditions, we found a larger BOLD response to asynchronous trials in a small region within the left pSTS, suggesting its involvement for multisensory temporal processing in ED individuals. This specific left-lateralized area, posterior to the auditory association cortex, showed activation for an auditory rhythm working memory paradigm in NH individuals, and for a matched visual flashing rhythms task in ED (Bola et al., 2017). In keeping with the functional constancy hypothesis, our data support preserved function in this part of left pSTS for processing of VT instead of auditory information in ED individuals. This might reflect adaptation for VT sign language capacity rather than for spoken language, although this has not been directly tested in ED individuals (Jones et al., 2009).

4.3. Significantly Larger BOLD Activation in Bilateral pSTS and Other Regions in ED During Active Experiment

We found significantly larger bilateral pSTS activation in the ED group than in NH during our VT TOJ task (‘Easy’ + ‘Hard’ vs. baseline). This is consistent with previous findings that pSTS was recruited for tactile and visual processing in ED individuals (Karns et al., 2012). We found no difference in response magnitude between ‘Hard’ and ‘Easy’ SOA conditions, suggesting overall compensatory recruitment of the pSTS in ED individuals that was not modulated by task difficulty. In addition, there was no clear hemispheric dominance for this larger pSTS activation in the ED group. This differs from previous reports of right lateralization of pSTS in ED individuals for processing of tactile motion (Scurry et al., 2020b), directional visual motion (Retter et al., 2019), and reorientation of attention between overlapping visual and somatosensory stimulation (Cardin et al., 2020).

Note that increased bilateral pSTS activation was found in ED individuals while they performed similarly to NH in the visuotactile TOJ task. A lack of behavioral change in the visuotactile TOJ task was similarly reported in ED by Scurry et al. (2020a), despite their larger visual (P100) and tactile (N140) ERP responses. Even with enhanced BOLD activation in bilateral pSTS, ED individuals showed no behavioral change in a Japanese sign language comprehension task (Sadato et al., 2004). Combined, these larger responses in ED individuals might be indicative of compensatory recruitment as a result of sensory deprivation (Bola et al., 2017; Güdücü et al., 2019). These compensatory responses might have contributed to the lack of evidence for the auditory scaffolding hypothesis, i.e., worse temporal performance in the ED group than in NH (Conway et al., 2009).

The present study found the following regions were activated only in ED individuals during the VT TOJ task: bilateral pSTS, right inferior parietal lobule (IPL), bilateral posterior superior temporal gyrus (pSTG), and bilateral insula cortex. These areas have been activated in previous studies in ED individuals that used multisensory, temporal, and language-based tasks (Twomey et al., 2017; Li et al., 2014; Allen et al., 2008). The VT TOJ task we used involves sequential multisensory processing, which is also required for language. The bilateral insula cortex was activated during VT integration for sign language production in an ED group and not in NH (Allen et al., 2008). Bilateral pSTG activation occurred in ED individuals (not in hearing signers) while viewing British sign language (BSL) phonological content (Twomey et al., 2017). Silent reading and writing tasks activated bilateral IPL in an ED group and not in NH (Li et al., 2014). This difference was attributed to the ED group’s reliance on sign language-equivalent letter-phoneme mapping to aid reading rhythm in the absence of auditory imagery. Spoken language is the most prominent multisensory temporal experience in NH individuals, as is sign language in ED. Combined, these regions activated solely in the ED group during our VT TOJ task and previous language tasks might illustrate altered multisensory temporal processing as a result of auditory deprivation (Cardin et al., 2020).

The only region activated in both groups was the left supplementary motor area (SMA). This region is implicated for temporal processing in NH, particularly sensorimotor synchronization (SMS), due to enhanced connections between auditory areas and SMA that underly human receptiveness to rhythm. NH individuals might be able to perform temporal tasks without recruiting additional areas reported in ED individuals.

4.4. Similar Resting-State Functional Connectivity Between Groups

Our resting-state findings are consistent with reports of overall reduced spontaneous connectivity between regions in ED individuals (Ding et al., 2016; Bonna et al., 2019). In the present study, weaker connected regions with the left pSTS as seed included the right precentral gyrus and right angular gyrus. These regions form part of a multisensory associative area, known to be involved in semantic processing in NH individuals (Seghier, 2013). In addition, we found weaker FC in the ED group between right pSTS and bilateral motor areas (right supplementary motor area, right precentral gyrus, and left Rolandic operculum). Bonna et al. (2019) also reported reduced resting-state FC between auditory and motor areas in ED individuals. Their interpretation was that circuits required for speech production do not develop in ED individuals, since they do not need the feedback language-speech mechanism involved in vocal movements. Left Rolandic operculum is specifically implicated in the synchronization of external rhythm to one’s heartbeat (Blefari et al., 2017) and the lesion in the left Rolandic operculum is known to cause speech disturbance in NH individuals (Triarhou, 2021). Thus, reduced FC with this area might be expected in ED individuals.

4.5. Task-based Functional Connectivity Readily Revealed More Between-Group Differences for Bilateral pSTS Seeds

We predicted that changes in FC would be more readily revealed by task than resting-state fMRI data. Indeed, our FC analysis during the VT TOJ task using anatomically defined pSTS seeds and the overlap of anatomically and functionally defined pSTS seeds similarly revealed more between-group differences. These differences are task-specific, allowing us to gain insight into the effect of deafness on FC underlying multisensory temporal processing.

The most interesting finding was weaker connectivity for the ED group than in NH between bilateral pSTS as seed (left and right separately) and visual areas. For left pSTS, we found reduced connectivity with visual areas (left calcarine sulcus and left superior occipital gyrus). With the right pSTS as seed, these weaker connected visual areas were left middle occipital gyrus and right calcarine. In contrast, we found stronger FC between bilateral pSTS seeds and the right supplementary motor area (SMA) in our ED group than in NH during the VT TOJ task. ED individuals have been found to synchronize with superior accuracy than NH for visual sensorimotor synchronization (Iversen et al., 2015). Further neuroimaging studies are necessary to determine whether the stronger connectivity between bilateral pSTS and right SMA we found in ED individuals contributes to their superior synchronization abilities.

Given deaf individuals’ heightened susceptibility to touch-induced double-flash illusion (Karns et al., 2012), touch may have relied upon more than vision when processing the two in a multisensory temporal scenario. This was also suggested by Xiao et al, (2021) who found a reduced visual influence over tactile motion perception in ED individuals when motion direction was defined using temporal asynchrony cues. Our finding of altered connectivity between pSTS and unisensory areas in the ED group might reflect the increased reliance on touch over vision to achieve comparable TOJ performance. This is consistent with the modality appropriateness hypothesis (Welch & Warren, 1980), where sensory processing would be optimal when the most salient stimulus modality is prioritized. Indeed, it has been shown that in the absence of audition, touch is prioritized for temporal tasks, whereas vision is prioritized for spatial tasks (Karns et al., 2012; Heimler et al., 2017).

4.6. Resting-State and TOJ Task Functional Connectivity for Bilateral PAC Seeds

The PAC is the prime cortical area studied for reorganization in ED individuals (Karns et al., 2012; Almeida et al., 2015). However, using left and right anatomically defined PAC as seeds, we did not observe the altered connections to somatomotor or visual areas. One notable stronger connected region in the ED group than in NH to the left PAC during the VT TOJ task was the right putamen. This area has been implicated in the perceptual aspects of pain and pain-related motor responses (Starr et al., 2011). Since our tactile stimulation was not accompanied by any pain sensations, stronger FC might represent a heightened state of alertness to potential tactile cues in ED individuals than in NH during our TOJ task.

4. Conclusions

Overall, we found compensatory reorganization in the pSTS in ED individuals, despite comparable behavioral performance to NH in the multisensory TOJ task. Both left and right pSTS were activated significantly more in the ED group than in NH for VT stimulation. Using anatomically defined pSTS seeds, our FC analysis revealed stronger somatomotor and weaker visual regional connections in the ED group than in NH during the TOJ task. These results suggest that a lack of auditory input might alter the balance of tactile and visual area FC with pSTS when a multisensory temporal task is involved.

Supplementary Material

1

Supplementary Figure 1. Anatomically Defined Bilateral PAC Seed-Based Voxel-Wise Functional Connectivity Maps for Active Experiment. Illustrated for ED group: FC for left (a) and right (b) PAC seeds, and for NH group: FC for left (c) and right (d) PAC seeds, and stronger connectivity for ED group than in NH for left (e) and right (f). Color bars represent t-values. Positive (negative) t-values represent stronger (weaker) connectivity in ED group.

Table 4.

Functionally defined left pSTS Seed to Whole-Brain FC Differences Between ED & NH in Resting-State and During Active Experiment.

Resting-State			MNI Coordinates
Left pSTS with region	Extent	t-value	x	y	z
Frontal_Sup_2_R	177	−4.406	18	−20	72
Precentral_R	45	−4.557	28	−14	56
During Active Experiment			MNI Coordinates
Left pSTS with region	Extent	t-value	x	y	z
Frontal_Inf_Orb_2_R	228	6.880	34	26	−6
Frontal_Inf_Oper_R	50	6.691	62	14	8
Vermis_4_5	179	5.922	−2	−58	−14
Temporal_Sup_R	105	5.452	54	−36	12
Insula_L	118	4.928	−34	22	−8
Precentral_L	47	4.840	−40	−2	60
Cuneus_R	92	−7.614	20	−84	42
Occipital_Sup_L	268	−6.368	−20	−82	44
Temporal_Pole_Mid_L	90	−6.225	−24	14	−36
Calcarine_R	152	−5.646	20	−64	10
Frontal_Sup_Medial_L	109	−5.131	−14	52	−2
Precuneus_L	55	−4.518	−10	−60	14

Displayed are region labels corresponding to peak coordinate of cluster based on AAL, size of cluster, t-statistic, and MNI coordinates. All entries represent stronger (grey) or weaker (white) connectivity in ED than in NH. Analyses represent between-group differences at uncorrected p<0.001.

Table 5.

Functionally defined right pSTS seed to whole-brain FC Differences between ED & NH in Resting-State and During Active Experiment.

Resting-State			MNI Coordinates
Right pSTS with region	Extent	t-value	x	y	z
Supp_Motor_Area_R	195	−5.457	10	−16	72
Rolandic_Oper_L	62	−5.281	−54	0	8
Postcentral_L	48	−4.692	−54	−6	26
During Active Experiment			MNI Coordinates
Right pSTS with region	Extent	t-value	x	y	z
Insula_L	165	7.713	−36	14	−4
Supp_Motor_Area_R	445	7.343	4	24	54
Cingulate_Mid_L	445	4.628	−10	20	38
Temporal_Sup_L	155	6.166	−60	−32	16
Frontal_Mid_2_R	156	5.320	48	44	20
Insula_R	99	5.097	38	20	0
Parietal_Inf_L	59	4.995	−52	−38	46
Temporal_Sup_R	45	4.931	62	−18	0
Occipital_Mid_L	71	−5.137	−24	−102	0
Temporal_Pole_Mid_L	74	−4.748	−38	18	−32
Temporal_Mid_L	61	−4.391	−56	0	−24

Displayed are region labels corresponding to peak coordinate of cluster based on AAL, size of cluster, t-statistic, and MNI coordinates. All entries represent stronger (grey) or weaker (white) connectivity in ED than in NH. Analyses represent between-group differences at uncorrected p<0.001.

Highlights.

• Similar temporal order judgment task performance between early deaf and controls

• Altered functional connectivity in early deaf during temporal order judgment task

• Deafness might change the balance between tactile and visual processing