Graph theoretical analysis of functional network for comprehension of sign language

Abstract

Signed languages are natural human languages using the visual-motor modality. Previous neuroimaging studies based on univariate activation analysis show that a widely overlapped cortical network is recruited regardless whether the sign language is comprehended (for signers) or not (for non-signers). Here we move beyond previous studies by examining whether the functional connectivity profiles and the underlying organizational structure of the overlapped neural network may differ between signers and non-signers when watching sign language. Using graph theoretical analysis (GTA) and fMRI, we compared the large-scale functional network organization in hearing signers with non-signers during the observation of sentences in Chinese Sign Language. We found that signed sentences elicited highly similar cortical activations in the two groups of participants, with slightly larger responses within the left frontal and left temporal gyrus in signers than in non-signers. Crucially, further GTA revealed substantial group differences in the topologies of this activation network. Globally, the network engaged by signers showed higher local efficiency (t₍₂₄₎ = 2.379, p = 0.026), small-worldness (t₍₂₄₎ = 2.604, p = 0.016) and modularity (t₍₂₄₎ = 3.513, p = 0.002), and exhibited different modular structures, compared to the network engaged by non-signers. Locally, the left ventral pars opercularis served as a network hub in the signer group but not in the non-signer group. These findings suggest that, despite overlap in cortical activation, the neural substrates underlying sign language comprehension are distinguishable at the network level from those for the processing of gestural action.

1. Introduction

Signed languages are natural human languages expressed through movements of hands, face and body. On the surface, signed languages look similar to non-linguistic communicative actions such as gestures and pantomimes. In contrast to gestural actions, signed languages have an intricate compositional structure identified at the levels of phonology, morphology, syntax and discourse (Emmorey, 2002; Tang, 2006; Valli and Lucas, 2000). Studying how sign language is processed in the brain could provide important insights into understanding to what extent language processing builds upon the general human action perception system, which encompasses a wide range of human actions including imitation, social intent, and human language (Corina and Knapp, 2006; MacSweeney et al., 2008; Rizzolatti and Arbib, 1998).

Based on univariate activation analyses of neuroimaging data, previous studies using MEG (Levanen et al., 2001), PET (Corina et al., 2007) or fMRI (Emmorey et al., 2010; MacSweeney et al., 2004, 2006; Newman et al., 2015) have revealed both extensive overlap and linguistic-specific cortical activations between sign language comprehension and gestural action observation. Overlaps in cortical activation are mainly observed in the superior and middle temporal cortex, the inferior frontal cortex, the superior/inferior parietal lobe, and the fusiform gyrus. For example, MacSweeney et al. (2004) compared neural correlates of viewing British sign language and a manual-brachial code in deaf signers, hearing signers and hearing non-signers. A very high similarity in brain activation relative to a low baseline (viewing the model at rest) between the two conditions was found regardless the hearing status or sign language knowledge of subjects. Studies comparing signers and non-signers viewing sign language have also revealed extensive overlap in cortical activation (Levanen et al., 2001; MacSweeney et al., 2004, 2006). In this paradigm, as non-signers have no access to the linguistic meaning of signs, signs are likely to be processed in a similar way as non-linguistic gestural actions (Levanen et al., 2001; MacSweeney et al., 2004). Therefore, the differences between signers and non-signers can be considered to reflect the differences between sign language comprehension and gestural action observation (Levanen et al., 2001), meanwhile perceptual level discrepancies between language and non-linguistic stimuli are ideally controlled. These overlapped cortical activations are suggested to reflect the processing for visual motor sequences and communication intent that are involved in both sign language comprehension and gestural action observation. Beyond these shared neural correlates, sign-specific cortical activation was also revealed in previous studies (Corina et al., 2007; Emmorey et al., 2015; Newman et al., 2015), mostly at the left posterior perisylvian cortex. For example, Newman et al. (2015) found that the left inferior frontal gyrus (IFG) and the middle superior temporal gyrus (STS) in deaf native signers were more strongly activated by American sign language (ASL) than gestures expressing approximately the same content. In the same study, deaf signers also showed stronger activation for ASL than hearing non-signers in the anterior/middle STS bilaterally and in the left IFG.

While studies based on univariate activation analysis have delivered rich insights into the cortical localization for sign language comprehension versus gestural action perception, whether and how the two types of processing would differ at neural network level remain poorly understood. As accumulating evidence confirms that complex cognitive functions are supported by dynamic interactions and integrative processes across multiple distributed regions (Park and Friston, 2013; van den Heuvel and Sporns, 2013a,b,c), it is possible that the configuration of the neural network involved in the comprehension of sign language by signers is different from the one involved in the observation of sign language by non-signers, despite the overlap in cortical activation.

Here we move beyond previous studies by examining the network configuration of neural circuitry involved in the comprehension of sign language by signers versus the observation of sign language by non-signers using fMRI and GTA, which provides insights into neural substrates underlying sign language comprehension versus gestural action perception. In GTA, brain networks are mathematically characterized as graphs, essentially comprising sets of nodes (brain regions, voxels or other neuronal elements) and edges (their interactions). The arrangement of nodes and edges defines the network’s topology (He and Evans, 2010; van den Heuvel and Sporns, 2013a,b,c). It is widely accepted that functional segregation and functional integration are the key organizational principles of brain network (Sporns, 2013). In GTA, functional segregation can be characterized by network local efficiency and modularity, and functional integration can be characterized by network global efficiency. The balance between functional segregation and functional integration is essential for the operation of networks underlying cognitive functions, and it can be characterized by the graph property of small-worldness (Rubinov and Sporns, 2010). Furthermore, it is proposed that important integrative functions are enabled by a specific set of regions that are often referred to as network hubs, which are generally characterized by a high degree of connections with other nodes and a central placement in the network (van den Heuvel and Sporns, 2013a,b,c).

In the present study, we compared brain activities and functional organization of the activated network between a group of hearing signers (bimodal bilinguals proficient in Chinese Sign Language and spoken Chinese) and a matched group of hearing non-signers (monolinguals proficient in spoken Chinese) during the perception of Chinese Sign Language (CSL). In the first step analysis, cortical activations that were elicited by sign language relative to a static model baseline condition were examined for the signer and non-signer groups. Based on prior literature (Corina et al., 2007; Levanen et al., 2001; MacSweeney et al., 2004, 2006), we expected that sign language would yield similar activations for the two groups in a widely distributed set of brain areas, including the occipital-parietal regions and regions within the perisylvian cortex, with differential activations in focal regions of the temporal and frontal cortices. In the second step analysis, we applied GTA to test whether the commonly activated network would be differentially organized in signers compared to non-signers. At global level, we examined network efficiency, modularity and small-worldness. At local level, we examined nodal degree, nodal efficiency and nodal betweenness. Through these measures, we comprehensively explored the functional segregation and functional integration aspects of the network, and identified regions that playing a central role in the network. Considering that the CSL is linguistically meaningful for the signers, but linguistically meaningless for the non-signers, we expected that the network engaged by signers would present different topologies from that engaged by non-signers. We also performed two control analyses including the examination of the topological properties of the network in signers versus non-signers either during the baseline phase (in which no linguistic processing was involved) or during passive spoken Mandarin comprehension (in which common linguistic processing was engaged). The preprocessing strategy, nodes components and thresholds adopted were identical to the main analyses.

2. Results

2.1. Local activations

Using conventional activation analysis, we observed highly similar cortical activations in signers and non-signers in response to the signed sentences (see Fig. 2). Those activated regions included the inferior frontal gyrus, the inferior parietal, posterior temporal and occipital regions and the cerebellum in both hemisphere, as well as several sub-cortical regions including the putamen and thalamus. For the between group contrast, no region survived multiple comparison corrections (p < 0.05, FDR corrected). However, at a more lenient threshold of p < 0.005 without correction, we observed enhanced activation in focal areas within the left superior/middle temporal gyrus, left precentral gyrus and the bilateral supplementary motor area for signer group than for the non-signer group (see Table 1 in supplementary material)

Fig. 2.

Cortical activations in the hearing signer group and the non-signer group during sign language observation relative to the baseline. Threshold: p < 0.05, FDR corrected.

Table 1.

Regions of interest used to define nodes in the network analysis, and their corresponding module assignment.

	MNI coordinate	Module Assignment
		Signer	Non-signer
Occipital_Mid_L	−24, −91, 13	Module 1	Module 1
Fusiform_L	−42, −55, −17	Module 1	Module 1
Cerebelum_6_R	21, −67, −20	Module 1	Module 1
Occipital_Mid_L	−33, −88, −2	Module 1	Module 1
Temporal_Mid_R	48, −73, 4	Module 1	Module 1
Temporal_Sup_L	−54, −1, −8	Module 1	Module 1
Temporal_Mid_L	−45, −67, 10	Module 1	Module 1
Putamen_L	−21, 2, 10	Module 1	Module 1
Putamen_R	21, 5, 10	Module 1	Module 1
Thalamus_R	18, −16, 7	Module 1	Module 1
Temporal_Inf_R	48, −58, −8	Module 1	Module 1
Temporal_Mid_R	57, −58, 7	Module 1	Module 1
Occipital_Inf_L	−39, −73, −11	Module 1	Module 1
Temporal_Sup_L	−57, −46, 16	Module 1	Module 1
Frontal_Inf_Tri_R	33, 29, 1	Module 1	Module 1
Frontal_Inf_Tri_R	51, 20, 1	Module 1	Module 1
vFrontal_Inf_Oper_L	−51, 11, 4	Module 1	Module 1
Parietal_Inf_R	33, −43, 52	Module 2	Module 2
Precentral_R	45, 5, 34	Module 2	Module 2
Frontal_Mid_R	42, −4, 52	Module 2	Module 2
Frontal_Mid_L	−24, −4, 49	Module 2	Module 2
Precentral_L	−39, −4, 55	Module 2	Module 2
Supp_Motor_Area_L	−9, 11, 52	Module 2	Module 2
Supp_Motor_Area_L	−12, 2, 64	Module 2	Module 2
Precentral_L	−18, −16, 70	Module 2	Module 2
Parietal_Inf_L	−36, −43, 49	Module 2	Module 2
Parietal_Sup_L	−24, −61, 55	Module 2	Module 2
SupraMarginal_L	−57, −22, 34	Module 2	Module 2
dFrontal_Inf_Oper_L	−48, 8, 25	Module 3	Module 2
Frontal_Inf_Tri_R	51, 14, 22	Module 3	Module 2
Frontal_Inf_Tri_L	−51, 26, 10	Module 3	Module 1
Frontal_Inf_Tri_L	−39, 29, 19	Module 3	Module 2
Occipital_Sup_L	−24, −76, 37	Module 3	Module 2

2.2. Network topologies

2.2.1. Global network properties

Networks consisting of regions that were activated to the same extent in the signer and the non-signer groups were subjected to graph theoretical analysis. In both groups, the networks displayed significant small-world (expressed by σ > 1) and modular (expressed by z-score of modularity >2.58) organization. There was no significant group difference in global network efficiency at any point of the threshold range (0.3 ≤ T ≤ 0.6). However, the network engaged by signers exhibited significantly higher local efficiency (for 0.3 ≤ T ≤ 0.34 and 0.4 ≤ T ≤ 0.48), modularity (for 0.3 ≤ T ≤ 0.6) and small-worldness (for 0.3 ≤ T ≤ 0.6) than that engaged by non-signers. T-tests on the sparsity-integrated measures revealed similar results, with the network engaged by signers showing significantly higher local efficiency [${\int }_{0.3}^{0.6}{\mathrm{E}}_{(\text{local})}$, t₍₂₄₎ = 2.379, p = 0.026, Cohen’s d = 0.933], modularity [${\int }_{0.3}^{0.6}\mathrm{Q}$, t₍₂₄₎ = 3.513, p = 0.002, Cohen’s d = 1.378] and small-worldness [${\int }_{0.3}^{0.6}\sigma$, t₍₂₄₎ = 2.6042, p = 0.016, Cohen’s d = 1.022], while the network global efficiencies(${\int }_{0.3}^{0.6}{\mathrm{E}}_{(\text{glob})}$) did not differ between the two groups (t₍₂₄₎ = −0.255, p = 0.801, Cohen’s d = 0.100). See Fig. 3 for a summary of these findings. The results of permutation test were highly consistent with these derived from the t-tests (see supplementary information).

Fig. 3.

Between-group comparisons in graph properties including local efficiency (A), global efficiency (B), modularity (C) and small-worldness (D). The left column: results for the graph properties that were obtained over a range of thresholds (0.3–0.6). Right column: results for the sparsity-integrated graph properties.

2.2.2. Network modules

The modularity analysis was further performed in group-level networks to determine the modular structures in signers and non-signers. Fig. 4(left) shows the module assignments over a range of thresholds. We identified a modular partition that showed the highest similarity with other modular partitions obtained across the thresholds as the representative modular structure of the network. In the signer group, module partitions obtained at thresholds ranging from 0.36 to 0.46 had identical nodal assignments, and showed the highest similarity (NMI = 0.84) with other modular partitions across the threshold range. This representative modular structure consisted of three modules. The first module was located in frontal and parietal cortices, including bilateral middle frontal gyrus, bilateral precentral gyrus, left supplementary motor area, bilateral inferior and superior parietal lobule, and supramarginal gyrus. The second module was composed of anatomically distributed regions, including bilateral inferior frontal gyrus, inferior and middle temporal regions, occipital regions, putamen, right thalamus and right cerebellum. The third module consisted of the bilateral pars triangularis in the inferior frontal gyrus and the left superior occipital gyrus. For the non-signer group, the representative modular structure was the partitions obtained at thresholds ranging from 0.48 to 0.60, with a NMI value of 0.84. This representative modular structure consisted of two modules that have very similar nodal assignment to the first two modules in the signer group. Fig. 4(right) shows the representative modular structures mapped onto the brain surface for the signer and non-signer groups.

Fig. 4.

Group–wise modular structure. Left: module assignments in signer and non-signer groups over a set of sparsity thresholds (0.3–0.6). Right: representative modules mapped onto the brain surface. The representative modules for signer and non-signer groups corresponded to the partitions obtained at the sparsity thresholds ranging from 0.36 to 0.46 and ranging from 0.48 to 0.6 for each group, respectively. Note: nodes with larger size signify network hubs. The numeric coding for modules is in agreement with that in Table 1. The 3D surface visualizations of the results were implemented using the BrainNet Viewer (www.nitrc.org/projects/bnv) (Xia, Wang, & He, 2013).

2.2.3. Network hubs

Based on the group-mean nodal degree, efficiency, and betweenness, we identified hubs in networks engaged by signers and non-signers separately. In the signer group, network hubs were located in the left middle temporal gyrus (MNI coordinates for the center: −24, −91, 13), the left superior occipital gyrus (MNI coordinates for the center: −24, −76, 37) and the left ventral pars opercularis (MNI coordinates for the center: −51, 11, 4). In the non-signer group, the left middle temporal gyrus and the left superior occipital gyrus also served as hubs, and an additional hub was located in the right precentral gyrus (MNI coordinates for the center: 45, 5, 34). The hub regions are illustrated in Fig. 4, highlighted in larger size. Fig.5 plots the node-specific values in efficiency, betweenness, and degree. For the convenience of visualization, raw scores for each nodal property were transformed into z scores. The z score was calculated as (node_i – node_m)/node_std, where node_i was the degree (efficiency or betweenness) of node i, and node_m and node_std were the mean and standard deviation of degree (efficiency or betweenness) across all nodes within the network.

Fig. 5.

Node-specific values in efficiency, degree and betweenness. The nodes were sorted by efficiency in descending order for each group. The top three nodes in each plot were regions identified as hubs. For the convenience of visualization, the raw values for each nodal property were transformed into z scores.

2.2.4. Node-specific analysis

When FDR correction for multiple comparisons was applied (p < 0.05), there was no node showing significant group differences in node-specific properties. We then performed a targeted analysis for the left ventral pars opercularis, which was identified as a hub in the signer group but not in the non-signer group in the above analysis. One-tailed t-test showed that the left ventral pars opercularis in the signer group presented significantly higher nodal betweenness (t₍₂₄₎ = 1.790, p = 0.043, Cohen’s d = 0.702) and a tendency of higher nodal degree (t₍₂₄₎ = 1.405, p = 0.086, Cohen’s d = 0.551) and higher nodal efficiency (t₍₂₄₎ = 1.489, p = 0.075, Cohen’s d = 0.584) than that in the non-signer group. These results may suggest that, compared with the non-signer group, the left ventral pars opercularis in the signer group tends to have more connections with other regions in the network, and makes a greater contribution to facilitating communication among other regions, but these post hoc findings will need future replication.

2.3. Results of control analyses

During the baseline condition where no linguistic processing was involved, there was no significant group differences (p < 0.05, two-tailed t-test) in either network local efficiency, global efficiency, modularity, or small-worldness at any point of the pre-selected sparsity thresholds. In addition, in the spoken Mandarin comprehension condition, where common linguistic processing was engaged, no significant group difference was found for the above network properties at any point of the pre-selected sparsity thresholds. The absence of between-group differences for the two control conditions indicates that the differences in network configuration between signers and non-signers when viewing signed sentences were generated by the different processing they engaged (linguistic processing versus gestural action perception).

2.4. Results of validation analyses

Given that weighted matrix carries different information about network organization (Rubinov and Sporns, 2010), we re-performed GTA on weighted networks to assess the reliability of the main results based on binary networks. With one-tailed t-tests, we found that the network engaged by signers displayed significantly higher modularity (${\int }_{0.3}^{0.6}\mathrm{Q}$, t₍₂₄₎ = 2.185, p = 0.019, Cohen’s d = 0.857), small-worldness (${\int }_{0.3}^{0.6}\sigma$, t₍₂₄₎ = 2.386, p = 0.013, Cohen’s d = 0.936) and local efficiency (${\int }_{0.3}^{0.6}{\mathrm{E}}_{(\text{local})}$, t₍₂₄₎ = 1.906, p = 0.034, Cohen’s d = 0.748) than non-signers. However, we also found significantly higher global efficiencies (${\int }_{0.3}^{0.6}{\mathrm{E}}_{(\text{glob})}$, t₍₂₄₎ = 1.946, p = 0.032, Cohen’s d = 0.763) in the network engaged by signers than non-signers, and this effect was not found in the main analyses. This finding may suggest that taking into account the strength of functional connectivity in GTA can improve the sensitivity of measurement.

For the network constructed based on shorter time courses, similar results as the main analyses were obtained. The network engaged by signer group exhibited significantly higher local efficiency [ ${\int }_{0.3}^{0.6}{\mathrm{E}}_{(\text{local})}$, t₍₂₄₎ = 2.838, p = 0.009, Cohen’s d = 1.113], modularity [${\int }_{0.3}^{0.6}\mathrm{Q}$, t₍₂₄₎ = 4.316, p< 0.001, Cohen’s d = 1.693] and small-worldness [${\int }_{0.3}^{0.6}\sigma$, t₍₂₄₎ = 4.176, p < 0.001, Cohen’s d = 1.217] than that engaged by non-signers, while the network global efficiencies ((${\int }_{0.3}^{0.6}{\mathrm{E}}_{(\text{glob})}$) did not differ between the two groups (t₍₂₄₎ = 1.227, p = 0.232, Cohen’s d = 0.481). The hub analysis also yielded the same pattern as the main analyses, with the left ventral pars opercularis being a hub in the signer group but not in the non-signer group. Together, these results demonstrated the reliability of our main findings.

3. Discussion

In this study, we investigated how the large-scale functional brain network is organized in hearing signers in contrast to hearing non-signers when viewing sign language. Using conventional activation analysis, we observed that the sign language elicited highly similar activation patterns in signers and non-signers, with focal differential activations within the left frontal and temporal regions. Next, GTA revealed that the overlapped activation network was differentially organized in the two groups. Specifically, at the global level, the network engaged by signers presented higher local efficiency, small-worldness and modularity, and exhibited different modular structure as compared to the network engaged by non-signers. At the regional level, the left ventral pars opercularis served as a hub in the network engaged by the signer group, but not in the non-signer group. Implications of these findings are discussed below.

3.1. Similar cortical activations in signers and non-signers

The activation analysis revealed highly similar activation patterns in signers and non-signers in response to sign language relative to the baseline. Regions commonly activated in the two groups included the inferior parietal, posterior temporal, occipital regions and the cerebellum, which are implicated in visual-spatial encoding of moving stimuli. More interestingly, a part of the classical language areas including the left inferior opercularis and middle temporal gyrus were also activated in both groups. These findings are in line with previous studies (Andric et al., 2013; Courtin et al., 2011; Levanen et al., 2001; Xu et al., 2009), suggesting anatomically shared neural substrates for sign language comprehension and gestural action perception and supporting a tight link between the language and action systems (Arbib, 2005). With a lenient threshold, we observed that focal regions within the perisylvian cortex as well as the bilateral supplementary motor area were more strongly activated in signers than in non-signers. These results are largely consistent with previous findings (Levanen et al., 2001; Newman et al., 2015), though the between-group effect is weaker in our study. This weaker between-group effect might be attributed to the fact that in the present study hearing non-native signers (bimodal bilinguals) are recruited. Native deaf signers recruited in previous studies have sign language as their dominant language and are more fluent in sign language than bimodal bilinguals. Thus, native deaf signers may have a much stronger activation network and differ more greatly from non-signers than that of bimodal bilinguals. Another possible reason for stronger between-group activation difference revealed in previous studies than in this study is that, high baselines (e.g. backward-played video stimuli in Newman et al., 2015) were adopted in previous studies, which might be more sensitive to detect focal differential brain activations associated linguistic processing. In contrast, in the current study, only a low-level baseline was adopted and the activation pattern due to biological motion perception or other non-linguistic aspects of the stimuli was not controlled. Therefore, the effects of linguistic processing might be small compared to the overall strength of activation, and could be masked in particular in a between-subjects design.

3.2. Different network topology between signers and non-signers

While sign language elicited highly similar cortical activations in signers and non-signers, these activated regions were organized differently. The functional network consisting of the commonly activated regions presented higher small-worldness and modularity in signers than in non-signers. Crucially, both small-world and modularity topologies are thought to reflect optimal network configuration (Pan and Sinha, 2007; Rubinov and Sporns, 2010). The small-world topology features higher local clustering coefficient than random networks, yet comparable characteristic path length as random networks, reflecting an optimal balance between functional segregation and integration (Rubinov and Sporns, 2010). The modularity topology is characterized by the presence of intensive intramodular connections and sparse intermodular connections. Such a configuration allows for efficient local processing while reducing interdependence of modules, which leads to enhanced robustness and specificity (Bassett and Gazzaniga, 2011). No previous study has established the significance of small-worldness or modularity property of functional brain network for linguistic processing compared with other cognitive processes. Nonetheless, evidence from other domains shows that a brain network with higher small-worldness and modularity is associated with cognitive advantages. For example, greater network small-worldness and modularity are associated with better short-term memory capacity (Stevens et al., 2012). There has also been evidence for aging-related and disease-related decreases in network small-worldness and modularity (He et al., 2009; Onoda and Yamaguchi, 2013). In the context of this study, we assume that the signed sentences evoke automatic linguistic operations in the signer group, such as to unify the movements of hands, faces and other parts of body into larger units and to map them onto lexical and syntactic representations according to grammatical rules. These computations may demand a high level of both local specificity and global integration in the brain network of signers.

Yet, more investigation is required to identify how different components of linguistic processing produce specific network configuration. The different network topology in the signer group compared to the non-signer group, together with their shared cortical activation, supports the idea that high-level cognitive systems could emerge from a fixed anatomy via a reconfiguration of its connections (Park and Friston, 2013). Whether changes in functional connectivity in the brain of signers are preceded by changes in structural connectivity remains an open question.

3.3. A unique module in signers and the modules shared between signers and non-signers

Based on the group-level modular analysis, we observed two modules (sub-networks) that were similar in the networks engaged by signers and non-signers. The first module was consisting of the bilateral middle frontal gyrus, bilateral precentral gyrus, left supplementary motor area, bilateral inferior and superior parietal lobule, and supramarginal gyrus. This sub-network corresponds well to the frontal-parietal mirror neuron system (Molnar-Szakacs et al., 2006; Rizzolatti and Craighero, 2004) and is probably involved in the representation of complex action sequences. Another module was composed of anatomically distributed regions, including the bilateral inferior frontal gyrus, inferior and middle temporal regions, occipital regions, putamen, right thalamus and right cerebellum. The role of this module is less clear. The inferior frontal gyrus, putamen, and cerebellum have been implicated in spoken language production (Price, 2012), while inferior and middle temporal regions are related to meaning processing (Price, 2012). The presence of this module may reflect the process of trying to derive the meaning in the sign language stimuli (rather than actually finding the correct meaning) and to translate signs into spoken words.

We further observed a unique module in the network engaged by signers, which had constant nodal assignment across the sparsity thresholds but not presented in the non-signer group. This module consisted of the bilateral pars triangularis, the left dorsal opercular of the inferior frontal gyrus, and the left superior occipital gyrus. The bilateral pars triangularis and dorsal opercular regions have long been revealed to play a crucial role in lexicosemantic integration (Price, 2012) and syntactic processing (Caplan, 2001; Caplan et al., 2000). The left superior occipital gyrus is usually engaged in tasks involving the process of visual motion (Emmorey et al., 2010; Sadato et al., 2005). The presence of this unique module in the signer group may highlights the interaction between high-level linguistic areas and visual-motion perception area for sign language comprehension.

To summarize, the presence of similar functional modules may be related to visual-motor representation of signs and the processing for communicative intention, which are shared by sign language comprehension and gestural action observation. The presence of the unique module may be involved in lexicosemantic integration and syntactic processing, which are specific to sign language comprehension.

3.4. Differential role of the left ventral pars opercularis for signers and non-signers

The left ventral pars opercularis was identified as a hub in the network engaged by signers but a periphery node in the network of non-signers. In graph theory, hubs are proposed to play a crucial role in integrating information and coordinating the communication across different subsystems (van den Heuvel and Sporns, 2013a, b, c). Lesions to hub nodes could significantly disrupt modularity structure (van den Heuvel and Sporns, 2013a, b, c), reduce network efficiency (Hwang et al., 2013), and have pronounced effects on behavioral performance (Liu et al., 2014; Merkley et al., 2013; Pandit et al., 2013). The dissociable role of the left ventral pars opercularis for language and gestural action processing revealed by GTA is in concordance with neuropsychological studies showing that patients with lesion to the Broca’s area suffer language loss but preserve action function (Corina et al., 1992; Goschke et al., 2001; Kean, 1977), though the exact locations between our study and the previous ones may differ. Two recent studies applying GTA explored semantic networks of spoken language and identified the left triangular of the inferior frontal gyrus as one of their network hubs (Vandenberghe et al., 2013; Xu et al., 2016). However, since the left ventral pars opercularis was not included in their analyses, it remains unknown about whether this region plays a similar role for sign and spoken languages. The homogeneity and heterogeneity between the functional networks underlying sign language and spoken language should be further investigated. It is worth noting that, unlike the left ventral pars opercularis, the left dorsal pars opercularis of the inferior frontal gyrus did not show significant group differences in nodal degree or nodal efficiency. This region displayed relatively high nodal degree, efficiency and betweenness in the networks of both groups (see fig.5), suggesting that it might be equally important for sign language and gestural action processing. The dissociable roles of left ventral pars opercularis and dorsal pars opercularis in sign language and gestural action processing provide novel evidence supporting the functional segregation within the left pars opercularis of the inferior frontal gyrus (Fedorenko et al., 2012; Molnar-Szakacs et al., 2005).

3.5. Limitations

Several limitations of this study should be noted. First, participants recruited in this study are hearing bimodal bilinguals. While the recruitment of bimodal bilinguals allows us to match the hearing status as well as other factors such as the level of education and native language background between signers and non-signers, bilingualism per se could introduce potential confounding effect on our results. Indeed, previous studies have shown that second language experience could produce changes in brain functional connectivity (Li et al., 2015; Zou et al., 2012), though neural plasticity expressed at the complex network level is still undetermined. Thus, it is possible that the changes in network configuration we observed in the signer group relative to the non-signer group are joint effects of linguistic processing and neural plasticity associated with bilingualism. However, considering that signers and non-signers did not differ in network topologies either when engaging no linguistic processing (the baseline condition), or engaging common linguistic processing (the condition of spoken Mandarin comprehension), we infer that the between-group differences revealed in the main analysis are dominated by the linguistic effect of sign comprehension versus gestural action perception. Nevertheless, the addition of a bilingual control group that is naive to signing, and the addition of non-hearing monolingual signers, are required to tease apart the effect of bilingualism from linguistic effect. Secondly, the surrogate group studied reported that they were aware that the videos of signing contained information and that they attempted to extract information while viewing signs. This would suggest that both signer and non-signer groups were engaging in linguistic processes. Therefore, the differences in network topology between groups may reflect the degree of linguistic processing, rather than pure linguistic versus non-linguistic processing. Besides, since both groups were trying to comprehend signing, but at different skill levels, the between-group differences in network topology might also reflect effort-related effects. Third, by applying the conjunction analysis to define the network nodes, brain regions with differential activation were excluded priorly. While guaranteeing that the nodes were unbiasedly chosen for signers and non-signers, this approach risks missing regions which might carry important information differentiating networks subserving the processing of sign-language versus gestural action observation. Richer information will be obtained by including a more complete set of relevant regions into the network analysis. Finally, we removed weak, spurious connections at individual level, and set the number of network nodes and connection density identical across all subjects. While this approach can eliminate the effect of network size and density, it may lead to a modification of the network by ignoring significant connections (Van Wijk et al., 2010). Further study adopting a different thresholding strategy is needed to validate our findings.

4. Conclusions

The present study revealed that hearing signers and non-signers presented similar cortical activations when viewing sign language. However, the commonly activated network was differently organized in the two groups. Specifically, the network engaged by signers displayed a higher degree of small-worldness and modularity than that of non-signers, with the left ventral pars opercularis playing a central role in the network. Our study suggests that while a shared anatomical network is engaged by comprehension of sign language and observation of gestural action, this network is differently configured for the two types of processing. Our study also shows that GTA can provide an important complementary perspective to the activation analysis on the neural basis underlying cognition.

5. Experimental procedure

5.1. Participants

Fourteen hearing signers (3 males, aged 33–65 years old, mean age = 49 years) and fifteen hearing non-signers (3 males, aged 31–67 years old, mean age = 48 years) took part in the experiment. The signers were spoken-sign bimodal bilinguals who taught CSL at schools for the deaf. They acquired CSL, on average, at the age of 19, and they were highly proficient in sign language. They used CSL for at least 3.3 h each day and had a mean CSL experience of 30 years. In addition, a self-rating scale of 1–5 was administered to assess sign language proficiency, with 5 signifying highly proficient. The mean scores of the hearing signer group were 4.5 (standard deviation = 0.52). The non-signer group was monolingual speakers who had no knowledge of a sign language. They were administrative staff of Beijing Normal University. Both groups acquired Mandarin as their native language. The two groups were matched in age (t₍₂₇₎ = 0.204, p = 0.84), and education level (t₍₂₇₎ = 0.144, p = 0.89). No participants reported history of neurological or psychiatric disorders.

5.2. Stimuli and experimental design

Twenty short declarative sentences produced by a deaf native CSL signer were used in this study (see Fig. 1 and supplementary material). Hand movements and facial articulations required by CSL were involved in these signed sentences. There were four task blocks alternating with four baseline blocks, each block lasting about 30 s. During each task block, a silent video of five signed sentences was presented. Participants were told to watch and comprehend these signed sentences, and no explicit response was required. During the baseline blocks, videos showing the same CSL model standing still were presented. Note that using such a low-level baseline can avoid “washing out” domain-general regions and regions supporting sensory-perception. Those regions might constitute a periphery of language network (Fedorenko and Thompson-Schill, 2014). The presentation of experimental stimuli was fixed across participants. The complete scanning session included another two experiments which involved passive spoken language listening and passive written sentence viewing. These two experiments were not reported in this study. After the scanning session, participants were given an unexpected recognition test, where they were asked to indicate how familiar a signed sentence was on a 4-point scale, with 1 as definitely new and 4 as definitely old. Our analysis showed the signer group scored significantly higher than the non-signer group (M_(signer) = 2.85, SD = 1.18; M _(non-signer) = 1.68, SD = 1.12; p < 0.05).

Fig. 1.

Stimuli and experimental design. Both hearing signers and non-signers viewed silent videos showing a native deaf signer producing signed sentences during the task phase and standing still during the baseline phase. No explicit response was required.

To gain insights into the mental processes in hearing non-signers when viewing sign language, we conducted one additional post hoc experiment. A new group of hearing non-signers (including 10 college students and 4 people aged above 50 years) were recruited to view the same videos as used in the above experiment, and then we carried out a short interview about their experience viewing the sign language videos. These participants reported that their attention was primarily focused on the movement of hands and secondly the movement of lip of the signer. They thought that the gestures in the video were communicative rather than meaningless. They attempted to extract the meaning conveyed by the signer but failed. Given that the reports were highly consistent across the 14 participants, we assumed that similar mental processes could be involved for those participants in the fMRI experiment.

5.3. Image acquisition

Scans were acquired with a 3T Siemens Trio Scanner at the MRI Center of the Beijing Normal University. For functional scans, a gradient echo planar imaging (EPI) sequence was applied with the following parameters: time repetition = 2000 ms, time echo = 30 ms, flip angle = 90°, FOV = 200 mm, matrix size = 64 × 64, 32 interleaved slices per volume with slice thickness = 4.8 mm, and voxel size = 3.12 × 3.12 × 4.8 mm. Parameters for anatomical images were:MPRAGE sequence, time repetition = 2530 ms, time echo = 3.39 ms, flip = 7°, FOV = 256 mm, scan order = interleaved, matrix size = 256 × 256, slice thickness = 1.33 mm, and voxel size = 1.0 × 1.0 × 1.33 mm.

5.4. Image preprocessing

Image preprocessing was conducted using SPM8 (statistical parametric mapping) (www.fil.ion.ucl.ac.uk/spm/). First, slice-timing correction was performed to correct for varied sampling time of slices, with the middle slice in time being used as a reference slice. Second, all functional images were spatially realigned and co-registered to their corresponding anatomical images. The resultant images were then spatially normalized to Montreal Neurological Institute (MNI) space. After normalization, all images were resampled into 3 × 3 × 3 mm voxel size, and were further spatially smoothed using a Gaussian kernel with 8 mm full-width at half maximum (FWHM). The dataset of one non-signer and one signer were deleted for excessive head motion (>3 mm or 3 degree).

5.5. Activation analysis

An activation analysis was performed to locate cortical regions that were engaged when signers and non-signers viewed the signed sentences. The effect of task versus baseline was first assessed for individual participants, using a general linear model (GLM) by convolving the design matrix with the canonical hemodynamic response function, with six motion parameters regressed out. Next, a second level analysis was carried out to assess the group mean of brain activation. A one-sample t-test was conducted for the signer group and non-signer group separately to identify regions significantly activated in the CSL task relative to baseline. Then a two-sample t-test was used to examine to what extent the two groups differed in cortical activations.

5.6. Graph theoretical analysis

5.6.1. Node definition

To avoid potential bias caused by group differences in regional activation, we confined the network node definition to brain regions that showed comparable activations between the signers and non-signers. For this purpose, we performed a conjunction analysis for the effect of task relative to baseline in signers and non-signers using the SPM8, with the “conjunction null hypothesis”. This approach identified cortical regions that were activated in both groups and excluded regions for which activation differed significantly between the two groups (Price and Friston, 1997). For the conjunction analysis, we applied a relatively lenient threshold, with p < 0.005 at voxel level combined with a cluster size of >20 voxels. Then, local maxima that were located at least 15 mm apart from each other were extracted from the conjunction map, and spheres with a radius of 5 mm centered on each local maxima were drawn (Vandenberghe et al., 2013). We thus identified 33 regions (nodes) of interest (see Table 1).

5.6.2. Network construction

To calculate functional connectivity, three additional processes were performed on the pre-processed datasets: (1) high-pass filtering with a cutoff of 1/128 Hz; (2) removal of linear trends; and (3) regression to remove potential sources of head motion. Mean nodal BOLD time series from the task blocks were extracted (shifting 6 s to account for the hemodynamic lag) (Aguirre et al., 1998). The time series in discontinuous task blocks were normalized within blocks, with a mean of zero and a deviation of 1, and were then concatenated (Ekman et al., 2012), yielding a total of 57 time points. While concatenating data from different blocks could cause discontinuities in the time series, a previous study on the “resting state” functional connectivity suggests that the connectivity pattern obtained from concatenated data are similar with that of continuous data (Fair et al., 2007). For each participant, Pearson’s correlation coefficients were calculated for every possible pair of time series. The resultant correlation matrices were thresholded to generate binary brain graphs, using a set of sparsity thresholds ranging from 0.3 to 0.6 with a step of 0.02 (0.3 ≤ T ≤ 0.6), where sparsity is defined as the proportion of actual number of edges to the maximum possible number of edges in a network. The lowest threshold (T = 0.3) was determined to ensure that the resultant networks were not severely fragmented: on average across all participants, 98.48% of the nodes in the network were connected with other nodes by direct or indirect paths. The highest threshold (T = 0.6) was set to remove weak connections: for each participant, all possible connections in the correlation matrix were subjected to a t-test, and only connections that were significantly stronger than zero (p < 0.05) were retained (Liang et al., 2015). We then computed the network sparsity of each participant and set the mean of network sparsity across all participants as the highest threshold. As most graph theoretic measures are contingent on the number of nodes and connections of a graph, we set an equal number of nodes and sparsity between signers and non-signers to make their network topologies comparable (Fornito et al., 2013)

5.6.3. Network measures

The graph theoretical analysis was performed using the GRETNA toolbox (Graph theoretical network analysis: htpp://www.nitrc.org/projects/gretna) (Wang et al., 2015). For each participant, we calculated graph properties characterizing the global-level network organization, including global and local efficiency, small-worldness, and modularity. We also examined the graph properties of each region (node), including nodal degree, nodal betweenness and nodal efficiency.

Global network properties

Global network efficiency (E_glob): the global network efficiency measures how efficiently information transmits across the global network, which is computed as

$${\mathrm{E}}_{\text{glob}}(\mathrm{G})=\frac{1}{\mathrm{N}(\mathrm{N}-1)}\sum _{\mathrm{i}\ne \mathrm{j}\in \mathrm{G}}\frac{1}{{\mathrm{d}}_{\text{ij}}}$$ (1)

where N is the total number of nodes in network G, and d_ij the shortest path length between node i and node j (Latora and Marchiori, 2001).

Local network efficiency (E_loc): local network efficiency shows how efficient the communication is among the neighbors of each node. In a network with high local efficiency, nodes tend to cluster together to form connected local structures. Local efficiency is computed as

$${\mathrm{E}}_{\text{loc}}(\mathrm{G})=\frac{1}{\mathrm{N}}\sum _{\mathrm{i}\in \mathrm{G}}{\mathrm{E}}_{\text{glob}}({\mathrm{G}}_{\mathrm{i}})$$ (2)

where G_i is the subgraphs (neighbors) of node i, E_glob (G_i) is the global efficiency of G _i (Latora and Marchiori, 2001).

Small-worldness (σ): Small-world networks are characterized by higher local clustering coefficient than random networks, yet comparable characteristic path length as random networks (Watts and Strogatz, 1998). The small-worldness of a network can be measured as:

$$\sigma =\frac{\mathrm{C}∕{\mathrm{C}}_{\text{rand}}}{\mathrm{L}∕{\mathrm{L}}_{\text{rand}}}$$ (3)

where C and C _rand are the clustering coefficients, and L and L_rand are the characteristic path lengths of the tested and the random networks respectively. In this study, 1000 equivalent random graphs with the same number of nodes and edges and the same degree distribution as the real network were sampled. A network with σ > 1 is generally accepted as ‘small-world’ (Sporns and Honey, 2006).

Modularity (Q): Modularity quantifies the degree to which a network can be partitioned into densely connected subgroups, with only sparse connections between subgroups. Unlike most other network measures, modularity is typically assessed with optimization algorithms, rather than with exact computations (Danon et al., 2005; Rubinov and Sporns, 2010). Here, we used the modified greedy optimization algorithm to identify modules in the functional network that optimize the modularity value (Clauset et al., 2004; Danon et al., 2006; Guimera and Sales-Pardo, 2006; Newman and Girvan, 2004). For a given partition p, the modularity is calculated as:

$$\mathrm{Q}(\mathrm{p})=\sum _{\mathrm{s}=1}^{\mathrm{N}}\left[\frac{{\mathrm{l}}_{\mathrm{s}}}{\mathrm{L}}-{\left(\frac{{\mathrm{d}}_{\mathrm{s}}}{2\mathrm{L}}\right)}^2\right]$$ (4)

where N is the number of modules, L is the sum of connections in the network, l_s is the number of connections in module s, and d_s is the sum of the node degrees in module s (Chen et al., 2008). To test whether the observed modular structure arises from random interactions, we calculated the z score of the maximum modularity as (Q_real – Q_rand)/Q_std, where Q_real is the maximum modularity of the brain network, and Q_rand and Q_std are the mean and the standard deviations of the maximum modularity of 1000 randomized networks (Chen et al., 2008). The randomized networks had the same number of nodes and edges and the same degree distribution as the real network.

The modularity analysis was first conducted for each individual network. Considering the between-participants variance in module assignment and module numbers, we also conducted the modularity analysis at the group level to determine modular structures in the signer and the non-signer groups (Liang et al., 2015). To obtain the group-level brain networks, we first averaged all connectivity matrices across participants in each group and then binarized the group-mean matrices using the pre-selected sparsity thresholds (0.3 ≤ T ≤ 0.6). After the group-level modularity analysis, we calculated the similarity between modular partitions across thresholds using normalized mutual information (NMI) (Danon et al., 2005). The value of NMI ranges from 0 to 1, where 0 signifies that the two modular partitions are totally independent, and 1 signifies that they are identical. For the modular partition obtained at each threshold, we computed the averaged NMI of this modular partition with any other modular partitions obtained across the threshold range. Then, the modular partition with the highest NMI was defined as the representative modular structure of the network.

Regional nodal properties

Nodal degree (d_i): for a network G with N nodes, the degree for node i is defined as the sum of the edges connected to it.

$$d_i=\sum _{j\in N}^N{d}_{ij}$$ (5)

Nodal efficiency (e_i): Nodal efficiency is defined as the shortest path length between a given node i and other nodes in the network.

$$e_i=\frac{1}{N-1}\sum _{i\ne j}\frac{1}{d_{ij}}$$ (6)

Nodal betweenness (b_i): Nodal betweenness is defined as the fraction of the shortest paths between any pair of nodes that travel through the node.

$$b_i=\frac{1}{(n-1)(n-2)}\sum _{k,j,i\in N,k\ne j\ne i}\frac{g_{jk}(i)}{g_{jk}}$$ (7)

where g_jk is the number of shortest paths between node j and k, and g_jk(i) is the number of shortest paths between j and k that pass through i. A higher nodal betweenness indicates greater contribution to facilitating the communication between other regions.

Identification of hubs

Efficient communication and integration across distributed regions are enabled by a set of specific regions that serve as network hubs (van den Heuvel and Sporns, 2013a,b,c). Typically, network hubs are characterized by high degree, efficiency and betweenness (Rubinov and Sporns, 2010). In this study, hubs were identified by the following procedures outlined in (van den Heuvel et al., 2010). First, node-specific degree, efficiency and betweenness were calculated for each participant, and then these values were averaged across all participants in each group. Next, all nodes were sorted according to their values in the group-mean nodal degree, nodal efficiency and nodal betweenness, respectively. Finally, nodes that fulfilled two of the following criteria were identified as hubs: (1) those belonging to the top 10% of nodes showing the highest degree; (2) those belonging to the top 10% of nodes showing the highest betweenness; or (3) those belonging to the top 10% of nodes showing the highest global efficiency.

5.6.4. Statistical analysis

We tested the null hypothesis of no difference between signers and non-signers in any measures of global or regional network properties. The global-level network properties including local efficiency, global efficiency, small-worldness and modularity were first tested over a range of sparsity values (0.3 ≤ T ≤ 0.6) (Fornito et al., 2013). Two-sample t-tests (two-tailed, p < 0.05) were applied to examine group differences in these sparsity-integrated measures, and Cohen’s d (Cohen, 1988) was calculated to measure effect size. Since tests conducted at neighboring sparsity are strongly dependent, we did not perform corrections for multiple tests at individual sparsity points (Fornito et al., 2013). Instead, integrated network measures over the sparsity range were estimated by calculating the area under the curve (AUC) and statistical inferences were further performed on the AUC. The AUC provides a summary measure that is independent of a single threshold, and avoids the need for multiple comparisons correction (Fornito et al., 2013). Given the exploratory nature of GTA, we also applied a nonparametric permutation test (N = 1000) to assess the statistical significance of between-group differences in global network properties (see supplementary material for details).

For node-specific analysis, statistical inferences were performed only on the sparsity-integrated measures, as ${\int }_{0.3}^{0.6}$ d_i for nodal degree, ${\int }_{0.3}^{0.6}$ b_i for nodal betweenness, and ${\int }_{0.3}^{0.6}$ e_i for nodal efficiency. A significant level of p < 0.05 (FDR corrected for multiple tests performed on 33 nodes) were used.

5.7. Control analyses

Two control analyses were performed to test the causal link between sign language processing and brain network organization. First, we examined the topological properties of the network in signers versus non-signers during the baseline phase, in which no linguistic processing was involved. Second, we examined the topological properties of the network in the same participant groups during passive spoken Mandarin comprehension, in which common linguistic processing was engaged. The preprocessing strategy, nodes components and thresholds adopted were identical to the main analyses. Then we tested group differences in overall graph properties including local network efficiency, global network efficiency, modularity, and small-worldness.

5.8. Validation analysis

We performed the validation analysis on weighted networks to assess the reliability of our main analysis. In this approach, the individual connectivity matrices were thresholded by the same set of sparsity thresholds as in the main analyses (0.3 ≤ T ≤ 0.6), and values below the threshold were set to zero, whereas values above the threshold kept their original values.

A previous study suggests that scan length can have an effect on the estimate of resting-state functional connectivity (Birn et al., 2013). However, no study has examined the vulnerability of task-state functional connectivity to scan length. In order to estimate the effect of scan length on our main results, we computed the functional connectivity matrices using the scans from the first three task blocks (45 volumes, with the last task block removed) and re-performed the network analysis.