Multimodal imaging of brain reorganization in hearing late learners of sign language

Abstract

The neural plasticity underlying language learning is a process rather than a single event. However, the dynamics of training‐induced brain reorganization have rarely been examined, especially using a multimodal magnetic resonance imaging approach, which allows us to study the relationship between functional and structural changes. We focus on sign language acquisition in hearing adults who underwent an 8‐month long course and five neuroimaging sessions. We assessed what neural changes occurred as participants learned a new language in a different modality—as reflected by task‐based activity, connectivity changes, and co‐occurring structural alterations. Major changes in the activity pattern appeared after just 3 months of learning, as indicated by increases in activation within the modality‐independent perisylvian language network, together with increased activation in modality‐dependent parieto‐occipital, visuospatial and motion‐sensitive regions. Despite further learning, no alterations in activation were detected during the following months. However, enhanced coupling between left‐lateralized occipital and inferior frontal regions was observed as the proficiency increased. Furthermore, an increase in gray matter volume was detected in the left inferior frontal gyrus which peaked at the end of learning. Overall, these results showed complexity and temporal distinctiveness of various aspects of brain reorganization associated with learning of new language in different sensory modality.

The current study focuses on sign language acquisition in hearing adults who underwent an 8‐month long training and five neuroimaging sessions. Using multimodal approach, we assessed what neural changes occurred over the course of learning—as reflected by task‐based activity, connectivity changes, and co‐occurring structural alterations. Overall, these results showed complexity and temporal distinctiveness of various aspects of brain reorganization associated with learning of new language in different sensory modality.

1. INTRODUCTION

Research in the last two decades has provided compelling evidence for a lifelong capacity of the human brain to alter in response to a changing environment. This phenomenon, referred to as neuroplasticity, was initially investigated in seminal studies that used in vivo brain imaging, such as magnetic resonance imaging (MRI), and cross‐sectional comparisons (e.g., Gaser & Schlaug, 2003; Jacini et al., 2009; Maguire et al., 2000). Today, the study of neuroplasticity has moved to documenting how reorganization occurs and what are its dynamics. Longitudinal, within‐participant paradigms in controlled learning settings not only allow for causal inferences about neural mechanisms linked to complex cognitive functions and experience‐induced plasticity, but also for delineating temporal properties of neural plasticity (Draganski, Kherif, & Lutti, 2014; Johansen‐Berg, 2012; Li, Legault, & Litcofsky, 2014; Lövdén, Wenger, Mårtensson, Lindenberger, & Bäckman, 2013; Valkanova, Rodriguez, & Ebmeier, 2014 for reviews).

One elegant example of neural and behavioral plasticity is learning a second language (L2), which requires high‐level cognitive resources and is usually effortful, especially when learning takes place in adulthood (Rodriguez‐Fornells, Cunillera, Mestres‐Misse, & de Diego‐Balaguer, 2009). Despite the fact that late bilingualism is fairly common, the dynamics of neural changes following L2 learning remain poorly understood. Prior findings indicate language proficiency as one of the critical factors modulating functional organization of L2 (e.g., Consonni et al., 2013; Del Maschio & Abutalebi, 2017; Stowe & Sabourin, 2005); however, only a few studies have aimed to explore the effects of proficiency longitudinally. For example, Stein et al. (2009) showed that increasing language proficiency after 5 months of learning German correlates negatively with the level of frontal activity (e.g., bilateral inferior frontal gyri [IFG]) during L2 word reading, which could be explained as less effortful processing for more proficient L2 readers. In contrast, Barbeau et al. (2017) reported increased activation in the left inferior parietal lobule in response to written stimuli after an intensive, 12‐week‐long French course. Thus, it appears that language learning may result in both increases and decreases in neural activation when comprehending a newly learned L2.

The longitudinal evidence for gray matter (GM) plasticity associated with language learning is also rather limited. Mårtensson et al. (2012) collected structural scans of interpreters before and after 3 months of a new language course (Russian, Arabic, or Dari). The authors reported increases in hippocampal volume and in cortical thickness of IFG, middle frontal, and superior temporal gyri (STG). Stein et al. (2010) examined a group of exchange students before and after 5 months of learning German. Immersive linguistic training resulted in a positive correlation between L2 proficiency and GM density in the IFG and anterior temporal lobe in the left hemisphere. Finally, 16 weeks of learning English resulted in an increase in GM volume (GMV) in the right IFG in a group of Japanese speakers (Hosoda, Tanaka, Nariai, Honda, & Hanakawa, 2013).

Learning a new language also requires engaging executive control functions (e.g., inhibiting L1; see Declerk & Philipp, 2015, for a recent review). Grant, Fang, and Li (2015) reported activation decreases in cognitive control areas (such as cingulate cortex) and stronger functional connectivity within the lexical reading network (e.g., IFG and middle temporal gyrus [MTG]), across two semesters of learning Spanish by native English speakers. Very recently, Legault, Grant, and Li (2019) extended this finding by examining structural changes in the same group of participants and within identical functionally connected areas reported by Grant et al. (2015). This study revealed greater cortical thickness in the right MTG and left cingulate cortex after the yearlong Spanish course. Moreover, Legault et al. (2019) reported a correlation between cortical thickness and cortical connectivity changes. This finding illustrates that a relation between cortical activity and morphological changes exists during L2 acquisition. However, the spatial and temporal correspondences between these different aspects of neuroplasticity remain unknown. A multimodal approach using multiple time points (TPs) could provide a more comprehensive view of the brain reorganization dynamics that support the acquisition of new linguistic skills.

Although acquisition of a new spoken language is a highly demanding and complex process, it engages the same perceptual and articulatory systems as those used for the native language of hearing speakers. However, meanings and linguistic structures can also be coded in manual movements through space and decoded through vision. Sign languages represent a special case of L2 learning—requiring not only acquisition of new vocabulary and linguistic rules, but also switching to a different sensory modality. Thus, these languages provide a particularly interesting framework for neuroplasticity research. Studies on bimodal bilingualism (the knowledge of both spoken and sign language) have identified remarkable parallels between spoken and signed languages, including left‐hemisphere lateralization and overlapping perisylvian neural circuits—such as IFG, which form a modality‐independent language system (for reviews, see Emmorey, Giezen, & Gollan, 2016; Poeppel, Emmorey, Hickok, & Pylkkanen, 2012). Perceptual and articulatory differences are reflected in a greater degree of neural activity in superior/inferior parietal lobe (SPL) and occipital cortex during sign language processing compared to spoken language processing; these differences are most likely due to the visual–spatial and manual articulation demands of signing (e.g., Corina, Lawyer, & Cates, 2013; Emmorey et al., 2016; Emmorey, McCullough, Mehta, & Grabowski, 2014; Johnson et al., 2018; MacSweeney et al., 2002; MacSweeney, Capek, Campbell, & Woll, 2008; Sakai, Tatsuno, Suzuki, Kimura, & Ichida, 2005). So far, only one study has examined brain reorganization following acquisition of a sign language by hearing adults (Williams, Darcy, & Newman, 2016). In this study, brain activity to American Sign Language (ASL) was measured at three TPs: at the beginning of learning (TP0), after 5 months (TP1, ~44 hr of instruction), and after 10 months of ASL instruction (TP2, ~89 hr of instruction). In the scanner, participants (12 native English speakers) decided whether single signs were produced near the face or the body area. The results suggested enhanced activation in modality‐independent (linguistic) areas including bilateral supramarginal gyri after 5 months, with an additional increase in neural activity in left inferior frontal gyrus after 10 months from the onset of learning. However, as the task required simple spatial decisions, it is plausible that linguistic processing occurred implicitly, and a clear separation between spatial attention and linguistic analysis is problematic. Therefore, further research with a larger sample size of L2 learners that explicitly tests comprehension of more complex linguistic constructs might shed light on neural mechanisms mediating late acquisition of visuospatial language.

Another important question is how the visual system integrates with the language system following the process of sign language acquisition and how this perceptual‐cognitive integration manifests at the functional level. Further, it is unknown whether functional brain changes in the language network are accompanied by morphological adaptations. Here, we employed a unique multimodal neuroimaging approach in order to understand the complexity of brain reorganization following sign language acquisition. Twenty hearing participants completed an 8‐month long course of Polish Sign Language (polski język migowy [PJM]) and underwent five functional MRI (fMRI) and structural MRI (sMRI) sessions, performed at 2.5‐month intervals, including pre‐exposure (TP0) and follow‐up scans performed 3 months after the course ended (TP4); see Figure 1a. To study changes in task‐based functional activity, we used a sentence comprehension task in PJM. Additionally, we investigated whether the strength of functional connectivity between spatially distinct regions was related to the proficiency of the PJM learners. In addition, we investigated whether alterations in GMV were associated with sign language learning and/or with PJM proficiency.

FIGURE 1.

(a–c) The design of the functional magnetic resonance imaging (fMRI) procedure (L2). (a) A timeline of fMRI sessions. (b) An overview of the experimental paradigm. (c) An example of stimuli and timing in the block. (d) Polski język migowy (PJM) material distribution. For a detailed description, see Section 2. (e,f) Behavioral results. (e) Accuracy scores for the in‐scanner L2 SJT task at all TPs (%). (f) Correlations between the sum of in‐scanner SJT accuracy scores at TP1–TP4 and the sum of all six proficiency classroom test scores, r = .47, p < .05). *p ≤ .005, **p ≤ .001, Bonferroni corrected. Error bars represent SDTP: time point; EXP, explicit condition (semantic processing); IMP, implicit condition (gender discrimination); SJT, semantic judgment task

Growing proficiency in sign language could result in a linear increase of brain activation, which would be in line with the Williams et al. (2016) findings. However, other studies report a negative relation between neural activation and L2 learning (Grant et al., 2015; Stein et al., 2009). Another plausible trajectory would be an inverted U‐shaped pattern with highest activation at early stages of learning, declining as proficiency increases (see the neural efficiency theory by Haier, Siegel, Tang, Abel, & Buchsbaum, 1992; Buschkuehl, Jaeggi, & Jonides, 2012; expansion‐renormalization model by Wenger, Brozzoli, Lindenberger, & Lövdén, 2017). Previous L2 acquisition studies suggest that brain reorganization can occur within 3 months (12 weeks) from the onset of learning (Barbeau et al., 2017), which corresponds to TP1 in the current study. It remains to be determined if additional functional changes are visible later on, or if after the initial reorganization the neural circuit prevails without major modifications. Changes in the brain response are expected within IFG—a modality‐independent lexico‐semantic region—as well as in modality‐dependent system that subserves visuospatial processing, for example, SPL and occipital regions. We also predict an enhancement of task‐based connectivity following sign language learning, with the strongest coupling between functionally and spatially remote visual and language areas occurring at the peak of PJM skills. In the domain of anatomical changes, sMRI research consistently suggests increases in GMV induced by both nonlinguistic and linguistic training in task‐relevant areas. Hence, we predict a significant increase of GMV in the regions involved in sign language processing.

We also examined whether plastic changes remain stable after the sign language course is ceased, a question that has so far been investigated only in a limited number of nonlinguistic studies (Draganski, Gaser, Schuierer, & May, 2004; Driemeyer, Boyke, Gaser, Buchel, & May, 2008; Scholz, Klein, Behrens, & Johansen‐berg, 2009). However, the results of these studies are inconsistent. Draganski et al. (2004) and Driemeyer et al. (2008) studied neural changes associated with learning to juggle and reported GM decreases after a period of 2–4 months without juggling training. In contrast, Scholz et al. (2009) did not observe significant GM changes after 4‐week break from juggling. We expected that at the follow‐up session (after a 3‐month break from signing), the level of brain activation and GMV would differ compared to the previous session, when participants were at the peak of their PJM skills. Specifically, we hypothesized that if we observe a positive relationship between language proficiency and the hemodynamic brain response, then the follow‐up session would reveal decreased neural activity. Conversely, a negative relationship (reflecting less effort over time) would result in stronger activity following the end of the sign language practice.

2. MATERIALS AND METHODS

2.1. Participants

Thirty‐three hearing females were recruited to participate in the study (mean age at pre‐exposure = 23.5, SD = 1.6; range = 20.3–28.3). Eleven participants dropped out of the study due to personal or medical reasons. Two were excluded from the fMRI analysis due to incomplete data caused by technical problems. Additionally, one person was excluded from the Polish control (first language, L1) analysis (see below) due to a high rate of incorrect responses (40%) at TP3. Therefore, 20 participants were included in the fMRI analysis of PJM learning (mean age at pre‐exposure = 23.0, SD = 1.4, range = 20.3–25.7), 19 participants were included in the Polish control (L1) fMRI data analysis, and 22 participants were included in the sMRI data analysis. The present experiment was a part of a larger MRI study on second language acquisition.

All of the participants were healthy, right‐handed, naïve to PJM, and native speakers of Polish. All had normal or corrected‐to‐normal vision. They had 14 or more years of formal education (20 were students, and one had completed higher education) and had nonverbal IQ within the age norms (assessed with the Raven Progressive Matrices test). At the time, participants were not learning any new languages besides PJM; however, they reported knowledge of at least one spoken L2. Participants had no contraindications to undergoing MRI, gave written informed consent, and were financially reimbursed for their time and effort (~500 euro in total). The study was approved by the Committee for Research Ethics of the Institute of Psychology of the Jagiellonian University.

2.2. Polish Sign Language course and behavioral measurements

For the purpose of the experiment, we implemented an 8‐month long PJM course led by two deaf professional instructors. The classes were 1.5 hr long and took place twice a week (57 meetings, 86 hr, M = 73.5 hr of instruction [range = 45.0–84.0, SD = 9.9], due to absences). The program of the course provided an increasing complexity of applied themes and activities. At the end, learners reached A1/A2 proficiency level, being able to describe immediate environment and matters, hold a conversation, and comprehend a simple monologue. In order to monitor students' learning, six vocabulary classroom tests (~every 5 weeks) were performed. Each test consisted of 30 signs introduced recently, presented by one of the teachers—learners had to write down the Polish translation of the sign.

2.3. fMRI tasks and stimuli

The experimental task was a Semantic Judgment Task (SJT) that required sentence‐level processing (Binder, Desai, Graves, & Conant, 2009) and was performed at five TPs (TP0–TP4), where TP0 was a pre‐exposure scan and TP4 a follow‐up scan, 3 months after the end of the course. TP1, TP2, and TP3 took place after ~32, ~58, and ~ 86 hr of instruction, respectively. Participants were asked not to practice PJM for the 3 months between TP3 and the follow‐up scan at TP4. An additional control task in Polish (the L1 of the participants) was implemented at each TP, presented in two conditions—reading (linguistic task) and visual search (nonlinguistic task).

PJM task (L2): In a condition that required explicit linguistic processing (EXP), participants performed the SJT, deciding whether a two‐sign phrase was semantically correct (e.g., DAD KNOW) or anomalous (CHAIR DRINK). In an implicit condition (IMP), sentential stimuli of the same type were presented (in blocks), but participants were asked to indicate the gender of a signing person for each sentence.

A set of signs was selected to be introduced in each learning period between fMRI scans at different TPs. For each TP, stimuli were adjusted to match participants' skills—only signs that had already been learned were included (Figure 1d). At each TP the stimuli that were presented were selected from all signs learned prior to that TP, so at TP1 they included signs learned during the first 3 months, while the stimuli presented at TP2 and TP3 consisted of signs acquired not only in the last learning period, but also earlier during the course. The purpose of such material distribution was to establish a similar level of task difficulty (and therefore cognitive demand) across all TPs. Stimuli presented at TP0 and TP4 were also taken from all learning periods, but were different from those presented at TP3. Since at TP0 participants were naïve to PJM, stimuli presented at TP4 were identical, although presented in a different order.

In total, 320 video clips were recorded in PJM by native Deaf signers (one male, one female), with full‐face and torso exposed. The sign models were dressed in black t‐shirts and stood in front of a grey screen. They were asked not to produce strong mouthing in order to avoid lip reading by the participants. Videos were displayed using Presentation software (Neurobehavioral Systems, Berkeley, CA) on a screen located in the back of the scanner, reflected in the mirror mounted on the MRI head coil. Participants' responses were collected using an MRI compatible ResponseGrip device (NordicNeuroLab; https://nordicneurolab.com/nordic-fmri-solution/). Participants kept the ResponseGrip device in their left hand and were asked to press a button with their thumb for one decision (semantically correct phrase/woman) and their index finger for the other decision (semantically anomalous phrases/man). All of the answers were saved in log files, which contained the list of correct, incorrect, and missing responses, as well as the timing of the responses. Sample stimuli (presented at TP0 and TP4) are listed in Supplementary Materials 1.1, and the experimental materials are available at: https://osf.io/6uf8g/.

Polish control task (L1): Analogously to PJM task, participants performed a SJT, deciding whether a two‐word written Polish phrase was semantically correct (e.g., “Chłopiec biega,” ang. “Boy runs”) or anomalous (“Stół pije,” ang. “Table drinks”). In the visual search, condition random letter strings were displayed on the screen. Half of the strings contained two “#” (GT#J T#PK) and half did not (RGSH TNCF). Participants had to indicate whether or not hashtags were present in the letter strings. Stimuli were three to six letters words/strings. Only the reading condition (the linguistic control task) was analyzed over time in order to test for nonspecific effects of task repetition.

2.4. Procedure

Stimuli were presented in a mixed block/event design. All of the tasks were presented in separate runs, and conditions were alternated. In order to exclude effects of material, stimuli in the EXP and IMP/reading and visual search were counterbalanced across participants. A task consisted of five EXP and five IMP blocks with eight trials per block. Each block consisted of four correct and four anomalous phrases, presented in pseudorandomized order. Before each block a fixation cross was presented for 6–8 s, followed by 2 s of a visual cue informing participants about the type of upcoming block (EXP/IMP or reading/visual search), followed by another fixation cross (1–2 s). The total duration of L2 SJT was 8.8 min: mean block duration = 47 s; mean stimuli length = 2.7 s; ISI = 3 s (starting with blank screen = 1 s, followed by a fixation cross indicating answer window = 2 s; Figure 1b,c). Total duration of L1 SJT was 7.9 min: mean block duration = 44 s; stimuli length = 2 s; ISI = 3 s (blank screen = 1 s, fixation cross indicating answer window = 2 s).

2.5. Imaging parameters

MRI data were acquired on a 3 T Siemens Trio Tim MRI scanner using a 12‐channel head coil. T1‐weighted (T1w) images were acquired with the following specifications: 176 slices, slice‐thickness = 1 mm, TR = 2,530 ms, TE = 3.32 ms, flip angle = 7°, FOV = 256 mm, matrix size: 256 × 256, voxel size: 1 × 1 × 1 mm. An echo planar imaging sequence was used for functional imaging. Forty‐one slices were collected with the following protocol: slice‐thickness = 3 mm, TR = 2,500, flip angle = 80°, FOV = 216 × 216 mm, matrix size: 72 × 72, voxel size: 3 × 3 × 3 mm.

2.6. Structural and fMRI preprocessing

The preprocessing and statistical analyses of MRI scans were performed using SPM12 (Wellcome Imaging Department, University College, London, UK, http://fil.ion.ucl.ac.uk/spm), run in MATLAB R2013b (The MathWorks Inc., Natick, MA). First, if needed, structural and functional images were manually reoriented to the Anterior Commissure. Within‐subject registration and bias‐correction of T1‐w images from five TPs was performed, using a Serial Longitudinal Registration approach implemented in SPM12 (Ashburner & Ridgway, 2013). This method avoids potential TP asymmetry biases by using spatially warped, intensity‐corrected versions of mid‐point average T1w (avgT1w) scan. For each participant, functional images from all TPs were realigned, coregistered to their avgT1w image and normalized to Montreal Neurological Institute (MNI) space using deformation fields acquired from avgT1w images. Finally, normalized images were smoothed with 6 mm full width at half maximum Gaussian kernel. For structural data, avgT1w images were segmented into tissue classes using the enhanced tissue probability maps for optimal delineation of subcortical structures (Lorio et al., 2016) followed by diffeomorphic anatomical registration (DARTEL; Ashburner, 2007), which creates a study‐specific template. Segmented GM probability maps were multiplied by Jacobian determinants from Serial Longitudinal Registration to obtain TP specific, bias corrected maps of volume deformation between the particular TP and the avgT1w image data. The product image was subsequently aligned with the standard stereotactic space defined by the MNI using DARTEL parameters and smoothed with 8 mm full width at half maximum Gaussian kernel.

2.7. Statistical analysis—Behavioral measurements

Behavioral results were analyzed using repeated‐measures analyses of variance (rmANOVA) in the SPSS software package (SPSS Inc., Chicago, IL, version 18.0). Scores from the vocabulary classroom tests were entered in the rmANOVA 6 × 1 model (six vocabulary tests), and in‐scanner accuracy was entered in a 5 (TP0–TP4) × 2 (EXP and IMP conditions) model. For the post hoc pairwise comparisons, Bonferroni correction at p < .05 was used.

Finally, a 5 (TP0–TP4) × 2 (EXP and IMP conditions) rmANOVA was computed to test differences in reaction times, which were measured as the time from the sentence onset to button press (see Supplementary Materials 1.3 and Table S1).

2.8. Statistical analysis—Task‐based activation

PJM task (L2): Statistical analysis was performed on subject (first) and group (second) levels using a general linear model (GLM). First level models were computed by convolving task and TP‐specific timings of all conditions (EXP/IMP correct/incorrect answers, missing responses and visual cues) and six head movement regressors with canonical hemodynamic response function. The data were high‐pass filtered with cut‐off period 1/210 Hz. At the second level analysis contrasts from correct answers in the EXP and IMP conditions from all TPs were entered into a flexible factorial model, with 5 (TP) × 2 (condition) factors—both specified with unequal variance—and subject factor, specified with equal variance. Then, several contrasts were computed, testing the main effects of time and condition as well as the TP x condition interaction. Task‐related responses were considered significant at a voxelwise threshold p < .05, corrected for multiple comparisons across the whole brain using a voxel‐level family wise error (FWE), with an additional extent threshold of >20 voxels. In the second level model, masks of the task‐positive activations from both conditions at each TP were added in order to create one binary mask for all subsequent analyses (Brennan, Cao, Pedroarena‐leal, Mcnorgan, & Booth, 2013).

Additionally, for each task, a series of post hoc pairwise comparisons between consecutive TPs were performed via paired t tests (TP0 vs. TP1, TP1 vs. TP2, TP2 vs. TP3, and TP3 vs. TP4, EXP condition; p < .05, cluster‐level FWE (FWEc) corrected; the results can be found in Supplementary Materials 1.4) with an inclusive mask from the main effect of time. Anatomical structures were identified with the probabilistic Harvard‐Oxford Atlas (http://www.cma.mgh.harvard.edu/) for cortical and subcortical areas and the AAL atlas (Tzourio‐Mazoyer et al., 2002) for cerebellar areas.

Finally, to further investigate the pattern of differences between TPs, region of interest (ROI) analyses were performed. Independent ROIs were defined anatomically using the Harvard‐Oxford Atlas. These structures were the modality‐independent region of left IFG (e.g., Emmorey et al., 2014; Friederici, 2012; Hickok & Poeppel, 2007), and modality‐dependent visuospatial areas—left SPL and occipital regions (lateral occipital cortex [L LOC]; Corina et al., 2007; Emmorey et al., 2014). Beta values extracted from the 5 (TP) × 2 (condition) model were then analyzed using rmANOVA in the SPSS software. All post hoc pairwise comparisons used Bonferroni correction at p < .05. For clarity, in the main text, we focus only on the main effect of time analysis, addressing research questions directly related to our hypotheses. Results and discussion of the main effect of condition, as well as the TP by condition interaction, can be found in Supplementary Materials 1.5 and 1.6.

Polish control task (L1): At the first level task and TP‐specific timings of all conditions (reading and visual search correct/incorrect answers, missing responses and visual cues), together with six head movement outliers and seven regressors were entered in the model. We computed a second level one‐way within‐subject 5 (TP) × 1 (reading condition) ANOVA that included only correct trials. Task‐related responses were considered significant at a voxelwise threshold p < .05, FWE (k = 20). Beta values extracted from the same ROIs (L IFGoper, L SPL, and L LOCsup) from the 5 (TP) × 1 (condition) model were analyzed using rmANOVA in the SPSS software.

Interaction language × time: At the second level the EXP condition data in both L1 and L2 from all TPs were entered into a flexible factorial model, with 5 (TP) × 2 (language) factors—both specified with unequal variance—and subject factor, specified with equal variance. Then, the TP × language interaction analysis was computed. Task‐related responses were considered significant at a voxelwise threshold p < .05, corrected for multiple comparisons across the whole brain using an FWEc. For the results of this analysis, see Supplementary Materials 1.7.

2.9. Statistical analysis—Task‐based connectivity

In order to investigate whether there was enhanced coupling between visual and language areas associated with learning a visual–spatial language, we conducted a general psychophysiological interaction analysis (McLaren, Ries, Xu, & Johnson, 2012). The deconvolved time series from the seed ROI in L LOC (defined independently using the Harvard‐Oxford Atlas)were extracted to create the physiological variable for each participant. New first level models (analogous to the first level GLM) were computed, including the physiological regressors of the seed ROI, the psychological regressor representing the EXP condition (for L2 SJT and Polish L1 control task), and the interaction of interest between both physiological and psychological regressors.

Beta coefficients for the interaction term were further aggregated at the second level random‐effects analysis into a flexible factorial model, with 5 (TP) × 2 (language) factors—both specified with unequal variance—and subject factor, specified with equal variance. To test whether the coupling occurs between the visual (L LOCsup) and language regions (L IFGoper and the left posterior part of MTG [L pMTG]) and to determine the temporal pattern of this coupling, we performed ROI analyses. Beta‐values from the flexible factorial mode were extracted using L IFG and L pMTG anatomically guided masks using the Harvard‐Oxford Atlas. Analyses were performed with rmANOVA models for each ROI using the SPSS software with TP (TP0–TP4) and language (L2, L1) as factors, with post hoc pairwise comparisons using Bonferroni correction at p < .05.

2.10. Statistical analysis—Voxel‐based morphometry

A one‐way within‐subject ANOVA with TP as a factor was conducted, in order to assess regional variations in GMV indicators over time. To achieve maximal statistical sensitivity and to estimate voxel residual smoothness correctly, a comparison‐specific optimal threshold GM mask was created using the Masking toolbox (Ridgway et al., 2009) and was used as an explicit mask for voxelwise comparisons. Because our a priori hypothesis was that GMV changes would be observed in the regions that changed their pattern of activity over time, the search volume was restricted to the same anatomically guided ROIs that were used in the task‐based activation analysis (defined using the Harvard‐Oxford Atlas). Voxels were considered significant when falling below a cluster‐corrected threshold of .05, adjusted for the small volume (small‐volume correction method) within each ROI.

To further perform the statistical analysis, the beta values obtained from significant results were exported to an rmANOVA model in the SPSS. All post hoc pairwise comparisons used Bonferroni correction at p < .05. When the sphericity assumption was not met, Greenhouse–Geisser correction was used and noted in the results.

3. RESULTS

3.1. Behavioral results

Vocabulary classroom tests: The rmANOVA analysis revealed a significant main effect of test (F(5,95) = 3.63, p ≤ .005, eta‐squared = 0.16); however, post hoc tests showed only a trend for the difference between Tests 1 (91.0%) and 2 (83.0%), p = .066. No significant differences in the scores between other tests were found (see Table S1 for details about participants' scoring).

PJM task (L2): The rmANOVA analysis with TP (TP0–TP4) and condition (EXP and IMP) showed a significant main effect of TP (F(4,76) = 16.84, p < .001, eta‐squared = 0.47). Post hoc tests showed an increase in accuracy between TP0 and TP1 (p < .001) and TP2 and TP3 (p < .005). Moreover, there was a significant accuracy decline at TP4 (p ≤ .001). As gender discrimination is relatively automatic and easier than language processing, there were also significant main effects of condition—differences between EXP and IMP conditions were significant at each TP (F(1,19) = 11,714.57, p < .001, eta‐squared = 0.98). Such a strong main effect of condition is likely due to ceiling effects in the IMP condition (see Figure 1e). In all TPs, accuracy in the IMP condition was higher than in the EXP condition at p < .001. Additionally, an interaction of TP × condition was found (F(4,76) = 17.84, p < .001, eta‐squared = 0.48), resulting from significant changes in accuracy only in the EXP condition.

We observed a positive correlation between participants' accuracy scores from the in‐scanner task (collected during PJM course from TP1 to TP4) and proficiency classroom vocabulary tests scores (r = .47, p < .05; mean score = 66.2%, proficiency tests = 87.7%; Figure 1f).

Polish reading task (L1): Behavioral analyses were also performed for the control L1 reading task. RmANOVAs with TP (TP0–TP4) as factor revealed a trend toward a significant main effect of time (F(4,72) = 3.15, p < .05, eta‐squared = 0.15); however, post hoc tests showed a significant difference only between TP0 and TP2, p < .005 (where there was a decline in accuracy; see Table S1).

3.2. Neuroimaging results—Task‐based activation

3.2.1. Main effect of TP

PJM task (L2): Over the course of PJM learning brain activation during SJT changed in left hemisphere cortical regions—IFG pars opercularis (L IFGoper), superior part of LOC (L LOCsup) and in bilateral superior parietal lobule (L SPL; Figure 2a and Table 1.). Additional clusters were found in L pMTG; Figure 2a). The full list of clusters can be found in Table 1).

FIGURE 2.

(a) Results from the task‐based activity analysis: Main effect of time point for L2 SJT (p < .05; family wise error [FWE]). (b) The results of independent anatomical region of interest (ROI) analyses (for a detailed description of ROIs, see Section 2). The Polish reading task (L1) is presented for visualization purposes. (c) Results of the task‐based connectivity between the seed L LOCsup and L IFGoper; repeated‐measures analyses of variance (rmANOVA), p < .05 Bonferroni corrected. (d) gray matter volume (GMV) changes over time in the L IFGoper. *p ≤ .05; **p ≤ .005; ***p ≤ .001, Bonferroni corrected. Error bars represent SD, adjusted to reflect within‐subject variance

TABLE 1.

Results from the main effect of the TP contrast—PJM learning

			MNI coordinates
Brain regions	Cluster size	F‐value	x	y	z
SJT
Left hemisphere
LOC (superior)	2,746	39.6	−26	−70	36
SPL		39.1	−28	−54	42
Precentral/inferior frontal gyri	2,626	42.4	−46	2	38
MTG (temporo‐occipital)	1,015	20.4	−52	−58	−4
LOC (inferior)		18.7	−48	−66	−6
Insular cortex	48	14.9	−28	22	−4
Paracingulate gyrus	46	16.1	−6	10	52
Precuneus	44	14.5	−4	−54	16
Right hemisphere
SPL	244	18.3	26	−54	58
LOC (superior)	171	29.6	30	−66	32
LOC (superior)	91	13.5	38	−76	18
Putamen	62	14.2	28	−6	−4
MFG	50	14.9	30	−4	48
Precentral gyrus		12.1	32	−6	56
Superior frontal gyrus	46	23.9	12	24	60
Precentral gyrus	33	13.3	36	6	30
Frontal pole	21	13.0	16	56	34

Abbreviations: MFG, middle frontal gyrus; MTG, middle temporal gyrus; LOC, lateral occipital cortex; PJM, polski język migowy; SJT, semantic judgment task; SPL, superior parietal lobule; TP, time point.

The ROI analysis using rmANOVA with TP (TP0–TP4) as a factor revealed a significant main effect of TP in L IFGoper (F(4,76) = 15.92, p < .001, eta‐squared = 0.47); L SPL (F(4,76) = 19.59, p < .001, eta‐squared = 0.51) and L LOCsup (F(4,76) = 15.64, p < .001, eta‐squared = 0.45). Post hoc tests showed that the most pronounced brain reorganization induced by learning occurred during the first 3 months of the PJM course—at TP0, the activity was significantly lower than in other TPs in all of the reported structures (Figure 2b).

Finally, pairwise comparisons between consecutive TPs (see Supplementary Materials 1.4) revealed a significant activation increase at TP1 compared to TP0. No significant brain activation changes were observed at later TPs. The comparison between TP4 and TP0, which included the same stimuli, produced qualitatively similar results to the comparison between TP1 and TP0. Importantly, no significant differences were observed between TP3, when participants were at the peak of PJM skills (after 8 months of learning), and TP4, the follow‐up scan after a 3‐month break from learning.

Additionally, a post hoc analysis of hemispheric lateralization using the lateralization index (LI) toolbox in SPM (Wilke & Lidzba, 2006) was performed comparing TP0 and TP1—the TPs that exhibited significant activation change. Statistical maps obtained at the first level of GLM analysis for each participant at both TPs were analyzed with the bootstrap option. Lateralization was calculated within a mask built from the following anatomically guided ROIs (using the Harvard‐Oxford Atlas): L IFGoper, LSPL, and L LOCsup and their right‐hemispheric analogues. The analysis resulted in an LI for each participant; left‐lateralized activations were indicated with positive values, and right‐lateralized activations were indicated with negative values. The lateralization indices were then compared between TPs TP0 and TP1 in SPSS via a paired t test. The comparison showed a significant difference in LI between TPs (t(19) = 4.64, p < .001) with the mean LI for the TP0: = −0.04, SD = 0.40; range = −0.54 to 0.70; for the TP1 mean LI = 0.38; SD = 0.25; range = −0.23 to 0.80. Since the common threshold for assessing hemispheric dominance is ±0.20 (Seghier, 2008), this result indicates that the activation significantly changed from bilateral at TP0 to left‐lateralized at TP1.

Polish reading task (L1): Whole‐brain and ROI analyses were performed with TP as a factor and as expected, no significant main effect of time was revealed.

3.3. Neuroimaging results—Task‐based connectivity

RmANOVA ROI analysis for the seed L LOCsup and L IFGoper (defined anatomically) revealed a significant TP × language interaction (F(4,76) = 2.98, p < .05, eta‐squared = 0.14). Post hoc tests showed a trend toward an increase of functional coupling between TP0 and TP2 (p = .059) and a significant increase between TP0 and TP3 in L2 SJT (p = .008; see Figure 2c). The ROI analysis for the L pMTG did not reveal any significant results for PJM learning. No changes in connectivity between L LOCsup and L IFGoper or L pMTG were observed for L1.

3.4. Neuroimaging results—GMV

Structural analysis based on voxel‐based morphometry revealed that significant changes occurred only in L IFGoper (peak coordinates −57 12 27, F = 6.43, k = 30). We did not observe any significant results in the other tested areas: L SPL and L LOC. The rmANOVA with TP (TP0–TP4) as a factor showed a significant main effect of TP (F(2.18, 45.84) = 6.14, p < .005, eta‐squared = 0.23). Post hoc tests revealed increases in the GMV between TP2 and TP3 (p ≤ .001). A significant decline was observed at TP4 in comparison to TP3 (p < .05). There was no significant difference in GMV between TP0 and TP4 (Figure 2d).

4. DISCUSSION

In the current study, we sought to characterize for the first time the neural dynamics of incorporating a newly acquired visuospatial sign language into an existing spoken language network, as reflected by functional and structural reorganization. We asked (a) how growing proficiency in a sign language impacts neural activity in modality‐independent and modality‐dependent brain regions and (b) what are the correspondences between functional and structural neural changes over time. We used a multimodal design with multiple TPs during learning in order to provide a more comprehensive understanding of the mechanisms of experience‐driven plasticity and its temporal pattern. Additionally, we tested whether the observed neural changes remained stable after sign language learning had ceased.

In line with our predictions, the functional brain reorganization following PJM learning in hearing novice learners occurred within a classic language‐related region located in the left hemisphere—the inferior frontal gyrus (e.g., Binder et al., 2009; Friederici, 2012; Hickok & Poeppel, 2007) and within regions associated more specifically with sign language processing, for example, the SPL and lateral occipital gyrus. Crucially, the lack of temporal changes in L1 processing emphasizes the specificity of learning‐induced alterations in L2 comprehension.

Left IFG (pars opercularis and triangularis; in the literature also referred to as BA44/45) has been previously described as a language region mediating both production and comprehension, regardless of language modality (Binder et al., 2009; Corina & Knapp, 2006; Emmorey et al., 2014, 2016; Friederici, 2012; Johnson et al., 2018; MacSweeney et al., 2002; MacSweeney, Capek, et al., 2008; Sakai et al., 2005; Williams et al., 2016). This region has been hypothesized to be a key node subserving unification, integration, and memory retrieval at various linguistic levels (Hagoort, 2013). The change in activation in L IFG with learning extended into the precentral gyrus, a region that has been previously reported to be engaged during the observation and understanding of human action and movement (Emmorey et al., 2014; Schippers & Keysers, 2011).

Moreover, as a result of learning, extensive clusters in temporo‐occipital regions (MTG and ITG) together with occipital regions (LOCsup and LOCinf) were engaged in PJM processing. Activation in this temporo‐occipital network likely reflects motion‐related perception of the body (Emmorey et al., 2014; Liu et al., 2017; MacSweeney, Capek, et al., 2008; Williams et al., 2016), as it overlapped with bilateral V5/MT+. Functional plasticity was also found in parietal regions including bilateral SMG, which has been shown to be involved in the phonological analysis of signs (e.g., MacSweeney, Waters, Brammer, Woll, & Goswami, 2008) and to support working memory demands of sign language (e.g., Rönnberg, Rudner, & Ingvar, 2004). In addition, we observed stronger engagement of SPL after learning (increased activation at TP1 in comparison to TP0). This finding contrasts with Williams et al. (2016) who found no significant changes in SPL with sign language learning. We speculate that Williams et al. (2016) may have not observed SPL changes because their experiment employed a simple, implicit task. This would be supported with our findings showing that the sentence‐level semantic judgment task evoked greater response in SPL than the implicit processing condition in which participants made a gender decision (for more information, see Supplementary Materials). Prior studies with deaf and hearing native signers report activation in SPL during sign language processing (Emmorey et al., 2014; Jednoróg et al., 2015; MacSweeney et al., 2002; McCullough, Saygin, Korpics, & Emmorey, 2012). SPL has also been associated with a number of nonlinguistic processes related to space and movement, for example, human action observation (Buccino, Binkofski, & Riggio, 2004; Calvo‐Merino, Glaser, Grèzes, Passingham, & Haggard, 2005; Corina et al., 2007; Goodale, 2011). In addition, SPL activation has been reported when sign‐naïve participants perceive sign language (Corina et al., 2007; Courtin et al., 2011; MacSweeney et al., 2006). Given the fact that in visuospatial languages movement and space convey linguistic information, it is not surprising that SPL belongs to a sign language comprehension network.

We found the most pronounced alterations in task‐based activity at an early stage of learning—during the first fMRI session (TP1) after ~32 hr of instruction, in comparison to the pre‐exposure scan (TP0). Later scans, performed during the PJM course did not reveal any significant changes in the level of brain activation (between TP1 and TP3), despite the fact that the learners did show increasing proficiency in their behavioral performance. Importantly, no significant changes in brain activity over time occurred in control L1 reading tasks, providing strong evidence that functional changes observed in L2 were indeed training‐specific and not a consequence of task repetition.

It has been previously shown that alterations in the neural response to newly learned words in an L2 can occur rapidly (e.g., McLaughlin, Osterhout, & Kim, 2004), and our study suggests that rapid neural changes can also occur when more complex, sentential processing is tested. Our results, however, are in contrast to Williams et al. (2016) who reported an increase of activation in the L IFGoper after two semesters of sign language course (TP2) in comparison to one semester (TP1). One possible explanation for the different results is that Williams et al. (2016) presented the same set of 30 stimuli at each TP; therefore, the increase in activation at TP2 might be a consequence of repetition or overlearning of the signs. The present study was designed to exclude effects of stimuli repetition by distributing the material across sessions in a manner that was adjusted to the program of the course. The lack of an effect of training from TP1 to TP3 suggests that once the elemental knowledge about the linguistic aspects of a new language is developed and incorporated into the comprehension system, this knowledge might be consolidated and retrieved in a stable manner. Nevertheless, it is quite possible that greater immersion in the L2, for example living in an L2‐speaking country (Stein et al., 2009), could result in stronger proficiency effects within this time period (e.g., 3 months) and thus greater changes in neural reorganization.

However, we did observe an increase in functional neural coupling between visual cortex and language‐related cortex that extended beyond the first TP. Specifically, we found increasing functional coupling between L LOCsup and L IFGoper that reached its peak when learners gained their highest level of proficiency in PJM (TP3). This result supports our hypothesis that because sign language requires encoding linguistic information through visual–spatial distinctions, its acquisition will entail the integration of spatially and functionally distinct visual and language systems. Furthermore, this finding is consistent with the idea that hierarchical processing is a key organizational aspect of the language system. For spoken languages, acoustic input reaches early auditory cortex and is subsequently decoded and interpreted in supramodal, higher‐level language areas (e.g., IFG; Davis & Johnsrude, 2003; Friederici, 2012). Indeed, when learning words in a spoken L2 enhanced coupling is observed between auditory cortex (STG) and IFG, underlying successful learning (Yang, Marie, & Molenaar, 2015). Here, we show for the first time that analogous bottom‐up processing most likely occurs also during sign language processing and that the integration of the system supporting efficient decoding of the linguistic information from visual input manifests with growing proficiency when learning visuospatial language.

With respect to GMV, our findings support previous results showing an increase in GMV following learning (e.g., Draganski et al., 2004; Hosoda et al., 2013; Schmidt‐Wilcke et al., 2010). We expected to find a correspondence between functional and structural changes in both modality‐independent and modality‐dependent brain areas. This prediction was partially supported, as we observed a significant GMV increase in the language‐related region of L IFGoper. This finding is consistent with the results of Mårtensson et al. (2012) who also reported an increase in GMV in L IFGoper after L2 language learning. In contrast to the initial changes in functional activity, the change in GMV did not occur rapidly (between TP0 and TP1), but was more prolonged in time, with the peak at the end of the PJM course. A number of studies have shown that GMV changes can be detected as early as within a few days or even hours of training as a result of improving, for example, visuomotor skills (e.g., Driemeyer et al., 2008; Kodama, Ono, Yamashita, Ebata, & Liu, 2018; Landi, Baguear, & Della‐Maggiore, 2011) or learning new colors (Kwok et al., 2011). We extend these findings by showing that cognitively demanding and time‐consuming L2 acquisition also results in GMV increases after just a few months of learning.

To sum up our results thus far, we found support for the idea that incorporation of a visuospatial L2 into the language comprehension system results in a rapid increase of brain activation when processing signed sentences for meaning, and we hypothesize that this functional change reflects the transition from sensory to higher‐level linguistic processing (in line with Williams et al., 2016). We did not confirm the predictions of the neural efficiency theory (Haier et al., 1992) and the expansion‐renormalization hypothesis (Wenger et al., 2017), which suggest that functional plastic changes associated with learning should follow an inverted U‐shaped pattern. For example, Wenger et al. (2017) propose that the learning‐related plasticity follows a sequence of neural expansion (e.g., through the generation of new dendritic spines or synaptogenesis), optimization (selection of the most efficient neural circuitry) and renormalization (stabilization through practice and elimination of the redundant neural pathways). However, it seems doubtful that the PJM learners reached a level of proficiency that would allow for such optimization to take place. Indeed, we found no alterations of brain activity after the first 3 months of PJM learning, despite continued improvements in performance. Nonetheless, enhanced connectivity between visual and language systems (L LOCsup and L IFGoper) was observed with the highest coupling occurring at the peak of sign language skills. Our findings show that the brain of novice L2 learners adapts to demands introduced by novel experience through an increased level of neural activity at the earliest stages of learning, followed by morphological and connectivity alterations (Wenger et al., 2017).

Finally, we employed an additional follow‐up scanning session in order to test whether the cessation of L2 exposure would have an impact on brain activity and/or morphology. We found that a 3‐month long break did not affect the functional brain responses, although the behavioral results showed a decrease in accuracy at TP4. This finding contrasts with Tu et al. (2015) who reported that early, highly proficient spoken language bilinguals showed a significant increase in activation after 30‐day break from L2 usage in frontal language control areas for their L2 in comparison to their L1. However, the difference between our findings and results reported by Tu et al. (2015) is presumably related to distinct cognitive control mechanisms supporting L1 and L2 processing between novice L2 learners in our study and early proficient bilinguals in Tu et al. (2015).

The pause in PJM learning did result in a significant decrease of GMV in the L IFGoper, which receded back to the pretraining baseline state, in line with previous studies that examined nonlinguistic training (Draganski et al., 2004; Driemeyer et al., 2008; Hosoda et al., 2013). This result suggests, that after an initial increase of demand for neural supplies triggered by novel experience, when learning is terminated, the brain must again adapt to lower demands by getting rid of the unused neural supplies (in other words, “use it or lose it”; see Lövdén, Bäckman, Lindenberger, Schaefer, & Schmiedek, 2010 for discussion). Although in the case of GMV the reason for the observed decrease is intuitive, the lack of changes in the level of neural activity at TP4 remains puzzling. One of the major challenges in future studies would be to separate the possible effects of optimization/consolidation of L2 language representation from the effects of the cessation of learning and to explore the relation between functional and structural characteristics of these effects.

4.1. Limitations

Several limitations of the current experiment should be noted. First, longitudinal paradigms capturing learning‐dependent processes are susceptible to the effect of task repetition. We attempted to address this problem by including a control L1 reading task which was presented at each TP. However, a better control might be to include a separate control group of participants who undergo the MRI sessions within the same time intervals but in the absence sign language training. Inclusion of such a control group would provide more compelling evidence for the structural plasticity (see Thomas & Baker, 2013). Thus, a future longitudinal study controlling for morphological alterations with a separate group would be helpful.

In addition, we note that the strategy of material distribution is a crucial decision in longitudinal studies. Learning a new language is a demanding skill and efficient communication requires retrieving the acquired vocabulary from a dynamically expanding lexicon. Therefore, we attempted to create an ecologically valid task design by keeping task difficulty comparable across TPs. However, we cannot fully rule out the possibility that the lack of functional changes between TP1 and TP3 is, at least partially, linked to our methodological strategy. The alternative approach would be presentation of the same set of stimuli throughout all of the sessions (as done by Williams et al., 2016). However, since the current experiment includes five fMRI sessions, we suggest that this strategy is suboptimal because the risk of stimulus set overlearning is high and repetition effects would very likely be present.

Future studies could also consider exploring distinct linguistic components (e.g., phonological, lexical) to identify the neural dynamics of processing different streams during sign language acquisition. Moreover, applying multivariate methods could access a finer‐grained level of linguistic processing.

5. CONCLUSIONS

Here, we observed rapid functional reorganization after just 3 months of sign language learning in the modality‐independent perisylvian language‐related network, together with additional reorganization in modality‐dependent visuospatial and motion‐sensitive regions. Connectivity analyses revealed enhanced coupling within the occipital and frontal network associated with increased proficiency. At the same time, GMV increase was detected in the left inferior frontal gyrus peaking at the end of learning. The results suggest that the brain adapts to demands introduced by novel experiences through multiple, temporally distinct mechanisms. Precisely how the initial changes in functional activation during language learning contribute to a tighter integration between cortical regions and their structural reorganization are critical questions for future research.

Supporting information

Appendix S1: Supporting information

Banaszkiewicz A, Matuszewski J, Bola Ł, et al. Multimodal imaging of brain reorganization in hearing late learners of sign language. Hum Brain Mapp. 2021;42:384–397. 10.1002/hbm.25229

DATA AVAILABILITY STATEMENT

The datasets generated during the current study are available from the corresponding author upon request and are available at: https://osf.io/6uf8g/.