Anatomical Substrates of Visual and Auditory Miniature Second-language Learning

Abstract

Longitudinal changes in brain activity during second language (L2) acquisition of a miniature finite-state grammar, named Wernickese, were identified with functional magnetic resonance imaging (fMRI). Participants learned either a visual sign language form or an auditory-verbal form to equivalent proficiency levels. Brain activity during sentence comprehension while hearing/viewing stimuli was assessed at low, medium, and high levels of proficiency in three separate fMRI sessions. Activation in the left inferior frontal gyrus (Broca’s area) correlated positively with improving L2 proficiency, whereas activity in the right-hemisphere (RH) homologue was negatively correlated for both auditory and visual forms of the language. Activity in sequence learning areas including the premotor cortex and putamen also correlated with L2 proficiency. Modality-specific differences in the blood oxygenation level-dependent signal accompanying L2 acquisition were localized to the planum temporale (PT). Participants learning the auditory form exhibited decreasing reliance on bilateral PT sites across sessions. In the visual form, bilateral PT sites increased in activity between Session 1 and Session 2, then decreased in left PT activity from Session 2 to Session 3. Comparison of L2 laterality (as compared to L1 laterality) in auditory and visual groups failed to demonstrate greater RH lateralization for the visual versus auditory L2. These data establish a common role for Broca’s area in language acquisition irrespective of the perceptual form of the language and suggest that L2s are processed similar to first languages even when learned after the ‘‘critical period.’’ The right frontal cortex was not preferentially recruited by visual language after accounting for phonetic/structural complexity and performance.

INTRODUCTION

Most imaging studies of second language have focused on the characterization and comparison of stable brain representations resulting from many years of exposure and experience. This research demonstrates interactions between the effect of age of acquisition (Wartenburger et al., 2003; Hasegawa, Carpenter, & Just, 2002; Chee, Caplan, et al., 1999; Chee, Tan, & Thiel, 1999; Klein, Minler, Zatorre, Zhao, & Mikelski, 1999; Neville et al., 1998; Kim, Relkin, Lee, & Hirsch, 1997) or proficiency (Wartenburger et al., 2003; Chee, Hon, Lee, & Soon, 2001; Perani et al., 1998; Perani et al., 1996) and the extent of spatial overlap between brain representations of native language (L1) and second language (L2), as well as spatial variability of activation associated with L1 and L2 (Dehaene et al., 1997), and the functional correlates of L1–L2 switching (Hernandez, Dapretto, Mazziotta, & Bookheimer, 2001; Price, Green, & von Studnitz, 1999). An equally important aspect of L2 research is the characterization of the learning process itself. Defining the brain changes accompanying second-language acquisition (SLA) is critical for the development of complete and accurate models of the functional localization of language during normal development and has potential implications for the rehabilitation of persons recovering from brain injuries affecting language skills.

The two goals of the current experiment were to identify brain regions that underlie comprehension in a recently acquired L2 and to establish if different brain substrates were recruited for visual versus auditory forms of the language. For the first goal, it was hypothesized that the left inferior frontal cortex (Broca’s area) known to be involved in native language syntactic processing would also be engaged for SLA. Broca’s area (Brodmann’s area [BA] 44 and BA 45) has been implicated in L1 syntactic processing by both lesion (Grodinsky, Pinango, Zurif, & Drai, 1999; Caramazza & Zurif, 1976) and imaging data (Muller and Basho, 2004; Musso et al., 2003; Sakai, Homae, & Hashimoto, 2003; Hashimoto & Sakai, 2002; Sakai, Noguchi, Takeuchi, & Watanabe, 2002; Grezes & Decety, 2001; Caplan, Alpert, Waters, & Oliviera, 2000; Caplan, Alpert, & Waters, 1999; Stromswold, Caplan, Alpert, & Rauch, 1996). Only recently has the role of Broca’s area in the language acquisition process become the focus of brain imaging experiments. Musso et al. (2003) demonstrated that activity in Broca’s area and its right-hemisphere (RH) homologue increased as participants were explicitly taught a finite set of ‘‘real’’ grammatical rules. Similar increases in activation were not observed in Broca’s area for a separate set of ‘‘unreal’’ grammatical rules (rules that were not compatible with universal grammar and occurred in no natural languages), although activation increases were observed in the RH homologue. These findings led Musso et al. (2003) to hypothesize that Broca’s area is specialized for the acquisition and/or processing of specific types of grammatical rules found in natural language, whereas its RH homologue is involved in rule acquisition more generally. Based on this hypothesis, we predicted that increases in activation would be observed in both Broca’s area and its RH homologue as participants learned a novel finite grammatical system that we refer to as ‘‘Wernickese.’’

Because detection and representation of sequentially recurring patterns was critical to the acquisition of Wernickese, we also predicted that brain areas associated with sequence learning would exhibit learning-dependent changes in activity during SLA. A large number of experimental paradigms involving perceptual/motor sequence learning report changes in the presupplementary motor area (pre-SMA), SMA, lateral premotor cortex, and basal ganglia (Hikosaka et al., 2001; Toni, Krams, Turner, & Passigham, 1998; Hikosaka et al., 1996; Kettner, Marcario, & Clark-Phelps, 1996; Sadato, Campbell, Ibanez, Deiber, & Hallett, 1996; Gordon et al., 1995; Grafton, 1995; Grafton et al., 1992; Mushiake, Inase, & Tanji, 1991; Halsband & Freund, 1990).

Our second goal was to determine if language areas were differentially recruited as a function of the perceptual modality of the language. Although it is generally accepted that auditory and visual language processing predominately recruits brain areas in the left hemisphere (LH; Dehaene-Lambertz, Dehaene, & Hertz-Pannier, 2002; Frost et al., 1999), brain imaging experiments involving sign language have reached conflicting conclusions regarding the role of the RH in visual–spatial language processing (Braun, Guillemin, Hosey, & Varga, 2001; Newman, Bavelier, Corina, Jezzard, & Neville, 2001; Bavelier et al., 1998; Neville et al., 1998; McGuire et al., 1997). An untested hypothesis for this difference is that the LH is recruited for the processing of temporal frequencies on which auditory languages rely most heavily, whereas the RH is recruited to process spatial information on which visual languages rely more heavily. Based on this hypothesis, we predicted laterality differences in the final brain representations of auditory versus visual versions of Wernickese. An alternative hypothesis to explain hemispheric differences of activity is that auditory and visual languages have topological differences due to their grammatical structures. Because we used a finite-state grammar with identical topological structure in the auditory and visual modalities, any differences observed between the two L2 modalities could not be ascribed to topological differences. A third hypothesis is that differences in proficiency between two languages might cause differential recruitment of the two hemispheres. Our subjects were trained to identical levels of proficiency, so any differences could not be ascribed to this explanation.

To examine these goals rigorously, auditory and visual versions of a miniature language, Wernickese (Figure 1), were created and taught to two separate groups of participants. The two versions were matched in terms of grammatical complexity, morphosyntactic structure and duration, and subject proficiency throughout learning. In order to compare the learning mechanisms in these two maximally different modalities of a second language we assessed learning-related changes in magnetic resonance imaging (MRI) blood oxygenation level-dependent (BOLD) signal within these groups at three stages of proficiency (low, moderate, and high) during the course of an intensive (1 hr/day, Monday to Saturday) 4-week training period. Finally, we tested whether brain activation during visual or auditory L2 processing was more right lateralized than brain activation during L1 processing. This required that in the final functional MRI (fMRI) session, participants be scanned not only during processing of auditory or visual Wernickese sentences at equal proficiencies, but also during processing of similar sentences presented in their native language (English) to characterize the magnitude of L1 laterality.

Figure 1.

Schematic representation of Wernickese grammar. Grammatical sentences were constructed by randomly selecting two noun phrases, a preposition, and a verb. One determiner (d) was used for unmodified nouns, whereas a separate determiner (D) was used for nouns followed by a modifier. S = sentence; NP = noun phrase; VP = verb phrase; N = noun; M = modifier; P = preposition; d = determiner; V = verb.

METHODS

Participants

Twenty-two healthy right-handed (Oldfield, 1971) adults (aged 18–21 years) participated in a 6-week-long experiment after informed consent was obtained. We selected only native English speakers with comparable English proficiency. No participant was fluent in any language other than English. These participants were randomly assigned to learn either an auditory/spoken or a visual/gestural version of Wernickese. Eight participants in the auditory group (4 women, 4 men), and 10 in the visual group (6 women, 4 men) completed all experimental requirements. Participants were paid $10 per hour for behavioral sessions, $20 per hour for fMRI sessions.

Stimuli Creation

The auditory and visual versions of Wernickese were designed to be structurally and grammatically similar to natural language, but to be produced and perceived in maximally different modalities (auditory and visual).

For the auditory version of Wernickese, 56 two-syllable words (8 nouns, 8 modifiers, 8 prepositions, 2 verbs, 2 determiners, and 28 meaningless control words), each consisting of two consonant–vowel pairings (CVCV) were created. Structure was assigned to the language by assigning patterns to words within each word category. For example, nouns ended in ‘‘IG,’’ modifiers ended in ‘‘TO.’’ All sounds were first recorded on a G-4 Macintosh (Apple, Cupertino, CA) equipped with a standard microphone using the program Sound-Edit 16 (Macromedia, San Jose, CA). Length of the two-syllable sound files was adjusted (lengthened) such that all words were the same length (600 msec). Words were recorded by a speaker blind to the word category in a natural-sounding manner. Only entire words were matched for duration. Sounds were then normalized and saved as 44-kHz stereo digital files.

Short video animations of bimanual gestures were used as ‘‘signs’’ (equivalent to ‘‘words’’) in the miniature visual language version. These bimanual gestures were constructed by using hand shapes taken from a listing of standard American Sign Language (ASL) hand shapes (Tennant & Brown, 2000). Importantly, the signs we used were not real ASL signs, but were instead combinations of arm movements and real ASL ‘‘hand shapes.’’ The resulting words were thus possible but nonexisting manual forms. Discernable structure was also given to this visual version of the language. For example, all nouns consisted of bimanual movements in which identical hand shapes were moved in a circular path and returned to their original positions. All gestures were produced while listening to a metronome beating at approximately 120 beats per second to ensure accurate timing. A special gesturing box was constructed that allowed the language user to place his or her arms in two holes and perform movements against a contrast-maximizing background. Movies of the visual language stimuli were recorded using a Sony miniDV video recorder (30 fps) on a tripod, and were digitized on a G-4 Macintosh using iMovie. Each gesture file was then converted into 30 .jpeg images. Subsequent presentation programs presented the gestures at 50 fps, or 650 msec per gesture, resulting in a sign duration identical to the word duration in the auditory stimuli. Because 50 msec were inserted between words in the auditory version of Wernickese, stimuli were sized to 600-msec duration. Thus, one word plus 50 msec of silence equaled the duration of one word in the visual version of Wernickese. The visual stimuli were continuous (no 50-msec duration between words) and lasted for 650 msec. Both hands always started and ended at the same position. Hands were stationary for 50 msec at this starting/ending position between all movements. Thus, the 50 msec of silence inserted between auditory Wernickese words can be thought of as equivalent to the 50-msec pause in hand movement for visual Wernickese words.

A novel finite-state grammar with real grammatical rules (see Musso et al., 2003) was constructed that was identical for both auditory and visual versions of Wernickese (Figure 1). Within this framework, 82,944 unique grammatical sentences were constructed with MATLAB (The Mathworks, Natick, MA). This set was randomly sampled for presentation during the experiment.

Training Procedure

Because participants had extensive experience with auditory language perception and little to no experience with perception of a visual language, we thought it critical to ensure that both groups were capable of processing the basic vocabulary of Wernickese equally well before beginning with grammar training. Thus, participants first learned the vocabulary of Wernickese over a 2-week period via a picture–word matching game. Grammar training began immediately after vocabulary mastery, with the first exposure to sentences occurring during the fMRI grammar session 1. Participants were presented with four 10-min runs. Each run consisted of presentation of 40–60 Wernickese sentences. Following this fMRI session, participants received additional practice and instruction in the grammar of Wernickese (via a suite of training programs). When medium proficiency (60–80% accuracy) had been achieved (1–2 weeks into training) the second grammar fMRI session was conducted. A third and final grammar fMRI session was conducted when participants had mastered sentence processing (85–100% accuracy).

fMRI Testing Procedure

Functional activity during L2 sentence processing was recorded during the grammar fMRI sessions. Each session consisted of four to five 10-min sentence-processing (SP) runs. Each 10-min SP run consisted of (1) SENT epoch in which participants viewed/heard sentences, (2) PROC epoch consisting of the time between the end of the sentence and a subject’s response signaling readiness to proceed, (3) GRID epoch beginning with the appearance of a 4 × 4 test grid containing a number of shapes of various colors and ending with a subject evaluation of the veracity of the Wernickese sentences’ claim concerning the relative positions of two specific colored shapes on the basis of the presented grid, and (4) REST epoch consisting of the first and last 45 secs of each functional run during which subjects fixated on a small cross. These runs were self-paced, and, depending on the speed of the participants’ decisions, between 40 and 60 sentences were presented during each SP run. A control task consisted of grammatical English sentences with semantic content similar to that of Wernickese sentences (Figure 2).

Figure 2.

Percent correct for the Wernickese sentence comprehension task during three fMRI sessions (S1_W, S2_W, and S3_W). Also shown is percent correct on the English sentence comprehension control task administered during the third fMRI session (S3_E). The fMRI sessions took place approximately 0, 2, and 4 weeks into sentence comprehension training.

Imaging Protocol

BOLD images were acquired using a 1.5-T GE SIGNA Echospeed MRI scanner (General Electric, Milwaukee, WI) equipped with high-performance gradients (revision LX 8.3; maximum amplitude 4.0 mT/m; slew rate 150 mT/m/sec). A self-molding pillow and small pads were used to restrict head motion. Functional scans were preceded by acquisition of a T1-weighted localizer scan and coplanar scan. During the task, whole-brain gradient-echo (EPI) scans (25 contiguous axial slices, 4.5 mm thick, skip 1, repetition time [TR] = 2500 msec, echo time [TE] = 35 msec, flip angle = 90°) were acquired. Finally, a 124-slice, high-resolution, T1-weighted anatomical scan was obtained by using a 3-D SPGR pulse sequence (TR = 25 msec, TE = 6 msec, radio frequency flip angle = 25°, bandwidth = 156 Hz, voxel size = 0.9375 × 1.25 × 1.2 mm) for all subjects.

Data Analysis

All data were moved to an off-line computer and preprocessed by using statistical parametric mapping (SPM 99) software developed by the Wellcome Department of Cognitive Neurology (London, UK). Data were first adjusted for slice timing by reordering the slices to coincide with the actual time course of the experiment. Next, subject motion was corrected by using a rigid-body least squares fitting algorithm. Participants were not rejected on the basis of head motion. Head motion parameters (pitch, roll, and yaw) were entered into SPM99 models as regressors to account for variance in brain signal due to head movement. Following motion correction, the high-resolution anatomical image was coregistered with the functional data using a mutual information algorithm. All functional images were then normalized to a standard anatomic template in SPM99 using 3 × 3 × 3-mm voxels. Spatial smoothing (gaussian kernel, 6.0-mm full width at half maximum) was applied to all functional images.

For the grammar fMRI sessions, the ‘‘REST’’ epoch was modeled as the first and last 40 sec of each functional run. The ‘‘SENT’’ events were modeled as the time during which sentences were presented. The duration of this epoch ranged from 3250 to 5200 msec, depending on the number of elements in the randomly generated sentence. The ‘‘PROC’’ events were modeled as the time from the end of the SENT event to a button press signaling readiness to proceed to the next task. Finally, the GRID event was modeled as the time between the appearance of a test grid and the participants’ veracity judgment (via a button press).

Fixed effects multiple regression of the BOLD activity was calculated for each individual. Resultant contrast images were subjected to a group-level random effects model. To assess changes in functional activity associated with grammar acquisition, BOLD signal during L2 sentence processing in Session 1 (S1) and Session 3 (S3) were compared using independent sample t tests [S1–S3] and [S3–S1]. Significant clusters of activation from these two contrasts were considered parts of the ‘‘SLA network.’’ Significant activation clusters in the SMA, basal ganglia (bilateral), inferior frontal gyrus (IFG), pars opercularis (bilateral), and planum temporale (PT, bilateral) were subjected to additional region-of-interest (ROI) analysis in order to further specify learning-related changes in activation. For this analysis, BOLD signals recorded during all three sessions were plotted.

RESULTS

During the initial vocabulary acquisition stage (a set of 10 one-hour training sessions conducted over a 2-week period) participants in the auditory and visual groups learned to associate individual Wernickese words with pictographic representations of their meanings placed on the buttons of a standard computer keyboard. Performance, as measured by percent correct in the last day of training, was not significantly different in the auditory (M = 94.62, SD = 4.72) and visual (M = 94.25, SD = 3.32) groups, t(17) = 0.20, p = .85. Performance, as measured by reaction time on the final day of training, was also similar in auditory (M = 797.03, SD = 239.68) and visual (M = 825.67, SD = 174.32) groups, t(17) = .31, p = .76. These data suggest that at the end of vocabulary training, participants in the auditory and visual groups were equally proficient at retrieving the meanings of Wernickese words in the learned modality.

During the course of grammar training, which consisted of decoding Wernickese sentences, all participants were able to achieve high proficiency (85–100% correct). A 2 × 3 (Group × fMRI Session) repeated measures analysis of variance (ANOVA) using percent correct as the dependent variable revealed a significant main effect of fMRI session, F(1,17) = 212.82, p < .001, but no main effect of group, F(1,17) = 2.96, p = .10, although a trend was observed in the direction of superior performance in the visual group (Figure 1). The interaction between group and session was not significant F(1,17) = 0.19, p = .825, indicating that both groups showed similar rates of improvement across these three fMRI sessions. Follow-up statistics revealed a significant improvement in the auditory group between fMRI Sessions 1 (M = 44.77, SD = 4.53) and 2 (M = 72.36, SD = 9.95), as well as between fMRI Sessions 2 and 3 (M = 89.3, SD = 5.3), t(7), all comparisons p < .001. Participants in the visual group also showed significant improvements between Sessions 1 (M = 51.2, SD = 12.11) and 2 (M = 75.07, SD = 14.23), as well as 2 and 3 (M = 93.87, SD = 5.06), t(9), all comparisons p < .001. A Group × fMRI Session repeated measures ANOVA was also conducted on sentence processing time. Again, there was a main effect of fMRI session, F(1,17) = 12, p < .005, but no significant main effect of group, F(1,17) = 0.080, p = .78, or interaction between group and fMRI session, F(1,17) = 0.347, p = .79 (Figure 2). In the auditory group, improvement was not significant between Sessions 1 (M = 3.88, SD = 0.3) and 2 (M = 3.13, SD = 1.15), t(7) = 1.73, p = .106. Nor was it significant between Sessions 2 and 3 (M = 2.32, SD = 1.53), t(7), p = .252. In the visual group, improvement was not significant between Sessions 1 (M = 3.56, SD = 1.02) and 2 (M = 3.01, SD = 1.45), t(9)= .921, p = .368, or between Sessions 2 and 3 (M = 2.32, SD = 1.65), t(9) = 1.31, p = .271.

The SLA network was defined as the set of areas where BOLD signal during L2 sentence processing in Session 1 (S1) and Session 3 (S3) was significantly different (both [S1–S3] and [S3–S1]). As predicted, activation in areas associated with sequence learning including SMA, basal ganglia (putamen), and native language processing, for example, Broca’s area, increased between S1 and S3 (Table 1). In contrast to previous findings (Musso et al., 2003), BOLD signal in the RH homologue of Broca’s area decreased over S1 to S3 in both auditory and visual groups (Figures 3 and 4). Some changes in BOLD signal associated with SLA were dependent on L2 modality. In the auditory group, regional activity in the PT activity during sentence processing decreased bilaterally over all three grammar sessions. In the visual group, the direction of change was more complex, with a net increase in right PT activation from Sessions 1 to 3 (Figures 5 and 6), and an increase (Session 1 to 2) followed by a decrease (Session 2 to 3) in left PT activation.

Table 1.

Regions Showing Increases in Activation during Second Language Acquisition

	Voxel Coordinates
Region	x	y	z	z Score
Auditory group
L/R SMA	6	0	57	5.07
L anterior cingulate cortex	9	12	36	4.44
L circular insular sulcus/gyrus	48	0	12	3.36
L IFG (pars opercularis)	45	15	21	3.99
L lingual gyrus	24	−75	6	3.86
L putamen	18	3	3	4.19
R anterior cingulate cortex	−6	27	21	4.92
R precuneus	−15	−63	24	7.14
R circular insular sulcus/gyrus	−45	3	9	4.79
R ventral precentral gyrus	−60	−3	15	3.70
R inferior precentral sulcus	51	9	9	4.02
R lingual gyrus	−12	−75	6	3.98
R thalamic nuclei	−6	−24	6	3.47
Visual group
L/R SMA	−3	6	54	4.16
L/R occipital gyrus	0	−84	6	5.70
L/R lingual gyrus	0	−84	6	5.70
L left superior frontal gyrus	30	45	27	2.90
L basal ganglia, putamen	24	0	3	3.31
L pars opercularis	57	9	18	3.49
L ventral precentral sulcus	63	3	6	3.49
L circular insular sulcus/gyrus	42	−3	15	4.94
L precentral gyrus	42	−12	54	5.09
L angular gyrus	18	−75	45	3.01
L thalamic nuclei	24	−24	6	4.33
R circular insular sulcus/gyrus	−42	−3	12	2.66
R precentral gyrus	−42	−15	60	4.39
R right superior frontal gyrus	−9	69	−3	3.31
R right superior frontal gyrus	−9	54	27	3.42
R putamen	−21	0	9	4.22
R fusiform gyrus	−27	−9	−42	3.39

L = left; R = right. SMA = supplementary motor area; IFG = inferior frontal gyrus.

Figure 3.

Changes in BOLD signal across three fMRI sessions in the IFG. L = left hemisphere; R = right hemisphere. (A) Increases (S3–S1) in BOLD signal in the visual group. (B) Increases in the auditory group. (C) Decreases (S1–S3) in BOLD signal in the visual group. (D) Decreases (S1–S3) in the auditory group. Areas shown surpass significance threshold of p < .05, uncorrected. Extraneous activation was filtered from these statistical images by masking with 18-mm spheres centered on the left and right IFG (Talairach coordinates: x, y, z = −46, 19, 13 and 46, 19, 13, respectively; Talairach & Tournoux, 1988). Images are overlaid on a normalized (1 × 1 × 1 mm voxels) high-resolution image from a single participant.

Figure 4.

Statistical comparison of percent signal change (based on extracted beta values) during sentence processing epoch of fMRI Sessions 1, 2, and 3. Results are plotted for Broca’s area and its RH homologue in the auditory and visual groups. Above each graph are p values for t tests (corrected for multiple comparisons) between Sessions 1 and 2, 2 and 3, and 1 and 3, respectively.

Figure 5.

Changes in BOLD signal across three fMRI sessions. (A) Left (L) and right (R) PT regions exhibiting learning-dependent changes in activation in the visual group (black areas). Activation in the left PT increased then decreased, whereas activation in the right PT increased consistently across the three grammar fMRI sessions. (B) Decreases in activation (Session 1–Session 3) at bilateral PT sites in the auditory group (white areas). Areas shown surpass significance threshold of p < .05, uncorrected. Images are overlaid on a normalized (1 × 1 × 1 mm voxels) high-resolution image from a single participant.

Figure 6.

Statistical comparison of percent signal change (based on extracted beta values) during sentence processing epoch of fMRI Sessions 1, 2, and 3. Results are plotted for PT and its RH homologue in the auditory and visual groups. Above each graph are p values for t tests (corrected for multiple comparisons) between Sessions 1 and 2, 2 and 3, and 1 and 3, respectively.

In order to assess laterality of the brain representation of Wernickese and its auditory and visual modalities relative to each other and to L1, BOLD signal in the LH and RH were compared across a series of ROIs established a priori (Table 2). Spherical ROIs were positioned to encompass areas strongly linked to L1 and L2: SMA, pre-SMA, dorsal premotor cortex (dPMC), ventral premotor cortex (vPMC), IFG, superior temporal gyrus (STG), middle temporal gyrus (MTG), angular gyrus, and supramarginal gyrus (see Table 2 for center and radius of these spherical ROIs). Our goal in creating these ROIs was to determine regional laterality throughout large-scale functional circuits. To this end, spherical ROIs were created and positioned so as to encompass the full extent of each of these areas on a brain template. LH and RH ROIs were symmetric mirror copies. Within each ROI, L2 laterality was then assessed relative to L1 laterality. This was done by computing the number of suprathreshhold pixels ( p < .05, uncorrected) within each ROI during L1 and L2 processing and then entering the results into the laterality equation: (L1 pixels + L2 pixels)/(L1 pixels + L2 pixels) × 100. This resulted in a laterality index (LAT) with negative values signifying greater right lateralization for L2, and positive values signifying greater left lateralization of L2 (as compared to L1). This analysis, conducted with data from all spherical ROIs, did not reveal significantly greater right lateralization for the visual L2 version of Wernickese (see Table 2, row ‘‘ALL’’). Laterality was also compared between auditory and visual groups in each ROI. This analysis failed to reveal greater right lateralization of the visual L2 as compared to the auditory L2 in all but one (middle STG/MGT) of our preestablished ROIs.

Table 2.

Comparison (Multiple t Tests) of L1 and L2 Laterality Indices (L1 − L2/(L1 + L2) × 100) in Auditory and Visual Groups within Specific a priori Regions of Interest (ROIs)

Auditory Group						Visual Group
ROI	t	p	LAT	Center	r (mm)	ROI	t	p	LAT
ALL	0.047	.39	−0.01	N/A	N/A	ALL	0.97	.24	0.12
dPMc	0.52	.33	0.13	46, −3, 46	15	dPMc	0.63	.31	−0.09
vPMc	0.43	.37	−0.07	51, −5, 16	15	vPMc	3.6	.0024a	0.64
SMA	0.19	.38	0.06	15, −15, 60	15	SMA	0.74	.29	0.11
pre-SMA	0.1	.39	−0.03	15, 17, 54	15	pre-SMA	0.17	.38	0.03
IFG	0.34	.36	0.08	46, 19, 13	17	IFG	1	.23	0.22
Insula	1.19	.19	0.28	31, 22, 5	15	Insula	0.89	.26	−0.16
ANG	0.11	.39	0.02	54, −35, 22	15	ANG	2.1	.048a	0.32
SMG	1.06	.22	0.28	50, −58, 19	15	SMG	0.35	.36	0.08
aSTG/MTG	0.13	.39	0.03	50, 2, −18	15	aSTG/MTG	0.81	.28	−0.21
mSTG/MTG	0.14	.39	0.03	50, −18, −5	15	mSTG/MTG	2.81	.012a	−0.65
pSTG/MTG	1.6	.1	0.23	50, −33, 9	15	pSTG/MTG	2.83	.012a	0.44

A negative laterality index (LAT) value signifies greater right lateralization for L2, and a positive LAT value signifies greater left lateralization of L2 in a given area. Significant areas in bold. Center = center of ROI sphere in MNI space; r = radius of ROI; ALL = all areas combined; dPMc = dorsal premotor cortex; vPMc = ventral premotor cortex; SMA = supplementary motor area; pre-SMA = presupplementary motor area; IFG = inferior frontal gyrus; ANG = angular gyrus; SMG = supramarginal gyrus; aSTG/MTG = anterior superior temporal gyrus; mSTG/MTG = middle superior temporal gyrus and middle temporal gyrus; pSTG/MTG = posterior superior temporal gyrus and middle temporal gyrus.

^aDenote areas in which L1 and L2 laterality were significantly different.

DISCUSSION

The current study identified the neural systems involved in SLA using fMRI. Analysis of learning-dependent changes in BOLD signal revealed that brain systems known to be involved in native language syntax (Broca’s area), phonotactic processing (PT), and sequence learning (SMA, PMC, putamen) are also the neural systems upon which L2 grammar acquisition and processing are built. The finding that activation in Broca’s area (BA 44) increases during syntax acquisition is consistent with the general hypothesis that Broca’s area is involved in syntactic processing for both L1 and L2 and is supported by previous imaging research involving acquisition of artificial grammars (Optiz & Friederici, 2004). More specifically, our results were consistent with the Musso et al. (2003) experiment involving acquisition of real grammatical rules. Our findings regarding the RH homologue of Broca’s area, however, departed from predictions. Rather than increases in brain activity over time, as would be predicted by Musso et al., we observed decreases in activity in this brain region across the three grammar-training fMRI sessions. This difference may be a result of differences in the learning context experienced by participants. Subjects in the Musso et al. experiment were given explicit instructions during 1-min ‘‘breaks’’ between scanning runs. Participants were presented with new grammatical rules and, ostensibly, relied heavily on episodic memory to retrieve these rules during subsequent grammaticality judgments throughout the experiment. This heavy reliance on explicit recall could be responsible for the observed increase in activation in the RH homologue of Broca’s area observed in their experiment (Kapur et al., 1994). In the current experiment, no explicit information concerning the rules of the grammar was provided; that is, participants were not given written or visual descriptions of the rules governing word order in Wernickese sentences. Subjects were forced to learn these rules implicitly. Because rule learning was based on experience, rather than the recall of specific rules, we did not expect any activity in the right IFG. What we found was a trend toward decreasing activation in this area during learning. One explanation for this effect is that participants formulated/used explicit rules concerning the structure of Wernickese sentences during the initial fMRI session, and that their reliance on these rules diminished as they became more familiar with the patterns embedded in Wernickese sentences.

The findings regarding Broca’s area are interesting from a theoretical perspective. First, these results have implications for neurocognitive theories of the function of LH and RH. Recently Jung-Beeman (2005) has proposed that the two hemispheres code similar items at different levels, with the LH coding in a fine manner (supported by small, focused ‘‘receptive fields’’) and the RH coding coarsely (supported by large, overlapping receptive fields). More specifically, Jung-Beeman claims that the IFG (bilaterally) is involved in semantic selection, that is, selecting one concept from among a number of competing concepts. According to this proposal, the initial reliance observed in the RH homologue of Broca’s area may represent the fact that learners were drawing on information from a broad range of brain systems in order to interpret the meanings of the (then) incomprehensible Wernickese sentences. Faster semantic selection required during the rapid presentation of entire sentences required that processing functions be taken over by the more focused semantic fields present in the LH, and hence, a corresponding shift in activation from the right to left IFG. Interestingly, semantic selection is only one of three proposed components of natural language comprehension, with the other two being semantic activation and semantic integration. These two components, however, are proposed to be represented in brain areas that, in the current experiment, did not show exhibit transitions toward representations in the LH as a consequence of learning. This finding suggests that SLA may critically rely upon changes in brain representations at the level of semantic selection, whereas brain areas supporting components such as semantic activation and semantic integration may be similar to those used during native language processing.

It is also interesting to note that this shift in lateralization occurred in both visual and auditory modalities. This suggests that brain mechanisms supporting the semantic integration of meaningful stimuli function in a modality-independent manner. It may be that semantic integration critically depends on the assignment/extraction of the lexical category of incoming words. This claim is supported by numerous experiments showing that detection of word category errors results in an early left anterior negativity (ELAN) originating in or around Broca’s area (Friederici, & Kotz, 2003; Hahne & Friederici, 1999; Friederici, Pfeifer, & Hahne, 1993; Neville, Nicol, Barss, Forster, & Garrett, 1991).

In addition to the IFG, significant learning-related changes in brain activation were also localized to the PT. Within the auditory group, activation within bilateral PT decreased across grammar training trials (Figures 5 and 6). However, participants in the visual group showed a markedly different pattern of PT activation across sessions, with bilateral BOLD signal increasing between fMRI Sessions 1 and 2, and then decreasing only to the left PT between Sessions 2 and 3 (Figures 5 and 6). The finding that a shift from bilateral to left lateral PT activation accompanied visual L2 grammar acquisition in the current experiment is significant for a number of reasons (Table 2). The shift from bilateral to left-lateralized PT activity provides an intriguing novel perspective on the hypothesized spatial (RH) versus temporal (LH) processing differences between the hemispheres. Here, the observed activation patterns may reveal the participants’ shift from initial nonlinguistic spatial–visual analyses of the miniature visual language (hence, the bilateral activation in Sessions 1 and 2) to more stable linguistic processing (Session 3), as would be supported by the recruitment of left-lateralized PT cortex classically associated with the sublexical (componential) analysis of the linguistic stream (Binder, Frost, Hammeke, Rao, & Cox, 1996). That this more language-like shift occurred further suggests that the participants were processing our miniature language Wernickese like a native language, engaging the classic ‘‘deep’’ processing neural sites involved in the processing of the subcomponents of language (as represented by PT activation).

That participants in the visual group showed greater left-hemisphere PT activation after language mastery demonstrates that participants were using L1 cortex to process an L2 in a maximally different modality. Moreover, the fact that changes in PT activation were not identical across auditory and visual groups suggest that the exact role of the PT (and homologous cortex in the RH) during SLA depends on the modality of the newly acquired language. Learning-dependent changes in PT activation were also found in an experiment in which participants learned to produce (as opposed to observe) gestural sentences (Newman-Norlund, Johnson, & Grafton, 2002).

The differences in activation patterns between participants in the auditory and visual groups raise tantalizing questions about the nature of PT involvement in language acquisition and language in general. For instance, why is it that right-hemisphere PT changes mirrored left-hemisphere PT changes in the auditory group? How is it possible that the PT cortex, long regarded as an auditory processing cortex, showed activation during the processing of the visual form of Wernickese?

Recent evidence suggests that the PT plays a much broader role in language processing than traditionally thought. Originally regarded as a unimodal secondary auditory cortex based on activation during processing of speech sounds (Callan et al., 2003; Zatorre, Meyer, Gjedde, & Evans, 1996), there is now evidence that PT is also involved in the processing of complex visual stimuli. For example, it is activated in visual language (reading) and visual patterned light paradigms (Finney, Fine, & Dobkins, 2001; Nakada, Fujii, Yoneoka, & Kwee, 2001, respectively), as well as in native signers when processing silent phonetic/syllabic units on the hands in natural signed languages (Petitto et al., 2000). A revised interpretation of the PT is that it is crucial to any task involving segregation and matching of specific spatio-temporal patterns found in natural languages with the goal of accessing comprehension systems located in parietal and temporal sites (Griffiths & Warren, 2002; Petitto et al., 2000). The PT, which Petitto et al. (2000) hypothesize is privileged to process temporal units occurring at a frequency of ~1 Hz, could be the neural mechanism which humans rely on to extract phonetic/syllabic units from incoming visual or auditory streams. This view may be able to explain certain aspects of the current experimental results, insofar as both the auditory and visual forms of Wernickese contained identical temporally patterned segments that were language-like. What remains the subject for future research will be to determine precisely how the PT, which receives projections from the primary auditory afferent system, and which is considered to constitute unimodal secondary auditory cortex in structure and function based on cytoarchitectonic, chemoarchitectonic, and connectivity criteria, is able to process visual signals.

The idea that structures involved in implicit learning come online during SLA is supported by our findings of increasing activity in areas known to subserve implicit sequence learning. Increases in SMA proper activation are often seen as learners become faster and more accurate at implicit motor tasks involving production of sequential movements (see Grafton, Hazeltine, & Ivry, 1998; Hazeltine, Grafton, & Ivry, 1997; Grafton, 1995; Grafton et al., 1992). Although learning-related changes in this area are usually associated with movement production, in the present experiment, SMA increases accompanied improvement at a purely perceptual task, as participants were making no articulatory movements during sentence-comprehension epochs. This difference can be reconciled by establishing that the critical common process is retrieval of syntactic information. In the same way that complex movements consist of specific actions occurring in a set order, so too Wernickese sentences were composed of linguistic elements occurring in a predefined order. Participants relied on this information to more quickly and accurately comprehend the meaning of the presented sentences. It is likely that retrieval of knowledge concerning the structural constraints of incoming linguistic streams made it easier for participants to predict upcoming words and thus speeded sentence comprehension. Take, for example, a noun phrase beginning with a determiner. The occurrence of this word category signaled to the participant that the object currently being described would have a characteristic color. This, in turn, prepared them for the occurrence of a ‘‘color’’ word at a later point in the sentence. It is our claim that SMA activity, which has previously been shown to be directly related to retrieval of sequential information in motor sequencing tasks (Bischoff-Grethe, Goedert, Willingham, & Grafton, 2003), supported this linguistic expectancy. Interestingly, observed SMA proper activation increases were left lateralized in the auditory group and bilateral in the visual group. The difference in SMA laterality observed between groups may be related to the modality in which the sentence was presented. For example, previous research suggests that bilateral SMA activation accompanies processing of visually presented linguistic stimuli (Chee, Tan, et al., 1999).

In addition to SMA increases, large portions of lateral dPMC and vPMC evinced learning-related increases in activation during grammar acquisition. Traditionally, this area was thought to subserve sensorimotor transformations required for dynamically relating sensory input to motor plans during movement, exclusively. However, recent evidence from experiments conducted by Schubotz and von Cramon (2001) and Schubotz, von Cramon, and Friederici (2001) suggests that these same transformations may be critical to any planning task requiring ‘‘anticipation of structured (nonrandom) perceptual events’’ (Schubotz, 2003, pp. 1–2). According to this hypothesis, language comprehension may be regarded as a ‘‘special case’’ of perceptual/motor matching. In a similar manner, Broca’s area may be regarded as a special case of BA 6, a functional and anatomical extension of BA 6 that is unique in its proximity to brain areas controlling language effectors (hand and motor areas of the motor cortex). In the context of this hypothesis, we suggest that premotor activity observed in our experiment is related to general perceptual/motor matching required by our sentence processing task.

Sites in the basal ganglia (putamen) showed increasing activity as participants improved at the SP task. This change occurred at bilateral sites in the visual group, and left putamen in the auditory group. This activation may be due to the learning of sequential mental operations required by L1–L2 translation. This hypothesis is supported by the following. First, learning-related increases in activation in basal ganglia sites are typically observed in experiments involving sequence acquisition (Mueller, Kleinhans, Pierce, Kemmotsu, & Courchesne, 2002). Second, putamen activity (specifically left putamen) has been observed when participants translate from a first language to a second language they acquired after age 5 (Klein et al., 1994) and may reflect increased articulatory demands associated with L2. Our results extend those of Klein et al. (1994) by suggesting that bilateral (as opposed to left) putamen activation is associated with L1–L2 translation when L2 is visual/bimanual. We believe this may be due to the bilateral nature of the gestural stimuli used in this experiment. Future experiments might address this hypothesis by examining brain activation during L1–L2 translation in native signers who have been taught a novel visual language. We also note that the observed putamen activation serves as further evidence that Wernickese was being processed as a second language.

Laterality in Auditory and Visual Groups

The second goal of the current experiment was to assess the effect of L2 modality on the representation of L2. Based on the more general distinction between LH (frequency) and RH (spatial) processing, we predicted that greater RH lateralization of activity would be observed in participants processing visual–manual Wernickese sentences as compared to participants processing aural–oral Wernickese sentences. Contrary to our predictions, the visual version of Wernickese did not rely more heavily on RH structures than the auditory version. Although L2 was more right lateralized than L1 for one ROI (middle STG/MTG), it was more left lateralized than L1 in three other ROIs (pSTG/MTG, angular gyrus, and vPMC) (Table 2). These data suggest that in native-speaking learners, even languages with high visuospatial demands come to be under the jurisdiction of LH processes as mastery is achieved.

Previous findings regarding the existence of RH activation during sign language processing have been inconclusive (Braun et al., 2001; Newman et al., 2001; Bavelier et al., 1998; Neville et al., 1998; McGuire et al., 1997). Differences in tasks used by these experimenters had been cited as one possible explanation for these conflicting results, with tasks involving pure recognition resulting in RH activation (Newman et al., 2001; Bavelier et al., 1998; Neville et al., 1998; Neville et al., 1997), whereas those involving production tasks (Braun et al., 2001; McGuire et al., 1997), overt or covert, respectively, have failed to find such activation. In the current experiment, using a purely observational task, we failed to observe greater RH lateralization in areas cited by these experiments (e.g., Broca’s area, angular gyrus, and posterior superior temporal sulcus). Although the current experiment is not directly comparable to the experiments mentioned above, it does demonstrate that miniature visual–manual second languages do not necessarily recruit RH components. Modality alone is not enough to engage RH brain areas as part of the end-state brain basis of a second language. It is perhaps more likely that RH activations observed in previous experiments are the product of topological differences in the grammar of spoken and signed languages.

Future experiments in this vein may teach children miniature visual second languages in order to build upon the adult work and extend our knowledge to understand the impact of the age of acquisition on the presence of RH involvement during visual language processing. Miniature visual languages specifically designed to incorporate this aspect of sign language might be used in future studies to address this issue.

Conclusions

The primary goal of the current experiment was to understand the neural regions that participate in adult SLA and their change over time. This was accomplished by teaching participants either an auditory or visual miniature L2. A shift in activation from the RH homologue of Broca’s area to Broca’s area proper was thought to reflect decreasing reliance on explicit rule recall resulting from neural dedication and recruitment of specialized rule-based knowledge necessary for native language as well as fast and accurate L2 processing. More importantly, these data support the idea that Broca’s area is critical to syntactic processing, while clarifying the role of its RH homologue in explicit rule processing necessary in the early stages of L2 grammar acquisition. In addition, we highlight differences between the auditory and visual groups regarding the PT, which evinced markedly different learning-dependent activation as a function of L2 modality. We suggest that the shift from bilateral to left-lateralized L2 processing observed in PT resulted from a general shift from visual–spatial processing to linguistic processing. That this area evinced learning-dependent changes in activation during the acquisition of a miniature visual language raises important questions concerning the exact role of PT in language processing. That activation in areas involved in both native language processing (Broca’s area, PT) and language translation (putamen) were implicated in the acquisition of both versions of our miniature language provides strong evidence that Wernickese was indeed being processed as a second language. This is a surprising finding considering that our L2 learners were far beyond the ‘‘critical age.’’ It may be that the observed pattern of brain activity was due to the high level of proficiency achieved in Wernickese, a level made possible by the small size of the vocabulary and the extensive training program undertaken by participants. In general, a common reliance on sequence acquisition areas (SMA, PMC, and putamen) was also observed. The recruitment of these areas during miniature L2 processing was, for the most part, found to be independent of L2 modality (although there were some laterality differences, specifically in SMA).

The secondary goal of the present experiment was to examine the effect of modality on the laterality of L2 representation. Although the visual–manual version of Wernickese was significantly more right lateralized in only one ROI, the middle STG/MTG, it was significantly more left lateralized in three other ROIs (pSTG/MTG, angular gyrus, and vPMC). These laterality data add to our current understanding of the roles of the LH and RH in language comprehension by suggesting that visual L2s do not necessarily recruit RH areas. Indeed, it may be that early experience or spatial grammar is responsible for RH activity observed in previous experiments involving sign language.

Lastly, the current experiment further demonstrates the potential usefulness of miniature languages, in conjunction with modern imaging techniques, in examining the process of language acquisition. In addition, it demonstrates the usefulness of examining the learning process itself in better understanding critical issues in the field of neurolinguistics.