Language switching decomposed through MEG and evidence from bimodal bilinguals

Significance

Bilingual individuals have the ability to switch between two languages, which requires engaging cognitive control processes to simultaneously “turn off” (disengage) one language and “turn on” (engage) their other language. The unique ability of American Sign Language (ASL)–English bilinguals to simultaneously produce a word and a sign (a code-blend) allows us to unpack these two processes (e.g., switching into a code-blend involves only turning on a language). Magnetoencephalography (MEG) evidence indicates that activating a new language does not recruit brain regions involved in cognitive control, but having to disengage from the previous language does. Additionally, this study reveals that, without motor constraints, simultaneously retrieving two words in different languages is less cognitively effortful than having to inhibit the production of the dominant language.

Abstract

A defining feature of human cognition is the ability to quickly and accurately alternate between complex behaviors. One striking example of such an ability is bilinguals’ capacity to rapidly switch between languages. This switching process minimally comprises disengagement from the previous language and engagement in a new language. Previous studies have associated language switching with increased prefrontal activity. However, it is unknown how the subcomputations of language switching individually contribute to these activities, because few natural situations enable full separation of disengagement and engagement processes during switching. We recorded magnetoencephalography (MEG) from American Sign Language–English bilinguals who often sign and speak simultaneously, which allows to dissociate engagement and disengagement. MEG data showed that turning a language “off” (switching from simultaneous to single language production) led to increased activity in the anterior cingulate cortex (ACC) and dorsolateral prefrontal cortex (dlPFC), while turning a language “on” (switching from one language to two simultaneously) did not. The distinct representational nature of these on and off processes was also supported by multivariate decoding analyses. Additionally, Granger causality analyses revealed that (i) compared with “turning on” a language, “turning off” required stronger connectivity between left and right dlPFC, and (ii) dlPFC activity predicted ACC activity, consistent with models in which the dlPFC is a top–down modulator of the ACC. These results suggest that the burden of language switching lies in disengagement from the previous language as opposed to engaging a new language and that, in the absence of motor constraints, producing two languages simultaneously is not necessarily more cognitively costly than producing one.

One of the most complex skills of the human species is to be able to quickly adapt behavior to ever-changing environmental demands. An impressive example of such an ability is multilingual individuals’ capacity to switch languages in response to the language profile of the interlocutor. This ability is generally thought to be supported by so-called cognitive control, a set of functions governing adaptive behavior. Switching between languages in response to external cues is typically associated with increased activation in dorsolateral prefrontal cortex (dlPFC) and anterior cingulate cortex (ACC) (refs. 1–6; see ref. 7 for a review). However, we do not know how this pattern of prefrontal activity relates to the subcomponents of switching, which are generally thought to consist of (at least) goal recognition, disengagement from the old task, and engagement in a new task (8).

Of these task components, goal recognition is perhaps the best understood, with several studies aiming to isolate this process by varying how far in advance a relevant cue is presented (9–11). In contrast, dissociating disengagement from the previous language and engagement in a new language has proven harder to investigate. In experimental paradigms that involve switching between languages, these two processes occur simultaneously on switch trials, preventing their isolation. However, there is one instance of bilingual language switching that allows for a natural dissociation of disengagement and engagement processes: namely, the language switching of bimodal bilinguals who are fluent in both a signed language [e.g., American Sign Language (ASL)] and a spoken language. Since the two languages of bimodal bilinguals use different articulators, it is possible to produce both a sign and a word simultaneously, and in fact this type of “code-blending” is common in bimodal bilingual communication (12). The existence of code-blending enabled us to design a picture-naming paradigm in which participants switched between producing one vs. two languages (Fig. 1A). This type of language switching naturally dissociates disengagement and engagement processes. Specifically, switching from code-blending to producing a single language requires only the disengagement of one of the languages. That is, switching from an ASL–English code-blend to producing English alone involves just “turning off” or disengaging ASL. Switching from producing one language to code-blending involves only the engagement of the other language, that is, switching from English to an ASL–English code-blend only involves engaging or “turning on” ASL. Behaviorally, it has been shown turning off a language is effortful while turning on a language is not (13). However, whether and how this dissociation is instantiated in the brain is unknown.

Fig. 1.

Experimental design for (A) the switching task and (B) the blocked task.

Additionally, this population offered the unique opportunity to study double language production, given that code-blending involves the simultaneous production of two languages. Intuitively, producing two languages at once should be harder than only producing one, but if this were the case for the language processing of bimodal bilinguals, their frequent code-blending would be a puzzle. Speaking two languages at once is physically impossible for languages that both use the mouth as the articulator—thus, it is interesting that as soon as this constraint is removed, simultaneous production of two languages is actually quite natural, although with some constraints (e.g., one language serves as the “matrix” language providing the syntax; ref. 12). This phenomenon suggests that, in principle, the human language system may be quite comfortable expressing a meaning with multiple different externalizations at once—perhaps simultaneous speaking and signing is some variant of this. Previous research has found that there is in fact no behavioral cost associated with dual lexical retrieval (14); and further, it has been found that the comprehension of simultaneous words and signs facilitates processing compared with the perception of ASL or English alone (14–16). However, how cognitive control activity is affected by the control demands posited by double production remains unclear. To examine the general relationship between producing one language vs. two, our design included not only a switching task, in which each trial contained a language cue that varied from trial to trial, but also solid blocks of English, ASL, and both (code-blends), with no switching in these blocks (Fig. 1B). Thus, we aimed to characterize both the transient (switch related) and sustained control components involved in language control (cf. ref. 17). Based on previous research, we would expect increased activity related to cognitive control when recruiting transient control components (in the switching condition) compared with applying sustained control (in the blocked condition) (18, 19).

We capitalized on the temporal resolution of MEG to track the unfolding of neural activity, both at the language cue and during stimulus presentation, to inform the precise spatiotemporal account of the individual computations involved in task switching. Source-localized neural activity was analyzed in the temporal and frontal lobes, and these analyses were supported by a multivariate pattern decoding analyses. Additionally, we performed a Granger causality analysis to determine the connectivity patterns between our regions of interest.

Results

Dissociation of Disengagement and Engagement Processes: Turning a Language “Off” Elicits Increased Prefrontal Activity, but Turning a Language “On” Does Not.

The analysis of switch type (nonswitch, switch-on, and switch-off) revealed that there is a fundamental difference between the off and on procedures involved in switching: while turning a language off required increased engagement of the dlPFC and ACC, turning a language on did not elicit any more activity than nonswitch trials. This effect was biphasic: it was first observed after the cue indicating the type of production (sign, speech) to be performed (−66–0 ms; P = 0.05), and then again after stimulus presentation (106–194 ms; P = 0.04; Fig. 2 A and B), suggesting that task reconfiguration processes begin at the point when a switch is cued, before the information required to actually retrieve the lexical item is presented. Our decoding analyses provided convergent data supporting the hypothesis that even though some flavor of language control has to mediate both turning a language on and turning a language off, the neural underpinnings of these two processes are different. Furthermore, they start to diverge ∼100 ms after a to-be-named picture is presented, temporally overlapping with our poststimulus univariate effect. Specifically, we found that our classifier could successfully decode the type of switch from 110 to 230 ms, from 245 to 265 ms, and from 315 to 490 ms (Fig. 2C). The timing of the windows at which decoding was significantly above chance reveals that the difference between turning on and off processes lasts until overt articulation has begun. However, the source localization of the activity at the peak of the classification accuracy showed that this activity emerged exclusively from the dlPFC and ACC, from more restricted although overlapping sources as our univariate analysis. Hence, the pattern analysis in our decoding procedure revealed longer-lasting although similarly grounded differences between the disengagement and engagement processes involved in task switching.

Fig. 2.

Analyses of differential activity for switch-on and switch-off trials. A and B show increases in activity for switching a language off in the left ACC and dlPFC, time-locked to the cue (A) and stimulus (B), respectively. The FreeSurfer average brains on the Left side illustrate the spatial distribution of the reliable cluster (every source that was part of the cluster at some point in time is color-coded with the sum F or t statistic). On the waveform plots, we show the time course of activity for the sources in the cluster, where 0 is the presentation of the to-be-named stimulus. The shaded regions indicate that the difference in activity between the tested conditions was significant at P = 0.05 (corrected). Significance was determined using a nonparametric permutation test (61) performed from −300–0 and 0–450 ms (10,000 permutations). The bar graphs on the Right side illustrate the average activity per condition for the sources and time points that constitute the cluster. Pairwise significance is indicated with an asterisk. C shows multivariate pattern analysis of switch type using generalization across time (62). The brains on the Left side show the source localization of the pattern weights at the peak of the classification accuracy. The time course plot indicates classifier accuracy over time, when the classifier was trained and tested on the same time point. Shading along the decoding accuracy indicates 95% confidence intervals. The graph on the Right side of C shows classifier accuracy trained and tested at every time point. D and E show pairwise conditional Granger causality (63) across all ROIs for switch-on and switch-off trials, respectively. F shows the areas that diverged in connectivity between turning a language on or off. The average brains on the Right side show the location of the reliable connections.

Finally, we ran a Granger causality analysis on our region of interest (ROI) data to determine whether the connectivity patterns between regions in our data varied from switching-on to switching-off processes. The complete set of connected regions for switch-on and switch-off processes at P < 0.01 is displayed in Fig. 2 D and E, respectively. Importantly, we found that switching a language off required reliable connectivity both between left and right dlPFC and between dlPFC and ACC, while switching a language on did not (Fig. 2F). Furthermore, Granger causality was able to establish the directionality of the connectivity, revealing that it was the dlPFC that caused subsequent activity in the ACC.

Language Inhibition vs. Dual Lexical Retrieval: Increased Prefrontal Activity for Dominant Language Suppression over Dual Language Production.

We ran a second analysis directed at testing (i) whether activity in control regions increases when participants produce two languages at once (no inhibition is required, but two lexical items must be simultaneously retrieved) and (ii) whether control demands increase when participants had to inhibit one language and produced just one language at a time. To do this, we ran one-way ANOVAs with the main factor language type [English, ASL, or code-blending (CB)] within the blocked condition and within the nonswitch trials of the switching condition.

This analysis revealed a reliable cluster of activity in the left temporal lobe during the blocked task [187–316 ms; P = 0.05], where ASL elicited increased activity over CB [t₍₂₀₎ = 2.22; P = 0.038], and CB elicited increased activity compared with English trials [t₍₂₀₎ = 4.11; P < 0.001] (Fig. 3A).

Fig. 3.

Analyses of differential activity for English, ASL, or code-blending. A and B show the univariate analysis of the MEG activity time-locked to the presentation of the stimulus, which revealed a significant cluster of activity in (A) the left temporal lobe and (B) dorsolateral and anterior prefrontal cortices, reflecting a main effect of language. The FreeSurfer average brains on the Left side illustrate the spatial distribution of the reliable cluster (every source that was part of the cluster at some point in time is color-coded with the sum F or t statistic). On the waveform plots, we show the time course of activity for the sources in the cluster, where 0 is the presentation of the to-be-named stimulus. The shaded regions indicate that the difference in activity between the tested conditions was significant at P = 0.05 (corrected). Significance was determined using a nonparametric permutation test (61) performed from −300–0 and 0–450 ms (10,000 permutations). The bar graphs on the Right side illustrate the average activity per condition for the sources and time points that constitute the cluster. Pairwise significance is indicated with an asterisk. C shows multivariate pattern analysis of language using generalization across time (62). The brain on the Left side shows the source localization of the pattern weights at the peak of the classification accuracy. The time course plot indicates classifier accuracy over time, when the classifier was trained and tested on the same time point. Shading along the decoding accuracy indicates 95% confidence intervals. The panel on the Right side shows classifier accuracy trained and tested at every time point.

The analysis of language type within the nonswitch trials of the switching condition also revealed a main effect of language type, but in this case, the cluster was located in the ACC and dlPFC [301–450 ms; P = 0.04]. This cluster also followed the same pattern: ASL trials elicited more activity than CB [t₍₂₀₎ = 4.35; P < 0.001] and English [t₍₂₀₎ = 7.99; P < 0.001], and CB also elicited reliably more activity than English [t₍₂₀₎ = 3.10; P = 0.006; Fig. 3B].

Our decoding analyses (Fig. 3C) showed that even though the semantic representations of the items to be retrieved were the same across all language response types (ASL, English, CB), we could reliably decode the language(s) in which the utterances were going to be produced. Specifically, we found distinct activity for different languages starting 260 ms after cue presentation (40 before picture presentation) and lasting until the end of the trial. Hence, we were able to successfully decode the language(s) to be produced before any lexical access processes were initiated, suggesting proactive engagement to prepare for the language(s) to be retrieved before lexical selection processes were initiated (as in refs. 20–23).

Transient and Sustained Cognitive Control During Task Switching.

Switching tasks has been hypothesized to recruit transient cognitive control, in contrast with blocked tasks, thought to rely on sustained control. However, we did not find differential activation of dlPFC or ACC between these two conditions. The 2 × 3 ANOVA on MEG data with main factors condition (blocked or switching) and language type (English, ASL, and CB) did not reveal any reliable effect of condition (all clusters P > 0.1).

Discussion

By studying a unique bilingual population, whose two languages do not employ the same articulators, this work aimed to decompose and characterize two fundamental processes involved in language switching: disengagement from the previous language and engagement in a new language. Our results revealed that, in language switching, disengagement (turning off a language) elicits increased activity in the dlPFC and ACC, but engaging in a new language (turning on a language) does not incur a neural switch cost. Additionally, by comparing the production of a single language (ASL or English) to the simultaneous production of both languages, we found that activity associated with lexical retrieval increased during simultaneous production of ASL and English compared with only producing English (our participants’ dominant language), but neural activity decreased compared with producing only ASL (the nondominant language) while inhibiting English. These results suggest that dual lexical retrieval does not necessarily incur any additional neural cost; on the contrary, neural activity during lexical retrieval of the nondominant language (ASL) is reduced by simultaneous production of English, a result that is consistent with behavioral data from ref. 13. These effect patterns were observed in prefrontal and anterior cingulate regions (during language switching) and in the temporal lobe (during the blocked task).

Bimodal Language Switching in the Context of General Models of Task Switching.

According to the predominant task-switching model proposed by ref. 24, switching costs arise from (i) inhibition of elements of the prior task or (ii) from activation of the required task set. Our results showed that, in fact, it is inhibiting elements of the prior language that causes this effect, as illustrated by increased activity in the dlPFC and ACC while turning a language off, but not while turning a language on. Since both switch-on and switch-off trials were preceded by a cue, our results also inform an additional controversial topic: whether the processing of a new cue on switch trials (i.e., cue-switching effects) may be the underlying cause of task switch effects (9–11, 25, 26). Our data suggest that cue switches do not by themselves account for switch costs, given that switch-on trials involved a cue switch but did not differ from nonswitch trials. Hence, although cue processing may have an influence on language-switching effects (e.g., more natural cues lead to smaller switch effects; refs. 1 and 3), language-switching costs cannot be fully explained by changes in cue (11).

Switch effects in our experiment were primarily localized to the ACC. This region has been long hypothesized to underlie conflict resolution generally (27–29), as well as conflict during lexical retrieval (30, 31) and language switching particularly (7, 32). Specifically, in the bilingual language control model proposed by ref. 33, the authors identify task disengagement and engagement as separate independent control processes, and suggest that a more anterior part of the ACC may be involved in language disengagement, while a more posterior part may be recruited during language engagement (32, 34). Our results provide direct evidence that their hypothesis was correct in proposing engagement of the anterior part of the ACC in disengaging from the previous task (i.e., turning a language off); however, contra their hypothesis, the process of readying the cognitive system for an upcoming language did not elicit any increased activation. The localization and timing of these switch effects were consistent with previous research on unimodal language switching in production (2, 3, 5, 6, 17), suggesting that activity associated with disengagement processes may underlie these effects as well. Furthermore, there is reason to believe that bimodal and unimodal bilinguals engage the same cognitive control processes to manage coactivation of their two languages. For example, inhibitory control ability (assessed by a Stroop task) is correlated with the amount of cross-language activation (assessed in a visual world paradigm) for both bimodal and unimodal bilinguals: bilinguals with better nonlinguistic inhibitory control exhibit less cross-language interference (35, 36).

In addition, the effect of “switching off” was temporally biphasic (manifesting both after the cue and after the picture were presented), but both clusters emerged in spatially overlapping sources, presumably indexing a reinstantiation of control processes exerted by the same mechanisms. In combination with the successful decoding of language before stimulus presentation, these results suggest (i) that bilinguals can use proactive control to prepare for upcoming language requirements (20–23), and (ii) that task reconfiguration processes begin at the point when a switch is cued, but are called upon again when the information required to actually retrieve the lexical item is presented. This contrasts with, for example, Reverberi et al. (37), who found distinct activation pattern after cue and after picture presentation. However, very likely the briefness of our cue-stimulus interval (300 ms) prevented us from capturing the differences identified in ref. 37.

Additionally, several researchers have suggested the ACC and the dlPFC are closely interconnected (38, 39), and that a flow of information between the two regions is necessary for efficient cognitive control (40, 41), including during lexical retrieval in language production (30, 31) and language control (33). However, there is little empirical evidence addressing the directionality of this connection (42). The causal nature of the Granger connectivity analysis allowed us to address this question. Our results revealed that (i) the connectivity between the dlPFC and ACC was significantly higher when participants had to disengage a language (i.e., a language was being turned off) than when they engaged an additional language, and (ii) the signals from the dlPFC were directed to the ACC during the performance of the demanding task of cued language switching. In other words, although the two regions work in coordination to successfully execute demanding control processes, our analysis indicates that it is the dlPFC that provides downstream modulation of the ACC. Some previous research suggested a reverse directionality of dlPFC–ACC activity (e.g., refs. 40 and 43). However, in those studies, researchers only tested whether ACC activity on behavioral error trials predicted activity in the PFC on the following trial (i.e., they measured whether ACC error signal correlated with dlPFC signal in a subsequent trial). It is possible then that the directionality of the activity may be context dependent: the ACC may monitor activity and send readjustment signals to the dlPFC when there has to be a behavior correction on trials following an error, but the dlPFC may issue top–down monitoring signals to the ACC when current goal information can be retrieved first (such as in cued switching tasks). This idea fits neatly within the so-called conflict-monitoring account, which suggests that the dlPFC engages the top–down processes required to identify conflict whereas the ACC resolves it (40, 41), perhaps reactively (44). Hence, our results provide evidence that Brown and Braver’s (27) model may be accurate in challenging the notion that the role of the ACC is restricted to conflict or behavioral error monitoring. However, the conflict-monitoring model in ref. 41 may be correct in describing the interaction between these two areas and the description of the role of the dlPFC as engaging top–down control preceding ACC engagement (45).

Role of Shared Modality and Cognitive Domains During Dual Language Production.

The second question addressed by our study was how neural activity associated with lexical retrieval and cognitive control would be modulated by the simultaneous production of English and ASL. Clear predictions for this follow from working memory models proposing that two actions can only be executed simultaneously if they are sufficiently dissimilar from each other (46). Since the lexical retrieval of English and ASL overlaps to a large extent (47–50), this hypothesis would predict that simultaneous production of both languages would require additional neural resources. On the other hand, an alternative hypothesis would suggest that, in the absence of external or output constraints, the suppression of the nontarget task could be more effortful than producing both, given that response inhibition has been associated with increased cognitive effort (e.g., refs. 51 and 52; for a review, see ref. 53). Our question was whether code-blending would pattern like two similar tasks that interfere with each other (54–56), or whether there is something special about double lexical production such that it is in fact relatively easy—with bimodal bilinguals’ behavior and intuitions clearly favoring the latter account.

Our results fell somewhere in-between these two predictions. Although producing both languages elicited more activity than only producing English (our subjects’ dominant language), simultaneous production required less activity than producing ASL alone. It is unsurprising that the retrieval of ASL would require more extensive cognitive engagement compared with the retrieval of English, given that ASL is the nondominant language. However, that double production elicits less activity than the production of ASL (but not English) alone was more unexpected. There are two plausible explanations for this pattern of results. On the one hand, the production of English may prime ASL in a code-blend, making the retrieval of ASL faster in this situation (14). On the other, since double production does not necessitate the inhibition of the dominant language, increased activity for ASL may be a consequence of the inhibition applied to English—the dominant language. It is also possible that both of these hypotheses are valid to some extent, and that the most accurate explanation is a combination of the two. Unfortunately, our paradigm does not allow us to tease apart these different possibilities.

This pattern of activity was found both during the switching condition and the blocked condition. However, the location of the effects varied. When code-blends were produced in the blocked condition, in which overall executive control demands were lower than in the switching condition, the locus of the increased activity for ASL production was localized in the temporal lobe. The location and timing (∼200–300 ms) of this effect neatly fit within models of lexical access and retrieval (ref. 57; for a review, see ref. 58). This result suggests that in the case of language, the relative effort associated with retrieving the nondominant language (ASL) may be partially alleviated by the synchronous use of the dominant language (English). Perhaps in the absence of competition for the articulators, all lemmas may be active and dual lexical retrieval may be less effortful than inhibiting the dominant response for that stimulus. This proposal is in line with studies on the perception of code-blends, which show that redundant semantic content leads to more efficient neural processing in modality-specific regions (16).

During the switching condition, in which transient cognitive control was required, we found that the same pattern emerged from the prefrontal regions associated with cognitive control instead of the temporal lobe: ASL productions elicited increased activity compared with code-blends and to English alone. This pattern may be in some ways similar to that observed during a Stroop task, where the prefrontal cortices are engaged to exert sufficient top–down support of color-naming pathways to avoid interference from the dominant response: word reading (59, 60). Hence, these results suggest that when the task demands on cognitive control are already high, such as during the switching task, the prefrontal regions need to be more intensely recruited to exert top–down control that results in successful inhibition of the dominant language response.

Conclusion

These findings show that, during language switching, disengaging from the previous language requires executive control, whereas engaging in an additional language is relatively cost-free. Additionally, we characterized an instance of multitasking in which simultaneously producing two languages that rely on the same conceptual system not only is possible but requires decreased engagement of language control areas compared with inhibiting a dominant response. In congruence with the connectivity analyses, which showed that when exogenous information allows for top–down executive control, neural activation flows from the dlPFC to the ACC, we provide a step toward a full characterization of the specific contributions of the dlPFC and ACC to the common and much investigated language-switching behavior of multilingual individuals.