Parallel language activation and inhibitory control in bimodal bilinguals

Abstract

Findings from recent studies suggest that spoken-language bilinguals engage nonlinguistic inhibitory control mechanisms to resolve cross-linguistic competition during auditory word recognition. Bilingual advantages in inhibitory control might stem from the need to resolve perceptual competition between similar-sounding words both within and between their two languages. If so, these advantages should be lessened or eliminated when there is no perceptual competition between two languages. The present study investigated the extent of inhibitory control recruitment during bilingual language comprehension by examining associations between language co-activation and nonlinguistic inhibitory control abilities in bimodal bilinguals, whose two languages do not perceptually compete. Cross-linguistic distractor activation was identified in the visual world paradigm, and correlated significantly with performance on a nonlinguistic spatial Stroop task within a group of 27 hearing ASL-English bilinguals. Smaller Stroop effects (indexing more efficient inhibition) were associated with reduced co-activation of ASL signs during the early stages of auditory word recognition. These results suggest that the role of inhibitory control in auditory word recognition is not limited to resolving perceptual linguistic competition in phonological input, but is also used to moderate competition that originates at the lexico-semantic level.

1. Introduction

Previous research has suggested that bilinguals with two spoken languages may develop selective advantages in nonlinguistic cognitive control abilities compared to monolinguals, for instance in conflict monitoring, conflict resolution, and task-switching (e.g., Bialystok, Craik, Klein, & Viswanathan, 2004; Bialystok, Craik, & Luk, 2008; Hernández, Costa, Fuentes, Vivas, & Sebastián-Gallés, 2010; Kushalnagar, Hannay, & Hernandez, 2010; Prior & MacWhinney, 2010; Salvatierrra & Rosselli, 2011). One possible explanation for these advantages is that bilinguals engage domain-general cognitive control mechanisms to manage the cognitive demands of bilingual language processing (e.g., Blumenfeld & Marian, 2011; Linck, Schwieter, & Sunderman, 2012; Pivneva, Palmer, & Titone, 2012; Prior & Gollan, 2011; Soveri, Rodriguez-Fornells, & Laine, 2011). Over time, growing experience with managing these demands might enhance nonlinguistic cognitive control abilities (e.g., Blumenfeld & Marian, 2013; Luk et al., 2011; Singh & Mishra, 2012).

One such cognitive demand that bilinguals commonly experience is cross-linguistic competition during auditory word recognition. For bilingual listeners, auditory input in one language activates possible word candidates regardless of language membership (e.g., Marian & Spivey, 2003a, 2003b). This input-driven language co-activation is observed across different proficiency levels, ages of onset of language acquisition, and highly diverse language pairs (e.g., Blumenfeld & Marian, 2007, 2013; Canseco-Gonzalez et al., 2010; Cutler, Weber, & Otake, 2006; Ju & Luce, 2004; Marian, Blumenfeld, & Boukrina, 2008; Weber & Cutler, 2004). Resolving such cross-linguistic competition has been posited to require cognitive inhibition skills (e.g., Green, 1998; Shook & Marian, 2013). For example, Blumenfeld and Marian (2011) showed that, in Spanish-English bilinguals but not monolinguals, efficiency of nonlinguistic conflict resolution (as measured by a spatial Stroop task) was associated with inhibition of English within-language phonological distractors after word identification. The authors suggested that the bilingual participants routinely engage domain-general cognitive control mechanisms to resolve linguistic conflict because they must control activation of a second language, perhaps increasing overall involvement of cognitive control mechanisms during language processing (cf. Mercier, Pivneva, & Titone, 2014). In a more recent study, Blumenfeld and Marian (2013) showed that efficient conflict resolution was indeed associated with how unimodal bilinguals manage between-language activation during auditory word recognition (also see Mercier et al., 2014). For Spanish-English bilinguals, better performance on a nonlinguistic spatial Stroop task was associated with increased cross-linguistic activation during the early stages of word recognition (300-500 ms after word-onset) and decreased cross-linguistic activation during later stages of word recognition (633-767 ms after word-onset). That is, better inhibitory control was associated with earlier cross-linguistic distractor activation, followed by efficient resolution of such competition.

Blumenfeld and Marian (2011, 2013) suggested that the association between perceptual linguistic competition and Stroop-type inhibition for bilinguals may reflect similar underlying cognitive mechanisms. Specifically, both tasks involve processing bivalent perceptual aspects of the same stimulus (e.g., cat-cap upon hearing ca-), i.e., they represent perceptual conflict. Indeed, neuroimaging studies have shown that the neural substrates for Stroop-type inhibition and bilingual language control are largely shared (Abutalebi, 2008; Liu, Banich, Jacobson, & Tanabe, 2004), and that bilingual experience modulates these neural substrates (e.g., Luk, Anderson, Craik, Grady, & Bialystok, 2010). Bimodal bilinguals (i.e., bilinguals with a spoken and a signed language) do not experience within-modality perceptual competition between their languages. Therefore, the absence of Stroop-type advantages in bimodal bilinguals may serve as additional evidence that recruitment of inhibition in bilingual comprehension is linked to perceptually generated competition (Emmorey, Luk, Pyers, & Bialystok, 2008).

The present study investigates whether the recruitment of Stroop-type inhibitory control mechanisms during bilinguals’ auditory word recognition exclusively depends on perceptual competition in phonological input. We do this by examining the association between nonlinguistic inhibitory control and language co-activation for bimodal bilinguals. For such bilinguals, the two languages have completely distinct, non-overlapping phonological systems, and co-activation through perceptual overlap in linguistic input is therefore not possible. As a result, if bimodal bilinguals engage inhibitory control to resolve cross-linguistic competition, then it suggests that recruitment of cognitive control processes is not exclusively driven by perceptual conflict.

1.1 Cross-linguistic activation and language-cognition interactions in bimodal bilinguals

Despite the absence of overlap at the phonological level, there is some evidence for co-activation between a spoken and a sign language during bilingual language processing for deaf and hearing bimodal bilinguals, possibly through top-down conceptual and lateral lexical connections between the two languages (i.e., cross-linguistic competition between lexico-semantic representations). Morford, Wilkinson, Villwock, Piñar, and Kroll (2011) found that phonological overlap between sign translation equivalents affected semantic judgments to written English word pairs in deaf ASL-English bilingual adults. Semantically related word pairs (e.g., apple and onion) were judged more quickly when their ASL sign translation equivalents overlapped in sign phonology (the ASL signs APPLE and ONION overlap in all phonological features except location). Furthermore, semantically unrelated word pairs were judged more slowly when their ASL sign translation equivalents overlapped in sign phonology (also see Kubus, Villwock, Morford, & Rathmann, 2014; Ormel, Hermans, Knoors, & Verhoeven, 2012).

Shook and Marian (2012) examined co-activation of signs during spoken word recognition instead of written word recognition in an eye-tracking study with hearing ASL-English bimodal bilinguals. They used a bilingual visual world paradigm to present participants with spoken words while they were looking at displays with four pictures: the target picture (that matched the spoken word) and three distractor pictures. Some of the displays included a picture of a cross-linguistic phonological distractor, for example a picture of ‘paper’ in a trial with the English target word cheese. Although cheese and paper are phonologically unrelated in English, the ASL signs for CHEESE and PAPER share the same location and handshape features and only differ in movement features. ASL-English bilinguals looked more at the cross-linguistic distractor than at unrelated distractors in the first 500ms post word-onset, suggesting they were co-activating ASL signs in the English listening experiment (see Van Hell, Ormel, Van der Loop, and Hermans (2009) for evidence of co-activation in the opposite direction, that is, spoken word activation during sign processing by sign language interpreters in training).

Given that bimodal bilinguals co-activate spoken and signed lexical items during auditory word recognition in the absence of perceptually driven linguistic competition, they might also engage nonlinguistic inhibitory mechanisms to resolve cross-linguistic competition that originates at the lexico-semantic level. However, direct links between bimodal bilingual language processing and executive function have not been examined and findings regarding possible enhancements in nonlinguistic cognitive control abilities of bimodal bilinguals have been mixed so far.

Emmorey, Luk, et al. (2008) compared the performance of hearing ASL-English bilinguals who had learned ASL from an early age as CODAs (i.e., children of deaf adults), unimodal bilinguals who learned two spoken languages from an early age, and English monolinguals on a conflict resolution task (an Eriksen flanker task). The researchers found that, whereas the unimodal bilinguals were faster than the other two groups, the bimodal bilinguals did not differ from the monolinguals, suggesting that (hearing) bimodal bilinguals may not experience the same advantages in cognitive control as unimodal bilinguals. To explain these results, Emmorey, Luk, et al. (2008) suggested that the enhanced executive control observed for unimodal bilinguals might stem from the need to attend to and perceptually discriminate between two spoken languages, whereas perceptual cues to language membership are unambiguous for bimodal bilinguals. Furthermore, the researchers argued that the possibility for bimodal bilinguals to produce signs and words concurrently (code-blending) places lower demands on language control than for unimodal bilinguals, because less monitoring is required to ensure that the correct language is being selected.

Indeed, hearing bimodal bilinguals frequently code-blend in conversations with other bimodal bilinguals (Emmorey, Borinstein, Thompson, & Gollan, 2008), and sometimes even in conversations with non-signers (Casey & Emmorey, 2009). Interestingly, bimodal bilinguals prefer code-blending to code-switching, that is, switching between speaking and signing that would likely require inhibition of the non-target language (Emmorey, Borinstein, et al., 2008). Emmorey, Petrich, and Gollan (2012) compared picture-naming times for ASL-English code-blends compared to English words and ASL signs alone and found that, although code-blending slowed English production because participants synchronized ASL and English articulatory onsets, code-blending did not slow ASL retrieval. Furthermore, during language comprehension (indexed by a semantic decision task), code-blending facilitated lexical access, as compared to either language alone. Bimodal bilinguals are thus able to simultaneously access lexical signed and spoken items seemingly without additional processing costs. Since cross-linguistic inhibition has been associated with processing costs (e.g., Meuter & Allport, 1991), this suggests that bimodal bilinguals may not inhibit their other language to the same degree as unimodal bilinguals.

Despite evidence against extensive recruitment of inhibitory control during bimodal bilingual language processing, cognitive control may nevertheless guide some aspects of processing in ASL-English bilinguals. For example, Kushalnagar, Hannay, and Hernández (2010) compared the performance of balanced and unbalanced deaf ASL-English bilingual adults on a selective attention task and an attention-switching task. Whereas the two groups performed similarly on the selective attention task, the balanced bilinguals performed better than the unbalanced bilinguals on the attention-switching task, suggesting that there might be enhancements in cognitive flexibility for bimodal bilinguals who are highly proficient in both languages. However, this study did not include comparison samples of unimodal bilinguals or monolinguals. Another study tested ASL simultaneous interpreter students on a battery of cognitive tests at the beginning of their program and two years later (MacNamara & Conway, 2014). The interpreter students improved on measures of task switching, mental flexibility, psychomotor speed, and on two working memory tasks that required the coordination or transformation of information (but not on working memory tasks requiring the storage and processing of information or on a task measuring perceptual speed). While suggestive of a modulating effect of bimodal bilingual interpreting experience on the cognitive system, this study also did not include monolingual or unimodal bilingual controls, which leaves open the possibility that these improvements came about for reasons other than increased experience with bilingual language management demands.

1.2 The current study

The aim of the present study was twofold. The primary goal was to investigate competition mechanisms during auditory word recognition in bilinguals (Blumenfeld & Marian, 2013; Mercier et al., 2014), by examining whether bimodal bilinguals engage inhibitory control to resolve cross-linguistic competition between languages without overlap in phonological input. The secondary goal was to replicate findings of parallel language activation in hearing bimodal bilinguals. Although several studies identified co-activation of a signed and a written language in deaf bimodal bilinguals (Kubus et al., 2014; Morford et al., 2011), only one published study so far has shown co-activation between a spoken and a signed language in hearing bimodal bilinguals (Shook & Marian, 2012). To this end, we examined both language co-activation during auditory word recognition and nonlinguistic conflict resolution in a group of hearing ASL-English bilinguals. Further, we then directly linked individual differences in inhibitory control to the degree and time-course of cross-linguistic competition. More specifically, we used a bilingual visual world eye-tracking paradigm to index language co-activation in bimodal bilinguals (based on Shook & Marian, 2012), and a nonlinguistic spatial Stroop task to index inhibitory control ability which can be linked to individual co-activation patterns (Blumenfeld & Marian, 2011, 2013).

If the association between cross-linguistic competition and nonlinguistic inhibitory control abilities during auditory word recognition is exclusively related to underlying similarities in the resolution of perceptually driven conflict, then bimodal bilinguals should not show an association between language co-activation and performance on the spatial Stroop task. In this case, such an association should only be found in the context of cross-linguistic activation between two languages with overlapping phonological input within a single modality (Blumenfeld & Marian, 2013). Alternatively, if inhibitory control mechanisms are recruited to resolve cross-linguistic competition that originates at non-perceptual (e.g., lexico-semantic) levels of processing during auditory word recognition, then bimodal bilinguals are expected to show a similar association between language co-activation and performance on the spatial Stroop task.

2. Methods

2.1 Participants

Twenty-seven proficient bilingual users of English and ASL (15 females, mean age = 27.8 years, SD = 8.4) participated in the study. Twenty participants were Children of deaf adults (Codas) and had learned ASL from an early age. The other seven participants had learned ASL as a second language as adults (L2 learners, mean age of exposure: 18.3 years, range = 15 – 21). Self-rated ASL proficiency (on a 1-7 scale for signing and understanding) and information on language exposure and socioeconomic status (Hollingshead, 1975) were collected with a language background questionnaire. In addition, scores on the ASL Sentence Reproduction Test (Supalla, Hauser, & Bavelier, 2014) were available for 20 bimodal bilinguals (14 Codas and 6 L2 learners). Self-proficiency ratings and ASL-SRT scores did not differ significantly between the Codas and L2 learners (non-parametric Mann-Whitney U, all ps > .24). Two additional bilingual participants were tested, but excluded from the study because they rated their ASL proficiency less than 5 (out of 7). English receptive vocabulary skill was measured with the Peabody Picture Vocabulary Test (Dunn & Dunn, 1997) and nonverbal reasoning was measured with the Matrices subtest of the Kaufman Brief Test of Intelligence (Kaufman & Kaufman, 2004). English receptive vocabulary was tested to ensure that the bimodal bilinguals were English-dominant and that their English receptive vocabulary knowledge was similar to that of English monolinguals, in order to exclude the possibility that correlations with inhibitory control simply reflected group differences in English word comprehension abilities.

Twenty-seven monolingual users of English (23 females, mean age = 26.0 years, SD = 7.0) were selected (out of a total of 30 who were tested), that matched the bilingual sample as closely as possible on age, number of years of education, socioeconomic status, English receptive vocabulary, and nonverbal intelligence (see Table 1). Three additional monolingual participants were tested, but excluded because of either too much ASL exposure (N = 1), less than 75% overall accuracy on the spatial Stroop task (N = 1), or missing background information (N = 1). The monolingual group was included to exclude the possibility that observed co-activation for the bimodal bilinguals was an artifact of the selected visual stimuli, as well as to confirm that any associations between the degree and time course of (cross-linguistic) activation for bimodal bilinguals and nonlinguistic cognitive control abilities were indeed specific to the bilingual participants. Descriptive statistics for the bilingual and monolingual participants are provided in Table 1.

Table A.1.

Target, cross-linguistic distractor and matched unrelated distractor triplets.

Target	Freq.	Cross-linguistic distractor	Freq.	Matched unrelated distractor	Freq.
alligator	2.10	hippo	1.60	Dresser	2.20
bread	2.96	wood	2.88	Flashlight	2.32
broom	2.26	pie	2.90	Scarf	2.22
butter	2.83	soap	2.71	Hammer	2.60
candy	2.96	apple	2.84	Stapler	1.49
chair	3.19	train	3.25	Ball	3.32
cheese	3.05	paper	3.42	Stamp	2.35
chocolate	2.95	island	2.84	Leaf	2.34
clown	2.65	wolf	2.58	Match	3.19
glasses	3.00	camera	3.14	Wrench	2.11
gorilla	2.20	bath	3.00	Pirate	2.21
knife	3.10	egg	2.86	Cake	3.08
movie	3.32	school	3.66	Car	3.71
napkin	2.14	lipstick	2.52	Dinosaur	2.07
newspaper	2.88	magnet	1.98	Dress	3.34
nurse	3.04	sushi	2.16	Bear	3.18
owl	2.14	binoculars	1.85	Thumb	2.61
parachute	2.08	mushroom	1.92	Envelope	2.51
pig	3.02	frog	2.43	Mitten	0.95
poison	2.84	bone	2.93	Fish	3.23
potato	2.59	church	3.13	Door	3.72
screwdriver	1.93	key	3.34	Orange	2.84
shower	3.11	lamp	2.59	Beard	2.60
skunk	2.03	lion	2.52	Battery	2.60
subway	2.53	iron	2.79	Banana	2.51
thermometer	1.90	carrot	2.09	Gum	2.65
umbrella	2.32	coffee	3.48	Peach	2.33
witch	2.68	doll	2.81	Mirror	2.94
M	2.64		2.72		2.62
SD	0.43		0.51		0.60

Note. Spoken word frequency (log-10) obtained from SubtLex-US (Brysbaert & New, 2009)

2.2 Materials

A visual world eye-tracking paradigm was used to measure cross-linguistic activation of ASL during English auditory word recognition in the ASL-English bimodal bilinguals. Twenty-eight sign pairs were selected that were highly similar on three out of the four major sign parameters, i.e., handshape, location, movement and orientation of the hand (see Appendix A). Twenty of these pairs had previously been used by Shook and Marian (2012). Target, cross-linguistic distractor and unrelated objects were represented by black-and-white line drawings obtained from Shook and Marian (2012) and from the International Picture Naming Database (Székely et al., 2004). Isolated auditory target words were recorded at 44.1 kHz, 32 bits by a female, monolingual speaker of English and amplitude-normalized.

Target and cross-linguistic distractor items did not significantly differ in English frequency (SubtLex-US Brysbaert & New, 2009), phoneme length and concreteness ratings (all ps >.09) and did not overlap in English phonology. A set of 28 unrelated items was chosen as matched unrelated distractor items in the statistical analyses (see Appendix A). These unrelated distractor items did not significantly differ from the target and cross-linguistic distractor item in English frequency, phoneme length and concreteness (all ps > .11). In addition, 15 English monolinguals (10 females, mean age = 26.5 years, SD = 9.3) who did not take part in the eye-tracking study rated the target - cross-linguistic distractor and target - unrelated distractor pairs for semantic similarity and visual similarity on a 1-7 scale (ranging from “not similar at all” to “very similar”) to ensure that any observed co-activation could not be explained by semantic or visual relationships between the targets and cross-linguistic distractors. Mean semantic similarity ratings were low for both the target - cross-linguistic distractor pairs (2.0; SD = 0.5) and the target - unrelated distractor pairs (1.7; SD = 0.4). Mean visual similarity ratings for these pairs were also low, with 2.6 (SD = 0.6) and 1.6 (SD = 0.4), respectively. Although the remaining unrelated distractor items did not differ significantly from the cross-linguistic distractor item in English frequency, phoneme length and concreteness (all ps > .22), they were marginally higher in English frequency (p =. 07) and marginally shorter in phoneme length (p = .08) than the target items. Mean semantic similarity and visual similarity ratings for these remaining unrelated distractor items and the target items were 1.4 (SD = 0.2) and 1.7 (SD = 0.4), respectively.

The eye-tracking design consisted of twenty-eight critical trials in which displays were presented that contained an image of the target item, a cross-linguistic distractor item, a matched unrelated distractor item and another unrelated distractor item. The location of target, cross-linguistic distractor and unrelated distractor images on the displays was counterbalanced across trials. In addition, 88 filler trials were presented, which contained a target and three unrelated distractors. This resulted in a total of 116 eye-tracking trials. Filler trials were not analyzed. The target, cross-linguistic distractor and unrelated distractors from the critical trials were all presented two times in the experiment (cf. Shook & Marian, 2012). The target and unrelated distractors were repeated in the same location in a subset of the filler trials in which the cross-linguistic distractor item had been replaced with another unrelated distractor item. The cross-linguistic distractor item appeared once in the critical trial and another time as the target in a filler trial with three unrelated distractors. See Figure 1 for examples of critical and filler trials.

Figure 1.

Examples of displays for critical and filler trials in the eye-tracking experiment. Critical trials included the target item (chair), a cross-linguistic distractor item (train), a matched unrelated distractor item (ball) and another unrelated distractor item (goat). In a subset of the filler trials, the cross-linguistic distractor item from the critical trial was replaced with an unrelated distractor item in the same location (dog). The cross-linguistic distractor appeared another time as the target in a filler trial with three unrelated distractor items. Participants responded by pressing one of four buttons on a button box that corresponded to the four picture quadrants in the displays.

For each visual display, the black-and-white images were presented in the four corners of a 3 × 3 grid (1280 × 1024 pixels) with a fixation cross in the center. Each trial was preceded by a fixation period of 500 ms and a 1500 ms inter-stimulus interval before presentation of the visual display, which remained visible until a response had been given by the participant. The target word was presented 600 ms after onset of the visual display. Participants responded by pressing one of four colored buttons on a button box in front of them that corresponded with the four positions of the images on the display. The experiment was divided into four blocks of 29 trials each, preceded by five practice trials. Trial presentation in each block was pseudo-randomized to ensure that each critical trial was separated by one or more filler trials. Furthermore, half of the critical trials preceded the corresponding filler trial in which the cross-linguistic distractor was substituted by an unrelated distractor, and half followed the corresponding filler trial, and critical trials and corresponding filler trials were never presented in the same block. Two different block orders were used and counterbalanced across participants.

2.3 Spatial Stroop task

The nonlinguistic spatial Stroop task used in the present study as a measure of inhibitory control was adapted from the task used by Blumenfeld and Marian (2011, 2013) and originally adapted from Peterson et al. (2002) and Liu, Banich, Jacobson, and Tanabe (2004). In this task, a black arrow is presented on the screen and participants are asked to respond to its direction (either left or right) while ignoring its location on the screen (either left, right or center). The stimulus dimensions, direction and location, were combined in different ways to create baseline trials, congruent trials and incongruent trials (see Figure 2). Baseline trials (42) consisted of a right- or leftward-facing arrow in the center of the screen. Congruent trials (126) consisted of a leftward-facing arrow on the left side of the screen or a rightward-facing arrow on the right side of the screen. Incongruent trials (42) consisted of a leftward-facing arrow on the right side of the screen or a rightward-facing arrow on the left side of the screen. The ratio of congruent to incongruent trials was 3:1. Reaction times for the baseline condition were subtracted from the reaction times for the incongruent condition to obtain a measure of efficiency of inhibitory control (the Stroop effect).

Figure 2.

Illustration of baseline, congruent and incongruent trials in the spatial Stroop task.

Direction (left, right) and location of the arrow (center, left, right) were counterbalanced for baseline, congruent and incongruent trials. Each trial started with a 500 ms central fixation cross, followed by presentation of the stimulus display until participants gave a response (with a 1200 ms time-out) and ended with a 500 ms blank screen before the next trial started. Participants gave a response by pressing a left or right button on a button box in front of them. The task was divided into two blocks separated by a brief pause, and was preceded by 20 practice trials (4 baseline, 4 incongruent, 12 congruent). Trial presentation was pseudo-randomized such that there were not more than three consecutive left or right responses. Furthermore, an equal number of the baseline and incongruent trials were stay trials (8) and switch trials (34). Twenty-eight of these switch trials involved a switch from a congruent trial (into a baseline or incongruent trial). The remaining six switch trials were between the baseline and the incongruent condition.

2.4 Procedure

Both the bilingual and monolingual participants received verbal and written instructions in English. The target words were presented through headphones. Participants’ eye movements were recorded at a rate of 1000Hz using an SR Research Eyelink^® 1000 tower system with chin rest support to stabilize the head, and in combination with Experiment Builder stimulus presentation software (SR Research^©). The distance between the participants’ eyes and the screen was approximately 70cm. A 12-point calibration and validation check were performed at the beginning of the experiment and again after the instructions and practice trials. If needed, re-calibration took place after short breaks in between the different blocks of the experiment. All participants completed the word recognition task before the spatial Stroop task. Participants were instructed to keep their fingers on the relevant response buttons during the task to be able to respond without looking down.

2.5 Data analysis

Behavioral data (accuracy and reaction time) from the spatial Stroop task were analyzed with Analysis of Variance methods using IBM^© SPSS^© 21. To analyze the eye movement data from the word recognition task, Eyelink’s built-in software was used to parse fixations from the gaze position data through standard parsing algorithms that track changes in velocity and position. Fixations were automatically categorized in interest areas that corresponded with the four quadrants of the grid that contained the images. The proportion of fixations to target items, cross-linguistic distractor items and unrelated distractor items during critical trials was analyzed in 20ms bins starting at 200ms post word-onset (the approximate time it takes to plan an eye-movement) until 700ms post word-onset. To analyze the time-course of activation, Growth Curve Analyses (Mirman, Dixon, & Magnuson, 2008) were conducted using the statistical software program R v.3.02 (R Development Core Team, 2007) and the lme4 package v.1.1-7 (Bates, Maechler, Bolker, & Walker, 2013). Normal approximation (z-distribution) was used to estimate p-values.

To examine if individual differences in inhibitory control were associated with the degree and time-course of language co-activation during auditory word recognition for bimodal bilinguals, correlations were analyzed between individual Stroop effects and proportion of looks to cross-linguistic distractors for each 20 ms time frame in pre-specified time-windows (for similar approaches, see Blumenfeld & Marian, 2013; Costa, Strijkers, Martin, & Thierry, 2009).

All participants completed a naming task after the eye-tracking task. Because we were interested in the relationship between nonlinguistic cognitive control mechanisms and early automatic co-activation of ASL signs, we only included critical trials with target - cross-linguistic distractor pairs in the eye-gaze analyses that were readily named by individual bimodal bilingual participants in a separate picture-naming experiment (Giezen & Emmorey, in press) . That is, trials were excluded if the participant did not produce one of the expected phonologically overlapping signs for the target and cross-linguistic distractor items, with the rationale that spontaneous production of the intended sign correlated with strong knowledge of it and thus automatic activation of it during comprehension. Note that this does not necessarily mean that these participants did not know the sign for that target or cross-linguistic distractor item, but only that they did not readily produce the intended phonologically-related sign or preferred to use fingerspelling for that item. Regional variation is ubiquitous in ASL (Lucas, Bayley, & Valli, 2003), and many concepts have different possible signs. Although all participants were tested in San Diego, many had moved there from other places and hence used variable signs for our target and cross-linguistic distractor items. Based on the production data, nine trials on average were excluded per participant, resulting in nineteen remaining trials on average per participant, which is similar to the number of critical trials (20) used in the original study by Shook and Marian (2012). Exclusion of incorrectly answered critical trials resulted in omission of 0.3% of data for the eye gaze-analyses.

For the analysis of response times on the spatial Stroop task, incorrect trials were excluded. Furthermore, trials were excluded with response times 2.5 or more standard deviations above or below the mean response time (across baseline, congruent and incongruent trials) for each individual participant. After excluding incorrect trials, this resulted in further exclusion of ~2.5% of the data for both bilingual and monolingual participants.

3. Results

On the word identification eye-tracking task, overall accuracy was 97.8% correct (SE = 0.2%) for bilingual participants and 97.8% correct (SE = 0.1%) for monolingual participants across critical and filler trials. Accuracy rates for critical trials only were 99.5% (SE = 0.3%) for bilingual participants and 99.6% (SE = 0.2%) for monolingual participants. Mean response time across all trials was 1,500 ms (SE = 30 ms) for bilingual participants and 1,449 ms (SE = 83 ms) for monolingual participants.

3.1 Growth curve analyses

We used growth curve analyses to examine the time course of fixations on the cross-linguistic distractor items and matched unrelated distractor items from 200 until 700 ms post word-onset. Visual inspection of the time course suggested that this was the time-window where eye-movements to targets and distractors varied before stabilizing (see Figure 3). The time course of proportion of cross-linguistic distractor and matched unrelated distractor fixations for bimodal bilinguals and monolinguals was modeled with orthogonal polynomials (first and second order time terms). To determine how fixations changed over time as a function of participants and conditions, the model included a fixed effect of Distractor (matched unrelated distractor, cross-linguistic distractor), and random-effects of Participants and Participants X Distractor on all time terms. The distractor factor was contrast-coded using deviation coding, with the proportion of fixations to cross-linguistic distractors over time contrasted with the listeners’ fixations to the matched unrelated distractors, which was present on the same display as the competing item. Group membership (bimodal bilinguals vs. monolinguals) was not included in this analysis because of the large number of model parameters this comparison would require in relation to the small sample size, and because of the overall higher fixation proportions by the monolinguals, regardless of condition (see Fig.3). That is, the monolinguals tended to fixate all pictures more than the bimodal bilinguals, including the cross-linguistic distractor. To further account for group differences in the overall number of fixations, the time course of the difference of the proportion of cross-linguistic distractor and matched unrelated distractor fixations was also modeled. This model included a fixed effect of Group (monolinguals, bimodal bilinguals), and random-effects of Participants on all time terms. In addition to by-participant analyses, parallel by-item analyses were conducted in which the random-effects of Participants in the models were replaced by random-effects of Items. Variance of the random-effects in the analyses is reported in Appendix B.

Figure 3.

Proportion fixations on target, cross-linguistic distractor and matched unrelated distractor items for bimodal bilinguals (left panel) and monolinguals (right panel). Error bars indicate ±1 SE.

For the bilinguals, the by-participant growth curve analysis yielded a significant effect of Distractor on the intercept term (Estimate = .029, SE = .010, p < .01), reflecting a higher overall proportion of looks to the cross-linguistic distractor than to the matched unrelated distractor. In addition, a significant effect of Distractor on the quadratic term was observed (Estimate = −.094, SE = .032, p < .01), indicating sharper curvature for cross-linguistic distractor fixations. The estimated coefficients for all the parameters in the by-participant model for the bilingual data are presented in Table 2 and the observed data and model fits are illustrated in Figure 4. The by-item analysis yielded effects of Distractor on the quadratic time term (Estimate = −.090, SE = .044, p < .05). For the monolingual participants, as expected, neither the by-participant nor by-item analysis yielded a significant effect of Distractor on the intercept term or any of the time terms (all ps > .15), suggesting that monolinguals treated cross-linguistic distractors and matched unrelated distractors similarly. The estimated coefficients for all the parameters in the by-participant model for the monolingual data are presented in Table 3.

Table B.1.

Variance of the random effects in the growth curve analyses.

Model	Group	Random-effect	Model term	Variance
Fixed effect of Distractor (by-participants)	Bimodal bilinguals	Participants	(Intercept)	.002
			ot1	.007
			ot2	.002
		Participants:Distractor	(Intercept)	.001
			ot1	.035
			ot2	.002
		Residual	--	.001
Fixed effect of Distractor (by-participants)	Monolinguals	Participants	(Intercept)	.001
			ot1	.003
			ot2	.001
		Participants:Distractor	(Intercept)	.002
			ot1	.021
			ot2	.009
		Residual	--	.001
Difference curve (by-participants)	--	Participants	(Intercept)	.003
			ot1	.063
			ot2	.023
		Residual	--	.003
Fixed effect of Distractor (by-items)	Bimodal bilinguals	Items	(Intercept)	<.001
			ot1	.001
			ot2	<.001
		Items:Distractor	(Intercept)	.002
			ot1	.034
			ot2	.026
		Residual	--	.002
Fixed effect of Distractor (by-items)	Monolinguals	Items	(Intercept)	.001
			ot1	<.001
			ot2	<.001
		Items:Distractor	(Intercept)	.003
			ot1	.053
			ot2	.014
		Residual	--	.001
Difference curve (by-items)	--	Items	(Intercept)	.003
			ot1	.083
			ot2	.029
		Residual	--	.008

Note. ot1=linear time term, ot2=quadratic time term

Figure 4.

Observed data and model fits (lines) for fixations on cross-linguistic distractor and matched unrelated distractor items for bimodal bilinguals (left panel) and monolinguals (right panel). Error bars indicate ±1 SE.

Table 1.

Descriptive statistics for the bimodal bilingual and monolingual participants.

	Bimodal bilinguals	Monolinguals	t test
Age (years)	27.8 (8.4)	26.0 (7.0)	p = .38
Years of education	15.3 (2.4)	15.0 (1.3)	p = .61
English receptive vocabularya	112.4 (12.7)	108.5 (12.2)	p = .25
Nonverbal reasoningb	55.7 (7.9)	56.3 (6.9)	p = .78
Social-economic statusc	5.8 (1.1)	5.9 (1.0)	p = .71
Age of L2 exposure (N = 7)	18.3 (1.9)
% Time ASL used	31.3 (16.7)
% Time ASL exposured	38.2 (20.0)
ASL production proficiencyd	6.3 (0.7)
ASL comprehension proficiencyd	6.4 (0.7)
ASL Sentence Reproduction Teste	15.4 (5.6)

^aPPVT-IIIb (Dunn & Dunn, 1997), standard score.

^bK-BIT2 Matrices Subtest (Kaufman & Kaufman, 2004) or WASI Matrix Reasoning (PsychCorp, 1999), T score.

^cBased on caretakers' education score (Hollingshead, 1975); available for 27 bimodal bilinguals and 16 monolinguals.

^dSelf-ratings from a language background questionnaire; proficiency and mixing were rated on a 7-point scale, ranging from ‘almost none’ to ‘like native’ (proficiency) and from ‘never’ to ‘often’ (mixing).

^eAvailable for 20 bimodal bilinguals; maximum score = 35, mean scores for all signers who have taken this test in our lab are 15.0 (SD = 5.7) for Codas (N = 25), 10.1 (SD = 4.1) for L2 learners (N = 56) and 22.4 (SD = 5.9) for deaf signers (N = 105).

In addition, by-participant and by-item growth-curve models were fitted to the time course of the difference between the proportion of cross-linguistic distractor fixations and matched unrelated distractor fixations (see Figure 5 for the observed data and model fits in the by-participant analysis). The by-participant analysis yielded a significant effect of Group on the quadratic term (Estimate = −.089, SE = .044, p < .05), indicating a more peaked difference curve for the bilinguals, providing further support for parallel language activation in the bilingual participants only. The estimated coefficients for all the parameters in the by-participant model are presented in Table 4. The by-item analysis yielded significant effects of Group on the linear (Estimate = .065, SE = .023, p < .01) and quadratic (Estimate = −.109, SE = .023, p < .001) time terms, again suggesting that bilinguals, but not monolinguals, experienced cross-linguistic competition.

Figure 5.

Observed data and model fits (lines though the data points) for the difference curve of fixations on cross-linguistic distractor and matched unrelated distractor items for bimodal bilinguals (red) and monolinguals (blue). Error bars indicate ±1 SE.

Table 2.

Parameter estimates for fixed effects of Time (1^st order, 2^nd order), and Distractor (matched unrelated distractor, cross-linguistic distractor) on fixation proportions for bimodal bilinguals.

Fixed effects	Estimate	SE	t	p
(Intercept)	.090	.010	8.791	< .001
ot1	−.091	.030	−2.973	< .01
ot2	−.002	.018	−0.010	.92
Distractor	.029	.010	2.960	<.01
ot1:Distractor	.062	.051	1.204	.23
ot2:Distractor	−.094	.032	−2.910	<.01

Note. ot1=linear time term, ot2=quadratic time term

In summary, the bilingual participants fixated the cross-linguistic distractor items more than the unrelated distractor items, suggesting that they co-activated the ASL translations of the spoken target words in the experiment. These results are an important replication of the findings of Shook and Marian (2012) with a larger group of bimodal bilinguals.

3.2 Spatial Stroop task

Accuracy. For the bilingual participants, mean percentage correct scores were 98.8% (SE = 0.4%, 95% CI [98.0, 99.5]) on baseline trials, 99.1% (SE = 0.2%, 95% CI [98.6, 99.6]) on congruent trials, and 90.8% (SE = 1.3%, 95% CI [88.1, 93.6]) on incongruent trials. For the monolingual participants, mean percentage correct scores were 98.6% (SE = 0.5%, 95% CI [97.6, 99.6]) on baseline trials, 98.9% (SE = 0.2%, 95% CI [98.5, 99.4]) on congruent trials, and 91.1% (SE = 1.6%, 95% CI [87.8, 94.3]) on incongruent trials. A 3 × 2 repeated measures ANOVA on arcsine transformed proportion correct scores with Condition (baseline, congruent, incongruent) as within-subjects factor and Group (bilingual, monolingual) as between-subjects factor only revealed a main effect of Condition (F(2,104) = 96.49, p < .001, ŋ^p2 = .65). Bonferroni-corrected post-hoc comparisons revealed significantly lower scores on incongruent trials than congruent trials (p < .001, d = −1.78, 95% CI [−.34, −.22]) and baseline trials (p < .001, d =−1.70, 95% CI [−.36, −.22]), which did not differ significantly from each other. There was no significant main effect of Group (F(1,52) < 1, p = .96) and the Condition by Group interaction was also not significant (F(2,104) < 1, p =.87).

Reaction times. A 3 × 2 repeated measures ANOVA with Condition (baseline, congruent, incongruent) as within-subjects factor and Group (monolingual, bilingual) as between-subjects factor revealed a significant main effect of Condition (F(2,104) = 209.79, p < .001, ŋ ² = .80). There was no significant main effect of Group (F(1,52) = 1.25, p = .27), and the Condition by Group interaction was also not significant (F(2,104) = 1.87, p = .17). Bonferroni-corrected post-hoc comparisons showed that, as expected for both the monolingual and the bilingual participants, response times on incongruent trials were slower than response times on baseline trials (p <.001, d = −.53, 95% CI [−49.2, −32.9]) and congruent trials (p <.001, d = −0.99, 95% CI [−86.0, −62.4]), i.e., an inhibition effect for incongruent trials was observed. Furthermore, response times on congruent trials were faster than response times on baseline trials for both groups, i.e., a facilitation effect for congruent trials was observed (p <.001, d = .46, 95% CI [27.2, 39.1]).

Next, the Stroop effect was calculated as a measure of inhibitory control by subtracting response times on baseline trials from response times on incongruent trials, which resulted in a mean Stroop effect of 35.6 ms (SE = 3.7 ms, 95% CI [28.0, 43.2]) for the bilingual participants and 46.5ms (SE = 5.5 ms, 95% CI [35.2, 57.7]) for the monolingual participants. This difference was not significant (t(52) = −1.65, p = .11), suggesting no performance differences in inhibition of incongruent information between our bilingual and monolingual samples.

3.3 Relationship between parallel language activation and inhibitory control

To investigate whether there was an association between the degree and time course of cross-linguistic distractor activation and inhibitory control for the bilingual participants, correlations were conducted between the proportion of looks to the cross-linguistic distractor and the Stroop effect (response times on incongruent trials minus baseline trials) across the 200-300 ms post word-onset time-window where language co-activation appeared to be strongest (see Fig. 3) and the 100 ms immediately preceding and following this time-window (i.e., 100-400 ms post word-onset). Using 20 ms time-frames, this resulted in correlations across 15 time frames in total. To correct for multiple comparisons, we used Guthrie and Buchwald’s (1991) statistical significance thresholds (p < .05) for the probability of finding a series of consecutive significant tests in the analysis of time-series with high auto-correlation. Based on their thresholds, adopting an auto-correlation (lag 1) of .70 (auto-correlation for this time-series was estimated at .62), with a sample size of N=25 and window length of 25 frames, four adjacent time frames < .05 are required to result in an adjusted significance level of .044 (for a window length of 10 frames, three adjacent time frames < .05 yield an adjusted significance level of .047). This means that four or more significant correlations in adjacent time-frames in our dataset would unlikely have occurred by chance. We observed significant positive correlations across four adjacent time-frames between 180ms and 260ms post word-onset (.39 ≤ r ≤ .43, mean r =.43, p < .05; see Fig. 6, Panel A).¹ Bimodal bilingual participants with smaller Stroop effects, i.e., those with better inhibitory control, experienced reduced competition from ASL during early stages of auditory word recognition. As expected, we found no significant correlations (all ps >. 20) between cross-linguistic distractor activation and the Stroop effect in the 100-400 ms post word-onset time-window for the monolinguals, for whom the cross-linguistic distractor was an unrelated distractor that did not require inhibition (see Fig. 6, Panel B, for the corresponding correlation for monolinguals across the 180-260 ms post word-onset window that yielded significant effects for the bimodal bilinguals).

Figure 6.

Correlations between percentage of cross-linguistic distractor fixations and Stroop effect 180-260 ms post word-onset (reaction times on incongruent minus baseline trials) for bimodal bilinguals (A) and monolinguals (B).

To investigate the possibility that the bimodal bilinguals’ cognitive control skills were related to spatial selection rather than to inhibiting the cross-linguistic distractor, we calculated the correlation between the Stroop effect and the proportion of fixations to the matched unrelated distractor fixations across the same 100-400 ms window for the bimodal bilinguals as well as the monolinguals and found no significant correlations (see Fig. 7, Panels A and B, for correlations with the matched unrelated distractor fixations across the 180-260 ms window for the bimodal bilinguals and monolinguals, respectively).

Figure 7.

Correlations between percentage of matched unrelated distractor fixations and Stroop effect 180-260 ms post word-onset (reaction times on incongruent minus baseline trials) for bimodal bilinguals (A) and monolinguals (B).

To further confirm the validity and interpretation of the correlations between cross-linguistic distractor fixations and the Stroop effect, we conducted a similar correlation analysis across another 300 ms window in the bilingual time course (400-700 ms post word-onset), where correlations with cognitive control were no longer expected because parallel language activation was no longer evident, and found no significant correlations.

Finally, although unlikely given the absence of significant group differences on baseline and congruent trials in the Stroop task, it could be argued that the positive correlations might not be linked to inhibition processes specifically, but to some other factor such as overall speed differences among the bilingual participants. To examine this possibility, we correlated the proportion of cross-linguistic distractor fixations across the 100-400 ms window with reaction times in the baseline condition of the spatial Stroop task (centrally-presented arrows, acting as a control condition where no inhibition was required) and found no significant correlations.

In summary, the results of the correlation analysis showed an association between better nonlinguistic inhibitory control performance and reduced cross-linguistic competition during auditory word recognition. This association suggests that bimodal bilinguals engage nonlinguistic inhibitory control to resolve competition from ASL during English spoken word recognition.

4. Discussion

Using a bilingual visual world paradigm, we confirmed that hearing ASL-English bimodal bilinguals co-activate ASL signs during English spoken word recognition, replicating Shook and Marian (2012). Critically, we showed that bimodal bilinguals with better nonlinguistic inhibitory control (as measured by a spatial Stroop task) looked less at cross-linguistic distractors, suggesting that they either exhibited less language co-activation and/or resolved such co-activation more quickly. These results are in line with recent findings with unimodal bilinguals (Blumenfeld & Marian, 2013; Mercier et al., 2014) and for the first time provide evidence that perceptual competition is not required for the engagement of inhibitory control during the early stages of auditory word recognition in bilinguals.

Our finding that ASL-English bilinguals co-activate ASL signs during English spoken word recognition is also in line with other studies showing co-activation between sign language and written language in deaf bimodal bilinguals (Kubus et al., 2014; Morford et al., 2011; Ormel et al., 2012). The fact that language co-activation can readily occur between two languages without overlapping phonologies has important implications for theories of bilingual language processing. Most models of bilingual word recognition such as the Bilingual Interactive Activation model (Dijkstra & Van Heuven, 2002), the Bilingual Model of Lexical Access (Grosjean, 1988, 2008) or the Bilingual Language Interaction Network for Comprehension of Speech (Shook & Marian, 2013), assume that the primary source of parallel language activation in unimodal bilinguals is at a shared phonological level (consisting of speech sounds and/or phonological features). However, bottom-up phonological activation cannot be the origin of parallel language activation in bimodal bilinguals, whose two languages have fully distinct phonological systems. To account for co-activation in bimodal bilinguals these models need to be adjusted to allow language-specific phonological information from each language to be incorporated at separate phonological levels that feed forward to the respective lexical levels for spoken and signed items. Co-activation can then arise at the lexical level, either through direct lateral links or through feed-forward and feedback connections between the lexical representations of each language and shared semantic/conceptual representations (for more detailed discussion of these possibilities, see Shook and Marian (2009)).

Although our sample included both native signers and proficient second language learners, it was not feasible to statistically compare co-activation patterns in the two groups because of the small sample of second language learners (N = 7), which is a limitation of the current study. However, in unimodal bilinguals, language proficiency has been shown to drive parallel activation (e.g., Blumenfeld & Marian, 2007; Ju & Luce, 2004), and the second language learners in the present study were highly proficient. Future research exploring the differences between these two groups would provide interesting insight into how language learning history influences language co-activation in bimodal bilinguals.

Another potential methodological limitation of the study is that the targets and cross-linguistic distractors were rated significantly higher than the targets and unrelated distractors in semantic similarity (2.0 vs 1.7) and visual similarity (2.6 vs 1.6). However, the differences were small and on the low end of the rating scale (1-7). Furthermore, these differences should theoretically have affected the bimodal bilinguals and monolinguals to the same extent. The fact that they did not suggests that increased semantic and/or visual similarity was not the driving force behind the observed effects. Indeed, for the bimodal bilinguals, semantic similarity and visual similarity of the cross-linguistic distractor item to the target item did not correlate with the proportion of looks to the cross-linguistic distractor across the 200-700 ms post word-onset window that was used in the analysis of the time course of fixations (semantic similarity: r = .08, p = .70; visual similarity: r = −.02, p = .90). Furthermore, after excluding six items that were rated more than one standard deviation higher than the average across the two rating scales, the growth curve analyses still yielded a significant effect (p < .05) of the cross-linguistic distractor on the quadratic time term for the bimodal bilinguals, and the correlations between looks to the cross-linguistic distractor and the Stroop effect in the 180-260 ms post word-onset window were also still significant.

Importantly, the present study shows that despite the different sources of language co-activation for unimodal and bimodal bilinguals, both types of bilinguals engage similar nonlinguistic inhibitory control processes to resolve cross-linguistic competition during the early stages of auditory word recognition. In the unimodal bilingual literature, it has been argued that such specific links between bilingual language processing and inhibitory control might provide explanatory mechanisms for the widely reported – but also debated – bilingual advantages in cognitive control abilities (e.g., Blumenfeld & Marian, 2011, 2013; Festman, Rodriguez-Fornells, & Münte, 2010; Linck et al., 2012; Prior & Gollan, 2011; Soveri et al., 2011). It was not our primary aim to look for bimodal bilingual advantages in cognitive control, and our analyses of reaction times on the spatial Stroop task did not yield significant differences between the bimodal bilingual participants and the monolingual controls (closely matched on age, years of education and nonverbal intelligence).

Although this result is in line with findings by Emmorey, Luk, et al. (2008), who reported no advantage in inhibitory control for bimodal bilinguals compared to monolinguals using a variant of the Eriksen flanker task, it should be interpreted with caution. Only three previous studies have investigated possible cognitive consequences of bimodal bilingualism (Emmorey, Luk, et al., 2008; Kushalnagar et al., 2010; MacNamara & Conway, 2014) and only one of these directly compared bimodal bilinguals to monolinguals and unimodal bilinguals (Emmorey, Luk, et al., 2008). In addition, although the spatial Stroop and Eriksen flanker tasks are both considered inhibitory control tasks, several studies that compared interference scores across these tasks have reported non-significant cross-correlations (Paap & Greenberg, 2013), suggesting that these tasks may not measure identical cognitive constructs. Moreover, it should be noted that the evidence for cognitive advantages in unimodal bilinguals is highly variable, and the factors that contribute to and modulate advantages are currently unclear (for discussion, see e.g., De Bruin, Treccani, & Della Sala, 2015; Hilchey & Klein, 2011; Kroll & Bialystok, 2013; Paap & Sawi, 2014; Valian, 2015). The same factors may account for variation in bimodal bilinguals and further research in this area is warranted.

Crucially, regardless of whether or not bimodal bilinguals may develop enhanced cognitive control abilities compared to monolinguals, we show for the first time that, similar to unimodal bilinguals, bimodal bilinguals engage nonlinguistic inhibitory control processes during auditory word recognition. We cannot rule out the possibility that the correlations between cross-linguistic activation and inhibition are related to the shared visual-spatial nature of the Stroop and eye-tracking tasks. However, Blumenfeld and Marian (2011) identified links between within-language competition and the same Stroop task in bilinguals, but not monolinguals, suggesting that the effect is specific to bilinguals. Future research can further address whether correlations between cross-language activation and inhibitory control in bilinguals reflect purely cross-linguistic competition resolution or whether this relation extends to more general types of competition resolution. Furthermore, it is also possible that, although bimodal bilinguals recruit nonlinguistic inhibitory control mechanisms during language processing, they may not do so to the same extent as unimodal bilinguals. For example, code-blending (in contrast to code-switching) does not require inhibition of the non-target language and often occurs when speaking with non-signers (Casey & Emmorey, 2009). Therefore the cognitive control abilities of bimodal bilinguals might not be as practiced as those of unimodal bilinguals.

Our finding that the degree and time course of cross-linguistic competition for bimodal bilinguals was associated with their performance on a nonlinguistic spatial Stroop task indicates that perceptual similarity between word candidates from the two languages is not necessary to trigger the recruitment of Stroop-type inhibition during auditory word recognition. To explain the association between Spanish cross-linguistic distractor activation and performance on the spatial Stroop task for Spanish-English bilinguals, Blumenfeld and Marian (2013) suggested that auditory word recognition and spatial Stroop inhibition both involve the processing of perceptually bivalent stimuli. More specifically, Blumenfeld and Marian (Blumenfeld & Marian, 2013, 2014) adopt the framework of the Dimensional Overlap Model (Kornblum, Stevens, Whipple, & Requin, 1999) to distinguish between Simon-type inhibition, which reflects conflict between stimulus and response mappings (Stimulus-Response conflict), and Stroop-type inhibition, which reflects perceptual conflict between overlapping stimulus dimensions (Stimulus-Stimulus conflict). They hypothesize that perceptual similarities between word candidates within and between languages may drive bilingual recruitment of Stimulus-Stimulus inhibition during auditory word recognition. Furthermore, they suggest that cross-linguistic competition at the conceptual, lexical and phonological levels may drive bilingual recruitment of Stimulus-Stimulus inhibition during language production. The present study shows that actual perceptual conflict, i.e., overlapping phonological input, is not required to trigger Stimulus-Stimulus type inhibition during auditory word recognition. The results furthermore suggest that competing language-internal representations at the lexical or semantic level can trigger Stimulus-Stimulus inhibition not only during bilingual language production (e.g., Kroll, Bobb, & Wodniecka, 2006), but also comprehension.

Unlike the current study, Blumenfeld and Marian (2013) found negative correlations with the Stroop effect 300-500 ms post word-onset, and positive correlations 633-767 ms post word-onset. That is, Spanish-English bilinguals with better inhibitory control showed increased parallel activation early in the time course and reduced activation later in the time course. Blumenfeld and Marian (2013) explained the shift as reflecting two separate processes: early and automatized co-activation for highly proficient bilingual participants, and engagement of cognitive control mechanisms later between 633 ms and 767 ms. Although we also argue that the positive correlations in the present study reflect engagement of cognitive control mechanisms, these correlations were observed much earlier in the time course (180-260 ms post word-onset) for ASL-English bilinguals, despite similar onsets of parallel language activation (around 300 ms post word-onset for the Spanish-English bilinguals in the Blumenfeld and Marian (2013) study and around 250-300 ms post word-onset for the ASL-English bilinguals). This difference in the time-course of cognitive control between the present study and Blumenfeld and Marian (2013) is likely due to differences in the nature of competition across studies. There is evidence to suggest that the early negative correlation seen in Blumenfeld and Marian’s study is driven by perceptual competition – in their study, the auditory signal presented to participants initially mapped to more than one object in the display. Thus, until the auditory signal reached some “uniqueness point,” where it became exclusively consistent with the target item, the auditory input activated both targets and cross-linguistic distractors. Their time-course data indicate that participants began preferentially looking at target items at approximately 500 ms - if we consider that an eye-movement takes an average of 150-200 ms to initiate (Altmann, 2011; Hallett, 1986), this places the uniqueness point at approximately 300-350 ms post word-onset. This is the same time-window where Blumenfeld and Marian found the negative correlation between Stroop performance and lexical competition, suggesting that competition in this region may be driven by perceptual ambiguity occurring before the uniqueness point in the auditory stream.

The notion that Blumenfeld and Marian’s early negative correlation relates to perceptual ambiguity provides an explanation for why no such correlation was seen in the present study with bimodal bilinguals. Namely, the bimodal participants in our study did not experience this sequence of temporary, perceptually-based ambiguity followed by disambiguation at a uniqueness point. Because our task was completely in English, and the overlap between targets and cross-linguistic distractors was based on similarity in ASL, the auditory information corresponded to only one object in the display from the moment the trial began – thus, the incoming phonological information provided unambiguous cues to a single target item.

The lack of a perceptually-defined uniqueness point in the present task may also help explain why the positive Stroop-by-competition correlation occurred earlier in time for the bimodal relative to the unimodal bilinguals. We believe that in both studies, this positive correlation reflects the engagement of cognitive control mechanisms to inhibit currently-active, non-target lexical items - a critical requirement of this inhibition is that the listener must have some knowledge of what should or should not be inhibited. For the unimodal bilinguals in Blumenfeld and Marian’s study, the initial time-point at which participants are likely to begin receiving cues to target identity is at approximately 300-350 ms post word-onset (i.e., at the perceptual uniqueness point). If we therefore consider the latency of the onset of the positive correlation with respect to the perceptual uniqueness point, the difference between the time-scale of the effect across studies is greatly reduced. Specifically, the post uniqueness-point timescale of the positive correlation is 333-467 ms (assuming a uniqueness point of 300 ms), compared to 160-260 ms in our bimodal bilinguals (a difference in onset of only 173 ms). The similar post uniqueness-point onsets and temporal durations of this positive Stroop-by-competition correlation suggest that both the bimodal bilinguals in our study and the unimodal bilinguals in the Blumenfeld and Marian (2013) study were engaging comparable cognitive control mechanisms to resolve cross-linguistic competition during auditory word recognition (see Fig. 8 for a schematic representation of the time course of co-activation and inhibitory control in both studies).²

Figure 8.

Schematic representation of the time course of parallel language activation and inhibitory control in Blumenfeld and Marian (2013) and the current study. The uniqueness point is identified at 200ms preceding preferential looks to the target items in Blumenfeld and Marian (2013) and at 200 ms post word-onset in the current study.

In summary, the present study confirms and extends previous findings of language co-activation during auditory word recognition in bimodal bilinguals by showing a direct link between nonlinguistic inhibitory control abilities and the degree and time course of linguistic co-activation. Our findings thus suggest that bimodal bilinguals, similar to unimodal bilinguals, engage domain-general cognitive control processes to resolve cross-linguistic competition during the early stages of word comprehension. Because the two languages for bimodal bilinguals have clearly distinct phonological properties, our results indicate that the recruitment of nonlinguistic inhibitory control abilities during auditory word recognition is not dependent exclusively on perceptual competition between the two languages.

Highlights.

• Bimodal bilinguals co-activate ASL signs during auditory word recognition

• Bilinguals with better inhibition show reduced language co-activation

• Inhibitory control during word recognition does not depend on perceptual competition

• Without perceptual overlap, inhibition can occur earlier for bimodal bilinguals

Table 3.

Parameter estimates for fixed effects of Time (1^st order, 2^nd order), and Distractor (matched unrelated distractor, cross-linguistic distractor) on fixation proportions for monolinguals.

Fixed effects	Estimate	SE	t	p
(Intercept)	.118	.008	15.170	< .001
ot1	−.140	.023	−6.021	< .001
ot2	−.016	.015	−1.034	.30
Distractor	.017	.012	1.379	.17
ot1:Distractor	.014	.041	0.340	.73
ot2:Distractor	−.004	.027	−0.155	.88

Note. ot1=linear time term, ot2=quadratic time term

Table 4.

Parameter estimates for fixed effects of Time (1^st order, 2^nd order), and Group (monolinguals, bimodal bilinguals) on the difference between cross-linguistic distractor fixations and matched unrelated distractor fixations.

Fixed effects	Estimate	SE	t	p
(Intercept)	.023	.008	2.895	<.01
ot1	.038	.035	1.093	.27
ot2	−.049	.022	−2.242	<.05
Group	.012	.016	0.756	.45
ot1: Group	.048	.070	0.694	.49
ot2: Group	−.089	.044	−2.050	<.05

Note. ot1=linear time term, ot2=quadratic time term