Speakers of different languages remember visual scenes differently

Abstract

Language can have a powerful effect on how people experience events. Here, we examine how the languages people speak guide attention and influence what they remember from a visual scene. When hearing a word, listeners activate other similar-sounding words before settling on the correct target. We tested whether this linguistic coactivation during a visual search task changes memory for objects. Bilinguals and monolinguals remembered English competitor words that overlapped phonologically with a spoken English target better than control objects without name overlap. High Spanish proficiency also enhanced memory for Spanish competitors that overlapped across languages. We conclude that linguistic diversity partly accounts for differences in higher cognitive functions such as memory, with multilinguals providing a fertile ground for studying the interaction between language and cognition.

Linguistic competition within and across languages changes memory for visual items, showing language and memory are intertwined.

INTRODUCTION

We often rely on language to remember the details of past events. For example, mnemonic devices that are linguistic in nature use acronyms and rhymes to help improve memory. Memory and language are so closely linked that even hearing single words can change how we remember events. When hearing a spoken word unfold over time, a listener activates several related candidates before ultimately accessing the correct word (1, 2). For example, as the word "clock" is heard, related words that sound similar (e.g., clown) are also activated and act as phonological competitors (3). Words with many competitors (i.e., those with higher phonological neighborhood density) are generally identified more slowly than words with few competitors [see (4) for review]. Critical for the present study, bilinguals have been shown to activate competing words in both of their languages, resulting in linguistic competition within and between languages. For example, bilinguals are slower to recognize interlingual homophones (words that overlap in phonology but not meaning across languages, e.g., English "sue" and French "sous") (5). Similar effects of competition across languages have been observed for words with partial phonological overlap (e.g., clock-clavo, nail in Spanish) (6–8).

While the dual-language activation of competitors is an established effect, little is known about the long-term cognitive consequences of continuously accessing competing words in two languages. Recent research indicates that the activation of competing labels within a single language can enhance memory for corresponding visual images (9). Despite evidence for the interactivity of language and memory within the monolingual mind, our current understanding does not account for the diversity in language experiences seen throughout the world. Here, we test whether knowing multiple languages improves visual memory for linguistic competitors through the coactivation of labels that overlap within and across languages.

The activation of language in the mind can be studied by tracking eye movements (10, 11). In visual search experiments, participants typically hear a word and find the matching item among an array of object images. Crucially, the other objects in the array can be manipulated to resemble the target item visually or linguistically. For example, when asked to find a beaker among other objects, participants look more at objects whose names overlap (e.g., beetle) or rhyme (e.g., speaker) with the target word than at unrelated objects (e.g., carriage) (1). Increased eye movements toward related objects reflect activation of competing labels, showing that linguistic overlap with a target can affect visual search (2, 12).

During visual search, bilinguals look more at competitor objects that overlap phonologically in both of their languages (6, 7, 13). Effects of between-language competition are robust across languages (14, 15), modalities (16, 17), and levels of processing (e.g., phonological and lexical) (6, 12) and can even be observed without overt linguistic cues (18, 19). There is also substantial interactivity between linguistic and nonlinguistic systems, and bilingualism is known to play an important role in higher-order cognitive domains such as decision-making, creativity, and memory (20–24). The current study examines the relationship between language and other cognitive systems by measuring the impact of phonological competition on episodic memory.

Episodic memory is the recollection of specific events and their contexts, and it is often measured by presenting participants with words or items to be remembered (i.e., memory encoding) and asking them to recall and recognize them later in the experiment (i.e., memory retrieval). Visual search experiments show that the more objects are looked at during memory encoding, the better they are remembered later on (25, 26). Since linguistic competition between items increases looks to competitors, we predicted that phonological competitors encountered during a visual search task would be remembered better than control objects without phonological overlap (Fig. 1).

Fig. 1. Bilinguals activate phonological competitors in both languages.

If this lexical activation facilitates visual memory for corresponding objects, then bilinguals will remember phonological competitors better than objects without phonological overlap.

Eighty-four Spanish-English bilinguals and 42 English monolinguals first completed a visual search task while their eye movements were tracked (see Table 1 for participant demographics). On each trial, an English auditory target (e.g., a clock) was identified from among four visual objects that included an English within-language competitor (e.g., a clown), a Spanish between-language competitor (e.g., a nail, clavo in Spanish), or a nonoverlapping control item (e.g., a mirror; Fig 2A). Targets and their respective phonological competitors shared at least two initial phonemes at onset (also known as cohort competitors). Participants were then tested on their recognition memory of previously seen items (within-language English competitors, between-language Spanish competitors, and control items; Fig 2B). The effects of bilingualism, phonological competition, and eye gaze on item memory were examined with generalized linear mixed-effects models using recognition memory accuracy as a binomial outcome variable. Spanish proficiency was used as a measure for bilingualism with participants being split into three groups: high-Spanish bilinguals (n = 43), low-Spanish bilinguals (n = 41), and English monolinguals (n = 42).

Table 1. Participant demographics.

Note that values represent means with SDs in parentheses. The last two columns show t test comparisons between the two bilingual groups and between bilinguals and monolinguals.

Measure	High-Spanish bilinguals	Low-Spanish bilinguals	English monolinguals	High versus low Bilinguals	Bilingual versus monolingual
n	43	41	42
Age	29.6 (5.4)	28.9 (6.9)	30.0 (8.8)
English AoA (LEAP-Q)	7.3 (4.5)	5.1 (4.6)	0.7 (0.7)	*	***
Spanish AoA (LEAP-Q)	1.1 (1.1)	4.7 (6.3)	−	***	***
English exposure % (LEAP-Q)	23.3 (19.7)	43.8 (27.0)	99.7 (1.0)	***	***
Spanish exposure % (LEAP-Q)	71.6 (24.2)	53.0 (28.9)	−	**	***
English proficiency (LexTALE)	81.5 (10.3)	82.0 (11.6)	94.1 (6.8)		***
Spanish proficiency (LexTALE-Esp)	95.2 (3.1)	75.5 (11.2)	−	***	***
Nonverbal IQ (matrix reasoning; WASI)	26.0 (3.0)	25.2 (2.8)	25.8 (3.1)
Verbal working memory (digit span; CTOPP)	14.2 (3.2)	15.1 (2.6)	17.3 (2.4)		***

***P < 0.001.

**P < 0.01.

*P < 0.05.

Fig. 2. Participants completed a visual search task and then a recognition memory task.

(A) Examples of visual search trials for each condition. Competition trials (top row) included either English within-language (e.g., clock-clown) or Spanish between-language (e.g., clock-clavo) phonological competitors. Competition trials were compared to control trials without phonological overlap (e.g., clock-mirror), while filler trials masked the experimental manipulation during the encoding phase. (B) Recognition memory task. Memory for each competitor and control item was assessed by asking participants whether they remembered seeing items previously (old) or not (new).

RESULTS

Recognition memory for visual items

We examined the effects of competition type (within-language, between-language, and controls) and language group (high-Spanish bilinguals, low-Spanish bilinguals, and English monolinguals) on participants’ recognition memory of competitor and control items.

Within-language competition

Recognition memory for English competitors (M = 28.8%, SE = 3.3) was significantly greater than for control items (M = 18.5%, SE = 2.3, estimate = 0.58, SE = 0.20, P = 0.004; see table S1 for full model output), suggesting that within-language competition during visual search facilitated memory for competing objects. Tukey-adjusted pairwise comparisons showed that English competitors were remembered significantly better than control items by monolinguals (estimate = −0.64, SE = 0.26, z = −2.47, P = 0.037; see Fig. 3) and high-Spanish bilinguals (estimate = −0.73, SE = 0.25, z = −2.96, P = 0.009), but not by low-Spanish bilinguals (estimate = −0.38, SE = 0.24, z = −1.62, P = 0.238). Prior work has shown that visual (e.g., shape and color) or semantic (e.g., category) overlap with targets can facilitate encoding of competitor items into memory (27, 28). The observed effect of phonological competition indicates that spreading activation from visual objects to overlapping linguistic representations can alter memory even when overlapping features are not present in the display itself.

Fig. 3. Recognition memory accuracy (%) for competitor and control items.

Memory for within-language English competitors (orange) was better than for control items (gray). Memory for between-language Spanish competitors (blue) was better than for control items (gray) for high-Spanish bilinguals. Note that significance denotes Tukey-adjusted pairwise comparisons while controlling for verbal working memory and nonverbal IQ (**P < 0.01 and *P < 0.05). Error bars represent the SEM.

Between-language competition

The effect of between-language competition on recognition memory was influenced by language group and cognitive abilities, as seen in a three-way interaction between competition type (control versus between), language group (monolinguals versus bilinguals), and verbal working memory. When controlling for cognitive abilities, Tukey-adjusted follow-up comparisons revealed that high-Spanish bilinguals remembered Spanish competitors (M = 31.4%, SE = 4.8) significantly better than control items (M = 18.0%, SE = 3.4; estimate = −0.53, SE = 0.22, z = −2.41, P = 0.042; see Fig. 3). Memory for Spanish competitors and control items did not differ for bilinguals with low Spanish proficiency (estimate = 0.14, SE = 0.21, z = 0.66, P = 0.788) or monolinguals (estimate = 0.34, SE = 0.23, z = 1.48, P = 0.300), indicating that the effect of between-language competition is contingent on the activation of Spanish labels rather than a product of confounding visual or semantic features. Previous findings have demonstrated that bilinguals show greater competition from their first language upon hearing a second language as compared to competition from a second language when hearing their first language (6, 7). In the context of our experiment, in which only English was used, higher levels of Spanish proficiency were likely needed to activate Spanish labels and promote item encoding.

When exploring the effects of cognitive abilities, we found that higher verbal working memory predicted a greater between-language competition effect on memory in bilinguals but not monolinguals. This trend was driven primarily by high-Spanish bilinguals, where the effect of between-language competition on memory increased as verbal working memory increased (estimate = −0.33, SE = 0.14, z = −2.33, P = 0.052). Verbal working memory did not affect the effect of phonological competition on memory for monolinguals or low-Spanish bilinguals (Ps > 0.1). Together, these results suggest that both high Spanish proficiency and high verbal working memory promote activation of Spanish labels.

Eye movements during encoding

To further investigate the role of label activation during encoding, we examined whether competitors were fixated more than controls during the visual search trial.

Within-language competition

Growth curve analyses (GCA) (29) revealed that participants spent more time looking at English competitors than at control items (estimate = 0.006, SE = 0.001, t = 6.42, P < 0.001), confirming that within-language competition promoted greater attention toward competitor items that overlapped with the target in English (Fig. 4A, table S2, and Supplementary Results for GCA model output). Monolinguals showed a greater within-language gaze effect (i.e., more looks to English competitors than controls) than bilinguals (estimate = 0.007, SE = 0.002, t = 3.53, P < 0.001). Tukey-adjusted follow-up comparisons revealed that the effect of English competition was significant for monolinguals (estimate = −0.010, SE = 0.012, z = −5.88, P < 0.001) and high-Spanish bilinguals (estimate = −0.004, SE = 0.001, z = −2.80, P = 0.014) and marginal for low-Spanish bilinguals (estimate = −0.003, SE = 0.001, z = −2.22, P = 0.068).

Fig. 4. Phonological competitors were looked at more, which predicted subsequent recognition memory.

(A) Time course of eye movements toward competitor and control items during encoding. Monolinguals and bilinguals looked more at English competitors (orange) than at control items (gray) during the visual search task. High-Spanish bilinguals (but not monolinguals or low-Spanish bilinguals) looked more at Spanish between-language competitors (blue) than at control items (gray). (B and C) Effects of eye gaze on recognition memory accuracy. (B) Recognition memory for English competitors increased with more time spent looking at English competitors (relative to control items). (C) High-Spanish bilinguals’ recognition memory for Spanish competitors increased with more time spent looking at Spanish competitors (relative to control items), suggesting that both attention and high language proficiency are needed to encode between-language competitors.

Between-language competition

High-Spanish bilinguals showed a greater between-language competition effect on fixations than low-Spanish bilinguals (estimate = −0.010, SE = 0.002, t = −4.89, P < 0.001). Tukey-adjusted follow-up comparisons revealed that high-Spanish bilinguals spent more time looking at Spanish competitors than control items (estimate = 0.007, SE = 0.001, z = −4.61, P < 0.001), suggesting that between-language competition from Spanish promoted greater attention toward Spanish competitors. There were no significant differences in looks toward Spanish competitors and control items in bilinguals with low Spanish proficiency or monolinguals (Ps > 0.1). These findings support our interpretation that the effect of Spanish proficiency on memory is likely driven by variable degrees of between-language competition experienced during the encoding stage. High, but not low, proficiency bilinguals showed phonological competition from Spanish competitors during encoding, which subsequently enhanced competitor memory at retrieval.

Effects of eye movements on item memory

To test whether effects of competition on memory were predicted by visual attention, we added a measure of relative competitor gaze during encoding to our memory models. As expected, the enhanced memory of competitor items was partially explained by increased eye movements to competitors during the visual search task, indicating that greater attention to a competitor during encoding resulted in better subsequent memory for that item.

Within-language competition

A two-way interaction revealed that the effect of competition type (control versus within) on recognition memory was moderated by relative competitor gaze to English competitors (estimate = 0.20, SE = 0.09, z = 2.23, P = 0.026; see table S3 for full model output). Tukey-adjusted follow-up comparisons showed that recognition memory for English competitors increased with more time spent looking at English competitors (relative to control items; estimate = 0.17, SE = 0.06, z = 2.76, P = 0.006; Fig. 4B). In contrast, relative competitor gaze did not predict memory for control items (estimate = −0.03, SE = 0.07, z = −0.43, P = 0.665). Therefore, the effect of within-language competition on memory for English competitors was predicted by the effect of within-language competition on visual fixations. The effects of competition type and competitor gaze did not significantly differ across the three groups (competition type × competitor gaze × language group: χ² (2) = 1.32, P = 0.516).

To determine whether the memory advantage for English competitors was contingent on preferential fixations to the competitor, we examined the effect of competition type at the median relative competitor gaze (−0.15) when competitors were not looked at more than the controls. A significant effect of competition type revealed that even without a competitor gaze effect, participants still showed better recognition memory for English competitors (M = 26.9, SE = 3.4) than for control items (M = 17.1, SE = 2.4, estimate = −0.58, SE = 0.23, z = −2.54, P = 0.011). It is possible that within-language phonological competition may facilitate memory to some extent without differences in overt fixations (30). Last, we found a two-way interaction between competition type (control versus within) and nonverbal intelligence quotient (IQ) (estimate = 0.25, SE = 0.10, z = 2.57, P = 0.010), suggesting that greater nonverbal IQ predicts a larger recognition memory effect for English competitors when controlling for eye movements.

Between-language competition

A comparable effect of relative competitor gaze was observed for Spanish competitors, which was moderated by language group. A significant interaction between relative competitor gaze and the second language group contrast (estimate = −0.53, SE = 0.23, z = −2.27, P = 0.023; see table S4 for full model output) indicated that the positive effect of Spanish competitor gaze on competitor memory was significant for high-Spanish bilinguals (estimate = −0.42, SE = 0.14, z = −3.07, P = 0.002), but not for low-Spanish bilinguals (estimate = −0.28, SE = 0.17, z = −1.68, P = 0.093) or for English monolinguals (estimate = 0.03, SE = 0.18, z = 0.155, P = 0.877; Fig. 4C). The dissociation between Spanish competitor gaze and memory for the latter groups suggests that spending more time looking at Spanish competitors is not in itself sufficient to elicit a Spanish competition effect on memory. Rather, it may be the case that once a Spanish competitor is looked at, the listener’s level of Spanish proficiency moderates the extent to which it is encoded into memory.

DISCUSSION

The current study was designed with two goals in mind. The first was to investigate how phonological competition during encoding affects memory for distractor items in a visual scene. The second was to understand the role of language experience and dual-language activation on memory. We found that both English monolinguals and Spanish-English bilinguals remembered competitor items that overlapped within-language in English (e.g., candle-candy) better than control items without overlap (e.g., candle-wing). We also found that, in bilinguals with high Spanish proficiency, between-language competition from Spanish (e.g., candle-candado) facilitated recognition memory of competitor items. Higher Spanish-language proficiency likely lowered the activation threshold of Spanish labels, resulting in greater between-language competition, which translated to better recognition memory for Spanish competitors. The enhanced memory for competitor items was partially explained by visual attention to competitors compared to control items during encoding. Overall, we found that the effects of phonological competition on memory were influenced by intrinsic characteristics such as participants’ language backgrounds and cognitive abilities, as well as by extrinsic properties such as the language of the experiment and the type of competition (within-language versus between-language).

Our finding that within-language competitors are remembered better than control items is consistent with research showing that feature overlap with targets during visual search can facilitate encoding of competitor items into memory. Most previous studies, however, have shown an effect of visual or semantic competition on memory, manipulating the competitors to resemble target items in category, shape, or color (27, 28). Our results show that phonological competition during visual search affects long-term memory. This adds to the small but growing body of evidence that coactivated labels during speech comprehension can have long-term consequences for higher-order processes such as memory (9).

As it has been found for semantically and visually similar items, our findings suggest that phonological overlap may promote greater attention toward competitors, facilitating encoding and subsequent memory. During the visual search task, our participants looked at competitors more than control items, which then predicted how well competitors were remembered. This is in line with visual search experiments showing that incidental encoding of distractors is largely predicted by fixations (26, 27). Eye movements are considered to be a behavioral marker of attentional deployment, a crucial cognitive process in memory encoding (31).

Our findings suggest that language proficiency plays a key role in how phonological competition affects memory. Previous studies have demonstrated that bilinguals show greater competition from a first language (L1) upon hearing a second language (L2) as compared to competition from an L2 when hearing an L1 (6, 32). Furthermore, the strength of L2 activation while processing an L1 depends heavily on L2 proficiency (33, 34). Similarly, we found that Spanish-English bilinguals with high Spanish proficiency showed phonological competition from Spanish competitors during encoding, but those with low Spanish proficiency did not. In the context of our experiment, in which only English was used, higher levels of Spanish proficiency were likely needed to activate Spanish labels and promote item encoding. Coactivation of Spanish competitors during encoding (i.e., fixations) predicted subsequent recognition memory only in bilinguals with high Spanish proficiency, suggesting that both high language proficiency and fixations may be necessary to encode between-language competitors. This could explain why even when Spanish competitors were looked at by participants with low Spanish proficiency (i.e., English monolinguals and low-Spanish bilinguals), they were not remembered better than control items.

We did not find a significant memory effect for English competitors in the low-Spanish bilingual group despite having comparable English proficiency to the high-Spanish bilingual group. One possible explanation stems from the relative language balance of our low-Spanish bilinguals. This group was largely balanced in English and Spanish across proficiency, exposure, and age of acquisition (AoA) (see Table 1). Recent evidence suggests that balanced bilinguals may have increased inhibitory control compared to unbalanced bilinguals due to more frequent exposure and use of both languages (35, 36). Inhibitory control is a core executive function that allows the suppression of task-irrelevant information and behaviors. If our balanced bilingual group had increased inhibitory control, then they could have suppressed distractor items in the visual search task better than other groups, leading to reduced memory for competitor items. The current study did not include measures of inhibitory control, limiting our ability to test this hypothesis. Future work could include objective measures of inhibitory control to examine the interaction between language experience and executive function on episodic memory. Our findings suggest that coactivation of phonological competitors drives their encoding into memory, but this relationship may be modulated by absolute and relative language proficiency levels and executive function abilities.

Although inhibitory control was not measured in the present study, individual differences in verbal working memory and nonverbal IQ influenced the relationship between phonological competition and memory. Recognition memory of between- but not within-language competitors varied as a function of verbal working memory. In contrast, greater nonverbal IQ was associated with a larger within- but not between-language competition effect. Together, these findings raise the possibility that between- and within-language competitors may be encoded differently, with relatively stronger verbal memory traces for between and visual traces for within. In line with this hypothesis, dual coding theory posits that both visual and verbal information can independently contribute to memory, meaning that items in a visual search task could be encoded visually and verbally (37). The bilingual extension of dual coding theory further proposes that bilinguals can encode information via two distinct verbal codes, one for each language, providing an additional linguistic route in relation to monolinguals (38, 39). To the extent that cross-linguistic phonological overlap promotes dual-language coding, memory for Spanish competitors may have been disproportionately guided by verbal (as opposed to visual) representations. Moreover, this could explain in part the increased role of verbal working memory and language proficiency in memory for between-language competitors. Together, our findings suggest that the effect of between-language competition on visual item memory may vary because of the interaction of individual cognitive abilities and memory processes such as dual-language encoding.

Our findings contribute to the growing body of evidence demonstrating the interactivity of linguistic and nonlinguistic cognitive function, but we note a number of limitations. One potential limitation of the present study is that the extent of Spanish activation may have been amplified by cuing participants into the Spanish nature of the task. As part of the recruitment process participants were asked to indicate their level of Spanish proficiency, which may have increased the salience of their Spanish knowledge and lowered their threshold for activating Spanish labels. To minimize this possibility, all instructions and audio were presented in English following screening. Furthermore, although evidence suggests that brief exposure to a language can affect subsequent linguistic activation in experimental tasks (40), we did not find that current exposure to Spanish predicted memory for competitors (whereas Spanish proficiency did). To provide a more rigorous test of how the relative levels of activation of participants’ two languages may moderate the extent of cross-linguistic interaction, future studies may experimentally manipulate the language environment through the use of blocked versus mixed-language designs. Another limitation of our study is that the experimental design precluded us from examining effects of competition on memory for targets. Although the present study was designed to assess the effects of language activation on memory for competitors, there is reason to expect that the activation of within- and between-language competitors could influence memory for their associated targets. If phonological competition results in greater attention to competitor items, then it follows that attention to target items may be reduced and memory for targets could subsequently suffer. This would be in line with subliminal priming experiments that have demonstrated that increasing the activation of a phonological competitor can suppress identification of neighboring target words (41) [see (42) for a review of facilitative and inhibitory effects of lexical neighbors]. Unfortunately, it was not feasible to assess memory for targets in the current study as each target item was seen and heard three times, each time with a different competitor or control item, yielding ceiling effects in recognition memory of target items. Therefore, whether phonological competition during visual search affects memory for target items remains a question for further research.

Our study examined phonological competition induced by overlapping phonemes at onset, but overlapping phonology can be experimentally manipulated to influence visual attention in other ways. Rhyme competitors, for example, elicit weaker and later visual attention than cohort competitors (43), which may affect memory differently than cohort competitors. Other variables such as proportion of overlap (43), covert overlap (8), and multimodal overlap (44) all influence the strength and timing of competitor visual attention. Beyond phonology, linguistic variables such as neighborhood density (45), morphology (46), and grammatical gender (47) also affect language processing and may have downstream effects on attention and memory. Computational models of both monolingual (48) and bilingual (49) language processing have begun to incorporate this interactive complexity. Multilink, for example, considers the language, orthography, phonology, and semantics of a word to make predictions on several psycholinguistic tasks (e.g., lexical decision, word naming, and translation). With the increasing capabilities of computational modeling, future models could make predictions of visual attention during speech processing, as well as its effects on memory.

Our study provides evidence of significant interactivity in the cognitive system, not only across different languages but also across domains of cognitive function. In contrast to the modular view that language and memory operate independently of each other (50, 51), our findings reveal that coactivating linguistic labels when processing speech directly alters how monolinguals and bilinguals encode visual memories. Extending prior work demonstrating that language experience can alter perceptual processes (consistent with the linguistic relativity hypothesis that language shapes perception and thought) (52), we show that language experience influences not only how people see their current environment but also what they remember long term. This may partially explain why the same event can be remembered differently by different people and illustrates how the diversity of experiences in the world can shape higher-order cognitive outcomes.

These results have potential implications for legal, educational, and clinical practices with linguistically diverse communities. Eye-witness memory, for example, has been shown to be subject to language effects in bilinguals (53), but the role of object names in a scene has not been explored. In clinical populations with memory loss, it may be possible to leverage the effects of phonological competition to develop strategies for improving memory, such as grouping similar sounding objects together to facilitate later retrieval. Similarly, overlapping phonology could be used to inform strategies for learning foreign languages. Our results show that language and memory are intertwined and suggest that it may be possible to capitalize on language to address everyday challenges in other cognitive domains.

MATERIALS AND METHODS

Participants

Participants were recruited online through the Prolific platform (prolific.co) and completed the experiment through the Pavlovia (pavlovia.org) and Qualtrics (qualtrics.com) platforms. Inclusionary criteria for monolinguals consisted of self-reported monolingual status and no experience with Spanish. Criteria for bilinguals included self-reported bilingual status and experience with both Spanish and English. All participants reported having normal hearing and no language-related disorders. Audio checks (e.g., typing correct audio to proceed) were done to ensure that participants could hear stimuli clearly. Eleven participants (three monolinguals, three low-Spanish bilinguals, and five high-Spanish bilinguals) were excluded from analyses due to technical problems or for scores on postexperiment cognitive tests (nonverbal IQ and verbal working memory) and vocabulary assessments (LexTALE and LexTALE-Esp) that fell two or more SDs below the mean. Eighty-four Spanish-English bilinguals (mean age = 29.3 years; 42 men) and 42 English monolinguals (mean age = 30.0 years; 19 men) were included in the final analysis (Table 1). During the experiment, participants completed a nonverbal IQ test [Matrix Reasoning subtest of the Wechsler Abbreviated Scale of Intelligence (WASI)] (54) in between experimental tasks. After the experiment, all participants completed the Language Experience and Proficiency Questionnaire (55), an English vocabulary test (LexTALE) (56), and a verbal working memory test (digit span subtest of the comprehensive test of phonological processing) (57). Bilingual participants additionally completed a Spanish vocabulary test (LexTALE-Esp) (58). Relative to bilinguals, monolinguals reported an earlier AoA for English, higher self-rated English proficiency, and scored higher in English proficiency. For analyses, Spanish proficiency (LexTALE-Esp) was used as a measure of bilingualism: first as a continuous variable and then as a categorical variable with three levels (high-Spanish bilinguals, low-Spanish bilinguals, and English monolinguals). Bilingual participants were designated as being part of either the low or high Spanish proficiency group based on their LexTALE-Esp scores. Compared to low-Spanish bilinguals, high-Spanish bilinguals had an earlier AoA in Spanish and later AoA in English, more current exposure to Spanish and less current exposure to English, and higher Spanish proficiency. Monolinguals scored higher in verbal working memory than bilinguals, likely due to the task being in English. There were no significant group differences in nonverbal IQ.

Experiment procedures were approved by the institutional review board of Northwestern University. Informed consent was obtained from all participants.

Design

We conducted an a priori power analysis for a linear multiple regression random model using G*Power 3.1 (59). With an assumed power of 0.8, α level of 0.05, and H1 p² of 0.14 [based on (9) and pilot data], the recommended total sample size to obtain a similar effect was 80. To account for variability in language experience among bilinguals, we doubled the size of the bilingual group. The study followed a 3 × 3 mixed design with language group (high-Spanish bilingual, low-Spanish bilingual, and English monolingual) as a between-subject variable and phonological competition type (within-language, between-language, and none) as a within-subject variable. During the encoding phase, participants completed a series of visual search trials in which they had to identify an English auditory target from a four-item search display. Memory for critical items (targets, competitors, and controls) was later assessed using a surprise recognition test.

Fifteen critical sets were constructed for the four-item search displays in the encoding phase (see table S5). Each display included a target item (e.g., candle), one of three possible critical items (competitors and controls), and two unrelated filler items. Critical items overlapped phonologically with the English target item either in English (e.g., candle-candy), in Spanish (e.g., candle-candado), or did not overlap in either language (e.g., candle-wing). During critical search trials, participants saw each set three times: once with the within-language English competitor, once with the between-language Spanish competitor, and once with the control (no competition) item (see Fig. 2). Throughout the encoding phase, participants completed 45 critical trials (15 sets × 3 conditions) and 45 filler trials with no competition for a total of 90 encoding trials.

Materials

For every critical set, the English target and phonological competitor shared at least two phonemes at onset (known as cohort competitors). Cohort competitors were chosen over rhyme competitors for two reasons. First, cohort competitors have been shown to elicit stronger competition than rhyme competitors (43). Second, cohort competitors are more common across languages than rhyme competitors, which permitted more stringent matching across sets on phonological and lexical characteristics. Phonemic overlap with the English target was matched between the within-language (English competitor) and between-language (Spanish competitor) competitor conditions (onset target-competitor overlap of 2.3 and 2.1 phonemes, respectively; paired P > 0.05). Competitor and control items in critical trials were matched on English (SUBTLEXUS) (60) and Spanish (SUBTLEX-ESP) (61) frequency, phonological and orthographic neighborhood size (CLEARPOND) (62), concreteness, familiarity, and imageability (MRC Psycholinguistic Database, Glasgow Norms) (63–65). Items within a set were controlled for semantic and physical similarity to avoid confounding factors during encoding.

Items were depicted visually by black and white drawings from the International Picture Naming Project database (66) or Google Images. Pictures from Google Images were normed independently for name reliability by English monolinguals and Spanish-English bilinguals online (Amazon Mechanical Turk, www.mturk.com; Prolific). Name reliability for all items used in the experiment was 94% (SD = 7.7) in English and 92% (SD = 9.6) in Spanish. English target words were recorded on Praat (67) at 44.1 Hz on a MacBook Pro by a bilingual Mexican-American female speaker with no detectable non-native accent in either Spanish or English.

Procedure

Visual search task

Participants first completed a visual search encoding task, during which their eye movements were remotely tracked using the webcam-based library WebGazer.js (68) modified for online use in PsychoPy (69). Following instructions for the visual search task, participants started with three practice trials before completing the 90 experimental trials. Each trial began with a fixation cross that participants clicked on to center their mouse and gaze. After clicking, participants saw a four-picture visual display and heard an English auditory target 500 ms after the visual display onset. Participants were instructed to click on the correct target as quickly as possible. The location of all items was pseudo-randomized, with critical items (competitors and controls) always appearing adjacent to the target item. Upon clicking, a black border appeared around the selected item. The visual display remained on screen for 5000 ms regardless of when the response was made to ensure equal encoding time across trials and participants. Participants were not informed that their memory for the visual items would be tested following the search task. Between encoding and retrieval, participants completed the matrix reasoning subtest of the WASI (54). The subtest served the dual purpose of being a measure for nonverbal IQ and a nonlinguistic distractor task to prevent primacy and recency effects during retrieval.

Recognition memory task

After the encoding phase and the distractor task, participants were shown 135 items in a random order and asked to indicate whether they had seen each one previously. The items included the 45 critical items (15 English competitors, 15 Spanish competitors, and 15 controls) and 30 target items (15 targets from critical trials and 15 targets from filler trials) seen during encoding, as well as 60 unseen “foil” items. Each recognition trial began with the participant clicking a fixation cross in the center of the screen, after which an item would immediately appear. Participants were instructed to click on a box labeled “old” if they recognized the item from the encoding phase and on a box labeled “new” if they did not. After the experimental tasks, participants completed a Qualtrics survey that included a linguistic background questionnaire, language vocabulary tests, and a verbal working memory test. All participants named the critical competitor items in English and bilinguals named competitor items in English and Spanish. Trials for which participants did not provide the correct competitor label were removed from analyses (4.1%). There was no significant effect of language group for false alarm recognition of unseen items [F(2,123) = 1.191, P = 0.307], suggesting no response bias between groups.

Statistical analysis

To determine the effects of bilingualism and phonological competition on item memory, we conducted three sets of analyses looking at gaze, memory, and the effect of gaze on memory. We analyzed gaze using participants’ eye movements to competitor and control items during the visual search task (i.e., encoding phase). Memory was analyzed using participants’ recognition accuracy for those critical items. Last, we explored the relationship between eye movements during encoding and subsequent item memory.

Eye movements to each picture in the display were recorded for each millisecond of critical trials (0 to 5000 ms). In preparation for growth curve analyses (29), fixations at each time point were aggregated across trials into 100-ms bins. The proportion of time spent looking at competitors and controls was first examined with linear mixed-effects models using the R lmer function from the lme4 package (70). Competition type was coded as a categorical predictor variable with three levels (within-language English competitors, between-language Spanish competitors, and control items) and simple-coded to create two contrasts: controls versus within-language competitors and controls versus between-language competitors.

We ran a series of models including fixed effects of competition type and continuous measures of language experience (self-rated and objective proficiency, exposure, and ages of acquisition in English and Spanish) to identify the most relevant individual difference measures of bilingualism. Model summaries and comparisons of Akaike information criterion (AIC) model fit revealed that a continuous measure of Spanish Proficiency was the most predictive language experience measure of eye movements during encoding. Spanish proficiency was calculated by mean centering LexTALE-Esp scores, with English monolinguals receiving a score of 0 before mean centering. Significant interactions with the continuous measure of Spanish proficiency (see table S6 and fig. S1) were subsequently examined in greater detail using GCA and a categorical variable of language group (English monolinguals, low-Spanish bilinguals, and high-Spanish bilinguals). Bilingual participants with LexTALE-Esp scores below 90 [one SD below the mean of Spanish L1 speakers in the original validation study (58) and the median in the present sample] were classified as low–Spanish proficiency bilinguals, while those who scored 90 or above were classified as high–Spanish proficiency bilinguals. Language group was Helmert-coded to create two contrasts: (i) monolinguals (+0.67) versus high-Spanish bilinguals and low-Spanish bilinguals (−0.33) and (ii) high-Spanish bilinguals (−0.5) versus low-Spanish bilinguals (+0.5). Cognitive measures of mean-centered nonverbal IQ (matrix reasoning subtest) and verbal working memory (digit span subtest), along with their two- and three-way interactions with competition type and language group, were included in all models as covariates. Models of competitor fixations included a random intercept by participant.

We examined the effects of bilingualism and phonological competition on item memory with generalized linear mixed-effects models using the R glmer function from the lme4 package (70). Recognition memory accuracy for critical distractor items (i.e., competitors and controls) was coded as binomial outcome variable (0 = incorrect and 1 = correct). The recognition memory model included fixed effects of competition type, language group, and their interactions with verbal working memory and nonverbal IQ, as well as the maximal random effects structure justified by the design (71), with random intercepts for participants and stimulus set, a by-participant random slope for competition type, and by-set slopes for competition type and language group.

We examined the role of eye movements in item memory by adding a measure of competitor fixations during encoding as a fixed effect in the memory models. Relative competitor gaze was calculated as the scaled proportion of fixations to either the English or Spanish competitor minus fixations to their respective control item for each set. As measures of relative English and Spanish competitor gaze were inherently correlated with each other (due to comparisons to the same control item within a given set), two separate models were constructed to examine the effects of gaze on memory for (i) English within-language and (ii) Spanish between-language competitors. Final models therefore included fixed effects of competition type, language group, within- or between-language relative competitor gaze, verbal working memory, nonverbal IQ, and all two- and three-way interactions. Random effects included random intercepts for participants and stimulus set, a by-participant random slope for competition type, and by-set slopes for competition type and language group.

Supplementary Materials

This PDF file includes:

Fig. S1

Tables S1, S3 to S6

Legend for table S2

Other Supplementary Material for this manuscript includes the following:

Table S2