Memory after Visual Search: Overlapping Phonology, Shared Meaning, and Bilingual Experience Influence What We Remember

Abstract

How we remember the things that we see can be shaped by our prior experiences. Here, we examine how linguistic and sensory experiences interact to influence visual memory. Objects in a visual search that shared phonology (cat-cast) or semantics (dog-fox) with a target were later remembered better than unrelated items. Phonological overlap had a greater influence on memory when targets were cued by spoken words, while semantic overlap had a greater effect when targets were cued by characteristic sounds. The influence of overlap on memory varied as a function of individual differences in language experience -- greater bilingual experience was associated with decreased impact of overlap on memory. We conclude that phonological and semantic features of objects influence memory differently depending on individual differences in language experience, guiding not only what we initially look at, but also what we later remember.

1. Introduction

In our daily lives, we frequently rely on our ability to locate specific objects in complex visual scenes. In addition to internally motivated search tasks (e.g., looking for a landmark while driving), visual search can be prompted by external stimuli, including linguistic instructions (e.g., “watch out for that car”) and environmental sounds (e.g., a car horn). These examples illustrate the multimodal nature of visual search, which often involves access to not only visual representations stored in long-term memory, but also other associated attributes, including semantic, auditory, and linguistic features of a target object.

Evidence of such multimodal activation can be found using the Visual World Paradigm (VWP), during which individuals are prompted to look for a target among a set of visual objects. Eye-tracking reveals that individuals often make visual fixations toward objects that are visually, semantically, or linguistically similar to target objects, indicating that objects in a visual scene activate multiple levels of associated representations (see Huettig, Rommers & Meyer, 2011 for a review). Though the impact of cross-modal interactions on visual perception and attention has been well established (Huettig & McQueen, 2007; Iordanescu, Grabowecky, & Suzuki, 2011; Marian & Spivey, 2003; Salverda & Altmann, 2011; Talsma, Senkowski, Soto-Faraco, & Woldorff, 2010; Tanenhaus, Spivey-Knowlton, Eberhard, & Sedivy, 1995), including the influence of task-related variables (e.g., language context; Marian & Spivey, 2003a, b) and individual differences (e.g., bilingual experience; Blumenfeld & Marian, 2007; Conseco-Gonzalez et al., 2010; Mercier, Pivneva, & Titone, 2014), relatively less is known about the potential downstream consequences for long-term memory.

The aims of the present study were twofold: first, to examine whether semantic and phonological competition during visual search impacts how well objects are later remembered, and second, to explore contextual and individual moderators, with a particular emphasis on the impact of bilingual language experiences such as acquisition, exposure, and proficiency.

1.1. Memory retrieval in visual search

Visual search and the mechanisms underlying how cross-modal inputs interact to elicit fixations to visual targets (as well as their semantic and phonological associates) involve a combination of bidirectional sensory, attentional, and memory processes. Visual attention can be guided by bottom-up, stimulus-driven processes (e.g., the perceptual salience of objects in a visual scene; e.g., Itti & Koch, 2000; 2001), as well as top-down, observer-driven influences such as task goals and internal representations retrieved from and maintained in memory (Bahle et al., 2020; Bravo & Farid 2014; Cowan, 1995; Kerzel, 2019; van Loon et al., 2017; see Wolfe, 2000 for review). According to biased competition models of visual search (e.g., Desimone & Duncan, 1995; Duncan & Humphreys, 1989), objects in a visual scene compete for representation and control in the visual cortex, and representations held in working memory (WM; i.e., “attentional templates”) can be used to bias visual and attentional processing towards those that are most relevant to the task at hand. For instance, Chelazzi et al. (1998) found that after cuing monkeys with a target image, cells in the inferior temporal cortex (an area associated with object recognition) tuned to properties of the target produced stronger responses than those of non-targets once the search display was presented.

Similar evidence of anticipatory sensory enhancement has been found when humans are cued, not with the visual target itself, but with a symbol representing the object category (e.g., representing “people” or “cars”; Peelen & Kastner, 2011). Preparing to respond to a familiar category elicited category-specific patterns of neural activation in the object-selective visual cortex, which subsequently facilitated detection of previously unseen category exemplars. In addition to demonstrating that representations held in WM can bias visual competition toward corresponding inputs, these findings show that representations stored in long-term memory (LTM; e.g., previously learned conceptual categories) can be used to enhance processing of related stimuli. At the behavioral level, biased competition elicits preferential fixations to possible targets (Chelazzi et al., 1998), which can subsequently increase the resolution of visual representations and facilitate the process of matching them to internal target representations.

Though the term “top-down” is often associated with volitional and effortful control, observer-driven influences can also operate implicitly, such as when attention is guided by a previously seen visual prime (e.g., Huang, Holcombe, & Pashler’s (2004) Episodic Theory of Priming in Visual Search) or the cascaded activation of task-irrelevant object features stored in LTM (e.g., phonological and semantic information; see Huettig & McQueen’s 2007 Cascaded-Activation Model of language-vision interactions). Critically for the present investigation, the processing of visual scenes can be influenced by the activation of cross-modal representations stored in memory (e.g., spoken words, environmental sounds), which can guide attention to facilitate or interfere with visual search depending on whether they overlap with target or non-target objects in the display (Iordanescu, Grabowecky, & Suzuki, 2011; Marian & Spivey, 2003; Salverda & Altmann, 2011). Here too, WM is thought to play a key role in directing attention and eye-movements by binding activated representations to each other and to particular locations (see Huettig, Olivers, & Hartsuiker, 2011 and Wolfe, 2021). The retrieval and maintenance of WM and LTM traces thus play a significant role in guiding visual attention and search.

1.2. Memory encoding during visual search

There is now convincing evidence that visual search can be influenced by features of earlier trials, indicating that information regarding previously seen objects and their visual contexts are encoded to some form of memory during the search task itself. For instance, when search displays are repeated, participants are able to exploit memory for targets’ previous locations (Chun & Nakayama, 2000; Shore & Klein, 2000), as well as the locations and identities of paired distractor objects to facilitate subsequent performance (i.e., a contextual cueing effect; Chun & Jiang, 1998, 1999; cf. Horowitz & Wolfe, 1998). Neurophysiological and brain imaging studies indicate that such contextual cueing effects are supported by neural regions implicated in LTM function (e.g., the medial temporal lobe and hippocampus; Chun & Phelps, 1999; Greene et al., 2007), as well as cortical areas associated with visual attention (Giesbrecht, Sy, & Guerin, 2013; Kasper et al., 2015; Olson, Chun, & Allison, 2001).

Though the medial temporal lobe and hippocampus have traditionally been thought to support explicit, declarative memory (e.g., Moscovitch et al., 2006), contextual cuing effects are generally believed to reflect implicit learning, as memory-based facilitation is often observed despite chance performance on explicit recognition tests (e.g., Chun & Nakayama, 2000). Other studies have found, however, that objects and scenes viewed during visual search are later recognized above chance (Beck et al., 2006; Castelhano & Henderson, 2005; Hollingworth & Henderson, 2002; Hout & Goldinger, 2010; Lavelle et al., 2021) and to varying degrees depending on similarity to the target (Thomas & Williams, 2014; Williams et al., 2005, 2009; Williams, 2010). Williams, Henderson, and Zacks (2005) found that, following a conjunction search task with written target cues (e.g., “white telephone”), recognition of target images exceeded that of category-matched distractors (e.g., a blue telephone) and color-matched distractors (e.g., a white pipe), which were in turn recognized more often than unrelated distractors (e.g., a green wallet). Explicit memory for distractors additionally appears to be associated with the duration of time spent fixating the objects (Hout & Goldinger, 2012; Lavelle et al., 2021), which tends to increase with greater similarity to the target (Becker, 2011; Williams, 2010).

Together, these findings indicate that irrelevant objects viewed during visual search can be incidentally encoded into LTM, and that the strength of encoding increases with similarity to the target. To date, however, the majority of existing studies have either tested memory for unrelated distractors or manipulated dimensions of visual similarity (e.g., shape or color, as in Williams et al., 2005, 2009). Therefore, while there is evidence that visual relationships between objects influence incidental encoding, little is known regarding potential interactions with the linguistic system and representations retrieved from memory (such as object labels and meanings), or the incidental encoding of visual objects in response to cross-modal inputs (e.g., auditory target cues). Similarly, while there has been significant interest in understanding the role of individual differences in guiding visual search (including language experience, e.g., Blumenfeld & Marian, 2011) and domain-general cognitive function (e.g., WM, Huettig & Janse, 2016), much less is known regarding the impact of prior language experience on incidental encoding of visual objects. The present study therefore examines how memory for previously seen objects is moderated by the type of competitors encountered during visual search (semantic vs. phonological), the type of auditory input used to cue the target (spoken words vs. environmental sounds), and individual differences in bilingual experience.

1.3. Auditory cues and visual search

When searching for a visual object (such as a picture, letter, or number), hearing a corresponding auditory cue can facilitate its identification and localization (Chen & Spence, 2010; Lupyan, 2008; Lupyan & Spivey, 2010; Iordanescu, et al. 2010; Iordanescu, 2011). According to functional (Lupyan, 2007, 2012; Iordansedcu et al., 2011) and structural (Chen & Spence, 2011) accounts, the interactive effects of auditory and visual inputs arise from converging activation of object features across multiple levels of representation. Hearing a spoken word (e.g., “cat”) or environmental sound (e.g., “meow”) can activate associated linguistic and semantic information (e.g., the word and concept of a cat) stored in LTM, as well as visual features of the cued object (e.g., ears, tail, whiskers). When combined with bottom-up activation stemming from the visual scene itself, an auditory input’s top-down activation of visual features can strengthen the perceptual salience of the cued object and draw visual attention toward its location. Patterns of cascading and converging activation could similarly account for why individuals often make visual fixations toward objects that share semantic features with a cued target (e.g., to a picture of a bone when cued with the word “dog”), as well as toward objects whose labels sound similar (e.g., a picture of a “doctor” when cued with “dog”; Allopenna, Magnuson, & Tanenhaus, 1998; Huettig & McQueen, 2007; Yee & Sedivy, 2006). Indeed, linguistically-based competition has been observed even during entirely non-linguistic tasks (e.g., a picture of a cloud cued by a picture of a clock; Chabal & Marian, 2015). Though evidence suggests that both semantic and linguistic information can become activated by linguistic and nonlinguistic stimuli, it is likely that word forms and meanings are activated via different routes of processing and to varying degrees depending on the input (Chen & Spence, 2011). Bartolotti et al. (2020) found that the timecourse of looks to semantic and phonological competitors differed depending on whether targets were cued by a word or a sound, with words eliciting early phonological competition followed by semantic competition, and sounds eliciting early semantic competition followed by phonological competition.

Based on work demonstrating that greater attentional deployment and visual fixations to objects strengthens encoding into long-term memory (e.g., Hout & Goldinger, 2012; Intraub, 1984; Lavelle et al., 2021; Loftus, 1972; Potter & Levy, 1969; Williams et al., 2005), there is reason to expect that effects of input and competition observed during visual search would be reflected in subsequent recognition of visual objects. We therefore hypothesize that long-term memory for visually presented semantic and phonological competitors will differ depending on whether their associated targets were previously cued by a word or a sound. Specifically, we predicted that recognition of semantic competitors would be enhanced when initially viewed concurrently with a sound compared to a word. Likewise, we would expect that direct, bottom-up activation of target labels should make phonological similarities especially salient, potentially resulting in better recognition of phonological competitors cued by words compared to sounds.

1.4. Individual differences in bilingual experience

In addition to contextual variables, individual differences in cognitive abilities and experiences can moderate the activation and subsequent inhibition of competitors during visual search. Much like interference that can arise during visual processing, comprehension of spoken language requires the management of competing representations. As a spoken word unfolds (e.g., “candle”), other words with similar phonological features (e.g., “casket,” “can,” “candy”) are activated until the correct candidate can be identified (Luce & Pisoni,1998; McClelland & Elman, 1986; McQueen & Cutler, 2001). Once accessed, any given word can go on to activate other phonologically or semantically related representations, each of which has the potential to influence where an individual directs their attention and subsequently, what they remember. Research with bilinguals suggests that the additional processing demands associated with managing competition within and between multiple languages may elicit both online (e.g., Friesen, Chung-Fat-Yim, & Bialystok; 2016; Kotz, 1997) and long-term (e.g., Bialystok, Craik, Green & Gollan, 2009) changes to cognitive processing, which could go on to modulate the influence of overlapping features on memory.

Specifically, there is reason to expect that while bilinguals may routinely need to manage a higher number of competitors (due to the additional activation of lexical candidates across languages), this could effectively dilute the level of activation associated with any given competitor. Dahan, Magnuson, and Tanenhaus (2001) showed that when participants were tasked with matching a picture to a spoken word (e.g., “bench”), the salience of phonological competitors in the visual scene varied as a function of word frequency – competitors with high-frequency labels (e.g., a bed) were fixated more often than those with lower-frequency labels (e.g., a bell). Given that bilinguals are likely to store and activate a larger number of lexical representations compared to monolinguals (each of which may be less frequently encountered due to reduced exposure to each language; see the frequency-lag or weaker links hypothesis; Gollan, Montoya, & Werner, 2002; Gollan et al., 2008, 2011), it is possible that phonological, and possibly even semantic, overlap between any given pair of stimuli may have a weaker effect on later memory.

Support for this hypothesis comes from Friesen et al. (2016) who had bilingual and monolingual participants complete a simple lexical selection task identifying which of two images corresponded to a simultaneously presented auditory word (e.g., “monkey”). While semantic overlap (e.g., pictures of a monkey and a gorilla) resulted in slower RTs for both bilinguals and monolinguals, there were distinct patterns of event-related potentials (ERPs) depending on language background. Specifically, the authors looked at the N400 component, which is sensitive to semantic incongruities and is typically reduced in response to semantically-related stimulus pairs compared to unrelated stimuli (Kutas & Federmeier, 2011); the reduced N400 negativity in response to related pairs has therefore been interpreted as an index of semantic integration (Holcomb & McPherson, 1994). Friesen et al. (2016) found that while monolinguals showed the expected N400 attenuation in response to semantically-related pictures (compared to unrelated pictures), no such effect was observed for bilinguals, leading the authors to propose that the greater number of lexical candidates that must be considered by bilinguals reduces (or perhaps delays) integration of semantic representations. We propose that such differences at early stages of phonological and semantic processing could go on to affect long-term memory, such that overlap between targets and competitors during encoding may have a less notable influence on subsequent memory for those who have substantial knowledge of and experience with multiple languages.

Furthermore, the greater processing demands associated with managing multiple languages could go on to have more long-term consequences, such as through the enhancement of domain-general cognitive control and working memory processes (Bialystok, Craik, Green & Gollan, 2009; Bialystok, Craik & Luk, 2008; Costa, Hernández & Sebastián-Gallés, 2008; Grundy & Timmer, 2017). Bilinguals’ cognitive control could facilitate the management of phonological and semantic competitors during encoding (Blumenfeld & Marian, 2011; Chabal, Schroeder, & Marian, 2015; Friesen et al., 2015; Hernández, Costa, & Humphreys, 2012; Mercier, Pivneva, & Titone, 2014) and subsequently reduce the impact of overlapping features on long-term memory. Speaking to this possibility, a number of studies have found that recognition of previously seen distractors is greater following tasks that impose a high (e.g., multiple targets) than low (e.g., single target) WM load (Guevara Pinto, Papesh, & Hout, 2020; Hout & Goldinger, 2010, 2012, cf. Lavelle et al., 2021), which has been posited to result from a reduced capacity inhibit irrelevant distractors.

Individual differences in WM capacity have similarly been shown to predict the activation and inhibition of competing representations during language comprehension (e.g., Gunter, Wagner, & Friederici., 2003) and linguistically-mediated visual search (e.g., Huettig & Janse, 2016). ERP studies indicate that, compared to readers with high WM spans, those with low WM spans are more likely to co-activate competing representations when processing syntactically- (e.g., Vos & Friederici, 2003) or lexically-ambiguous (Gunter et al., 2003) linguistic stimuli. For instance, Gunter et al. (2003) examined the effects of WM capacity on the N400 component in response to sentences that included a homonym and a subsequent disambiguating cue (e.g., “Since Ken really liked the boxer, he took a bus to the nearest pet store [sports arena]”). Low-span readers showed comparable N400s in response to cues inconsistent with the dominant and subordinate meanings, indicating that both representations were active in WM. High-span readers, on the other hand, had larger N400s in response to cues inconsistent with the dominant than subordinate meaning, suggesting that less probable representations were inhibited with higher WM capacity. To the extent that bilingual experience enhances the ability to suppress competing representations during visual search, we may expect that subsequent effects of competition on recognition memory would be reduced for bilinguals relative to monolinguals.

Yet, due to significant variability in how bilingualism is operationalized, studies comparing groups of individuals idiosyncratically classified as “monolingual” vs. “bilingual” have yielded inconsistent findings regarding the impact of language experience on cognitive function (see Bedore et al., 2012; Luk & Bialystok, 2013; Marian & Hayakawa, 2020; Sabourin et al., 2016; Surrain & Luk, 2019 for discussions). A growing consensus within the field has therefore been to treat multilingualism not as a homogenous group or category, but rather as a dynamic and interactive set of experiences and abilities that exist along a spectrum. To this end, an increasing number of bilingualism researchers have begun to step away from the monolingual-bilingual dichotomy to instead examine the different factors that characterize bilingual experience (e.g., Anderson et al., 2018, 2020; Gullifer et al., 2021; Li, Sepanski, & Zhao, 2006; Luk & Bialystok, 2013; Marian, Blumenfeld, & Kaushanskaya, 2007; Marian & Hayakawa, 2020), as well as the aspects of bilingual experience that affect cognition (Beatty-Martinez et al., 2019; Chung-Fat-Yim, Sorge, & Bialystok, 2020; Hartanto & Yang, 2016), behavior (Beatty-Martinez & Dussias, 2017; Tiv et al., 2020), and the brain (DeLuca et al., 2019; Gullifer et al., 2018; Pliatsikas et al., 2016; Sulpizio, 2019).

The ability to identify meaningful variables that capture distinct aspects of bilingual experience is critical for advancing our understanding of the role of language in cognitive and neural function more broadly (e.g., the impact of different contexts of language use, see Green & Abutalebi’s 2013 Adaptive Control Hypothesis). Illustrating the latter, DeLuca et al. (2018) found that different aspects of language experience (e.g., L2 age of acquisition (AoA), duration of language use, contexts of use) were associated with distinct forms of functional and structural neuroadaptations (e.g., resting-state connectivity, white matter volume, reshaping of subcortical structures). Particularly relevant for the present investigation, the authors observed that duration of L2 use (AoA) was associated with greater functional connectivity in the visual network and adaptations in subcortical structures (e.g., thalamus), which are respectively associated with more efficient visual and language processing. Duration of L2 immersion was also associated with subcortical adaptations in regions associated with automatized language control (e.g., caudate nucleus, thalamus).

Taking a similar approach, Sulpizio et al. (2019) observed that longer durations of L2 experience were associated with increased connectivity between the posterior superior temporal gyrus (associated with the integration of phonological and conceptual processing; Bonilha et al., 2017; Hickok & Poeppel, 2004) and the left precuneus (implicated in attentional processing, Cavanna & Trimble, 2006), which the authors suggest may facilitate attentional control over the activation of lexico-phonological representations. Other studies have shown that effects of different forms of bilingual experience on neural function predict behavioral measures of cognitive control (e.g., effects of AoA and diversity of language use on functional connectivity and proactive control; Gullifer et al., 2018). As attested by these findings, taking an individual differences approach to studying the consequences of bilingualism can provide substantial insight into the relationships between language, cognition, and the brain, including the impact of specific language experiences on particular cognitive processes.

With this goal in mind, the present study includes participants with varying degrees of English and Spanish experience, and examines the relationships between memory for phonological and semantic competitors and language background using two approaches: first by treating bilingualism as a categorical construct (monolinguals vs. bilinguals), and then with the use of multiple continuous measures tapping into different dimensions of bilingual experience. In addition to elucidating the aspects of bilingualism that may moderate the overall impact of competition on memory, specific effects of competition type (phonological vs. semantic) and auditory input (word vs. sound) may contribute to understanding the extent to which the consequences of language experience are restricted to linguistic contexts or result in cross-domain changes to the cognitive system.

2. Methods

2.1. Participants

Fifty-two participants with varying degrees of multilingual experience (82.7% female, mean age = 21.48, SD = 3.36) were included in the analysis, with an additional two participants excluded due to missing dataⁱ. Participants were native speakers of English (N = 32), Spanish (N = 17), or both (N = 3) and completed the experiment in either the Sound (N = 25) or Word (N = 27) condition. As determined by the Language Experience and Proficiency Questionnaire (LEAP-Q; Marian, Blumenfeld, & Kaushanskaya, 2007), all participants were highly proficient in English with scores ranging from 7 to 10 (out of 10) (M = 9.26, SD = 0.95), and had Spanish proficiency scores ranging from 0 to 10 (M = 4.53, SD = 4.01). English age of acquisition (AoA) ranged from 0 to 25 (M = 2.31, SD = 4.08), and Spanish AoA ranged from 0 to 24, with AoA set to the current age among those with no Spanish knowledge (M = 10.17, SD = 8.76)ⁱⁱ. Current English exposure ranged from 25% to 100% of the time (M = 83.37, SD = 19.91) and current Spanish exposure ranged from 0% to 75% of the time (M = 15.17, SD = 19.33). Four native English speakers with no Spanish experience reported minimal second language proficiency (< 3/10) in another language (Italian, French, Latin). Among all participants, 18 reported some level of proficiency with 3 or more languages (French, Hebrew, Italian, Latin, Portuguese, Chinese). Participants in the Sound and Word conditions did not differ from each other in gender (p = .515), age (p = .140), English proficiency (p = .120), Spanish proficiency (p = .622), English AoA (p = .697), Spanish AoA (p = .463), English exposure (p = .823), Spanish exposure (p = .696), or number of known languages (p = .680).

2.2. Design and materials

The study was designed as a 2 (Input) × 2 (Competition Type) × 3 (Object Type) mixed design with Input (Sounds, Words) varying between-subjects, and with Competition Type (Phonological, Semantic) and Object Type (Target, Competitor, Control) varying within-subjects. For the encoding phase of the experiment, participants completed a series of visual search trials during which they were presented with an auditory target cue (either a word or a sound) and a visual display depicting four objects. After all search trials, participants completed a surprise recognition memory test of the previously seen critical objects (targets, competitors, and controls). Fifteen sets of stimuli were compiled for phonological trials, which included three critical objects for the memory test: a target (e.g., a cat), an English phonological competitor (e.g., a cast), and an unrelated control object (e.g., an iron). Similarly, the fifteen semantic sets included a target (e.g., a dog), a categorical semantic competitor (e.g., a fox), and an unrelated control object (e.g., a flashlight; see Table A1 in the Appendix for full list of critical items in each set). Each set additionally included filler items (see below), which did not appear on the memory test. All depicted objects across the 15 phonological and 15 semantic sets were matched on English and Spanish word frequency (SUBTLEXUS; Brysbaert & New, 2009; LEXESP; Sebastián-Gallés, Martí, Cuetos, & Carreiras, 2000), English and Spanish phonological neighborhood size (CLEARPOND; Marian, Bartolotti, Chabal, & Shook, 2012), English and Spanish orthographic neighborhood size (CLEARPOND; Marian, Bartolotti, Chabal, & Shook, 2012), familiarity, concreteness, and imageability (MRC Psycholinguistic Database; Coltheart, 1981; BuscaPalabras; Davis & Perea, 2005).

Each of the 30 stimulus sets was used to construct four different trial types for the visual search task: 1) Target-Present/Competitor-Present, which included a target, a competitor adjacent to the target, and two unrelated filler objects, 2) Target-Present/Competitor-Absent, which included a target, an unrelated control adjacent to the target and two fillers, 3) Target-Absent/Competitor-Present, which included a competitor and three fillers, and 4) Target-Absent/Competitor-Absent, which included a control and three fillers. Each of the critical objects therefore appeared an equal number of times, with the target always appearing adjacent to either the competitor or the control object and the competitors and controls never appearing within the same display (see Figure 1 for an example of each trial type). Each trial type for each set was presented twice (with different fillers) and the position of objects in the display were counterbalanced across trials; each object therefore appeared a total of four times, each time in a different position. Participants completed a total of 240 trials (30 stimulus sets × 4 trial types × 2 presentations), with phonological and semantic trials interspersed.

Figure 1.

Example visual search displays for Target-Present/Competitor-Present (top left), Target-Present/Competitor-Absent (top right), Target-Absent/Competitor-Present (bottom left), and Target-Absent/Competitor-Absent (bottom right) trials constructed from a phonological stimulus set. Critical items for the subsequent memory test included the target (e.g., a cat), competitor (e.g., a cast), and control (e.g., an iron) objects.

Visual objects were positioned in the four corners of a 3 × 3 square grid with a fixation cross in the center. Objects were depicted as black and white line drawings obtained from the International Picture Naming Database (Szekeley et al., 2004) and were similar in saturation and line thickness. Spoken words cueing the 30 target objects were recorded at 44.1 Hz by a Midwestern female speaker of Standard American English and environmental sounds were selected from previous studies in our lab and from online databases. All auditory cues were amplitude normalized and presented to participants through closed-back headphones. Participants were seated approximately 80 cm away from the computer screen (2560 × 1440 resolution).

2.3. Procedure

2.3.1. Encoding phase

During the encoding phase, participants were presented with a set of practice trials to familiarize them with the visual search task before completing the 240 test trials. Each trial began with a fixation cross for 1500 ms, followed by a four-picture visual display for 5000 ms. An auditory target cue (either a spoken word or environmental sound) began playing 500 ms after the onset of the search display and participants were instructed to click on the target picture as quickly as possible if it was present, and on the central fixation cross if it was absent. Following a response, a green border appeared around the selected quadrant. The search display remained on screen for 5000 ms regardless of when a response was made so that the encoding time was fixed across all trials and participants (see Figure 2 for example trial timeline).

Figure 2.

Example timeline for a Target-Present/Competitor-Present visual search trial completed during the encoding phase. Participants were presented with a fixation cross for 1500 ms, followed by a four-object search display. An auditory target cue (either a spoken word or environmental sound) was presented 500 ms following the onset of the search display and participants were instructed to click on the target if it was present or the central fixation cross if it was absent. The search display remained on screen for a total of 5000 ms regardless of when a response was made.

2.3.2. Retrieval phase

Following the encoding phase, participants were presented with a surprise recognition memory test, which included each of the critical 90 images (30 target, 30 competitor, and 30 control pictures), as well as 60 “foil” pictures that were not previously seen. On each trial, participants were presented with a single picture and asked to click on a box marked OLD if it had been previously seen during the encoding phase and on a box marked NEW if it had not. Following each trial, participants clicked on a box in the center of the screen to center the mouse position. Memory for each of the three critical object types was assessed as the proportion of items that were correctly recognized as “old.” The order of presentation for the recognition test was randomized for each participant.

2.4. Data analysis

Two sets of analyses were conducted to 1) determine the impact of auditory input, competition, and a categorical variable of bilingualism on subsequent memory, and 2) assess whether effects of competition are moderated by continuous measures of individual differences in bilingual expertise and experience. Effects of auditory input, phonological/semantic competition, and (categorical) bilingualism were first examined using generalized linear mixed-effects models in the R environment (glmer function of the lme4 package, Bates et al., 2015), with recognition accuracy entered as a binomial outcome variable (0 = incorrect, 1 = correct). Subsequent effects of individual difference measures were examined using linear mixed-effects models (lmer function of the lme4 package, Bates et al., 2015) with continuous outcome variables representing participants’ mean competition effects, calculated as the proportion of correctly recognized competitor items minus correctly recognized control items. In each case, we began with the maximal random effects structure (Barr et al., 2013), as justified by the design, and resolved convergence errors by first removing random correlations and then sequentially dropping random effects explaining the least variance until convergence was achieved. Details for specific models are presented in sections 3.1 and 3.2.

3. Results

3.1. How auditory input (words or sounds) and language group (monolinguals or bilinguals) impact memory for semantic and phonological competitors

We began by examining the effects of Object Type (Target, Competitor, Control), Competition Type (Phonological, Semantic), Input (Sound, Word) and Language Group (Monolingual, Bilingual) on participants’ recognition of previously viewed visual objects. English speakers with no Spanish experience were designated as Monolingual (N = 17), while those with both English and Spanish experience were designated as Bilingual (N = 35). The two groups did not differ in gender, age, or English proficiency (averaged across reading, understanding, and speaking; ps > .05). Age of English acquisition was earlier in the Monolingual than Bilingual group (Ms = 0.71 and 3.09 years old, respectively; t(44.9) = 2.74, p = .009). Object Type was reverse Helmert coded to create two contrasts: (1) Targets (−0.67) vs. Competitors and Controls (+0.33) and (2) Competitors (-0.5) vs. Controls (+0.5). Competition Type was contrast coded to compare Phonological (-0.5) vs. Semantic (+0.5), Input was contrast coded to compare Sounds (−0.5) vs. Words (+0.5), and Language Group was contrast coded and weighted by the number of participants to compare Monolinguals (−0.67) vs. Bilinguals (+0.33). The maximally-converging model included random intercepts for participant and stimulus set, as well as a by-participant random slope for Object Type and by-set random slopes for Object Type and Input.

Participants remembered competitor objects (M = 83.9%, SD = 14.6) significantly better than control objects (M = 65.4%, SD = 20.5; Estimate = −1.70, SE = 0.35, z = −4.95, p < .0001; see Figure 3), indicating that similarities to targets impacted subsequent memory for irrelevant items. Recognition of target objects (M = 98.8%, SD = 3.7) was greater than that of both controls and competitors (Estimate = −9.70, SE = 2.30, z = −4.22, p < .0001), and objects in semantic sets (M = 87.2%, SD = 16.7) were remembered better than those in phonological sets (M = 78.1%, SD = 22.0; Estimate = 2.40, SE = 1.22, z = 1.96, p = .049).

Figure 3.

Effects of Input (Sound, Word), Competition Type (Phonological, Semantic), and Language Group (Bilingual, Monolingual) on recognition accuracy (%) for competitor (gray) and control objects (white). Memory of competitors exceeded that of controls, with larger effects of competition for monolinguals than bilinguals and larger effects of phonological competition on memory in the word than sound condition. Error bars represent standard errors. Simple effects were Tukey-corrected for multiple comparisons. *<.05, **<.01, ***<.001.

The effect of competition (competitors – controls) on recognition was greater among monolinguals (M = 26.5%, SD = 17.0) than bilinguals (M = 14.6%, SD = 15.3), reflected by a significant interaction between Language Group and the second Object contrast comparing competitors to controls (Estimate = 0.57, SE = 0.22, z = 2.59, p = .009). Tukey-adjusted follow-up comparisons revealed that memory for control objects was greater among bilinguals than monolinguals (Estimate = 0.97, SE = 0.23, z = 3.51, p < .001), while memory for competitors was comparable in the two groups (Estimate = 0.40, SE = 0.34, z = 1.16, p = .245). The difference between competitor and control recognition was therefore greater for monolinguals than bilinguals.

Lastly, there was a significant three-way interaction between Competition Type, Input, and the second Object contrast comparing competitors to controls (Estimate = 1.12, SE = 0.32, z = 3.50, p < .001; see Table A2 of the Appendix for full output). Follow-up comparisons revealed that the effects of competition depended on both the type of competition and the auditory input. Recognition of phonological competitors exceeded that of phonological controls when the target was cued by a word (Estimate = −1.60, SE = 0.49, z = −3.30, p = .013), but not by a sound (Estimate = −1.22, SE = 0.49, z = 2.49, p = .126). Semantic competitors exceeded semantic controls in both the word (Estimate = −1.80, SE = 0.50, z = −3.58, p = .005) and sound conditions (Estimate = −2.59, SE = 0.52, z = −4.98, p < .0001). In sum, we find that competition during visual processing does indeed impact subsequent memory, with its effect moderated by both language background and auditory input. When bilingualism is treated as a categorical variable, effects of competition are reduced for bilinguals compared to monolinguals, and the effect of phonological competition is greater when targets are cued by words than sounds. This latter finding is consistent with the idea that words and sounds access lexical information through different routes of cascading activation, with smaller effects of competition on memory with more indirect pathways of processing.

3.2. How individual differences in language experience impact memory for semantic and phonological competitors

3.2.1. Variable selection

To quantify participants’ levels of bilingual experience and ability, composite measures of dual-language acquisition, usage, and proficiency were calculated by averaging LEAP-Q responses across English and Spanish (see Table 1 for specific measures). To illustrate, a participant with full proficiency (10/10) in both English and Spanish would receive a dual-language proficiency score of 10, while a participant with full proficiency in English and no proficiency in Spanish would receive a score of 5. As the sample included participants with no second language experience, age-related variables (i.e., ages of first acquisition and fluency) were converted to measures representing the proportion of an individual’s lifespan during which they had been exposed to or fluent in each language (0% to 100%). As with proficiency, acquisition measures were aggregated across English and Spanish so that an individual who was exposed to both English and Spanish from birth would receive a score of 100 while someone exposed to English from birth with no exposure to Spanish would receive a score of 50.

Table 1.

Language Experience and Proficiency Measures

Measure	Dimensions	Score
Language Proficiency	Speaking, Reading, Understanding Spoken Language1, Foreign Accent2	10 (none) - 10 (perfect) 20 (never) - 10 (always)
Duration of Language Proficiency (AoA)	Speaking, Reading, Speaking Fluency, Reading Fluency	(current age - AoA)/current age
Duration of Language Immersion	Country, Family, School	Years
Current Language Exposure Across Contexts	Friends, Family, Television, Radio, Reading, Formal Study	0 (never) – 10 (always)
Contributions to Language Acquisition	Friends, Family, Television, Radio, Reading, Formal Study	0 (not a contributor) – 10 (most important contributor)

As many language background measures are often highly correlated, the inclusion of each measure in the same model could introduce issues of multicollinearity. We therefore subjected the measures to a principal components analysis in order to extract uncorrelated metrics of language experience that captured a significant proportion of variance across the raw measures. Importantly, this approach allowed us to reduce the number of variables based on the observed relationships among the raw measures (i.e., using an unsupervised dimension reduction algorithm) rather than on theoretical or arbitrary judgments regarding each measure’s relevance and relative importance. As principal component analyses rely on correlations among variables to achieve dimension reduction, we began by examining the correlational structure of the mean-centered and scaled dual-language measures in order to identify clusters of correlated variables, which was followed by principal component analyses within each cluster (see Baayen, 2008). The reduced number of measures contributing to individual components additionally simplified interpretations of each metric while minimizing the likelihood that components extracted from different clusters would be significantly correlated with each other.

The varclus function in the Hmisc package (Harrell, 2017) was used to visually cluster variables based on Spearman’s rank correlation, revealing three groups of associated measures (see Figure A1 of Appendix). Principal components were then extracted using the prcomp function in the stats package (R Core Team, 2017) to derive uncorrelated composite measures representing each cluster. Rotation matrices were used to determine each component’s relationship to the original measures (see Table 2). The loadings described by the rotation matrices are proportional to the correlation between each raw measure and each component, providing insight into the factors most strongly represented by each metric.

Table 2.

Rotation Matrices for Principal Component Analyses

Cluster 1
	Dual-Language Proficiency + Acquisition (PC1)	Single-Language Exposure (PC2)	Informal Dual-Language Exposure (PC3)
speaking	0.29	0.19	−0.04
reading	0.28	0.19	−0.02
understanding	0.28	0.22	0.04
propAgeStart	0.28	0.16	0.13
propAgeFluent	0.27	0.07	−0.02
propAgeStartRead	0.26	0.21	0.14
propAgeFluentRead	0.25	0.14	−0.03
acqTV	0.25	−0.15	−0.30
acqFam	0.24	0.07	0.03
acqRead	0.22	0.11	−0.11
acqFriend	0.22	−0.17	−0.38
acqRadio	0.22	−0.32	−0.28
expRadio	0.21	−0.41	−0.01
expFriends	0.20	−0.21	−0.25
expFam	0.20	−0.27	0.46
expRead	0.18	−0.38	0.25
expTV	0.09	−0.31	0.48
accent	−0.21	−0.29	−0.27
Proportion of Variance	0.57	0.09	0.07
Cumulative Proportion	0.57	0.66	0.73
Cluster 2
	Dual-Language Immersion (general) (PC1)	Dual-Language Immersion (family) (PC2)	Dual-Language Immersion (country) (PC3)
yrsFam	0.55	0.83	−0.07
yrsSchool	0.59	−0.44	−0.68
yrsCountry	0.59	−0.33	0.73
Proportion of Variance	0.64	0.20	0.16
Cumulative Proportion	0.64	0.84	1.00
Cluster 3
	Formal Dual-Language Acquisition/Exposure (PC1)
acqTapes	0.71
expLab	0.71
Proportion of Variance	0.76
Cumulative Proportion	0.76

Three components were extracted from the first cluster, with the most significant representing dual-language proficiency and early dual-language acquisition. This component was most strongly (positively) correlated with measures of bilingual proficiency and duration of bilingual experience (i.e., our proportion measure of AoA). The second component was associated with single-language exposure and was most strongly (negatively) correlated with measures of bilingual exposure through radio and reading. The third component indexed dual-language exposure in informal contexts and was most strongly (positively) correlated with measures of bilingual exposure through family and television. An additional three components were extracted from the second cluster, each representing durations of dual-language immersion in different contexts (general, family, country). Lastly, a single component was taken from the third cluster, representing dual-language acquisition and exposure in formal contexts (language platforms, laboratories). The collin.fnc function in the languageR package (Baayen, 2013) was used to verify that there was no collinearity among the final selected variables (κ = 2.18).

3.2.2. Effects of bilingual experience on memory

Mean competition effects (memory accuracy for competitors minus controls) were entered as outcome variables in a linear mixed-effects model with fixed effects of Input, Competition Type, each of the seven individual difference measures (unrotated principal components), and all two- and three-way interactions between Input, Competition Type, and each measure, as well as a random intercept for participant. There was a significant two-way interaction between Input and Competition Type (Estimate = −0.30, SE = 0.09, t(36) = −3.38, p = .002), with greater effects of phonological competition on memory in the word condition (M = 0.22, SD = 0.22) than in the sound condition (M = 0.15, SD = 0.12; Estimate = 0.15, SE = 0.08, t = 2.04, p = .048), and greater effects of semantic competition on memory in the sound condition (M = 0.22, SD = 0.14) than in the word condition (M = 0.15, SD = 0.16; Estimate = −0.15, SE = 0.06, t = −2.57, p =.014).

A number of interactions emerged with bilingual individual difference measures, including a two-way interaction between Competition Type and Dual-Language Proficiency + Acquisition (Cluster1_PC1; Estimate = −0.04, SE = 0.01, t(36) = −2.53, p = .016), between Competition Type and general Dual-Language Immersion (Cluster2_PC1; Estimate = 0.08, SE = 0.03, t(36) = 2.42, p = .021), and a three-way interaction between Input, Competition Type, and Informal Dual-Language Exposure (Cluster1_PC3; Estimate = −0.17, SE = 0.07, t(36) = −2.29, p = .028; see Table A3 of the Appendix for full output). Follow-up analyses revealed that individuals with greater dual-language proficiency and acquisition were significantly less impacted by semantic competition at memory retrieval (that is, less likely to have better recognition of semantic competitors relative to controls) compared to those with less dual-language experience (Estimate = −0.03, SE = 0.01, t = −2.99, p = .005). Dual-language proficiency and acquisition had little effect on how much phonological competition affected memory (p = .476). There was, however, a marginally significant effect of Dual-Language Immersion (general) for the impact of phonological (but not semantic; p = .250) competition (Estimate = −0.06, SE = 0.03, t = −1.96, p = .057), where experience living in multilingual environments was associated with reduced effects of competition on memory (see Figure 4). We therefore find evidence that the impact of both semantic and phonological competition on memory is reduced with greater bilingual experience, but that specific aspects of language history and ability have differential effects depending on the source of the competition. While the impact of semantic competition declines with earlier acquisition and greater proficiency across multiple languages, the effect of phonological competition declines with longer durations of immersion in bilingual environments.

Figure 4.

Interactive effects of competition and language experience on visual memory. Effects of semantic competition on memory decreased with greater dual-language proficiency and earlier acquisition (left), while effects of phonological competition on memory decreased with greater dual-language immersion (right).

When the effects of individual differences were broken down by input condition, we observed that the influence of Dual-Language Proficiency/Acquisition on memory for semantic competitors (relative to controls) was more pronounced when the target was cued by a word (Estimate = −0.04, SE =0.01, t = −2.72, p = .014) than by a sound (Estimate = −0.01, SE = 0.01, t = −1.40, p = .181), whereas the effect of Dual-Language Immersion on memory for phonological competitors (relative to controls) was greater for sounds (Estimate = −0.06, SE = 0.02, t = −2.30, p = .034) than for words (Estimate = −0.06, SE = 0.05, t = −1.21, p = .240). Lastly, Formal Dual-Language Acquisition/Exposure was found to reduce the impact of phonological competition on memory in the sound (Estimate = −0.08, SE = 0.02, t = −3.73, p = .002), but not the word condition (p = .666)ⁱⁱⁱ.

In sum, we observed that the influence of semantic competition was greater in response to sounds than words, and of phonological competition in response to words than sounds. This pattern is consistent with our prediction that effects of competition on memory should be most pronounced when competing representations are activated via more direct routes of processing (i.e., sounds to semantic representations and words to phonological representations). We additionally found that individual differences in bilingual experience predicted the size of competitor effects, and that the impact of bilingualism differed depending on the type of competition (semantic, phonological), type of input (sounds, words), and form of dual language experience (proficiency/acquisition, immersion, exposure).

4. Discussion

The goals of the present research were twofold. First, we sought to determine whether phonological and semantic similarities between objects seen during a visual search would impact how well the objects were later remembered. Second, we examined whether auditory input (sounds vs. words) and individual differences in language experience (bilingual proficiency, acquisition, and exposure) would influence the degree to which memory was influenced by semantic and phonological competition.

We discovered that objects that were phonologically (e.g., cat-cast) or semantically (e.g., dog-fox) related to targets were later remembered better than control objects. Furthermore, the effect of competition on memory was moderated by the type of auditory input used to cue the target during visual search – the effect of phonological competition on memory was most robust for spoken words (e.g., “cat”), while the effect of semantic competition on memory was stronger for environmental sounds (e.g., “meow’). We also find that bilingual language expertise and experience lessened the degree to which memory was influenced by the presence of competing distractors, with different aspects of bilingual experience reducing the impact of phonological and semantic feature overlap.

4.1. Auditory cues and visual search

The better recognition of previously observed competitor objects extends earlier work on visual perception and attention, showing that patterns of cascading activation that had been shown to influence online processing of visual scenes have downstream consequences for how information is stored in long term memory. The moderating effects of auditory input are consistent with models of multisensory cognitive processing (e.g., Chen & Spence, 2011), as well as with prior eye-tracking research (Bartolotti et al., 2020) suggesting that listening to words vs. sounds elicits different routes and magnitudes of phonological and semantic activation. Specifically, hearing a word representing a target object should directly activate phonological representations, making similarities with competitor labels especially salient and subsequently, the effects of phonological competition on memory greater for words than for sounds. Hearing a characteristic sound of the target, on the other hand, is hypothesized to directly activate semantic features, highlighting conceptual relationships between target and competitor objects and resulting in greater semantic competition in response to sounds compared to words.

The interactive effects of auditory cue and competition type indicate that words and sounds elicit different degrees of competition through their indirect routes of processing. Recognition accuracy for phonological and semantic competitors was higher than for control items regardless of auditory input, providing tentative evidence that lexical and conceptual information about visual objects can be indirectly activated by either sounds (e.g., “meow” [input] → cat [concept] → “c-a-t” [word] → “c-a-s-t” [word]) or words (e.g., “dog” [input] → “d-o-g” [word] → dog [concept] → fox [concept]). The effect of phonological competition on memory, however, did not reach significance when the target was cued by a sound, whereas effects of semantic competition on memory emerged for both words and sounds (albeit to a lesser extent when the target was cued by a word).

We additionally observed that objects in semantic sets were generally recognized better than those in phonological sets. While this finding should be interpreted with caution given that different items were used for displays containing phonological and semantic competitors, it is largely consistent with priming studies showing that drawing attention to semantic (vs. phonemic) features of items elicits deeper levels of processing and encoding to explicit memory (Craik et al., 1994; Richardson-Klavehn & Gardner, 1996; 1998). Though such effects have most often been produced by instructing participants to respond to semantic vs. phonemic characteristics of words, the present findings may indicate that the mere presence of semantic vs. phonological competitors in a visual display can elicit corresponding levels of processing for all visible objects, producing superior explicit memory for those in semantic than phonological sets.

In sum, we provide evidence that cuing targets of a visual search with a characteristic sound increases the influence of semantic overlap on visual memory, while cueing targets with a spoken word increases the influence of phonological overlap. How well visual objects are remembered therefore depends on auditory features that were present during encoding, revealing the impact of the brain’s cross-modal interactive architecture on long term memory.

4.2. Individual differences in bilingual experience

We propose that bilingualism may reduce the impact of competing representations on memory through a number of different mechanisms, including 1) more distributed activation of lexical and semantic representations (i.e., related words within and across languages), which could dilute the salience of any given competitor, 2) less frequent exposure to the words and sounds of a single language, which could attenuate the activation of any given competitor, and 3) better cognitive control over competing representations due to long-term experience managing co-activation within and across languages.

Indeed, we found that when language background was operationalized as a categorical variable, effects of competition on recognition memory were greater among monolinguals (i.e., native English speakers reporting no Spanish experience) compared to bilinguals (i.e., native English or Spanish speakers with proficiency in both languages). Notably, however, the smaller difference between competitor and control recognition in the bilingual group was primarily driven by enhanced memory for unrelated controls relative to the monolingual group, rather than by reduced memory for competitors. One possibility is that this pattern stems from distinct effects of bilingualism on top-down vs. bottom-up attentional guidance during visual search. Hernández et al. (2012) observed that, compared to monolinguals, bilinguals experienced less interference from irrelevant objects held in WM, but were equally distracted by salient, irrelevant objects in the display (e.g., a singleton). Bilingual experience may have therefore enhanced the control of interference from competing phonological and semantic representations held in WM (reducing the difference between memory for competitors vs. controls) without attenuating, and perhaps even increasing, stimulus-driven attentional processing of the visual objects in the display. Indeed, given that the search display preceded the onset of the target cue, there was a 500 ms period during which selective attention to a particular object would have been unnecessary and even counterproductive.

Bilinguals’ better memory for unrelated controls may also reflect a more general enhancement in episodic memory encoding (Kerrigan et al., 2017; Schroeder & Marian, 2012; Wodniecka et al., 2010). Schroeder and Marian (2012) observed that incidental memory for previously-seen pictures was greater among older bilinguals compared to monolinguals, which increased with earlier second language acquisition and longer durations of dual-language use. Consistent with neuroimaging studies implicating frontal lobe activity in episodic encoding (Blumenfeld & Ranganath, 2007; Buckner, Kelley, & Petersen, 1999; Kapur et al., 1994), the authors found that memory improved with better executive functioning, which was likewise greater among bilinguals than monolinguals. It may therefore be the case that the overall pattern observed in the present study stemmed from a combination of reduced competition from internal representations and enhanced encoding of visual stimuli for bilinguals compared to monolinguals.

Importantly, while we observed that the cumulative impact of bilingualism could be detected even when variability in language background was reduced to a coarse categorical metric (monolinguals vs. bilinguals), a more nuanced approach was necessary to detect the contributions of different forms and degrees of bilingual experience. When bilingualism was instead treated as a multidimensional continuous variable, we observed that individual differences in language experience modulated the influence of semantic and phonological competition on memory, with variable effects depending on the type of competition, as well as the type of experience. Earlier and greater bilingual proficiency was associated with a reduced effect of semantic competition on memory (i.e., smaller differences between competitors and controls), while greater bilingual immersion reduced the impact of phonological competition on memory. Similar patterns were observed when analyses were restricted to those with dual-language experience, indicating that individual differences among bilinguals predict effects of competition on memory, which would have been obscured by a categorical treatment of monolinguals vs. bilinguals. Together, these findings suggest that in order to obtain a more complete picture of how bilingualism influences cross-domain cognitive processes, we must account for the demands associated with particular tasks, as well as for specific language experiences.

Interestingly, the impact of bilingualism also varied depending on whether targets were cued by words or sounds, with relatively stronger effects of bilingualism on memory for phonological competitors in response to sounds and memory for semantic competitors (relative to controls) in response to words. While greater cognitive control could presumably reduce the effects of competition in response to either words or sounds, the impact of either more distributed or weaker activation of competing representations would be expected to differ depending on the interaction between bottom-up inputs and top-down experiences. For phonological competition in response to sounds, more extensive immersion and exposure to multiple languages may increase the likelihood that items in a visual display will automatically activate corresponding linguistic labels in not only English (e.g., “cat,” “cast,” “fork,” “apple”), but also other known languages (e.g., “gato,” “enyesado,” “tenedor,” “manzana”). Consequently, phonological relationships that exist in one language (e.g., cat-cast) may be less salient overall, resulting in a smaller effect of phonological overlap on memory. It may be the case, however, that the overt presentation of an English label could boost the salience of English competitors (relative to cross-linguistic competitors), thereby attenuating the effect of bilingual immersion when targets are cued by words rather than sounds. In the case of semantic competition in response to words, because the identification of a picture based on a spoken word (e.g., “dog”) requires access to linguistic representations associated with the auditory and visual stimuli, the activation of a greater number of lexical candidates (both within and across languages) by those with advanced bilingual proficiency could reduce semantic integration and the subsequent impact on memory. On the other hand, because it is not necessary to access linguistic labels in order to identify a picture based on a sound (e.g., <woof>), bilingual experience may have a lesser effect on how semantic competitors are processed and remembered. A similar pattern was observed by Friesen et al. (2016), who found that, compared to monolinguals, bilinguals had reduced semantic integration of related pictures (e.g., a fly and a bee) when the visual stimuli appeared concurrently with a verbal target cue (e.g., “fly”), but not when the pictures were presented without the verbal cue.

In addition to potential mechanisms attributable to the activation of competing representations, the present findings may reflect effects of different forms of bilingual experience on the ability to inhibit task-irrelevant representations. Given that participants consistently heard the target cue as a single type of input (either words or sounds), it would be reasonable to expect that participants would have activated the task-relevant features of the visual objects during the display preview more (i.e., phonological representations in the word condition and semantic representations in the sound condition). While bilingualism has already been shown to enhance control over irrelevant representations held in memory (e.g., Chabal, Schroeder, & Marian, 2015; Friesen et al., 2015; Hernández et al., 2012; Kuipers & Westphal, 2021; Macnamara & Conway, 2014), the effects observed in the sound condition may indicate that the ability to inhibit task-irrelevant representations (in this case, object labels) is particularly facilitated by longer immersion in bilingual environments. Such an effect would be consistent with prior work indicating that greater experience in dual-language contexts and more frequent language-switching facilitates the ability to suppress irrelevant goals when switching tasks (i.e., task-set reconfiguration, Gullifer et al., 2018; Hartanto and Yang, 2016). Similarly, the reduced competition in the semantic-word condition with greater dual-language proficiency and acquisition could reflect an enhanced ability to suppress task-irrelevant information (in this case, semantic features), which would be largely consistent with previous findings that higher bilingual proficiency and earlier bilingual acquisition can enhance reactive inhibitory control (e.g., such as in AX-CPT (Beatty-Martinez et al., 2019; Gullifer et al., 2018), Simon (Kousaie et al., 2017) and Flanker tasks (Chung-Fat-Yim et al., 2020; Luk, De Sa, & Bialystok, 2011)).

Together, our findings indicate that in addition to the competition that can arise from co-activations of overlapping linguistic, semantic, and visual representations, specific experiences with more than one language influence the extent to which competition during visual search has a lingering impact on subsequent memory. In addition to obtaining performance measures of domain-general cognitive control and lexical access, future research incorporating methods such as neuroimaging and ERP (Cansino & Trejo-Morales, 2008; Duarte et al., 2006; Gullifer et al., 2018; Guo et al., 2006; Ranganath et al., 2003; Wilding & Sharpe, 2003), eye-tracking (Chanon & Hopfinger, 2008; Holm & Mäntylä, 2007; Laeng & Teodorescu, 2002; Ryan, Hannula, & Cohen, 2007; Pertzov, Avidan, & Zohary, 2009; Van der Linde et al., 2009) and measurements of pupillary responses (Goldinger, He, & Papesh, 2009; Kafkas & Montaldi, 2011) could further elucidate the perceptual, attentional, and memory processes mediating the effects of bilingualism on memory.

4.3. Conclusion

The present study found that related meanings and labels of objects in a visual search can influence subsequent memory for what was previously seen. We additionally provide evidence that the impact of form and meaning overlap on visual memory differs as a function of input and of individual differences, with effects varying depending on whether targets of the search are cued by a word or a sound, and whether individuals engaged in the search come equipped with bilingual abilities and experiences. Specifically, bilingualism was found to attenuate the influence of phonological and semantic overlap on memory, which we suggest may be due to a more distributed network of activation across the two languages, less frequent exposure to items in a single language, and differential deployment of cognitive control.

In conclusion, the present study demonstrates that semantic and phonological features associated with the objects that we see can impact not only what we attend to in the moment, but also what we remember after the fact. The degree to which our memories are influenced by related information varies as a function of our immediate multisensory environment interacting with our language experiences accumulated over a lifetime.