Skip to main content
NCBI home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2026 Apr 18.
Published in final edited form as: Psychon Bull Rev. 2026 Feb 18;33(2):86. doi: 10.3758/s13423-025-02842-y

Task irrelevant sounds influence visual attention through graded crossmodal semantic modulation

Kira Wegner-Clemens 1, George L Malcolm 2, Dwight J Kravitz 1, Sarah Shomstein 1
PMCID: PMC13088902  NIHMSID: NIHMS2151847  PMID: 41709062

Abstract

The role of semantics in guiding attentional prioritization visually is well established, but how semantic information shapes attention in multisensory, rather than unisensory, contexts has been less well characterized. Task-irrelevant sounds can speed search for matched visual targets; e.g., you find a dog more quickly if you hear it barking. However, matched sounds and images may be a unique case, with the attentional benefit derived by integration into a combined multisensory event. To establish if semantic information influences attention in audiovisual contexts beyond direct matches, we systematically varied sound-image semantic relatedness while participants searched for target images. Search speed scaled with sound-image semantic relatedness, such that target images were found faster when the sound was more closely related to the image. To examine whether sound-image semantic scaling hinges on task relevance, we conducted two follow up experiments while participants completed an orthogonal Gabor discrimination task. Observed results demonstrate that semantic knowledge guides audiovisual attention, even when task-irrelevant and not matched, suggesting a robust underlying cognitive mechanism processing semantics both within and across sensory modality that goes on to influence attentional allocation.

Keywords: attention, audiovisual, multisensory, semantics

Introduction

Only a subset of the sensory information available at any moment can be processed due to constraints on neural processing. Recently, research aimed at uncovering mechanisms of attentional selection focused on elucidating the degree to which semantic information, intrinsic to objects and scenes, influences attention (Awh et al., 2012; Shomstein et al., 2022). Everyday environments are not random assortments of sensory information but, rather, are meaningfully organized. A shopper in a grocery store knows which items belong to an overarching category (e.g., lemons and limes are citrus fruits), what items often co-occur (e.g., mints and magazines are both found the checkout line), and what items share a function (e.g., corkscrew and a bottle of wine). This semantic understanding varies not only in type of relationship, but also in degree, with some items more related than other items (e.g., lemons and limes are more closely related than oranges and dragonfruit, but all belong to the same category of fruit). This information guides attention: semantically meaningful locations in a scene are fixated longer (Peacock et al., 2019) and gaze is more likely to shift between semantically related objects (Wu et al., 2014). Even when semantic meaning is not task relevant, targets near objects semantically related to previously seen objects are found more quickly (Malcolm et al., 2016) and semantically related objects are better decoded in early visual cortex (Nah et al., 2021). Interestingly, this research has focused largely on binary definitions of semantic relatedness such as shared and unshared category but has not examined how semantic guidance of attention might vary by degree of semantic relatedness.

Semantic relatedness could also impact attention in multisensory contexts, where some sensory signals can be integrated into a multisensory object. Prior research has established an attentional benefit for congruent audiovisual pairs. When a congruent sound is played, target images are found more quickly in search arrays (Iordanescu et al., 2008, 2010) and real-world scenes (Kvasova et al., 2019; Maezawa et al., 2025) and participants perform better in a Gabor discrimination task at the same location (Mastroberardino et al., 2015). Even when freely viewing a scene, objects are more likely to be fixated if a matched sound is played (Kvasova et al., 2024). While these results show congruent identity pairs modulate attention, it has not been shown whether other semantic relationships, such as shared category, can modulate attention (Fig. 1A; Wegner-Clemens et al., 2024). Distinguishing the effect of congruent identity pairs from more distantly semantically related pairs is particularly important in a multisensory context because a congruent pair could potentially be integrated into a multisensory event, which is preferentially attended over unisensory information (Lunn et al., 2019; Santangelo & Spence, 2007; Van der Burg et al., 2008). The attentional priority for congruent pairs could reflect a benefit for integrated multisensory objects, rather than a benefit for semantically related sounds and images. More distantly related pairs must be investigated to establish semantic guidance of attention across audition and vision.

Figure 1.

Figure 1.

Open in a new tab

Two possible approaches to defining semantic relatedness. Top row, items are characterized as either related or unrelated to the orange in a binary approach; Bottom row, items are characterized in a continuous manner in reference to the orange. B. Semantic relatedness values from the Sound-Sight Semantics Database, derived from human judgments about these sounds and images. Values are only available for items within the same category. C. Semantic similarity values extracted from a fastText model, trained on a large text corpora. Values are available for both items within the same category and items across categories.

To establish crossmodal semantic guidance of attention, we systematically varied semantic relatedness between a sound and a target image in both a visual search task and an orthogonal Gabor discrimination task. If the semantic relationship between a sound and image can influence attention, search speeds should scale with degree of audiovisual semantic relatedness. Furthermore, if this is a broad modulation of attention rather than a task strategy, the relationship between degree of audiovisual semantic relatedness and speed should hold even when the semantic identity of the sound is task irrelevant, such as in an orthogonal Gabor task. In both of our task types, we see modulation that scales with degree of semantic relatedness, providing strong evidence for task irrelevant semantic guidance of attention in a graded crossmodal manner.

Materials and Methods

To test the influence of audiovisual semantic relatedness on attention, we conducted three experiments where participants listened to sounds during a visual task. In the visual search experiment (Exp. 1), participants identified the location of an image, while in the two gabor discrimination experiments (Exps. 2 and 3), participants made judgments about a Gabor patch that appeared in the same location as a previously presented image. The gabor experiments differed by the relationship between the sound and the images. In the same category experiment, all sounds and images were in the same category. In the different category experiment, the sound matched one of the images and was in a different category from the other.

Participants:

All participants were recruited from the university subject pool and compensated with course credit. All reported normal hearing and normal or corrected-to-normal vision. Participants provided informed consent and experiments were performed according to a protocol approved by the institution’s review board. Sample sizes were preregistered on OSF (experiment 1: osf.io/qxz6g; experiments 2 and 3: osf.io/qrhe7)

Experiment 1 (Visual Search):

The final sample included in the analysis was 122 participants (94 women, 26 men, 3 other or not reported; mean age 20). Seventeen participants were removed for low accuracy (<80%) on either catch or task trials.

Experiment 2 (Gabor Discrimination, Same Category):

The final sample included in the analysis was 200 participants (160 women, 35 men, 5 other or not reported; mean age 20). A total of 181 participants were removed for low accuracy (<80%; 167 participants) or because more than 10% of trials were removed after RT trimming (14 participants).

Experiment 3 (Gabor Discrimination, Different Category):

The final sample included in the analysis was 200 participants (161 women, 36 men, 3 other or not reported; mean age 21). A total of 135 participants were removed for low accuracy (<80%; 125 participants) or because more than 10% of trials were removed after RT trimming (10 participants).

Power analysis:

To determine the necessary sample size in Experiment 1 (Visual Search), a simulation-based power analysis was conducted using data from a pilot Experiment and the “simr” package (Green & MacLeod, 2016) in R. This analysis showed 120 participants as a sufficient sample size to achieve 90% power of a true effect half the size of the pilot data.

To determine the necessary sample sizes for Experiment 2 and 3 (Gabor Discrimination), power was calculated separately for the distinct analyses in each task. A G*Power analysis determined that 30 participants was sufficient to achieve 95% power on a paired t-test between valid and invalid trials, which reflects the planned analysis of the validity effect in both studies. In Experiment 3, there was a second planned analysis examining how validity benefit scales with semantic relatedness. To determine sufficient power for this planned analysis, we conducted a simulation-based power analysis using pilot data from the task. Based on 1000 simulations of pilot data on the task, a sample size of 200 was sufficient in order to observe an effect in a linear mixed model in at least 80% of simulations. To allow for exploratory analyses comparing the within and between studies, an equal 200 participants were collected for Experiment 3.

Stimuli and procedure:

Participants accessed the studies through a direct link on the university’s experiment platform. To ensure that participants could hear the sounds on their device, participants completed an auditory pretest that involved listening to an auditory clip of a word and identifying it among three distractors. In the main task, catch trials were also implemented to ensure that the sound was not muted during the visual discrimination trials - participants heard a stimulus sound and identified the sound from a set of four words. Participants who did not answer 80% of the catch trials correctly were removed from the dataset.

Images, sounds, and semantic relatedness values were taken from the Sight Sound Semantics database (Wegner-Clemens et al., 2022). The database values were selected as the primary measure of semantic relatedness over text corpora models as they were directly derived from human similarity judgments and allow for a more direct interpretation of the meaning of values (Fig 2B). As an additional exploratory analyses, and in order to explore similarities/differences with human semantic rating, text-based measures of semantic relationships were also used to further understand the data and to estimate semantic relatedness of items in different categories, where values were not available in the database (Fig 2C). Similarity values were extracted with the Gensim library (Řehůřek & Sojka, 2010) using a pretrained fastText model, “fasttext-wiki-news-subwords-300” (Mikolov et al., 2017). This model uses a text-corpora of 650 billion words from Wikipedia, news data sets, and websites to create vector representations of each word. The model uses a combination of word co-occurrence, word position weighting, phrase representations, and subword information for improved performance on a semantic analogy test relative to other models (Mikolov et al., 2017). Text derived semantic measures are referred to as semantic similarity for clarification and to distinguish the measure from the database values of semantic relatedness derived from human judgements.

Figure 2.

Figure 2.

Open in a new tab

A. Experiment 1 trial display, with cue word shown and the four possible image options shown around it. Below, Depiction of the possible sound-image relationships for an image of a cat. In a match vs related analysis approach, the sound associated with cat (“meow”) is a match and all other sounds are related (non-match). In a continuous analysis approach, non-match sounds are analyzed by the degree of their relationship. The numbers shown below the images representing each sound are the semantic relatedness values (derived from the Sight Sound Semantic database) between the given sound and the target image of a cat. B. Experiment 2 and 3 trial design. Images and sound were presented for 1000 ms, followed by discrimination task until response or 5000 ms. Participants reported the rotation direction of the Gabor patch (left). Below, Schematic of conditions. On a valid trial (top), the image congruent to the sound appears in the same location as the target, while on an invalid trial (middle), the sound congruent image and target appear in different locations. The sound congruent image is shown by a pink solid outline. The target location is shown with a dashed yellow outline.

Experiment 1 Design (Visual Search):

Participants were presented with a display of four object images and a cued word in the middle (Fig. 2A) and instructed to press the arrow key corresponding to the location of the image that matched the cued word. A sound from the target image’s category played simultaneously with the visual display. Responses were collected after the images and the sound were presented for at least 500 ms. The sound and display continued for 1000 ms or until the participant responded, at which point a fixation appeared before the next trial started. Sounds were always from the same category as the target image. For each target image, there were 10 possible sounds from the same category that could play, each of which varied in semantic relatedness (Fig. 2). Each participant saw every possible sound-image pair twice, for a total of 600 trials. Trial order was randomized. To avoid interaction between the sound and the distractor images, all four images within a trial were from different categories (animals, instruments, household items, and a miscellaneous category of additional images that did not have an associated sound). On 30 catch trials, instead of a cued word and set of images, four words appeared along with a sound that matched one of the words. Participants were instructed to select the word that matched the sound. Every sound appeared as a catch trial. These catch trials could appear at any point during the experiment and ensured that participants did not mute the sound during the task.

Experiment 2 and 3 Design (Gabor Discrimination):

The same images and paradigm were used in both experiments, with only the relationship between the sound and the sound incongruent image changing. On each trial, a sound and two images were presented simultaneously. The sound always matched at least one of the images, which we refer to as the “sound congruent image.” If the sound was a drum roll, the congruent image would be a drum and the incongruent image could be an animal (for the different category manipulation, in Experiment 2) or an instrument (for the same category manipulation, in Experiment 3). Only two categories (animals and instruments) were used in these experiments. On neutral trials, which appeared only in Experiment 3, the sound would match both images, so if it was a drum roll, two drums would be presented. The images and sound were presented for 1000 ms, then removed from the screen. Participants then immediately saw a Gabor patch and a checkered patch that consisted of two overlapped Gabors presented at the same locations as the images and needed to report whether the Gabor patch was rotated clockwise or counterclockwise of vertical. The Gabor stimuli remained on screen until either the participant responded or 5000 ms had passed. Each participant completed 400 task trials, such that every image was paired with every other image in the other category (for Experiment 2) or in the same category (for Experiment 3), including in this case pairing the image with itself in the neutral trials. Participants saw each sound-image pair twice, once in the target valid and once as invalid. Stimulus location, target direction, and stimulus order were counterbalanced through the creation of distinct sequences per experiment such that every sound image pair appeared in a trial with each target location (left vs right) and each target rotation (clockwise vs counterclockwise). Each sequence was then reversed to avoid any effect of stimulus order, for a total of 8 distinct experiences per experiment. Participants additionally completed 20 catch trials, where a sound was presented and participants needed to select the matching option among 4 words. Catch trials appeared randomly.

Trials could either be valid or invalid depending on the relationship between the target location and the location of the image matching the sound. In valid trials, the sound matches the image that is at the location where the target later appears, while in an invalid trial, the sound matches the image where the distractor later appears (Fig 2B). If hearing a sound leads to increased attentional prioritization for the location of a matching image, the response times should show a validity effect, where participants report the Gabor patch’s direction more quickly on valid trials than invalid trials. Neutral trials were expected to intermediate RTs between valid and invalid trials since attention would not be biased towards either location.

In addition to the validity effect, this paradigm can also investigate graded semantic scaling. In Experiment 2, the sound matched the congruent image and was in a different category than the incongruent image, while in Experiment 3, the sound again matched the congruent image but now the sound was in the same category as the incongruent image. The sound is always expected to increase attentional priority for the congruent image, which would appear as a validity effect. Interestingly, though, there is also a semantic relationship between the sound and incongruent image in Experiment 3 that could modulate attention. If graded semantic scaling modulates attention in task irrelevant contexts, then attentional priority to the incongruent image is expected to vary by the semantic relationship between that incongruent image and the sound. In the example trial presented in Figure 2B, a meow sound is expected to prioritize the cat image as a match, but is also expected to prioritize the dog image to a lesser extent as an image with a strong semantic relationship. The attentional priority to both images should result in distributed attention priority across both locations and a weaker validity effect in scale with degree of semantic relatedness between the sound and incongruent image. In this case, even though it is both task irrelevant and detrimental to task performance to attend to the incongruent image, the sound modulates attentional priority. If the validity effect scales with the degree of semantic relatedness between the sound and incongruent image, there is strong evidence that sounds modulate attention as part of ongoing processing.

All t-tests, correlations, and analyses of variance (ANOVAs) were conducted in Python with the Pinguion statistical package (Vallat, 2018). All RT analyses were conducted on correct trials only. In Experiment 1, trials with response times (RTs) over 3000 ms were removed to eliminate outliers (1.7% of accurate trials). In Experiment 2 and 3, trials under 250 ms and over 2000 ms were removed to eliminate outliers (Expt 2: 1.8%; Expt 3: 1.7%). Different trimming criteria reflect the nature of tasks: slow search task in Experiment 1 and faster orientation discrimination task in Experiments 2 and 3.

To better understand how various factors contribute to attention allocation, a series of mixed effect models were constructed in R with the lme4 statistical package (Bates et al., 2015). Participant, image, and target location were included as random effects. Raw RTs were used in these analyses rather than z scored RTs since participant is included as a random effect. Semantic relatedness and category were modeled as fixed effects both alone and in combination. Participant, image, target location, and category were modeled as factors (categorical variables), while semantic relatedness was modeled as a continuous variable. A base model was constructed and then combinations of random and fixed effects were iterated through. Models that reached a stable convergence were evaluated with the Aikake Information Criteria (AIC) which accounts for prediction error and number of parameters to prevent overfitting. Further details about the models tested as well as evaluation with Bayesian Information Criterion (BIC) is available in the supplement.

Results

Experiment 1 (Visual Search):

Participants found at the target image a high level of accuracy (mean: 97% ± 3%) and at a similar level on matched (97%) and unmatched (97%) trials.

Response times, Match vs. non match:

Participants were significantly faster on match trials than non-match trials in a paired t-test of z-scored participant averages (t(121) = −8.93, p<10−15). Averaged search speed by match condition is shown as both RT in ms (Fig. 4A) and z-scored RT (Fig. 3B). This result confirms our paradigm elicits the previously described congruent search benefit (Iordanescu et al., 2008; Kvasova et al., 2019). In a two-level repeated measures ANOVA of z-scored participant average RTs with sound-image congruency (match or non-match) and category (animals, instruments, household), we observed main effects of sound-image relation (F(1,121)=78, p=10−14), category (F(2,242)=80, p=10−27), and a significant interaction (F(2,242)=4.3, p=0.01). Average search speed by sound-image congruency and category are shown as RT in ms (Fig. 3C) and z-scored RT (Fig. 3D). In planned post hoc paired t-tests, z-scored RT was significantly faster for match over non-matched in each category (animals: t(121)=−7.5, p= 10−11; instruments: t(121)=−7.5, p=10−11; household items: t(121)=−6.1, p=10−8), suggesting that the matched sound benefit is not category specific. There was a categorical difference in overall z-scored RT, with animal targets found fastest on both match (animal vs. instrument: t(121)=−5.7, p=10−8; animal vs. household items: t(121)=−4.6, p=10−7); and non-match trials (animal vs. instrument: t(121)=−17.8, p=10−35; animals vs. household items: t(121)=−12.1, p=10−22). There was no significant difference in z-scored RT between instruments and household items on match trials (t(121)=1.7, p=0.09), but was for non-match trials (t(121)=3.1, p=0.002).

Figure 4.

Figure 4.

Open in a new tab

A. Mean RT for valid and invalid conditions in Experiment 2 (Different Category context). Error bars are standard error of the mean. B. Mean z-scored RT for valid and invalid conditions in Experiment 2. C. Mean RT for valid, invalid, and neutral conditions in Experiment 3 (Same Category context). D. Mean z-scored RT for valid and invalid conditions in Experiment 3. E. Mean validity benefit (RTinvalid – RTvalid) for sound-image pairs correlated with each pair’s semantic relatedness. Each dot represents a sound-image pair. The correlation line is overlaid in gray. F. Mean validity benefit (RTinvalid – RTvalid) for sound-image pairs by the pair’s semantic similarity (text corpora). Each dot represents a sound-image pair. The correlation line is overlaid in gray. Sound-image pairs in different categories shown in pink. Sound-image pairs in the same category shown in teal.

Figure. 3.

Figure. 3.

Open in a new tab

A. Mean RT for match and related conditions. Error bars for all bar charts in this figure are standard error of the mean. B. Mean z-scored RT for match and related conditions, by target category. Animal is shown in blue, instrument in green, household items in red. Match is shown in darker shaded and related (every pair except match) in lighter shades. C. Mean RT for match and related conditions, by target category. D. Mean z-scored RT for match and related conditions, by target category E. Mean z-scored RT for a target image and sound pair correlated with each pair’s semantic relatedness, as defined by the Sight Sound Semantic database. Each dot represents a sound-image pair. The correlation line is overlaid in gray. Correlation without match trials is shown separately for F. instrument trials, G. animal trials, and H. household items.

Graded effect of semantic relatedness on response times:

To examine if search speeds are sensitive to modulation in semantic relatedness, we correlated average RTz for each sound-image pair with that pair’s semantic relatedness and found a significant correlation. Images were found more quickly when the sound was more closely related (r=−0.16, p=0.003; Fig. 3E). When separated into categories, the correlation held for instruments (r=−0.33, p=0.007) and animals (r=−0.32, p=0.001), but not household items (r=−0.04, p=0.63). Each correlation was then repeated with match trials removed to test the effect of semantic relatedness alone. With the match trials removed, semantic relatedness correlated with RTz for instruments (r=−0.22, p=0.03; Fig. 3F) and animals (r=−0.22, p=0.03; Fig 3G), but not household items (r=0.17, p=0.1; Fig. 3H).

To better understand how factors combine to influence attention, a series of models was constructed with one set predicting RT including match trials and one set not including match trials. The best fit model including match trials had semantic relatedness as a fixed effect and participant, image, and target location as random effects. There was a significant negative effect of semantic relatedness on RT (estimate: −50.9; t=−8.22, p<2−16). The largest variability in random effects was from participant, then image, then target location. The best fit model without match trials had semantic relatedness, category, and the interaction between them as fixed effects and participant, image, and target location as random effects. Even with the match trials removed, there was a significant negative effect of semantic relatedness on RT (estimate: −38.50, t=−2.705, p=0.006).

Text derived semantic relatedness:

In an exploratory analysis, we found a significant correlation between RTz for a given sound-image pair and fastText semantic similarity for all trials (r= −0.2, p = 0.0004). When separated by category, this relationship held for animals (r= −0.3, p= 0.001) and instruments (r= −0.38, p= 0.0001), but not household items (r= −0.16, p= 0.09). When the match trials were removed, semantic similarity and RTz only correlated for instruments (r= −0.36, p = 0.00001), but not for animals or household items. Text-derived semantic similarity may capture a different aspect of semantics or simply be a noisier measure, but regardless of the reason, it raises concerns about text corpora similarity as a proxy of semantic relatedness of visual stimuli.

Experiment 2 (Gabor Discrimination – Different Category):

Participants heard a sound and saw two images, then reported the rotation direction of a subsequent Gabor patch. Gabor discrimination was expected to be faster when the Gabor appeared at a location where the image matched the sound. Participants reported the Gabor direction with high accuracy (95%) on both valid (96%) and invalid (94%) trials. As predicted, participants were significantly faster on valid trials (where the target appeared in the same location as the sound congruent image) than on invalid trials (where the target appeared in the same location as the sound incongruent image) in a two-tailed paired t-test of participant z-scored averages (t(99)=−9.7, p=10−16). Averaged search speed by validity condition is shown as z scored RT (Fig. 4A) and raw RT (Fig. 4B). This result suggests that sound modulates attention to a matched image, even when the sound and the image are irrelevant to the task.

Experiment 3 (Gabor Discrimination – Same Category):

The task was the same as in experiment 2, except the non-matching image and sound shared a category. Accuracy was high overall (95%) and similar for valid (96%) and invalid (94%) trials. Discrimination was expected to be faster on valid (e.g., sound matched image at the location where the target Gabor later appeared) than invalid trials (e.g., sound matched image at the location where the distractor later appeared), with the increase in speed modulated by the relationship between the sound and the incongruent image. Participants were significantly faster on valid trials than on invalid trials in a two-tailed paired t-test of z-scored RTs (t(199)=−7.6, p=10−12; Fig. 4D). Participants were faster on neutral trials (both images matched the sound) than invalid trials in a two-tailed paired t-test (t(199)=5.5, p=10−8; Fig. 4D), but not valid trials. This pattern of results suggests that distraction on invalid trials may influence RT more than prioritization on valid trials. When semantic relatedness between the sound and second image was correlated with the average validity effect (invalid z-scored RT – valid z-scored RT), higher semantic relatedness values produced weaker validity effects (r=−0.19, p=0.01; Fig. 4E).

To better understand how factors interact, a series of mixed effect models predicting RT were constructed. In the winning model, semantic relatedness, validity, and their interaction were modeled as fixed effects while participant, rotation, target location, sound, and sound incongruent image were modeled as random effects. In this model, both validity (p<10−14) and the interaction between relatedness and validity (p=0.0003) significantly predicted z-scored RT. Relatedness alone did not significantly predict z-scored RT (p=0.06). This pattern where relatedness only interacts with validity fits with the expectation that greater relatedness will increase priority for the unmatched image and decrease priority for the matched image (e.g., higher semantic relatedness results in slower RT for valid trials and faster RT for invalid trials).

Relationship between semantic similarity and discrimination speed:

The largest effects of semantic similarity are expected within category (e.g., between instrument sounds and images), as is tested in Experiment 3 (Gabor Discrimination – Different Category) and semantic relatedness values were only available within category, so initial analysis focuses on attention modulation in the same category context. However, images and sounds can still be related across category (e.g., piano and dog are more likely to co-occur together than a tuba and a horse). To better understand cross category semantic effects, we correlated average validity benefit for all possible pairs (RTz invalid – RTz valid) with semantic similarity and found a significant negative correlation (r=−0.12, p=0.02; Fig. 4F). However, when experiments were analyzed separately, there was no correlation between text similarity and same category pairs (r=−0.02; p=0.76) or the different category pairs (r= −0.11; p = 0.1).

Discussion

Semantic relationships are processed rapidly and go on to influence attention (Belke et al., 2008; Eckstein et al., 2006; Hwang et al., 2011; Moores et al., 2003; Neider & Zelinsky, 2006), even when not task relevant (Cornelissen & Võ, 2017; Malcolm et al., 2016; Nah et al., 2021; Peacock et al., 2019, 2019). However, prior research has focused on binary (related or not) relationships between objects and unisensory visual information. Work in multisensory contexts focused on sound and image pairs that matched in identity (Iordanescu et al., 2008; Kvasova et al., 2019; Maezawa et al., 2025; Mastroberardino et al., 2015). Here, we manipulated the semantic relationship between sounds and images in various degrees to ask a question of whether semantic relationships beyond exact matches influence attention in a graded crossmodal manner.

Consistent with prior studies (Iordanescu et al., 2008; Kvasova et al., 2019; Maezawa et al., 2025; Mastroberardino et al., 2015), we found that matching congruent identity sounds speed visual search, even when the sound does not provide any target location information. This result alone could be a consequence of increased attention to integrated audiovisual objects, so we additionally investigated how sounds that do not share a semantic identity, but may still be related, influence attention to a visual target. By varying the relationship of the sounds and images, in a graded manner, we can disambiguate an effect specific to congruent semantic identities and a more general semantic effect. In an analysis that included matched pairs, target images were found more quickly as semantic relatedness with the sound increased. Crucially, for two categories (animals and instrument), semantic relatedness significantly correlated with RTz, even when removing congruent identity pairs, establishing for the first time that semantic relationships modulate attention in a graded manner and that audiovisual semantic relationships influence attention. Importantly, even when semantics was rendered task irrelevant in an orthogonal Gabor discrimination task, participants were significantly faster when the sound matched the image at the location the target later appeared (valid trials) and the validity effect still scaled with semantic relatedness (as in the search task), even though a non-matched image with a weakly related pair was present. This result suggests semantic influence on attentional allocation is a default signal that is used to prioritize attentional allocation and even weak non-matched semantic relationships modulate priority.

In the search task (Expt 1), the effect was highly sensitive to category and type of semantic measure. This category sensitivity may reflect broader neural processing differences, such as the differences observed for animate items and tools in visual temporal cortex (Grill-Spector & Weiner, 2014). Alternatively, it may reflect categorical differences in the type or strength of audiovisual semantic relationships. For example, location defined categories (like household items) may influence attention less than categories defined by function (like instruments which function to make music) or inherent characteristics (like animals which are living). Prior experience could strengthen certain semantic relationships (Adam & Noppeney, 2010). Instruments and animals may be more likely to be seen and heard simultaneously (e.g., at a concert, you hear and see the piano), while the household items may be more likely to be experienced unimodally (e.g., paper is usually quiet). Certain categories may also be better captured by different methods of measuring semantic relatedness. Here, human judgement based semantic relatedness better predicted behavior than text based semantic similarity. The different measures may reflect different aspects of semantic knowledge, particularly because the text measure includes sub-word information that may not be relevant to semantics (Mikolov et al., 2017). Future research will be necessary to better understand differences by category and measure.

Uncovering the underlying mechanism of cross-modal semantic influence on attentional selection will be another important question for future research. The present study established that semantics modulates attentional priority in a continuous way, but several underlying mechanisms could instantiate such attentional modulation. For example, in the search task, it is possible that the sound’s relationship with the cue word has a priming effect, leading to faster search times for images related to the cue word. This priming account would be consistent with semantic facilitation effects observed between pictures and words (Kircher et al., 2009). However, priming via a cue word does not effectively explain the results from the Gabor discrimination task, since there is no relationship between the sound and the spatial location or Gabor patch. Another possibility is that the sounds and images may be processed concurrently and independently contribute to attentional priority for a location, rather than through a priming mechanism or direct multisensory processing. This separate-then-shared process fits with variable dual task results, where studies using spatial based tasks suggest a shared attentional capacity across modalities and object based tasks suggest separate attentional capacities (Wahn & König, 2017). Since this possibility does not rely on a cue word, it could explain the results from both the search and Gabor discrimination tasks. Another possibility is that unmatched sounds and images could be partially bound together or otherwise the sound and image directly influence each other to varying degrees, in line with results showing slowing in congruency judgments for partially perceived multisensory cues (Noel, et al 2019). Future research will be able to disambiguate these possibilities.

Ultimately, with these three experiments, we establish that crossmodal semantic processing influences attentional priority. Semantic relationships between signals from different sensory modalities (such as images and sounds) shapes attention in a similar manner to visual attention, suggesting the possibility of a shared mechanism by which semantic processing between signals, regardless of sensory origin, impinge on attention. We also show that audiovisual semantic information is not simply used in a goal directed manner, but rather, as in vision, broadly modulates attentional priority independent of whether it is relevant to the observer’s current goal. We also show semantic guidance operates in a graded manner where the magnitude of attention priority benefit scales with the degree of semantic relatedness between the image and sound. This result serves as evidence for the new reconceptualization of the way attentional guidance is instantiated, where various signals simultaneously contribute to an overall dynamic priority map that guides behavior.

Supplementary Material

Supplement

References

  1. Adam R, & Noppeney U (2010). Prior auditory information shapes visual category-selectivity in ventral occipito-temporal cortex. Neuroimage, 52(4), 1592–1602. 10.1016/j.neuroimage.2010.05.002 [DOI] [PubMed] [Google Scholar]
  2. Awh E, Belopolsky AV, & Theeuwes J (2012). Top-down versus bottom-up attentional control: A failed theoretical dichotomy. Trends Cogn. Sci, 16(8), 437–443. 10.1016/j.tics.2012.06.010 [DOI] [PMC free article] [PubMed] [Google Scholar]
  3. Bates D, Mächler M, Bolker B, & Walker S (2015). Fitting Linear Mixed-Effects Models Using lme4. J. Stat. Softw, 67, 1–48. 10.18637/jss.v067.i01 [DOI] [Google Scholar]
  4. Belke E, Humphreys GW, Watson DG, Meyer AS, & Telling AL (2008). Top-down effects of semantic knowledge in visual search are modulated by cognitive but not perceptual load. Percept. Psychophys, 70(8), 1444–1458. 10.3758/PP.70.8.1444 [DOI] [PubMed] [Google Scholar]
  5. Cornelissen THW, & Võ ML-H (2017). Stuck on semantics: Processing of irrelevant object-scene inconsistencies modulates ongoing gaze behavior. Atten. Percept. Psychophys, 79(1), 154–168. 10.3758/s13414-016-1203-7 [DOI] [PubMed] [Google Scholar]
  6. Eckstein MP, Drescher BA, & Shimozaki SS (2006). Attentional cues in real scenes, saccadic targeting, and Bayesian priors. Psychol. Sci, 17(11), 973–980. 10.1111/j.1467-9280.2006.01815.x [DOI] [PubMed] [Google Scholar]
  7. Green P, & MacLeod CJ (2016). SIMR : an R package for power analysis of generalized linear mixed models by simulation. Methods Ecol. Evol, 7(4), 493–498. 10.1111/2041-210x.12504 [DOI] [Google Scholar]
  8. Grill-Spector K, & Weiner KS (2014). The functional architecture of the ventral temporal cortex and its role in categorization. Nat. Rev. Neurosci, 15(8), 536–548. 10.1038/nrn3747 [DOI] [PMC free article] [PubMed] [Google Scholar]
  9. Hwang AD, Wang H-C, & Pomplun M (2011). Semantic guidance of eye movements in real-world scenes. Vision Res, 51(10), 1192–1205. 10.1016/j.visres.2011.03.010 [DOI] [PMC free article] [PubMed] [Google Scholar]
  10. Iordanescu L, Grabowecky M, Franconeri S, Theeuwes J, & Suzuki S (2010). Characteristic sounds make you look at target objects more quickly. Atten. Percept. Psychophys, 72(7), 1736–1741. 10.3758/APP.72.7.1736 [DOI] [PMC free article] [PubMed] [Google Scholar]
  11. Iordanescu L, Guzman-Martinez E, Grabowecky M, & Suzuki S (2008). Characteristic sounds facilitate visual search. Psychon. Bull. Rev, 15(3), 548–554. 10.3758/pbr.15.3.548 [DOI] [PMC free article] [PubMed] [Google Scholar]
  12. Kircher T, Sass K, Sachs O, & Krach S (2009). Priming words with pictures: Neural correlates of semantic associations in a cross-modal priming task using fMRI. Human Brain Mapping, 30(12), 4116–4128. 10.1002/hbm.20833 [DOI] [PMC free article] [PubMed] [Google Scholar]
  13. Kuznetsova A, Brockhoff PB, & Christensen RHB (2017). lmerTest Package: Tests in Linear Mixed Effects Models. J. Stat. Softw, 82, 1–26. 10.18637/jss.v082.i13 [DOI] [Google Scholar]
  14. Kvasova D, Coll L, Stewart T, & Soto-Faraco S (2024). Crossmodal semantic congruence guides spontaneous orienting in real-life scenes. Psychological Research, 88(7), 2138–2148. 10.1007/s00426-024-02018-8 [DOI] [PubMed] [Google Scholar]
  15. Kvasova D, Garcia-Vernet L, & Soto-Faraco S (2019). Characteristic Sounds Facilitate Object Search in Real-Life Scenes. Front. Psychol, 10, 2511. 10.3389/fpsyg.2019.02511 [DOI] [PMC free article] [PubMed] [Google Scholar]
  16. Lo S, & Andrews S (2015). To transform or not to transform: Using generalized linear mixed models to analyse reaction time data. Frontiers in Psychology, 6. 10.3389/fpsyg.2015.01171 [DOI] [PMC free article] [PubMed] [Google Scholar]
  17. Lunn J, Sjoblom A, Ward J, Soto-Faraco S, & Forster S (2019). Multisensory enhancement of attention depends on whether you are already paying attention. Cognition, 187, 38–49. 10.1016/j.cognition.2019.02.008 [DOI] [PubMed] [Google Scholar]
  18. Maezawa T, Kiyosawa M, & Kawahara JI (2025). Interference from semantically distracting sounds in action scene search. Attention, Perception & Psychophysics, 87(2), 498–510. 10.3758/s13414-025-03023-8 [DOI] [PMC free article] [PubMed] [Google Scholar]
  19. Malcolm GL, Rattinger M, & Shomstein S (2016). Intrusive effects of semantic information on visual selective attention. Atten. Percept. Psychophys, 78(7), 2066–2078. 10.3758/s13414-016-1156-x [DOI] [PMC free article] [PubMed] [Google Scholar]
  20. Mastroberardino S, Santangelo V, & Macaluso E (2015). Crossmodal semantic congruence can affect visuo-spatial processing and activity of the fronto-parietal attention networks. Front. Integr. Neurosci, 9, 45. 10.3389/fnint.2015.00045 [DOI] [PMC free article] [PubMed] [Google Scholar]
  21. Mikolov T, Grave E, Bojanowski P, Puhrsch C, & Joulin A (2017). Advances in Pre-Training Distributed Word Representations. In arXiv [cs.CL]. http://arxiv.org/abs/1712.09405 [Google Scholar]
  22. Moores E, Laiti L, & Chelazzi L (2003). Associative knowledge controls deployment of visual selective attention. Nat. Neurosci, 6(2), 182–189. 10.1038/nn996 [DOI] [PubMed] [Google Scholar]
  23. Nah JC, Malcolm GL, & Shomstein S (2021). Task-Irrelevant Semantic Properties of Objects Impinge on Sensory Representations within the Early Visual Cortex. Cereb Cortex Commun, 2(3), tgab049. 10.1093/texcom/tgab049 [DOI] [PMC free article] [PubMed] [Google Scholar]
  24. Neider MB, & Zelinsky GJ (2006). Scene context guides eye movements during visual search. Vision Res, 46(5), 614–621. 10.1016/j.visres.2005.08.025 [DOI] [PubMed] [Google Scholar]
  25. Peacock CE, Hayes TR, & Henderson JM (2019). Meaning guides attention during scene viewing, even when it is irrelevant. Atten. Percept. Psychophys, 81(1), 20–34. 10.3758/s13414-018-1607-7 [DOI] [PMC free article] [PubMed] [Google Scholar]
  26. Řehůřek R, & Sojka P (2010). Software Framework for Topic Modelling with Large Corpora. Proceedings of the LREC 2010 Workshop on New Challenges for NLP Frameworks, 45–50. [Google Scholar]
  27. Santangelo V, & Spence C (2007). Multisensory cues capture spatial attention regardless of perceptual load. J. Exp. Psychol. Hum. Percept. Perform, 33(6), 1311–1321. 10.1037/0096-1523.33.6.1311 [DOI] [PubMed] [Google Scholar]
  28. Shomstein S, Zhang X, & Dubbelde D (2022). Attention and platypuses. Wiley Interdiscip. Rev. Cogn. Sci, e1600. 10.1002/wcs.1600 [DOI] [PubMed] [Google Scholar]
  29. Vallat R (2018). Pingouin: Statistics in Python. J. Open Source Softw, 3(31), 1026. 10.21105/joss.01026 [DOI] [Google Scholar]
  30. Van der Burg E, Olivers CNL, Bronkhorst AW, & Theeuwes J (2008). Pip and pop: Nonspatial auditory signals improve spatial visual search. J. Exp. Psychol. Hum. Percept. Perform, 34(5), 1053–1065. 10.1037/0096-1523.34.5.1053 [DOI] [PubMed] [Google Scholar]
  31. Wahn B, & König P (2017). Is Attentional Resource Allocation Across Sensory Modalities Task-Dependent? Adv. Cogn. Psychol, 13(1), 83–96. 10.5709/acp-0209-2 [DOI] [PMC free article] [PubMed] [Google Scholar]
  32. Wegner-Clemens K, Malcolm GL, & Shomstein S (2022). How much is a cow like a meow? A novel database of human judgements of audiovisual semantic relatedness. Atten. Percept. Psychophys, 84(4), 1317–1327. 10.3758/s13414-022-02488-1 [DOI] [PubMed] [Google Scholar]
  33. Wegner-Clemens K, Malcolm GL, & Shomstein S (2024). Predicting attentional allocation in real-world environments: The need to investigate crossmodal semantic guidance. Wiley Interdiscip. Rev. Cogn. Sci, e1675. 10.1002/wcs.1675 [DOI] [PubMed] [Google Scholar]
  34. Wu C-C, Wang H-C, & Pomplun M (2014). The roles of scene gist and spatial dependency among objects in the semantic guidance of attention in real-world scenes. Vision Res, 105, 10–20. 10.1016/j.visres.2014.08.019 [DOI] [PubMed] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

Supplement
NIHMS2151847-supplement-Supplement.docx (2.7MB, docx)

RESOURCES