Cross-modal facilitation of auditory discrimination in a frog

Abstract

Stimulation in one sensory modality can affect perception in a separate modality, resulting in diverse effects including illusions in humans. This can also result in cross-modal facilitation, a process where sensory performance in one modality is improved by stimulation in another modality. For instance, a simple sound can improve performance in a visual task in both humans and cats. However, the range of contexts and underlying mechanisms that evoke such facilitation effects remain poorly understood. Here, we demonstrated cross-modal stimulation in wild-caught túngara frogs, a species with well-studied acoustic preferences in females. We first identified that a combined visual and seismic cue (vocal sac movement and water ripple) was behaviourally relevant for females choosing between two courtship calls in a phonotaxis assay. We then found that this combined cross-modal stimulus rescued a species-typical acoustic preference in the presence of background noise that otherwise abolished the preference. These results highlight how cross-modal stimulation can prime attention in receivers to improve performance during decision-making. With this, we provide the foundation for future work uncovering the processes and conditions that promote cross-modal facilitation effects.

1. Introduction

Sensory perceptions in one modality are routinely impacted by stimulation in other modalities [1,2]. For instance, interactions between vision and hearing create many illusions in humans [3–9]. The sources of cross-modal interactions are myriad, including interactions in primary sensory processes as well as higher level cognitive processes [3]. While most research has focused on visual/auditory interactions, these effects likely occur across all sensory modalities. For instance, tactile stimulation can affect visual perception, and odours can affect tactile perception [10,11].

Stimulation in one modality can improve performance in a separate modality. For instance, visual input improves noisy speech comprehension [12,13]. In addition, hearing the word for an object can allow participants to visually detect an otherwise unseen object [14]. However, the cross-modal stimulus need not always be so relevant. Simple auditory stimulation can reveal otherwise unseen images when temporally and spatially aligned to the image presentation, and vice versa [15,16]. Even a non-spatial auditory ‘pip’ can improve performance on a visual search task [17,18] and a light flash can improve detection of low-intensity sounds [19]. Here, we refer to the process in which stimulation in one modality improves performance in a separate modality as ‘cross-modal facilitation’, a term that has been used to describe a variety of processes, often akin to this phenomenon [2,20–22]. Many aspects of when and how cross-modal facilitation occurs remain unexplored.

Investigations in non-human animals have revealed similar cross-modal impacts on perception and identified some of the neuronal processes responsible. Pioneering studies in owls and cats revealed extensive brain areas that respond to and integrate stimuli from multiple modalities [23–26]. Cats demonstrate a behavioural result similar to humans, in which a spatially and temporally aligned auditory stimulus enhances performance on a visual detection task [27,28]. However, little is known about the range of conditions and species where cross-modal facilitation occurs, particularly using naturalistic stimuli and non-domestic animals.

The túngara frog offers an excellent system to study cross-modal facilitation. In this species, groups of males call from shallow pools at night to attract female frogs, which also attracts predators and parasites [29–31]. Males can produce simple calls consisting of a downward sweeping whine, or complex calls which consist of a whine followed by one or more short chucks. Across decades of two-choice phonotaxis experiments, wild-caught females have shown consistent preferences for a speaker playing a complex call over a simple call [32]. To produce these calls, males inflate and deflate a large vocal sac, creating a temporally aligned visual cue as well as a water surface ripple ‘seismic’ cue [33]. These additional cues can be used by female frogs [34–37], generally enhancing preference. Integration of the acoustic and visual components can also occur nonlinearly and create emergent percepts [36,38,39]. However, how these stimuli may promote cross-modal facilitation remains largely unknown.

Here, we demonstrated that cross-modal facilitation could improve performance of a biologically relevant auditory choice task in the túngara frog. We first investigated what type of cross-modal stimulation was behaviourally relevant for female frogs. We then found that, when we used acoustic noise to abolish the preference for a complex call, cross-modal facilitation restored the acoustic preference.

2. Methods

(a) . Animals

We collected pairs of túngara frogs (Engystomops (=Physalaemus) pustulosus) from ephemeral pools in and around Gamboa, Panamá shortly after sunset between September and December 2021. Phonotaxis experiments were conducted with females in a laboratory at the Smithsonian Tropical Research Institute. Frogs were acclimated to darkness in a cooler for at least 30 min prior to testing, and toe clipped following testing to ensure that frogs were not recaptured and tested again. All procedures were approved by the University of Texas at Austin (IACUC: AUP-2019-00067), STRI (IACUC: 2018-0411-2021) and the Ministry of the Environment of Panamá (MiAmbiente: SE/A-40-19). We used 38 frogs in experiment 1, and 50 frogs in experiment 2. Following testing, all frogs were returned to the site of capture within 24 h.

(b) . Apparatus

Experiments were conducted in a wading pool inside a dimly lit acoustic chamber (figure 1a). For acoustic stimulation, we placed speakers in holes cut in the side of the pool, directly above the water line. For visual stimulation, we attached three-dimensional-printed model frogs (RoboFrogs [34]) in front of each speaker (figure 1b,c). The RoboFrogs housed a silicon vocal sac replica that we dynamically inflated simultaneously with the acoustic call for trials with visual stimulation. For seismic stimulation, we placed custom-built ripple generators on elevated platforms in front of each speaker. The generator rested just below the water surface and created a ripple simultaneously with the acoustic call for trials with seismic stimulation. Finally, a pair of speakers on the wall above and behind the call speakers were used for continuous playback of green noise stimuli on trials with noise. See electronic supplementary material for additional details on the materials and methods used.

Figure 1.

(a) Experimental set-up. (b) RoboFrog with an inflated silicone vocal sac. (c) The RoboFrog controller. See methods and electronic supplementary material for additional details.

(c) . Experimental stimuli

For experiment 1, the speakers alternated broadcasting the same complex call stimulus (whine chuck) on a 1.2 s loop. Each female was tested in three conditions: visual, seismic and visual + seismic. In each condition, the call from one speaker (randomly assigned each trial) was paired with the dynamically inflating vocal sac of a RoboFrog (visual), the generation of a ripple (seismic) or both (visual + seismic).

For experiment 2, speakers again alternated broadcasting call stimuli on a 1.2 s loop. One speaker (randomly assigned each trial) played a simple call (whine) and the other played a complex call (whine chuck). The whine portion of both stimuli was identical. Each female was tested in four conditions: acoustic, trimodal, acoustic + noise and trimodal + noise. In the acoustic condition, no other stimuli were presented besides the calls. In the trimodal condition, the calls from both speakers were paired with a simultaneous visual and seismic cue. These two conditions were repeated in the presence of continuous playback of green noise for the acoustic + noise and trimodal + noise conditions. The duration of both the visual and seismic stimuli was based on a simple call (whine) and was identical regardless of the acoustic stimulus with which it was paired (i.e. the visual and seismic stimuli gave no indication of what acoustic stimulus played from the speaker).

In both experiments, the order of conditions was randomly assigned to each female.

(d) . Phonotaxis testing

We conducted phonotaxis experiments using standard protocols in this system [32,35]. In brief, each female was exposed to that trial's stimuli from the starting platform for 2 min before the restraining cage was lifted. Experiments were monitored live and terminated when the female made a choice by remaining in the ‘choice zone’ near one speaker for at least 4 s or failed to choose after 10 min elapsed (foul out). Foul out trials were removed from analysis. She was then placed back on the starting platform and the next trial was started. The latency to respond was recorded (we did not detect any significant differences in latency across conditions; see electronic supplementary material).

(e) . Statistics

All analyses were conducted in R. For each condition, comparisons to chance were conducted using binomial tests. Because the same females were used across conditions, we compared conditions using generalized linear mixed effects models (GLMMs) with frog ID as a random effect and a binomial link function using the ‘lme4’ package [40].

3. Results

(a) . Experiment 1

We used a two-choice phonotaxis protocol in a large pool of water where both speakers broadcasted the same complex call, and we randomly paired one speaker with a dynamic visual, seismic or combined stimulus on each trial (see Methods). By presenting each cue in isolation and combined, we discovered that only the combined stimulus with both the visual and seismic cue evoked a preference reliably different from chance (binomial test: 25 out of 35; p = 0.0167; figure 2a).

Figure 2.

(a) A bar graph from experiment 1 demonstrating that the combined visual and seismic stimulus causes the largest effect. The y-axis indicates the percentage of females that chose the multi-sensory stimulus over the unisensory stimulus. For all conditions, an identical whine chuck (complex call) was played from both speakers. (b) A bar graph from experiment 2 depicting significant cross-modal facilitation. The y-axis indicates the percentage of females that chose a whine chuck (complex call) over a whine (simple call). For all conditions, the only difference between the choices was in the acoustic modality. For (a,b), error bars indicate 95% confidence intervals from binomial tests, with darker bars and asterisks above bars indicating conditions significantly different from chance (50% dashed line). Asterisks with horizontal lines indicate significant differences between conditions from GLMMs. Numbers at the bottom of each bar indicate the sample size.

(b) . Experiment 2

We next asked whether cross-modal stimulation could enhance auditory discrimination. We capitalized on the reliable and natural acoustic preference in túngara frogs for a complex call over a simple call [32], which we reproduced in an aquatic arena for the first time (binomial test: 38 out of 48; p < 0.0001; figure 2b). Next, we paired both speakers (one broadcasting a simple call, the other broadcasting a complex call) with the same combined dynamic visual and seismic cues, simultaneously with the acoustic stimuli, and found that preferences remained stable and high (binomial test: 40 out of 50; p < 0.0001; figure 2b). Note that the visual and seismic cues were identical at both speakers, providing no information to the females about what acoustic stimulus was playing from each speaker. Given the results of Experiment 1, we used only the combined stimulus to ensure that the stimulus was perceptible and behaviourally relevant for the female frogs. Then, we added green background noise (see electronic supplementary material) at a volume equal to the call stimuli at the starting platform, which was sufficient to abolish the acoustic preference for a complex call (binomial test: 26 out of 50; p < 0.8877; figure 2b). Finally, we found that adding the cross-modal stimuli to both speakers in the presence of noise was sufficient to rescue the preference (binomial test: 36 out of 48; p = 0.0007; GLMM: p < 0.02 for all pairwise comparisons with noise only condition; figure 2b).

4. Discussion

Here, we demonstrated cross-modal facilitation in a frog, where the presence of visual and seismic stimuli rescued an auditory preference in the presence of noise. This finding is reminiscent of cross-modal facilitation in other domains, particularly the finding that visual cues improve noisy speech comprehension [12,13]. However, it is important to note that in the current study, the identical visual and seismic stimuli were present at both speakers, and thus could not bias decision-making on their own. Rather, the mere presence of these additional stimuli caused female frogs to act on the acoustic differences between two stimuli, despite the presence of noise that previously abolished the acoustic preference.

The processes governing cross-modal facilitation remain poorly understood and are likely complex. The related concept of multi-sensory integration, which can enhance overall performance, has received extensive theorizing [1,41,42], but how stimuli can improve discrimination or detection in a separate modality has received considerably less attention. From human studies, it has been hypothesized that hearing a word primes subjects to expect particular shapes, and thus participants can more easily identify those objects in a visual detection task [14]. Cross-modal facilitation could also occur from one stimulus improving attention to another modality. For instance, an acoustic cue will improve visual detection in humans and cats, but only when spatially and temporally aligned to the visual stimulus [15,20,27,28], and an acoustic ‘pip’ can cause a temporally aligned visual stimulus to ‘pop’ out [17,18,21]. In particular, cross-modal input may improve selective attention for the especially relevant aspects of stimuli in another modality [43]. More generally, any cue that provides temporal or spatial information to a receiver can help unmask stimuli from noise. We hypothesize that these processes occurred in our experiments with túngara frogs, where cross-modal stimuli prime females to temporal and spatial aspects of the acoustic stimuli. Indeed, previous research on multi-sensory preferences in female túngara frogs has found that the temporal and spatial alignment of the visual and acoustic stimuli are important for whether females prefer or even recognize the visual stimulus [38,44,45].

Within the vertebrate brain, the optic tectum (OT; superior colliculus in mammals) is a region that has been identified as a key area for multi-sensory integration in multiple taxa as well as for goal-oriented movement [23,25,46–51]. Electrophysiological results in the superior colliculus of cats closely match behavioural responses during cross-modal facilitation when auditory cues improve visual detection [27,47,52], suggesting that the OT could also be important for the cross-modal facilitation we observed in a frog. Indeed, other species of frogs with ablations to the OT fail to respond to relevant visual motion [53,54]. These results highlight that cross-modal effects can appear without a mammalian cortex. Given that multi-sensory integration occurs in invertebrates, we believe that cross-modal facilitation effects are likely, indicating that other, radically different, neural architectures can produce such effects [55–57].

Our results have important implications for sexual signalling, mate choice and multi-sensory processing in frogs. Multi-sensory integration of acoustic and visual components has been shown across numerous frog species [58–64]. The current study indicates a novel importance for water surface ripples in mate choice. In addition, our data suggest that cross-modal facilitation can serve to maintain species-typical preferences for complex calls in noisy conditions, an important task for many species to solve [65]. Frog-eating bats also attend to all three components tested here [66–68], and future work will be essential to understanding how cross-modal effects might impact predation risk and calling behaviour.

Overall, we demonstrate that visual and seismic stimuli can cause cross-modal facilitation in a naturalistic auditory choice task. This is a special case of the general theory that multi-sensory signalling leads to enhanced performance in animal communication [69]. However, overstimulation in one modality can also reduce performance in another modality, leading to cognitive overload [70], a process well understood by drivers who turn the radio down when parking the car. Understanding where this line is between enhanced performance and cognitive overload, as well as how this line varies across different receivers, provide intriguing avenues for future inquiry.

Ethics

All procedures were approved by the University of Texas at Austin (IACUC: AUP-2019-00067), STRI (IACUC: 2018-0411-2021) and the Ministry of the Environment of Panamá (MiAmbiente: SE/A-40-19).

Data accessibility

Data used for analysis and example code are available in the electronic supplementary material [71].

Authors' contributions

L.S.J.: conceptualization, formal analysis, investigation, methodology, visualization, writing—original draft and writing—review and editing; A.L.B.: conceptualization, methodology and writing—review and editing; R.A.P.: conceptualization, funding acquisition, methodology, resources, supervision and writing—review and editing; P.C.: methodology, resources and writing—review and editing; K.L.H.: conceptualization, funding acquisition, methodology, resources, supervision and writing—review and editing; R.C.T.: conceptualization, funding acquisition, methodology, resources, supervision and writing—review and editing; M.J.R.: conceptualization, funding acquisition, methodology, supervision and writing—review and editing.

All authors gave final approval for publication and agreed to be held accountable for the work performed therein.

Conflict of interest declaration

We declare we have no competing interest.

Funding

The research was funded through a grant from the National Science Foundation (IOS-1914646). A.L.B. was supported by the Alexander von Humboldt Foundation (Feodor Lynen Research Fellowship).