The addition of a spatial auditory cue improves spatial updating in a virtual reality navigation task

Abstract

Auditory cues are integrated with vision and body-based self-motion cues for motion perception, balance, and gait, though limited research has evaluated their effectiveness for navigation. Here, we tested whether an auditory cue co-localized with a visual target could improve spatial updating in a virtual reality homing task. Participants navigated a triangular homing task with and without an easily localizable spatial audio signal co-located with the home location. The main outcome was unsigned angular error, defined as the absolute value of the difference between the participant’s turning response and the correct response towards the home location. Angular error was significantly reduced in the presence of spatial sound compared to a head-fixed identical auditory signal. Participants’ angular error was 22.79° in the presence of spatial audio and 30.09° in its absence. Those with the worst performance in the absence of spatial sound demonstrated the greatest improvement with the added sound cue. These results suggest that auditory cues may benefit navigation, particularly for those who demonstrated the highest level of spatial updating error in the absence of spatial sound.

Introduction

Spatial updating is the ability to keep track of the location of an initially perceived stationary target during one’s own movement (Loomis et al., 2002). Spatial updating is a complex undertaking in which navigators maintain an internal representation of a spatial location given dynamic changes in their self-position and orientation. Individuals can generally form and update spatial representations effectively using multiple sensory systems. Despite this general ability, all navigators occasionally exhibit a significant amount of spatial updating error, particularly in settings of reduced sensory cue reliability (Chen et al., 2017). As such, supplementing sensory cues may be a way of improving spatial updating performance.

In real-world contexts, numerous available cues can be used for orientation during navigation. Those generally argued to have the greatest impact on orienting are visual cues (both optic flow and landmarks) and vestibular and proprioceptive cues (also known as body-based self-motion cues when grouped), though there is growing evidence that auditory information can play a role as well. Auditory cues have been shown to be useful for yaw-plane direction-recognition (Shayman et al., 2020) as well as dynamic behavioral tasks such as maintaining gait (Hamacher et al., 2018; Kaipust et al., 2013; Karim et al., 2017) and balance (Marme-Karelse & Bles, 1977; Zarei et al., 2022). For instance, researchers have shown that auditory cues are critical for self-motion perception (the ability to distinguish the direction of passive body rotation), particularly at frequencies where vestibular information is less useful (Shayman et al., 2020). Other studies have examined the role of audition on orientation. Seiwerth et al. (2018) demonstrated that spatial audition can effectively stabilize inherent drift in gait. Karim et al. (2017) showed that front-facing spatial audition is better at stabilizing in-place walking than sounds at larger azimuthal angles. Finally, Stevens et al. (2017) demonstrated reductions in force-plate sway with spatial audition. These studies all suggest that auditory cues may play a role in tasks that rely on self-motion perception and also require more complex spatial updating, such as navigation.

Compared to vision and body-based self-motion cues, there has been less emphasis on the role of auditory cues for the spatial updating component of navigation. This may be partially due to the relatively low spatial resolution of the auditory sense compared to vision and self-motion (Grantham et al., 2003; Long & Giudice, 2010). Most of the existing literature is focused on distance perception to auditory landmarks, often using a spatial updating task. This literature has examined the role of auditory cues in helping form accurate static and dynamic spatial representations. Two decades ago, Loomis et al. (1998) demonstrated large spatial distance compression on the order of almost 100% when navigating to a target with a single auditory landmark. These data demonstrated a limited ability to use auditory landmarks for accurate distance perception during navigation, but audition clearly factors into spatial updating in some circumstances. Kolarik et al. (2016) highlight a general ability to use auditory cues, such as sound level, sound frequency (particularly as frequency shifts with movement), and reverberance for distance judgments (with systematic over- and under-estimations). Other work has shown that the presence of auditory information in addition to visual cues improves the accuracy of distance judgments and reduces auditory distance compression (Anderson & Zahorik, 2014). Taken together, these findings suggest that multisensory information can improve distance judgments and that concordant multisensory cues may help facilitate spatial encoding when auditory- or visual-only encoding is insufficient.

The distance perception work suggests that humans use auditory cues for targeted walking, but few studies have examined the role of auditory cues for spatial updating more broadly. The same task that showed distance compression for auditory landmarks was also analyzed for spatial updating performance (Loomis et al., 1998). Although distance error to auditory targets was greater than to visual targets, Loomis et al. (1998) argued that once the initial percept or “spatial image” was formed, it was updated just as accurately regardless of modality (see also Loomis et al. (2002)). A number of related studies motivated by the development of route navigation systems for blind individuals showed that spatialized sound had an advantage over spatial language in wayfinding tasks performed without vision (Loomis et al., 1998; Klatzky et al., 2006). Recently, Zanchi et al. (2022) examined mechanisms of auditory-visual integration for spatial updating using a Bayesian cue-combination paradigm. Single visual or auditory landmarks or the combination of these cues were used to assess accuracy and sensory weighting in a homing task. Precision was higher when both auditory and visual landmarks were available compared to auditory (but not visual) landmarks alone. The authors found high inter-individual differences in the weighting of auditory landmarks with only some individuals relying significantly on a spatially co-localized auditory stream.

More broadly, navigation involves more than spatial updating, and some studies have used large-scale virtual environments to test the influence of multisensory cues on wayfinding and landmark recognition. These studies generally support a positive influence of auditory cues on navigation performance. For example, Karimpur and Hamburger (2016) showed, in a tunnel-like maze environment, that decisions about turns during wayfinding were equally accurate with either visual or auditory landmarks presented near the intersections and that when combined, the semantic congruency of the visual and auditory targets did not matter for the wayfinding task, but did affect landmark recognition. Other studies using virtual mazes have found that audiovisual landmarks facilitate wayfinding and spatial learning over auditory or visual cues alone (Werkhoven et al., 2014; Gröhn et al., 2005). It is notable that these studies all necessarily used controller-based or passive locomotion through the environments because of the large-scale maze design, so it is unknown how audiovisual cues would have influenced performance if self-motion cues through actual locomotion had been provided. However, the findings are consistent with the advantages seen when providing auditory cues in real world settings (e.g., Loomis et al., 1998).

The current study uses a room-sized virtual environment to test whether a spatial auditory cue co-localized with a visual home target improves spatial encoding and updating performance for a point-to-origin homing task involving physical locomotion. The novelty of this work is that the task includes a condition that combines auditory, vision, and body-based self-motion cues together in comparison to vision and body-based self-motion cues alone. In addition, our paradigm manipulates the auditory cue presence at the home target location, not of a distant target (Loomis et al., 1998) or landmarks surrounding the targets (Zanchi et al., 2022). The auditory cue was a spatially localizable click-train, designed to provide salient interaural timing and level differences. Furthermore, we used a within-subjects design with and without the localized sound to allow us to determine whether a spatial auditory cue affects pointing to home with each participant acting as their own control.

Method

Participants

All procedures were approved by the University of Utah Institutional Review Board and adhere to the 1964 Declaration of Helsinki. An a priori power analysis performed in G*Power version 3.1 (Faul et al., 2007) based on effects found by Seiwerth et al. (2018)¹ suggested that 17 participants would be needed to measure an effect size of Cohen’s $d=0.88$ at an $a=0.05$ and a power of 0.9. Participants ($N=17$, eight female) had an average age of 25.9 years ($SD=3.6$, range = 21–36). Inclusion criteria were as follows: ages 18–39, normal or corrected-to-normal vision as defined by 20/20 visual acuity on a Snellen vision card and self-reported normal hearing without history of otologic surgery. All participants reported easily hearing the head-fixed and spatial auditory stimuli. All participants denied any falls, dizziness, or trouble walking in the last year. Participants all passed a cognitive screen (Mini-Mental Status Exam score of >24) and were able to give written, informed consent.

Materials and procedure

Participants completed the homing task in our carpeted immersive virtual reality laboratory (dimensions 8.53 × 11.58 m). The HTC Vive Pro head-mounted display (HMD) with wireless capabilities displayed the virtual environment. The HMD was powered by an Anker battery pack. The HMD was spatially calibrated to a fixed point in the room to ensure the co-location of the auditory stimulus during the spatial auditory condition. Auditory stimuli were presented via the HMD head-fixed over-the-ear speakers during the non-spatial auditory conditions and were presented from a JBL Charge2 Bluetooth speaker connected to an iPhone 6S. An experimenter controlled the sound from the mobile device during all trials to ensure that the spatial sound was presented during the correct intervals.

The homing task

Participants first walked to a starting location, specified visually as a black pole (see Fig. 1). No other visual landmarks were present, effectively limiting visual cue use to optic flow and horizon information. Upon arrival at the first pole location, participants walked towards a second pole (orange), and then to a third pole (orange). At the third pole, all visual and auditory cues were removed by shutting off the auditory cue and switching the display to black, and participants turned back toward their starting location (black pole) and took one step in that direction. Angular error was recorded by comparing the participant’s heading angle relative to the line formed by the starting location and the third pole.

Fig. 1.

A Screenshot of the virtual environment used from a raised perspective to illustrate all pole locations. Note that only one pole was visible at a time during the task. B Overhead diagram showing outbound path legs 1 and 2 and the participant’s response, which involved taking a step toward the remembered location of the path origin. The θ denotes the main outcome, angular error

Training and task familiarization

Participants experienced a brief training phase, consisting of turning the entire body toward various virtual objects in the virtual environment. Participants turned to face five objects before facing back to a starting object. A feedback line (solid blue) projected out of the front of the display to aim precisely. Participants then completed the same practice trials with the screen turning off during the turn, to practice responses without visual information. The training was performed with head-fixed click-train equivalent to the stimulus used during the non-spatial auditory condition described below. Participants were instructed to “use all available cues to orient towards the home location” to the best of their ability.

Trials and spatial layout

Eight different spatial triangular layouts were used. All layouts were repeated for each condition. Path leg length and turning angle were varied (between 1.5 and 2.5 m and 45 and 150°, respectively). Turns were varied between left and right. Triangular layouts were adopted from Barhorst-Cates et al. (2021) and are explained in greater detail there. Participants completed all eight triangles in each condition. Order of conditions was counterbalanced across participants.

The auditory stimulus

Participants completed two auditory conditions using: (1) a spatially localized click-train auditory cue in the environment or (2) head-fixed click-train as a control. The spatially localized auditory cue was a broadband jittered click train designed to be easily localizable and placed such that it was co-located with the black pole at the starting location. The stimulus was adopted from Shayman et al. (2020). It was calibrated to 60 dB SPL from a 1-m distance. The sound was only audible during the outbound navigation component of the triangle task. Upon arrival at the second orange pole (third target), the spatial sound was switched off such that participants could not continue to receive spatial cues during their orientation judgment. In the control condition, participants listened to the same click-train through headphones built into the head-mounted display. All participants confirmed that they could not hear any localizable room sound when listening to the click-train.

Results

Data (Fig. 2) were analyzed using multi-level modeling, which is appropriate for the nested structure of the data in this experiment where participants completed multiple repeated trials. Multi-level modeling also allows for the partitioning of variance within and between participants. Analyses were performed in R version 4.1.3 (R Core Team, 2022). Multilevel models were run using the lme4 (Bates et al., 2015) and lmerTest (Kuznetsova et al., 2017) packages. Intraclass correlations (ICC) were calculated using the performance package (Lüdecke, 2021). $R^2$ effect size measures were obtained using the r2mlm package, total $R^2$ is reported for each fixed effect Shaw et al. (2023). The equations below detail the model specifications in series of equation format, where $i$ represents individuals and $j$ represents trials.

$$\begin{gathered} Y(\mathit{angular}\mathit{error}{)}_{ij}={\beta }_{0i}+{\beta }_1(\mathit{condition}{)}_{ij} \\ +{\beta }_2(\mathit{pathlength}{)}_{ij} \\ +{\beta }_3{\mathit{(turningangle)}}_{ij}+{ϵ}_{ij} \end{gathered}$$ (1)

$${\beta }_{0i}={\gamma }_{00}+u_{0i}$$ (2)

Fig. 2.

Angular error in the presence and absence of spatial auditory cues. Box and whisker plots are inset with individual data to visualize within- and between-participant variability. The line in the middle of the boxes is plotted at the median, the boxes represent the 25th to 75th percentiles of the distributions, and the whiskers represent the minimum and maximum values. Means are denoted as plus symbols, and the asterisk denotes a significant difference between the means at $p<0.05$

The main outcome variable was unsigned angular error (in degrees) defined by the difference between the participant’s response direction and the vector formed by the final waypoint and the home location. The intraclass correlation coefficient (ICC) for angular error was 0.069, indicating that 6.9 % of the variance was between participants and 93.1% of the variance in angular error was within people ². A random intercept was added to the model to allow for individual participant intercepts to vary (Eq. 2). The path length $({\beta }_2)$ and turning angle $\left({\beta }_3\right)$ of each of the eight triangles used were entered into the model as control variables. On average, unsigned (absolute) angular error, as measured between the target heading and the perceived target heading was 30.09° in the absence of spatial auditory cues and 22.79° in the presence of spatial auditory cues. The significance of this difference was evaluated by looking at the ${\beta }_1$ term within the model (outlined in Eq. 2), which revealed a significant effect of condition $\left(\beta =-7.306,{\chi }^2\left(1\right)=4.706,p=.030,R_{\mathit{total}}^2=.016\right)$, while controlling for path length and turning angle³. This suggests that the condition with the spatial sound cue was associated with an accuracy improvement of approximately 7°, compared to the condition with no spatial sound cue, and that condition explained 1.6% of the total variance in angular error. Compared to the null model neither path length nor turning angle significantly predicted angular error ($\beta =1.220,{\chi }^2(1)=.341,p=.559;\beta =-.024,{\chi }^2(1)=.186,p=.666$, respectively). While not significant, path length and turning angle described .01% of the total variance in angular error.

To better visualize the relationship between baseline spatial encoding performance (no spatial auditory cue) and performance with a spatial auditory cue, angular error data were transformed into % change as a function of baseline (Fig. 3). A Pearson correlation was performed on these data. The R-squared value was 0.53 and with a two-tailed p value < 0.001, suggesting that individuals with the poorest baseline (highest angular error in the non-spatial condition) improved the most.

Fig. 3.

Change in unsigned angular error (without spatial sound - with spatial sound) as a function of unsigned angular error without spatial sound. The shaded region represents a 95% confidence interval

Discussion

This study demonstrated that a localized auditory cue combined with a visual cue can improve performance over a visual cue alone in a visually guided spatial updating task. Prior work showed that auditory spatial cues can be used effectively in the context of distance perception (Kolarik et al., 2016) non-visual real-world navigation (Loomis et al., 1998; Loomis et al., 1998; Klatzky et al., 2006), ambulation (Karim et al., 2017; Seiwerth et al., 2018), and wayfinding tasks in virtual environments (Karimpur & Hamburger, 2016; Werkhoven et al., 2014; Gröhn et al., 2005). Our goal was to test whether a spatial auditory cue co-localized with the home location would help to improve spatial updating during a homing task that included both self-motion cues associated with locomotion and sparse visual cues (a ground plane and horizon). Our hypothesis was supported by the finding that the spatial auditory cue at the home location improved angular error by approximately 7° over visual cues alone. This finding is novel and extends the auditory multi-sensory integration literature to an active homing task using an immersive virtual environment.

It is notable in Fig. 3 that although the average mean improvement with an auditory spatial cue was about 7°, there appears to be a larger benefit for those who had the poorest performance without the additional auditory spatial cue. This benefit of spatial auditory cues has long been hypothesized (Maheu et al., 2017; Weaver et al., 2017), but has not been demonstrated in the spatial audition literature. Such a finding may help to explain why only some people appear to receive multisensory benefit with auditory cues (Zanchi et al., 2022). It is unknown why we find the differences in baseline performance without an auditory spatial cue, but the variation seen in the current study is consistent with large individual differences seen in navigation studies across the literature (Hegarty & Waller, 2005; Newcombe et al., 2023). It also appears that five of the participants show worse performance with the co-localized auditory cue compared to the baseline without the spatial cue. However, a close look at these participants shows that they generally showed low error without the spatial cue, so the small increase in error may be due to natural variation on trials. The differences observed across participants in magnitude of improvement with the spatial auditory cue supports the notion that individual differences typically studied in the use of visual cues (for example, strategies to use proximal versus distal landmarks (Padilla et al., 2017)), also emerge when considering multisensory cues. The current study was not designed to study individual differences, given the relatively small sample size, but suggests the importance of considering the use of non-visual cues in future individual differences navigation studies. An understanding of individual differences in reliance on specific sensory cues could help to provide individualized assistance for people of varying spatial navigational abilities. Given that our effect size was smaller than expected based on our a priori power analysis, further studies focused on individual differences may need to take into account differences in effect sizes with different experimental tasks (Seiwerth et al., 2018; Shayman et al., 2020).

At a broader scale, the results of this study suggest that auditory supplementation could be a useful strategy for rehabilitation in those with poor spatial updating performance or deficits in sensory processing. The current data, albeit limited, represent a link between the spatial auditory beacon literature (Clemenson et al., 2021) and the navigation literature. Taking this link further by testing rehabilitation in future work would be useful. For example, auditory rehabilitation devices (hearing aids or cochlear implants) may provide value to those with impairments in spatial cognition (such as those with dementia). Previous work has suggested the utility of these devices for balance and gait control (Shayman et al., 2017; Weaver et al., 2017), but whether access to spatial sound through these devices may affect navigation has not yet been explored. The ability to hear spatial cues, however, is certainly not the only factor influencing their use, as attention to the cues or explicit strategies to use specific cues are both likely to play a role in their effectiveness (Newcombe et al., 2023).

Further, benefit from auditory input may also have a potential diagnostic utility for those with sub-clinical vestibular impairment. Maheu et al. (2017) demonstrated that sound appears to differentially aid those with vestibular hypofunction, suggesting an up-weighting of spatial auditory cues for balance control for those with vestibular impairment. Although the present study does not include individuals with known sensory impairment, it may be that those with increased spatial-sound benefit have sub-clinical deficits in self-motion and/or navigation ability. Further work could assess whether increased reliance on auditory cues for navigation correlates with clinical or sub-clinical vestibular impairment. Likewise, spatial auditory cues when combined with visual cues may be even more important for people with visual impairment. Given the current study tested only normally sighted participants, future work could explore the benefit of co-located auditory cues for facilitating navigation in conditions of degraded visual information.

The current study provides initial evidence for the utility of audition in spatial updating, but many questions remain. Given the presence of an auditory cue during the entire encoding phase but not while returning home, it is an open question of whether the auditory cue would further facilitate performance during the turning phase. Further, our conclusions are limited to the specific auditory stimulus used and future work should test whether similar effects would be found with sound sources typically found in everyday navigation. Finally, in the current study, the auditory cue was placed in the exact location of the home target, but future studies should test whether auditory landmarks that are not co-located with the home target serve the same utility. These ample directions for future work are exciting and will be informed by the current study’s clear benefit of auditory cueing in spatial updating.