Crosstalk Between Auditory and Visual information in Motor Learning and Savings

ABSTRACT

Sensorimotor adaptation relies on the integration of multisensory feedback, yet the specific contributions of auditory signals to this process remain poorly understood compared to vision. We tested 46 participants on a reaching task using visual, auditory, or combined feedback about terminal errors. Participants alternated between feedback modalities during learning and relearning, revealing how adaptation transfers across sensory systems. Strikingly, although auditory feedback alone failed to support initial learning, prior visual learning enabled subsequent auditory adaptation. Further experiments showed that this cross‐modal transfer stems from a memory of the learned reaching direction, acting as an attractor for movement, independent of error‐based mechanisms, or explicit strategies. Our findings unveil a novel pathway for sensory integration in motor learning, with implications for designing cross‐modal training protocols.

Visual feedback enables motor adaptation, whereas auditory feedback alone fails to support initial learning. Crucially, prior visual learning allows for subsequent auditory adaptation. This cross‐modal transfer stems from a memory of the successful reaching direction, which acts as a movement attractor independent of sensory modality. These results suggest that savings are driven by reward‐based directional memories rather than error‐based recalibration.

Abbreviations

A → V

auditory‐to‐visual feedback group

V → A

visual‐to‐auditory feedback group

‐ V → A

visual‐to‐auditory group with reversed rotation

V + A

combined visual and auditory feedback group

1. Introduction

The brain relies on sensory feedback to correct inaccurate movements and improve the outcome of subsequent actions. The body is equipped with different biological sensors that serve this purpose. For instance, vision (Held and Freedman 1963) and proprioception (Wolpert et al. 1995; Sarlegna and Sainburg 2009) are known to yield critical and complementary information (van Beers et al. 2002). Surprisingly, little attention has been paid to audition (Opoku‐Baah et al. 2021) despite the fact this inflow is the main sensory feedback modality for speech movements. Humans modulate the loudness of their speech in function of the loudness of the environment and as a function of how they perceive their own voice (Lane and Tranel 1971). Similarly, artificially altering the auditory feedback results in a modification of the vocal output (Chen et al. 2007). If such modifications are maintained consistently for several exposures, then it leads to a long‐term change in speech production (Houde and Jordan 1998). Finally, a study also showed that auditory feedback can enhance movement timing (van Vugt and Tillmann 2015).

Both visual and auditory feedback modalities can be used to guide upper limb and speech movements. For instance, visual information about one's own speech movements can improve language training (Katz and Mehta 2015). Similarly, the ability to use auditory feedback to maintain optimal motor output extends beyond speech production. When sound is not consistent with an action (e.g., delayed auditory feedback), it may interfere with the smooth course of an action and even result in destructive interactions. Altering auditory feedback during the performance of sequential motor actions such as playing the piano interferes with the production of the melody (Pfordresher 2005, 2012; Furuya and Soechting 2010). Audio‐motor interactions are numerous while playing musical instruments (Zatorre et al. 2007; Cao et al. 2016) or, more generally, in other types of tasks (Sigrist et al. 2013). Proper use of auditory information can also result in constructive interaction. For instance, providing auditory feedback about motor performance is sufficient to induce reliable learning in a natural gait learning task (Zanotto et al. 2013) or in a trunk‐arm rowing task (Sigrist et al. 2014) and also in a classical experimental cursor motion tracking paradigm (Rosati et al. 2012). Auditory feedback can be used for sensorimotor adaptation of reaching movements even though such adaptation mainly relies on visual and proprioceptive information. Changing the location and/or loudness of the sound allows humans to derive the motor error and to adapt to a perturbation (Oscari et al. 2012; Schmitz and Bock 2014). This suggests that sensory substitution of visual information by auditory information is sufficient to induce motor adaptation. But are auditory and visual information treated equally by the brain or do they provide complementary information?

Our main interest was to study how learning with one modality (e.g., visual) could be transferred to relearning in another modality. Savings refers to faster relearning upon re‐exposure to a previously encountered perturbation, operationally defined as a larger reduction in error during the initial trials of relearning compared to initial learning. This well‐known phenomenon has been observed in a wide range of declarative (e.g., Ebbinghaus 2013) and motor learning tasks (Brashers‐Krug et al. 1996; Krakauer et al. 2005). In motor adaptation tasks, several mechanisms have been proposed to account for savings. Some authors suggested that savings could arise from a memory of errors (Herzfeld et al. 2014), from the reliance on previous motor memories (Smith et al. 2006), from the use of an explicit strategy (Haith et al. 2015; Morehead et al. 2015), and/or from a reward‐based memory of successful motor actions (Huang et al. 2011; Shmuelof et al. 2012; Orban de Xivry and Lefèvre 2015) as well. Although recent work has advanced our understanding of these mechanisms—for instance, Hadjiosif et al. (2023) demonstrated a double dissociation between savings and long‐term memory—the relative contributions of error‐, strategy‐, and reward‐based processes remain debated.

Whereas experimental manipulations exist to minimize the contribution of the explicit strategy (Taylor et al. 2014; Morehead et al. 2015) or the reliance on previous motor memories (Huang et al. 2011; Orban de Xivry and Lefèvre 2015), reducing the contribution of memory of errors mechanisms has proven more difficult. Here, we reasoned that if savings were observed across modalities, it could not be due to a memory of errors given their different modalities. Internal model theory predicts that adaptation should be modality specific, as the sensory prediction errors driving implicit recalibration depend on the feedback modality. In contrast, reward‐based memory theory predicts that savings could transfer across modalities, because the memory of successful motor actions is encoded in terms of the movement direction rather than the sensory feedback that signaled success. In a first experiment, we tested the transfer of motor adaptation between visual and auditory sensory modalities. In a second experiment, we tested whether the observed savings are due to an explicit strategy or to a memory of the successful motor actions learned during the first exposure (Huang et al. 2011; Orban de Xivry and Lefèvre 2015).

2. Materials and Methods

2.1. Participants

Forty‐six right‐handed adults were included in this study. Experiment 1 comprised 34 participants (17 females, 24.3 ± 3.9 years old), and Experiment 2 involved 12 different participants (8 females, 23 ± 2.2 years old). All participants were volunteers, healthy, without neuromuscular disease or auditory impairment, and with normal or corrected to normal vision. Furthermore, participants were musically trained. Musical training was defined as having received at least 3 years of formal musical instruction or regular practice on a musical instrument, assessed via self‐report questionnaire and used as an inclusion criterion. Participants reported an average of 6.2 ± 2.5 years of musical practice. The experimental protocol was carried out in accordance with the Declaration of Helsinki (1964), and the procedures were approved by the ‘Comité d'Ethique pour les Recherches’ (CER) of Université de Bourgogne. All participants were naïve as to the purpose of the experiments and were debriefed after the experimental session.

2.2. Apparatus and Stimuli

Participants were comfortably seated in front of a virtual environment equipment with the head on a chin rest in a quiet, dimly illuminated room. They looked into two mirrors that were mounted at 90° to each other, such that they viewed one LCD screen with the right eye and one LCD screen with the left eye. This stereo display was calibrated such that the physical location of the robotic arm was consistent with visual disparity information. Participants made 12‐cm movements while holding on to a robotic device with the right hand (Phantom 3.0, SensAble Technologies, United States). Movements were performed in the natural reaching space in an upward‐forward direction, involving shoulder and elbow movements, with the elbow pointing downward. The experiments took less than 75 min, including briefing and debriefing.

2.3. Experimental Procedure

Participants performed shooting movement toward one of five green targets positioned on a circle (radius = 12 cm) at five different eccentricities (−40°, −20°, 0°, 20°, and 40°). The 0° direction corresponded to the vertical upward direction (Figure 1A). The 3D positions of the robot handle were mapped in real time to a gray cursor (diameter = 3 mm). A trial started when the cursor was positioned inside a starting green sphere (diameter = 6 mm). Then, as soon as one of the five targets appeared (diameter = 6 mm), the participant performed a rapid shooting movement through the target. Continuous visual feedback of the cursor trajectory was not provided. The trial ended when the radial distance from the starting position and the cursor was larger than 12 cm. We instructed and trained participants to reach the target within around 250 ms. After each trial, the target turned red or blue if movement times were longer than 300 ms or shorter than 200 ms, respectively.

FIGURE 1.

Illustration of experimental procedures. (A) Scaled trajectories of reaching movements toward the five targets in Experiment 1 (top circles). The dotted line represents a successful shooting movement toward the 40° target after full adaptation to a +25° rotation in hand space. (B) Pitch of the feedback sound (y axis) in function of the angular error (x axis). Resolution is optimized around 0 error (largest slope). The two horizontal dashed lines represent the saturation pitches (240 and 640 Hz), and the horizontal solid line is positioned at the reference frequency of 440 Hz. The vertical cursor is positioned at the angular error of 0°. (C) In Experiment 1, each group (rows) received different feedback modalities but was exposed to the same perturbation schedule. An “eye” icon and a “speaker” icon correspond to the presence of visual (V), auditory (A), or both (V + A) feedback information. The numbers on top represent the number of trials for each period, separated by dashed vertical and by gray rectangles. (D) Experimental schedule for Experiment 2.

After 500 ms, feedback about shooting error was provided through two different modalities. In the visual modality, a yellow cursor (diameter = 6 mm) was flashed during 1 s at the position closest to the target. The error was therefore signed; the yellow cursor could appear on the left or on the right of the target. In the auditory modality, error information was provided through a sound delivered through headphones at a comfortable volume (JBL) defined the following way. A continuous sound was emitted for 1 s but with specific frequencies. During the first 500 ms, its pitch followed a sigmoid function defined by

$$f\left(\mathrm{\alpha }\right)=f_0\left(1-\frac{1}{1+\exp \left(-\mathit{\tau \alpha }\right)}\right)+f_1\frac{1}{1+\exp \left(-\mathit{\tau \alpha }\right)}.$$

Pitch saturated at frequency f ₀ or f ₁ for angular errors α larger than −30° or +30°, respectively (Figure 1B). The inflection point and largest slope of the sigmoid function were set at 0° error in order to maximize the resolution of the auditory feedback around target position. The parameters were set as follows: f ₀ = 240 Hz, f ₁ = 640 Hz, and τ = 0.15°⁻¹. During the 500 ms that followed, a single frequency tone of 440 Hz that corresponded to a “A4” in the main octave was played back. This tone provided a reference sound for a perfect shooting movement (error α = 0°). Participants interpreted the sign of the error by comparing the trial and reference pitches, these aptitudes being well developed in humans (Plack and Oxenham 2005). Trial frequencies lower (upper) than 440 Hz were interpreted as angular errors to the left (right) of the target, and no error resulted in no identified differences between pitches. The 500‐ms sound modulated by the angular error was emitted immediately after the trial in order to reinforce the link between the movement just performed and its auditory consequences. After the trial ended, the robot moved the hand back onto the starting circle. This sped up trials and prevented participants performing active return movements.

In Experiment 1, participants were randomly dispatched in one of three groups, depending on feedback modality. In the “VISUAL” group (V → A, n = 12), endpoint error was provided through visual modality and then through auditory modality. It was the opposite in the “AUDITORY” group (A → V, n = 11). A third group was provided with both modalities for all trials (“AUDITORY and VISUAL,” V + A, n = 11).

After briefing, participants were familiarized with a practice block of 20 trials to a single target at −15° that were not considered for analysis. Cursor trajectories were not displayed online, and feedback was provided both in the visual and auditory modalities. During the experimental session, participants performed 560 trials in the following order (Figure 1C). The baseline block consisted of 120 reaching movements (24 to each target) to randomly selected targets (Figure 1C, “baseline” sequence). During the first 80 trials, feedback was visual and auditory, whereas it was either visual, auditory or both, depending on the group, for the last 40 trials. Then, a visuomotor perturbation was introduced for the next 200 shooting movements (40 trials × 5 targets). The perturbation consisted of a +25° clockwise rotation of the cursor for all groups. However, groups relied on different feedback to adapt their shooting movements (Figure 1C, “learning” sequence). During the perturbed trials, participants had to move the hand 25° to the left of the target to perform a successful trial (Figure 1A, dotted lines). After this perturbation block, 140 trials were again performed in the same condition as baseline (Figure 1C, “washout” sequence), without visuomotor rotation. During the first 80 trials, only feedback modality of that group was provided, like in the previous sequence. Next, participants could use both feedback modalities to adjust their movement in a short block of 20 trials. Finally, during the last 40 trials of this “washout” sequence, participants switched to the other feedback modality (i.e., “auditory” for the “VISUAL” group and vice versa). The “VISUAL and AUDITORY” group had always received both feedback modalities. In the last sequence (Figure 1C, “relearning”), the same perturbation was reintroduced in the other modality for 100 trials.

In Experiment 2 (n = 12), we specifically tested for the contribution of an explicit strategy or of a reward‐based memory of the learned movement direction. The procedure was similar to the VISUAL group in Experiment 1 and differed only for target positions and number of presentations (Figure 1D). The baseline block consisted of 120 reaching movements to one of four targets (−90°, −40°, 10°, or 60°) based on Figure 1C (“baseline” sequence). In the “learning” sequence, only three targets were presented (−90°, −40°, and 10°), and hand cursors were rotated 25° counterclockwise (−25°). To avoid overlearning, block design ensured that the same number of trials per target was presented in this condition (40 trials per target × 3 targets = 120 trials). The “washout” sequence consisted in 136 shooting movements to the same four targets as in the baseline sequence without visuomotor rotation. Finally, relearning was tested in the “auditory” modality but with different targets (−40°, 10°, and 60°) and with the opposite perturbation (clockwise, 25°). The new set of targets was arranged so that the same hand trajectories (−65°, −25°, and 15°) as during the learning led to success. In other words, this experimental design allowed us to test savings. In hand space, the solution to the second perturbation of three targets ([−40°, 10°, 60°]−25° = [−65°, −15°, 35°]; “relearning”) was identical to the solution of the first perturbation ([−90°, −40°, 10°] + 25° = [−65°, −15°, 35°], “learning”).

2.4. Data Processing

Positions were recorded with a sampling rate of 500 Hz. Movement start was detected when movement velocity exceeded 3 cm/s for at least 100 ms. Angular error of each movement was defined as the angular deviation at the end of the movement from the straight direction toward the target. In Experiment 1, we quantified, for each participant, the amount of learning, washout, and relearning by looking at the difference between the mean of the angular errors during the first five trials (early) and the mean of the angular errors during the last 15 trials (late) in each sequence. We adopted the same procedure for Experiment 2 to account for the different number of targets among the three sequences. Namely, learning and relearning (using three targets) were assessed using the first 3 and last 9 trials, whereas washout (using four targets) was quantified by comparing the difference in angular errors between the first 4 trials and the last 12 trials in that sequence.

The existence of savings was assessed by quantifying the speed of learning. To do so, we fitted an exponential function of the form $a_1\mathit{exp}\left(-a_{2}t\right)+a_3$ to the angular error, where a ₁, a ₂, and a ₃ are constants quantifying the amplitude of adaptation, the adaptation rate, and the asymptote, respectively.

To compare learning and savings across groups, we adopted mixed‐design ANOVAs with group (V → A, A → V, V + A) as the between‐subjects factor and time (early vs. late adaptation) as the within‐subjects factor. One‐way ANOVAs were used to compare differences between groups at specific experimental phases. When ANOVAs yielded significant main effects or interactions, Tukey HSD post hoc tests were performed to identify specific pairwise differences. For within‐group comparisons of specific conditions, paired t tests were used, whereas independent t tests were applied for specific between‐group contrasts. Quantile‐quantile (Q‐Q) plots were used to assess the normality of the data. Effect sizes are reported as partial eta‐squared for ANOVAs and Cohen's d for t tests. Sample size was fixed at 10+ subjects per group before the start of the experiments, a number chosen to provide sufficient power for between‐subject effects. Data processing and statistical analyses were performed using Matlab (The Mathworks, Natick, MA).

3. Results

In this experiment, we test the effect of visual versus auditory feedback on the adaptation to a visuomotor rotation and on the transfer of learning between these two modalities. We reasoned that such transfer would highlight the role of nonerror‐based mechanisms in savings because error‐based saving mechanisms should depend on the modality of the error feedback. Participants from the V → A group received visual feedback during learning and auditory feedback during relearning. The second group (A → V) received auditory feedback during learning and visual feedback during relearning. Finally, we expected maximal adaptation for the participants who received both visual and auditory feedback during the entire experiment (V + A group). Except for the modality of the error feedback, the participants from these three groups were exposed to an identical schedule of visuomotor perturbations (Figure 1C).

3.1. Inability to Adapt Under Auditory Error Feedback

After baseline trials, each group was confronted with a 25° visuomotor rotation (Figure 2). Initially, the perturbation caused identical initial angular errors in the three groups (main effect of group on initial angular error: F _{2, 31} = 1.74, p = 0.19, ${\eta }_p^2=0.1$). However, adaptation to the perturbation was absent in the group learning under auditory feedback (A → V), whereas the two other groups (V → A and V + A) could adapt normally to the perturbation (learning period in Figure 2A). The absence of learning under auditory feedback about terminal errors resulted in a larger final error for the A → V group compared to the two other groups (Figure 2, second column), yielding a significant effect of group on the final error measure (F _{2, 31} = 56.25, p < 10⁻¹⁰, ${\eta }_p^2=0.78$). A Tukey post hoc test to compare the final error across groups revealed that the group that learned in the “auditory” modality (A → V) was different from the two other groups (A → V vs. V → A: p < 10⁻⁸ and A → V vs. V + A: p < 10⁻⁸). In other words, participants from the A → V group did not decrease their angular error as much as the two other groups (interaction between group and time, initial vs. late error: F _{1, 31} = 58.17, p < 10⁻⁷, ${\eta }_p^2=0.52$). Indeed, late angular errors were significantly smaller than initial errors in the V → A group (t test: t ₁₁ = 11.9, p < 10⁻⁶, d = 3.44) and V + A group (t ₁₀ = 9.31, p < 10⁻⁵, d = 2.81) but not in the A → V group (t ₁₀ = 1.05, p = 0.318) that received auditory feedback only during the learning period.

FIGURE 2.

Evolution of the angular error over trials for the three groups of Exp. 1 (first row: V → A; second row: A → V; third row: V + A). (A) Evolution of the angular error over the course of the experiment. Data are averaged across participants (solid line), and gray shaded area correspond to the standard error of the mean. Thin black line corresponds to cursor rotation. Green background: auditory feedback. Orange background: visual feedback. Blue background: auditory and visual feedback. (B) Average angular error early and late during the learning period. Early = first cycle (five trials); late = last six cycles (30 trials). (C) Average angular error early and late during relearning. (D) Average amount of learning during the learning (early) and relearning (late) periods. For (B)–(D), color code is similar to (A). On each plot, individual values are represented by dots for each measure. Rectangles represent the mean. Black error bars represent mean ± SE.

The lack of adaptation of the A → V group was confirmed by the limited after‐effect for this group in the ensuing washout period under the same feedback modality (mean ± SD: −2.5° ± 3.8°), whereas the two other groups demonstrated a much larger after‐effect (V → A: −13.57° ± 3.6° and V + A: −11.4° ± 5.9°). The after‐effect of the A → V group was significantly smaller than in the other two groups (main effect of group: F _{2, 31} = 19.03, p < 10⁻⁵, ${\eta }_p^2=0.55$; post hoc Tukey test: A → V vs. V → A: p < 10⁻⁵; A → V vs. A + V: p = 0.0002; and V → A vs. V + A: p = 0.49).

Participants from all groups then performed 20 trials with concurrent visual and auditory feedback without perturbation (Figure 1C), which were followed by 40 unperturbed shooting movements but with only the other feedback modality. In other words, the V → A and A → V groups switched to the other feedback modality (the V + A group still received V + A feedback). By the end of this washout phase, all groups performed reaching movements without significant errors (V → A: t ₁₁ = 1.13, p = 0.28; A → V: t ₁₀ = 0.76, p = 0.463; V + A: t ₁₀ = 0.57, p = 0.583).

3.2. Relearning Under Auditory Feedback Is Possible After Learning Under Visual Feedback

In the relearning period, participants from the V → A and A → V groups were exposed to the same perturbation a second time but under the other modality. That is, participants who experienced the first perturbation under auditory (visual) feedback received visual (auditory) feedback during the relearning period. Participants of the V + A group received both modalities during the relearning period as well.

Surprisingly, in this second learning period, adaptation to perturbation was observed under auditory feedback. Indeed, participants from the V → A group changed their reaching direction during the relearning period (change in angular error from start to end of relearning period: t ₁₁ = 3.4, p = 0.006, d = 0.98) even though they only received auditory feedback during that period. For this group, the difference between the learning under visual feedback and the relearning under auditory feedback did not reach significance (interaction between period of learning and time (early vs. late): F _{1, 11} = 4.62, p = 0.05, ${\eta }_p^2=0.3$). More importantly, this change in error was larger than the change observed during the first 100 trials of the learning period under auditory feedback in the A → V group (learning A → V group vs. relearning of the V → A group, interaction between group and time: F _{1, 21} = 7.88, p = 0.01, ${\eta }_p^2=0.27$).

The A → V group that received the auditory feedback during the learning period—and exhibited no learning—was well able to learn the visuomotor rotation under visual feedback (difference between initial and final errors in the relearning period: t ₁₀ = 4.2, p = 0.002, d = 1.27). This relearning was significantly larger than the change observed in the learning period of this group, under auditory feedback (interaction between period and time: F _{1, 10} = 23.7, p = 0.0006, ${\eta }_p^2=0.7$). This suggests that this group was able to adapt to a visuomotor rotation under normal visual feedback condition but that the auditory feedback as provided here is insufficient to drive motor adaptation. Overall, we could not find any difference in the amount of relearning across the three groups (ANOVA on angular error, interaction between group and time: F _{2, 31} = 0.35, p = 0.7, ${\eta }_p^2=0.02$).

Exponential fitting (mean R ² = 0.21) corroborated these findings. The adaptation rate (a ₂) was higher in the V → A group during auditory relearning (0.151 ± 0.062 SEM) compared to the A → V group during initial auditory learning (0.033 ± 0.015; t ₁₁ = 1.86, p = 0.089), where fits were often poor or nonconvergent. In contrast, visual adaptation rates remained consistent regardless of prior auditory exposure (V → A, initial a ₂ = 0.123 vs. A → V relearning a ₂ = 0.126), confirming that savings were specific to the V → A transition.

3.3. Relearning Under Auditory Feedback Is Not due to an Explicit Strategy but to a Memory of the Learned Directions

Two possible components of motor adaptation could account for the observed relearning under auditory feedback in the V → A group. Either participants adopted an explicit strategy and modified their reach angle consciously in the same direction as during the learning period, or they implicitly converged toward the learned reaching direction that was rewarded at the end of the learning period. To differentiate between these two hypotheses, we conducted an additional control experiment (Experiment 2, Figure 3) directly inspired by Experiment 3 of Huang et al. (2011). In our experimental design, rotation angles between the learning (−25°) and relearning periods (+25°) are reversed. Therefore, the use of an explicit strategy would lead to an increase in error early during the relearning period. In contrast, the targets are arranged in such a way that the hand directions that counteract the visuomotor rotation are identical during the two adaptation periods. Therefore, the hand direction learned during the first adaptation period could drive the learning in the second adaptation period through reward‐based learning even though the change in hand direction is opposite in these two periods.

FIGURE 3.

Evolution of the angular error over trials for Exp. 2. (A) Evolution of the angular error over the course of the experiment. Data are averaged across participants (solid line) and gray shaded area correspond to the standard error of the mean. Thin black line corresponds to cursor rotation. Green background: auditory feedback. Red background: visual feedback. (B) Average angular error early and late during the learning period. Early = first cycle (three trials); late = last six cycles (18 trials). (C) Average angular error early and late during relearning. For (B)–(D), color code is similar to (A). On each plot, individual values are represented by dots for each measure. Rectangles represent the mean. Black error bars represent mean ± SE.

Figure 3 presents results from Experiment 2. Participants from this group (−V → A) adapted first to a −25° visuomotor rotation under visual feedback (difference between early and late angular error, t ₁₁ = 13.74, p < 10⁻⁷, d = 3.97), which yielded a significant after‐effect (mean ± SD: 12.2 ± 6.2; t ₁₁ = 6.8, p < 10⁻⁴, d = 1.96). Most importantly, we observed a decrease in error during the relearning period (mean ± SD: 12.3 ± 9.1; t ₁₁ = 4.66, p = 0.0007, d = 1.35), consistent with the hypothesis that the reward‐based memory of hand direction was able to guide the learning during the second adaptation period and that the explicit strategy was not responsible for the observed learning under auditory feedback during the second adaptation period.

The use of explicit strategy in the control experiment could be seen in an increase in the angular error during the first cycles of the relearning periods. However, such a change in strategy was not apparent in our data as shown in Figure 4.

FIGURE 4.

Evolution of the angular error during the first three cycles of the learning and relearning periods for the four experimental groups. Orange, green, and blue correspond to visual (V), auditory (A), or both modalities (A + V) during learning (before the arrow) or relearning (after the arrow). Each cycle corresponds to one presentation of each target.

4. Discussion

In the present study, we demonstrated that substituting visual error information with auditory error information did not allow the participants to adapt to a 25° visuomotor rotation. However, if adaptation to the perturbation by means of auditory error information was preceded by a first exposure to the same perturbation combined with visual error information, then participants were able to adapt to the perturbation based on auditory error information. Our control experiment suggests that adaptation under auditory feedback in the relearning period is driven by a reward‐based memory of the rewarded hand direction and not by an explicit strategy. Together, these results suggest that transfer of adaptation under different sensory feedback modalities is possible through reward‐based motor memory that can act as an attractor for the reaching direction.

4.1. Absence of Learning Based on Auditory Error Information on the First Exposure

Humans are particularly skilled at detecting differences in pitch frequency (Bendor and Wang 2006). All participants played at least one musical instrument. However, on average, adaptation did not emerge when participants received auditory feedback during the first exposure. This finding is consistent with a previous study that showed that aftereffects were absent in auditory retention trials, whereas aftereffects remained high during visual retention trials (Kagerer and Contreras‐Vidal 2009). However, this also contrasts with previous studies (Oscari et al. 2012; Zanotto et al. 2013; Schmitz and Bock 2014). These differences might be due to the experimental setup and to the way feedback was provided. First, we provided endpoint auditory feedback, whereas previous studies used online auditory feedback. It is known that online and endpoint feedback influence motor adaptation in different ways (Taylor et al. 2013). Second, we used the content of the auditory stimulus rather than its localization to provide feedback (Schmitz and Bock 2014). This forces the participants to build an arbitrary mapping between the content of the auditory stimulus and the error they performed. This new relationship is very common when we cope with visuomotor rotations. For instance, moving a computer mouse on a horizontal table induces cursor displacements in the vertical plane (screen) with a gain usually different from 1. Third, there was a short delay between the end of the movement and the presentation of the auditory feedback, which might hamper the adaptation process (Brudner et al. 2016). This delay was the same in the VISUAL and AUDITORY modalities. Because the 500‐ms delay was identical across modalities and the trial design was self‐paced, timing alone cannot explain the initial auditory deficit, as participants had sufficient time to process feedback regardless of sensory network. Furthermore, our exponential fitting analysis confirms that the abstract mapping is not inherently “unlearnable” but becomes accessible to the motor system only once a spatial reference frame has been established through prior visual exposure.

The absence of adaptation when auditory feedback alone was provided suggests that implicit adaptation of the internal model (Mazzoni and Krakauer 2006; Morehead et al. 2017; Kim et al. 2018) might not take place under auditory feedback alone, as suggested by another study (Kagerer and Contreras‐Vidal 2009).

4.2. Reward‐Based Memory Acts as an Attractor

Interestingly, we found that, under auditory feedback, participants adapted to the perturbation during the second exposure but not during the first exposure. We believe that the learning on the basis of auditory feedback during the second exposure is driven by the spontaneous recovery of the motor memory (here a movement direction) elicited by the absence of reward (Pekny et al. 2011), which acts as an attractor for the hand movement (Shmuelof et al. 2012). Our findings suggest that the motor memory formed during initial visual learning does not simply fade but persists as a latent spatial attractor—a stored representation of the rewarded hand direction that biases subsequent motor commands toward the previously successful solution, even when the sensory feedback context changes entirely. Although internal models may facilitate the fine‐tuning of movements within a modality, the robust cross‐modal savings observed here suggest that the “memory of success” (reward based) is the primary driver of rapid readaptation when the sensory environment changes. It is unsure whether the content of the auditory feedback played any role in the observed phenomenon. Indeed, given that the A → V group demonstrated that participants were unable to use the auditory feedback to adjust their hand movement to the visuomotor rotation, the observed effect might be driven by the removal of the reward at the end of each movement. In our experiment, successful movements were rewarded by points. Removing this rewarding signal yielded to spontaneous recovery of the motor memory as observed in our study and in others (Pekny et al. 2011; Shmuelof et al. 2012). This is also consistent with the savings observed in Experiment 3 of Huang et al. (2011) and in Orban de Xivry and Lefèvre (2015). Indeed, our control experiment that adopted a design similar to Huang et al. shows that the sign of the error during the first exposure did not matter and that the learning during the second exposure was due to the recall of the reward‐based memory, which acted as an attractor for the hand movement. The learning in the second exposure through auditory feedback cannot be due to a memory of errors (Herzfeld et al. 2014), as the errors are radically different between the visual and auditory feedback modalities. Crucially, Experiment 2 demonstrates that the mere experience of errors is insufficient to trigger savings if those errors do not lead to a successful outcome. This further reinforces the reward‐based account: Savings in our task were not a result of ‘learning to learn’ through error repetition but were specifically tied to the reinforcement of the successful 30° compensated trajectory. It is also unlikely that it is due to an explicit strategy as the explicit strategy would have driven the hand in the opposite direction in the control experiment (Morehead et al. 2015), which was not observed in the present study.

4.3. Limitation

Several limitations should be acknowledged. First, the auditory feedback used a frequency‐based, nonspatial mapping. Although this was a deliberate choice to probe abstract auditory error coding, it remains possible that more ecologically valid auditory feedback—such as spatially lateralized sounds—could support robust initial adaptation and alter the pattern of cross‐modal savings observed here. Second, although all participants met a minimum criterion of musical training (≥ 3 years), no formal psychophysical pitch‐discrimination task was administered. Including such a measure would have provided direct confirmation that participants could reliably interpret the feedback signal. Third, the musically trained sample may limit generalizability to untrained populations. Fourth, the single‐session design precludes conclusions about the long‐term retention of cross‐modal savings. Future studies should examine whether the reward‐based motor memory identified here persists over days or weeks and whether it generalizes to other types of perturbations and feedback mappings.

4.4. Conclusion

We tested the independence of a reward‐based motor memory of movement direction on the sensory modality during which it was created and found that reward‐based memory can be formed and recalled independently of the sensory modality. This reward‐based memory allowed the participants to learn a task that they could not learn otherwise independently of any memory of errors or explicit strategy. We believe that this study presents a new way of testing the presence and strength of a reward‐based motor memory. In conclusion, the crosstalk between auditory and visual information in our task suggests that the brain builds a higher order representation of motor success. This suggests that motor memories are more abstract and flexible than previously thought, allowing for the seamless transfer of learned skills across different sensory contexts.

Author Contributions

Olivier White: conceptualization, data curation, formal analysis, funding acquisition, investigation, methodology, project administration, resources, software, supervision, validation, visualization, writing – original draft, writing – review and editing.

Funding

This work was supported by the Fonds Européen de Développement Régional (FEDER), the Conseil Général de Bourgogne, and the Institut National de la Santé et de la Recherche Médicale.

Ethics Statement

The experimental protocol was carried out in accordance with the Declaration of Helsinki (1964) and the procedures were approved by the “Comité d'Ethique pour les Recherches” (CER) of Université de Bourgogne.

Conflicts of Interest

The author declares no conflicts of interest.

White, O. 2026. “Crosstalk Between Auditory and Visual information in Motor Learning and Savings.” European Journal of Neuroscience 63, no. 7: e70502. 10.1111/ejn.70502.

Data Availability Statement

The data that support the findings of this study are available on request from the corresponding author.