Enhanced efficiency in visually guided online motor control for actions redirected towards the body midline

Abstract

Reaching objects in a dynamic environment requires fast online corrections that compensate for sudden object shifts or postural changes. Previous studies revealed the key role of visually monitoring the hand-to-target distance throughout action execution. In the current study, we investigate how sensorimotor asymmetries associated with space perception, brain lateralization and biomechanical constraints, affect the efficiency of online corrections. Participants performed reaching actions in virtual reality, where the virtual hand was progressively displaced from the real hand to trigger online corrections, for which it was possible to control the total amount of the redirection and the region of space in which the action unfolded. The efficiency of online corrections and the degree of awareness of the ensuing motor corrections were taken as assessment variables. Results revealed more efficient visuo-motor corrections for actions redirected towards, rather than away from the body midline. The effect is independent on the reaching hand and the hemispace of action, making explanations associated with laterality effects and biomechanical constraints improbable. The result cannot either be accounted for by the visual processing advantage in the straight-ahead region. An explanation may be found in the finer sensorimotor representations characterizing the frontal space proximal to body, where a preference for visual processing has been documented, and where high-value functional actions, like fine manipulative skills, typically take place.

This article is part of a discussion meeting issue ‘New approaches to 3D vision’.

1. Introduction

Reaching is a universal motor behaviour spanning a wide range of functions. In everyday life, we need to interact within a dynamic environment, where the object of interest and our own body could be moving smoothly or be suddenly displaced by unpredictable events. From grasping a fruit on a tree on a windy day to intercepting a flying object while running, successful reaching requires a continuous real-time monitoring of the spatial relationship between the reaching hand and the target. This perceptual monitoring is indeed key to the operation of online motor corrections [1,2] that continuously update and refine motor commands planned under sensorimotor noise [3,4], and are essential for compensating unpredictable perturbations [5] affecting the target [6] or the hand state [7].

Online motor corrections have been a central topic in motor control research. In particular, the systematic study of online corrections during reaching actions unravelled several fundamental aspects of how the brain controls movements. Empirical studies showed how online corrections are often operated outside of awareness. For example, if the visual feedback of the hand is deviated laterally (through the use of mirrors or virtual displays), participants automatically displace their hand in the opposite direction to comply with the assignment of drawing lines along the midline, while showing little if any awareness of such motor adjustments [7]. Analogously, if a force field is applied to the reaching hand, through a held manipulandum, in order to comply with the task and keep the hand on track, subjects typically engage in an anomalous pattern of muscles activity that they fail to notice and report [8]. The involuntary nature of online motor corrections has been further supported by introducing sudden perturbations of the hand's state or the target location: this triggers automatic corrections well ahead of when the perturbation is consciously processed [9] or intentional counteractions are instantiated [8]. The extremely short latencies of motor corrections have further highlighted the need to predict the effects of motor commands on the system state before sensory feedback from executed movement becomes available, thus providing evidence for the involvement of efference copies and internal forward models in the motor control loop [10,11].

Online motor corrections have been mostly explored in relation to their temporal dimensions (e.g. looking at their latencies). Nonetheless, the inhomogeneities that characterize spatial perception could plausibly affect the efficiency of online motor corrections, which may vary according to the region of space where the action is executed—for example, close versus away from the body midline (i.e. the region of the frontal midsagittal plane proximal to the trunk), or in the right versus the left hemispace, ipsilateral or contralateral with respect to the moving hand. Exploring the impact of these spatial factors on the efficiency of online corrections may provide novel insights into the interplay between space perception, self-centred perceptual processing and motor execution, and bring new insights into the sensorimotor processing underlying the fine tuning of online motor control.

There are different aspects of space perception that may potentially affect online corrections. The neurophysiological representation of space in primates is known to be strictly related to the body configuration [12–14] and to the egocentric perspective that the agent holds on the surrounding environment [15,16]. Within the reachable space, a rich cohort of multisensory body-centred (thus dynamic) representations is known to process visual stimuli according to the current body posture [17–19]. So, the same visual stimulus may trigger a larger activation when the hand is placed in its vicinity, reflecting the salience associated with the possibility for interaction [13,20]. In addition, there is ample evidence for a privileged visual processing of the straight-ahead direction in both primates [21] and humans [22,23]. Neurophysiological assessments have shown how gaze direction modulates the gain of peripheral neural populations in the visual system, which display increased responses as their receptive field moves closer to the body midline (straight-ahead direction) [21,22]. This enhanced processing of visual stimuli lying in the region proximal to the body midline has been confirmed with behavioural assessments [23], and has been linked to an optimization of the detection and avoidance of potential threats for the body while visually attending other regions in the surrounding space. Besides this, the proprioceptive encoding of the hand location was found to be misperceived closer to the body midline, with a bias that increases as the hand's true location moves away from it [24,25].

Brain lateralization also engenders spatial asymmetries in sensorimotor processing, substantiated in subtle effects that go well beyond handedness [26,27]. These asymmetries have been mostly attributed to the left and right hemispheres being specialized for temporal and spatial processing, respectively [27,28]. A specialization that is reflected, for example, in higher frequency detection thresholds for stimuli presented in the right hemispace [29,30], or in tasks based on spatial processing for which higher performances are found for stimuli presented in the left hemispace [31]. Related asymmetries, independent of hand dominance, have been reported for sensorimotor tasks [32,33]. For example, for both right and left handers, a right-hand advantage characterizes the control of movement's direction, and a left-hand advantage the control of movement's amplitude [34,35]. Furthermore, a right hemisphere specialization for the visual processing of motion has been reported [36], with associated enhanced performances in interceptive tasks performed with the left hand and for trajectory stimuli presented in the left hemispace [37].

In order to assess the potential impact of perceptual sensorimotor processing on the efficiency of online motor corrections, it is necessary to control for other potential modulating factors. In particular, handedness and biomechanical constraints can affect the way online motor corrections are planned and executed. The potential role of handedness is clear: the dominant hand outperforms in fine motor skills and dexterity [38,39]. Still, the ability to predict the sensory consequences of self-generated actions appears to be independent on handedness [40]. So, given the critical role of predictions in guaranteeing accurate goal-directed movements [11,41], the impact of handedness on the efficiency of online corrections is not obvious to anticipate. Complementary is the role of biomechanical constraints. Different experimental paradigms revealed that humans hold an implicit knowledge of how the arm inertia varies with movement direction, and that this knowledge is used in the planning of reaching actions [42–44]. The inertial anisotropies of the arm's musculoskeletal structure have also a direct impact on action execution: this is reflected, for example, by differences in the kinematics observed in abductive versus adductive reaching actions [45,46]. Similarly, modulations observed in reaching actions executed in the same versus the opposite hemispace of the moving hand, initially linked to latencies associated with interhemispheric communication [47,48], have been attributed to biomechanical constraints [49,50].

The present experiment has been designed to explore the interplay between inhomogeneities in sensorimotor processing and the efficiency of online corrections, with the aim of gaining novel insights into the mechanisms that fine tune the control of movement. To this aim, we implemented an immersive virtual reality (VR) set-up in which participants had to perform reaching actions towards a set of virtual targets. Online corrections were triggered by applying retargeting, a technique developed in VR applications. It consists in smoothly displacing the virtual hand along the reaching path, so to induce automatic corrections that redirect the real hand to a predefined location (the real target) when the virtual hand reaches the aimed virtual target [51,52]. In the current experiment, retargeting was adopted as a way to induce automatic online corrections in a controlled fashion, and we refer to it as redirection. By varying the reciprocal location of real and virtual targets, and their arrangement with respect to the participants' sagittal plane (figure 1a), it was possible to assess how the efficiency of online corrections varies with their amplitude and direction (towards or away from the body midline), and with the region of space in which the action unfolds (e.g. right versus left hemispaces). Participants performed repeated reaching actions both with their right and left hands. This, together with the geometry of the implemented configurations, allowed to control for possible confounds associated with handedness and biomechanical constraints.

Figure 1.

Experimental design and set-up. (a) Participants wearing a head-mounted display (HMD) and holding a virtual controller in each hand were asked to perform a sequence of redirected reaches. The design included three locations for the real targets (red, green and blue circles), each associated with seven virtual symmetrically targets arranged. (b) Left: during a redirected reach, the participant initially plans a movement pointing directly to the virtual target (grey circle). Right: to bring the virtual hand (red) to the virtual target the participant redirects his physical hand towards the location of the real target (blue circle). (c) Example of the virtual (grey) and real (blue) hand trajectories for a redirected reaching action.

In our experiment we used a realistic virtual model of the hands (figure 1b) that can be easily embodied and integrated in the self-body representation [53,54]. This is a major difference with respect to most previous studies on online corrections, in which the visual feedback from the moving hand is rendered as a small circle. Because embodiment is known to affect the perception and control of reaching actions under multisensory (visuo-proprioceptive) conflict [55,56], we could further test for possible confounds associated with the illusory embodiment of the virtual hands.

The impact of action redirection in the different conditions was evaluated both implicitly by quantifying the efficiency of online motor corrections from the action kinematics, and explicitly by asking participants to report if they noticed something weird when performing the action. While we expected that the efficiency of online corrections would decrease for larger redirection offsets, with a corresponding increase in the explicit awareness of the applied manipulation, we did not have any a priori hypotheses about the specific impact of the factors under inspection and their interactions. We therefore conducted an exploratory analysis of the collected data. In summary, results clearly show a motor advantage for actions redirected towards the body medial plane, irrespectively of the hand performing the reach and of the hemispace of action. This pattern of results suggests that the effects of handedness, biomechanical constraints and perceptual asymmetries associated with brain lateralization, play a minor role in determining the advantage of redirecting the hand towards the body midline. Results further show that the explicit awareness of operating online motor corrections is strictly related to their motor efficiency, so that a more efficient redirection has a larger probability of going unnoticed.

2. Results

Participants (n = 12) performed a set of reaching actions (420 trials) in an immersive VR environment, where both hands were represented by virtual hands holding the VR controllers (figure 1a). All reaching actions, performed in a pseudo-random alternation with either the right or left hand, started with both virtual hands spatially aligned with their real counterparts. Along the reaching path towards the virtual target, the virtual representation of the reaching hand was displaced so as to induce automatic corrections that redirected the real hand to the predefined location of the real target, which was always unknown to participants (see Methods).

The experiment was designed to collect an integrated set of data including the hand trajectory and the subjective experience of the redirected actions, as a function of (i) the virtual-to-real target distance, hereafter redirection offset; (ii) the reaching hand (right versus left hand); and (iii) the relative position of the real and virtual targets with respect to the body. The combination of the last two factors allowed us to test how the efficiency of online motor corrections varies with the movement direction, the hemispace of action and to assess the effect of executing the action in the contralateral hemispace. In addition, after each redirection, we assessed the proprioceptive recalibration of the perceived hand location (proprioceptive drift) induced by the redirection as an implicit measure of the virtual hand embodiment. For this, a blind reaching task (reaching in the absence of visual feedback from the moving hand) was adopted [57].

The experimental design included three real target locations to which the physical hand was redirected. Seven virtual targets were arranged symmetrically around each physical target, including a zero offset condition and three locations placed at different distances on each side of the real target (figure 1a). Each trial consisted of three steps: the redirected reach from which the kinematics data were recorded; a two-alternatives-forced-choice (2AFC) task, in which participants had to report whether or not they noticed the visuo-motor manipulation; a blind reach task, in which participants had to reach a second target in absence of visual feedback from the hand (see Methods for more details).

(a) . Kinematics of redirected reaches

To characterize the motor efficiency of a redirected action, we introduced the redirection cost C_R, a scalar parameter extracted from the reaching action kinematics, quantifying the deviation of the actual hand trajectory form the optimal straight path connecting the starting point (S) to the real target (T_R). Given the specific implementation of the redirection (see Methods), for a given real-to-target distance, the cost is higher if the online corrections start later in the reach, smaller if participants start correcting early on. In fact, early corrections result in reaching trajectories of the real hand closer to the optimal path (figure 4 and Methods).

Figure 4.

Geometry of action redirection. The figure depicts the implementation adopted for redirecting reaching actions. The real hand trajectory (green line) starts pointing to the virtual target, T_V. The location of the virtual hand H_V is shifted by the vector $w\hspace{0.17em}{\mathit{r}}_V$, with $w\in [0,1]$ and proportional to d_HR, i.e. the distance from S to the projection of the real hand location along $\mathit{S}{\mathit{T}}_V$. By definition ${\mathit{r}}_V={\mathit{T}}_R{\mathit{T}}_V$, so there exists a unique real hand configuration corresponding to the virtual hand being on the virtual target location T_V, which is the one in which the real hand is on T_R.

Reaching trajectories were visually inspected as a first step. The mean and standard deviation of the hand trajectory from a representative participant is shown in figure 2a, for a particular set of four symmetric conditions, all having the same absolute redirection offsets, θ_T. When actions are redirected towards the body midline online corrections appear more efficient, with hand trajectories closer to the straight path from S to T_R. The effect is independent on the reaching hand, and on the action being executed in the ipsilateral or contralateral hemispace. These trends were consistent across all subjects (electronic supplementary material, figure S1).

Figure 2.

Kinematics, efficiency and subjective experience of redirected reaching. (a) Examples of reaching trajectories from participant P8. (b) The redirection cost (C_R) is shown as a function of the virtual-to-real target offset for the three real target locations; negative/positive redirection offsets correspond to the virtual target being placed to the left/right of the real target. Points and bars show the mean and standard error of C_R across participants (with a lateral shift across target conditions added for better visibility). The curves show the fits of the nonlinear model (not including random effects). (c) C_R mean and standard error across participants are shown as a function of the absolute redirection offset, separately for action redirected towards versus away from the midline. (d–e) The same as in panel (b,c) but for the subjective experience of redirected reaches, quantified in terms of the probability of normal responses across repeated trials.

The redirection cost data, aggregated across all participants and hands, are shown in figure 2b. The mean redirection cost and its standard error is shown as a function of redirection offset (θ_T) for each real target location (left, centre, right). As the reaching hand is redirected further away from the planned target (i.e. for larger absolute values of θ_T), the redirection cost increases. Crucially, this dependence is modulated by the location of the real target: for actions redirected to the central target the trend of cost (C_R) versus offset (θ_T) appears to be symmetric and centred at zero offset (${\theta }_T=0$), while it appears skewed in opposite directions for the two lateral real targets. These trends hold independently of the reaching hand (electronic supplementary material, figure S2A) and are consistent across all participants (electronic supplementary material, figure S3). The data could be fit at a good approximation level (R² = 0.7) with the nonlinear mixed model in equation (4.2) (see Methods). The model's fit supports a significant dependence of the redirection cost (C_R) from the redirection offset (θ_T) modelled as a second-order polynomial (p < 10⁻¹⁵ for both polynomial coefficients), and from the real target location (T_R) treated as a fixed effect (p < 10⁻¹⁵). The fit further supports a significant interaction between real target location (T_R) and offset (θ_T) (p < 10⁻¹⁵ for both polynomial coefficients). Note that the significant interaction of T_R with the first-order polynomial coefficient corresponds to the different skewness of the curves associated with the three real target locations. The model output is given in the electronic supplementary material, table S1.

The significant interaction effect between the real target location and the offset can be attributed to an effect of movement direction, in line with the qualitative observations from the trajectory kinematics (figure 2a; electronic supplementary material, figure S1). To better appreciate this, the different combinations of real and virtual target configurations were tagged according to the direction of the implied online corrections (hereafter redirection direction D_R) as away versus towards the body midline. Figure 2c shows the cost C_R as a function of absolute offset $|{\theta }_T|$ and redirection direction D_R, for the subset of redirected actions (${\theta }_T\hspace{0.17em}\ne \hspace{0.17em}0$). It appears evident that, for a given $|{\theta }_T|$, redirecting actions towards the midline is more efficient, independently of the real target location, and of the reaching hand. Data could be fit at a good approximation level (R² = 0.7) with the linear mixed model in equation (4.3), which revealed: (i) a significant effect of $|{\theta }_T|$ (p < 10⁻¹⁵), as C_R increases with $|{\theta }_T|$; (ii) a significant effect of the redirection direction D_R (p = 10⁻⁶), as C_R is always lower for actions redirected towards the midline; and (iii) a significant interaction between $|{\theta }_T|$ and D_R, as the difference in C_R across conditions increases with $|{\theta }_T|$. The model output is summarized in the electronic supplementary material, table S2.

(b) . Subjective perception of redirected reaches

After each reaching action, participants had to keep their hand on the reached target, and state whether the performed action felt normal or weird, selecting one of two options using the controller buttons (see video in the electronic supplementary material). The probability of normal responses aggregated across participants is shown in figure 2d as a function of redirection offset θ_T, for the three locations of the real target (T_R). The probability to experience redirected actions as normal, so to not notice the virtual hand displacement along the reaching path, decreases with increasing values of $|{\theta }_T|$. However, as for the redirection cost C_R, this dependence is modulated by the real target location T_R, with a trend for curves associated with lateral real targets being skewed and shifted in opposite directions. This trend is consistent across reaching hands (electronic supplementary material, figure S2B), and across all participants, but one (electronic supplementary material, figure S4). That participant was excluded from this analysis as she always gave weird responses, also for non-redirected actions, indicating that she did not understand this part of the task. As for C_R, the data were fit with the nonlinear mixed model in equation (4.2) (R² = 0.72). The analysis revealed a significant dependence from offset θ_T (p < 10⁻¹⁵ for both polynomial coefficients), and a significant interaction between offset θ_T and real target T_R (p < 10⁻¹⁵) emerging from the different skewness directions associated with left and right real target. The model output is detailed in the electronic supplementary material, table S3.

As noticed above, the significant interaction between the real target location and the redirection offset reflects an effect of movement direction with respect to the body midline. For a given magnitude of the offset $|{\theta }_T|$, redirected actions were experienced as more natural (normal) when online corrections were made in the direction of the midline rather than away from it. This is explicitly shown in figure 2e where the probability of normal responses is shown as a function of absolute offset $|{\theta }_T|$ and of the redirection direction D_R. Data were fit with the linear mixed model described in equation (4.3). Results support a significant effect of $|{\theta }_T|$ (p < 10⁻¹⁵), as p(normal) systematically decreases with $|{\theta }_T|$, and a significant effect of redirection direction D_R, as p(normal) is always higher for action redirected toward rather than away from the body midline (p < 0.001). No significant interaction was detected as the slope of p(normal) versus $|{\theta }_T|$ was similar for both redirection directions. The model output is summarized in the electronic supplementary material, table S4.

These results are consistent with those found in the reaching kinematics analysis and provide evidence for lower redirection costs being associated with a more natural subjective experience of the actions, in which the visual manipulation of the hand location and the ensuing motor corrections are processed and executed outside of awareness.

(c) . Proprioceptive recalibration

A blind reach task, in which participants had to reach a visualized target without visual feedback from the moving hand [57], was adopted as an implicit measure of embodiment of the virtual hands [58], with the aim of controlling for the possible impact of different degrees of embodiment on the efficiency of online corrections. Results are fully presented in the electronic supplementary material as they support a control. In summary, a progressive increase of the proprioceptive recalibration with the redirection offset was found, which corresponds to the increased displacement between the real and the virtual hands at the end of the redirected reaching. However, no dependence on the direction of online motor corrections was found.

3. Discussion

The present study investigated the efficiency of online motor corrections in relation to the spatial inhomogeneities that characterize sensorimotor perceptual processing. Results show a strong effect of action direction, whereby the efficiency of online corrections is systematically higher for actions directed towards the body midline rather than away from it. The effect is robust and consistent across all participants. Crucially, it was also found to be independent on the reaching hand, on the hemispace of action, and on their combinations, which include actions executed in the hemispace ipsilateral or contralateral to the moving hand. Results further show that the subjective experience of the visuo-motor manipulation mirrors the motor efficiency of the online corrections, with higher motor efficiencies associated with larger probabilities of not noticing the experimental manipulation.

The higher efficiency of online corrections for actions redirected towards the body midline could be associated with different action costs. Previous studies have brought to light how the performance of reaching actions over extended trajectories is shaped by biomechanical factors, such as anisotropies in arm inertial properties and muscular structure that determine the effort required for action execution, and that make, for example, abductive actions requiring a larger effort with respect to adductive actions [42,44]. Nevertheless, the advantage of action redirected towards the midline was found to be independent of the reaching hand and of the hemispace of action, therefore independently on whether the performed action was adductive or abductive. In fact, depending on whether the action is operated in the hemispace ipsilateral or contralateral to the reaching hand, a movement towards (or away from) the midline could correspond to adductive as well as to abductive arm movements, which are associated with different motor costs. This evidence implies that the effect of movement direction on the retargeting cost cannot be attributed to differences in the biomechanical cost of the action.

The independence on the hemispace of action and on the reaching hand further indicates that perceptual and motor skill asymmetries associated with brain lateralization [26,59] do not play a relevant role in the facilitation of online corrections for actions redirected towards the midline. While participants were free to move their head, neutralizing possible effects of the perceptual asymmetries previously reported for conditions controlling for gaze fixation [29,31,37], no obvious confound could mask the possibility of a right-hand advantage in online motor adjustments associated with the left hemisphere specialization for temporal processing (independently on hand dominance). Our results therefore provide evidence for a marginal role of brain lateralization in determining the efficiency of online motor control.

In our experimental design, actions redirected towards the midline start in the periphery to end up closer to the body midline, while the opposite is true for actions redirected away from the midline. On the other end, the efficiency of online corrections depends on how early corrections start to be operated along the reaching path. Therefore, the privileged processing of visual information presented in the region of the body midline [22,23] should in principle facilitate early corrections—thus increasing the redirection efficiency—in our reaching actions redirected away from the midline. In these cases, in fact, the initial part of the virtual hand trajectory—the one providing visual input for the visuo-motor control—lies closer to the frontal midsagittal plane, where its early deviations from the real hand trajectory can be detected with higher accuracy. The fact that our results show the opposite trend suggests that the advantage for action redirected towards the midline found in the current study is not directly linked to visual perception.

With a similar reasoning we could expect that the systematic bias observed in the proprioceptively sensed location of static and unseen hands towards the body midline [25,58,60,61] may play a role in the reported effect. The effect of embodiment on the motor control of a virtual arm under visuo-proprioceptive conflict has been recently documented [55,62] and could be explained as an implicit motor component that acts to minimize multisensory prediction errors [56]. According to the model, this motor component is modulated by the perceived mismatch between the visually and proprioceptively sensed locations of the arm and acts akin to a small force in the direction of the visually perceived location. In the current study, for reaching actions redirected away from the midline the virtual hand is displaced closer to it, while the opposite is true for actions redirected towards the midline. The systematic proprioceptive bias should then correspond to a smaller visuo-proprioceptive conflict for actions redirected away, as the proprioceptively sensed location is biased in the direction of the virtual hand. The proprioceptive recalibration of the hand location at the end of the redirected reaching showed however no dependence on the direction of the redirection corrections. If present, such perceptual asymmetry would correspond to a weaker implicit action component in the direction of the virtual hand (opposite to the redirection direction) for action redirected away, and this would increase the redirection efficiency. Again, our results show the opposite trend, suggesting how the potential impact the multisensory processing associated with self-perception and embodiment has a negligible effect on the motor advantage of online corrections in actions directed towards the midline.

Keeping in mind the considerations above, a possible explanation for the higher efficiency of online corrections in hand movements directed towards the midline may be related to an intrinsic advantage of actions targeting the space proximal to the body centre. High-value functional actions, like self-defence, self-feeding and the fine manipulation of objects, are mostly operated in the vicinity of and/or towards the body midline. These ethologically relevant actions, observed in all primates, are known to be mapped onto dedicated cortical areas [63–65] and networks [17,66], which supports the possibility for a hardwired facilitation of the sensorimotor control of forelimbs movements directed towards the midline. This interpretation finds a resonance in early investigations attributing the faster and more accurate motor responses observed in adductive actions performed in the ipsilateral—rather than controlateral—space (thus towards the body midline) to the evolutionary advantage in performing fine manipulative control close to the body [28,45]. Further studies will be needed to corroborate this possibility.

The results presented may have important implications for the design of more effective redirection techniques for enhancing interactions in immersive VR applications. Several applications have adopted the controlled redirection of actions either as a way to make passive haptics more flexible [51,52,67], or as a way to enhance the possibility for interaction when the user is in a restricted place or has limited mobility [68–70]. All these applications rely on the visual dominance that characterize motor control under visual guidance. The specific design of these applications could thus benefit from the current results, particularly from the evidence that implicit redirections of arm movements are more likely to be unnoticed by the user if directed towards their body midline.

While in the current study the analysis of reaching kinematics was focused on the retargeting cost, here introduced as a scalar parameter quantifying the efficiency of action redirection, the experimental dataset is extremely rich and future works will be needed to go through it in more depth. Inspecting classical kinematics indexes (e.g. peak and time-to-peak velocities and accelerations) as in previous studies on reaching control may in fact add complementary insights. However, a direct comparison with previous works would not be straightforward, given that the continuous visuo-motor manipulation adopted in our study is profoundly different from those adopted in classical studies on online motor corrections using discrete perturbations [71,72]. A dedicated study will be therefore needed to advance in this direction. Another aspect that deserves attention and that will be the subject of a follow-up study is the relationship between implicit and explicit strategies in motor adaptation and learning [73–75]. Again, a comparison with previous studies is not straightforward, because the visuo-motor manipulation adopted for redirection changes dynamically throughout a single reaching, and is different for each spatial configuration of real and virtual targets. We expect that this would hinder the learning of the new forward mapping and the exploitation of explicit compensatory strategies. A dedicated study will be needed to assess this and to explore how the balance between implicit and explicit online corrections changes as the retargeting offset increases.

In summary, we explored the interplay between the efficiency of online corrections in visually guided actions, and the inhomogeneities in the sensorimotor processing associated with self-perception, biomechanical constraints and brain lateralization. The overall aim was to gain novel insights into the mechanisms underlaying fine tuning in motor control. Participants were consistently more efficient when the redirection implied movements towards the midline. The effect could not be associated with biomechanical constraints or laterality effects, neither motor nor perceptual. The effect could not either be reconducted to inhomogeneities in the multisensory processing associated with self-perception. All together, results favour an explanation rooted in the finer sensorimotor representations characterizing the space proximal to the body centre, where high-value functional actions, like fine manipulative skills and self-defence, typically take place.

4. Methods

(a) . Participants

Twelve participants took part in the experiment (five female; ages 23–52, 33.9 ± 7.9 mean and standard deviation). They read and signed a consent form before starting the experimental session. The procedure was approved by the Ethics Committee at Microsoft Research. Before entering VR, all participants filled a demographic questionnaire and completed the Edinburgh handedness questionnaire (http://www.brainmapping.org/shared/Edinburgh.php) [76]. All participants were right handed. Participants were given 50$ for the time spent in the experimental session.

(b) . Apparatus

The virtual environment was implemented and controlled by the Unity game engine and rendered on the HTC Vive Pro 2 headset (resolution: 2448 × 2448 LCD panels per eye, rate frame: 120 Hz, field of view: 120° horizontal). Participants sat on a chair throughout the experiment, and they were free to move their head. They held a HTC Vive controller in each hand. The virtual scene included a table, spatially aligned with a real table in the laboratory, and a rendering of the controllers held by virtual hands (figures 1 and 3). By default, the virtual controllers were spatially aligned with their real counterparts, and participants were instructed to hold them as shown in the rendering of the virtual hands. This assured a high degree of spatial alignment between real and virtual hands. Only during redirected reaches, the virtual controller, together with the attached reaching hand, was shown displaced from its real counterpart according to equation (4.1).

Figure 3.

The difference stages of a single trial are illustrated. In each panel, the upper left view shows the VR scene seen by participants; the upper right view shows the participant's real hands as filmed by a camera integrated on the HMD; the lower view shows the blending between the real set-up and the virtual representations as experienced by the participant.

(c) . General procedure

Participants sat with their chest placed at 3 cm from the edge of a 76 cm high table. Once fitted with the VR, they underwent a brief training of about 10 complete trials. Next, the experimental session started as a sequence of 420 trials, each consisting of a precise sequence of subtasks. First, participants had to place both controllers on two blue patches placed on the virtual table 48 cm from each other symmetrically with respect to the participant's midline (figure 1a). Once at this home position (figure 3a), one of the two virtual controllers turned red, indicating which hand should perform the reach. That controller should be brought on the start location S, a red patch placed on the table along the body midline, 30 cm away from the participant's chest (figure 1a).

When the active controller touched the start location, the virtual target (T_V) appeared as a green circle on the table (figure 3b); the real target (T_R) was instead never displayed on the virtual scene. Participants should next perform a reaching action towards the green target and stop there (figure 3c). Next, without moving the hand, they had to perform a 2AFC task, indicating if the performed reach felt normal or weird. We opted for these unusual categories, rather than more standard yes/no options, to keep the question presented after each trial as short as possible question, i.e. ‘How did the performed action felt?’ Responses were given by selecting one of the two options appearing on a virtual panel withthe controller's buttons (electronic supplementary material, video S1). In the onboarding procedure, participants were instructed about the procedure they should following indicating whether the performed action felt natural (normal) or whether they perceived some anomalies (weird). Once the subjective response was selected, the virtual controllers and hands disappeared from the scene, and a new orange target appeared. At this point, participants had to perform a blind reach (they could not see the hands nor the controllers) towards that new target, stop there and click on the controller to notify the task completion (figure 3d). The location of the blind reach target depended on the location of the physical target and was set 25 cm away, along the radial direction from the hand location to the start location (see the electronic supplementary material, figure S5). Participants could then bring back the controller to the home position to start a new trial.

Participants were free to pace the rhythm of the experiment and could take breaks at their will. Most participants took one or two breaks; few performed all the trials in one block. The time for completing the whole experiment ranged between 50 and 70 min including preparation and breaks.

(d) . Experimental design

The experiment consisted of a 2 × 3 × 7 repeated measures design (42 conditions), in which participants had to perform a series of redirected reaching actions in VR. The three factors were: (i) the hand performing the reaching task, hand (H), with two levels (right and left); (ii) the location of the physical target to which the real hand was redirected, real target (T_R), with three levels (centre, right and left); and (iii) the angular offset, redirection offset (θ_T), corresponding to the location of the virtual target (T_V) with respect to the real target (figure 3a), which could take seven different values. The geometry of the experimental design is shown in figure 1. All targets were arranged on a circle of 70 cm radius on the plane of the table, centred on the participant. The three locations of the real target were placed at −20° (left), 0° (centre), and 20° (right). For each real target, the seven virtual targets were arranged symmetrically around it, at [−30°, −20°, −10°, 0°, 10°, 20°, 30°]. This design corresponds all together to 11 different locations for the virtual target (figure 1a). Each participant performed 10 repetitions for each of the 42 conditions. The sequence of 420 trials was presented across 10 blocks, each consisting of a random presentation of all conditions.

(e) . Implementation of redirection

Action redirection consisted of a visual manipulation of the virtual hand so that for it to reach T_V, the real hand should be redirected to T_R [51]. To do so, as the virtual hand approaches the target, its representation is progressively displaced from the actual location of the real hand in the direction opposite to the real target. The geometry of the redirection is shown in figure 4. We define virtual displacement the vector ${\mathit{r}}_V={\mathit{T}}_R{\mathit{T}}_V$. Along the path, the virtual hand was displaced in the direction of ${\mathit{r}}_V$ by a fraction w of its length. For achieving a smooth redirection, w was set to vary linearly from 0 to 1 proportionally to the projected distance (d_HR) of the real hand position H_R along the $\mathit{S}{\mathit{T}}_R$ vector. The location of the virtual hand at a given time t depends therefore on $d_{HR}(t)$ as in the following:

$$T_V=\hspace{0.17em}{T}_R+w(t)\hspace{0.17em}{\mathit{r}}_V\hspace{0.17em}$$ 4.1

with $w(t)=d_{HR}(t)/|\mathit{S}{\mathit{T}}_R|$, and $d_{HR}(t)=\mathit{S}{\mathit{T}}_R\hspace{0.17em}\cdot \mathit{S}{\mathit{H}}_R$.

The redirection geometry was applied to the point centred on the lower part of controller surface touching the table.

(f) . Measurements

The experiment was designed to measure three different aspects involved in the experience of the action redirection: the efficiency of online motor corrections, the subjective experience of the performed action and the proprioceptive recalibration of perceived hand location associated with the visuo-motor manipulation.

(i) . Redirection cost

We introduced the redirection cost (C_R) as a novel metric to characterize the efficiency of online corrections in retargeted reaches. Figure 4 shows the relevant geometry involved. While the most efficient way to perform a redirected reach would be to follow a direct path from the start location S to the real target T_R, no a priori and explicit information about T_R is available to the users before starting the action. The reaching movement is instead planned and initially directed towards the virtual target (i.e. along $\mathit{S}{\mathit{T}}_V$). Only as the virtual hand gets displaced away from the intended direction, corrections to the initial motor plan should be made to bring the virtual hand towards the intended target T_V. We considered the total area between the actual hand trajectory and the ideal straight trajectory to the real target, $\mathit{S}{\mathit{T}}_R$, as a global scalar metric for the cost of the visuo-motor strategy implicitly adopted in the redirected action: the larger the area, the less efficient the redirection, and the higher its cost. By definition, we expect C_R to increase with the distance from T_R to T_V, i.e. with the redirection offset ${\theta }_T$. Also, the earlier corrections are made, the less the final trajectory will deviate from the optimal one and the lower C_R. Skewness of the curve and differences between curves associated with different T_R and H combinations could reveal the effect of perceptual and/or biomechanical asymmetries on the motor efficiency of the redirection.

(ii) . Subjective experience

The subjective experience was assessed via a 2AFC task. After completing each redirected reach, participants must indicate whether the performed movement felt normal or weird. This allowed us to quantify the probability of undergoing a perceptually transparent redirection (i.e. of not noticing the visuo-motor manipulation) as a function of the redirection offset ${\theta }_T$, for the different combinations of real target location and reaching hand. Overall, we expect to find a bell-shaped curve where the probability peaks for zero offset (i.e. no redirection) and decreases with increasing absolute offset $|{\theta }_T|$, with the slope characterizing how fast the redirection transparency degrades with increasing offset. Skewness of the curve and differences between curves associated with different T_R and H combinations could reveal the effect of perceptual and/or biomechanical asymmetries on the subjective experience of redirected actions.

(iii) . Proprioceptive recalibration

A complete description of the quantitative assessment of the proprioceptive drift as derived from the kinematics of blind reaching trajectory is given in the electronic supplementary material.

(g) . Data acquisition and processing

(i) . Kinematics

For each trial, the kinematics of the hand-held controller was recorded from the time at which the active controller was placed on the start location to the time at which the participants notified the end of the blind reach (see ‘General procedure’). Recordings of real controller kinematics were performed through the Unity engine at 120 Hz. Positional data were filtered with a zero-phase digital low pass-filter (Butterworth with 10 Hz cut-off) to remove noise.

Based on the tangential velocity profile, for each trial we extracted the two segments of interest: the kinematics of the redirected reach and of the blind reach. First, the kinematics was split from start to the time at which the response to the 2AFC was given, and from the latter to the end. Next, both reaching movements were delimited by identifying (i) the onset as the time at which the magnitude of tangential velocity (speed) exceeded the minimum threshold of 0.05 m s⁻¹, and (ii) the end of the reaching action as the time at which the speed decreased below 0.015 m s⁻¹. The kinematics associated with the two reaching actions was next processed to extract the relevant metrics. Both metrics were extracted from the reaching kinematics projected on the two-dimensional plane of the table, where the geometry of the different experimental conditions was fully defined.

(h) . Statistical analysis

The dependence of the redirection metrics from the experimental factors was tested with mixed models that account for inter-individual variability using random factors [77]. Different models were adopted according to the metric under scrutiny and the observed trend in the data. The dependence from the redirection offset ${\theta }_T$ was modelled with a second-order polynomial function, while the real target location T_R and the reaching hand H were treated as categorical factors and included as dummy variables when relevant.

The redirection cost C_R and the probability of normal response in the subjective 2AFC task (P_normal) were fitted with the following nonlinear mixed model:

$$x=\hspace{0.17em}{a}_0+u_0+{\beta }_0{T}_R+(a_1{\theta }_T+a_2{\theta }_T^2)\hspace{0.17em}+\hspace{0.17em}(a_1{\theta }_T+a_2{\theta }_T^2)\times ({\beta }_1\hspace{0.17em}{T}_R+u_1)\hspace{0.17em}+\hspace{0.17em}ϵ.$$ 4.2

Here a₁ and a₂ represent the coefficients of the second-order polynomial capturing the nonlinear dependence from ${\theta }_T$, u₀ and u₁ are the random effects accounting for different intercepts and shapes of the polynomial fit associated with individual participants (accounting for inter-individual difference), while [β₀, β₁] represent the fixed-effects, i.e. modulations of the intercept and shape associated with T_R. The reaching hand H was not considered as a fixed factor as none of the response variables displayed a significant dependence on the reaching hand.

As both C_R and P_normal results suggested that, for a given value of the absolute redirection offset $|{\theta }_T|$, the response variable was sensitive to the redirection direction, D_R, both response variables were further fitted with the following linear mixed model:

$$x=u_0+{\delta }_1{D}_R+a_1|{\theta }_T|+\hspace{0.17em}{a}_1|{\theta }_T|\times (\hspace{0.17em}{D}_R+u_1)\hspace{0.17em}+\hspace{0.17em}ϵ.$$ 4.3

Ethics

The procedure was approved by the Ethics Committee, IRB at Microsoft Research, officially registered with the U.S. Health and Human Services’ Office for Human Research Protections (OHRP) since 2017. Registration: IORG0008066, IRB00009672. Before entering VR, all participants gave informed consent following the Helsinki declaration and at the end of the experiment were given 50$ for the time spent in the session.

Data accessibility

The data are provided in the electronic supplementary material [78].

Authors' contributions

A.M.: conceptualization, data curation, formal analysis, investigation, methodology, software, validation, visualization, writing—original draft, writing—review and editing; E.O.: conceptualization, resources, software; B.C.: methodology, software; K.H.: funding acquisition, resources, supervision, writing—review and editing; M.G.-F.: conceptualization, investigation, methodology, project administration, resources, software, supervision, validation, visualization, writing—review and editing.

All authors gave final approval for publication and agreed to be held accountable for the work performed therein.

Conflict of interest declaration

The authors declare that the current manuscript presents a balanced and unbiased experiment on the field of VR and Visuo-Motor Control. The authors however report their affiliation to Microsoft, an entity with a financial interest in the subject matter or materials discussed in this manuscript. The authors have conducted the experimentation following scientific research standards.

Funding

This work was funded by Microsoft Research.