Supplemental vibrotactile feedback of real-time limb position enhances precision of goal-directed reaching

Abstract

We examined vibrotactile stimulation as a form of supplemental limb state feedback to enhance planning and ongoing control of goal-directed movements. Subjects wore a two-dimensional vibrotactile display on their nondominant arm while performing horizontal planar reaching with the dominant arm. The vibrotactile display provided feedback of hand position such that small hand displacements were more easily discriminable using vibrotactile feedback than with intrinsic proprioceptive feedback. When subjects relied solely on proprioception to capture visuospatial targets, performance was degraded by proprioceptive drift and an expansion of task space. By contrast, reach accuracy was enhanced immediately when subjects were provided vibrotactile feedback and further improved over 2 days of training. Improvements reflected resolution of proprioceptive drift, which occurred only when vibrotactile feedback was active, demonstrating that benefits of vibrotactile feedback are due, in part to its integration into the ongoing control of movement. A partial resolution of task space expansion persisted even when vibrotactile feedback was inactive, demonstrating that training with vibrotactile feedback also induced changes in movement planning. However, the benefits of vibrotactile feedback come at a cognitive cost. All subjects adopted a stereotyped strategy wherein they attempted to capture targets by moving first along one axis of the vibrotactile display and then the other. For most subjects, this inefficient approach did not resolve over two bouts of training performed on separate days, suggesting that additional training is needed to integrate vibrotactile feedback into the planning and online control of goal-directed reaching in a way that promotes smooth and efficient movement.

NEW & NOTEWORTHY A two-dimensional vibrotactile display provided state (not error) feedback to enhance control of a moving limb. Subjects learned to use state feedback to perform blind reaches with accuracy and precision exceeding that attained using intrinsic proprioception alone. Feedback utilization incurred substantial cognitive cost: subjects moved first along one axis of the vibrotactile display, then the other. This stereotyped control strategy must be overcome if vibrotactile limb state feedback is to promote naturalistic limb movements.

INTRODUCTION

Information arising from limb proprioceptors contributes importantly to the planning and ongoing control of movements (Sainburg et al. 1993; Sanes et al. 1984; Scheidt et al. 2005; Sober and Sabes 2003). Proprioceptors, predominantly muscle spindle afferents (Gandevia et al. 1992; Matthews 1988; cf. Proske and Gandevia 2012), give rise to sensations of limb position and movement. This is termed kinesthesia (Bastian 1887), which is essential for carrying out activities of daily life. Unfortunately, many people (including almost 50% of stroke survivors) experience deficits of kinesthesia in at least one arm (Carey and Matyas 2011; Connell et al. 2008; Dukelow et al. 2010). These sensation deficits contribute to impaired control of reaching and stabilization behaviors that are vital to an independent life style (Blennerhassett et al. 2007; Scheidt and Stoeckmann 2007; Tyson et al. 2008; Zackowski et al. 2004). Although such individuals often rely on visual feedback to guide movement, lengthy delays of visual processing [100–200 ms (Cameron et al. 2014)] yield slow, poorly coordinated actions that require focused attention (Ghez et al. 1995; Sainburg et al. 1993). Visually guided corrections come too late and result in jerky, unstable movements (Sarlegna et al. 2006).

Several research groups have proposed technological solutions to problems caused by somatosensory deficits, with notable examples being robotic retraining of proprioception (Cuppone et al. 2015, 2016; De Santis et al. 2015; Wong et al. 2011) and the application of supplemental performance feedback using vibrotactile (cf. Bark et al. 2011; Krueger et al. 2017; Lieberman and Breazeal 2007; Rinderknecht et al. 2013; Tzorakoleftherakis et al. 2015) or tactile stimulation (Ballardini et al. 2018; Bianchi et al. 2017; cf. Blanchard et al. 2011; Moscatelli et al. 2016; Terekhov and Hayward 2015). Building on that foundation, a long-term goal of our work is to reestablish or enhance closed-loop control of goal-directed behaviors in individuals with impaired kinesthesia by creating sensory substitution technologies that provide real-time feedback of the moving limb’s state (e.g., position and velocity of the arm and hand) to a site on the body retaining somatosensation (such as the ipsilesional arm).

We also consider the possibility of enhancing skilled manual performance in a variety of applications, including teleoperation tasks such as robotic surgery, where visual attention is constrained and precision of manual movements is desired (see also Ajoudani et al. 2014; de Jesus Oliveira et al. 2016; Sigrist et al. 2015). Previously, Krueger and colleagues (2017) showed that neurologically intact control subjects were able to enhance the accuracy and precision of goal-directed stabilization and reaching tasks performed with their dominant arm after only 3–5 min of practice with a vibrotactile sensory substitution system that applied feedback of dominant arm motion to the nonmoving arm and hand. A first set of experiments found that a limb state feedback scheme that encoded predominantly hand position information minimized performance errors during stabilization and reaching tasks. A second set of experiments compared this optimal limb state feedback to hand position error feedback [one of the simplest forms of “goal aware” feedback (Tzorakoleftherakis et al. 2015)] to determine the performance benefits of each encoding scheme. Both forms of synthesized feedback were able to enhance performance of stabilization and reaching behaviors in the absence of visual feedback. Although error encoding yielded superior results, likely due to the inclusion of information related to the spatial goal of movement, there are technological challenges associated with estimating the user’s motor intentions in nonstructured, dynamically changing environments. Therefore, based on the finding that subjects can use limb state feedback to enhance the control of stabilizing and reaching actions throughout the arm’s workspace, we sought to advance our long-term goal by characterizing the sensorimotor learning that leads to improved control of arm movements guided by supplemental limb state feedback systems.

Here, we explored how the human sensorimotor system may use supplemental limb state feedback to enhance the ongoing control of a moving limb. In particular, we assessed the extent to which supplemental vibrotactile feedback can be used to enhance the control of arm movements beyond performance limits imposed by intrinsic proprioceptive feedback. We exploited the fact that the vast majority of people, including neurologically intact individuals, exhibit imperfect somatosensory control of the arm and hand in the absence of ongoing visual feedback (Fuentes and Bastian 2010; Paillard and Brouchon 1968; Smeets et al. 2006; Wann and Ibrahim 1992) The most conspicuous and ubiquitous manifestation of imperfect somatosensation is “proprioceptive drift” (cf. Smeets et al. 2006; Wann and Ibrahim 1992), wherein bias in the perceived position of the unseen hand develops within a period of 12–15 s (a progressive degradation in the accuracy of proprioceptive sensation; cf. Paillard and Brouchon 1968). A second manifestation is a lack of precision in the proprioceptive estimation of limb state, whereby repeated estimates of hand position or joint configuration exhibit marked variability (cf. Fuentes and Bastian 2010).

We devised a planar reaching task wherein neurologically intact people grasped the handle of a 2-joint robotic manipulandum while making “target capture” movements between 25 spatial targets arranged in a 5 × 5 grid. To differentiate between the effects of short-term training with supplemental limb state feedback on subsequent performance of reaching movements made both with and without that feedback, we designed the intertarget distance of this grid to be less than the magnitude of spatial uncertainty in the proprioceptive assessment of hand position as reported by Fuentes and Bastian (2010). At the same time, we implemented a vibrotactile display of the arm’s planar work space wherein a hand displacement corresponding to the grid’s minimum intertarget distance also corresponded to a change in vibrotactile stimulation that was greater than three times the just-noticeable differences (JNDs) in the intensity of that stimulus, as reported by Shah and colleagues (2016, 2018). These design choices enabled us to test the following two main hypotheses: 1) that neurologically intact humans can learn to use vibrotactile limb state feedback (in contradistinction to error feedback) to enhance the ongoing control of a moving limb beyond the limits of intrinsic proprioception; and 2) that this training has aftereffects on subsequent movements performed without concurrent supplemental vibrotactile feedback (i.e., that training with supplemental limb state feedback can lead to improved proprioceptive control; cf. Cuppone et al. 2016). Preliminary aspects of this study have been presented in abstract form (Risi et al. 2016, 2017).

METHODS

Subjects.

Fifteen right-handed, neurologically intact human adults [6 female; 23.8 yr (SD 4.1)] were recruited from the Marquette University community. All provided written, informed consent to participate in the experimental procedures of this study, which were approved by Marquette University’s Office of Research Compliance. None of the subjects had known neurological disorders, all had normal or corrected-to-normal vision, and all were naïve to the purposes of the study. As described in the following paragraphs, all subjects participated in the same two experimental sessions on separate days. The two sessions differed in the sequence of tasks subjects were required to perform.

Experimental setup.

Each subject was seated in a high-backed, adjustable-height chair that was placed in front of a horizontal planar robotic manipulandum (Fig. 1A; cf. Scheidt et al. 2010). Subjects grasped the robot’s handle with their right hand. The left arm rested comfortably in a cloth sling with the forearm and hand pointed forward. A horizontal, opaque shield covered the workspace to block the subject’s view of the moving arm and the robotic apparatus. View of the stationary arm was not precluded. Seat height was adjusted such that the shoulder was aligned with the opaque shield, thereby allowing the subject to comfortably move the robot’s handle throughout the workspace. To display visual feedback, a 42-in. computer screen was oriented vertically above the shield within direct view (0.7 m in front of the subject). This screen provided real-time visual cues of hand and target positions when appropriate (the visual feedback schedule is described below).

Fig. 1.

Experimental setup and protocol. A: subject holding the end effector of a planar manipulandum, with visual occlusion shield. Red spheres indicate default configuration of the vibrotactile display. B: displayed grid and visual cues (cursor and targets). C: visual display was aligned with the subject’s midline. The subject was tested in both the right task space (i.e., trained task space) and the left task space (i.e., untrained contralateral task space). D: sequence of 12 experimental blocks performed on each experimental testing day. The familiarization, baseline, testing, and sham blocks each consisted of 25 reaches. The training block consisted of 5 sets of 25 reaches. Visual feedback (V) and vibrotactile feedback (T) of hand position were continuous (+), absent (−), or only displayed briefly before the onset of the reach (INIT). Each subject performed the 2 experimental sessions on separate days.

Prior work in haptic perception has shown that continuous tactile and kinesthetic signals from a stationary and the contralateral moving hand, respectively, are combined as if they came from the moving hand (Dupin et al. 2015). We seek to extend this idea into the application of sensory augmentation by providing supplemental kinesthetic feedback from one arm’s motion to the contralateral nonmoving arm. We stimulated the nonmoving arm because we ultimately seek to enhance closed-loop control of goal-directed behaviors performed in situations where the controlled end point (e.g., the hand or a robotic end-effector) provides little or no kinesthetic feedback aside from that provided through vision. This may happen, for example, in stroke survivors with impaired or absent somatosensation on the more involved side and in the expert telemanipulation of robotic tools during robotic surgery.

Supplemental vibrotactile feedback of the moving hand’s position was provided using a 2-channel (4 tactors) vibrotactile display attached to the nonmoving arm. The tactors (Pico Vibe 10-mm vibration motors; model no. 310-117; Precision Microdrives) have an operational frequency range of 50–250 Hz and a vibrational amplitude range of 0.20–0.97 N, which corresponds to an expected maximal forearm-plus-hand acceleration ranging between 0.53 and 0.77 m/s² depending on subject anthropometrics. For simplicity, we chose to represent vibrotactile stimulus intensity in terms of frequency even though the amplitude of the vibration covaries with frequency in the eccentric rotating mass tactors used in this study. The fact that stimulus frequency and amplitude are coupled in these low-cost tactors is not a limitation because experimental evidence shows that people perceive vibrotactile stimuli better when amplitude and frequency change coherently (e.g., both increasing or both decreasing together; Cipriani et al. 2012).

Tactors were taped to the skin over four different dermatomes (cf. Lee et al. 2008) with default locations indicated by red spheres in Fig. 1A. The minimum intertactor distance was always greater than 8 cm, which exceeded intertactor distances yielding minimal spatial interference (e.g., errors in localizing vibrotactile stimuli), as reported in prior related studies (5 cm: Cholewiak and Collins 2003; 6 cm: Cipriani et al. 2012). We confirmed via pilot testing in eight subjects (accelerometry data not shown) that the cross-coupling of vibration across stimulation sites was negligible when tactor separations are >6 cm.

Within the task space, the intensity of vibration increased in the vibrotactile display as a vector representation of the hand’s deviation from a fixed position—the center of the central target (described below). Thus, the vibrotactile display encoded information about limb state (i.e., hand position) rather than information about performance error relative to the current desired target. Two of the tactors encoded hand position along the x-axis of the workspace, whereas the other two tactors encoded hand position along the y-axis. The +y tactor was placed on the skin above the ulnar head (dermatome C8); the +x tactor was placed above the flexor carpi radialis muscle (dermatome T1); the −y tactor was placed above the brachialis muscle (dermatome C5); and the −x tactor was placed above the extensor carpi ulnaris muscle (dermatome C7). Elastic fabric bands were used to secure the tactors. The vibrotactile encoding scheme was designed such that placement of the robot handle to the right of the central target induced the +x tactor to vibrate, whereas placement of the robot handle away from the central target (i.e., toward the monitor) induced the +y tactor to vibrate. A detailed description of the vibrotactile encoding scheme is provided below. For additional details on tactor placement and their computer-controlled activation, see Krueger et al. (2017).

Task space and vibrotactile display.

Subjects performed reaches between 25 targets (0.6-cm radius dots) arranged in a 5 × 5 grid that had an edge length of 8 cm (Fig. 1B). We selected the minimum intertarget distance (2.0 cm) to be smaller than the range of uncertainty of proprioceptive perception of limb position [2.5 cm (SD 0.2)], as derived using joint angular uncertainty values reported by Fuentes and Bastian (2010). We constructed a mapping between the task space and its representation within the vibrotactile display such that very small displacements of the hand were clearly discriminable within the vibrotactile display (Fig. 2). Under this mapping, JND in intensity of sequential vibrotactile stimulations as reported by Shah et al. (2016) [i.e., a change of 16.3 Hz (SD 11.8)] corresponded to 0.3 cm (SD 0.2) of hand displacement. Therefore, the minimum intertarget distance was more than six times greater than the vibrotactile JND and considerably smaller than the range of uncertainty of proprioceptive perception of limb position.

Fig. 2.

Vibrotactile encoding scheme: position feedback. Left axis: activation of one tactor [percentage of full scale range (FSR)] plotted against handle displacement along the direction encoded by the tactor. The black staircase accurately represents the finite spatial resolution of the mapping between hand displacement and tactor activation. The height of the gray box indicates the average discrimination threshold for dermatome C7 (from Shah et al. 2016). Right axis: tactor vibration amplitude is a monotonic function of hand displacement along the encoding direction. JND, just-noticeable difference.

We employed a piecewise linear activation map (Fig. 2) wherein tactor activation intensity was a function of the hand’s distance from the center of the grid of targets. Specifically, tactor activation was 71% full-scale range (i.e., 3.5 V) when the hand was on a far target (4 cm from the center of the task space), 45% when the hand was on a near target (2 cm from the center), and 26% when the hand was on the outer edge of the central target (i.e., 0.6 cm from its center). The feedback was turned off in all tactors (i.e., the activation was 0.0 V) only when the hand was centered on the central target (i.e., within 0.06 cm from the grid center). This range of activations corresponded to tactor vibration amplitudes ranging from 0–0.78 N. The discontinuity at 0.06 cm served to counteract static friction within the tactor motor. The frequency of vibration (in units of hertz) was ~250 times the amplitude of vibration (in units of newton).

Experimental protocol.

Subjects performed two experimental sessions on separate days within 2 wk (range: 1–10 days). Before testing on each day, subjects were introduced to the vibrotactile display and invited to freely explore the robot’s workspace by displacing the robot’s handle along the two cardinal axes. During this period, subjects were frequently and repeatedly asked to report which tactors were activated at any given time. If subjects made errors in detecting vibration on any tactor, we adjusted that tactor’s location by 2 or 3 cm so that each subject could reliably detect and report vibration. Subjects were then encouraged to explore the vibrotactile display by making self-guided reaching movements. This introduction and exploration procedure took between 2 and 5 min to complete. This amount of time for exploration was sufficient to provide a basic understanding of how a portion of the arm’s larger workspace was encoded within the vibrotactile feedback.

During the main part of both experimental sessions, subjects performed 12 blocks of 25 reaching movements, 1 reach per trial. The target grid was always displayed on screen in low-contrast gray, with the current target presented in vivid green (Fig. 1B). Subjects were instructed to “capture the target as accurately as possible.” Upon completing a reach, subjects announced that they had arrived at the target and the experimenter registered the event by pressing a button. Subjects had 10 s to complete each reach. At the end of the trial, to emphasize the beginning of a new trial, the previous target became an empty green dot. After a few seconds [an intertrial interval of 2.3 s (SD 0.7)], a different dot turned green, providing the cue to move. The target sequences were pseudorandomized across each block of 25 trials, with the distance between consecutive targets in the range of 4.0–6.32 cm.

The sequence of trial blocks in the two experimental sessions was identical except for the last block (Fig. 1D). The first nine blocks were performed with the center of the physical target grid shifted 20 cm to the right of the subject’s midline (i.e., Fig. 1C: trained task space). The purpose of this shift was to sever usual correspondences between visual and intrinsic proprioceptive representations (memories) of target locations, thereby facilitating differential assessment of movement accuracy driven by visual, proprioceptive, and vibrotactile feedback of limb state. As all subjects used their right hand to move the robotic handle, subjects practiced primarily on the ipsilateral side of the arm’s reachable workspace. Subjects then performed two test blocks with the center of the physical target grid shifted 20 cm to the left of the subject’s midline (i.e., Fig. 1C: on the contralateral side of the arm’s workspace and in the contralateral task space). The last block was performed in either the trained task space (day 1) or the contralateral task space (day 2). By contrast, the center of the visual grid was aligned at all times with the subject’s midline (i.e., Fig. 1C: visual task space).

The 12 blocks were performed under various combinations of visual (V) and vibrotactile (T) feedback of hand position (Fig. 1D). Visual and vibrotactile feedback were either continuous (+), absent (−), or only displayed briefly before the onset of the reach (INIT). During the first trial block (Fig. 1D: familiarization, trained task space), subjects were familiarized with the task by performing 25 reaches with visual feedback but without vibrotactile feedback (V₊T₋); a small white visual cursor (0.5-cm radius) continuously tracked hand position during this block. The purpose of the familiarization block was to ensure that subjects understood the basic mechanics of the experimental task. Next, to assess baseline accuracy before training with supplemental vibrotactile feedback, subjects performed a block of reaches without any performance feedback (V₋T₋; Fig. 1D: baseline 1, trained task space). The purpose of baseline blocks was to assess reach performance when driven only by intrinsic proprioceptive feedback.

Five blocks of training with vibrotactile feedback followed the baseline assessment. Throughout the training movements, vibrotactile feedback of the state of the moving right arm was provided to the stationary left arm. Because we used a state encoding for the vibrotactile feedback (not error encoding), it was critical that training should establish a consistent cross-modal spatial correspondence between perceived hand position and the actual hand position encoded by the vibrotactile feedback (cf. Deneve and Pouget 2004). Thus, visual feedback of the cursor was provided only briefly when the cursor was inside the starting target, disappearing when the new target appeared (V_INIT T₊; Fig. 1D: training, trained task space). No cursor or other visual cues were provided during training trials to minimize visual distraction from the vibrotactile and intrinsic proprioceptive feedback. The initial glimpse of hand position was intended to reduce the development of “drift” between proprioceptive and visual estimates of hand position (Wann and Ibrahim 1992), which would degrade the desired correspondence between actual and encoded hand positions. Without a consistent correspondence, subjects would no longer interpret the vibratory signals as encoding limb state, thereby interfering with the acquisition of an association between vibrotactile and proprioceptive representations of hand position within the reachable workspace.

During training, subjects were also provided haptic knowledge of results in the form of robot-guided corrections to target capture errors. After each training reach (i.e., when the experimenter pressed the button indicating target capture or after the 10-s timeout), the desired target turned white and the robot gently guided the subject’s hand to the correct target location (unless the subject had correctly hit the target). The purpose of this manipulation was to eliminate the need for visually guided voluntary corrections that would have confounded the effect of vibrotactile training while still providing objective nonvisual information about performance errors, which subjects were to reduce with practice (i.e., to induce learning). Upon completion of the robotic correction, a new desired target turned green. On average, the five blocks of training lasted 30 min in total.

We then examined aftereffects of training by assessing posttraining baseline performance without either visual or supplemental vibrotactile feedback (V₋T₋; Fig. 1D: baseline 2, trained task space). By comparing performance in this block of trials with that in the first baseline block, we can quantify the extent to which training induced an improved ability to use intrinsic proprioception to guide reaching movements (i.e., perceptual learning). We also tested the extent to which subjects could recall what they had learned during training by having them complete one test block of reaches with only vibrotactile feedback and without visual feedback or haptic corrections (V₋T₊; Fig. 1D: testing, trained task space). The order of the baseline 2 and testing blocks were counterbalanced across subjects to mitigate potential order effects.

Next, we assessed the capability of subjects to apply what they had learned in the trained task space to reaches performed in the contralateral task space. Here, a baseline block without visual or vibrotactile feedback was performed in a physical task space centered 20 cm to the left of the subject’s midline (V₋T₋; Fig. 1D: baseline 3, contralateral task space). This block was followed by a “contralateral task space test block” of reaches with only vibrotactile feedback and no visual feedback (V₋T₊; Fig. 1D: testing, contralateral task space).

Finally, the last 25 reaches were performed with different feedback conditions in the 2 experimental sessions. On day 1, subjects reached in the ipsilateral (right) task space guided by sham vibrotactile feedback (V₋T_SHAM; Fig. 1D: sham, trained task space). In this feedback condition, the vibration recorded during a training block was played back without reference to the subject’s actual performance. Providing noninformative vibrotactile feedback allowed us to assess whether potential performance enhancements were due specifically to the information contained within the supplemental vibrotactile feedback and not by the mere presence of vibration (see also Krueger et al. 2017). Subjects who noticed that the vibration lost its information content were instructed to nevertheless attempt to use the “feedback” as best they could; we informed subjects only after the end of the experimental session that this last block involved noninformative vibration. During the second experimental session (day 2), the last block was meant to assess potential order effects of test blocks in the contralateral task space, i.e., the extent to which test block performance improvements in the contralateral task space might simply have been due to additional practice on the task during the posttraining baseline. Therefore, instead of sham feedback, subjects performed an additional 25 baseline reaches in the contralateral task space without either visual or vibrotactile feedback (V₋T₋; Fig. 1D: baseline 4, contralateral task space). At the end of both sessions, subjects were asked to report how they used the information encoded in the vibratory feedback to guide their reaching movements.

Data analysis.

Analysis of kinematic performance focused principally on hand position at two time points during each reach, i.e., when the target first appeared (“initial hand position”) and when the subject indicated that they had acquired the intended target (“final hand position”). Our primary measure of accuracy was the “target capture error,” which is defined as the absolute distance between the final hand position and the target location.

We derived eight additional task-related variables. We defined the “actual movement vector” as a vector pointing from the initial to the final hand position [Fig. 3A: (1)]. We defined the “ideal movement vector” as the vector pointing from the actual initial hand position to the intended target location [Fig. 3A: (2)]. We defined “directional error” β (Fig. 3A) as the magnitude of the angle between the actual movement direction (solid black line) and the ideal movement direction (dashed line) (Ghez et al. 1995; Gordon et al. 1995). β-Values are small if the subject correctly estimates his/her initial hand position and moves the hand in the proper direction (cf. Fig. 3A). We defined “extent error” as the difference between the actual movement magnitude and the ideal movement magnitude; extent error is positive if the actual movement extent exceeds the ideal extent (Ghez et al. 1995; Gordon et al. 1995).

Fig. 3.

Single trial performance in two distinct trials. Triangles: initial hand position; large gray circle: current intended target location; cross: final hand position. A: depiction of direction and extent error; the dashed line shows the ideal movement direction and extent; the solid black line shows the actual movement direction and extent. Movement extent error is defined as the length of line (2) subtracted from the length of line (1). β is defined as the angle between the ideal and actual movement vectors. The angle is positive if the rotation from the actual vector (solid line) to the ideal vector (dashed line) was counterclockwise. B: depiction of aspect ratio calculation. Gray path: a subject’s hand trajectory. Solid black lines: d₁ is the segment between the initial position and the final position; d₂: the perpendicular distance between line segment d₁ and the point of greatest hand path deviation from d₁.

We computed the “spatial contraction/expansion” of reach end points along both coordinates (i.e., cont/exp_xy) to assess the fidelity of the subject’s internal representation of extrapersonal space in both task spaces (cf. Dukelow et al. 2010). For the outer targets, spatial contraction/expansion was computed as the area of the quadrilateral spanned by the final hand positions of reaches made to the outer targets of the grid. This area was then normalized to (i.e., divided by) the actual area of the outer grid square. Spatial contraction/expansion was similarly computed for movements made to the inner targets of the grid.

We also evaluated the extent to which subjects’ trajectories became straighter as they practiced using the vibrotactile feedback. To this end, we defined “aspect ratio” as the ratio |d₂| / |d₁|, where |d₁| was defined as the distance between the initial hand position and its final position, and |d₂| was defined as the perpendicular distance between line segment d₁ and the point of greatest hand path deviation from d₁ (Fig. 3B). The aspect ratio is a larger number (closer to 1) when the subjects used an inefficient path and is a small number when the subjects used nearly a straight-line path. We also introduce a new scalar performance measure—the “decomposition index” [DI (Eq. 1)]—a unitless performance measure that, together with aspect ratio, can distinguish between different strategies for using vibrotactile feedback to solve the target capture task in the absence of visual feedback. Specifically, the DI quantifies the extent to which sampled-data hand paths in any given trial move exclusively parallel to the cardinal {x, y} axes of the vibrotactile display:

$$\text{DI}={\displaystyle \sum }_{n=2}^N\left\{\frac{1}{2}\left|\frac{x\left(n\right)-x\left(n-1\right)}{x\left(N\right)-x\left(1\right)}\right|\left|\frac{{\dot{y}}_{max}-\dot{y}\left(n\right)}{{\dot{y}}_{max}}\left|+\frac{1}{2}\right|\frac{y\left(n\right)-y\left(n-1\right)}{y\left(N\right)-y\left(1\right)}\right|\left|\frac{{\dot{x}}_{max}-\dot{x}\left(n\right)}{{\dot{x}}_{max}}\right|\right\},$$ (1)

where N corresponds to the maximum number of data samples within a given trajectory, n corresponds to the sample number in that trajectory, and ${\dot{x}}_{max}$ and ${\dot{y}}_{max}$ correspond to the peak hand speeds along each of the cardinal axes. If the hand first moves parallel to the x-axis [i.e., y(n) − y(n − 1) = 0] and then parallel to the y-axis [i.e., x(n) − x(n − 1) = 0], the initial motion along the x-axis will yield zero values of $\dot{y}\left(n\right)$; thus, the equation will “integrate” the x-axis displacements with unitary weight in the first half of the movement. In the second half of this hypothetical movement, displacements along the y-axis are integrated with unitary weight because the x-axis velocity is zero during this portion of the trajectory. Thus, the overall DI value would be high. By contrast, if a movement were to be directed along a diagonal or any off-axis movement, the largest sample displacements along the x-axis would occur at the same time as the peak speed along the y-axis, leading to small contributions to DI; a similar argument holds for the y-axis displacements. In this second hypothetical case, the DI would be small. As presented in the appendix, off-axis, straight-line trajectories with bell-shaped velocity profiles yield DI values equal to 0.24.

Finally, we computed “target capture time” as the difference between the moment when the target appeared and the moment when the subject indicated she/he had acquired the intended target.

Statistical hypothesis testing.

This study tested two main hypotheses. The first proposes that neurologically intact subjects can learn to use supplemental vibrotactile feedback of hand position to enhance the ongoing control of reaching movements beyond limits imposed by intrinsic proprioception. The second proposes that this training has aftereffects on subsequent movements performed without concurrent supplemental limb state feedback (i.e., that training induces improved proprioceptive control; cf. Cuppone et al. 2016). To test these hypotheses, we compared target capture error across feedback conditions and days.

More specifically, we designed the sequence of training blocks to facilitate planned paired t-test comparisons between specific training blocks (feedback conditions) either within or across days. To test for the presence of sensorimotor learning (the first hypothesis), we compared target capture errors in the initial training block to those in the final training block in each day. We also compared reach performance driven by intrinsic proprioceptive feedback alone against performances driven additionally by vibrotactile limb state feedback by comparing target capture errors in the two posttraining test blocks in the trained task space (i.e., comparing blind reach performance with and without vibrotactile feedback). These analyses were performed separately for the training blocks of days 1 and 2. To test for aftereffects of vibrotactile feedback training on subsequent movements performed without either visual or vibrotactile feedback (i.e., the second hypothesis pertaining to perceptual learning), we compared target capture errors between no-vibration baseline blocks performed immediately before and after the five training blocks on both days.

We performed ancillary paired t-test comparisons to examine the specificity of action for sensory augmentation using vibrotactile feedback. We used paired t-tests to compare target capture errors across test phase blocks with and without vibration in the contralateral (untrained) task space to test if the ability to use vibrotactile state feedback extends beyond the trained workspace. We compared the V₋T₊ condition with the V₋T_SHAM condition on day 1 to verify that any performance improvement observed when using supplemental feedback was due specifically to the information encoded within the vibrotactile display, rather than the mere presence of vibration. We also tested for retention of learning across days by comparing target capture errors within the initial training sets on days 1 and 2. We also compared day 2 baseline block performance within the contralateral task space immediately before and after the test block with vibration (i.e., block 9) to verify that any potential benefits of vibration in the contralateral task space were indeed due to the vibrotactile feedback rather than a possible order effect.

Additional supplemental statistical tests were performed on the secondary performance measures (spatial contraction/expansion, extent and directional errors, DI, target capture time) to address specific questions related to the strategies used by subjects to solve the target capture task using vibrotactile feedback. All statistical tests were carried out in the SPSS computing environment (SPSS version 24.0; IBM). We report results of the ancillary analyses after Bonferroni correction such that effects were considered statistically significant at the α = 0.05 level.

RESULTS

This study sought to characterize how human subjects learn to use supplemental vibrotactile feedback of performance to enhance reach accuracy beyond limits imposed by the inherent inaccuracies of intrinsic proprioception, as reported in the literature (cf. Fuentes and Bastian 2010). We asked subjects to perform a horizontal planar reaching task in two task spaces. We constructed a mapping between the position of the moving hand and its representation within the vibrotactile display such that small, goal-directed hand displacements were more easily discriminable using supplemental vibrotactile feedback than with intrinsic proprioceptive feedback in the moving arm. Subjects then made goal-directed (cued) reaches to specific locations within the task spaces.

Target capture error.

Figure 4 depicts all final end-of-reach hand positions (gray dots) achieved on day 1 by a selected subject; this performance was typical of the population (see group data reported below and in Fig. 5). During the familiarization block when visual cues were provided (Fig. 4: familiarization), the subject was able to accurately reach all targets (i.e., the reach end points sampled the target task space uniformly; shown in the gray shaded region). Target capture performance degraded dramatically when the visual cues were subsequently removed (Fig. 4: baseline 1, trained task space): the subject exhibited both a systematic drift toward the body midline as well as a spatial expansion of the performance space.

Fig. 4.

Performance of a selected subject in the reaching task on day 1. For each of the blocks, hand positions after reaching are plotted for inner targets (circles) and outer targets (squares). The spatial location of the target grid and its location relative to the center of the visual workspace (axis lines) is represented by the shaded square. A dashed line connects the outer targets. The progression of experimental blocks ranges from top to bottom in the figure.

Fig. 5.

Group results: root-mean-square target capture error vs. experimental block for both days. Black bars: the initial block with visual feedback (V₊) at the start of each experimental session; V₋ indicates blocks without visual feedback and V_INIT indicates visual feedback of the cursor only at the beginning of a trial while the subject was near the starting target. Gray bars: blocks with vibrotactile feedback (T₊); open bars: blocks without vibrotactile feedback (T₋); dashed bar: sham feedback (T_SHAM). Horizontal solid bars under the x-axis indicate blocks in the training workspace, and the horizontal dashed lines indicate blocks in the generalization workspace. Note that we did not compare target capture errors across the T₊ blocks in the V_INIT and V₋ conditions because differences in the visual feedback conditions at the onset of training trials can differentially affect the accumulation of proprioceptive drift (cf. Wann and Ibrahim, 1992). Error bars: 95% confidence interval about the mean.

Next, the subject performed five blocks of training with vibrotactile limb state feedback (~30 min). Although subjects were not provided concurrent visual feedback of cursor motion, they were provided a glimpse of the hand’s initial position just before movement and knowledge of results in the form of robot-guided corrections at the end of each reach. These manipulations were intended to prevent the accumulation of proprioceptive drift, which would degrade during training the requisite cross-modal (vibrotactile and proprioceptive) representations of hand position within the reachable workspace. Across the five training blocks on day 1, target capture errors decreased (accuracy improved) from the initial to the final block of training (Fig. 4: initial training and final training, respectively).

After training, the subject exhibited a reduced expansion of the task space during posttraining baseline assessment (Fig. 4: trained task space baseline 2), thereby demonstrating a beneficial aftereffect of training even without concurrent vibrotactile feedback. By contrast, proprioceptive drift toward the midline did reappear during the posttraining baseline 2 block. When vibrotactile feedback was reinstated, the drift immediately resolved (Fig. 4: trained task space testing). A similar pattern of behavior arose when the subject was tested in the contralateral task space: we observed drift toward the body midline and spatial expansion in the absence of vibrotactile feedback (Fig. 4: contralateral baseline 3). With concurrent vibrotactile feedback, there was an increased target capture accuracy and reduced proprioceptive drift (Fig. 4: contralateral testing). Target capture performance subsequently degraded when noninformative feedback was provided in the trained task space (Fig. 4: compare sham to trained task space testing). These results demonstrate that the performance enhancements seen with supplemental limb state feedback were specifically due to the information encoded within the vibrotactile display and not simply the mere presence of vibration applied to the nonmoving arm.

The individual-subject results were characteristic of the study cohort as a whole. Across the study population, subjects reached the targets with accuracy and precision in the presence of concurrent visual feedback, whereas performance degraded markedly when visual cues were subsequently precluded and no supplemental feedback was provided (Fig. 5: compare the two left-most bars in each panel). Remarkably, reach errors dropped precipitously and immediately after vibrotactile feedback was provided (Fig. 5: compare the second to the third left-most bars). To characterize the extent to which subjects might improve their ability to use vibrotactile limb state feedback, we examined differences in accuracy across the five training blocks (i.e., the five shaded bars above the horizontal bar marking the V_INIT trials). On day 1, errors decreased most rapidly within the first two to three blocks of training. Target capture errors appeared to reach a performance plateau by the third block of training, with little further improvement in performance in subsequent training blocks. Similar patterns of performance changes were observed across trial blocks in both experimental sessions.

We first challenged the hypothesis that neurologically intact subjects could learn to use vibrotactile limb state feedback to enhance the ongoing control of reaching movements beyond limits imposed by intrinsic proprioception. One-sided paired t-tests found within-day reduction of target capture errors during training to be significant on both days (t₁₄ > 3.99, P < 0.0005 in both cases). Performances immediately and similarly degraded when vibration was removed after training (i.e., when subjects had to rely solely on intrinsic proprioceptors to perform the task in the Fig. 4 baseline 2 block and in the first open bar to the right of the training bars in Fig. 5). During posttraining testing blocks, the performance-enhancing effects of supplemental vibrotactile feedback were immediately restored (cf. Fig. 5: gray bars above solid horizontal V₋ lines) such that target capture errors dramatically and significantly decreased relative to the posttraining baseline block (one-sided paired t-test: t₁₄ > 6.98, P < 0.0005 on both days). These findings confirm that subjects could learn to use vibrotactile limb state feedback to improve performance in the trained task space beyond levels typically observed when using intrinsic proprioception alone.

Across the five blocks of training on each day, the learning trend was reasonably well characterized by a falling exponential decay of performance error, with learning rate time constants averaging 1.4 blocks (SD 1.1) on day 1 and 1.3 blocks (SD 0.9) on day 2. We found evidence that root-mean-square target capture error decreased between the initial training block of day 1 to the initial training block of day 2 (one-sided paired t-test: t₁₄ = 2.25, P < 0.05), by which we infer that subjects retained knowledge gained through training from days 1 to 2. This increased accuracy was due to the specific information encoded within the vibrotactile display (and not merely the vibratory stimulation itself) because performance degraded once again when noninformative sham feedback was provided at the end of day 1 (Fig. 5, left: dashed bar; one-sided paired t-test between posttraining recall and sham: t₁₄ = 6.26, P < 0.0005).

We next tested that the proposed training has aftereffects on subsequent movements performed without such feedback. Consistent with this premise, when vibrotactile feedback was removed after training on day 1, target capture errors in the baseline 2 V₋T₋ block (Fig. 5: first open bar to the right of the training bars) were substantially lower than those in the baseline 1 V₋T₋ block (Fig. 5: second open bar from the left; one-sided paired t-test: t₁₄ = 4.1778, P < 0.0005). This beneficial aftereffect of training was evidently limited to the initial bout of training, as no further improvements were observed after a second bout of training (i.e., when comparing baseline 2 performance on day 2 to baseline 2 performance on day 1: t₁₄ = 1.1140, P = 0.284).

Consistent with the hypothesis that the utility of vibrotactile limb state feedback generalizes to movements performed in untrained regions of the reachable workspace, we observed immediate performance enhancements due to the vibrotactile feedback also in the contralateral workspace. Specifically, we observed a marked improvement in reach accuracy between the baseline 3 and testing blocks in the contralateral untrained workspace (Fig. 5: open vs. gray bars above dashed, bottom line; one-sided paired t-test: t₁₄ = 7.15, P < 0.0005). The observed benefits of vibrotactile feedback were not simply due to an order effect, as confirmed by the baseline 4 block (V₋) at the end of the day 2 (Fig. 5: second open bar above dashed, bottom line in right panel), wherein root-mean-square errors increased relative to the prior testing block (one-sided paired t-test: t₁₄ = 7.64, P < 0.0005). We observed no noticeable differences in target capture error within the same testing conditions across the two task spaces within each day. These results support the hypothesis that the benefits of supplemental vibrotactile feedback training generalize across the arm’s reachable workspace.

Direction and extent errors.

We analyzed directional and extent errors (see Fig. 3 for a description) to gain insight into the effect of concurrent vibrotactile feedback on proprioceptive drift. Because even a glimpse of hand position feedback can arrest or eliminate drift (Wann and Ibrahim 1992), we present in Fig. 6 only those blocks performed entirely without visual cues (pretraining: baseline 1; posttraining: baseline 2 and testing in trained task space and baseline 3 and testing in the contralateral task space). Whereas unsigned directional error varied markedly across experimental blocks, we observed no evident effect of days, nor any apparent interaction between experimental blocks and days. On both days, directional error decreased in the trained workspace from baselines 1 and 2 (no vibrotactile limb state feedback) to testing (with vibrotactile feedback) (Fig. 6A: open bars vs. gray bars with solid lines below). A similar trend was reported in the contralateral workspace (Fig. 6A: open vs. gray bars with subjacent dashed lines). Movement extent errors similarly varied across experimental blocks (Fig. 6B), with the most striking and consistent differences observed between baseline (no vibrotactile feedback) and testing (with vibrotactile feedback) in both the trained and generalization task spaces (Fig. 6B: open vs. gray bars to the right of the vertical dotted line in each day). Thus, we infer that subjects used limb state information encoded within the supplemental vibrotactile feedback signals to correct their initial drift and improve movement accuracy.

Fig. 6.

Group results for direction and extent errors across blocks for both days. A: direction error vs. blocks in experimental phases wherein visual cues were precluded (V₋). Pretraining: baseline 1; posttraining: baseline 2 and testing in the trained workspace (open and gray bars above solid horizontal line) and baseline 3 and testing in the generalization workspace (open and gray bars above dashed horizontal line). The vertical dotted line indicates where training occurred in the sequence of trial blocks. B: extent error vs. blocks in experimental phases wherein visual cues were precluded. The presentation of trial block performance is as described for A. Error bars: 95% confidence interval; gray bars: blocks with vibrotactile feedback (T₊); open bars: blocks without vibrotactile feedback (T₋).

Spatial contraction/expansion.

The analysis of spatial contraction/expansion (cont/exp_xy) revealed an expansion of target space when visual cues were not provided (Fig. 7). However, the cont/exp_xy index was closer to the ideal value of 1.0 when the vibrational feedback was provided in testing blocks relative to baseline blocks, both in the trained task space and in the generalization task space (Fig. 7: gray bars vs. posttraining open bars). Moreover, we observed no clear difference in testing block performances between the trained task space and the generalization task space. By contrast, comparison of the cont/exp_xy index between the baselines 1 (pretraining) and 2 (posttraining) blocks on day 1 revealed a training effect, i.e., a significant reduction toward the ideal value for the outer targets (t₁₄ = 5.432, P < 0.0001) and the inner targets (t₁₄ = 3.9876, P = 0.0013) (Fig. 7: compare open bars on either side of the vertical dotted line for day 1). Taken together, these results support the conclusion that practicing to use supplemental limb state feedback improved subjects’ short-term internal representations of extrapersonal space throughout the arm’s workspace, even when that feedback was no longer present.

Fig. 7.

Group results for spatial contraction/expansion vs. blocks in experimental phases wherein visual cues were precluded (V₋). Pretraining: baseline 1; posttraining: baseline 2 and testing in the trained workspace (open and gray bars above solid horizontal line) and baseline 3 and testing in the generalization workspace (open and gray bars above dashed horizontal line). Error bars: 95% confidence interval; solid bars: cont/exp_xy computed for the outer targets; dashed bars: cont/exp_xy computed for the inner targets; gray bars: blocks with vibrotactile feedback (T₊); open bars: blocks without vibrotactile feedback (T₋).

Aspect ratio.

We next scrutinized hand path aspect ratio to evaluate the extent to which training with vibrotactile feedback facilitated the production of rectilinear hand paths. Hand paths were relatively straight when subjects were provided visual feedback of hand motion (Fig. 8: black bars; aspect ratios less than 0.2). On the contrary, paths exhibited marked curvature in the presence of supplemental vibrotactile feedback (Fig. 8: gray bars; with aspect ratios greater than 0.4). Hand path curvatures assumed intermediate values with neither visual nor vibrotactile feedback. The fact that aspect ratio increased when vibrotactile feedback was provided suggests that subjects employed different control strategies to reach the intended target depending on current feedback conditions (we explore this observation further below). Moreover, we observed no meaningful difference between aspect ratios produced in the initial and final training blocks on either day, suggesting that subjects may need much more than two training sessions if they are to recover “natural” straight-line hand trajectories when incorporating vibrotactile limb state feedback into the ongoing control of reaching movements.

Fig. 8.

Group results for aspect ratio vs. blocks for both days. Error bars: 95% confidence interval; black bars: blocks with visual feedback (V₊). V₋ indicates blocks without visual feedback. Gray bars: blocks with vibrotactile feedback (T₊); open bars: blocks without vibrotactile feedback (T₋); dashed bar: blocks with sham feedback (T_SHAM).

Decomposition index.

Kinematic analyses of individual subject hand trajectories during each experimental phase reveal the use of different movement strategies before training with the vibrotactile feedback versus after training. Before training, subjects made goal-directed reaches that were reasonably straight (Fig. 9A: a representative baseline movement shown with dashed line), with hand speed profiles that were approximately bell-shaped (Fig. 9B, top: thin dashed trace). By contrast, movements performed after training typically had bimodal hand speed profiles (Fig. 9B, bottom: thin solid trace) corresponding to hand paths that first moved predominantly along one cardinal axis of the vibrotactile display, then the other. The selected baseline and training hand paths shown in Fig. 9A had unitless DI values (Eq. 1) equal to 0.40 and 0.82, respectively.

Fig. 9.

Decomposition index (DI) vs. experimental block. Sample hand paths (A) and the corresponding speed profiles (B) performed by a selected subject from a movement during the baseline 1 block (dashed traces) and the first training block (solid traces). On initial exposure to the reach task, the subject made movements that were reasonably straight (A) with speed profiles that were roughly bell-shaped (B). Also plotted for comparison in B are speed profiles from ideal minimum-jerk trajectories (thick black traces). Whereas the minimum-jerk trajectories always yield DI values equal to 0.24, regardless of movement extent, movement speed, and movement direction (excepting the singular cases described in the appendix), the baseline block movement with single speed peak yielded a DI value of 0.40 and the training block movement with 2 speed peaks yielded a DI value of 0.82. Group results for DI values vs. experimental block for both days (C). The meanings of bar shadings and labels are consistent with those described for Fig. 8. The long horizontal reference bar corresponds to the DI value obtained from ideal straight-line minimum-jerk movements. Error bars: 95% confidence interval.

As described in the appendix, off-axis minimum-jerk hand paths yield DI values equal to 0.24 (Fig. 9C: horizontal line). This is true regardless of movement extent, movement speed, or movement direction (excepting trajectories directed along one of the cardinal axes of the vibrotactile display, which yield singular DI values; see the appendix for more details). Numerical DI values for baseline movements deviate from the ideal value because individual baseline movements had hand trajectories that were not quite straight and hand speed profiles that were not quite smooth. By contrast, DI values for training trajectories were relatively high because hand paths were curved and speed profiles were bimodal.

The kinematic observations presented for individual trajectories (Fig. 9, A and B) generally held true across the study population across both days of training (Fig. 9C). Whereas DI values were relatively low before training with vibrotactile feedback on day 1, DI values immediately jumped high at the onset of training as subjects employed a “decomposed” movement strategy whereby they first moved along one cardinal axis of the vibrotactile display, then the other. At the onset of training, all 15 subjects made vibrotactile-guided reaches with DI values ≥ 0.60. By the end of day 2 training, 13 of 15 subjects persisted in making vibrotactile-guided reaches with DI values > 0.60. We did not observe any meaningful reduction in DI values across the 2 days of training (i.e., from the first training block on day 1 to the last training block on day 2; t₁₄ = 0.9039, P = 0.381). Remarkably, decomposition persisted in posttraining reaches performed without vibrotactile feedback, both in the trained and contralateral workspaces on both days.

Target capture time.

Subjects were instructed to capture the targets as accurately as possible, without constraint on time or hand path. We expected that initial attempts to integrate the novel vibrotactile feedback signals into the ongoing planning and control of arm movement would increase target capture time as subjects initially learned how best to interpret and use the vibrotactile information but that target capture times would decrease as learning progressed. We therefore examined target capture time across the different feedback conditions to verify that subjects did in fact attempt to integrate the vibrotactile feedback into their ongoing planning and control rather than ignore it. We found a marked increase in target capture time of ~2 s when supplemental sensory feedback was provided (Fig. 10: gray bars), suggesting that the integration of vibrotactile feedback into the ongoing planning and control of limb motion involved considerable cognitive processing relative to movements performed without that feedback. A comparison of initial training performance across days reveals that target capture time was longer on day 1 relative to day 2 (t₁₄ = 2.6205, P < 0.05). Although the magnitude of this decrease was quite small, the results add additional support to the conclusion that subjects did retain on day 2 some aspects of what they learned while training to use supplemental sensory feedback on day 1.

Fig. 10.

Group results for target capture time vs. blocks for both days. Error bars: 95% confidence interval; black bars: blocks with visual feedback (V₊). V₋ indicates blocks without visual feedback. V_INIT indicates blocks wherein the cursor was provided briefly when the cursor was inside the starting target but disappeared when the new target appeared. Gray bars: blocks with vibrotactile feedback (T₊); open bars: blocks without vibrotactile feedback (T₋); dashed bar: sham feedback (T_SHAM). Horizontal dashed line: visual guideline for comparing initial training blocks across days.

We observed no difference in target capture time between the sham condition and other blocks with vibration. This outcome suggests that although sham-block performance degraded dramatically relative to trials when informative vibration was provided, subjects did in fact attempt to use the sham feedback to drive performance and did not simply ignore it.

DISCUSSION

This study tested the application of vibrotactile stimulation as supplemental state feedback for enhancing the ongoing control of a moving limb. Subjects wore a two-dimensional vibrotactile display on their nondominant arm while performing horizontal planar reaching movements with their dominant arm. We constructed a mapping between the position of the moving hand and its representation within the vibrotactile display such that small, goal-directed hand displacements were more easily discriminable using supplemental vibrotactile feedback than with intrinsic proprioceptive feedback in the moving arm (cf. Fuentes and Bastian 2010; Shah et al. 2016). After mere minutes of training, subjects were able to use the information encoded within the vibrotactile feedback to perform blind reaches (i.e., to capture the visual target without concurrent cursor feedback of hand position). Subjects performed with levels of accuracy and precision that exceeded levels attained without the supplemental feedback (i.e., using intrinsic proprioceptive feedback alone). Reach accuracy continued to improve within and across 2 days of training with additional practice on using supplemental vibrotactile feedback. These improvements were due to partial resolution of distortions in the internal representation of reachable space (i.e., proprioceptive drift and task space expansion) that spontaneously arose when concurrent cursor feedback of hand motion was eliminated. We also observed a beneficial aftereffect of vibrotactile feedback training (a reduction of task space expansion) when the vibrotactile feedback was turned off; this observation was consistent with the hypothesis that supplemental vibrotactile limb state feedback training can improve internal representations of extrapersonal space. Finally, the utility of vibrotactile feedback was not limited to the trained workspace; subjects immediately capitalized on the information encoded within the vibrotactile feedback to perform blind reaches in an unpracticed region of the arm’s workspace.

In both workspaces, all subjects immediately adopted a movement strategy by which they attempted to capture targets by moving first along one cardinal axis of the vibrotactile display and then along the other (i.e., a “decomposition” strategy). Kinematic consequences of decomposition included elevated levels of hand path curvature and bimodal hand speed profiles after training. The decomposition strategy was persistent; 13 of 15 subjects continued to decompose their movements after 2 days of training. Thus, this limited amount of training did not suffice to allow subjects to integrate supplemental limb state feedback into the planning and control of goal-directed reaching in a way that would produce movements having straight hand paths and unimodal hand speed profiles (cf. Morasso 1981). The fact that posttraining reaches reflected the decomposition strategy even when vibrotactile feedback was turned off indicates that at least part of the persistence was due to altered movement plans and not just a change in the way the brain uses concurrent sensory information to guide the ongoing feedback control of movement. Future studies are warranted to identify training programs that will encourage integration of vibrotactile limb state feedback into the planning and control of goal-directed movements with straight hand paths and unimodal hand speed profiles. The results of such studies will have practical implications for the augmentation of human performance in a variety of applications where precision and efficiency of manual movements is desired, including rehabilitation of functional movement after stroke and the enhancement of skilled manual performance in teleoperation tasks such as robotic surgery.

Vibratory feedback of limb state enhances the online control of movement beyond limits imposed by intrinsic proprioception.

Subjects could reach to each of the visual targets with accuracy and precision when cursor feedback of hand position was provided, but dramatic drift of reach end points toward the midline and expansion of task space along both planar dimensions of task space occurred rapidly after cursor feedback was extinguished in the first baseline block of trials. These reach errors reflect distortions of an internal representation of reachable space (i.e., task space and/or body image) that evolve within a time frame of seconds. Such distortions could arise, for example, because of biases in the internal representation of limb configuration caused by the recent history of movement in the left or right workspace (Ghilardi et al. 1995) or by accumulating errors caused by noise inherent to the sensory and motor signals used for updating estimates of limb position (Wolpert et al. 1998; for additional discussion, see Brown et al. 2003). Remarkably, all subjects were able to reduce these distortions after mere minutes of prior exposure to the vibrotactile display and the information it encoded (see Fig. 5: day 1, training block 1). To do so, subjects had to integrate a completely novel set of limb-state-dependent stimuli into their plans for movement and its ongoing control. A previous study from Dupin and colleagues (2015) also showed that people can successfully combine tactile information from a nonmoving limb with proprioceptive information from a moving limb into a single coherent percept of limb motion. Within the context of our study, the phenomenon of bilateral integration of movement cues seems to apply also for vibrotactile information supplementing proprioceptive information to improve goal-directed reaching.

Although target capture errors decreased further with practice both within and across 2 days of training, target capture times did not decrease meaningfully over the 2 days, suggesting that this limited training was insufficient for subjects to transition far beyond the initial cognitive stage of sensorimotor skill learning and into the associative phase (cf. Fitts and Posner 1967). These two phases rely on the so-called “body image,” an internal representation of the body that is accessible to consciousness (see Proske and Gandevia 2012 for a description and discussion of the concept of body image). Further studies should explore how extended and structured training schedules can impact the rate at which integration of vibrotactile limb state feedback into the ongoing control of goal-directed movements progresses through the later, “automatic” phase of sensorimotor skill learning. The later phases of skill learning involve a separate internal representation of the body (the body schema) that works independently of consciousness (cf. Proske and Gandevia 2012).

Two novel aspects of our experimental approach likely impacted the rapid learning observed on day 1 of our study (i.e., reduction of drift and reduction in the expansion of end points, as seen in Figs. 4, 6, and 7). First, the lateral displacement of the physical workspaces from the visual workspace probably accentuated the evolution of hand position drift toward the midline in the absence of both visual and vibrotactile feedback (Brown et al. 2003; Ghilardi et al. 1995; Wolpert et al. 1998). Second and more importantly, vibrotactile feedback encoded limb state information in our study rather than performance error information, as used by most previous studies of sensory augmentation via vibrotactile stimulation (e.g., Bark et al. 2015; Cuppone et al. 2016). Because we used a state encoding for the vibrotactile feedback, it was critical that training should establish a consistent spatial correspondence between perceived hand position and the actual hand position encoded by the vibrotactile feedback. Whereas limb state encoding preserves a bijective (one-to-one) relationship between hand position in the workspace and the stimulus presented by the vibrotactile display, error encoding does not. In the present study, we carefully designed the training condition to minimize “proprioceptive drift” (Wann and Ibrahim 1992) such that initial movement trials could establish cross-modal spatial links between unimodal representations of initial or desired hand positions (cf. Deneve and Pouget 2004) and later trials could reinforce those relationships. By contrast, we would not expect that encodings of task performance error would induce persistent changes in internal representation(s) of reachable space because the relationships between visual, proprioceptive, and vibrotactile signals would reset each time the initial starting conditions and/or desired goal targets change.

In any case, beneficial effects of vibrotactile limb state feedback reflect its partial integration into the online control of movement because some benefits were seen only while the vibrotactile signals were present; upon removal of the vibrotactile signals on day 1, target capture errors increased, as did the amount of drift and the area spanned by the reach end points. The ability to integrate vibrotactile limb state feedback into ongoing control of movement was not limited to the trained workspace because the effects of vibration were reported also in the untrained contralateral workspace. When subjects were provided with concurrent vibrotactile limb state feedback on day 2, target capture errors again decreased, as did directional and extent errors and the phenomenon of spatial expansion. Because performances degraded during the last generalization baseline in the untrained workspace on day 2, we reject the possibility that the learning effect experienced with concurrent vibration feedback was due merely to more prolonged practice on the reaching task.

Vibrotactile feedback training elicits adaptive changes in motor plans.

Vibrotactile limb state feedback training resulted not only in short-term improvements in the internal representation of extrapersonal space, but also lasting changes in motor plans for reaching. When subjects were not provided visual or supplemental feedback about their movements, they made large movements that yielded large target capture errors and an expansion of the workspace spanned by the hand at the end of the reaches. After training, in the same reduced feedback conditions (i.e., with only proprioceptive feedback available), subjects exhibited improved reach accuracy and reduced spatial expansion, indicating that the training had had a persistent and beneficial impact on the sensorimotor transformation(s) relating visual targets to desired reach end points. The finding that subjects also persisted in decomposing their movements along the cardinal axis of the vibrotactile display in the posttraining baseline blocks demonstrates that subjects applied to these movements the same movement strategy adopted when the vibrational feedback was provided.

Cuppone and colleagues (2016) have also found a persistent improvement of motor performance after a training based on concurrent vibrotactile and haptic feedback. In that study, groups of subjects grasped the handle of wrist manipulandum and were tested pre- and posttraining on their ability to perform a two-degree-of-freedom (DoF; flexion/extension, abduction/adduction) position matching task and a two-DoF target-tracking task. One group received small forces (haptic cues) that guided the hand to the target during training. Two additional groups received the haptic cues along with one-DoF vibrotactile cues related to the “lateral deviation” of the wrist from the straight line connecting a neutral home position to the desired two-DoF target. The two groups differed according to the delivery site of vibrotactile stimulation. A control group received no training. Relative to the untrained control group, proprioceptive acuity was improved after training only in the groups that received both the haptic and vibrotactile cues. Improvement did not depend on which limb received the vibrotactile cues. The groups that received vibrotactile cues exhibited improved movement kinematics during goal-directed wrist movements from the onset of training (i.e., reduced lateral deviations), although improvement came at the cost of increased movement times and an increased number of “movement units” (i.e., velocity profile peaks) required to capture the target. Because the haptic and vibrotactile cues were not tested in isolation but presented concurrently, Cuppone et al. (2016) could not determine whether haptic force cues or vibratory feedback of movement errors were necessary elements of the proprioceptive training they described.

The results of the current study support and extend the findings of Cuppone and colleagues (2016). Also in our case, observed improvements in accuracy and representation of the workspace could be due both to the vibrational feedback and the terminal haptic cues that we used to provide knowledge of results during training (i.e., robotic translation of the hand to the intended final target positions). However, in the present study, the motor plan adopted by the subjects after training changed profoundly, demonstrating clear carryover effects of movement strategies adopted while subjects attended to the vibrational feedback. We believe the decomposition strategy was motivated primarily by the vibrotactile limb state feedback for four reasons. First, the decomposition strategy arose during the part of the movement that was not impacted by the terminal haptic feedback. Second, submovements of the decomposition strategy were aligned primarily along the cardinal axes of the vibrotactile display. Third, terminal haptic feedback was provided only after subjects had used the vibrotactile feedback to correct for movement errors and had indicated that they had captured the target. Finally, the terminal haptic feedback corrected for target capture errors by driving the subject’s hand straight to the target, not along a decomposed path.

The finding that subjects used a stereotyped, strategic approach toward integrating vibrotactile feedback into the planning and execution of goal-directed reaches is interesting in itself. When performing goal-directed reaches, the hand typically follows a relatively straight path between its initial and final positions, with a unimodal hand speed profile that is often described as “bell-shaped” (Abend et al. 1982; Flash and Hogan 1985; Morasso 1981; Sergio and Scott 1998). Such point-to-point hand trajectories are planned to be straight in visually perceived space (Wolpert et al. 1995) and can become markedly curved either when a nonlinear transformation is interposed between the motion of the end effector (hand) and its visual representation (Flanagan and Rao 1995) or when accounting for the geometrical properties of the object that the brain is controlling (Danziger and Mussa-Ivaldi 2012). In our case, all 15 subjects employed the same, unusual decomposition strategy to capture targets in the absence of visual feedback and in the presence of informative vibrotactile feedback of hand position. Instead of treating vibrotactile feedback of hand position as a vector quantity, i.e., by simultaneously reducing target capture error along both axes of the vibrotactile display, subjects first reduced error along one dimension and then along the other. Most subjects self-reported at the conclusion of the experiments that they had attempted to “process one feedback dimension at a time” or that they had planned hand movements “so that only one [pair of tactors] would provide meaningful vibration at any given time.” People are better able to discriminate small differences in the intensity of two vibrotactile stimuli when the stimuli are presented sequentially in time rather than simultaneously, potentially reflecting a bottleneck in the parallel processing of vibrotactile stimuli (Shah et al. 2016, 2018). Because changes in vibrotactile intensity map directly onto changes in hand location in the current study, the avoidance of simultaneous activation of two tactors can be a viable strategy to maximize perceptual acuity and thus the precision of hand localization.

Two subjects explicitly reported that the subjacent gray grid underneath the targets was a useful cue to interpret the vibrotactile feedback, suggesting that they were able to “feel” their way along the grid to the desired target. The grid we display might have influenced the adoption and retention of the decomposition strategy, but it could not have been the main contributing factor because that strategy emerged only when vibrational feedback was provided. Moreover, movement decomposition was not merely a reflection of how the vibrotactile feedback was processed in real time, but also must have reflected, in part, a training-dependent alteration in the movement plans because decomposition persisted in posttraining reaches in both the trained and contralateral workspaces, whether supplemental vibrotactile feedback of hand position was currently available or not. Future studies should explore the factors motivating adoption and retention of a decomposition strategy in training with multi-DoF vibrotactile displays and identify training strategies to encourage the vectorial interpretation of the vibrotactile stimuli and the simultaneous reduction of target capture error along all axes of the vibrotactile display (e.g., the production of goal-directed movements with straight hand paths and unimodal hand speed profiles). Additionally, these studies should assess the utility of vibrotactile sensory substitution systems and appropriate training strategies to promote successful completion of typical daily tasks by suitable patient populations, such as survivors of stroke who have deficits of proprioceptive sensation and yet retain residual movement capability.

Limitations and future directions.

Our study design has limitations worth noting. The vibrotactile stimuli used in the current experiments spanned a range of frequencies (approximately 60–240 Hz) that can excite a variety of mechanoreceptors. These include hair follicle afferents and Meissner corpuscles, which respond to low-frequency vibrations (less than ~80 Hz; cf. Zachariah et al. 2001), as well as muscle spindle afferents, Pacinian corpuscles, and (in some testing conditions) Golgi tendon organs (cf. Fallon and Macefield 2007), which respond to higher frequency vibrations greater than ~60 Hz (see Mahns et al. 2006 for a review). Therefore, there is uncertainty regarding the relative contribution of each sensory channel to the perception of stimulus intensity as it relates to hand displacement in the vibrotactile display. This limitation of our study design does not negate the significance of our findings because regardless of which sensory channels were recruited, the vibrotactile display successfully allowed subjects to use supplemental kinesthetic feedback to improve the accuracy of goal-directed reaches in the absence of visual feedback.

Another study limitation is that we studied performance changes motivated by the application of supplemental vibrotactile stimulation only to the nonmoving arm. It is possible that other stimulation sites might yield better reach performances than those described here. For example, a reasonable critic could suggest that subjects might not invoke a decomposition strategy if the supplemental feedback were instead applied to the moving arm. It could also be argued that a decomposition strategy was encouraged by the intuitive alignment between hand displacement and stimulus encoding along the x- and y-axes of the vibrotactile display. If subjects bias their movement strategies based on the feedback axes, then the manipulation of the feedback encoding scheme (e.g., rotating the display axes relative to the axes of hand motion) could encourage decomposition along different principal directions than those described here or perhaps eliminate decomposition altogether. The extent to which alternative stimulus display configurations, information encoding schemes, and application sites can enhance limb control is a question worthy of future experimental exploration.

Although our results demonstrated beneficial effects of training with vibrotactile feedback on reach performance in the absence of concurrent visual feedback, it remains unclear what aspects of the experimental approach may have driven the observed sensorimotor and perceptual learning effects. Training block trials included initial visual cues on hand position, vibrotactile information throughout movement, and knowledge of results feedback from the robot on final position accuracy. It is not clear which of the three (or combination thereof) induced the observed performance improvements during training and which induced the perceptual learning that occurred between the pre- and posttraining baseline assessments. All three forms of information were necessary for subjects to learn a consistent mapping between the actual and perceived limb states. A future study should conduct the experiments needed to properly and separately test the impact of initial visual cues, vibrotactile information, and robotic knowledge of results of final position accuracy on sensorimotor and perceptual learning within the context of vibrotactile sensory augmentation. Such work could have practical significance in the development of therapeutic interventions for patients contending with somatosensory deficits that may arise, for example, after stroke.

GRANTS

This study was supported by the National Institute of Neurological Disorders and Stroke and the Eunice Kennedy Shriver National Institute of Child Health and Human Development of the National Institutes of Health under award numbers R15HD093086 and R01 HD053727; the National Science Foundation under an Individual Research and Development plan; a Whitaker International Program Grant; FP7-PEOPLE-2012-CIG-334201; the Erasmus+ KA 107 action (USA-ITALY); the Italian Ministry of Foreign Affairs, Unit for Scientific and Technological Cooperation; and the Marquette University Opus College of Engineering.

DISCLAIMERS

Any opinions, findings, conclusions, or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation or the National Institutes of Health.

DISCLOSURES

No conflicts of interest, financial or otherwise, are declared by the authors.

AUTHOR CONTRIBUTIONS

N.R., V.S., M.C., and R.A.S. conceived and designed research; N.R. performed experiments; N.R., V.S., L.A.M., M.C., and R.A.S. analyzed data; N.R., V.S., L.A.M., M.C., and R.A.S. interpreted results of experiments; N.R. and R.A.S. prepared figures; N.R., L.A.M., M.C., and R.A.S. drafted manuscript; N.R., V.S., L.A.M., M.C., and R.A.S. edited and revised manuscript; N.R., V.S., L.A.M., M.C., and R.A.S. approved final version of manuscript.

APPENDIX

With the use of the MATLAB script presented in Fig. A1, we simulated ideal, “minimum-jerk” hand trajectory (Flash and Hogan 1985) in the horizontal plane. As expected for a dimensionless performance measure, we found the value of the DI of Eq. 1 to be invariant across movement direction, movement extent, and movement speed for these straight-line hand movements. When we simulated movements with N = 16 movement directions, a movement extent of 10 cm, and a movement duration of 5 s, each of the simulated reaches yielded a DI value of 0.24. We also simulated movements having durations of 3 s; these also yielded DI values of 0.24. Identical values were found for movements in all directions when we increased or decreased movement length or varied the number of movements simulated. The only exceptions to these general observations were trajectories for which there was no motion along one or the other of the cardinal directions, in which case the DI of Eq. 1 becomes singular. Because real-world hand trajectories will in all likelihood have some motion along both the x- and y-axes, the practical utility of the DI for evaluating human performance is unconstrained by the exceptional cases.

Fig. A1.

MATLAB script for simulating minimum-jerk hand trajectories and computing resulting decomposition index (DI) values.