Object Manipulation Improvements due to Single Session Training Outweigh the Differences among Stimulation Sites during Vibrotactile Feedback

Abstract

Most hand prostheses do not provide intentional haptic feedback about movement performance; thus users must rely almost completely on visual feedback. This paper focuses on understanding the effects of learning and different stimulation sites when vibrotactile stimulation is used as the intentional haptic feedback. Eighteen unimpaired individuals participated in this study with a robotic interface to manipulate a virtual object with visual and vibrotactile feedback at four body sites (finger, arm, neck, and foot) presented in a random order. All participants showed improvements in object manipulation performance with the addition of vibrotactile feedback. Specifically, performance showed a strong learning effect across time, with learning transferring across different sites of vibrotactile stimulation. The effects of learning over the experiment overshadowed the effects of different stimulation sites. The addition of a cognitive task slowed participants and increased the subjective difficulty. User preference ratings showed no difference in their preference between vibrotactile stimulation sites. These findings indicate that the stimulation site may not be as critical as ensuring adequate training with vibrotactile feedback during object manipulation. Future research to identify improvements in vibrotactile-based feedback parameters with amputees is warranted.

Introduction

Although individual users may be able to gain some insight for limb control by using the vibrations caused by the prosthetic motor of hand prostheses as they engage and disengage, most hand prostheses do not typically provide intentional haptic (proprioceptive and/or cutaneous) feedback about movement performance. Instead, users must rely almost completely on visual feedback, which requires constant cognitive attention and loses accuracy in degraded visual environments such as dim lighting. The addition of haptic feedback to hand prostheses and relief from visual attention to perform functions have both been noted by users as top design priorities [1, 2]. The addition of haptic feedback to prostheses may encourage self-attribution of the body part [3], and based on studies in unimpaired individuals, may aid in object manipulation [e.g., 4, 5].

There is currently technology to detect contact force on prosthetic hands and fingers during object manipulation [6–12], with many groups successfully integrating these technologies into prosthetic hands, e.g., the cybernetic hand [8, 9]. However, these technologies have not been widely implemented in products. Although implementation may be impeded by cost and weight, the most significant barrier is in the method of providing sensory information to users of prosthetic hands. Sensory substitution refers to transformation of sensation across or within sensory systems. A variety of both invasive and non-invasive approaches of sensory substitution have been suggested, such as vibrotactile, electrocutaneous, acoustic, and direct nerve stimulation [13–20]. Since force feedback may aid in object manipulation [e.g., 4, 5], providing force feedback through non-invasive sensory substitution such as vibrotactile stimulation [7, 21–23] is a particularly inviting approach. The foremost attraction being that the non-invasive nature of this approach would allow for immediate wide-scale implementation among users of prosthetic hands [13, 14].

However, the effects of vibrotactile stimulation on object manipulation performance and speed have yet to be systematically studied. Previous work in this area has been crucial in providing directions for future research, but has also often been limited by small subject numbers, lack of objective goal performance measures (task performance with typical sensory feedback), and examination of limited areas of vibrotactile stimulation [7, 21, 23, 24]. Another hurdle to understanding the effects of vibrotactile stimulation on object manipulation is that some of these studies resulted in conflicting performance results.

Promising results were found by two groups [7, 23]. In a case study utilizing the Boston Arm in 1970, Mann & Reimers used vibrotactile stimulation on the participant’s stump to signal tactual display of limb angle, combined with force feedback at the stump during an arm positioning task [23]. The vibrotactile stimulation was logarithmically related to the angular position of the limb, and was found to improve the accuracy of the participant’s ability to position his arm. Pylatiuk et al. [7] asked 5 users of myoelectric prosthetic hands to perform a simple object grasp task both with and without vibrotactile feedback relating to the contact force with the object. The vibrotactile stimulation increased in both amplitude and stimulation frequency with increases in contact forces and was applied directly to the prosthesis or to the skin of the residual limb. Their study found decreases in contact forces when the vibrotactile feedback was available.

On the other hand, other groups have found vibrotactile feedback to be less effective [21, 24]. Patterson & Katz utilized N = 25 unimpaired participants broken into 5 groups [24]. Participants received feedback related to the force applied to a robot arm through 5 methods: 1) a pressure cuff placed on the upper arm, 2) vibrotactile stimulation of the upper arm, 3) vision, 4) pressure + vision, and 5) vibrotactile stimulation + vision. Participants performed gripping trials, which consisted of a reference grip in which they chose the force, followed by a replication grip in which they attempted to match the force from the previous grip. Results did not indicate that the group that received vibrotactile + vision resulted in decreased error relative to the group that received vision alone. More recently, Chatterjee et al. [21] presented 8 unimpaired individuals with vibrotactile stimulation on the upper arm during use of a myoelectric prosthesis simulator to complete an interactive force-matching task. The vibrotactile stimulation was in the form of a square wave (pulse train) of a 200-Hz carrier frequency. Increases in force resulted in increases in the square wave pulse rate. Use of this feedback did not result in a consistent overall reduction in force-matching error.

It is well-known that humans have less subjective capacity for vibrotactile detection in the non-glabrous (hairy) skin than in glabrous skin [25]. Further, there are stark differences among locations on hairy skin [26, 27]. All previous work examining the use of vibrotactile stimulation for motor control applied vibrotactile stimulation to the upper arm, forearm and/or limb stump. There is a need to study the potential differences among different stimulation sites to determine whether there is an optimal skin location to provide vibrotactile stimulation for object manipulation using a hand prosthesis. Although the skin of the hand is the most sensitive [27], it is also the area for which function has been lost. The skin of the upper arm is closest to the area of lost function. However, since vibrotactile stimulation differs from the physical quantity being represented (vibration versus force) it is possible that the learning necessary in the central nervous system to utilize this type of feedback is relatively unaffected by proximity, and that relative sensitivity to the stimulus is all that is necessary. If so, the anterior neck and palmar surface of the foot are two body locations with possible promise as stimulation sites in terms of both vibrotactile sensitivity and cosmesis. The palmar surface of the foot is glabrous skin, and has been shown to have lower vibrotactile detection thresholds than the upper arm [27, 28]. The foot also represents a promising site in terms of cosmesis. Vibrotactile stimulators could be easily administered and hidden using insoles [29]. Although the anterior neck has not been studied in terms of vibrotactile detection, the sternum has been shown to have relatively low thresholds for vibration detection compared to other body sites (although less sensitive than the foot, forearm, and finger, more sensitive than the upper arm) [27]. The neck also offers a relatively discreet option for stimulator placement, in which electronics could be easily hidden by a shirt collar. Both the foot and neck offer the added benefits of being sites that are less likely to be diseased in upper limb amputees, while possible nerve damage in the upper arm could decrease sensitivity to vibrotactile stimulation.

This paper focuses on understanding the effects of learning and different stimulation sites when vibrotactile stimulation is used as intentional haptic feedback. This work uses a robotic interface with which both visual and direct haptic feedback can be experimentally controlled [22] to study virtual object manipulation. Our group has shown using this environment and one stimulation location (upper arm) that vibrotactile feedback in addition to visual feedback improves virtual object manipulation ability over visual feedback alone within one training session [22]. For this paper, we specifically addressed the effect of vibrotactile feedback at four body sites (fingerpad, upper arm, anterior neck, and palmar surface of the foot) as a function of time and simultaneous cognitive load. We hypothesized that performance might differ among body sites, that it would be decreased during a simultaneous cognitive task, and that participants would show increases in performance as a function of time. Given the uncertainty of the results from the literature, we also hypothesized that the relative differences in body location may not be as important as the effects of an added cognitive load, or the effect of learning to use vibrotactile stimulation as the replacement of natural haptic feedback. We started with a large population study (N=18) of unimpaired participants to get a baseline, with the hope to conduct an analogous experiment with amputees in the future.

Methods

Participants

Participants were 18 right-handed adults (8 male, 10 female; mean age = 24.4 years, SD = 4.4 years). The individuals reported normal hand function, with no complaints related to their hands. Informed consent was obtained from all participants in compliance with the Institutional Review Board of the University of Washington.

Physical Setup

Participants were asked to complete an object manipulation task, in which they used their index finger. They were asked to apply appropriate normal force to a virtual object to allow for translation, and to drag it to a target at the right of the screen as quickly as possible without breaking it. This task was chosen based on the assumption that it would be easy to understand, functional, and relatively difficult to perform without sensory feedback. It was specifically inspired by the demonstrated difficulties of prosthetic hand users with appropriately applying normal force to delicate objects (e.g., picking up and manipulating a disposable plastic cup [30]).

Participants placed their right index finger into a custom splint attached to the end effector of a PHANTOM Premium 1.0 robotic device (Sensable Technologies, Inc., Woburn, MA) to interact with the virtual environment. The PHANTOM was used only to measure the three-dimensional positions of the tip of the index finger, and did not provide force feedback at any time during the experiment. A projection system consisted of a frame above the PHANTOM, which supported an inverted video monitor, positioned at 45° toward the participant. A mirror was placed between the virtual environment and the monitor to permit reflection of images from the monitor to the user (see Fig. 1).

Fig. 1.

Study Methodology Participants placed their right index finger into a custom splint attached to the end effector of the PHANTOM (Panel A). Panels C and D show a schematic and photo of the physical set-up. Panel B shows a screenshot of the virtual environment.

The virtual environment was programmed in C++, with graphics driven by OpenGL. It consisted of one of two possible virtual objects at the left end of the workspace (see Fig. 1). Both objects were cylinders oriented with the flat surface parallel to the ground plane, referred to as boxes. Each of the two boxes used had distinct stiffness characteristics, and the difference between the two boxes was signaled to the participant by their color (red and blue). The stiffness characteristics of each box were defined as two continuous piecewise functions of vertical displacement as indicated below:

$$\begin{array}{l}{F}_{\mathit{blue}}=\{\begin{array}{cc}(0.34\text{N}/\text{cm})x,& \text{if}x<1.7\text{cm}\\ (4.65\text{N}/{\text{cm}}^2)x^2-(14.33\text{N}/\text{cm})x+11.55,& \text{if}x>1.7\text{cm}\end{array}\\ F_{\mathit{red}}=\{\begin{array}{cc}(0.68\text{N}/\text{cm})x,& \text{if}x<1.7\text{cm}\\ (9.59\text{N}/{\text{cm}}^2)x^2-(30.67\text{N}/\text{cm})x+25.56,& \text{if}x>1.7\text{cm}\end{array}\end{array}$$

These functions are scaled versions of a fit to the force-displacement curve acquired during pushing on a disposable plastic cup. The virtual normal force of the blue box was F_blue (N) and the virtual normal force of the red box was F_red (N), where x is the displacement of the finger into the box, in centimeters. Both the linear and quadratic portions of the stiffness characteristics were decreased for the blue relative to the red box.

Momentum was not included in the system dynamics. The force required to overcome friction to translate the cylinder, F_move, was arbitrarily defined as 1.2 times the force at the displacement of 1.7cm, and the force threshold to “break” each cylinder, F_break, was defined as 0.75N greater than F_move. Thus, normal force applied to the cylinder between F_move and F_break would allow the participant to slide the object to a target located 30 cm to the right. The “move” and “break” thresholds were chosen empirically, in order to provide a task that was difficult but possible to learn. Due to the difference in stiffness between the two boxes, the 0.75N window for moving the box resulted in differing allowable displacements of the finger during motion (1.6 mm for the red box and 2.7 mm for the blue box), creating differing difficulty for the 2 boxes. While this task appeared difficult, it was designed to simulate daily object manipulation tasks that require close pressure control.

Visual feedback of the task consisted of a real time depiction of the location of the finger in the virtual environment (shown as a small sphere), and the current position of the box (see Fig. 1). Deformations of the box were not shown, and visual feedback of finger location was occluded during penetration of the box. The role of the visual display was to provide the user with gross measures of position such as those that would be obtainable during operation of a prosthetic device. In particular, the finger position on the screen was used by participants to assess horizontal position in the virtual environment.

Vibrotactile Stimulation

During interaction with the virtual environment, increases in force were translated to increases in amplitude of vibrotactile stimulation. Stimulation at 250 Hz was provided using a C2 tactor (Engineering Acoustics, Inc.) mounted to the skin and secured with an elasticized cloth bandage. A 250-Hz carrier frequency was used, because human glabrous skin has been shown to be maximally sensitive to vibrotactile stimulation at this frequency [31, 32]. This frequency is in a region of maximal sensitivity of two of the four psychophysical channels: P and NP II, associated with Pacinian Corpuscle (PC) and Slowly Adapting (SA) II afferent fibers [33]. Sites used for vibrotactile stimulation were the right index fingerpad, right lateral upper arm, right anterior neck, and palmar surface of the right foot (see Fig 2). The foot stimulation site was chosen on the arch (see Fig 2), since this area has been shown to have lower detection thresholds at 250 Hz than the lateral borders of the foot and heel and the toes [34]. The possible range of amplitude of vibrotactile stimulation used was the same for all participants and was approximately 0 - 400 μm. However, the different skin surfaces used in the study have varying degrees of compliance and stiffness, and these differences may have led to differences in the distance traveled by the tactor head as a function of body location. The mapping of virtual force to stimulation amplitude was linear, and all participants reported the ability to feel and perceive differences in amplitude of the stimulus at all stimulation sites.

Fig. 2.

Stimulation sites used for vibrotactile feedback.

Cognitive Load

An auditory 2-back test was used to provide a simultaneous cognitive task during testing. This test involved listening to random 16 digit strings and responding verbally by identifying any numbers repeated with only one intervening number [35]. Before the experiment in the virtual environment began, participants practiced 20 sets of this task to ensure that they understood and could complete the task. During trials with the virtual environment, participants were asked to complete the cognitive task while simultaneously completing the motor task. Each number string was of a specific finite length, whereas the length of each trial was dependent on task performance. Therefore, completion of the entire 16 digits of the cognitive task was not always achieved by participants in cases of relatively quick “breaking” of the box.

Experimental Protocol

Over approximately 1.5 – 3.5 hours (including breaks), participants completed 160 trials (40 at each of the 4 stimulation sites) of interaction with the virtual system. The presentation order of each of the 4 stimulation sites was randomized for each participant. Within the 40 trials for each stimulation site, trials were presented in 10 blocks, randomized within block by box (blue, red), and cognitive load (ON, OFF). During interaction, participants sat with their forearm resting on the front of the workspace, and their hand was free to move about the 3D workspace. Trials ended when the box reached the target or was broken. At the end of each trial, the participant was asked to report the difficulty of completing the motor task on an ordinal scale 1 – 5, in which 1 was very easy and 5 very difficult. Participants were required to take 5 minute breaks between each of the 4 stimulation sites, and were encouraged to take breaks between any trials to avoid fatigue. Participants generally took between 4 – 6 breaks during the experiment.

During all trials, participants wore noise-canceling headphones (Bose, Framingham, MA), which were used to present the stimuli for the cognitive task, and to provide low-level masking noise. Since the vibrotactile feedback was provided at 250Hz, which is in the range of human hearing, the masking noise and noise-canceling headphones were used to ensure that participants were not using any auditory feedback from the tactor to complete the motor task.

At the end of their participation, each participant was asked to rate their preference 1 – 5 for each stimulation site, in which 1 was least preferred and 5 was most preferred.

Analysis

Performance variables were box displacement (the total distance toward the target that participants were able to translate the box during the trial), average box velocity (box displacement normalized by trial duration), and difficulty ratings.

Data analysis to determine the performance variables for each trial was performed using custom software in MATLAB (Mathworks, Natick MA), and statistical analysis was performed using Minitab Statistical Software (Minitab Inc., State College, PA). A four factor repeated measures analysis of variance (ANOVA) was performed to assess the effects of stimulation site, cognitive task, presentation order, and box on the performance variables, with post hoc two-sided Tukey’s Simultaneous tests when appropriate. A one factor repeated measures ANOVA was performed to assess the effects of stimulation site on user preference ratings. The correlation between the average change in each performance variable between the last stimulation site tested (presentation order 4) and the first stimulation site tested (presentation order 1) for each participant and the cumulative time that participant spent interacting with the system was assessed using Pearson’s R.

Results

Overall, out of 2880 combined trials, participants were able to successfully move the box to the target 238 times (8% of attempts). During these successful attempts, the average distance achieved was the full range of the task (30 cm), and the average velocity was 0.67 cm/s (SE = 0.02 cm/s). Conversely, during unsuccessful attempts, the average distance achieved was 3.96 cm (SE = 0.12 cm) and the average velocity was 0.02 cm/s. Results of four factor (stimulation site, cognitive task, presentation order, and box) repeated measures ANOVA on the three performance variables are shown in Tables 1 – 3. Main effects of each of the four factors used in the ANOVAs on the three performance variables are shown in Fig. 3. Selected factor interactions of interest are highlighted in Fig. 4: the influence of presentation order and stimulation site on the 3 performance variables, and the influence of cognitive task with stimulation site on box velocity.

TABLE 3.

Repeated Measures ANOVA for Difficulty Ratings

Source	DF	F	p
Box	1	445.09	<0.001
Cognitive Task	1	5.10	0.024
Stimulation Site	3	6.07	<0.001
Presentation Order	3	3.03	0.028
Box × Cognitive Task	1	0.14	0.712
Box × Stimulation Site	3	2.06	0.104
Box × Presentation Order	3	4.91	0.002
Cognitive Task × Stimulation Site	3	0.32	0.810
Cognitive Task × Presentation Order	3	0.07	0.977
Stimulation Site × Presentation Order	9	6.13	<0.001

Fig. 3.

Main effects of each of the four factors used in ANOVAs on the performance variables: presentation order, stimulation site, cognitive task, and box. Markers show means +/− SE. Brackets indicate statistically significant differences found during post hoc testing.

Fig. 4.

Selected factor interactions: A statistically significant interaction between stimulation site and presentation order was found for box displacement and difficulty ratings, but not for box velocity. Additionally, a statistically significant interaction between cognitive task and stimulation site was found for box velocity. Markers show means +/− SE.

Based on the results of the ANOVA on box displacement (see Fig. 3), two-sided Tukey’s Simultaneous tests were calculated to test the effects of stimulation site, presentation order, and box. The box was displaced significantly (p_adj < 0.05) further during interaction with the blue box (M = 8.74 cm, SE = 0.27) relative to the red box (M = 3.46 cm, SE = 0.17). The box was displaced significantly (p_adj < 0.05) further during stimulation at the arm site (M = 6.38 cm, SE = 0.35) relative to the finger (M = 6.00 cm, SE = 0.33) and foot (M = 5.58 cm, SE = 0.32) sites. Finally, box displacement was significantly (p_adj < 0.05) increased with each increase in presentation order, with the exception of the 4^th site relative to the 3^rd site (M_{site 1} = 3.89 cm, SE = 0.28; M_{site 2} = 5.56 cm, SE = 0.31; M_{site 3} = 7.08 cm, SE = 0.36; M_{site 4} = 7.88 cm, SE = 0.38).

Based on the results of the ANOVA on box velocity (see Fig. 3), two-sided Tukey’s Simultaneous tests were calculated to test the effects of stimulation site, cognitive task, presentation order, and box. Box velocity was significantly (p_adj < 0.05) increased during interaction with the blue box (M = 0.34 cm/s, SE = 0.0077) relative to the red box (M = 0.15 cm/s, SE = 0.0048). Box velocity was significantly (p_adj < 0.05) decreased during the cognitive task (M_off = 0.27 cm/s, SE = 0.0078; M_on = 0.22 cm/s, SE = 0.0058). Lastly, box velocity was significantly (p_adj < 0.05) increased during stimulation at the foot site (M = 0.25 cm/s, SE = 0.0112) and neck site (M = 0.27 cm/s, SE = 0.0102) relative to the finger (M = 0.24 cm/s, SE = 0.0088) site, and significantly (p_adj < 0.05) increased with each increase in presentation order, with the exception of the 4^th site relative to the 3^rd site (M_{site 1} = 0.17 cm/s, SE = 0.0080; M_{site 2} = 0.23 cm/s, SE = 0.0084; M_{site 3} = 0.29 cm/s, SE = 0.011; M_{site 4} = 0.30 cm/s, SE = 0.011).

Based on the results of the ANOVA on difficulty ratings (see Fig. 3), two-sided Tukey’s Simultaneous tests were calculated to test the effects of stimulation site, cognitive task, presentation order, and box. Difficulty ratings (1 – 5) were significantly (p_adj < 0.05) increased during interaction with the red box (M = 3.81, SE = 0.022) relative to the blue box (M = 3.24, SE = 0.026). Difficulty ratings were significantly (p_adj < 0.05) increased during the cognitive task (M_off = 3.50, SE = 0.026; M_on = 3.56, SE = 0.025). Finally, difficulty ratings were significantly (p_adj < 0.05) increased during stimulation at the finger site (M = 3.62, SE = 0.038) relative to the arm (M = 3.49, SE = 0.038) site, and were significantly (p_adj < 0.05) increased during the first presentation site (M = 3.59, SE = 0.034) relative to the second presentation site (M = 3.44, SE = 0.035).

The mean user preference rating for the foot (M = 2.94, SE = 0.29) was slightly lower than for the other three stimulation sites (M_arm = 3.11, SE = 0.24; M_finger = 3.06, SE = 0.34; M_neck = 3.33, SE = 0.30); however, a one factor repeated measures ANOVA found no effect of stimulation site on user preference ratings (DF = 3, F = 0.28, p = 0.842).

The correlation between the cumulative time each participant spent interacting with the system and the average change (presentation order 4 - presentation order 1) in each performance variable for each participant was: R = 0.36 for box displacement, R = 0.69 for box velocity, and R = −.27 for difficulty ratings.

Discussion

Effects of training

Apart from the difference between the difficulty of the two boxes, the most significant factor on performance variables was the presentation order (eg., training time). Performance results improved over the first 120 trials (3 body locations), with generally sustained performance from location 3 to location 4 (see Fig. 3). The average cumulative time spent actively engaging with the virtual environment (discarding breaks) at the end of first 120 trials was 52.0 minutes (SE = 2.6). Interestingly, correlations were seen between the total cumulative time each participant spent interacting with the system and the average change of the box displacement and velocity (presentation order 4 - presentation order 1) for each participant. In essence, the more time participants spent interacting with the device, the faster they were able to eventually learn to move the box, and to a lesser degree the further they were able to move it. Thus, additional training helps individuals use the vibrotactile feedback to become faster at the task.

The training in this study was performed across body locations, requiring the individuals to leverage learning to use a sensation at one body site for later performance at another site. It has been shown that humans plan movement based on multi-sensory integration, such as visual and haptic feedback [e.g., 36, 37], and that cross-modal transfer (CMT) can occur during motor learning [38]. CMT has been studied extensively in animal models and is defined as skills acquired during learning of a motor task with specific stimulus information being transferred to a new stimulus condition [e.g., 39, 40]. Kehoe and Napier found that, in rabbits, CMT increases when the two stimuli share the same temporal pattern [39]. CMT has been studied in humans by Erni and Dietz, who found significant CMT between auditory to somatosensory feedback conditions during a lower limb motor learning task [38]. Although the site of stimulation in the present work changed over time, individuals were able to transfer learning achieved at one site, and use it to achieve increased performance at a subsequent site (see effects of presentation order in Fig. 3). This may be due to the fact that the nature of stimulation and the specific temporal pattern were not changed as a function of time. Our future work will examine whether training with visual + vibrotactile stimulation can transfer skill to performing the motor task with vision alone.

If vibrotactile-based feedback were provided for users of prosthetic hands, the potential for sustained learning may be even greater than in the single sessions reported on in this study. Apart from issues of skill transfer across stimulation sites, this experiment was performed in a single sitting. It is unclear whether the apparent saturation in performance between presentation orders 3 and 4 is a result of a true maximum in performance, or the effect of fatigue and boredom from 1.5 – 3.5 hours of experimentation. Training using a cross-modality sensory substitution paradigm (electrotactile stimulation for visual perception) has shown improvements in perceptual task performance and changes in brain activation with 7 hours of training applied over 7 days [41]. We plan to examine the effects of training across multiple days at a single body location in the future; however, the current results are encouraging for the utilization of vibrotactile feedback for long-term use.

Effects of stimulation site

Overall, although significant differences were seen in some performance variables as a function of the site of stimulation, differences between sites were small when compared to differences as a function of order of presentation, and were not consistent between performance variables. This suggests that stimulation on the arm or even the finger may offer no substantial benefit relative to the neck and foot. Further, user preference ratings showed no difference in their preference between sites. If amputees show the same results found here in unimpaired participants, discreet body locations such as the foot and neck can be used for vibrotactile feedback.

It is interesting that the finger location did not result in better overall performance than the other locations. The data indicate that when individuals first begin the task, the finger location may offer an initial advantage compared to the other sites, but that this advantage dissipates when users have time to learn to interpret the sensory feedback. Further, there was a significant interaction effect between stimulation site and the addition of a cognitive task on the box velocity (see Fig. 4). This interaction indicates that performance during stimulation on the finger site was the least affected by a simultaneous cognitive load. One possible interpretation of this finding is that stimulation of the finger may offer an initial advantage in the face of cognitive load due to the naturalness provided by the proximity between the substituted feedback and the original feedback source. This interpretation matches well with the finding that when individuals first begin the task, the finger location may offer an initial advantage compared to other, more remote sites. These findings are similar to the previous work indicating that more natural haptic feedback during robotic surgery leads to shorter task times than feedback supplied through sensory substitution (augmented visual feedback about force interactions), due to the increase in the necessary cognitive processing by users [42]. On the other hand, the smaller effect of cognitive load for the finger location could also be a result of the increased sensitivity of the index finger relative to the other body sites [27]. Future studies comparing stimulation of the left versus the right finger could elucidate further the effects of sensitivity versus proximity. Another issue in comparing performance of the finger site to other body locations is the relative curvature of the skin. The curvature of the finger caused a smaller area of contact between the vibrotactile stimulator and the skin, which may have resulted in a reduction in the perceptual dynamic range and thus decreased performance from what could have otherwise been achieved.

Task completion versus speed, and the effects of cognitive load

Although the addition of a cognitive task did not have an effect on box displacements, it led to decreased velocities and increased difficulty ratings, agreeing with previous work [22]. Although box velocity using vision and the vibrotactile feedback increased significantly with training, the velocities achieved were much slower than those previously reported using this platform with vision and haptic (force) feedback provided by the PHANTOM directly to the finger. For instance, the average box velocity seen previously during visual and direct haptic feedback for the two boxes reported on here was 1.82 cm/s (SE = 0.078 cm/s), whereas the average velocity seen during the final quarter of experimentation in this study using visual and vibrotactile feedback was 0.30 cm/s (SE = 0.010 cm/s). It has been suggested that in addition to the increased naturalness of direct haptic feedback, that it provides physical constraints that could aid in task performance relative to force feedback provided via sensory substitution [43]. In addition, even with direct haptic feedback, only highly trained individuals improve performance without a significant increase in trial time [44]. It is possible that more long-term training is necessary to enable more “typical” speeds using this feedback, especially given that the current study found that increased training across a single session helped individuals to use the vibrotactile feedback to become faster at the task.

Conclusions

Based on the previous work in our group [22, 45], the promise indicated by the findings of this study is tempered by the relatively poor performance using vibrotactile feedback to interact with this platform when compared with individuals using direct haptic feedback. Specifically, using vision and force feedback applied directly to the finger by the PHANTOM, users are able to perform this task with box displacements that are over twice as large and box velocities that are over 5 times those seen during the final quarter of experimentation (presentation order 4) in this study using vibrotactile feedback [45]. Conversely, during the final quarter of experimentation in this study using vibrotactile feedback, users were able to perform this task with box displacements over 3 times as large and box velocities over 1.5 times those reported previously using vision alone with an identical experimental paradigm [22, 45]. Training time was a critical factor on performance variables, with improvement throughout experimentation. The finding that additional training helps individuals use the vibrotactile feedback to become faster at the task suggests that differences in the improvement using vibrotactile feedback between the current study and previous work may be a result of training time. In the case of Chatterjee et al., an additional difference was the vibrotactile stimulation paradigm: their work used pulse train frequency modulation rather than amplitude modulation to convey motor changes [21]. Although there is promise in this approach when compared to a lack of non-visual sensory feedback, future research to identify improvements in vibrotactile-based feedback parameters is warranted.

In addition, the current study examined the ability of young unimpaired individuals (mean age: 24.4 years) to use vibrotactile-based feedback to perform object manipulation. However, the most common causes of upper limb amputation are trauma and peripheral vascular disease, with the risk of amputation due to either cause increasing steadily with age [46]. Thus, the integrity of vibrotactile sensitivity in the elderly population is necessary for the viability of the use of vibrotactile-based feedback. Unfortunately, previous work has shown an increase in vibrotactile detection thresholds as a function of age [26, 34, 47, 48]. However, the present work did not see a correlation between the ability of participants to use vibrotactile stimulation to perform the object manipulation task and the relative sensitivity of the site. For instance, although the finger is known to have the lowest vibrotactile detection thresholds, participants were not significantly better at performing the task using stimulation on the finger. This suggests that some reduction in vibrotactile sensitivity in the elderly population may not impede their ability to use vibrotactile-based feedback for object manipulation. Another possible issue for the viability of vibrotactile stimulation for long-term sensory substitution is the potential for habituation to the stimulus. Habituation to vibrotactile stimuli can be both centrally-mediated and a result of sensory peripheral adaptation [49]. Adaptation of sensory afferents has been shown to occur during continuous vibrotactile stimulation with time constants ranging from 10 – 40 s, with recovery time constants ranging from 10 – 30 s [49]. The paradigm of encoding contact force as vibrotactile stimulation means that stimulation is only given during contact, leaving time for subsequent recovery after habituation. Further, the results of the current study do not show evidence of desensitization, with increases in box displacement and velocity occurring throughout experimentation (see Fig. 3). However, future work to study the long-term habituation to vibrotactile stimulation for object manipulation as well as to compare the ability of elderly individuals and individuals with a history of limb amputation to integrate vibrotactile stimulation is necessary to more accurately determine the feasibility of vibrotactile-based feedback for users of prosthetic hands.

Overall, visual + vibrotactile feedback at the four body sites showed a strong learning effect within a single session. The effects of stimulation site were less dramatic, and user preference ratings showed no difference between sites. These findings are pragmatically promising, indicating that this method of feedback may be utilized even in cases where there is residual damage to the upper arm after amputation, and that individuals may be able to increase their ability to use vibrotactile feedback considerably with training. However, future research is necessary to identify improvements in information encoding of object manipulation using vibrotactile-based feedback.

TABLE I.

Repeated Measures ANOVA for Box Displacement

Source	DF	F	p
Box	1	428.26	<0.001
Cognitive Task	1	0.11	0.745
Stimulation Site	3	2.85	0.036
Presentation Order	3	45.12	<0.001
Box × Cognitive Task	1	0.04	0.841
Box × Stimulation Site	3	2.61	0.050
Box × Presentation Order	3	17.21	<0.001
Cognitive Task × Stimulation Site	3	0.68	0.562
Cognitive Task × Presentation Order	3	0.03	0.994
Stimulation Site × Presentation Order	9	3.47	<0.001

TABLE 2.

Repeated Measures ANOVA for Box Velocity

Source	DF	F	p
Box	1	671.39	<0.001
Cognitive Task	1	47.86	<0.001
Stimulation Site	3	3.80	0.010
Presentation Order	3	67.44	<0.001
Box × Cognitive Task	1	2.47	0.116
Box × Stimulation Site	3	1.00	0.393
Box × Presentation Order	3	11.02	<0.001
Cognitive Task × Stimulation Site	3	3.11	0.026
Cognitive Task × Presentation Order	3	1.58	0.191
Stimulation Site × Presentation Order	9	1.15	0.324