Object Shape and Surface Topology Recognition using Tactile Feedback Evoked through Transcutaneous Nerve Stimulation

Abstract

Tactile feedback is critical for distinguishing different object properties. In this study, we determined if tactile feedback evoked by transcutaneous nerve stimulation can be used to detect objects of different shape and surface topology. To evoke tactile sensation at different fingers, a 2×8 electrode grid was placed along the subject’s upper arm, and two concurrent electrical stimulation trains targeted the median and ulnar nerve bundles, which evoked individually modulated sensations at different fingers. Fingertip forces of the prosthetic hand were transformed to stimulation current amplitude. Object shape was encoded based on finger-object contact timing. Surface topology represented by ridge height and spacing was encoded through current amplitude and stimulation time interval, respectively. The elicited sensation allowed subjects to determine object shape with success rates > 84%. Surface topology recognition resulted in success rates > 81%. Our findings suggest that tactile feedback evoked from transcutaneous nerve stimulation allows the recognition of object shape and surface topology. The ability to recognize these properties may help improve object manipulation and promote fine control of a prosthetic hand.

I. Introduction

Touch sensation is essential for object interaction and understanding of our surroundings. It promotes essential object manipulation [1], and allows us to detect object physical properties without the need for visual/auditory feedback [2], [3]. A lack of tactile feedback, as in upper limb amputees, limits the performance in the control of assistive devices [4], [5]. As prosthetic limbs advance to replicate the motions of the human hand, it is essential to provide tactile feedback to the user to reduce future device abandonment [6], [7]. Improvements in object manipulation or identification have been reported when feedback describing joint angle and/or grasp force are delivered to prosthesis users [8]–[11].

Neurologically intact individuals use a series of mechanoreceptors embedded in our skin for tactile sensing [12]–[15]. Stimuli varying in location, frequency, and intensity transmit different tactile information by recruiting different types of mechanoreceptors [13]. During object shape recognition using multi-finger grasps, the timing of sensation between fingers can be used. For example, with an identical grasp pattern, a cylindrical object will lead to concurrent tactile feedback across fingers, and a spherical object will lead to a timing difference in sensation across fingers due to different timing of object contact across fingers. Object surface topology can be detected based on the sensation intensity and especially the variation of intensity in tactile feedback as a finger runs across a surface.

Similarly, object properties can be encoded through sensory feedback delivered artificially. Previous studies have used mechanotactile or electrotactile stimuli for object recognition [9], [16]–[19]. In order to provide somatotopically matched feedback, a recent work [19] elicited tactile sensation using peripheral nerve stimulation via an intrafascicular electrode, and objects of varying shape can be discriminated by varying the timing of sensation between fingers. Texture or surface topology recognition has also been performed using invasive intraneural stimulation [20], [21]. Although promising, these invasive procedures can only be tested on a small number of cases, and wide clinical applications are limited due to the required surgery procedure and long-term care.

Accordingly, the purpose of this study was to determine if shape and surface topology recognition could be performed using transcutaneous electrical stimulation of peripheral nerves. Using a 2×8 electrode array placed along the subjects’ upper arm, current pulses can be delivered to different electrode pairs to activate selective sets of axons in the median and ulnar nerves [22]–[25]. Distinct axon recruitment produces spatially distinct sensation on the palm of the hand. Spatially separated sensations from different electrode pairs can also be directly summated using concurrent multi-channel stimulation, forming complex sensation patterns [23]. Building on this approach, two concurrent stimulation trains were used to target the median and ulnar nerve bundles, which generated individually modulated sensation at different fingers. The prosthetic hand interacted with different objects, and the fingertip forces were translated to stimulation amplitudes that induced tactile sensation to different fingers. The object shape was encoded by a difference in the onset timing of sensation between fingers. Surface topology, represented by a range of ridge height and ridge spacing, was encoded by stimulation peak intensity (height) and time interval between stimulation peaks (spacing), respectively.

II. Methods

A. Subjects

This study recruited ten neurologically intact subjects (9 Male, 1 Female, 21–37 years of age). Before the experiment each subject gave informed consent via protocols approved by the Institutional Review Board of the University of North Carolina at Chapel Hill.

B. Experimental Setup

Subjects were seated with one arm comfortably placed on a table. Alcohol pads were used to clean the skin surface prior to electrode placement. A 2×8 electrode grid was placed collaterally to the vector connecting the center of the axilla and the medial epicondyle of the humerus (Fig. 1C) in order to maximize the superficial access to the median and ulnar nerves. Sensation from the palmar side of the hand is transmitted through these afferent pathways with the median nerve innervating the index, middle, and a portion of the ring fingers, while the ulnar nerve innervates the pinky and the remainder of the ring fingers. To ensure secure electrode-skin contact, mild inward pressure was applied using a custom vice. Subjects were instructed to report any discomfort throughout the experiment. The electrode grid allowed for the delivery of stimulus to different pairs resulting in unique electric field generation, and in turn the activation of different sets of axons innervating different regions of the hand.

Fig. 1.

Diagram of obtaining the fingertip forces from a sensorized prosthetic hand (A) and transforming the forces using user- and finger-specific sigmoid functions to current amplitude. Individually and concurrently modulated stimulation trains to evoke sensation (B). Example sensations are shown. Stimulation diagram to elicit haptic sensation using a 2×8 electrode grid (C).

Electrode pair selection was conducted using a custom MATLAB (v2016b, MathWorks Inc, Natick, MA) interface that controlled a switch matrix (Agilent Technologies, Santa Clara, CA). The matrix linked one of sixteen Ag/AgCl gel-based electrodes (1 cm in diameter) among a 2×8 grid to either the anode or the cathode of a stimulator.

Single- and dual-channel stimulation was delivered using a multi-channel stimulator (STG4008, Multichannel Systems, Reutlingen, Germany). Charge-balanced biphasic square wave stimuli (Fig. 1B) were delivered. Although pulse width, frequency, and dual stimulation delay were adjustable, these parameters were fixed at 200 μs, 150 Hz, and 3.33 ms, respectively, based on previous studies [22], [23].

Current amplitude was determined through a mapping [25] based on one axial force (orthogonal to contact surface) calculated from internal torque readings of a sensorized prosthetic hand (The LUKE DEKA RC ARM, Modius Bionics, Manchester, New Hampshire). Unprocessed fingertip forces from the prosthetic hand were converted to a stimulation current amplitude using a finger-specific and subject-specific sigmoid function [25]. Sigmoidal function parameters were determined using the stimulation current range for a given electrode pair, the minimum and maximum finger forces, and the steepness of the function. For a given electrode pair, the sensory threshold and just below the motor threshold were employed as the stimulation range. The sensory and motor thresholds were identified by altering the stimulation amplitude in steps of 0.1 mA until finger sensation or finger motion occurred, respectively. This process was repeated three times, and the average values were calculated to determine each threshold. The steepness (1) and the minimum (0.5 N) and maximum force (6 N) were kept consistent across subjects and fingers. This range of finger forces is commonly used in activities of daily living [26]. The sigmoid function is shown in Equation 1, where I, I_Max, and I_Min represent the actual current, maximum current (i.e. motor threshold), and minimum current (i.e. sensory threshold), respectively. Steepness, actual force, maximum force, and minimum force were represented by k, F, F_Max, and F_Min, respectively.

$$I(x)=\frac{\left(I_{Max}-I_{Min}\right)}{1+e^{\left(-k\ast \left(F-\frac{F_{Max}+F_{Min}}{2}\right)\right)}}+I_{Min}$$ (1)

In order to create a double-blinded experiment, prosthesis-object interaction forces were pre-recorded from the prosthetic hand’s index and middle finger sensors (Fig. 1). Only the index and middle fingers were selected due to inconsistent force recordings from the ring and pinky during preliminary testing and prior work [25]. The saved force traces were drawn randomly from a pool during the actual study. Prosthesis-object interactions for shape recognition involved the grasping of two dimensionally similar shapes at various closing speeds. During each grasp, the experimenter programmed fingers of the hand to close on a cube or sphere (Fig. 2E) at one of the four specified closing speeds (8, 13, 20, or 40 degrees per second). The closing speeds were determined based on the fingers to flex 40 degrees in 1, 2, 3, or 5 seconds. Different objects could be encoded by the different contact onset timing between index and middle fingers. Force recordings were repeated 5 times for each combination of shape and speed to account for any variability in the system. Representative force trace examples from different combinations of shape and hand closing speeds and their average onset delays are shown in Fig. 2.

Fig. 2.

Force traces recorded when grasping the cube and sphere (E) with the prosthetic’s index (blue) and middle fingers (red). Forces were recorded when the hand closing speed was set to 8 (A), 13 (B), 20 (C), and 40 (D) degrees per second, respectively. The bar graph shows the average onset delay between the middle/index finger contact with the object (F). Error bars represent standard deviations (std).

To pre-record the interaction force for surface topology, only the index finger was used. Each surface was slide across the fingertip at approximately 10 cm/s, or 1 Hz using a metronome. Surfaces were varied in ridge height and ridge spacing. Ridge height recognition was evaluated using surfaces with 5 ridges (2 cm apart) at different height (1, 2, or 3 mm). Ridge spacing recognition was tested using 2 mm ridges at different spacing (2, 1.5, or 1 cm) over a 10 cm span. Lastly, combined ridge configuration was tested using 4 distinct surfaces: two ridge heights (2 or 3 mm) and two ridge spacings (2 or 1 cm). Combinations of ridge height and spacing were selected based on preliminary testing regarding the sensitivity of the prosthetic hand’s sensors. Force trace recordings were repeated 5 times per surface (Fig. 3).

Fig. 3.

Diagram showing representitive force traces recorded for the surface topology recognition task, which were used during combined ridge height and spacing test.

C. Procedure

We first searched through the electrode grid to identify two electrode pairs that evoked sensation on different fingers. Five subjects were randomly selected to have pairs that elicited sensation in the median and ulnar region of the hand, while the other five subjects had sensations along the index and middle fingers. This allowed for a direct comparison with prior work, while allowing for a comparison across the two groups. Each electrode pair was then coupled to either the prosthetic hand’s index or middle finger sensors based on the medial/lateral position of the sensation along the hand. Each pair’s stimulation intensity was regulated using their coupled finger forces. Details of the electrode pairs selected, and sensation regions are described in Table S1 in the Supplementary Materials.

Once the electrode pairs were selected, their corresponding sigmoid functions were used to convert the pre-recorded forces to user-specific stimulation patterns, fixed throughout the experiment. For each experiment block, the force traces were randomly selected, resulting in a double-blinded test. Examples of the force-current transformation for representative trials are shown in Fig. S1 in the Supplementary Materials. The detailed description of the electrode pair selection, task familiarization, and stimulation scheme can be found in the Supplementary Materials.

The first block evaluated the shape recognition, which consisted of 4 hand closing speeds and 5 repetitions per speed for each object (40 trials in total). At the beginning of the test, the subject familiarized themselves with the two shapes using an exemplar speed-specific stimulation pattern that was repeated 3–5 times depending on the subject’s confidence. In each trial, subjects were given a random force trace (object) and were then asked to report the perceived shape.

The second block evaluated the surface topology recognition using varied ridge height, ridge spacing, or both, organized in three sub-blocks. Prior to the test of each sub-block, all corresponding surfaces were given to the subject 3–5 times each for task familiarization. The first sub-block required subjects to discern the ridge height relation of three surfaces based on the stimulation intensity. Stimulation patterns associated with two random surfaces were delivered, and the subjects were asked to report if the first or second surface had a higher ridge, or if they were the same. A total of 18 trials involving all possible ridge height combinations were evaluated. The second sub-block evaluated the recognition of ridge spacing based on the stimulation spike interval. A higher ridge spacing corresponded to a shorter spike interval. In each trial, subjects were given two ridge spacing conditions, and were asked to report if the first or second surface had a higher spacing, or if they were the same. Due to the use of a fixed 10 cm surface and constant sliding velocity, subjects could make the decision based on the number of stimulation spikes. To tease out this confounding, stimulation patterns with the same number of spikes were also used. A total of 30 trials were tested. The final sub-block evaluated the recognition of ridge height and spacing simultaneously. In each trial, stimulation patterns resembling one of the four surfaces were randomly delivered to the subject, and the subjects were asked to report the surface condition (high/low height and high/low spacing). A total of 16 trials were tested. Throughout the experiment, 10-second of rest time was provided between trials. Example force traces used during all familiarization phases are shown in Fig. S2.

D. Data Processing

To evaluate the recognition accuracy of shape and surface topology, confusion matrices were constructed for each block or sub-block to compare the perceived object to the ground truth. The average and standard error of the accuracy were also calculated across subjects.

E. Statistical Analysis

To determine if the recognition accuracy was significantly greater than chance values, one sample t-tests were performed for each experimental block or sub-block. During shape recognition, the chance of identifying the shape of a given object was 0.5. For the surface topology recognition, the chance of identifying the ridge height relation, spacing relation, or both was 0.33, 0.33, and 0.25, respectively. A chance of 0.5 was also used to evaluate the recognition of individual surface topology, i.e. ridge height or spacing. These accuracies were compared using paired t-tests. Lastly, a paired t-test was also performed between stimulation patterns involving the same number of ridges and the stimulation time during the ridge spacing comparison. A logit transformation [27] was applied to the proportion data prior to the statistical analysis, in order to obtain normal distribution of the residual.

III. Results

A. Shape Recognition

For the shape recognition block, involving different hand closing speeds, Fig. 5A shows a confusion matrix comparing the actual and perceived object shape across all subjects with the rows corresponding to individual hand closing speeds. The results showed that subjects could identify the shape of the object at all four hand closing speeds. Specifically, the closing speeds of 8, 13, 20, and 40 degrees per second resulted in an accuracy and standard error of 85.0% ± 3.4%, 89.0% ± 3.1%, 83.0% ± 3.3%, and 82.0% ± 3.9%, respectively (Fig. 5B). All closing speeds resulted in a recognition accuracy that was significantly greater than the chance value (p<0.001). Lastly, when comparing recognition accuracies across median & ulnar and index & middle sensation groups (Fig. S3), no statistical difference was found for any of the closing speeds (p>0.05).

Fig. 5.

Confusion matrix (A) of the perceived shape against the ground truth at different hand closing speeds. The individual (Indv) and mean accuracy (Avg) for both shapes and the standard error (SE) across subjects (B), with asterisks indicating significance of p<0.001, comparing with chance.

B. Surface Topology Recognition: Ridge Height

For the ridge height recognitioin between two given surfaces, the confusion matrix in Fig. 6 reports the ridge height recognition results across all subjects. The results showed that subjects were able to correctly identify the ridge height relation in 161 out of 180 trials, resulting in an accuracy of 89.4% ± 1.9%. Errors largely arose from either subjects perceiving different heights as the same (right colume) or, to a less degee, same height perceved as different (bottom row). Overall, the performance was found to be significantly greater than the chance level (p<0.001).

Fig. 6.

Confusion matrix quantifying the perceived ridge height in relation with the ground truth across all subjects. ‘First’ means the first surface had higher ridges. ‘Second’ means the second surface had higher ridge. ‘Same’ means both surfaces had the same ridge height.

C. Surface Topology Recognition: Ridge Spacing

For the ridge spacing recognition, Fig. 7 shows the recognition accuracy across subjects during trials with equal stimulation time, and with equal number of ridges. The results showed that subjects correctly identified the relative spacing with an accuracy of 81.7% ± 1.7% and 85.6% ± 1.5% with equal stimulation time, and with equal number of ridges, respectively. Similar to ridge height, most errors that arose involved the “same” spacing condition. However, both types had accuracies significantly greater than chance (p<0.001) with no statistical difference between the two encoding conditions (t=1.74; p>0.05).

Fig. 7.

Confusion matrix (A) quantifying comparisons between the perceived ridge spacing relation with the ground truth. The average accuracy and standard error across the subjects (B) are reported with the asterisk indicating significance of p<0.001, comparing to chance. ‘# of Ridges’ denotes the stimulation patterns had equal number of ridges/spikes, while ‘Time’ denotes the stimulation patterns lasted the same amount of time.

D. Combined Ridge Height and Spacing Recognition

Lastly, we evaluated the surface topology with combined ridge height and spacing parameters. The surfaces were labeled based on their ridge characteristics with ‘LL’ denoting low height and low spacing, ‘LH’ denoting low height and high spacing, ‘HL’ denoting high height and low spacing, and ‘HH’ denoting high height and high spacing. The recognition accuracy across subjects is illustrated in Fig. 8. The results showed that most surface topologies were correctly identified (138 out of 160 trials), resulting in an accuracy and standard error of 86.3% ± 1.8%. Each of the four surface topologies were detected with an accuracy significantly greater than chance (p<0.001). Additionally, the recognition accuracy of individual ridge characteristics were found to be significant greater than chance as well (p<0.001). A significant difference was also found across the two surface characteristics (p<0.05).

Fig. 8.

Confusion matrix (A) quantifying the perceived surface topology with the ground truth. The average accuracy and standard error across the subjects (B) are reported with a single asterisk indicating significance of p<0.001. ‘LL’ corresponds to a surface with a low ridge height and spacing, ‘LH’ a surface with a low ridge height and high ridge spacing, ‘HL’ a surface with a high ridge height and low ridge spacing, and ‘HH’ a surface with a high ridge height and spacing. ** indicates significant difference of p<0.05.

IV. Discussion

This study sought to determine if shape and surface topology recognition could be performed based on transcutaneous nerve stimulation. independently modulated tactile feedback at different fingers was evoked concurrently using a multi-electrode grid. The object shape and surface topology were encoded by temporal or amplitude information of the stimulation. Our results showed that different shape and surface topology can be recognized with accuracies >81%. The evoked tactile information may improve the quality of object manipulation/interactions, and potentially promote user acceptance and embodiment of prosthetic device.

Our results demonstrated that shape recognition could be performed at all four hand closing speeds without training, which suggest that the nerve stimulation and the encoding strategy were highly informative and intuitive. Our results showed that the lowest closing speed led to the lowest accuracy for the cube recognition. This can arise from the large variability of the stimulation onset timings between fingers, which led to a lack of difference from the sphere onset timing difference between fingers.

Prior studies have shown the ability to perform shape recognition using invasive or non-invasive stimulation techniques [9], [19], which showed similar recognition performance compared with the current study. Although hand closing speed was not available in previous studies, the recognition accuracy tended to be lower at a higher closing speed shown in the current study, which may be due to several factors. First, the differences in shapes and grasp patterns used in different studies may have resulted in different force traces and different timing of the onsets. second, the experimenter operated the prosthesis in our current study. Therefore, recognition was exclusively based on the tactile perception and was not biased by other information during sensorimotor integration. Direct control of the prosthetic hand by a user may provide additional information regarding finger position, which can further assist shape recognition. Overall, our findings suggest that non-invasive stimulation of proximal nerve bundles can reach reasonable performance on shape recognition. The non-invasive approach allows routine testing across a large population without the need of surgery procedure as described in [19]. The more proximal site compared with the previous study [9] also reduces the potential for myoelectric control signal interference, when the transcutaneous nerve stimulation was delivered near the muscle activity recording site.

Our results showed that surface topology recognition could be performed based on ridge height, spacing, or both simultaneously. Similar to the shape recognition, training was not needed, suggesting that the evoked tactile feedback was highly informative. When ridge height and ridge spacing were evaluated concurrently. We found that ridge height recognition accuracy was significantly higher than ridge spacing recognition accuracy. This suggests that stimulation intensity perception was more accurate compared with temporal-related spike interval perception. Differences in just noticeable difference or psychometric tests across encoding methods would likely affect their corresponding recognition accuracy. Further studies will be required to evaluate the recognition resolution of each encoding method.

As of now, few studies have quantified surface topology recognition via non-invasive nerve stimulation. In previous studies, tactile feedback is delivered using invasive approaches [20], [21], and ridge spacing is the only variable evaluated. Compared with prior studies, our accuracies are slightly higher with disparities due to multiple factors. First, surface characteristics and the sliding speed of the surface/fingertip can influence the recognition performance. In the current study, the sensors embedded in the DEKA hand and the size of the finger limited the size of the ridges, with ridge height at the millimeter scale and spacing at the centimeter scale and sliding speed of approximately 10 cm/s. In contrast, Oddo et al. [20] utilized ridge spacing and ridge height on the sub-millimeter and millimeter scale, respectively, when utilizing a sliding speed of 10 mm/s. Although the ridge parameters and sliding speeds vary across studies, the chosen speed of the current study led to similar inter-ridge intervals, compared with the previous work, which suggests that the recognition difficulty level was similar across studies. The performance of topology recognition in the current study was higher compared with the previous work. Differences in the stimulation approaches could affect the recognition performance. Prior studies only stimulated using a set stimulation level at or around the times when a ridge crossed the fingertip of an avatar hand [21] or when the force reaches peak values, i.e. instances when ridges contact the finger [20]. In essence, the participants could use stimulation on/off patterns to recognize surface ridges. In contrast, the force-modulated stimulation in our current study was delivered continuously when the surface was slid on the fingertip, because the surface and the fingertip was in contact throughout a trial. As a result, the participants had to rely on the change of sensation intensity for surface ridge recognition. Compared with the event-triggered stimulation approach, the stimulation directly modulated by the force can provide more intuitive information about object surface properties.

Multichannel stimulation can be further exploited using several sensors located across the prosthetic hand. Each of which would map to the different electrode pairs that produce sensations along the fingers or the palm. Complex percepts can then be elicited that realistically replicate the complex interactions of the human hand with different objects. Reductions in recognition accuracy were observed (Fig. S3 of the Supplementary Materials) during shape recognition, which can be caused by overlapping of sensation regions evoked by different electrode pairs. [28] Modulating stimulation in a more biomimetic strategy may help improve the amount of transferable information. [29] This has been discussed further in the Supplemental Materials.

Limitations and Future Work

The current study has several limitations. First, a previous study has showed similar evoked haptic perceptions between neurologically intact participants and individuals with arm amputations [22], suggesting that the results found in intact participants can characterize those in amputees. Additionally, work by Makin et al [30] has shown that several years after an amputation, the somatosensory cortex representation of the phantom hand remains unchanged, in contrast to the motor cortex representation. Nonetheless, further studies involving upper limb amputees are necessary to quantify the performance of shape and surface topology recognition. Second, the prosthesis was controlled by the experimenter, and the object-prosthesis interaction forces were recorded prior to the testing, in order to ensure a double-blinded experimental design. An experimenter-controlled hand also ensured that other sensory modality such as proprioceptive feedback during muscle activation or incidental feedback, such as vibration of the prosthesis, would not bias the results. Lastly, shape recognition only involved two discrepant shapes, i.e., a cube and a sphere. If a larger number of objects were used, we expect that the performance would potentially worsen. The ability to use this encoding method in a range of objects would need further investigation.

Conclusions

Overall, we demonstrated that sensory feedback via transcutaneous nerve stimulation targeting the proximal segments of the median and ulnar nerves could be used to perform object shape and surface topology recognitions. The findings suggest that these encoding methods could potentially be employed when using a sensorized prosthesis or a remotely controlled device to provide insight about object properties. This sensory input to the users may also help facilitate a user’s ability to execute dexterous motions.

Fig. 4.

Diagram illustrating the experimental protocol, involving the electrode pairs selected for the prosthesis’s index/middle finger sensors, and the experimental blocks for object shape and surface topology recognition.