Tactile Feedback in Upper Limb Prosthetic Devices Using Flexible Textile Force Sensors

Abstract

Many upper limb amputees are faced with the difficult challenge of using a prosthesis that lacks tactile sensing. State of the art research caliber prosthetic hands are often equipped with sophisticated sensors that provide valuable information regarding the prosthesis and its surrounding environment. Unfortunately, most commercial prosthetic hands do not contain any tactile sensing capabilities. In this paper, a textile based tactile sensor system was designed, built, and evaluated for use with upper limb prosthetic devices. Despite its simplicity, we demonstrate the ability of the sensors to determine object contact and perturbations due to slip during a grasping task with a prosthetic hand. This suggests the use of low-cost, customizable, textile sensors as part of a closed-loop tactile feedback system for monitoring grasping forces specifically in an upper limb prosthetic device.

I. INTRODUCTION

The human hand is an exquisite and crucial part of our bodies. It permits a wide range of objects to be grasped and manipulated in a dexterous manner. Besides an excellent actuation system, the human hand is densely populated with mechanoreceptors that provide timely tactile information during object manipulation. Despite significant advances in mechanical design, commercial prosthetic hands are still far from achieving comparable levels of functionality [1]. In particular, tactile sensing is lacking in most upper limb prosthetic devices.

Tactile sensors have been shown to significantly enhance the capabilities of prosthetic hands. In [2], tactile sensors were used to determine the state of object held by the hand. Force sensors also enhance safety during low-force interactions, such as shaking hands [3]. In addition to object contact, some tactile sensing methods can detect object slippage [4–6]. With tactile feedback, more intelligent control algorithms can also be implemented to improve the usability of the hand [7–9]. However, some of these studies were conducted using specialized tactile sensors that are complicated or difficult to fabricate. In particular, the hands would need to be modified, sometimes extensively, to include the sensors [9–11]. As a result this can increase costs, rendering them impractical or difficult to incorporate in commercial products.

A more practical approach would be to design tactile sensors that can be retrofit onto existing prosthetic hands. Textile based tactile sensors present an interesting alternative to traditional MEMS sensors that have been used for tactile sensing [10]. Robust, flexible and easily constructed, these textile sensors can be easily customized to mount on existing upper limb prosthetic devices. On the downside, fabric sensors often lack the precision and accuracy of more specialized sensors. However, the additional resolution and accuracy of specialized sensors are arguably of little benefit in many use cases of a prosthetic hand. Similar to how tactile feedback in the human hand can be non-linear and noisy [12–14], a crude force sensing system based on fabric tactile sensors may in fact be sufficient. The goal is that with tactile sensing capabilities a prosthesis could become more of a dynamic extension of an amputee’s body.

In this paper, we explore the feasibility of fabric based force sensing arrays for tactile feedback. While previous studies have proposed textile-based sensors [15, 16], none have been developed for use specifically on prosthetic hands. The rest of the paper is organized as follows: Section 2 describes the fabrication process of such a sensor, as well as characterization of range and sensitivity of the sensor. Section 3 describes the experimental methods and Section 4 provides results and discussion, which is followed by a conclusion in Section 5.

II. TACTILE SENSOR

Commercial force sensors are readily available and have been used for prosthetic tactile sensing applications; however, there is a need to develop a tactile sensing system that is low-cost, customizable, and compatible with any upper limb prosthesis [7–9, 17]. The goal of this work is to investigate the use of customizable force sensors for use with prosthetic technology.

A. Sensor Design

Based on previous designs the sensor is made by sandwiching a piezoresistive transducing fabric (Eeonyx, Pinole, USA) in between row and column conductive traces (Stretch conductive fabric, LessEMF, Latham, USA). Each intersection between traces constitutes a pressure-sensing element, where the resistance between the traces is reduced with increasing applied pressure. The sensor is then held together by non-conductive fusible stretch interfaces. Solderable pads made from ripstop, conductive, polyester based Ni/Cu plated fabric (LessEMF, Latham, USA) terminate each conductive trace, acting as an interface between fabric and metal wires. An exploded view of a sensor designed for a prosthetic finger is shown in Fig. 1.

Fig. 1.

Finger cuff sensor design. Conductive traces are fused to the inner textile cuff. It is then covered by an outer fabric layer and rubber.

The physical design of the sensors is customized such that an array of sensing elements can be placed on the phalanges and palmar surface of a myoelectrically controlled prosthetic hand. For this work, a bebionic v2 (RSL Steeper, Leeds, UK) prosthetic hand was used. The dimensions of the textiles can easily be changed to make different sized cuffs, thus making it easy to fix on any prosthesis (Fig. 1). Due to the stretchable nature of the textiles used, the sensor can be made to fit securely by having a smaller diameter than that of the prosthesis’ phalanges. The sensing area for each pressure sensing element is approximately 2.7 × 2.7 mm². Multiple sensing areas along each phalanx of the prosthesis can be made to increase spatial resolution. Fig. 2 shows possible sensor placements on a bebionic v2 prosthetic hand. For this study, sensors were placed on the index and middle fingers of the prosthesis as these are the regions that primarily contact an object during grasping due to the closing dynamics of the hand. The joints and actuators of prosthetic hands changes between manufacturers and models, thus it is important to create cuffs that can be easily placed on different sections of the prosthesis. The sensor cuffs can stretch to conform to a variety of phalanx dimensions.

Fig. 2.

Sample sensor placement on a bebionic prosthetic hand. Palm and finger cuffs create multiple sensing areas along the device.

It was shown that a compliant surface on the distal regions of the phalanges on a robotic hand enhances the stability of grasped objects while also increasing the probability of completing a grasping task with a fragile object [9]. This can be applied to the fabric tactile sensors by coating them with silicone-based rubbers. A curved mold was designed to coat and seal the sensor with a 3mm rubber layer. The cost of materials to make one sensor is less than $2 USD.

B. Characterization

The textile force sensitive resistors (FSRs) were characterized using an ADC (LabJack U12). A normal force was applied directly to the sensing area of the textile cuff using a mechanical probe with a tip the same size as the sensing element. The applied force was measured using a USB electronic scale (RadioShack, Fort Worth, USA) with a range of 0–50 N and a precision of 0.01 N. Each sensor was tested 3 times at each applied load. Average sensor response was recorded and is shown in Fig. 3. The operating range of the sensors is quantified as the linear section of the response curve, which is approximately 0.5–20 N. This can be fit using a power trendline of y = 9.97x^0.87. This operating range is comparable to force sensors that are available commercially [10, 18]. The textile sensors are suited for applications with prosthetic hands as grasping forces tend to range from 0.5–20 N for these devices, although this range can have some variation as it is largely dependent on object size, shape, and weight [7]. When no stimulus is applied the standard deviation of the sensor is 0.004 V, which corresponds to less than 1 mN.

Fig. 3.

Sensor response curve for a range of applied normal forces.

The rubber is a multi-functional component of the system as it provides extra protection between the sensing elements and the environment while also offering additional compliance to the textile cuff surface. To investigate the effects of having an additional rubber coating, we repeated the experiment with 5 different rubber materials (Smooth-On, Easton, USA) placed between the sensor and the probe. The properties of the materials are listed in Table I. Fig. 4 shows the response of the textile sensors when coated with a 3 mm layer of silicone rubber. It is apparent that increasing the durometer (hardness) of the rubber layer decreases the sensor sensitivity, while extending the operating range. This agrees with the classic JKR elastic contact model [19]. Dragon Skin 10 was chosen as the most suitable rubber coating for the sensors because it maintains sensor’s sensitivity while offering appropriate compliance under an applied load. The addition of the rubber layer shifts the operating range to 0.8–30 N. Although Ecoflex’s low durometer offers a more compliant rubber layer, its low viscosity causes the rubber to seep through the textile fibers and soak the internal conductive traces of the cuff. This either completely removes any connection between the traces or the conductance between traces is reduced. This phenomenon is seen in Fig. 4 where the sensor with Ecoflex requires a larger activation force. This may be prevented by more advanced fabrication techniques, but we recommend using more viscous rubbers to reduce fabrication costs. Conversely, Smooth Sil 945’s high durometer results in an unresponsive sensor, due to its hardness and lack of compliance.

TABLE I.

COMPARISON OF SILICONE RUBBERS USED ON TEXTILE SENSORS

Material	Mixed Viscosity (cPs)	Elongation at Break	Shore Hardness
Ecoflex	3,000	900%	00–30
Dragon Skin 10	23,000	1000%	10A
Dragon Skin 20	20,000	620%	20A
Dragon Skin 30	30,000	364%	30A
Sil 945	40,000	320%	45A

Fig. 4.

Sensor response for different silicone rubbers.

III. EXPERIMENTAL METHODS

To assess the functionality of the textile sensors for use with a prosthetic hand, a series of experiments were performed to evaluate the ability of the cuffs to detect forces while a prosthetic hand grasps an object. Two of the most important aspects of determining the state of the object are when it is contacted by the prosthesis and when it begins to move within the grasp, indicating object slip. The goal of the grasping experiments is to evaluate the responsiveness of the sensors under standard use cases, focusing on the ability of the sensors to detect object contact as well as the movement of a grasped object when slippage occurs. These are critical events in object manipulation, as they can aid a controller in determining the state of the grasped object.

A. Grasping Experiments

The sensor cuffs were mounted to the distal region of the index and middle fingers of a bebionic v2 prosthetic hand. The movement of the prosthesis was managed using a customized control board as opposed to traditional electromyogram (EMG) inputs, to reduce variability across experiments and trials. A tripod grasp was used by the prosthesis to grab an ice hockey puck (0.16 kg, 70mm diameter) as well as a ceramic coffee mug (0.35 kg, 80mm diameter) one at a time (Fig. 5). The objects were chosen because they offer the size, shape, and weight of a medium sized object that an amputee might grasp with his or her prosthesis.The objects were stationary when grasped by the prosthesis. For each object, a stable grasp is maintained for a few seconds before the hand is slowly opened to simulate object slip. The rate of increasing hand aperture is controlled by modulating the input of the prosthesis motors with a 5% duty cycle for a signal period of 42 ms. Opening at this controlled rate causes a slow decrease in the applied grip force, which will eventually cause the grasped object to slip from the prosthetic hand. The task was repeated at least 4 times for each object.

Fig. 5.

Prosthetic hand with attached sensor cuffs grasping an ice hockey puck and a ceramic coffee mug.

1). Contact

Contact between the prosthetic hand and the object is characterized as a positive increase in the sensor output. A minimum threshold is set to differentiate between signal noise and object contact; this threshold is chosen as 0.15 N above the resting state of the sensor once it is mounted on the prosthetic finger. Once contact is initiated, the prosthesis control board can then start actively monitoring the sensors signals for object slip.

2). Slip Detection

For these experiments, slip is determined by a first order time derivative of the sensor signal, which is calculated using a backward finite difference. Traditional methods for detecting object slip in robotic hands involves multi-axial force sensing to compare normal and tangential loads [7]. Multi-axial sensors can increase the cost of a system while also requiring modifications to be made to properly attach to a robotic hand. A simple time derivative of the normal force signal helps reduce the on-board computation time within the prosthesis, thus maintaining quick response times. For this experiment, object contact initialization and slip are defined as occurring when the force derivative signal is above 0.01 N/ms or below −0.008 N/ms, respectively. These values were chosen in an effort to reduce the number of false positives from the sensor output signal.

B. Simulated Prosthesis Controller

In order to create a closed-loop tactile feedback system that is capable of detecting and preventing grasped object slip, it is necessary to be able to determine the state of the object while it is being manipulated by the prosthetic hand. A prosthetic hand uses signals, either from the users’ EMG signals or else the on-board control unit to open and close the terminal device. Signals sent from the controller when slip is detected would result in hand closure, thus reinstating the stability of the grasped object. For these experiments, instances of slip trigger simulated electrical pulses, which could in turn be used to close the prosthesis to prevent an object from additional slip. Each instance of slip generates a simulated prosthesis closing pulse of 100 ms.

C. Data Acquisition and Processing

An instrumentation amplifier was used to boost the sensor signal while reducing noise and drift of the output. The sensor was sampled at 500 Hz using a NI USB-6009 DAQ (National Instruments, Austin, USA) and processed using LabView and MATLAB on a PC.

IV. RESULTS & DISCUSSION

Results from a typical trial are shown in Figs. 6 and 7, for the grasping of the puck and mug respectively. Variation between trials were minimal, and were thus omitted for clarity. The top charts in each figure are the normal force signal from the index and middle fingers of the prosthetic hand while the bottom chart shows the force derivative of those signals. It should be noted that the initial force values measured by the sensor, before grasping the object, are nonzero. This is because the sensors are stretched to fit over the phalanges of the prosthesis, which causes a reaction force between the phalanx and the sensor. For the hockey puck grasping task the object is primarily contacted by the index finger (Fig. 6). Initial contact is made around 8 s, which can be easily detected by applying a threshold on the time derivative of the force signal. This reflects the change in forces applied on the sensor. Moreover, by evaluating the changes of the force signal, there is no need to zero the sensors.

Fig. 6.

Results from the puck grasping task. The top chart shows the normal force grip, as measured by the sensors. The bottom chart shows the time derivative of the force signal. Large positive spikes and negative spikes indicate object contact and slip, respectively.

Fig. 7.

Results from the mug grasping task. The top chart shows the applied normal force grip signal, as measured by the sensors. The bottom chart shows the time derivative of the force signal. Large positive spikes and negative spikes indicate object contact and slip, respectively.

The grasped puck shows instability at the 13 s mark as the hand is closed tighter around the object. There is a slight, yet quick, decrease in the force signal, which results in a noticeable negative spike in the force derivative signal. This indicates that the object is moving across the force sensors. The prosthesis regains a stable grasp on the puck at 14.5 s, which is seen by the positive spike in the force derivative at the same time. The grasped object is held stationary until 24 s at which time the object begins to slip from the hand.

The onset of object slip is realized with the synchronous decrease in applied grip force on both the index and middle fingers of the prosthetic hand. The step like force signal is characteristic of an object undergoing stick-slip [20]. The abrupt reduction of the force is translated into negative spikes in the force derivative signal. The object fell completely from the grasp of the hand at 26.5 s. In a similar fashion, the ceramic mug grasping task shows positive increases in the force derivative signal, indicating object contact. Onset of the mug slipping from the prosthesis grasp is characterized by the negative spikes of the force derivative signal, similar to that seen during the grasping task with the hockey puck. The sturdy nature and geometry of the mug allowed for a higher grasping force from the prosthetic hand; however, instances of slip are still characterized as signals below −0.008 N/ms, as previously mentioned. It is interesting to note that the applied grip force for both objects is slightly less for the middle finger than the index finger. This can be attributed to the nature of the prosthetic hands closing mechanics as well as object shape. The first area of contact with the object made by the prosthesis is with the thumb and index finger. This results in higher grip forces being applied to these particular areas.

As seen from Figs. 6 and 7, the customizable textile cuffs are capable of detecting changes in the force signal that is related to object slip. A basic controller was designed to provide a proof of concept for making hand adjustments based on the force derivative signal. Fig. 8 shows the digital output from the prosthesis controller that can be generated from the force derivative signal and Fig. 9 shows a diagram of the proposed system. It should be noted that results for the prosthesis controller output are presented using one grasping task as the results from other grasping tasks are very similar. Multiple instances of slip in a short period, as seen from 24.5–26 s in Fig. 8, will cascade multiple hand closure signals. Essentially, repeated detection of slip will send a longer close pulse to the prosthesis. It is seen from the results that the low-cost, textile cuffs offer a viable option for creating a closed-loop tactile feedback system for monitoring grasping forces in an upper limb prosthetic device.

Fig. 8.

Simulated prosthesis controller results. The bottom chart shows the electrical pulses that can be sent to close the prosthetic hand. An instance of slip can generate a close pulse of 100 ms.

Fig. 9.

Diagram for a feedback system that would provide real-time hand adjustments based on tactile information.

We conclude that the textile cuff sensors are capable of detecting contact and perturbations due to object slip during a grasping task. The sensor cuffs themselves are universal in the sense that they can be made to fit any diameter prosthesis phalanx, thus making them suitable for a variety of different devices. To improve efficiency of the control algorithms running on the prosthesis control board, one can limit the amount of time spent sampling the sensor signal in order to preserve standard functionality and battery life of the prosthesis. The system is designed to contain multiple sensing elements along various parts of the prosthesis. The main area of contact when grasping an object with the prosthesis is distal region of the index finger. As a result, a single sensor on the index finger could be continuously sampled until contact is detected. Once the prosthetic hand initiates contact with the desired object, the control unit can begin monitoring the remaining sensors. This allows multiple sensing elements of the system to become active only when they are needed. Subsequent work will focus on the contact and deformation mechanics of the textile sensor during relative surface movement as well as grasping tasks. In addition, all of the objects grasped in this study have smooth, curved surfaces. A natural progression would be to investigate if the same techniques presented would be applicable to deformable objects and objects of other shapes.

V. CONCLUSION

This study presents a low-cost, easily customizable, textile force sensing cuff for use with an upper limb myoelectrically operated prosthetic device. In particular, contact and slip detection during a grasping task with a prosthetic hand is realized with the use of the textile sensors. Applied normal force is measured during a grasping task with a prosthetic hand, and the force derivative signal provides information regarding the contact between an object and a prosthetic hand. Instances of slip detected at the object and phalanx interface can be used to generate electrical pulses by a control unit to close the prosthetic hand in order to prevent further slip. Results show that a simple fabric based sensor mounted on a prosthetic hand is capable of measuring grip forces with enough sensitivity to detect perturbations due to object slip. Despite their simplicity, tactile feedback from these sensors would prove valuable in improving control algorithms and ultimately creating prosthetic hands that are more intuitive and usable. Additionally, the customizability and low-cost nature of the sensor would allow tactile integration with prosthetic hands without excessive modifications to the existing hardware, thus extending its user-base significantly.