Slip Detection and Grip Adjustment Using Optical Tracking in Prosthetic Hands

Abstract

We have designed a closed loop control system that adjusts the grasping force of a prosthetic hand based on the amount of object slip detected by an optical tracking sensor. The system was designed for the i-Limb (a multi-fingered prosthetic hand from Touch Bionics Inc.) and is comprised of an optical sensor embedded inside a silicone prosthetic glove and a control algorithm. In a proof of concept study to demonstrate the effectiveness of optical tracking in slip sensing, we record slip rate while increasing the weight held in the grasp of the hand and compare two cases: grip adjustment on and grip adjustment off. The average slip rate was found to be 0.314 slips/(s·oz) without grip adjustment and 0.0411 slips/(s·oz) with grip adjustment. This paper discusses the advantages of the optical approach in slip detection and presents the experiment and results utilizing the optical sensor and grip control algorithm.

I. Introduction

In the human body, the central nervous system constantly adjusts the grip force in the hand so that there is a safety margin of about 10–40% to prevent objects from slipping out of the grasp [1]–[3]. This grip control is reflexive and the processing occurs at the spinal cord level [4], [5], and the initial gripping forces are based on anticipation of the load and change reflexively based on sensation feedback in the hand [6]. Most current hand prostheses simply grip with a high grip force in order to prevent slipping. The problem with this technique is that delicate objects can easily be broken if the grip force is too large, and heavy objects can slip out of the grasp if the grip force is not large enough. An automated grip system that reacts to slipping by automatically tightening the grasp force would help solve this problem and prevent prosthetic hand users from dropping objects.

Because objects come in a variety of shapes, sizes, and weights, it is important for a prosthetic hand to be able to adapt its grip to the type of object it is grasping. Many prosthetic hands initially grasp with enough grip force to hold somewhat heavier objects [5]. This enables users to pick up a wide range of objects, but becomes a problem when attempting to manipulate exceptionally heavy or delicate objects. A slip detection system with automatic grip adjustment could potentially eliminate this problem by allowing the initial grip force to start low for gripping delicate objects but increase when slipping is detected for heavier objects.

Previous research into slip sensing has been done using pressure, force vector control, and vibration [7]–[9]. Force vector control is the method of observing the 3-dimensional vector of the force applied to the object by the hand and constantly measuring its angles. These angles are then used to see if the force vector is outside the “adhesive frictional cone,” which is made up of the vectors beyond which slip will occur [7]. As the vector reaches the friction cone, the grasp is tightened, and the force vector moves back inside the friction cone. The disadvantage is that it does not work as well for objects that are very smooth or rough, because the coefficient of friction used to define the frictional cone is pre-determined [7]. This method is currently used by Otto Bock, the largest manufacturer of prosthetic hands worldwide.

BioTac fingertips developed by Loeb et al. monitor slip by using vibration [8]. The BioTac fingertip consists of a solid core with impedance electrodes on the exterior of the core and a pressure sensor and thermister on the inside. The sensor covered core is covered by a silicone “skin” and a conductive fluid layer is injected between the silicone skin and the core. As the exterior silicone covering is pressed against the core, the impedance measured by the electrodes changes. These impedance changes are related to the tangential forces. Furthermore, the pressure sensor detects vibrations that result from contact of the fingertips with surfaces. The moment of slip is defined when vibration occurs and the tangential forces plateau [8]. However, impacts to the hand can also trigger these vibrations, and the signal of actual slip has yet to be distinguished from vibrations from impacts to the hand and vibrations from the motors controlling the hand [8].

Tremblay and Cutkosky pursued vibration by making foam filled fingertips and using accelerometers to detect impending slip [9]. When impending slip was about occur, small ridges on the skin would slip independently of one another before complete slip occurred, sending vibrations to be picked up by the accelerometers. Using multiple accelerometers, it was possible to distinguish between actual impending slip and outside impacts, though vibration from motors could still be an problem.

Some research [10] has also been done into optical solutions, but none have led to an effective solution because optical systems are typically too large, cannot be easily fitted to prostheses, and lighting conditions must be acceptable [10]. Optical sensors depend on light, so they do not function well in dark settings. One group [10] used force sensitive resistors to detect whether the hand was gripping an object, and then used optical sensors embedded in the palm to detect if the object was moving. However, this method would cause problems with gripping small objects.

The work presented here introduces a method for detecting slipping using an off-the-shelf optical sensor from a PC-based mouse in order to perform automatic grip adjustment to help maintain stable grasping in the i-Limb prosthetic hand. The main advantages of this approach are that slip is only detected if the sensor is in direct contact with a moving surface, it supplies its own light, and slip rate information is obtained. Monitoring this information allows the hand to automatically tighten its grip if slipping is detected. The automated grip idea is not new [6], [9], but it has not been implemented using optical slip detection, and slip rate has not previously been quantified to the extent of the authors’ knowledge.

II. Methods

A. Optical Sensing

In optical mice, light from an LED is sent through a lens and reflects off surfaces such that the angle of reflection sends the light into a complimentary metal-oxide semiconductor (CMOS) sensor (Figure 1). The CMOS sensor captures images of the surface illuminated by the LED light at a rate of 1500 frames per second. Next, a digital signal processor (DSP) processes the images to identify patterns in the images and evaluate how they have changed from the previous image. These pattern changes are translated into magnitude and direction of mouse movement [11].

Fig 1.

Optical Sensing. Light from the LED is sent through a lens and reflects off surfaces such that the angle of reflection sends the light into a CMOS sensor that takes 1500 pictures per second. Image courtesy of EEHomePage.com.

B. Sensor and Silicone Glove Fabrication

To obtain an optical sensing device, we extracted the lens, LED, and CMOS sensor from a Logitech SBF-96 Optical Wheel Mouse (Freemont, CA). The lens was cut down to an oval shape and combined with the LED and CMOS sensor to make it one piece. The overall sensor dimensions were about 32 × 17 × 15 mm. The CMOS sensor was then reattached to the rest of the mouse circuitry using wires so that the sensor could be extended to the hand without having to embed the PCB.

The sensor was embedded in a custom-made silicone glove using Dragon Skin^® 10 Medium silicone from Smooth-On Inc. The silicone glove was fabricated to fit directly over the i-LIMB prosthetic hand (Touch Bionics, Inc.). To fabricate the glove we painted two layers of silicone directly onto a positive mold of the i-Limb. This resulted in a very smooth texture and relatively even thickness throughout the glove (1–1.5 mm).

In order to accommodate for sensor size, the sensor was placed in the palm of the glove, close to ring finger. Silicone was painted around the sensor leaving the sensing area bare, as shown in Figure 2.

Fig. 2.

Embedded optical sensor in silicone glove. Wires run from CMOS sensor to the PCB, and the USB cable is attached.

Using the existing USB interface and closed loop algorithm, the change in x and y coordinates of the optical sensor were obtained and, using Microsoft Visual Studio C# 2008 Express Edition, the authors wrote a closed local loop that looks for a change in slip magnitude over a period of time and tightens the grip if any is detected.

Slip is defined as the mouse position displacement: $\mathrm{s}\mathrm{l}\mathrm{i}\mathrm{p}=\sqrt{{\left(x_2-x_1\right)}^2+{\left(y_2-y_1\right)}^2}$ (1). The program utilizes a timer and calculates the magnitude of slip by the change in mouse position every fifty milliseconds, and tightens the grip of the hand if slip is detected.

III. Experiment

To demonstrate the effectiveness of combining optical slip detection and grip adjustment, we conducted a proof of concept study wherein we quantified slip rate as object weight is increased in two tests: one with grip adjustment on, and one with grip adjustment off.

A. Experiment Set Up

In order to make sure the initial grip force was the same on cylinder during each trial, a Flexi-Force^® pressure sensor was placed on the thumb where it contacted the cylinder. Flexi-force^® pressure sensors are thin film flexible circuits with a piezoelectric sensing area. The resistance of this area is inversely proportional to the applied force. This sensor was used with additional circuitry to measure the amount of grasping force applied on the cylinder at the thumb point in terms of volts (0–5 V). The grip on the sensor was tightened until the voltage over the pressure sensor was equal to 4 V. A DC Power Supply HY3003D-2 (Shanghai Yi Hua V&A Instrument Co., Ltd) was used to supply power and a Tektronix Oscilloscope TDS 3012 to monitor the voltage.

The cylinder was covered with non-reflective printer paper to give it a uniform surface. Plastic bags were attached to hold weight from the bottom of the cylinder by string that ran through holes drilled in the lid. The total weight of the cylinder, bags and string combined was 9.2 oz. See Figure 3 for experimental set up.

Fig. 3.

Experimental set up. i-Limb with the slip sensing glove on and the cylinder with pressure sensor underneath the thumb. Bags to hold added weight were attached by string to the bottom of the cylinder.

B. Experimental Protocol

The experiment was run as follows: tighten grip on the cylinder to 4 V, add weight to the plastic bags in 4 oz. increments, and record total slip during the first five seconds after each weight addition.

Hunters Specialties 4 oz. Decoy Strap Weights (Model 00206) were used to add weight to the plastic bags. The time of five seconds was used because the authors assumed slip to be linear during each weight addition, so the average slip rate can be obtained by dividing the total slip rate by five.

The experiment was conducted using a control case (slip sensor on, grip adjustment off) and a test case (slip sensor on, grip adjustment on). Each case was repeated two times and the results were averaged.

C. Results

Figure 4 plots the average slip rate vs. object weight with grip adjustment on and off. Linear trend lines were used to approximate slip rate. It was found that slip began at about 60 oz. for both cases and occurred at an overall average rate of 0.314 slips/(s·oz) with grip adjustment off, but only 0.0411 slips/(s·oz) with grip adjustment on (663% difference).

Fig. 4.

Average slip rate vs. object weight for grip adjustment (GA) on and off. As weight gets above 50–60 oz., slipping occurs at an average rate of 0.314 slips/(s·oz) with GA off, and only 0.0411 slips/(s·oz) with GA on.

The reason the data for grip adjustment off ends at about 90 oz. is because at that point the slipping became so rapid that the cylinder slipped through the grasp of the hand. This did not occur for grip adjustment on. However, it can be assumed from the data for grip adjustment off that if more data could have been collected that more slip would have occurred at higher weights.

D. Discussion

In this experiment, optical sensors were used to detect if an object was slipping through the grasp of a prosthetic hand, and the grip force of the hand was adjusted if slipping was detected. The results from the experiment indicate that optical sensing can be used to this effect and that automated grip adjustment can be easily implemented to control slipping. Directional information can also be obtained from these sensors, although this was not pursued in this experiment.

There are experimental limits that should be noted. Only one surface type was used in the experiment (printer paper), and although these optical sensors track on many surfaces, they have difficulty with some. They cannot track on glass and have trouble on some glossy surfaces. Surfaces that are smooth can also prove difficult to track on.

The targeted initial grip condition of 4 volts across the Flexi-Force^® pressure sensor varied slightly (0.05–0.15 Volts) during each test setup.

Only one sensor was used. This sensor was placed in the palm of the hand, so only larger objects held in a specific grasp will come into contact with it. This could be improved if multiple smaller sensors embedded throughout the glove.

Only one shape and size of object was used. However, these limitations focus on the ability of the i-limb to grip objects, and not the effectiveness of the sensor to track slipping.

One disadvantage to the optical method is that moving light across the sensor can trigger a “false positive” (slip detected by the sensor when is none is occurring). This should not be an problem with most objects because when the sensor comes into contact with the object, outside light will not be able to reach the sensor, however, this may prove to be a problem with objects of clear materials (such as drinking glasses), because light from outside sources can travel through the glass and reach the sensor.

E. Future Work

Since directional information can be obtained from the sensors in addition to slip rate, it is possible to develop an algorithm that uses this directional information to determine if motion detected by the slip sensor in the hand is actually slip or just slight movement within the grasp, such as rotation. This could be accomplished using multiple optical sensors in the hand and accelerometers to determine the orientation of the hand.

The system is also still fairly large, even with the smaller solution of the CMOS sensor, LED, and lens, so the system must be miniaturized further to embed multiple sensors.

Laser tracking, as opposed to optical tracking, provides a possible solution. Philips^® has combined the entire sensor system (laser, lens, sensor) into a 6.8 × 6.8 × 3.86 mm sensor in their Twin Eye Laser sensors (PLN1523). Laser sensing will allow tracking on almost all surfaces (possibly including some types of glass), provide higher resolution, and allow for greater maximum tracking speed than optical sensing. The small size of the Twin Eye Laser sensors could enable the embedding of multiple sensors throughout the hand to provide spatial resolution so that slip can be localized to the finger where it occurs. This information could then be used to create an algorithm to tighten only the fingers where slip is occurring, thereby saving power and creating more natural grasp with minimized grip force. This would also allow for better manipulation of objects that are smaller or easy to break.