Robot-Assisted Guitar Hero for Finger Rehabilitation after Stroke

Abstract

This paper describes the design and testing of a robotic device for finger therapy after stroke: FINGER (Finger Individuating Grasp Exercise Robot). FINGER makes use of stacked single degree-of-freedom mechanisms to assist subjects in moving individual fingers in a naturalistic grasping pattern through much of their full range of motion. The device has a high bandwidth of control (−3dB at approximately 8 Hz) and is backdriveable. These characteristics make it capable of assisting in grasping tasks that require precise timing. We therefore used FINGER to assist individuals with a stroke (n = 8) and without impairment (n = 4) in playing a game similar to Guitar Hero©. The subjects attempted to move their fingers to target positions at times specified by notes that were graphically streamed to popular music. We show here that by automatically adjusting the robot gains, it is possible to use FINGER to modulate the subject’s success rate at the game, across a range of impairment levels. Modulating success rates did not alter the stroke subject’s effort, although the unimpaired subjects exerted more force when they were made less successful. We also present a novel measure of finger individuation that can be assessed as individuals play Guitar Hero with FINGER. The results demonstrate the ability of FINGER to provide controlled levels of assistance during an engaging computer game, and to quantify finger individuation after stroke.

I. Introduction

Suffered by more than 700,000 people each year, stroke is the major cause of neurological injuries in the United States and is frequently followed by serious, long-term disability. Risk of stroke is correlated to age and thus a higher incidence rate is expected as the population ages. More than 80% of victims lose partial control of their upper extremity. As of 2005, more than 30% of stroke survivors in 21 states received outpatient rehabilitation [1]. Traditional physical therapy can improve post-stroke recovery, but it is expensive, labor-intensive and likely dosage-dependent. Moreover, traditional therapy methods can sometimes be boring and do not provide therapists with quantitative data of the patient’s performance and improvement.

To address these problems, many researchers have created robotic rehabilitation devices, many of which have been targeted at upper extremity and hand/wrist therapy. Most of these devices are either simplified pneumatic, spring or cable driven approaches with only rudimentary control ability [2-4] or include complex exoskeletal mechanisms [5, 6]. Control strategies used for these robots range from rigid trajectory tracking controls to soft controls that adaptively learn the patient’s impairment level [7].

Although use of robotic devices can aid hand movement recovery[8], it is still unclear which behavioral factors robots should modulate during robot-assisted training to improve recovery. Higher fidelity robots that can implement a range of sophisticated control strategies may help identify those behavioral factors. For example, it is thought that increased patient effort during motor training may enhance recovery [9-11]. Robots that “over-assist” movement may thus cause patient slacking [12, 13]. Designing robots that can precisely modulate effort may help rigorously determine the relationship between effort and recovery, and provide insight into how to optimize that relationship.

The goal of this project was therefore to develop a lightweight robot capable of assisting in naturalistic grasping movements of individual fingers with high control fidelity. The resulting robot, FINGER, uses an eight-bar mechanism to control the orientation and position of the proximal phalanx and the position of the medial phalanx. Identical eight-bar mechanisms can be stacked to allow control of multiple fingers. This paper describes the design of FINGER and characterizes its control capability. In addition this paper reports the results of a pilot study in which we used FINGER to assist individuals with a stroke in playing a video game similar to Guitar Hero©. We developed a control algorithm to modulate success at the game, and hypothesized that by modulating success we could vary subject effort during the game. We also used the game to study whether FINGER could assess finger individuation after stroke.

II. Methods

A. Mechanical Design

FINGER uses an eight-bar mechanism to curl the finger in a natural grasping motion using a single actuator in the plane of finger flexion [14]. In order to facilitate easy attachment of the hand to the robot, and to allow possible contact of the volar surface of the hand with objects during therapy, the mechanism and its actuators are located behind the hand (Fig. 1). The mechanism includes mechanical hard stops to limit the range of motion of the fingers as an inherent safety factor. Moreover, the computer control system includes some velocity and force limits to guarantee the subjects’ safety. The mechanism is designed to be attached only to the proximal and medial phalanges, leaving the distal phalanges free for sensory feedback if needed during exercises. In order to make sure that the finger and the mechanism are aligned during the planar motion, the mechanism controls the location and angle of the proximal phalanx and only the position of the medial phalanx (Fig. 1).

Figure 1.

FINGER robot with two 8-bar finger curling mechanisms and two actuators (top), and close-up of index finger aligned with the two endpoints of the mechanism (bottom). The proximal phalanx finger-cup is fixed at an angle but the medial is free to rotate.

The trajectory design for the mechanism was based on motion capture [15] of finger curling motions during power grasp. The data was regressed against a 2 revolute joint finger model, producing dimensions of and the angular relationship between the proximal and medial phalanges during curling motions. The phalanx dimensions were compared to published values [16, 17] in order to determine appropriate size for each of the different design sizes. The dimensions were then used to generate 15 target positions for both the proximal (position and angle) and medial (position only) phalanges for use in the design of the curling mechanism This was repeated for the four different design sizes, so that the final mechanism would be easily adjustable to fit different hand sizes. After an exploratory trial and error process, an eight-bar mechanism was selected to achieve the finger curling motion. The preliminary design process is described in [14], with significant improvements made to allow the mechanism to be easily adjusted to fit different finger sizes. The version of FINGER used in this study includes two of the 8-bar mechanisms that are parallel to each other for assisting the middle and index fingers.

Finger cups with hook-and-loop straps were located at the two end effectors of each mechanism to attach the robot to the subjects’ proximal and medial phalanges. The medial finger-cup allows for rotation while the proximal finger cup is fixed, as per desired kinematic design. The robotic device also includes vertical adjustment for both finger 8-bar mechanisms so they may be moved to the plane of the subjects’ index and middle fingers (see Fig 2 below). The subjects’ wrists are secured in a trough with a hook-and-loop strap during game play.

Figure 2.

Finger size adjustment of the proximal phalanx (left) and vertical adjustment of the 8-bar mechanisms for the middle and index fingers (right).

Effort was taken during the manufacturing of each 8-bar mechanism to reduce mass and friction. Each member in the 8-bar linkage was built as either an “inner” link or a pair of “outer” links. The inner links were designed with two small ball-bearings that allow the outer links to rotate with low friction on shoulder bolts connecting the outer links. All of the links were manufactured using CNC machining to ensure repeatability and axis alignment.

B. Actuation

The goal of this project was to make a lightweight robot capable of assisting in naturalistic grasping movements of individual fingers with high control fidelity. To achieve high control fidelity, we designed FINGER to use a type of linear brushless motor called a Servo-Tube actuator (Dunkermotoren STA116-168-S-S03C). Its thrust element is a rod consisting of multiple permanent north-south magnetic pairs placed back-to-back, moving inside the actuator body which contains electromagnetic coils. The actuator force has a linear relationship with the current that can be controlled with current drives specifically designed for these actuators, allowing for easy force control. An internal dry bearing provides gear-free smooth performance, and thus the actuator has low-friction and is backdrivable. The actuator can reach accelerations up to 42g. It also has a built in position measurement system that measures the position of the rod very accurately using the Hall effect inside the actuator and sends emulated quadrature encoder data to the computer, that depending on the amplifier, can achieve 8 microns of resolution. Some specifications are given in Table 1. In final robot assembly, the two actuators are mounted on top of each other such that each one can independently rotate around an axis that is normal to the plane of the mechanisms. This minimizes actuator friction from off-centered loading.

Table I.

Actuator Specifications

Specification	Value
Peak force (N)	91.2
Continuous force (N)	26.75
Peak Acceleration (m/s2)	422
Maximum speed (m/s)	4.7
Stroke (mm)	168

C. Determining Control Fidelity

A closed loop frequency response test was performed in order to identify the control fidelity of FINGER when not connected to a human hand. A proportional-integral-derivative (PID) controller was used with proportional, integral, and derivative gains of K_P= 8 N/m, K_I = 8 N/m·s and K_D = 2 N·s/m, respectively. These gains were chosen by trial and error to get a good tracking performance at low speeds. Sine waves with frequencies ranging from 0.15 to 100 Hz were tested, all within 75% of the stroke to avoid hitting the hard stops at both ends.

D. Human Subjects

The device was also tested on a group of eight volunteers with stroke (6 male and 2 female, average age of 56.5 +/− 13.8 SD), and four unimpaired subjects (3 male/1 female, average age 33.5 ± 9.4 SD). For the individuals with stroke, upper extremity impairment was assessed using both the Fugl-Meyer (FM) test and the Box and Block (BB) test [18, 19]. Average scores were found to be 40.6 ± 18.4 out of 66 and 22.5±23.1 respectively. The average BB score for healthy subjects within the same age range is reported in literature to be 75.2±11.9 [19]. Based on these scores, four of the subjects were classified as highly impaired (FM < 40 & BB < 20), and the remaining four subjects were classified as moderately impaired. All subjects provided informed consent, and all procedures were approved by the institutional review board at U.C. Irvine.

E. Training Task

To evaluate its potential as a therapeutic tool, FINGER was used to assist subjects in playing a video game similar to Guitar Hero©. In previous studies with an instrumented glove, we found this game creates an enjoyable and motivating context for repetitive therapy [20]. The goal of the game was to play along with a song by hitting notes displayed on a screen. The song selected was Happy Together by the Turtles, and it required a total of 104 notes to be hit in 160 seconds. In the version of the game implemented here, three possible notes were controlled by flexing the index finger, the middle finger, and both fingers together as shown in Fig. 3. Visual feedback of finger position was given by small balls that hovered in front of the targets. In order to hit the notes, subjects had to move the ball for the correct finger into the center of its corresponding target at the correct time. After hitting a note, the subjects were required to extend their finger(s) back to a neutral position before the game would allow them to attempt another note.

Figure 3.

Screen-shot of the game, which is similar to Guitar Hero©. The green target was controlled by the index finger, the yellow target by the middle finger, and the blue target by both fingers together. The other two targets were not used in this study

Since a separate note was assigned to the flexion of both fingers together, notes for which the subjects were supposed to move only one finger but instead moved both were considered as misses. This inclusion of the separate and combined notes allowed subjects to practice finger individuation. However, since the subjects received assistance from the robot during these trails additional measures were necessary to measure the subjects unassisted finger individuation.

F. Assistance Algorithm

During each song, the amount of assistance provided by the robot was adjusted using a success rate algorithm described in detail in [21]. This algorithm adaptively adjusted the robot’s control gains so as to manipulate the subjects’ probability of success. After each note passed, the algorithm checked whether the note was hit or missed. If the note was hit, the gains for the corresponding finger(s) were reduced by an amount ρ. In contrast, if the note was missed then the corresponding gains were increased by an amount α·ρ. After a number of trials, the algorithm would cause the subject’s probability of success to converge on a value dependent only by α as shown in Equation 1 below [21].

$${\stackrel{‒}{P}}_{i\to \infty }=\frac{\alpha }{\alpha +1}$$ (1)

The variable ρ in this algorithm affects both the rate of convergence and the amount of expected variance around the desired success rate. If ρ is very large, the average success rate will converge quickly but the variance around the success rate will be high. Alternatively, if the value for ρ is very low the success rate will take longer to converge to the desired value but will have a lower variance [21]. For this experiment, ρ was set to a value of 0.5. Separate gains were used for each finger. The ratio between proportional and differential gains was fixed, and the gains were prevented from going above or below their approved operation range.

A formal proof of this algorithm can be found in [21]. Intuitively however, this algorithm slowly reduces the control gains as long as the subjects are successful and then more rapidly increases them when the subjects become unsuccessful. The rates at which the gains are increased (α·ρ) and decreased (ρ) determine how long the subjects will operate within the range of difficulty levels in which they are able to play the game sucessfully.

G. Experimental Protocol

Subjects were seated comfortably in front of a visual display with their forearm and hand attached to the robot. Care was taken to ensure that the end effectors of the robot were properly aligned with the proximal and middle phalanges of the subjects’ index and middle fingers. Subjects then played through one song with the success rate set to 75% to familiarize themselves with the game. Data from this initial trial was not included in the analysis.

After the familiarization trial, the robot was locked in a neutral position and used to measure the maximum isometric force that the subjects could produce in both flexion and extension. In similar fashion, the robot was used to measure the subjects’ range of motion. Measurements were taken from the index finger, the middle finger, and both fingers together.

Once the initial measurements were complete, subjects were asked to play through the same song twice at each of the three desired success rates (50%, 75%, and 99%). A trial consisted of playing through a song once at a given success rate. The order in which the different success rates were presented to the subjects was randomized. On roughly 15% of the notes within these six songs (randomly selected) the robot would block the subjects’ movements rather than assisting them. During the blocked notes, the control gains on both fingers were set to a consistent value and the force applied by the subjects was taken as a measure of the subject’s engagement in the game. Once a blocked note passed, the gains were returned to their previous values. Notes that were blocked by the robot were not evaluated by the success rate algorithm and were not used to adapt the robot’s gains.

Once these six trials were completed, the subjects’ maximum isometric force and range of motion were measured again using the same process described above.

H. Data Analysis

For each song, the instantaneous success rate was calculated at each note by first defining a moving window containing the 25 most recent notes and then dividing the number of successful notes within that window by the total number of notes within the window. Force applied against the robot during blocked trials was used as a measure of subject engagement. Peak force during each blocked note was normalized by the subject’s maximum force for the corresponding finger as measured during isometric trials. These normalized force measurements were then averaged to obtain an estimate of subject engagement at each desired success rate.

During blocked notes for the index and middle fingers, the robot restricted the motion of both the correct finger and the incorrect finger. An estimate of finger individuation was thus obtained by comparing the force applied by the finger that was supposed to move to that of the finger that was not. Forces measured from both fingers were first normalized by their corresponding maximum force values. These normalized forces were then regressed against one another and the resulting slope was taken as a measure of individuation for a single blocked note. For blocked notes in which the regression showed that the subjects applied more force with the incorrect finger than the correct finger (slope > 1.25) it was assumed that the subjects accidentally tried to hit the wrong note. These blocked notes were not included in the analysis. Similarly, trials in which the subjects did not apply any measurable force with either finger (i.e. missed notes) were not included in the analysis.

III. Results

A. Control Fidelity of the FINGER robot

The Bode plots of FINGER’s frequency response are shown in Fig. 4. The magnitude ratio stays above −3dB until approximately 8Hz. The medial finger-cup of FINGER travelled through about a 10 cm motion for this test, which was performed without a human hand connected. Higher control fidelity would be expected for smaller motions.

Figure 4.

Bode plot of the robot under closed loop control without a human hand connected.

B. Assisting Individuals with Stroke in Playing Guitar Hero

The average success rates in hitting desired notes during Guitar Hero for the 8 stroke subjects and the four unimpaired subjects are shown in Fig. 5. For the stroke subjects, the average actual success rates for songs with desired success rates of 50%, 75%, and 99% were 44%, 74.8%, and 99.6% respectively. For the unimpaired subjects, the actual success rates at convergence were 72.1%, 79.2%, 99%; that is, the unimpaired achieved higher success rates than desired for the lower target success rates.

Figure 5.

Actual success rates of stroke (top) and unimpaired (bottom) subjects for songs with desired success rates of 50% (red), 75% (green), and 99% (blue). Plots to the left show time progression of success rates. Plots to the right show desired vs. actual success rates at convergence.

We used the force generated against FINGER on blocked trials, normalized by the maximum voluntary force, to assess effort during game play. Effort estimates for the four highly impaired subjects, the four moderately impaired subjects, and the four unimpaired subjects are shown in Fig 6. A two factor ANOVA with repeated measures on one factor was used to test the significance of success rate and impairment level on subject engagement. Effort was significantly less for more impaired subjects (p = 0.0006), but did not depend on success rate (p = 0.215). The interaction between impairment level and success rate was not significant (p = 0.555)

Figure 6.

Average normalized force measured during blocked trials at each success rate. A significant difference was found between impairment levels, but not between success rates

We used the correlation between force generated on the incorrect finger and the force generate on the correct finger during blocked trials to assess finger individuation. Finger individuation estimates for the three groups are shown in Fig. 7. A two factor ANOVA with repeated measures on one factor was used to test the effects of impairment level and success rate on finger individuation. Finger individuation decreased (i.e. correlation between finger forces increased) significantly as a function of impairment level (p = 0.048), but success rate and the interaction between impairment level and success rate were not significant (p = 0.922 and p = 0.864 respectively).

Figure 7.

Average finger force correlation measured during blocked trials. Scores close to 1 indicate a high correlation between forces applied by the correct and incorrect fingers. As such, low scores indicate good individuation and high scores indicate poor individuation.

IV. Discussion

This paper described the design and pilot testing of FINGER (Finger Individuating Grasp Exercise Robot). We designed FINGER to be a high-performance robotic platform for implementing and testing control strategies for hand rehabilitation. Our goal is to better identify the behavioral factors associated with training in robotic devices that may help promote functional recovery after stroke. We designed FINGER to have a high level of control fidelity to allow testing of the greatest possible range of training strategies. To achieve these goals, we designed a lightweight planar mechanism to guide each finger through a naturalistic grasping motion, actuated by a backdriveable, low friction, and high-bandwidth linear electric actuator. Closed loop testing with a simple position controller demonstrated that FINGER is able to achieve near 8 Hz tracking prior to a −3 dB loss in magnitude for a large curling motion (path > 10cm). Since human finger motion is bandlimited around 4 Hz [22], FINGER provides control fidelity to keep up with the most rapid human finger movements.

The control fidelity of FINGER makes it a viable candidate for assisting patients in therapy tasks requiring precise timing. Therefore, we implemented a music game environment similar to Guitar Hero, in which subjects attempt to perform finger movements to match the timing of musical notes. This type of gameplay has the additional advantage of being engaging even for repetitive motions: Guitar Hero is the third most popular video game franchise in History [23].

Since the level of success during game play is likely an important factor that influences effort and engagement during movement training [24], we devised a way to control success at the Guitar Hero game with FINGER. Specifically, during game play, we used FINGER to provide assistance to the subjects based on their in-game performance. By modulating the gains of a standard position feedback controller, we accurately controlled the stroke subjects’ success level. However, the unimpaired subjects achieved higher success levels than desired. This is likely due to a combination of the subjects’ inherent ability to perform the task without assistance and the fact that FINGER was programmed to assist subjects rather than resisting them. Future research should study how the difficulty of the Guitar Hero task can be increased for subjects that do not require physical assistance so that greater challenge can be introduced into the task when steady-state success rates are too high.

Given a way to control success levels, we hypothesized that the level of success that subject’s experienced would modulate their engagement in the task. Specifically, based on previous studies that found that individuals with a neurologic impairment slacked when robot therapy devices over-assisted their movements [12, 13], we expected that subjects would exert less effort if they were too successful at the task. We confirmed this hypothesis for the unimpaired subjects, but not for the stroke subjects. One possible explanation is that the lower resolution of hand motor control in individuals with stroke limited their ability to vary their output; i.e. they exhibited “all” or “nothing” motor behavior of the hand in response to the demand for a rapid sequence of individuated finger movements.

We did find that more impaired subjects in general exerted less effort in FINGER. A possible explanation is that the song we selected required notes to be frequently played (about 1 note every 1.5 second). More impaired subjects appeared to have slower finger flexion force relaxation times, and this may have caused them to exert less effort, in order to be able relax the finger to the home position quickly enough to be ready for the next note. Future studies will test this possibility by varying the frequency of notes that the subjects experience.

Since FINGER allows individual fingers to move independently, it has the ability to assess finger individuation. We assessed finger individuation during game play by periodically blocking the subjects’ movements and correlating the force output between the fingers when motion in only one finger was required by the game. The results show that more impaired subjects individuated their finger movement less. Given such a real-time measure of finger individuation, it may be possible to feed back this measurement to the subjects in order to encourage development of isolated finger control. We plan on improving FINGER by including additional actuation for the thumb and remaining fingers, as well as force sensing to allow for impedance control to reduce friction, lower apparent mass, and improve backdriveability. Such improvements will also enhance the ability of the device to measure neuromuscular hand impairment, and to provide multiple-finger grasp training. Furthermore, FINGER will allow a wide range of training algorithms to be implemented and tested in order to investigate which robot control characteristics most improve movement recovery after stroke.