Development of an Interactive Upper Extremity Gestural Robotic Feedback System: From Bench to Reality

Abstract

Development of an interactive system to treat patients with movement impairments of the upper extremity is described. Gestures and movement of patients as instructed by therapists are detected by accelerometers and feedback is provided directly to the patient via a robot.

I. Introduction

Therapy to help patients with upper extremity movement impairments requires focused practice and uses repetition to change motor control patterns and extremity use. Physical and occupational therapy can address issues such as lack of coordination, delayed development of gross/fine motor control and difficulties with activities of daily living. Creative use of ordinary objects such as blocks, puppets and balls help therapists engage their patients in performing a task which is therapeutically beneficial. These toys, unfortunately, have no inherent therapeutic value and patients, particularly children, become bored with both using the same object and repetition of the same task making it difficult for therapists to keep the children engaged in their therapy. This is a problem of significant magnitude as it is estimated that over 10% of all school children have one or more physical or learning disabilities (5.2 million children [1]) with about 10,000-12,000 new cases of cerebral palsy annually [2], [3]. Clearly, there is a need for an upper extremity treatment system which can assist therapists to engage their patients in therapeutic movements.

II. Historical Perspective

Robotic systems designed to support productivity of clinicians by providing movement therapy (such as the MIT-MANUS) [4] in an effort to train patients by forced repetition and supporting movement error reduction have proven useful and represent an improvement over assistive technologies which merely support or assist movement. A feedback system based on sensors which respond to specific gestures or movements produced by patients as directed by their physical or occupational therapist was proposed as a potential solution. Other systems utilizing feedback from gestural interfaces which have been used in rehabilitation settings are tethered to a fixed location such as SenseAble [5] Technologies (Woburn, MA) or focus more on whole body movements like GestureTek [6] Health (Toronto, ON) and EyeToy [7] for Sony PlayStation2 (New York, NY) or focus on shifting the center of mass like the platform based NeuroCom [8] (Clackamas, OR). We desired a system which could be used in the many different environments where upper extremity therapy traditionally occurs, such as schools, homes, and clinics. In addition, we desired a system which could be used by children with limited ability to interact with the visual images on a computer monitor. The CosmoBot Learning System (AnthroTronix, Silver Springs, MD) is a portable, high-tech, integrative system originally designed and developed for therapy, education and play (Fig. 1). The system was developed over a five year period through an iterative process which focused on the needs of children with disabilities, their parents, teachers and clinical professionals. Consideration was given to the needs of all stake holders with the overall goal to create a motivational tool for use in education and physical, occupational and speech therapies. The core component of the CosmoBot Learning System was a robot designed in consideration of parameters outlined by Woods, et. al. [9], as an interactive tool to interface with the child. The robot is controlled via Mission Control, a command unit with four jumbo-sized buttons which provides an interface both with software and the robot. Teachers and therapists can use a microphone to create a ”voice“ for the robot and engage the child in interactive play using the control unit.

Fig 1.

Original CosmoBot™ Learning System which provided a basis from which a robotic feedback gestural therapy system was developed. The original system has a graphical user interface and was used for therapy, education and play.

Good results were reported from initial use of the CosmoBot Learning System as an interaction tool for children with Autism Spectrum Disorders, children in speech therapy and in classrooms assisting children with basic concept such as order (first, second, etc.) and physical location (before, after, behind, etc.). Overall, the therapists, children and parents agreed that the technology was highly motivating and the provision of immediate feedback without the therapist needing to use verbal language as feedback was beneficial.

However, use of the Mission Control unit was a key shortcoming of the original CosmoBot Learning System limiting use for upper extremity therapy. Although the universal accessibility design was easy for patients to use, the motion of pressing the buttons engaged wrist flexion with forearm pronation which is opposite from the direction which is therapeutically needed to support functional use of the wrist, hand and forearm, specifically, wrist extension and forearm supination. A different sensor was required to more fully realize the goal of using a CosmoBot system for treating upper extremity movement impairments.

A second problem was the computer interface for the therapist. Many games within the learning system and computer based components did not allow the therapist to adapt the system for individual patient capabilities or limitations. A different interface was needed to allow therapist to change the thresholds and adapt the system for use with a variety of patients.

Finally, several limitations related to the fragile nature of the robot were experienced during pilot testing. The shell of the robot was prone to cracking as a result of the enthusiastic interactions by children. The drive unit became overburdened with the weight of the overall robot structure and the arms of the robot had limited range of motion due to weight of the servos driving limb movements.

III. Current Configuration

A. Robot

For the upper extremity therapy system, the original CosmoBot robot provided the frame work around which a gestural interface for movement feedback was created. Rather than making the robot mimic movements, the decision was made to mount an inanimate version of CosmoBot onto a mobile base (Fig. 2). A commercially available mobile base, ”Roamer-Too“ (Valiant Technologies USA, Forest Park, IL 60130) was selected to provide movement of CosmoBot. The mobile base is remotely controlled by computer via Bluetooth.

Fig. 2.

CosmoBot™ robot mounted on mobile base.

B. Sensor

A triaxial accelerometer was integrated into the CosmoBot system to allow control of the robot as a feedback type of device. The accelerometer responds to specific gestures or movement produced by patients as directed by their physical or occupational therapist. We focused initially on the movements which therapists indicated are most needed by patients with upper extremity movement impairments secondary to cerebral palsy, brain injury or stroke, specifically, wrist extension and forearm supination. The choice of avoiding any exoskeleton type design was to allow flexibility in use of the system beyond the currently envisioned application.

The accelerometer was embedded in a sealed case for protection of the sensor from any liquid or body fluid contamination and to allow the sensor to be cleaned between patients (Fig. 3). Hook type Velcro was affixed to one side of the sensor box to allow the sensor to be attached to any body location via a cuff made from athletic wrist/forehead sweat bands. The sensor communicates via a USB connection with a laptop computer.

Fig. 3.

Triaxial accelerometer enclosed in sealed case and attached on the dorsum of the hand.

C. GUI

A graphical user interface (GUI) was developed to provide therapists with control of sensor parameters and the CosmoBot robot. One component of the GUI is the sensor definition and set up (Fig. 4). After the therapists determines the therapeutic movement required of the patient for the treatment session, a reference position for the sensor and identification of sensor orientation is determined. Threshold limits are set individually for each of four directions of movement away from the reference position and are mapped to a 3×3 display grid used by the therapists as a quick visual reference of the sensor position with respect to the thresholds. The process of mapping can accommodate both right and left sides of the body. Therapists can return to this screen at any time during the treatment session and reset to a different reference position or change the threshold for any movement direction

Fig. 4.

Image of GUI used to establish sensor reference position and define thresholds for movement to trigger a response from the CosmoBot. For this patient, a motion in supination greater than 30 degrees from the rest position would trigger CosmoBot. Because the threshold for flexion, extension and pronation are beyond the motion limits for this patient, supination, would be the only movement which would control the robot.

A second component of the GUI addresses control of the CosmoBot. Signals are transmitted via Bluetooth from the computer to the mobile base triggering movements of CosmoBot. Two modes of play are available to therapists via the GUI, ”Free Play“ and ”Patterned Play“. During ”Free Play“, each movement of the sensor which results in crossing the predefined threshold, results in a single movement of CosmoBot (a step). Default movement of CosmoBot is forward one length of the mobile base. Turning the robot to the right or left is accomplished using a wireless mouse/trackball. Left clicking the mouse prior to the patient moving the sensor sufficiently to cross the threshold and activate the robot, will result in CosmoBot turning to the left. Right clicking the mouse causes a right turn when the sensor detects movement in the appropriate direction greater than the threshold. During ”Patterned Play“ the therapist must define both the feedback which the robot will provide as well as the task required of the subject. Feedback consists of the CosmoBot moving along a prescribed course, also established by the therapist. Using a 10×10 grid (Fig. 5), therapists define a starting location for the robot and a path in orthogonally directed steps for the robot to move. Each block in the grid represents a step equivalent to one length of the robot base. Turning the robot to move in a new direction is counted as a step. Therapists can link multiple steps so that successful completion of the task provides several movements of the robot as feedback.

Fig. 5.

Image of GUI used to establish track for CosmoBot to follow during the ”patterned play“. In this example, the robot would be taking 27 ”steps“ with turns counted as a step.

For each task, the therapist defines two parameters of sensor positioning which are used to determine the ”go“ – ”no go“ state for the robot (Fig. 6). The first parameter defines the direction of movement required of the patient such as extension or supination. The magnitude of motion required of the patient is set by the threshold as previously described. The second parameter set by the therapist for the task is the length of time the selected motion must be sustained above the threshold for the ”go“ condition to be met. The task is then linked to an action or actions the CosmoBot robot will perform as feedback.

Fig. 6.

Image of GUI used to define feedback action and task required to trigger movement of CosmoBot as it follows the path established in Fig. 5. For this patient, an extension motion sustained for 5 seconds would result in the CosmoBot moving through the first six steps in the track shown in Fig. 5.

D. Clinical Testing

The current configuration was tested in an outpatient rehabilitation clinic. Three children with cerebral palsy (ages 4-11) which affected their upper extremity functioning participated in a pilot project which was approved by the Mayo Clinic IRB. Each child was seen for two-20 minute sessions. The therapist who regularly treated each subject identified a therapeutic goal for that child and defined a specific therapeutic movement (gesture). The sensor was placed and reference position and threshold limits were established. Threshold limits were reset as needed during all treatment sessions to both challenge and allow success of the child in performing their therapeutic gesture.

Subjective data was recorded from the patients, their parents and the therapists providing the intervention. Considerations in evaluating the system included 1) robustness; 2) usability by therapist; 3) child motivation; and 4) efficacy in treatment. Compared with the previous version of CosmoBot, the current configuration is considered more robust as repair was required only once over a 6 month period (with usage approximately 5 hours/week including patients outside of this pilot study). Therapists related that the system was easy to use and the computer interface was intuitive. Therapists were very complimentary about the flexibility in how the system could be used to meet a variety of therapeutic goals and was useful with patients of all ages. As an indication of motivation, therapists and parents identified that the children in the study remained engaged in their therapy and performed the requested gestures more times when using the CosmoBot feedback system as compared with their conventional therapy sessions. Therapists related that during conventional therapy sessions, the children frequently required redirection to the therapeutic tasks which was not needed at all during the robotic feedback sessions. Examples of improved functional task performance by the children in the pilot project included a child being able to consistently maintain wrist extension to allow use of finger flexors to manipulate blocks and another child who was able to perform forearm supination allowing them to open doorknobs.

IV. Conclusion

This use of a robotic feedback system for habilitation/rehabilitation represents a creative use of robotics to facilitate patient's movement rather than supplant patient movement with robotically assisted movement. The current configuration of the CosmoBot is being used in a rigorous clinical trial for patients with brain injury, cerebral palsy or stroke needing upper extremity intervention. Future reports will provide quantitative measures of movement and strength and objective measures of functional ability acquired in a repeated measures crossover study.