NAO robot for cooperative rehabilitation training

Abstract

Introduction

The aim of this research is to develop a robot-assistive training approach for the disabled individuals with impaired upper limb functions. People with impaired upper limb function can regain their motor functionality undergoing intense rehabilitation exercises. With increasing number of disabled individuals, we face deficiency in the number of expert therapists. One promising remedy could be the use of robotic assistive devices.

Method

To instruct and demonstrate rehabilitation exercise, this research used NAO robot. A library of recommended rehabilitation exercises involving shoulder (i.e., abduction/adduction, vertical flexion/extension, and internal/external rotation), and elbow (i.e., flexion/extension) joint movements was formed in Choregraphe (graphical programming interface). For this purpose, a kinematic model of human upper-extremity was developed based on modified Denavit-Hartenberg notations.

Result

In experiments, NAO robot gave voice instruction and was maneuvered to cooperate and demonstrate the exercises from the library. NAO also plays some complex game with the subject that represents a multi-joint movement's exercise, which was also included in the library.

Conclusions

Experimental results with healthy participants reveal that the NAO robot can successfully instruct and demonstrate upper-extremity rehabilitation exercises for single and multi-joint movements. It implies a technical development of cooperative rehabilitation system for which target group will be individuals with upper limb impairment.

Background

Loss of upper limb function either full or partial is a very common impairment due to geriatric disorders and/or following a stroke or other conditions such as trauma, sports injuries, occupational injuries, and spinal cord injuries. In the past two decades, the total number of stroke-affected people, survivor, related death, and the overall burden of stoke is greatly increasing. As such, strokes have become a major cause of disabilities worldwide.¹ Stroke is the second-leading cause of death worldwide. In the United States, approximately 795,000 strokes occur each year; on average, for every 40 s, someone has a stroke, and for every 4 min, someone dies of a stroke.² Stroke plays an important role in the global burden of diseases and creates an emotional and financial burden for the sufferer and their relatives.^3,4 It is estimated that 85% of stroke survivors suffer arm impairment and 40% are chronically impaired, which makes a burden for the family and the communities as well.⁵ Based on studies, it is found that rehabilitation programs are the main key to regain the motor functionality of disabled people. Intensive and repetitive therapies are one of the major strategies for improving the motor skill significantly.⁶

Numerous rehabilitation hypotheses exist as to how upper extremity impairment may be improved. Passive rehabilitation therapy shows promising improvement to the people who lost partial functionality of their upper limb. It does not contribute to muscle building but helps to increase the range of motion, prevent muscle contraction, and improve mobility of the patient.⁷ After diminishing the resistance to the passive arm movement, individuals have to practice active movement. In this situation, a therapist guides the patient to perform a variety of functional tasks independently, such as grasping and reaching movement (also termed as “active rehabilitation therapy” mode). Studies reveal that this enhanced motor learning.^8,9

However, with this increasing number of people with disability, only 30% of stroke survivors in the United States can receive rehabilitation.¹⁰ On the other hand, the number of rehabilitation facilities is not sufficient to facilitate a large number of people.¹¹ Therefore, robot-assistive rehabilitation devices could significantly contribute to facilitating rehabilitation for such an increasing number of people with disability. The use of the assistive robot in upper extremity rehabilitation has already been emerging and developing as many potential advantages appear out there. To date, the researcher has developed plenty of robotic devices for the rehabilitation of patients with upper extremity mobility impairment (e.g. ETS-MARSE,¹² CADEN-7,¹³ LIMPACT,¹⁴ RAD-HR,¹⁵ etc.).^16–19 For instance, Rahman and coworkers²⁰ have developed a robotic exoskeleton which can use muscle activity (electromyogram signal) to detect the intention of the patient and use it to control the given therapeutic movement. Now researchers would like to extend the robotic devices’ role from assisting in rehabilitation to working as a coach and do the therapeutic role as human–computer interaction has been shifting to human–robot interaction. One of the reasons behind this shift is that using a robot in therapeutic role keeps patients motivated. The study finds socially assistive robotics promising and positive interaction response.^21–24

In general, rehabilitation robotics has mainly focused on orthoses which required physical interaction with an individual with disability. Such as InMotion2 developed by Interactive Motion Technologies, Inc., Boston, MA (the commercially available version of the MITManus), is used for upper limb rehabilitation using reaching movement.²⁵ The Mirror Image Movement Enabler was the first robot-assistive device used for combined unilateral/bilateral therapy and was able to reduce abnormal synergies.²⁶ For bilateral therapy of wrist and forearm, Reha-Stim, Berlin, provides Bi-Manu-Track robotics device.²⁷ The Assisted Rehabilitation and Measurement guide provides both assistive and resistive reaching therapy for the patient and improves better functionality.²⁸ In GENTLE/s system, a HapticMASTER robot was used. The experimental results evidenced a greater improvement rate of active range of motion and post-stroke motor recovery for upper arm.²⁹ For decreasing spasticity of the upper limb, REHAROB system provides augmented rate of improvement.³⁰ None of these systems facilitate verbal communication with the participants. All those systems basically work based on user’s input force/torques (detected by force sensors). Such assistive device produces similar clinical outcomes those obtained using human-administered therapies.

In recent decade, the use of socially assistive robots in cooperative rehabilitation increases significantly. TAIZO,³¹ an exercise demonstrator robot, was developed to help human demonstrators during simple arm exercises with a training group. Fasola and Matari³² developed a socially assistive robotic exercise coach that effectively motivated and engaged elderly users to perform physical exercises. Their experimental results have strongly justified a clear preference to use a real robot coach over a virtual coach in terms of enjoyableness, helpfulness, and social attraction. Eriksson and coworkers³³ developed a socially assistive robotic system for stroke and mild traumatic brain injury rehabilitation. Their pilot test results validated user acceptance of the robot. Bhuvaneswari et al.³⁴ developed a physiotherapeutic-assistive trainer using a humanoid robot NAO that can teach elderly individuals different physiotherapy exercises like shoulder bracing exercises, isometric neck exercises, and isometric knee exercises. Magyar and coworkers³⁵ developed a movement rehabilitation trainer using a humanoid robot NAO controlled by an android smart phone. Simonov and Delconte³⁶ developed an approach in which a humanoid robot was used to assist individuals with a disability in home settings to perform rehabilitation exercise. In this approach, rehabilitation exercises are encoded as formal knowledge on the robot. It provides better sustainable rehabilitative care services in remote settings. Several other kinds of literature provide evidence that socially assistive robots can significantly be used for social skill training, daily life assistance, elderly care, and physical therapy.^37–41

To contribute in this area, we have developed a socially assistive robotic system for cooperative rehabilitation training. We define cooperative rehabilitation, where a participant actively participates in the rehabilitation process, e.g. in our proposed cooperative rehabilitation training program, a participant can verbally communicate with the robot and can request the robot to demonstrate the exercise(s) he/she prefers. Once the robot gets the request, it demonstrates the exercice(s) from its library of exercises named “rehab robot exercise library.” For this purpose, we have used a humanoid robot named NAO. Literature reviews reveal that the real robot provides better performance to interact with human compared to a virtual agent.^42,43 Such a non-contact system can demonstrate rehabilitation exercise and verbally communicate with the users. Due to the lack of physical contact, such a system requires minimum safety concern and increased accessibility.⁴⁴ Whereas, verbal communication is highly effective for communicating emotion and provides ground for social interaction.^45,46 In our social-assistive robotic system, the NAO robot has been used to instruct and demonstrate the exercise to the people with partial loss of upper extremity functionality. The humanoid robot NAO is able to interact verbally and can perform complex manoeuver. A set of rehabilitation exercises like daily activities and some complex interactive games were built in Choregraphe as a behavior for NAO. Details about the NAO robot, library building, and behavior implementation using Choregraphe are presented in the “Methods” section. The next section presents the results and discussion. The paper ends with the conclusion and future improvements are in given in the “Conclusion” section.

Methods

In this research, we hypothesize that a humanoid robot named NAO can be effectively used to instruct and demonstrate rehabilitation exercise for upper limb impairment and verbally communicate with the participants. This leads to the technical development of a cooperative rehabilitation system for which the targeted group will be the individuals with upper limb impairment. For this purpose, first, we have developed a variety of motion trajectories that represent human upper limb rehabilitation exercises. Based on these trajectories (exercises), in the next step, we developed NAO’s exercise library named “rehab robot exercises library” in Choregraphe. In addition, we have developed a code (communication instruction) in Choregraphe for NAO to communicate with the participants. The developed Choregraphe model was implemented on a NAO robot as a “NAO’s behavior.” To evaluate the performance of the developed cooperative rehabilitation system, we have tested NAO’s communication skills and exercise demonstration skills with a healthy male adult participant, where a participant verbally requests NAO to demonstrate a variety of exercises. The benchmark of efficient performance of NAO was how well the NAO could interact with a participant and follow the commands to demonstrate the exercises.

Humanoid robot NAO

The current research employs one of the most promising autonomous programmable social robot NAO, developed by Aldebaran Robotics. The NAO robot is equipped with different sensors that provide different communication features such as vision, speech, hearing, and touch-sensing. The 25 degrees of freedom (Figure 1) provide NAO the capability to mimic almost all human-like movement. Due to such features, NAO can be used in a multitude of the research environment, including assistive robotics,^42,47 healthcare robotics,^34–36 educations robotics,^48,49 etc. NAO is equipped with two cameras, four microphones, two loud speakers, nine tactile sensors, and eight pressure sensors (Figure 2).⁵⁰ This work utilized NAO V5, which is 574 mm tall and 275 mm width with 5.4 kg body weight. Its upper arm length is 105 mm and lower arm length is 55.95 mm. Its thigh length is 100 mm, tibial length 102.90 mm, and foot height 45.19 mm. In this research, we mainly focused on upper extremity rehabilitation scheme, in particular, the right upper arm of the NAO (Figure 3). NAO’s upper arm joints’ range of motions is different from that of human upper limb joints (Table 1). For example, NAO’s elbow joint range of motion is less compared to the human elbow joint motion. In addition, its shoulder joint roll (abduction/adduction) motion range is smaller compared to that of human shoulder joint motion. On the other hand, NAO’s shoulder pitch (vertical flexion/extension) range of motion is larger compared to the human shoulder joint pitch motion. Human upper limb joints motions are depicted in Figure 4.

Figure 1.

All joints in NAO robot and initial position.⁵⁰

Figure 2.

All sensors in NAO robot.⁵⁰

Figure 3.

Right arm joints and range of motions.⁵⁰

Figure 4.

General motion: (a) shoulder flexion/extension (shoulder pitch); (b) shoulder abduction/adduction (shoulder roll); (c) elbow flexion/extension (elbow roll); and (d) shoulder internal/external rotation (elbow yaw).

There are several ways to build behavior for NAO robot. Choregraphe is a high-level block-based programming environment for this purpose.⁵² One can also use other programming languages, such as C++, Python, among others, for which appropriate application programming interface is available.

Table 1.

Upper extremity range of motion of NAO robot (SoftBank Robotics)⁵⁰ and human (Winter, 1990).⁵¹

Joint name	Motion	Range for NAO	Range for human51
RshoulderPitch	Right shoulder joint front and back (Y)	−119.5 to 119.5	−150 to +30
RshoulderRoll	Right shoulder joint right and left (Z)	−76 to 18	−50 to +180
RElbowRoll	Right elbow joint (Z)	2 to 88.5	0 to +150
RElbowYaw	Right shoulder joint twist (X)	−119.5 to 119.5	−90 to +15

Choregraphe: A behavior-based software platform

Choregraphe is a multi-platform desktop application that is able to create complex behaviors using a set of basic behavior blocks. It is a user-friendly graphical programming interface, where different functionalities of the NAO are represented as behavior blocks. Each block represents a specific task for NAO. Multiple tasks can be combined to generate a new behavior block. Similar to LabVIEW⁵³ and Simulink,⁵⁴ Choregraphe employs the concept of signal flow and executes behavior blocks in the order they are connected to each other. Blocks in Choregraphe provide access to all sensors and actuators of NAO. Choregraphe also facilitates access to the NAO memory. Using programming language Python with the provided software development kit, it is possible to create a new behavior block in Choregraphe. To perform the simulation, it provides a virtual NAO robot. Some advance behavior, such as speech recognition, face recognition, learning face, and detect face, is also provided in Choregraphe. The workflow in Choregraphe follows a parent–child relation. Each block has input and output ports. They are connected through lines/wires and program flow in sequential order. Another important functional block library in Choregraphe is the “Timeline” block. Using this block, each motor was controlled in a time frame along with executing different other functional blocks. The joint trajectory for different types of motion exercises was generated using human arm kinematic and set those in a library. There also exist some conditional blocks such as “if-else,” “for loop,” “switch,” etc., which are used to generate a cooperative rehabilitation exercise library. Figure 5(c) is an example of a behavior created in Choregraphe. If we execute this behavior, “stand-up” block will be executed first which will result in NAO to stand-up. Then, “Say” and “Wipe Forehead” blocks will be executed simultaneously which will result in NAO to speak and wipe its forehead simultaneously. Next, the output will travel to “Hello” block. This “Hello” block will command NAO to say Hello and to waive its hand. Once “Hello” block is executed, NAO will exist the behavior.

Figure 5.

(a) Choregraphe programming interface; (b) functions blocks in Choregraphe; (c) program flow in Choregraphe.

Trajectory generation for human upper extremity

To develop rehabilitation exercise for human upper extremity, the trajectory for different arm movements needs to be generated. For this reason, kinematics of upper extremity should be analyzed. For kinematic modeling of the human upper extremity, consider a serial robotic manipulator with five degrees of freedom (shoulder abduction/adduction, vertical flexion/extension, internal/external rotation, elbow flexion–extension, and forearm pronation–supination). Modified DH conventions were used to develop the kinematic model of the human upper limb, and coordinate frames were assigned in every joint.⁵⁵ As shown in Figure 6, the joint axes of rotation of the human right upper limb are indicated by dark black arrow heads (i.e. z_i). In this model, joints 1 and 2 together constitute the shoulder joint, where joint 1 corresponds to abduction/adduction; joint 2 represents vertical flexion/extension; joint 3 corresponds to internal/external rotation of the shoulder joint; and joint 4 represents the flexion/extension of the elbow joint. The elbow joint is located at a distance $d_{\mathit{upper}\_\mathit{arm}}$, and wrist joint is located at a distance $d_{\mathit{forearm}}$. The modified DH parameters corresponding to the placement of the link frames (in Figure 6) are summarized in Table 2.

Figure 6.

Link frame attachments to the human right limb.

Table 2.

Modified Denavit–Hartenberg parameters.

Joint (i)	αi −1	di	ai −1	θi
1	0	0	0	θ1 − π/2
2	π/2	0	0	θ 2
3	π/2	dupper_arm	0	θ 3
4	−π/2	0	0	θ 4
5	π/2	dforearm	0	0

We know that the general form of a link transformation that relates frame $\{i\}$ relative to the frame $\{i-1\}$ is

$$\begin{array}{c}ii-1T=[\begin{array}{cc}ii-1R^{3\times 3}& \hfill ii-1P^{3\times 1}\hfill \\ 0^{1\times 3}& \hfill 1\hfill \end{array}]\end{array}$$ (1)

where $ii-1R$ is the rotation matrix that maps frame $\{i\}$ relative to frame $\{i-1\}$ and can be expressed as

$$\begin{array}{c}ii-1R=[\begin{array}{cc}\text{cos}{\theta }_i& \hfill -\text{sin}{\theta }_{i}0\hfill \\ \text{sin}{\theta }_i\text{cos}{\alpha }_{i-1}& \hfill \text{cos}{\theta }_i\text{cos}{\alpha }_{i-1}\hfill & \hfill -\text{sin}{\alpha }_{i-1}\hfill \\ \text{sin}{\theta }_i\text{sin}{\alpha }_{i-1}& \hfill \text{cos}{\theta }_i\text{sin}{\alpha }_{i-1}\hfill & \hfill \text{cos}{\alpha }_{i-1}\hfill \end{array}]\end{array}$$ (2)

and $ii-1P$ is the vector that locates the origin of frame $\{i\}$ relative to frame $\{i-1\}$ and can be expressed as

$$ii-1P=[\begin{array}{ccc}{a}_{i-1}& \hfill -s{\alpha }_{i-1}{d}_i\hfill & \hfill c{\alpha }_{i-1}{d}_i\hfill \end{array}{]}^T$$ (3)

Using equations (1) to (3), the individual homogeneous transfer matrix that relates two successive frames can be found.

The homogenous transformation matrix that relates frame $\{5\}$ to frame $\{0\}$ can be obtained by multiplying the individual transformation matrices

$$50T=[10T·21T·32T·43T·54T]$$ (4)

The single transformation matrix thus was found from equation (4) represents the positions and orientations of the reference frame attached to the wrist joint (axis 5) with respect to the fixed reference frame {0}. The equation obtained from this transformation matrix is known as the forward kinematics equation. If the joint variable of each joint (${\theta }_1,{\theta }_2,{\theta }_3,\text{\hspace{1em}and\hspace{1em}}{\theta }_4$) is given, then from this forward kinematics equation, the position of list frame can be determined.

Results and discussion

The exercises formed in Choregraphe can be grouped under three categories: “single joint movement,” “multi joint movements,” and “co-operative exercise.” NAO will instruct and demonstrate participants to perform those exercises. A typical NAO’s instruction is given below

• {

• NAO: Hello friend,

• NAO: Let’s get ready. Stay normal and don’t worry. I am here for you mate. Believe me, it’s going to be fun.

• NAO: Anyway, we will do some exercise together

• NAO: In this session, I will show you how to do elbow flexion/extension exercise in a minute. After that, you will be asked to do perform the exercise…

• ⁞

• }

Single joint movements

Single joint movement exercises include shoulder joint abduction/adduction, shoulder joint vertical flexion/extension, shoulder joint internal/external rotation, and elbow joint flexion/extension motion. Since NAO does not have wrist joint flexion/extension and radial/ulnar deviation, we have excluded wrist joint motions from this study.

Figure 7 shows the experimental results of shoulder joint abduction/adduction where a coordinated movement of shoulder horizontal and vertical flexion/extension motion was performed. The top plot of Figure 7(a) shows NAO’s abduction/adduction angle as a function of time. The bottom plot of Figure 7(a) shows the joint velocity. As shown in Figure 7(b), the exercise began with NAO’s adduction angle 0°, and then abduction motion was performed; finally, the exercise ends with the NAO’s adduction to 0°. Maximum abduction angle observed in this case was −75°.

Figure 7.

(a) Shoulder joint abduction/adduction motion (joint trajectory and velocity); (b) abduction/adduction motion of NAO with human.

A typical rehabilitation exercise involving shoulder joint flexion/extension movement is depicted in Figure 8 where the NAO raises its arm (from the initial position, i.e. shoulder joint-2 is at 90°) straight up to a specific position over the head (the elevation was set to a range of 180°, i.e. from 90° to −90°) and then slowly moves the joint back to its initial position.

Figure 8.

(a) Shoulder vertical flexion/extension motion (joint trajectory and velocity); (b) vertical flexion/extension motion of NAO with human.

Depending on the participants, it is often required to change the speed of such exercises. Figure 9 shows the similar exercises that were performed with different speeds of motion.

Figure 9.

Shoulder joint vertical flexion/extension motion (fast movement).

Figure 10 demonstrates a co-operative movement of the elbow (flexion/ extension) and shoulder joint (internal/external rotation). The objective of this exercise is to demonstrate shoulder joint internal/external rotation while maintaining the elbow at 90°. As shown in Figure 10, the exercise begins with elbow flexion at 90°, and then internal/external rotation was performed.

Figure 10.

(a) Shoulder joint internal/external rotation motion (joint trajectory and velocity); (b) shoulder joint internal/external rotation of virtual NAO; (c) Shoulder joint internal/external rotation motion of NAO with human.

A typical rehabilitation exercise involving elbow joint flexion/extension movement is depicted in Figure 11. The exercise began with the elbow joint at 4°, and then a repetitive flexion/extension motion was performed.

Figure 11.

(a) Elbow joint flexion/extension motion (joint trajectory and velocity); (b) elbow joint flexion/extension of virtual NAO.

Multi joint movements

Multi joint movement exercises include a combination and co-operative movements of shoulder joint (abduction/adduction, vertical flexion/extension, internal/external rotation), and elbow joint (flexion/extension) motion.

Reaching movements are widely used and recommended for multi joint movement exercises.⁵⁶ A forward reaching movement and a diagonal reaching movement exercise are depicted in Figures 12 and 13, respectively.

Figure 12.

(a) Forward reaching movements (joint trajectory and velocity); (b) forward reaching movements performed/demonstrated by NAO.

Figure 13.

(a) Diagonal reaching movements (joint trajectory and velocity); (b) diagonal reaching movements performed/demonstrated by NAO.

Forward reaching movement involves shoulder joint flexion/extension motion and elbow joint flexion/extension motion. Experimental results of forward reaching movement showing the elbow and shoulder joint angles are illustrated in Figure 12.

Diagonal reaching movement involves shoulder joint flexion/extension motion, shoulder joint abduction/adduction, and elbow joint flexion/extension motion. Experimental results of diagonal reaching movement showing the elbow and shoulder joint angles are illustrated in Figure 13. Typically, this exercise is repeated approximately 10 times; therefore, a few repetitions are depicted in Figure 13. NAO will instruct the participants to perform a repetitive motion of this exercise.

Cooperative exercise

Cooperative exercises in NRL involve NAO’s interaction with participants. Figure 14 shows the schematic diagram of a cooperative exercise “touch and play,” where points A, B, C, and D represent NAO’s hand position in 3D space at different time. The objective of this exercise is to reach different targets one after another which involve the movement of the entire upper limb’s joints. Figure 15 shows a few snapshots while NAO was playing “touch and play” game with the participant. Experiments were conducted with a healthy participant in the seated position. In this experiment, NAO instructs the participant to touch its hand in a 3D space (e.g. point A, Figure 15(b)). When the participant touches NAO’s hand, it can sense it with its tactile sensor, move its hand at a new position, and ask the participant to touch its hand again.

Figure 14.

Schematic of “touch and play” exercise.

Figure 15.

Cooperative exercise with NAO robot.

Finally, to perform more complex cooperative exercises with NAO, we combined all the functional behaviors (single joint movement exercises, multi join movement exercise, co-operative movement exercise) described earlier. A sample of such cooperative exercise (developed in Choregraphe) is shown in Figure 16.

Figure 16.

NAO’s behavior programming for single and multi-joints movements and cooperative tasks.

Conclusion

From the experimental results, it can be concluded that the NAO robot can be used as a cooperate trainer for rehabilitation therapy for the patient with partial disability. Rehabilitation exercise library developed in this study can be executed anytime and a multitude of the environment using the NAO robot. User-friendly Choregraphe software makes it possible to introduce new exercises in this library. To perform a training session, NAO can be operated verbally which means anyone with no expertise with NAO robot can be able to use NAO robot for instructing and demonstrating upper-extremity rehabilitation exercises for single and multi-joint movements. Such nonphysical contact-assistive system requires to vary little concern about safety. As a socially assistive robot, NAO can facilitate an effective means for rehabilitation.

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

Guarantor

AZ

Contributorship

MA conducted the overall study, data collection, analysis, and interpretation of the results. MRI contributes as a study participant as well as performing the data collection, analysis, and manuscript editing. SM is the co-supervisor of the study and works on manuscript editing. MHR supervised the overall study and manuscript writing. All authors read and approved the final manuscript.