Mechanism of Kinect-based virtual reality training for motor functional recovery of upper limbs after subacute stroke

Abstract

The Kinect-based virtual reality system for the Xbox 360 enables users to control and interact with the game console without the need to touch a game controller, and provides rehabilitation training for stroke patients with lower limb dysfunctions. However, the underlying mechanism remains unclear. In this study, 18 healthy subjects and five patients after subacute stroke were included. The five patients were scanned using functional MRI prior to training, 3 weeks after training and at a 12-week follow-up, and then compared with healthy subjects. The Fugl-Meyer Assessment and Wolf Motor Function Test scores of the hemiplegic upper limbs of stroke patients were significantly increased 3 weeks after training and at the 12-week follow-up. Functional MRI results showed that contralateral primary sensorimotor cortex was activated after Kinect-based virtual reality training in the stroke patients compared with the healthy subjects. Contralateral primary sensorimotor cortex, the bilateral supplementary motor area and the ipsilateral cerebellum were also activated during hand-clenching in all 18 healthy subjects. Our findings indicate that Kinect-based virtual reality training could promote the recovery of upper limb motor function in subacute stroke patients, and brain reorganization by Kinect-based virtual reality training may be linked to the contralateral sensorimotor cortex.

INTRODUCTION

Although stroke mortality is declining with the improvement of medical technology, this has led to an increased number of motor deficits in stroke survivors[1,2,3], especially affecting the function of the upper limbs. The most common measure is physical training. However, is consistent training beneficial for patients? Unfortunately, no significant improvement in upper limb function has been found after repetitive task training in previous studies, suggesting that further research should focus on the type and amount of training, and how to maintain functional gain[4].

Generally speaking, one therapist typically conducts rehabilitation activities with many patients at the same time. Patients often lack the enthusiasm to participate in the tedious rehabilitation process, resulting in continued muscle atrophy and insufficient muscle endurance. How can we design an optimal rehabilitation measure that maintains patients’ interest and achieves effective results?

Microsoft released an AVG console – the Xbox Kinect™ – in 2010, which is a commercial video game system, provides full-body control of animated virtual characters, and is a noninvasive, inexpensive virtual reality technique used to encourage people with motor disabilities to exercise repeatedly and actively[5,6,7,8,9,10]. Our study applied a Kinect-based rehabilitation system to assist therapists in rehabilitating stroke patients. Kinect enables users to control and interact with the game console without the need to touch a game controller, through a natural gesture-based user interface. The device comes with an RGB camera and a depth sensor, which in combination provides full- body three-dimensional motion capture capabilities and gesture recognition. A major advantage of the Kinect is that it allows patients to train at home or even at the office, thus allowing users the freedom to engage with complementary solutions to their physical impairments without requiring them to attend hospitals. Unlike several videogames available, this tool offers patients an interface for customizing posture parameters, so it can either be used by rehabilitation professionals or by the user through their guidance.

Popular methods of enhancing upper limb function include constraint-induced movement therapy[7] and feedback therapy[8]. More recently, the focus has been on virtual reality, a computer-generated environment in which people can feel and interact with various situations in three dimensions[9], which has been used in a neurological rehabilitation population to improve upper limb and global motor function after stroke[10]. Some studies showed that virtual reality training significantly improved motor impairment as measured by the Fugl-Meyer Assessment[11,12] and the Wolf Motor Function Test[13]. This evidence suggests that virtual reality techniques could improve motor function and the flexibility of the affected upper limbs. However, most studies recruited patients more than six months after stroke. In addition, the mechanisms of neuroplasticity used by virtual reality remain unknown. Some scholars have also studied its effects on balance rehabilitation[14]. In our study, Kinect was employed to investigate virtual reality mechanisms for the recovery of motor disabilities in subacute stroke patients.

Functional MRI (fMRI) can be used to study alterations in cortical and subcortical activation patterns in stroke patients[15,16]. This technique may therefore significantly contribute to the understanding of the neural reorganization that underlies functional recovery, predict outcomes, and monitor therapeutic strategies that promote brain repair[17]. However, to our knowledge, there is no study focusing on the use of fMRI to evaluate the effects of Kinect virtual reality training on subacute stroke.

The objective of this study was to 1) observe changes in activated brain regions in fMRI of hand-clenching in subacute stroke patients; 2) evaluate the effectiveness of Kinect-based virtual reality training on the recovery of upper limb function in subacute stroke patients; and 3) locate the target brain region for Kinect-based virtual reality intervention.

RESULTS

Quantitative analysis of subjects

Eighteen healthy people and five patients with subacute stroke were included in this study. All subjects were involved in the final analysis without dropout. Clinical data of the recruited patients are shown in Table 1.

Table 1.

Clinical data of the recruited patients

Behavioral changes of stroke patients after Kinect-based virtual reality training

Fugl-Meyer Assessment score and Wolf Motor Function Test score in patients increased (P < 0.05), while their log10 Wolf Motor Function Test time decreased at 3 weeks after training and at the 12-week follow-up (P < 0.05). There were significant differences in the Fugl-Meyer Assessment score, Wolf Motor Function Test score and log10 Wolf Motor Function Test time across the three groups (P < 0.05). This evidence indicates that Kinect-based virtual reality training improved motor recovery in stroke patients. The behavioral findings are shown in Table 2.

Table 2.

Effect of Kinect-based virtual reality training on Fugl-Meyer Assessment (FMA) and Wolf Motor Function Test (WMFT) results of stroke patients

Activated brain regions of stroke patients after Kinect-based virtual reality training

A statistically significant cortical activation pattern could be observed in all 18 healthy subjects in contralateral primary sensorimotor cortex, bilateral supplementary motor area and ipsilateral cerebellum (Table 3, Figure 1).

Table 3.

Regions of interest (ROI) in the brain that are activated during fist-clenching with the left or right hand in 18 healthy subjects

Figure 1.

Brain regions activated during fist-clenching with left or right hand in 18 healthy subjects using functional MRI.

(A) Averaged functional MRI activation maps of fist-clenching with left hand. Right side of the image corresponds to the left side of the brain.

(B) Averaged functional MRI activation maps of fist-clenching with right hand. The activated brain regions during fist-clenching include contralateral sensorimotor cortex, bilateral supplementary motor areas and ipsilateral cerebellum.

In this study, we found that there were no statistically significant differences due to age or sex among the healthy subjects in our study (aged 49 to 72 years old). This result allowed us to consider the healthy people as a group, rather than as individuals, and to establish a standard brain for this age range. The five stroke patients and 18 healthy people were in the same age group (aged 49 to 71 years). Changes were also noted in the blood- oxygen-level dependent (BOLD) signal volume of the five stroke patients between the pre-training, 3-week, and 12-week follow-up fMRI scans (Table 4, Figure 2) and were compared with those of the average healthy age- matched subjects. The differences of time-point fMRI images revealed significant changes. 1) The main activation areas were similar to average healthy age- matched subjects’ brain activation maps during hand- clenching movements under fMRI, such as primary sensorimotor cortex, supplementary motor area and cerebellum. 2) Primary sensorimotor cortex cluster sizes in patients 1–4 showed gradual increases, while those of patient 5 gradually decreased. 3) At the 12-week follow-up, the primary sensorimotor cortex cluster size of patients 1 and 3 were smaller than the averaged 18 healthy age-matched subjects, while that of patients 2, 4 and 5 were larger. 4) Primary sensorimotor cortex T-values from patients 1–4 gradually increased, while those of patient 5 gradually decreased. All T-values were significantly larger than the average 18 healthy subject benchmark. 5) No significant regularity was found for brain activation in the bilateral supplementary motor area or the ipsilateral cerebellum.

Table 4.

Regions of interest in primary sensorimotor cortex of five stroke patients before and after Kinect-based virtual reality training

Figure 2.

Effect of Kinect-based virtual reality training on regions of interest in primary sensorimotor cortex of five stroke patients.

Images are the average brain activation map (A) for patient 5 during fist-clenching with right hand before (B) and 3 weeks after training (C) and 12 weeks follow-up (D). Primary sensorimotor cortex cluster sizes of patient 5 gradually decreased.

DISCUSSION

This study assessed the effectiveness of Kinect-based virtual reality training for upper limb motor rehabilitation in subacute stroke patients, and compared brain activation during fist-clenching in healthy subjects with those found in patients after stroke, thus exploring the mechanism of brain reorganization in a virtual reality setting.

All five patients presented with improvements in upper limb function. These improvements included: the ability to raise their affected upper limb above their head, the speed at which they were able to operate the affected upper limb, the ability to separate fingers individually, and improved dexterity of grasping, pinching and other complex motor skills. Motor function test scores displayed significant improvements after virtual reality training. The valence and magnitude of these motor changes are similar to those reported in recent Kinerehab system studies using the same measures[14,18,19,20,21]. This finding suggests that Kinect-based virtual reality training could increase patients’ motivation to participate in rehabilitation. Possible reasons for this may be: first, auditory and visual stimulation in the games are attractive; and second, when patients win or lose, they can obtain feedback information and inspiring sounds from the Kinect-based virtual reality game to encourage them to repeat the same motion.

Our study focuses on the primary sensorimotor cortex, the supplementary motor area and the cerebellum, which are also regions of interest in other studies[22,23]. Stroke may trigger a number of cellular and molecular events in perilesional and remote brain regions, enabling cortical reorganization and recovery of function[24]. The function of the primary sensorimotor cortex is to control the speed, extent, direction, and force of movements, and the supplementary motor area is involved in both producing and the mental rehearsal of sequences of movements. The cerebellum is involved in the coordination of voluntary movements and motor learning. Our study explored the changes in the primary sensorimotor cortex, the supplementary motor area and the cerebellum with hand- clenching under fMRI.

We recruited 18 healthy subjects and five stroke patients, with an affected upper limb, to investigate the effect of Kinect-based therapy. When the healthy subjects performed hand-clenching under fMRI, we recorded areas and intensities of brain activation in the contralateral primary sensorimotor cortex, the bilateral supplementary motor area, and the ipsilateral cerebellum, consistent with other studies[25,26].

Loibl et al[27] have reported that functional activation was different in older and younger people. The brain activation of every patient was compared with that of the standard brain, and we found that the main activation areas and intensities during hand-clenching were similar to our standard brain under fMRI[28,29]. Dong et al[30] explored the evolution of fMRI activation of primary motor cortex and cerebellum on both pre- and post-stroke volunteers, and found that all patients showed higher activation magnitude in perilesional M1 than baseline controls with therapy. However, the activation magnitude of two cases decreased after long-term follow-up.

In our study, primary sensorimotor cortex activation areas increased for four patients[31], patients 1–4, while for patient 5 it gradually decreased over all three time points. Our findings suggest that these positive improvements might be the result of: 1) the requirement of a higher activation magnitude to finish the same motion (efficiency decreased); 2) patient 5, who had reduced activation in the primary sensorimotor cortex, cooperated well with their course of treatment and fMRI examination, and also had a higher education degree and displayed better hand function than the other four patients. Degree of education, compliance, and infarction site could have influenced these results. Regardless of how the brain activation areas change, it appears that the change in cluster size during 12 weeks of therapy was similar to that of a previous study[32]. It is very interesting to find extensively sporadic cortical activation in the whole brain of patients (Table 4, Figure 2), but also the hand function of all five patients displayed improvement 3 weeks after training[20]. This evidence suggests that the degree of recovery from a motor-impairing stroke appears not to be significantly correlated with brain activation, which is consistent with other studies[32]. For the supplementary motor area and cerebellum of patients, no obvious evolving pattern was observed[33,34], except in patient 5. The areas of activation in the supplementary motor area and cerebellum decreased and the activation magnitude increased over the course of the 12-week therapy. These results suggest that patient 5 experienced an increase in efficiency of using his right hand[32].

The mechanism of the Kinect system for physical rehabilitation of upper limb dysfunction may be connected to primary sensorimotor cortex reorganization. Kinect training strengthens the active movement of patients, especially for the affected upper limb. Throughout the duration of the therapy, the range and speed of movement of the affected upper limb increased. The effect of rehabilitation also continued until the 12-week post-intervention follow-up, and the function of upper limb improved. The virtual reality system may offer several advantages such as intensive task-specific practice, immediate feedback, and an enriched and more enjoyable environment. Patients were interested in the system and wanted to continue even after the training had been completed. They also indicated that the system increased their motivation to participate in rehabilitation. In addition, further suggestions were provided: the patients wanted us to increase the number of games, and they also wanted to have more choice. This would make rehabilitation more enjoyable and include the benefits of peer encouragement. The therapist also favorably assessed the system, indicating that it would reduce her workload and improve the effectiveness of rehabilitation for patients. The therapist also suggested that the system incorporated games to enhance the entertainment provided by the system. Enhancing the entertaining, amusing and effective elements of the system are targets for future improvement. Some patients even indicated that they would buy the Kinect system to facilitate their training in the home. Enhanced hardware or software assistive technology can repurpose many commercial high-technology products, turning them into high performance assistive devices to match the special needs of persons with disabilities[35,36]. We need a new training game for patients; the rehabilitation system in this study demonstrated the use of commercial off-the-shelf products and leveraged their many advantages, such as affordability, availability, good after-care service, good technical support, and low concern about social stigma[35,37,38]. In the study, a Kinect module in connection with an image processing software program was affordable for the average family. Evidence shows that the Kinect system with gesture recognition capabilities can be a viable rehabilitation tool that reduces impediments to individuals performing their rehabilitation exercises. This device reduced staff intervention and enabled the participants to enhance their motivation for physical rehabilitation. This achievement enhances the confidence of patients, increases their feeling of independence, and improves the quality of life for individuals with disabilities[39,40,41]. Although the data are encouraging, some study limitations should be noted. First, no control group was used. Our goal is to understand the neural mechanisms behind Kinect-based rehabilitation, so a control group may be less necessary. Second, the number of patients is too low for a comprehensive analysis, but this will be corrected in future work by increasing the amount. Also, matching subjects in lesion location, and focusing on cognitive function recovery via virtual reality training, will also be important in future work.

Overall, our results indicate the following: 1) Kinect- based virtual reality training could be an effective approach and achieved significant improvement in upper limb function in subacute stroke patients. 2) The patients showed significant activation changes in the primary sensorimotor cortex by Kinect-based virtual reality training. This indicates that the brain reorganization targeted by Kinect-based virtual reality training may be connected to the primary sensorimotor cortex. 3) Either a decrease or an increase in the BOLD signal during fMRI, so long as the brain activation map is similar to the healthy average brain map, may indicate upper limb functional recovery. Therefore, Kinect-based virtual reality facilitates functional neuroplasticity in the brain after stroke.

SUBJECTS AND METHODS

Design

A neuroplasticity study.

Time and setting

The experiment was performed in the Motor Recovery Research Laboratory of the First Affiliated Hospital, Sun Yat-sen University, China from March to December 2012.

Subjects

Patients who had suffered a subacute stroke were recruited from the Inpatients of Rehabilitation Department of the First Affiliated Hospital, Sun Yat-sen University, China in 2012. Five patients with subacute stroke were recruited according to the following criteria: 1) stroke confirmed by CT or MRI, and the first single clinical stroke, occurring in the last 3 months; 2) stroke was caused by a cortex or subcortical infarction (excluding infarctions of the brainstem and cerebellum); 3) wrist joint range of motion (ROM) ≥ 10°, metacarpophalangeal joints ROM ≥ 10°; 4) between 40 and 80 years of age. Patients were excluded if they had: 1) history of congestive heart failure; 2) history of recent deep vein thrombosis of the lower limbs; 3) malignant progressive hypertension; 4) reactive liver disease, hepatic or renal inadequacy; 5) respiratory failure; 6) brain, spinal cord or other nerve injury; 7) fracture of upper limb(s); 8) history of stroke with residual function impairment; 9) history of mental disorder or use of antipsychotic medication; 10) cognitive impairment; or 11) deaf or mute.

In accordance with the above inclusion and exclusion criteria, five patients were eligible (four males, aged 50–71 years, mean age 61.4 ± 7.9 years, time after onset ranging from 22–61 days, mean 44.8 ± 14.8 days; one female, aged 66 years, time after stroke was 61 days; all five subjects suffered right hemiparesis).

Healthy subjects

Eighteen healthy elderly subjects were recruited using advertisements placed in Rehabilitative Clinics with the following criteria: 1) without any previous psychiatric, neurological or chronic illness, and normal upper limb function; 2) aged between 40 and 80 years; 3) received primary school education or above. Healthy people were excluded if they had 1) dysfunction of the upper limb; 2) history of nervous system disease. All the subjects were right handed according to the Edinburgh Handedness Scale[18]. The subjects recruited were aged between 49 and 71 years, including 10 females and 8 males (mean age 59.2 ± 5.0 years). All subjects were required to provide written informed consent before commencing studies according to the Regulations on the Administration of Medical Institutions, issued by the State Council of China[19].

Methods

Kinect-based virtual reality training

The patients undertook virtual reality therapy using Kinect (XBOX360, Kinect, Microsoft Inc, Redmond, WA, USA), while playing the game Fruit Ninja. When fruit pops up on the screen, patients need to use their paretic hands to slice the fruit immediately. The Kinect training consisted of 1 hour-long therapy session (four 10-miniute movement periods interspersed with 5-minute rest periods) per day, 5 days per week, for 3 consecutive weeks.

Clinical assessment

A blind clinical assessor rated each participant using the Fugl-Meyer Assessment[11,12] and the Wolf Motor Function Test[13] before training, after 3 weeks of training, and a follow-up after 12 weeks. The Fugl-Meyer Assessment is a measure of upper limb function, with higher scores reflecting greater function (maximum 32 points). The Wolf Motor Function test was subsequently modified and contains 17 tasks, including two strength-based tasks and 15 function-based tasks, divided into two scales: performance time and functional ability. The shorter the performance time, the better the upper limb function. The higher the functional ability scores, the better the function of the upper limbs.

fMRI

All the subjects received hand-clenching training prior to the fMRI, and were instructed to keep stationary during the fMRI session, especially their heads. When proceeding with hand-clenching, the range of hand flexion and extension was as large as possible. All the subjects lay supine with earplugs to protect against the noisy environment, and the head, upper arm and leg were fixed on the bed to avoid body movement. Subjects were instructed to clench their hands for 20 seconds, and then relax for 20 seconds. All patients clenched their right fist first under fMRI, then their left fist[20,21].

Kinect system

Hardware setup: The system comprises a camera and a screen, and requires patients to stand or sit in front of a camera, keeping a suitable distance. The system, commercial video game technology that provides full-body control of animated virtual characters, is produced by Microsoft. The game is shown on the screen, and to play it, the patient moves the affected hand while the game begins. Sometimes, support for the affected upper limb was needed because of the patient's lack of strength and mobility of the impaired upper limb; it can be difficult to keep the upper limb raised for extended periods of time. The training approach is for patients to use their unaffected upper limb and to help their affected upper limb during the early disease stage. The Kinect system is a noninvasive approach and depends on the camera to capture the patients’ posture. A distance of 1.0–3.5 m is specified for tracking the skeletal configuration, and a shorter distance cannot be used. The patients can control the game through their posture without body contact or having to use hand-held controllers. This also helps patients avoid stumbling on the wiring.

Software setup: We chose the game according to the following standards: 1) the game was simple, and the patients could play using their affected upper limb; 2) the game was useful for affected upper limb function rehabilitation; 3) the patients actively liked playing the game; 4) playing the game need not cause drastic body movement, thus avoiding cardiovascular system stress.

Therefore, we chose the Fruit Ninja game produced by Halfbrick Studios. This game was designed and developed for entertainment primarily, but we found that it was very suitable for upper limb rehabilitation of stroke survivors because of its simplicity and enjoyability. Specifically, the objective of the game was to increase the range of the affected upper limb and hand separation movements, with the ultimate goal of improving motor functional recovery of the impaired upper limb. To meet this objective, the game required the patients to use their paretic hands to slice fruit immediately when it popped up on the screen. The goal of the game was to slice the fruit as much as possible while keeping away from the bomb, and finally to determine the total scores.

fMRI data processing and analysis

Eighteen healthy subjects performed one session of fMRI, and five patients underwent fMRI pre-training, 3 weeks following training and once more at a 12-week follow-up. The BOLD fMRI measurements were performed in a whole-body, 3.0 T, General Electric scanner (Siemens, Trio Tim, Germany). The fMRI protocols were based on multi-slice gradient echo-planar imaging (repetition time 2 000 ms, echo time 25 ms, flip angle 90°, field of view 200 × 200, image matrix 64 × 64, number of slices 36, thickness 3 mm, gap 0.3 mm). In addition, 3D T1-weighted magnetization-prepared rapid gradient echo sagittal images were obtained (repetition time 1 460 ms, echo time 2.54 ms, inversion time 900 ms, flip angle 9°, image matrix 256 × 256, number of slices 192, thickness 1 mm).

Statistical parametric mapping 8 software (SPM8, Wellcome Department of Imaging Neuroscience, London, UK) running on Matlab 2009a (Math Works, Natick, MA, USA) was used for MRI image analysis. The functional data sets of the healthy subjects and the five patients were pre- processed using the following steps: 1) Realignment: This corrected for the head motions of each subject during MRI, the images were realigned to the first echo-planar imaging volume. This wrote realigned images into the directory where the functional images are. All participants had less than 3 mm of translation in the X, Y, or Z axis and 1° of rotation in each axis. 2) Coregistration: SPM 8 software was applied to implement a coregistration between the structural and functional data that maximizes the mutual information. 3) Spatial normalization: The realigned volumes were spatially standardized into MNI (Montreal Neurological Institute) space by normalizing with the echo-planar imaging template via their corresponding mean image. Then, all the normalized images were resliced by 3.0 × 3.0 × 3.0 mm³ voxels. 4) Smooth: The normalized functional series were smoothed with a Gaussian kernel of 6 mm full width at half-maximum. Statistics were family-wise error corrected at P < 0.05.

Statistical analysis

Data analysis was carried out using SPSS 17.0 software (SPSS, Chicago, IL, USA). Because of the small sample size, individual data are presented and descriptive statistics were used. In addition, the mean change for the upper limb Fugl-Meyer Assessment, Wolf Motor Function Test level and quality, and number of tasks performed, using both arms, were examined 3 weeks after training using a repeated-measures analysis of variance. The Wolf Motor Function Test times were transformed using log10 to ensure the quality of the results. The intra-group difference was compared using analysis of variance. Data are expressed as mean ± SD. Effects were considered statistically significant for P < 0.05. The first level, for each smoothed individual image, was modified with a fixed effects analysis based on the general linear model, with a box-car response function as the reference waveform convolved with a canonical hemodynamic response function. In the fMRI data group analysis of the 18 healthy subjects, contrast images from the analysis of individual subjects were analyzed by one-sample t-tests. Then the resulting activation maps were created. From these averaged (across 18 subjects) fMRI activation maps and previously reported data, regions of interest were chosen. First-level SPM8 contrasts were then grouped with second-level independent-groups t-tests that compared men and women and one-sample t-tests that used age as covariates. For these analyses, the SPM {t} were thresholded at P < 0.05, family wise error and corrected for multiple comparisons across the whole brain.

Research background: The Kinect virtual reality training has been applied in the rehabilitation of post-stroke patients, and has produced promising results. However, the mechanism of Kinect-based virtual reality training for motor functional recovery of upper limb after subacute stroke was unknown.

Research frontiers: Some researchers have shown that Kinect virtual reality training could improve the upper limb function of poststroke patients. We studied the brains of subacute poststroke patients after Kinect virtual reality training using fMRI, and discussed the mechanism of Kinect-based virtual reality training for motor functional recovery of the upper limb.

Clinical significance: The results of this study could be the basis of the clinical application of Kinect-based virtual reality training, and could be used to define the target brain regions of Kinect-based virtual reality intervention and select suitable measures for the patients.

Academic terminology: Virtual reality is an artificial environment, a three-dimensional virtual world created by a computer, which provides users with simulations of senses such as vision, hearing, touch, and lets users observe things in three dimensions in time without limit. The Kinect Xbox360 from Microsoft is one kind of virtual reality system.

Peer review: The study focused on training subacute poststroke patients using Kinect virtual reality, defined the brain target area, and offered a basis for its clinical application.