Novel Clinically-Relevant Assessment of Upper Extremity Movement Using Depth Sensors

Introduction

Nearly eighty-five percent of individuals survive the initial event of a stroke and return to their homes and communities (Benjamin et al., 2019). Unfortunately, about half of these individuals report hemiparesis, or weakness on one side of the body, that affects full participation in daily activities (Benjamin et al., 2019). This includes areas of daily life such as bathing, dressing, feeding, grooming, sports and leisure, and paid/unpaid work (Benjamin et al., 2019). Rehabilitation providers, such as occupational and physical therapists, can address limitations in client function and participation through evidence-based rehabilitation interventions (Winstein et al., 2016). However, high demands are placed on the therapist’s time, and resources (insurance payments) are limited (Ayala et al., 2018).

To address many of these gaps in insurance coverage and demands on therapist time, most therapists prescribe home programs for their clients with stroke (Winstein et al., 2016). Practice of activities and exercises at home is crucial for rehabilitation. Home-based programs are often provided to clients in the form of a list of exercises with pictures and instructions. Adherence to self-guided exercise programs in the home setting is notoriously low (Jurkiewicz et al., 2011) and very difficult to quantify due to the reliance on clients’ subjective feedback. In the past two decades, virtual reality (VR)-based and video game-based rehabilitation interventions have been heralded as options to boost client motivation and overall adherence to home programs.

VR-based approaches show modest outcomes in improving post-stroke client function (Laver et al., 2017; Karamians et al., 2020). Many of these approaches utilize depth-based sensors, such as the Microsoft Kinect®, as the input device for tracking client movements. The Microsoft Kinect® integrates and infrared sensor for detection of movement in 3-dimensional space. The player does not have to hold a device; they simply use their body to interact with the virtual environment. The Microsoft Kinect® is portable and low cost, making it ideal for home-based rehabilitation.

In addition to providing an engaging method of intervention delivery, the Microsoft Kinect® sensor can also provide valuable data about client progress. These systems are able to discretely track and record full-body client movements while playing the rehabilitation game (Ma et al., 2018; Proffitt et al., 2015). This assessment of movement kinematics has become more important to the overall clinical assessment process. However, for the upper extremity post-stroke, most researchers have limited their use of the Microsoft Kinect® sensor to simple range of motion and velocity measurements (Cai et al., 2019; Wilson et al., 2017). Current studies assessing movement quality (e. g., reach paths, movement jerk) have been conducted in static, laboratory settings with expensive gold standard motion capture systems (Alt Murphy et al., 2015). Researchers have also studied movement kinematics using robotic systems (Balasubramanian et al., 2012); however, these systems are not as portable and are significantly higher in cost compared to portable VR sensors like the Microsoft Kinect.

In recent years, the Microsoft Kinect® V2 sensor has been validated against these gold standard systems for kinematic movement analysis (e. g., Vicon) (Cai et al., 2019; Ma et al., 2018). Given the increase in usage of these VR-based approaches, it is important to understand how sensors like the Microsoft Kinect® can be used to assess movement quality beyond simple range of motion. Although the Kinect is no longer available from Microsoft, the work reported here shows the potential of other similar systems. Therefore, the purpose of this study was to explore the utility of the Microsoft Kinect® in assessing measures of movement quality for individuals post-stroke through a pilot test.

Methods

Study Design

This study is an exploratory, pilot analysis of Microsoft Kinect® sensor data from multiple completed research studies. This study conforms to the STROBE Guidelines.

Microsoft Kinect® and Mystic Isle

The Microsoft Kinect® is a depth sensor that pairs an infrared sensor with an RGB camera. The sensor is able to locate the human form and 20 discrete joints. The skeletal joint data are tracked in 3-dimensional space (x, y, and z planes) and recorded at about 30 frames per second. The sensor tracks the participant and then displays an avatar on the screen (Figure 1). We paired the Microsoft Kinect® sensor with a customized rehabilitation game called Mystic Isle (Lange et al., 2011). The game utilized the Microsoft Kinect® V2 sensor which tracks the player’s movements in real 3-dimensional space. Using the coordinates data, Mystic Isle will create a virtual figure of the player in its 3D virtual game environment where the virtual figure, background, game objects and tasks all change with the player’s movements in real world. During the game, the player is asked to perform specific motions to acquire virtual targets. When the player successfully reaches the colored target, he/she will gain a point, and the target will disappear. The game terminated when the player reached all the targets. The participant control the on-screen avatar by “puppetering” and the location of in-game virtual objects are calibrated based on the participant’s movement abilities. The focus of the games is tailored based on the participant’s rehabilitation goals. All games involve some form of a reach in 3-dimensional space from a stepping forward, standing, seated, or tall kneeling stance. All in-game events are recorded as well as the skeletal joint data (x, y, z coordinates for 15 discrete joints at 30 frames per second for stroke individuals) (Figure 2).

Fig. 1.

Mystic Isle game environment. a) A virtual avatar collecting targets in the Mystic Isle game. b) The skeletal joint labels and positions tracked by the game.

Fig. 2.

Trajectory in lateral-vertical plane of both hands from a game trial. Target locations are marked using black stars. There are dense clusters around most targets.

Participants

Full methods and results for each of the studies described below have been published with the exception of Study 3. Brief methods are described below. Demographic data for all studies are combined and reported in Table 1. Respective Institutional Review Boards approved all studies and all participants provided written Informed Consent prior to enrollment.

Table 1.

Demographics for all participants across the four studies included in the analysis

					Side impacted by	Fugl-Meyer Assessment- Upper
		Age (years)	Sex	Handedness	stroke	Extremity baseline score
Study 1: Healthy		24.2 ± 6.6	M: 6, F: 24	R: 28, L: 2	--	--
Study 2
	P1	55	M	R	L	24
	P2	54	F	R	R	45
	P3	56	M	R	L	25
Study 3
	P1	61	F	R	L	64
	P2	47	M	R	R	66
	P3	36	M	R	L	64
Study 4
	P1	56	F	L	L	ND
	P2	57	M	R	R	ND

Notes: M = male, F = females, R = right, L = left, ND = no data available

Study 1: Healthy Individuals

For this study (Ma et al., 2018), thirty individuals without any hemiparesis or disability in the arm or hand interacted with the Mystic Isle game while being simultaneously tracked by the Vicon motion capture system. Six different types of trials were designed for healthy individuals including sitting close, sitting far, standing close, standing far, standing step and sorting game (Ma et al., 2018). Each individual played all six games twice.

Study 2: In-Home Post-stroke (moderate impairments)

For this study (Proffitt et al., 2015), three individuals with moderate upper extremity impairments post-stroke played the Mystic Isle game in their home for 6 weeks, 1 hour per day, 5 days per week. Each of the games was designed based on participant-identified goals using the Canadian Occupational Performance Measure (COPM) (Cup et al., 2003). Goals varied and included things such as cleaning the bathtub and walking with their spouse on the beach. All three participants reported improvements in performance of their daily occupations and there were minimal barriers to use of the system in the home setting (Proffitt et al., 2015).

Study 3: In-Home Post-stroke (minimal impairments)

The study followed the same methods as Study 2; however all three participants had mild upper extremity impairments post-stroke. No outcome data have been formally reported for this study.

Study 4: LSVT®BIG home exercises (moderate impairments)

For this study, two participants completed the Lee Silverman Voice Treatment®-BIG (LSVT®BIG) intervention 4 days per week for 4 weeks. LSVT®BIG is an intensive, amplitude-focused intervention designed for individuals with Parkinson’s disease and adapted for this study. The LSVT®BIG intervention has an associated home program. The two participants completed the home program via the Mystic Isle game, playing the game once on in-clinic treatment days and twice on non-treatment days. The exercises and functional tasks involved sustained reaches and movements, interacting with virtual objects in a similar way to the prior studies. Both participants had improvements in upper extremity motor function and self-rated occupational performance (Proffitt et al., 2018).

Data Analysis

A set of metrics were introduced to quantitively evaluate the quality of the hand movements. We’ve applied the extent of reach and hand speed metrics to the game in our previous work (Ma et al., 2018). The hand efficiency and smoothness were investigated in this paper.

Pre-processing

Mystic Isle recorded game event information and the 3D joint positions at 30 frames per second. Game trials with target location available from the game were used for the assessments. A sixth order Butterworth filter with a 3Hz cutoff was applied to all game files. Individual reaches were extracted from each game file using event start and end timestamps.

Defining valid paths

To assess the efficiency and the smoothness of the hand movement paths, it is necessary to understand the structure of hand joint samples. Figure 2 shows the visualization of the trajectories of both hands with target positions in lateral-vertical plane. Samples can be divided into two groups: a) curved paths between either targets or dense clusters; b) dense clusters surrounding the targets. Since the game records the samples at 30 frames per second, the more the hand appears in a location, the denser the samples are in this location.

The dense clusters around targets sometimes caused by the adjustment from the real coordinate space to the game virtual space. Thus, the data samples in the dense clusters around the targets should be excluded when performing assessments. This is to say, as shown in Figure 3., the colored data samples surrounding targets should not be considered when analyzing the path efficiency and smoothness. We only consider a valid path as a curved trajectory excluding clusters around targets.

Fig. 3.

Path segmentation results of both-hand samples from a stroke player. The dense clusters marked in colors were identified by the OPTICS algorithm. The valid paths were marked in black crosses.

To get the valid paths, we need to locate the clusters first. Since the sample structure is related to the density, we applied the Ordering Points To Identify the Clustering Structure (OPTICS) clustering algorithm (Ankerst et al., 1999) to perform the path segmentation. The dense clusters located by the OPTICS algorithm were marked in color and a valid path is a path between two adjacent targets with samples in the clusters excluded marked in black in Figure 3.

Kinematic variables

For movement efficiency, path ratio and average sway distance were calculated for each reaching movement valid path. Path ratio was defined as:

$$PathRatio=\frac{Thelengthoftheshortestpath}{thelengthofavalidpath}$$

where the shortest path is the straight line between the first point and the end point of the corresponding valid path.

To quantitatively describe the difference between a real or sample reaching path and an ideal reaching path, we propose a metric called average sway distance. During a hand movement between two targets, the sway distance of a sample point is the distance from the sample point to the shortest path between the two targets. Average sway distance was defined as:

$$\mathrm{A}\mathrm{v}\mathrm{e}\mathrm{r}\mathrm{a}\mathrm{g}\mathrm{e}\mathrm{s}\mathrm{w}\mathrm{a}\mathrm{y}\mathrm{d}\mathrm{i}\mathrm{s}\mathrm{t}\mathrm{a}\mathrm{n}\mathrm{c}\mathrm{e}=\frac{\sum distance(sampleposition,idealposition)}{numberofsamples}$$

For movement smoothness, normalized jerk (Yan et al., 2000) was calculated using the formula:

$$NormalizedJerk=\sqrt{\frac{1}{2}\times \frac{d^5}{l^2}{\int }_{t_i}^{t_f}{J}^2\left(t\right)dt}$$

where d denotes the overall movement duration, and l denotes the overall movement length, and J denotes the jerk function, the third derivative of position. The normalized jerk was evaluated on two different paths: paths between two adjacent targets (target paths) and valid paths (defined above). The interquartile range approach was applied to detect outliers for the outcomes calculated on target paths.

Statistical Analysis

For the healthy individuals in Study 1, a within subjects t-test was used to determine if there was a statistically significant difference in right vs. left upper extremity reaches. If there was no difference, the data were collapsed into one group and averaged for all kinematic variables. Given the small sample size, the data were averaged across all reaches of each hand (more affected side and less affected side) for each kinematic variable for each individual. ANOVA with Games-Howell post hoc adjustment was used to compare the average for each side for each kinematic variable to the healthy individuals.

Results

There were no differences in path ratio (p = 0.56), average sway distance (p = 0.45), and normalized jerk (p = 0.62, 0.49 and 0.86 for reaches on target paths, reaches without outliers and reaches on valid paths, respectively) between the right and left upper extremities for the healthy individuals. Therefore, the data were averaged across the right and left upper extremities for all three kinematic variables. Tables 2 and 3 display the results for all kinematic variables for healthy individuals as well as each post-stroke participant by sides more and less impacted by their stroke.

Table 2.

Average values for efficiency metrics across the four studies.

		Path Ratio		Average Sway Distance (mm)
Study 1		0.76 ± 0.11		80.15 ± 30.08
		More affected	Less affected	More affected	Less affected
Study 2
	P1	0.47 ± 0.25	0.63 ± 0.18	106.3 ± 65.9	146.0 ± 105.5
	P2	0.55 ± 0.22	0.51 ± 0.23	182.2 ± 87.7	115.4 ± 61.9
	P3	0.20 ± 0.12	0.15 ± 0.14	367.6 ± 102.8	271.4 ± 193.1
Study 3
	P1	0.56 ± 0.17	0.61 ± 0.11	151.9 ± 52.5	131.6 ± 40.8
	P2	0.47 ± 0.23	0.51 ± 0.21	170.9 ± 88.8	156.3 ± 86.0
	P3	0.53 ± 0.23	0.54 ± 0.21	182.4 ± 115.2	175.8 ± 98.8
Study 4
	P1	0.35 ± 0.24	0.38 ± 0.26	253.9 ± 126.0	233.0 ± 133.0
	P2	0.50 ± 0.20	0.51 ± 0.17	112.0 ± 76.6	122.0 ± 67.1

Note. All values reported as mean ± standard deviation.

Table 3.

Average values for normalized jerk across the four studies.

		Reaches on target paths		Reaches without outliers		Reaches on valid paths
Study 1		6.0E4 ± 8.9E5		4.3E3 ± 6.0E3		4.3E4 ± 5.7E5
		More affected	Less affected	More affected	Less affected	More affected	Less affected
Study 2
	P1	1.0E7 ± 1.4E8	7.4E6 ± 7.4E0	1.5E5 ± 2.5E5	2.6E5 ± 3.8E5	1.2E7 ± 1.8E8	1.8E7 ± 1.4E8
	P2	2.6E5 ± 5.7E6	4.4E5 ± 1.1E7	5.5E3 ± 8.8E3	6.3E3 ± 9.7E3	1.2E5 ± 1.6E6	8.8E4 ± 1.0E6
	P3	8.9E8 ± 5.6E9	1.0E9 ± 5.8E9	3.5E6 ± 6.3E6	6.9E6 ± 1.3E7	3.3E8 ± 1.2E9	8.8E8 ± 4.6E9
Study 3
	P1	3.1E6 ± 2.1E7	2.6E6 ± 1.9E7	2.5E5 ± 3.2E5	2.4E5 ± 3.3E5	2.0E6 ± 1.7E7	1.6E6 ± 1.9E7
	P2	1.4E6 ± 4.4E7	3.7E6 ± 1.3E8	1.7E4 ± 2.3E4	1.7E4 ± 2.3E4	1.5E6 ± 4.8E7	4.9E6 ± 1.7E8
	P3	1.7E6 ± 7.2E6	2.0E6 ± 9.9E6	2.2E5 ± 3.2E5	2.2E5 ± 3.2E5	1.1E6 ± 6.1E6	1.4E6 ± 7.4E6
Study 4
	P1	1.4E5 ± 1.4E6	1.3E5 ± 1.0E6	1.7E4 ± 2.4E4	4.2E3 ± 6.0E3	1.2E5 ± 1.0E6	9.8E4 ± 6.8E5
	P2	1.6E6 ± 4.4E7	1.8E6 ± 5.4E7	3.2E4 ± 4.4E4	3.2E4 ± 4.3E4	1.8E6 ± 4.9E7	1.9E6 ± 5.9E7

Note. All values reported as mean ± standard deviation.

Kinematic variable: Path ratio

The average movement path ratio for healthy individuals was 0.76 ± 0.11. For all participants post-stroke, even those with mild impairments, average path ratio was smaller (indicating less efficient movements). Most participants post-stroke also had a smaller path ratio for their more affected side post-stroke (Figure 4, Table 2).

Fig. 4.

The mean and standard deviation of path ratio of healthy individuals and participants post-stroke.

Kinematic variable: Average sway distance

The average sway distance for healthy individuals was 80.15 ± 30.08. Most participants post-stroke had larger sway distances (indicating a less efficient path taken by the upper extremity). Those with mild impairments had large average sway distances and large standard deviations (variance) in their movements (Figure 5, Table 2).

Fig. 5.

The mean and standard deviation of average sway distance of healthy individuals and participants post-stroke.

Kinematic variable: Normalized jerk

To evaluation hand movement smoothness, the normalized jerk metric was applied on each target path (a trajectory between two adjacent targets). Two outlier approaches were then investigated including removing outliers using interquartile range approach and performing normalized jerk assessment on valid paths. As a result, there are three groups of data named as “all reaches”, “no outliers” and “valid paths”. We report normalized jerk values for all data analysis approaches. The average normalized jerk on target paths for healthy individuals was 6.0E⁴ ± 8.9E⁵ for reaches on target paths, 4.3E³ ± 6.0E³ for reaches with outliers removed, and 4.3E⁴ ± 5.7E⁵ for reaches on valid paths. For the participants post-stroke, most values were 2–5 orders of magnitude larger than healthy individuals, even on the less affected side post-stroke (Figure 6, Table 3).

Fig. 6.

The mean and standard deviation of normalized jerk of healthy individuals and participants post-stroke.

Discussion

The purpose of this study was to explore the utility of the Microsoft Kinect® in assessing measures of movement quality for individuals post-stroke through a pilot test. We can calculate new assessments of movement quality, such as smoothness of movement and movement efficiency. Further, these kinematic variables are potentially sensitive in detecting less efficient movement in individuals with mild motor impairments post-stroke. We discuss each of these points below along with limitations and future research.

The clinical kinematic variables calculated in this study provide insight into the quality of movement for individuals post-stroke. For rehabilitation games played using a 3-dimensional depth sensor, the therapist programs the location of the virtual objects in 3-dimensional space (Lange et al., 2011; Proffitt et al., 2015). Individuals are only successful in the Mystic Isle game if they touch the virtual object at that full extent of reach. Therefore, traditional measures of range of motion or extent of reach provide little clinical insight. Additionally, for individuals post-stroke, reach velocity is often not indicative of movement quality. For those with spasticity or increased tone, asking them to move faster leads to compensatory movement strategies during reaching (Mandon et al., 2016). Movement efficiency and smoothness are measures of the quality of a reach and have implications for designing interventions in the clinic and home settings. A therapist can include treatment components that challenge movement efficiency and smoothness within a functional task. As an assessment tool, these clinical kinematic variables can be used to document performance over time and supplement existing clinical outcome measures. Additionally, these variables could be generalized to other games, activities, and devices that record similar data (x, y, z position and time).

For the mild stroke population, these kinematic variables are potentially sensitive to performance impairments that are unobservable, even with a trained therapist eye. Therapists often treat individuals with mild stroke and other mild brain injuries (e. g., post-concussive syndrome) who report difficulty in motor-based tasks that have an added cognitive or balance component. For example, individuals post-stroke who completed dual tasking during walking demonstrated poorer performance than walking alone (Patel & Bhatt, 2014). Therapists could use these kinematic variables to detect deficits in performance and intervene as appropriate. This is especially important for those with mild stroke who return sooner to work and other community activities (Wolf et al., 2009).

This study has some limitations. First, the sample size is very small; however, this is to be expected given the exploratory nature of the study. The next step is to recruit a larger sample of individuals with both mild and moderate motor impairments post-stroke to determine sensitivity and specificity of the kinematic variables. Second, the Microsoft Kinect® is a fairly robust tracking system; however, it is subject to variations in tracking quality based on the individual’s environment, lighting, and distance from the sensor. This impacts the overall reliability of the data. Additionally, the Microsoft Kinect® is no longer being produced; however currently available depth sensors use similar methods for skeletal tracking and the results shown here have applications regardless of the technology. Lastly, this study explored three new kinematic variables over short time periods. Future research will explore these variables over longer intervention periods.

More rehabilitation games are including a depth sensor as the input/tracking device (Laver et al., 2017). It is imperative that these systems include clinically applicable and useful assessments. These kinematic variables within game-based rehabilitation systems using depth sensors become more necessary as telehealth becomes more widespread and insurance companies demand measures of patient progress for reimbursement. The portability of this system and pairing with engaging rehabilitation-specific games adds new avenues for in-home stroke rehabilitation as both a stand-alone and adjunct to existing rehabilitation.