Evaluation of Upper Extremity Movement Characteristics during Standardized Pediatric Functional Assessment with a Kinect®-Based Markerless Motion Analysis System

Abstract

A recently developed and evaluated upper extremity (UE) markerless motion analysis system based on the Microsoft® Kinect® has potential for improving functional assessment of patients with hemiplegic cerebral palsy. 12 typically-developing adolescents ages 12–17 were evaluated using both the Kinect-based system and the Shriners Hospitals for Children Upper Extremity Evaluation (SHUEE), a validated measure of UE motion. The study established population means of UE kinematic parameters for each activity. Statistical correlation analysis was used to identify key kinematic metrics used to develop automatic scoring algorithms. The Kinect motion analysis platform is technically sound and can be applied to standardized task-based UE evaluation while providing enhanced sensitivity in clinical analysis and automation through scoring algorithms.

I. Introduction and Background

Hemiplegic-type cerebral palsy (HCP) is a common movement disorder caused by non-progressive disturbance in the developing brain. Individuals with HCP present with UE motor impairments including hypertonicity, weakness, loss of selective motor control, and reduced range of motion, resulting in lower performance during gross and fine motor activities of daily living (ADL). Functional UE impairments in children with HCP range from minor to severe. Individuals with HCP receive two primary methods of intervention to address UE dysfunction: rehabilitative therapies and surgery. Physical and occupational therapy interventions are designed to improve range of motion and motor performance, maximize activity levels, and enhance participation. Surgical treatment is indicated to improve joint stabilization, restore range of motion, or balance torque distribution across joints. Quantitative assessment is vital in the treatment of UE dysfunction as it facilitates identifying impairments, planning intervention and measuring progress.

The SHUEE is an evaluation that measures an individual’s ability to perform functional tasks based on ADL. It is a validated tool that provides ordinal scoring of spontaneous usage, alignment of UE segments, and object grasp and release capability of the hand [1]. Davids et al., the developers of the SHUEE, admitted that kinematic motion analysis during functional tasks would provide more accurate, reliable, and objective data than the currently ordinal-based SHUEE scoring methods. Kinematic scoring would also provide a more sensitive measure than ordinal scoring when tracking progress over time or following intervention [1]. However, limitations of lab-based UE motion analysis systems, including expense, time, and uncomfortable marker application, restrict the ready application of kinematic motion analysis to evaluations such as the SHUEE without technological advancement [5].

To improve standardized task evaluation in individuals with HCP, a motion analysis platform using the Kinect was developed, including UE and hand skeletal tracking software, providing the benefits of kinematic analysis technology without limitations of task-based evaluations [2]. The system accurately and reliably detects UE kinematics, as shown during a separate technical evaluation [2]. Benefits include low cost, portability, and markerless operation.

The purpose of this work is to develop a set of UE ADL scoring algorithms using data collected from typically-developing adolescents and statistically evaluated to extract key measures of UE kinematics for specific ADLs. These algorithms will be implemented to provide automated scoring of activities during a Kinect-based evaluation.

II. Methods and Materials

A. Participants

Twelve typically-developing adolescent participants, (n=7) male and (n=5) female, ages 12 to 17, with no injury or impairment to UE function, were recruited. The SHUEE was performed as described in its original guidelines [1] by a physical therapist. SHUEE data analysis was performed based on video recordings using standardized scoring. A final score was calculated for spontaneous functional analysis (SFA), dynamic positional analysis (DPA), and grasp/release analysis (GRA). Inclusion criteria used in this study was a score of 100% on each component of the SHUEE. All recruited subjects scored 100% on each of SFA, DPA, and GRA and were included in the study.

B. Data Collection

Kinect® evaluation consisted of collecting UE position data while subjects performed SHUEE-derived activities [2]. Activities, which included both broad UE and hand-specific activities, were designed to accommodate the unique characteristics of the Kinect® sensor. Staff provided participants with standardized instructions and guidance for each activity. Multiple trials were performed in succession to obtain an average kinematic trajectory for each activity.

The Microsoft® Kinect® sensor is a commercially available, low-cost video game accessory that uses depth imaging to track position of body segments and interpolate skeletal position. It contains a pair of infrared depth sensors and a standard RGB camera that together capture three-dimensional objects [3], and has been shown to be accurate in kinematic detection [4]. A 3D surface map of the body is used to interpolate skeletal joints and anatomical features and stores 3D coordinates for further processing, allowing real-time markerless skeletal tracking (Fig. 1).

Figure 1.

Kinect UE/LE Skeletal Tracking

A hand-specific component (Fig. 2) tracks hand features as 3D coordinates [2], including palm center, finger tips, and medial and lateral finger base points, to calculate broad level hand kinematics, not specific to individual joints of the fingers. Both hand and UE systems detect skeletal position at 30 Hz. Once the evaluation is complete, the system stores the 3D location of each detected point throughout the duration of testing, including all trials and any downtime between them.

Figure 2.

Kinect Hand Skeletal Tracking

C. Data Processing

3D position coordinate trajectories were filtered using a low-pass digital Butterworth filter (2^nd order, 1.5 Hz cutoff, 30Hz sampling), to remove noise in motion data without affecting location accuracy. A skeletal image displayed on-screen allowed selection of trial start and end points (Fig. 3).

Figure 3.

Skeletal Display for Trial Selection, UE (top) and Hand (bottom)

Once the trials were marked, angular position (θ) was calculated for each joint, using an arctangent-based method:

$${\theta }_x=\text{arctan}\left(\frac{|\mathit{\text{DIST}}\times \mathit{\text{PROX}}|}{\mathit{\text{DIST}}•\mathit{\text{PROX}}}\right)*\frac{180}{\pi }$$ (1)

where DIST and PROX in equation (1) are unit vectors representing the segments distal and proximal to the joint. Angular velocity (ω) and acceleration (α) were calculated from position using 1^st and 2^nd order finite difference:

$${\omega }_x=\frac{d{\theta }_x}{dt}=\frac{{\theta }_{x,\mathrm{t}+1}-{\theta }_{x,\mathrm{t}-1}}{2*dt}$$

$${\omega }_y=\frac{d{\theta }_y}{dt}=\frac{{\theta }_{\mathrm{y},\mathrm{t}+1}-{\theta }_{\mathrm{y},\mathrm{t}-1}}{2*dt}$$ (2)

$${\omega }_z=\frac{d{\theta }_z}{dt}=\frac{{\theta }_{\mathrm{z},\mathrm{t}+1}-{\theta }_{\mathrm{z},\mathrm{t}-1}}{2*dt}$$

$${\alpha }_x=\frac{d{\omega }_x}{dt}=\frac{d^2{\theta }_x}{d{t}^2}=\frac{{\theta }_{x,\mathrm{t}+1}-2{\theta }_t+{\theta }_{x,\mathrm{t}-1}}{d{t}^2}$$

$${\alpha }_y=\frac{d{\omega }_y}{dt}=\frac{d^2{\theta }_y}{d{t}^2}=\frac{{\theta }_{\mathrm{y},\mathrm{t}+1}-2{\theta }_t+{\theta }_{\mathrm{y},\mathrm{t}-1}}{d{t}^2}$$ (3)

$${\alpha }_z=\frac{d{\omega }_z}{dt}=\frac{d^2{\theta }_z}{d{t}^2}=\frac{{\theta }_{\mathrm{z},\mathrm{t}+1}-2{\theta }_t+{\theta }_{\mathrm{z},\mathrm{t}-1}}{d{t}^2}$$

In Eqs. (2) and (3), only the ω_x and α_x components are computed, effectively a 2D analysis. This simplified joint motion will not correlate with 3D kinematics, but is appropriate for relative comparisons among subjects and consistent with the algorithms developed here.

Each trial was normalized, with mean and SD computed, resulting in position, velocity, and acceleration trajectory plots for each joint, and statistics for each activity, including ROM, peak velocity, and peak acceleration for each joint.

D. Interpretation and Algorithm Development

Kinematic metrics were evaluated using the SAS® CORR procedure in logarithmic scale to compute Pearson correlation coefficients. In markerless detection, abnormal kinematics occur when segments are obstructed from view or misidentified, producing outliers. Correlation coefficients were computed for ROM, velocity, and acceleration of all joints. Outliers observed in correlation plots were removed. To identify metrics that characterize each activity, kinematic focus was considered, based on SHUEE literature and the intent of Kinect activities. Correlation coefficients were used to identify strongly correlated and semi-correlated metrics. Strongly correlated metrics had Pearson correlation coefficients greater than 0.9. Semi-correlated metrics had coefficients greater than 0.5. Statistical insight is combined with kinematic intent of each activity to define a subset of metrics that best characterize each UE activity performance. Mean and standard deviation values are calculated for each of the key kinematic metrics obtained for each task. Kinect scoring algorithms are proposed based on this analysis.

III. Results

Table I provides normal mean and standard deviation values (n=12) for selected activities in the Kinect evaluation. Kinematic plots are obtained from results (Fig. 4).

Table I.

Sample UE Metrics for Normal Population

Activity	Key Metric	Pop. Mean ±SD
Grasp/Release Extended	Dominant (D) Finger ROM	27.10°±12.80°
	Nondominant (ND) Finger ROM	28.20°±11.80°
Thumb-Index Pinch	D Index ROM	33.48°±12.97°
	D Thumb ROM	26.52°±14.56°
	ND Index ROM	36.21°±12.86°
	ND Thumb ROM	28.67°±11.62°
Cut Play-Doh	D Wrist ROM	33.41°±18.64°
	D Elbow ROM	25.41°±16.36°
Throw Ping-Pong Ball	D Wrist ROM	32.75°±13.94°
	D Elbow ROM	40.30°±22.24°
	D Shoulder ROM	21.66°±10.79°

Figure 4.

Angular Kinematics for Ball Throwing Activity

The resulting correlated metrics for each activity are presented in Table II.

Table II.

Key UE Metrics for Normal Population

Activity	Strongly Corr. Metrics	Semi-Corr. Metrics
Grasp/Release Neutral	D Finger ROM ND Finger ROM	Finger Peak Velocity Finger Peak Acceleration
Grasp/Release Flexed	D Finger ROM ND Finger ROM	Finger Peak Velocity Finger Peak Acceleration
Grasp/Release Extended	D Finger ROM ND Finger ROM	Finger Peak Velocity Finger Peak Acceleration
Thumb-Index Pinch	D Index ROM D Thumb ROM ND Index ROM ND Thumb ROM	Thumb Peak Velocity Index Peak Velocity Thumb Peak Acceleration Index Peak Acceleration
Wrist Range of Motion	D Wrist ROM ND Wrist ROM	Wrist Peak Velocity Wrist Peak Acceleration
Elbow Range of Motion	D Elbow ROM ND Elbow ROM	Elbow Peak Velocity Elbow Peak Acceleration
Shoulder Range of Motion	D Shoulder ROM ND Shoulder ROM	Shoulder Peak Velocity Shoulder Peak Accel.
Unscrew Bottle or Jar Cap	D Wrist ROM D Wrist Peak Vel. D Wrist Peak Acc.	D Elbow ROM D Shoulder ROM
Pull Play-Doh Apart	D Wrist ROM ND Wrist ROM D Elbow ROM ND Elbow ROM D Shoulder ROM ND Shoulder ROM	Wrist Peak Velocity Wrist Peak Acceleration Elbow Peak Velocity Elbow Peak Acceleration Shoulder Peak Velocity Shoulder Peak Accel.
Cut Play-Doh With Knife	D Wrist ROM D Elbow ROM	D Shoulder ROM D Vel. and Accel.
Throw Ping-Pong Ball	D Wrist ROM D Elbow ROM D Shoulder ROM	D Extremity Velocity and Acceleration
Place Sticker on Large Ball	D Elbow ROM D Shoulder ROM	D Wrist ROM D Vel. and Accel.
Put Socks On or Fasten Shoe	D Elbow ROM ND Elbow ROM	Wrist ROM Shoulder ROM

Table III describes the SHUEE scoring method and the proposed implementation of Kinect scoring algorithms. Algorithms were based on statistical analysis of normal population data and adapted from SHUEE scoring strategies, providing continuous-scale rather than ordinal scoring while maintaining correlation between scores and kinematic parameters for increased clinical relevance.

Table III.

Proposed Kinect UE Scoring Algorithm

Scoring Metric	SHUEE Scoring [1]	Kinect Primary Measures	Kinect Secondary Measures
Grasp/Release Analysis (GRA)	Scored 0–6 based on ability to grasp and release hand	Finger range of motion	Finger velocity and acceleration
Dynamic Positional Analysis (DPA)	Scored 0–3 based on alignment of segments during activities	ROM for each joint of interest	Velocity and acceleration for each joint of interest
Spontaneous Functional Analysis (SFA)	Scored 0–5 (Modified House Scale) based on usage spontaneity	Velocity and Acceleration for each joint of interest	ROM for each joint of interest

As an example, the “throw ping-pong ball” activity could be characterized by an algorithm that uses weighted kinematics of the shoulder (S), elbow (E), and wrist (W) to calculate the DPA and SFA components. Each metric is weighted based on correlation, with strongly correlated metrics (Primary Measures) comprising 90% and weakly correlated metrics (Secondary Measures) 10%. In SFA velocity and acceleration are both strongly correlated so velocity is given 60% total weighting and acceleration 30%, to account for greater variability in acceleration.

$$DPA(\%)=0.3W_{\mathit{\text{ROM}}}+0.3E_{\mathit{\text{ROM}}}+0.3S_{\mathit{\text{ROM}}}+0.05(W_{\mathit{\text{VEL}}}+E_{\mathit{\text{VEL}}}+S_{\mathit{\text{VEL}}})+0.05(W_{ACC}+E_{ACC}+S_{ACC})$$ (4)

$$SFA(\%)=0.2W_{\mathit{\text{VEL}}}+0.2E_{\mathit{\text{VEL}}}+0.2S_{\mathit{\text{VEL}}}+0.1W_{ACC}+0.1E_{ACC}+0.1S_{ACC}+0.1(W_{\mathit{\text{ROM}}}+E_{\mathit{\text{ROM}}}+S_{\mathit{\text{ROM}}})$$ (5)

Each variable in the above algorithms represents a linear function. As an example, the wrist ROM function is

$$W_{\mathit{\text{ROM}}}(\%)=\frac{|W-W_0|}{\left(\frac{W_{+1SD}-W_0}{0.25}\right)}$$ (6)

where W₀ is the population mean. The value of this function is 75% when W is 1 SD from W₀, 50% when W is 2 SD from W₀, 25% when W is 3 SD from W₀, and 0 when W is greater than 4 SD from W₀.

In algorithms proposed above, healthy population data for the activity set was analyzed using correlation to identify the kinematic metrics that best characterize the performance of each activity. Initial values for each coefficient are proposed based on the degree of correlation in each metric, with metrics more strongly correlated to activity performance weighted higher in proposed algorithms. Coefficients will need to be optimized through a significant study of children with CP and varying UE function.

IV. Discussion and Conclusions

The SHUEE can be improved clinically using the Kinect® system, without placing additional burdens on patients or therapists. The system accomplishes these improvements by adding quantitative, objective, kinematic data, using markerless kinematic analysis and algorithms developed in this study. Using the SHUEE with the Kinect® system provides clinicians with useful UE metrics, increases speed and repeatability of SHUEE analysis by removing subjective components, and improves the ability to monitor multiple joints simultaneously to observe trends in multi-joint coordination or neuromotor compensation strategies.

The current study integrated statistical analysis of UE kinematics from 12 typically-developing adolescents using the Kinect® UE system to provide an innovative algorithm-based platform that can enhance functional assessment of patients with HCP. SHUEE scores for all participants were 100% with no deviation. Significant variability in UE kinematics across the sample was observed further alluding to increased sensitivity of kinematic motion analysis in characterization of UE performance. The addition of kinematic data using the Kinect® can enhance these current scoring methods by providing an additional set of continuous, sensitive, scores. The Kinect® evaluation was as easy to use for both the therapist and subjects in a clinical UE evaluation capacity as the SHUEE.

This is a methodological development study whose results will be refined and implemented in future work. Only healthy subjects were tested to obtain normal kinematics that were used to create the scoring algorithms and determine weighting coefficients in those algorithms. These algorithms will need to be optimized through extensive testing of children with CP with varying levels of UE function to characterize the complex UE impairments in CP. It should be noted that the simplified calculation of joint motion described above is acceptable for the elbow but cannot differentiate planar motions of the wrist and shoulder, which may reduce the efficacy of the system in detecting and scoring more complex activities.

Algorithms developed in this work allow automatic calculation of SHUEE scores based on continuous kinematic variables, as opposed to manual scoring from pre-recorded video of the examination. An enhanced 3D Kinect system is proposed for future work that integrates motion analysis hardware and software improvements with gaming and therapy goal integration to provide a comprehensive system. Physical therapists will design games tailored to specific therapy goals based on performance deficiencies, provide games in clinical or home settings using a low-cost and highly portable system, and obtain detailed kinematic performance and patient usage evaluations from the system.