Assessing Upper Extremity Motor Function in Practice of Virtual Activities of Daily Living

Abstract

A study was conducted to investigate the criterion validity of measures of upper extremity (UE) motor function derived during practice of virtual activities of daily living (ADLs). Fourteen hemiparetic stroke patients employed a Virtual Occupational Therapy Assistant (VOTA), consisting of a high-fidelity virtual world and a Kinect^™ sensor, in four sessions of approximately one hour in duration. An Unscented Kalman Filter-based human motion tracking algorithm estimated UE joint kinematics in real-time during performance of virtual ADL activities, enabling both animation of the user’s avatar and automated generation of metrics related to speed and smoothness of motion. These metrics, aggregated over discrete sub-task elements during performance of virtual ADLs, were compared to scores from an established assessment of UE motor performance, the Wolf Motor Function Test (WMFT). Spearman’s rank correlation analysis indicates a moderate correlation between VOTA-derived metrics and the time-based WMFT assessments, supporting the criterion validity of VOTA measures as a means of tracking patient progress during an UE rehabilitation program that includes practice of virtual ADLs.

I. Introduction

The impact of stroke on individuals and on the economy is substantial. Approximately 7 million Americans who are 20 years of age or older have experienced a stroke, with an overall incident rate estimated at 2.8% [1]. Minorities are disproportionately affected, with 3.9% of non-Hispanic blacks and 5.9% of American Indian/Alaska Natives having a history of stroke. Each year, approximately 795,000 individuals suffer a new or recurrent stroke, which translates into one every 40 seconds. Approximately 70% of individuals suffering from stroke experience significant functional deficits [2] .

Current stroke rehabilitation practice is based on principles of cortical plasticity which emphasize the need for repetition and task specificity [3]. Training specificity is required, whether targeting a particular movement (e.g. anti-gravity elbow flexion) or an integrated skill (e.g. self-grooming). Task-directed therapy appears to be a critical element to the regeneration of cortical function in regions of the brain responsible for related neurological activity [4]. Non-specific repetitive motor activity alone appears to be less effective [5]. The need for training specificity has been widely accepted for decades in milieus from competitive sports to aviation, but only more recently as a systematic approach to upper extremity (UE) stroke rehabilitation. One example is the Accelerated Skill Acquisition Program (ASAP), which integrates constraint-induced therapy, skill acquisition, and motivational elements to reinforce normal motion patterns through massed, task-specific practice [6], [7].

Rapid advances in low-cost human motion tracking technology and advanced computer graphics, largely fueled by the video gaming industry, have also spurred interest in gaming and virtual reality (VR) as tools for stroke rehabilitation. Preliminary studies have shown that gaming systems can provide a high level of patient enjoyment, and thus lead to increased motivation to engage in physical activity [8]. These encouraging findings are tempered by observations that off-the-shelf games encourage non-specific (and sometimes undesirable) movements, do not permit patient-specific settings, and can be frustrating for individuals with more severe impairment. Therapists have also observed that it is hard to target clinical outcomes and/or desired movements with available games [9].

In an overview of the use of virtual environments for stroke rehabilitation, Holden found that stroke-affected patients are able to reacquire motor function in a virtual environment and that movements practiced in a virtual environment transfer to real-world tasks [10]. In a meta-analysis of the use of VR in stroke rehabilitation, Saposnik and Levin found that eleven of twelve studies considered showed a significant benefit in the selected outcome measure [11]. Another review of recent work within the field indicates task-specific virtual UE training resulted in superior outcomes, while game-based interventions produced less specific outcomes [12]. In a comparative investigation, a group of chronic stroke patients performing a simple pointing task in a virtual environment achieved similar improvements as a group receiving equivalent training in a real-world physical environment [13]. The authors of that study suggest virtual training is especially applicable to the chronic post-stroke stage for targeted upper-limb tasks. While these studies have demonstrated the potential of VR for task-specific stroke rehabilitation, barriers including system cost and burdens on personnel time (specialized training and cumbersome setup/calibration procedures) continue to impede widespread adoption [14].

This paper describes the design of a low-cost virtual world-based system for practice of meaningful activities that incorporate specific functional movements, and investigates the concurrent validity of motor performance metrics generated during virtual ADL practice. Correlation of computer-derived measures generated by a Kinect^™-based motion tracking algorithm to the Wolf Motor Function Test (WMFT) [15] [16] provides evidence of the suitability of these measures for tracking patient progress and reporting status. Note that although we expect a mature Virtual Occupational Therapy Assistant (VOTA) system to deliver therapeutic benefits, this paper focuses specifically on a hypothesized correlation between VOTA metrics and an accepted clinical measure (the WMFT). Assessing the efficacy of VOTA practice for UE motor recovery is the subject of ongoing research.

II. Approach

A. Virtual Occupational Therapy Assistant (VOTA)

VOTA is being developed to enable stroke patients to practice virtual ADLs as part of an in-patient, skilled nursing, outpatient, home health, and/or teletherapy rehabilitation program. The system employs a Kinect^™ sensor, kinematic pose estimation algorithms, and state-of-the-art game engine technology to create a compelling world in which patients can perform realistic virtual ADLs that target specific UE movements within integrated, multi-step activities. Figure 1 shows a prototype VOTA system during virtual ADL practice. Depending on the treatment context, VOTA may be employed in independent practice or under the supervision of a care provider.

Figure 1.

User performing a virtual ADL activity.

Patient experience in the VOTA virtual world is built around the theme of a metaphoric “Road to Recovery.” After an automated introduction to the island by a virtual occupational therapist (OT), patients move within the virtual world to activity areas in which virtual ADL practice takes place. Once in an activity area, the patient’s character transitions into an “activity mode” in which avatar arm motion matches the patient’s UE pose (see Figure 2). Tasks are explicitly constructed to emulate actions required for independent living. It was hypothesized that patients would be more likely to accept and use the technology if “games” had a direct link to real-world benefit [17]. Training specificity is achieved by incorporating functional movements and task sequences associated with real-world ADLs.

Figure 2.

Examples of elements of the VOTA Meal Preparation activity: (a) virtual OT introduces patient to the VOTA virtual kitchen, (b) Bring Second Slice to Toaster sub-task performed in “activity mode,” and (c) Put Pan in Sink sub-task.

Note that this paper employs a broad definition of activities of daily living (ADLs) that encompasses both the traditional definition of ADL behaviors [18] and instrumental activities of daily living (IADLs) [19]. The former comprise toileting, feeding, dressing, grooming, locomotion, and bathing. The latter include use of a telephone, shopping, food preparation, housekeeping, laundry, use of transportation, management of medication, and handling finances.

Virtual ADL activities are decomposed into tasks and sub-task elements that target a wide range of functional movements, enabling software to automatically parse and process activity segments into aggregate metrics. Each activity is constructed as a hierarchy of tasks and sub-tasks (see Figure 3). For example, Meal Preparation includes a Get Pan task (one of fourteen tasks in that activity). Get Pan includes a Bring Pan to Stove sub-task (one of four sub-tasks in that task). Tasks and sub-tasks are completed sequentially, each contributing to the desired end-state of the activity. Each level in the hierarchy includes attributes that define configurable elements in the interactive simulation (e.g. verbal instructions, position/orientation of objects and avatars, level of cueing provided, goals, and rewards). At the lowest level, sub-tasks are categorized as motor (MOTORSCORE), cognitive (COGSCORE), or unscored elements. MOTORSCORE sub-tasks are used in generation of VOTA motor function metrics. COGSCORE sub-tasks relate to success in tasks involving recall, sequencing, and situation awareness. As this paper focuses on the criterion validity of MOTORSCORE metrics, COGSCORE results are not presented nor discussed.

Figure 3.

VOTA is organized a hierarchy of activities, tasks, and sub-tasks that are categorized as motor (MOTORSCORE), cognitive (COGSCORE), or unscored elements.

In MOTORSCORE sub-tasks, the VOTA application provides both visual and verbal cues that are intended to minimize the impact of cognition on derived metrics. These sub-tasks incorporate point-to-point movements, with arrested motion (near-zero wrist speed in Cartesian space) at well-defined start and stop locations in the virtual task space. For example, the Bring Pan to Stove sub-task begins when the user moves the avatar’s hand to the location of the pan’s handle, and ends when the pan is placed on a burner on the stove top. By arranging the position and orientation of objects in the virtual environment, sub-tasks can be designed to target specific functional UE movements and ranges of motion. For example, in a Reach for Pan sub-task, the height of a cabinet determines the amount of shoulder flexion required to successfully complete the movement. Note that the VOTA task-space is mirror-imaged when activities are performed left-handed, thus the range of motions required to accomplish each task is consistent.

The combination of human motion tracking and a virtual world-based game enables activities to be tailored in ways that would be difficult to achieve in real life. For example, although VOTA tracks both right and left arms as well as torso motion, the virtual (avatar) manifestation of the patient’s unimpaired limb and torso are typically fixed (not animated) during practice of virtual ADLs. Undesired compensatory movement by a patient, such as forward trunk motion, is not realized in the virtual world. Individuals are thus required to use only the affected UE to complete the task.

B. UE Tracking Filter

At the time of writing, the Kinect^™ software development kit (SDK) includes general purpose, low-overhead filtering software that works well on a wide range of applications, particularly for gaming. However, it does not provide all information needed by the VOTA application. For kinematic tracking and smooth avatar animation, VOTA requires estimates of joint velocities and angular rates, which are not directly measured by the sensor nor provided by the SDK. The SDK quantifies measurement confidence only in categorical terms (high, low, none) and exposes only a limited set of variables for tuning the behavior of the filtering, smoothing, and outlier rejection algorithms. These provide a critical starting point for the application but do not constitute a sufficient solution for VOTA.

VOTA kinematic pose estimation algorithms are based on an adaptation of an Unscented Kalman Filter (UKF) [20] [21]. The UKF-based solution to the inverse kinematics problem produces angles and angular rates defining shoulder and arm motion, employing only Kinect^™ measurements. The real-time generated variables support both avatar animation, to immerse the patient in the virtual ADL activity, and recording of kinematic information for assessing motor outcomes. The tracking filter enforces realistic arm kinematics and joint angle constraints, and handles noisy measurements and sensor dropout. Bone lengths are estimated jointly as a by-product of filtering and therefore no manual measurement or calibration is required. The filter accounts for these factors by weighting the relative confidence in new measurements versus prior estimates, allowing it to optimally fuse the two sources of information.

Let us denote the dynamic state and measurements of the system as $\overrightarrow{x}\left(t\right)$ and $\overrightarrow{y}\left(t\right)$, respectively. For clarity, the temporal notation is omitted throughout the remaining discussion, but is always implied. The state vector is comprised of four primary sub-vectors,

$$\overrightarrow{x}={\left\{{\overrightarrow{a}}_{\text{left}}^T{\overrightarrow{a}}_{\text{right}}^T{\overrightarrow{b}}^T{\overrightarrow{c}}^T\right\}}^T,$$ (1)

where superscript-T indicates vector transpose. The “arm” state vectors, ${\overrightarrow{a}}_{\text{left}}$ and ${\overrightarrow{a}}_{\text{right}}$, denote UE joint angles, rates, and accelerations of the left and right arms, respectively. Thus

$$\overrightarrow{a}={\left\{{\overrightarrow{\theta }}^T{\dot{\overrightarrow{\theta }}}^T{\ddot{\overrightarrow{\theta }}}^T\right\}}^T,$$ (2)

employing Newton’s (dot) notation to indicate differentiation with respect to time. UE joint angles are estimated in a swing-twist parameterization that is described fully in [22] [23].

$$\overrightarrow{\theta }={\left\{{\theta }_{\text{swing}.y}{\theta }_{\text{swing}.z}{\theta }_{\text{twist}}{\theta }_{\text{elbow}}\right\}}^T,$$ (3)

The sub-vector $\overrightarrow{b}$ denotes distances between joints, which we abstractly refer to as “bone” lengths, defined as,

$$\overrightarrow{b}={\left\{b_{\text{collarbone}}{b}_{\text{humerus}}{b}_{\text{forearm}}\right\}}^T,$$ (4)

For convenience we use the term “collarbone” to indicate the line segment connecting the shoulder joints, “humerus” to indicate the segment connecting shoulder to elbow, and “forearm” to indicate the segment connecting elbow to wrist. Including bone lengths in the state vector allows them to be estimated jointly with the other quantities, eliminating the need to manually calibrate the filter for each user. Finally, the sub-vector $\overrightarrow{c}$ denotes the pose (position, yaw, and pitch) of the “collarbone” line segment that connects the shoulder joints.

$$\overrightarrow{c}={\left\{c_{\mathrm{pos}1}{c}_{\mathrm{pos}2}{c}_{\mathrm{pos}3}{c}_{\text{yaw}}{c}_{\text{pitch}}\right\}}^T,$$ (5)

The collarbone line segment forms part of a right-hand coordinate system from which the shoulder angles are referenced, and therefore must be filtered jointly with arm state. The global (sensor-fixed) coordinate system can be chosen arbitrarily, so the collarbone’s three translational degrees of freedom are denoted generically. For the results presented here, the global reference frame is defined with the origin at the sensor focal plane, and the x, y, and z axes pointing in the user’s right, up, and rear directions respectively.

To facilitate interpretation of tracking results in terms that are more intuitive (and consistent with common use in the medical community) arm motion can also be represented as a series of right-handed Euler angle rotations about the axes defined in Figure 4.

$${\theta }_{\text{Euler}}={\left\{{\theta }_1{\theta }_2{\theta }_3{\theta }_4\right\}}^T,$$ (6)

Positive (negative) rotation of the upper arm around Axis 1 by angle θ₁ corresponds to shoulder flexion (extension). Positive (negative) rotation by angle θ₂ about Axis 2 corresponds to shoulder abduction (adduction). Positive (negative) rotation by angle θ₃ about Axis 3 corresponds to internal (external) shoulder rotation. Positive (negative) rotation of the forearm by θ₄ about Axis 4 corresponds to elbow flexion (extension). Note that forearm pronation/supination, wrist, and hand articulation are not included in this model due to the limitations of the Kinect^™ sensor.

Figure 4.

UE motion can be defined as a series of Euler angle rotations applied sequentially about Axes 1 to 4. The illustrated pose represents all four angles set to zero.

The swing-twist parameterization [22] [23] generated by the UKF filter solution is locally diffeomorphic to this Euler parameterization, and therefore joint angles (and their derivatives) can be freely transformed between the two representations. The angular mapping is

$${\theta }_1=\mathrm{atan}2({\theta }_{\text{swing}.y},-{\theta }_{\text{swing}.z})$$ (7)

$${\theta }_2=\frac{\pi }{2}-\sqrt{{\theta }_{\text{swing}.y}^2+{\theta }_{\text{swing}.z}^2}$$ (8)

$${\theta }_3={\theta }_1+{\theta }_{\text{twist}}$$ (9)

$${\theta }_4={\theta }_{\text{elbow}}$$ (10)

The process model of the UKF describes how the state vector evolves over time. For this application, bone length is assumed constant and collarbone motion is modeled as a random walk,

$$\frac{d}{dt}\overrightarrow{b}=0,\frac{d}{dt}\overrightarrow{c}={\overrightarrow{w}}_c,$$ (11)

where ${\overrightarrow{w}}_c$ denotes a zero-mean random noise vector whose covariance defines the rate of the random walk. UE motion is modeled as a kinematic linkage with damped joints, soft mechanical limits, and acceleration biases that randomly walk (roughly modeling muscle activity),

$$\frac{d}{dt}\overrightarrow{a}=\left\{\begin{array}{c}\hfill \dot{\overrightarrow{\theta }}\hfill \\ \hfill {\ddot{\theta }}_{\text{swing}.y}-{\phi }_{\text{swing}.y}{\stackrel{.}{\theta }}_{\text{swing}.y}-{\gamma }_{\text{swing}.y}\left(\overrightarrow{a}\right)\hfill \\ \hfill {\ddot{\theta }}_{\text{swing}.z}-{\phi }_{\text{swing}.z}{\stackrel{.}{\theta }}_{\text{swing}.z}-{\gamma }_{\text{swing}.z}\left(\overrightarrow{a}\right)\hfill \\ \hfill {\ddot{\theta }}_{\text{twist}}-{\phi }_{\text{twist}}{\stackrel{.}{\theta }}_{\text{twist}}-{\gamma }_{\text{twist}}\left(\overrightarrow{a}\right)\hfill \\ \hfill {\ddot{\theta }}_{\text{elbow}}-{\phi }_{\text{elbow}}{\stackrel{.}{\theta }}_{\text{elbow}}-{\gamma }_{\text{elbow}}\left(\overrightarrow{a}\right)\hfill \\ \hfill \overrightarrow{0}\hfill \end{array}\right\}+{\overrightarrow{w}}_a,$$ (12)

where ${\overrightarrow{w}}_a$ is random noise that represents modeling imperfections and defines the rate at which the biases randomly walk. The terms in each row represent the kinematic relationship between positions, velocities, and accelerations. Joint damping is modeled using the linear ${\phi }_{\ast }{\stackrel{.}{\theta }}_{\ast }$ terms, and joint limits are denoted with the nonlinear ${\gamma }_{\ast }\left(\overrightarrow{a}\right)$ terms. Limits are modeled as ideal springs that apply a restoring torque only when the respective joint reaches an upper or lower mechanical limit, as in

$${\gamma }_{\ast }\left(\overrightarrow{a}\right)=\{\begin{array}{cc}{k}_{\ast }\mid {\theta }_{\ast }-{\theta }_{\text{lower}}\mid \hfill & \text{below lower limit}\hfill \\ k_{\ast }\mid {\theta }_{\ast }-{\theta }_{\text{upper}}\mid \hfill & \text{above upper limit},\hfill \\ 0\hfill & \text{between limits}\hfill \end{array}\phantom{\}}$$ (13)

where k_* represents the stiffness of the mechanical limits, and θ_lower and θ_upper are the joint limit angles.

The measurement model defines the mapping from the state, $\overrightarrow{x}$, to the outputs of the Kinect^™ sensor, $\overrightarrow{y}$. The measurement is comprised of the user’s shoulder, elbow, and wrist joint positions, as in

$$\overrightarrow{y}={\left\{{\overrightarrow{p}}_{\text{shoulder}}^T{\overrightarrow{p}}_{\text{elbow}}^T{\overrightarrow{p}}_{\text{wrist}}^T\right\}}^T,$$ (14)

where ${\overrightarrow{p}}_{\ast }$ indicates the 3-element position vector of each joint in global coordinates. The measurement model is thus the set of forward kinematic equations that relate joint angles and bone lengths to positions in world space. Though lengthy, these equations are straightforward to derive using standard trigonometric constructions. In the interest of brevity, the equations are denoted simply as,

$$\overrightarrow{y}=h_{\mathit{kin}}\left(\overrightarrow{x}\right)+{\overrightarrow{w}}_y,$$ (15)

where $h_{\mathit{kin}}\left(\overrightarrow{x}\right)$ represents the nonlinear kinematic (trigonometric) mapping and ${\overrightarrow{w}}_y$ represents random measurement noise.

With the state space modeled as described above, the remainder of the UKF implementation follows the standard formulation described in [20] and [21].

C. UE Motor Performance Metrics

For each MOTORSCORE sub-task, VOTA parses the kinematic data provided by the UKF solution to calculate metrics that represent motor performance. If these measures can be established as valid indicators of UE motor performance, then they may provide an automated means to assess patient status and progress during practice of virtual ADLs. Motor function-related metrics considered in this study include:

• a)Duration, sub-task completion time in seconds;
• b)Normalized speed (NS) (percent) – Mean speed achieved divided by peak speed during performance of each sub-task;
• c)Movement Arrest Period Ratio (MAPR) (percent) – Percentage of time that speed exceeds a threshold percentage of peak speed during performance of each sub-task.

Metrics were chosen for incorporation in the VOTA study based on two factors: (1) previous evidence of correlation to UE motor function; and (2) compatibility with automated calculation during virtual ADL practice. Task completion time (duration) is a commonly used measure in well-accepted assessments such as the WMFT [15] [16] and the Arm Motor Ability Test (AMAT) [24]. The inclusion of NS and MAPR in this study was inspired by the work of Rohrer et al. [25]. In trials involving 31 stroke patients, these two metrics were the most strongly correlated to a clinical scale of upper extremity function from among five candidate measures related to smoothness of motion, which also included a jerk metric, a peaks metric, and a tent metric. Both of the selected smoothness-related measures rely on the notion that impaired motion in point-to-point UE movements tends to be characterized by an episodic speed profile. NS is based on the presumption that non-smooth motion will have a speed profile with a mean value that is significantly less than peak speed. MAPR is based on the supposition that speed profiles characteristic of episodic motion will spend a greater percentage of total duration below a threshold value than smooth profiles. A threshold of 30% of peak is used in this study (compared to the 10% threshold selected by Rohrer et al.) to increase the sensitivity of the MAPR metric to speed fluctuations [25] [26]. Both smoothness-related measures are calculated such that higher values suggest better motor performance, and thus are expected to be negatively correlated to duration. In all cases, speed is defined as the instantaneous magnitude of the Cartesian-space velocity vector of the user’s wrist, as estimated by the real-time UKF tracking filter.

III. Methods

A. Participants

Participants were hemiparetic stroke patients meeting study inclusion criteria including: antigravity strength at the elbow to at least 45 degrees of active flexion; antigravity shoulder strength to at least 30 degrees each in active flexion, abduction/adduction; and 15 degrees in active shoulder rotation from an upright seated position. Participants included individuals in in-patient rehabilitation care and outpatient rehabilitation. As the intent of the study was to investigate concurrent validity of VOTA metrics to existing clinical measures, study participation did not replace or interfere with any prescribed course of treatment. The protocol was administered by licensed occupational therapists under the approval and supervision of University of Virginia (UVa) Institutional Review Board for Health Sciences Research (IRB-HSR). All study activity took place in the clinical facilities of the UVa-HealthSouth Rehabilitation Hospital, Charlottesville, VA. Study characteristics are shown in Table 1. A total of 14 individuals enrolled in and completed the study between September and December 2013. All consented individuals completed the study; there were no dropouts.

Table 1.

Study Characteristics.

Subjects	Gender	Age(median)	Age(range)
14	4 M / 10 F	69	48-87
Months since stroke (median)	Months since stroke (range)	Dominant hand	Impaired arm
30	0.5-96	14 R / 0 L	7 R / 7 L

B. Protocol

In each of four visits, participants were asked to use their stroke-affected arm to practice a Meal Preparation activity that included 17 MOTORSCORE sub-tasks (see Table 2) performed while making a breakfast of eggs, toast, and juice. During system use, participants remained seated at all times in a chair centered in front of the computer display at a distance of two meters. When possible, the four visits were scheduled to occur within a two week window. The first three visits provided patients with the opportunity to become familiar with VOTA system and the Meal Preparation activity. For this protocol, task difficulty level was fixed both between subjects and between sessions.

Table 2.

VOTA Meal Preparation MOTORSCORE Sub-Tasks

Sub-task	Description
1	Bring Pan To Stove
2	Bring Oil To Pan
3	Bring First Egg To Pan
4	Bring Second Egg To Pan
5	Bring First Slice To Toaster
6	Bring Second Slice To Toaster
7	Reach For Pepper
8	Bring Pepper To Pan
9	Bring Plate To Counter
10	Bring Glass To Counter
11	Bring Spatula To Pan
12	Bring Spatula To Plate
13	Put Spatula In Sink
14	Put Pan In Sink
15	Bring Pitcher To Glass
16	Bring First Slice To Plate
17	Bring Second Slice To Plate

It was expected that during initial VOTA sessions, lack of familiarity with the virtual world, the motion tracking interface, and the Meal Preparation activity would result in significant variability in user performance. The first three VOTA sessions thus served as training, permitting participants to become familiar with the VOTA system and the constituent tasks. Based on the authors’ previous observations on learning in manual task performance system [27], it was hypothesized that performance change due to learning would plateau by the fourth session. Participants’ performance in the fourth visit (with training completed) therefore served as the basis for assessment of the concurrent validity of VOTA measures of UE motor performance. VOTA Duration, NS, and MAPR metrics collected in the fourth session were aggregated over the set of all MOTORSCORE sub-tasks to create a record of each patient’s performance in completing the Meal Preparation activity. VOTA-Duration, VOTA-NS, and VOTA-MAPR scores were calculated as the median values over all MOTORSCORE sub-tasks.

Figure 5 shows an example of participant data captured and processed by VOTA in the fourth session. The record of speed vs. time for a Bring Second Slice To Toaster sub-task within the Meal Preparation activity is typical for a MOTORSCORE element. The point-to-point UE movement begins at near-zero speed at a starting position (location of a loaf of bread on a counter top), accelerates to some peak speed, and then decelerates back to near-zero speed at the target position (location of toaster). This particular profile appears to include two sub-movements, hypothesized by Rohrer et al. to constitute building blocks of more complex UE motion [25] [28].

Figure 5.

UE tracking data captured during virtual ADL performance is parsed by sub-task and used to generate motor performance metrics that are then aggregated over an entire session.

Following completion of the fourth VOTA session, a therapist administered the WMFT. WMFT is a well-accepted, functionally-oriented clinical and research UE assessment that has been shown to have high inter-rater reliability, validity, and internal consistency [15] [16]. The full WFMT consists of 15 timed function-based and 2 strength-based tasks. Only the timed tasks were administered in this study. Seven of these tasks involve isolated movements of either the affected elbow or shoulder. The remaining eight items are practical tasks that require coordinated UE movement (e.g. lifting can from table to mouth). Performance in each task was timed using a stopwatch. Maximum time permitted for any one task was 120 seconds. Each participant’s WMFT-TIME score was derived by taking the median task completion time across all 15 elements [16].

IV. Results

A. Task-level Analysis of UE Involvement

Real-time tracking of UE kinematics by the above-described UKF solution provides detailed histories of the actual movements used by patients in accomplishing the virtual ADLs. Recall that position/orientation of the user’s avatar within the virtual space and the location of involved objects for MOTORSCORE sub-tasks were chosen to elicit a range of functional UE movements and ranges of motion. Examples of tracking results for three different VOTA sub-tasks performed by a stroke patient in this study are shown in Figure 6. Bring Oil To Pan (sub-task 2) starts with acquisition of a bottle of olive oil from a cabinet shelf and ends when the user brings the oil to the frying pan on the stove top. The resulting UE movement involves significant active shoulder flexion and elbow extension, as well as active internal shoulder rotation (Figure 6a). Bring Second Slice To Toaster (sub-task 6, illustrated in Figure 2b) requires the user to acquire a piece of bread from a loaf on the counter and move it laterally to an empty slot in a toaster. The resulting UE motion shown in Figure 6(b) includes active shoulder extension and external rotation. Put Pan In Sink (sub-task 14, illustrated in Figure 2c) begins when the user acquires the frying pan from the stove top and finishes when the pan is moved to a sink in an adjacent countertop. The resulting motion is among the more complex in the Meal Preparation Activity, eliciting both active shoulder horizontal abduction (rotation about Axis 2 with the shoulder in flexion) and active shoulder external rotation and active elbow flexion (Figure 6c).

Figure 6.

Examples of Euler angles and rates generated by real-time UE tracking of a stroke patient during performance of three different virtual ADL sub-tasks. (a) Bring Pan To Stove generates active shoulder extension/flexion, shoulder internal rotation, and elbow flexion. (b) Bring Second Slice To Toaster produces significant active shoulder external rotation. (c) Put Pan In Sink elicits active shoulder horizontal abduction with active shoulder external rotation and elbow flexion.

Table 3 provides a summary of the median peak angular speed achieved across all participants for individual sub-tasks. Most virtual ADL sub-tasks involve some UE movement in all three axes. This observation is consistent with characterization of UE movements during performance of real-world ADLs by Rosen et al. [29].

Table 3.

Sub-task level summary across all participants (n=14) of median peak angular speed (in degrees per second) observed in each of the four Euler angles defining UE movement.

Sub-task	Axis 1 (deg/s)	Axis 2 (deg/s)	Axis 3 (deg/s)	Axis 4 (deg/s)
1	23.5	31.4	37.4	20.5
2	49.4	39.7	51.9	49.2
3	24.4	34.4	28.6	23.5
4	17.9	37.7	33.8	19.1
5	42.1	40.6	38.3	26.2
6	43.9	29.4	33.6	31.7
7	38.9	19.0	38.6	31.4
8	25.8	45.9	51.7	34.0
9	33.1	47.4	63.9	46.5
10	69.3	58.8	70.2	53.3
11	40.7	29.7	56.4	31.8
12	49.2	33.1	56.6	44.9
13	55.6	33.0	68.7	36.3
14	47.2	37.5	59.0	38.7
15	18.8	29.9	19.3	19.1
16	25.0	50.0	54.9	20.3
17	32.0	43.6	66.9	21.1

B. Primary Outcome

VOTA and WMFT-TIME metrics are interval variables that can reasonably be expected to have a monotonic relationship. We do not, however, have any expectation of linearity. Spearman’s rank-order correlation (r_s) [30] is thus used to analyze the bivariate correlations between VOTA measures and WMFT-TIME. Table 4 summarizes the results. VOTA-Duration has a moderate and statistically significant correlation with WMFT-TIME (r_s = 0.56, p=0.036). This degree of correlation is within the range (0.54 ≤ r_s ≤ 0.68) previously observed by Wolf et al. [15] between WMFT and Fugl-Meyer UE assessments of 19 stroke survivors.

Table 4.

Primary outcome - bivariate correlations between VOTA-derived metrics and WMFT-TIME.

X	Y	Spear- man rs	Pvalue	95% confidence
VOTA- Duration	WMFT- TIME	0.56	0.036	0.03	0.85
VOTA-NS	WMFT- TIME	−0.45	0.107	−0.80	0.12
VOTA- MAPR	WMFT- TIME	−0.34	0.233	−0.75	0.25

Of the two smoothness-related VOTA metrics, NS shows the strongest correlation to WMFT-TIME. This result is consistent with results from previous research showing a moderate correlation between NS and MAPR and the Fugl-Meyer (FM) UE assessment [25], with NS having the stronger correlation. These results may indicate that smoothness-related metrics would be more appropriately treated as adjuncts to, rather than predictors of, more traditional measures of UE motor function. Duration of task completion may be generally indicative of strength, while smoothness-related measures may be better gauges of coordination.

Analysis of Spearman’s rank-order correlation between VOTA-derived metrics, summarized in Table 5, reveals a strong correlation between all metric pairings. The extremely high correlation between NS and MAPR indicates that, in a practical implementation, only one need be used.

Table 5.

Bivariate correlations between VOTA-derived metrics.

X	Y	Spear-man rs	Pvalue	95% confidence
VOTA- Duration	VOTA- NS	−0.84	<0.001	−0.95	−0.54
VOTA- Duration	VOTA- MAPR	−0.75	0.002	−0.92	−0.35
VOTA- NS	VOTA- MAPR	0.91	<0.001	0.73	0.97

C. Training Effects

A session-by-session summary of estimated mean metrics across all subjects, shown in Table 6, reveals that all three VOTA-derived measures of motor performance monotonically improve in each of the four successive trials.

Table 6.

Mean VOTA-derived summary metrics across all subjects by study session.

Session	Mean VOTA-Duration	Mean VOTA-NS	Mean VOTA-MAPR
1	6.05	0.40	0.57
2	5.08	0.43	0.61
3	3.94	0.49	0.70
4	3.70	0.48	0.70

As hypothesized, by the fourth session, training-related improvement appears to have plateaued. Between-subject differences in VOTA-derived metrics are therefore expected to be primarily reflective of UE motor status. For each metric, there is a significant difference (with a p≤0.05 decision rule) in the means between sessions 1 and 4, and between sessions 2 and 4, but no significant difference between sessions 3 and 4. Table 7 provides the associated p-values for the test of equal means.

Table 7.

P-values for mean differences in VOTA-derived metrics between sessions.

Sessions	VOTA-Duration	VOTA-NS	VOTA-MAPR
1-4	0.001	0.001	0.002
2-4	0.037	0.019	0.038
3-4	0.699	0.778	0.932

D. Intra-Subject Variability

Plotting the subject-specific estimated mean duration and associated 95% confidence intervals for the VOTA Meal Preparation activity (Figure 7) reveals a strong correlation between individuals’ mean completion time and variability in that metric.

Figure 7.

Mean subject-specific VOTA sub-task duration for the Meal Preparation activity. Circles denote the estimated mean duration, and vertical lines denote the 95% confidence interval for mean task duration.

Calculation of the bivariate correlation between estimated mean and standard deviation in VOTA sub-task completion times by way of the Spearman test confirms the existence of a very strong relationship (r_s = 0.89, p<0.001). Further investigation reveals a similar, even stronger, correlation between estimated mean duration and standard deviation in the 15 task completion times for the WMFT (r_s = 0.95, p<0.001). Note that the relationship between estimated mean and variance is much less pronounced in the smoothness-related metrics of NS (r_s = −0.12, p=0.67) and MAPR (r_s = −0.46, p=0.09).

These results are consistent with findings by other researchers that both response times and variability tend to increase with level of impairment [31]. Increased variability has also been associated with deficits in executive control functions of the brain [32]. In the context of the present study, the apparent importance of variance in task completion times may indicate that variability-related metrics can provide complementary insight into motor performance during practice of virtual ADLs.

V. Discussion

To our knowledge, this work represents the first time a metric derived during performance of ADLs in a virtual environment has been shown to correlate to an established assessment of UE motor performance. The implication is that it may be possible to employ low-cost, off-the-shelf sensing technology and “smart” software to systematically monitor a patient’s progress over a course of treatment. The game-embedded assessment does not require wearable markers, calibration, or administration by a specially-trained clinician.

Both VOTA-Duration and WMFT-TIME metrics are based on median completion time over a set of tasks involving point-to-point UE movements. It is therefore not surprising that they exhibit a moderate, positive correlation. The importance of this relationship lies in the provenance of the underlying data. While WMFT metrics are the product of a traditional assessment instrument, VOTA values are extracted “behind the scenes” during virtual ADL practice. To the extent that a VOTA metric is a valid measure of UE motor performance, it may be used to automate tracking of patient progress over the course of an intervention involving the VOTA system.

Virtual ADLs were designed to emulate real-world tasks. As a consequence, the VOTA Meal Preparation activity (which required numerous forward reaching movements to interact with objects on virtual shelves) placed a greater demand on anti-gravity shoulder strength and range of motion than the WMFT tasks (which primarily involve interacting with objects on a planar table surface). This incongruity may have adversely impacted the observed degree of correlation between the two tests – but it is unclear if the effect is undesired. Therapists involved in the protocol observed that the sequenced VOTA tasks elicited UE movements that better represent functional use than the more isolated WMFT tasks.

Note that the absolute accuracy of the above-described UKF-based tracking solution is limited by the accuracy of the underlying Kinect sensor in providing UE joint positions. In a study led by the University of California at Berkeley, researchers found that mean errors for estimated shoulder, elbow, and wrist joint positions ranged from 44 to 76 mm between a Kinect-based measurement and an established marker-based tracking solution [33]. In the present study, the kinematic tracking solution based on these measurements was found to be sufficient, both to permit patients to successfully complete virtual ADL tasks and to support derivation of speed-based motor performance metrics. Future work will take advantage of improvements in sensor technology (e.g. Kinect for Windows v2) and investigate incorporation of multiple sensors to further improve tracking performance.

Other planned future research includes investigation of the efficacy of virtual ADL practice for post-stroke UE motor recovery. Ongoing development will permit the physical and cognitive challenge level of the virtual ADLs to be adapted to the capabilities of each patient. The difficulty level may be fixed by the care provider, or adapted automatically by software (within a prescribed range) in response to trends in VOTA-generated metrics. Alternative motor performance metrics to be investigated include dimensionless jerk parameterizations that exhibit monotonic correlation to submovement blending [26]. Future work will also expand the depth and breadth of activities in the VOTA application, benchmark accuracy and reliability of achieved tracking performance, and explore the usability of the system by patients and providers in multiple rehabilitation settings.

Biographies

Richard J. Adams received the B.S. degree as a Distinguished Graduate of the U.S. Air Force (USAF) Academy in 1989, the M.S. degree from the University of Washington in 1990, and the Ph.D. degree from the University of Washington in 1999. He served as an active duty commissioned officer in the USAF between 1989 and 2009. He returned to the USAF Academy in 2007 as Assistant Professor in the Department of Astronautical Engineering. In 2009, Lieutenant Colonel Adams retired from the Air Force and joined Barron Associates, Inc., where, as Principal Research Scientist, he currently leads multiple efforts spanning game-based rehabilitation systems, assistive technology for the blind and vision impaired, tactile human-computer interfaces, and spaceborne sensors. Dr. Adams currently serves as an Associate Editor for the IEEE Transaction on Haptics.

Matthew D. Lichter received the B.S. degree in mechanical engineering from the Pennsylvania State University, University Park, in 1999, where he graduated at the top of his class and with honors. He received the M.S. and Ph.D. degrees in mechanical engineering from the Massachusetts Institute of Technology (MIT), Cambridge, in 2001 and 2004 respectively. Following his PhD, he continued working in the MIT Field and Space Robotics Laboratory as a post-doctoral associate. In 2006, Dr. Lichter joined Barron Associates, Inc., where, as a Senior Research Engineer, he is currently leading software development on several game-based rehabilitation products and designing estimation algorithms for unmanned aerial and spaceborne systems.

Eileen T. Krepkovich received the B.S. degree in biomedical engineering from Case Western Reserve University, where she graduated summa cum laude, and received the M.S. degree in biomedical engineering from Northwestern University. She received an NSF Graduate Research Fellowship in 2005. After Northwestern, she worked as a research engineer at the Rehabilitation institute of Chicago on multiple projects for the rehabilitation of individuals with stroke and spinal cord injury. She joined Barron Associates, Inc. in 2011, where her research focuses on the development of assistive technology and human-computer interfaces for rehabilitation. She is a member of IEEE Engineering in Medicine and Biology Society.

Allison Ellington received the B.S. and M.S. degrees in occupational therapy from Ithaca College, Ithaca, NY. She is currently working toward her OTD at Chatham University, Pittsburgh, PA. She has been an occupational therapist for 7 years, primarily specializing in stroke rehabilitation. She earned advanced clinical certifications from the NEURO-IFRAH organization, emphasizing therapeutic intervention techniques for adults with hemiplegia. Additionally, she is trained and has used in practice numerous therapeutic technology systems including functional electric simulation systems as well as robotics. She is currently director of clinical education for the occupational therapy program at the Murphy Deming College of Health Sciences, Staunton, VA and serves as occupational therapist at UVa-HealthSouth, Charlottesville, VA.

Marga White graduated Cum Laude from Virginia Tech in 2004 with a Bachelor’s degree in Human Development. She received the Master’s degree in Occupational Therapy from Shenandoah University in 2007. She is currently working as an outpatient occupational therapist at UVa- HealthSouth Rehabilitation Hospital, Charlottesville, VA. For the past six years, she has worked in the inpatient and outpatient adult rehab setting, treating patients with a wide range of diagnoses including: stroke, brain injury, spinal cord injury, multiple sclerosis, Parkinson’s, visual impairments, orthopedic injuries and arthritis..

Paul T. Diamond, MD, has been Director of Neurorehabilitation at the University of Virginia since 1992. He is an Associate Professor with tenure in the Department of Physical Medicine and Rehabilitation. Dr. Diamond is board certified in both Physical Medicine and Rehabilitation and Internal Medicine. He received his undergraduate degree from Harvard College and medical degree from the University of Virginia (UVa). He completed his residency training in Internal Medicine and Physical Medicine and Rehabilitation at the Johns Hopkins Medical Institutions. Dr. Diamond serves as primary attending on the UVa-HealthSout neurorehabilitation service and directs UVa’s Neurorehabilitation program. He has authored over thirty publications and abstracts and has lectured extensively on neurorehabilitation and functional outcomes.