Assessment of Neurological Impairment and Recovery Using Statistical Models of Neurologically Healthy Behavior

Abstract

While many areas of medicine have benefited from the development of objective assessment tools and biomarkers, there have been comparatively few improvements in techniques used to assess brain function and dysfunction. Brain functions such as perception, cognition, and motor control are commonly measured using criteria-based, ordinal scales which can be coarse, have floor/ceiling effects, and often lack the precision to detect change. There is growing recognition that kinematic and kinetic-based measures are needed to quantify impairments following neurological injury such as stroke, in particular for clinical research and clinical trials. This paper will first consider the challenges with using criteria-based ordinal scales to quantify impairment and recovery. We then describe how kinematic-based measures can overcome many of these challenges and highlight a statistical approach to quantify kinematic measures of behavior based on performance of neurologically healthy individuals. We illustrate this approach with a visually-guided reaching task to highlight measures of impairment for individuals following stroke. Finally, there has been considerable controversy about the calculation of motor recovery following stroke. Here, we highlight how our statistical-based approach can provide an effective estimate of impairment and recovery.

Introduction

In most areas of medicine, there has been rapid and continued improvement in clinical tools and biomarkers to quantify body function and dysfunction. Blood, urine, and saliva samples provide a wealth of information on the function of many organs, and imaging techniques provide detailed information on their structure. In contrast, there has been relatively limited improvement in assessment of brain function. Some indirect measures have been developed to look at patterns of activity in the brain (or muscle) such as electroencephalography (electromyography), positron emission tomography, and functional magnetic resonance imaging. However, techniques to assess perception, cognition, and motor impairments have changed very little over the years. For example, assessment of motor impairments continues to be largely based on visual and physical inspection by a clinician. These approaches have been honed over the years to focus on key problems for a given patient group, and commonly use criteria-based, ordinal scales to quantify impairments. Such techniques have minimal cost (beyond the clinician’s time) and exploit the impressive capability of our visual system to identify atypical performance such as slight asymmetry of gait, indicative of a stroke.

While criteria-based, ordinal scales may be useful in the clinic for making treatment decisions, their use is more problematic when used for clinical research and clinical trials (e.g., the modified Rankin Scale or Fugl-Meyer (FM) Assessment). The challenge is that our visual system may have evolved to identify atypical behavior, but we are only able to crudely quantify severity of atypical behavior from visual inspection, ¹ limiting the levels of impairment that can be reliably defined. Thus, there is a clear need for approaches to assess objectively, reliably and with high precision, neurological impairments and changes in impairments due to stroke and other neurological injuries/diseases.

There is growing recognition that kinematic and kinetic-based measures can provide excellent quantification of impairments after stroke.^2-6 A primary challenge remains how to appropriately characterize these impairments compared to healthy control performance. Age, sex, and handedness can often impact kinematic performance. Many studies manage these effects by using age and sex-matched controls, permitting statistical comparison for differences between patient cohorts and healthy controls (group effects).^7-10 However, comparisons to a healthy mean do not allow one to identify if an individual with stroke is impaired (outside the range of performance expected for healthy individuals). Another approach is to compare individual patients to a distribution of healthy individuals. Cortes et al (2017) ¹¹ compared reaching performance of individual participants with stroke to 12 healthy individuals of a similar age distribution making reaches with their dominant hand. They calculated the Mahalanobis distance to measure the distance of each individual patient’s reaching performance from the distribution of healthy reaches in order to quantify impairment. This is a step in the right direction. However, using a very small number of age-matched individuals to estimate the performance of a healthy population is problematic as there can be substantial deviation between the estimated and actual distributions. This approach also does not account for the impact of age, sex, or handedness on performance.

In this article, we introduce a statistical approach for quantifying impairment based on measures of neurologically healthy individuals. We first review some of the challenges with using criteria-based ordinal scales for quantifying impairments and recovery in order to highlight how kinematic-based measures can overcome at least some of these challenges. We then describe our approach that develops a statistical model of healthy performance using a large number of healthy controls and then use this model to transform performance of individuals with stroke into standardized units. We illustrate our approach using a visually-guided reaching task. However, it is important to recognize that our approach can be used with any spatial or temporal features of motor performance, collected with wearable sensors, markered or markerless motion capture, or even as simple as the time to walk 10 m. Finally, there has been considerable controversy and debate on how to quantify motor recovery following stroke.^12-18 Here we highlight how we can use our statistical-based approach to provide a simple and effective estimate of recovery for a cohort of individuals.

The Problem: Criteria-Based Scales

While there are many different criteria-based, ordinal scales to quantify sensory, motor, and cognitive impairments for different neurological injuries/diseases, we will focus attention on the properties of the arm motor component of the Fugl-Meyer Assessment ¹⁹ (FM), as this scale is commonly used to quantify upper limb motor impairments and recovery following stroke. Measures included in this scale are based on the pattern of recovery described by Twitchell ²⁰ with 33 individual items related to arm function. For example, volitional movements within synergies. The examiner instructs the individual to move their hand from the contralateral knee to the ipsilateral ear. Performance on each task is quantified using a 3-point scoring system: “cannot perform” (0), “partially perform” (1), and “fully perform” (2). ¹⁹ While FM uses a rather coarse rating system, there is excellent test–retest and inter-rater reliability scores for FM for those trained in the implementation of this scale.^21-25

Nonetheless, this three-point scoring system creates many clear problems when trying to quantify impairment. Most obvious is a large rounding error when using just three values or bins to define performance. Specifically, there is a considerable range in performance that can be defined as “partially perform” the task. Further, the bottom and top bin give us floor and ceiling effects, respectively. Floor effects reflect that two subjects “cannot perform” the task, but one individual is clearly more impaired than the other (i.e., one cannot move their arm at all, whereas another is able to initiate some movement but not enough to attain “partially perform”). Ceiling effects are the opposite, where two individuals pass the criteria for “fully perform,” but one individual looks fluid and graceful, whereas the other is just a bit slower and uncoordinated. This coarseness also creates rounding error for measuring recovery as subjects may display substantial improvements in performance after therapy, but nevertheless, land in the same bin because they could not attain the next criteria. Again, this leads to excellent test–retest and inter-rater reliability at the expense of coarseness.

Another problem with criteria-based scales is converting these ordinal labels of motor performance into interval units. In other words, the transition from “cannot perform” to “partially perform” (0 to 1) is construed as numerically equivalent to “partially perform” to “fully perform” (1 to 2). The validity of this assumption hinges on the placement of the criteria in performance, or hurdles. If performance is viewed as a 100 m race, the scoring system numerically places the lower bounds of “partially” and “fully” at 33.3 and 66.7 m along the track. However, these criteria may better represent 20 and 75 m, respectively. This again is difficult to assess with criteria-based scales which are selected more because it is easier for raters to identify if a criteria was met than based on some equal space along some measure of performance.

FM sums together performance on 33 items, with some tasks being fairly easy and others harder. There has been some work using Rasch analysis to hierarchically categorize the items in order to determine the specific types of behaviors that patients can and can’t perform at certain levels. ²⁶ Overall, the inclusion of many items in the clinical tool may help smooth out the coarseness and asymmetries in the criteria inherent in each item, and also reduce floor and ceiling effects. However, previous work has shown that 20% or more of individuals with stroke will be impacted by floor or ceiling effects in commonly used clinical measures of motor impairment, including FM. ²⁷

Figure 1 A highlights the upper extremity FM scores extracted from several studies,^13,28-32 and initially published in Hawe and colleagues. ¹⁴ “Stroke Initial” denotes assessment completed within 2 weeks of stroke, whereas “Stroke Final” denotes assessment completed ~3–6 months following stroke. FM scores for Stroke Initial are fairly broadly distributed, however, there is a preponderance of individuals at the bottom of the range with fewer individuals receiving a score in the middle and higher range. The reverse problem is observed for Stroke Final where there is a preponderance of individuals near the very top of the scale. Approximately 70 (19%) of 373 participants received the maximum possible score of 66 meaning they could fully perform all 33 items of the scale and almost half received scores >60. However, it is likely that many of the individuals scoring 66 display atypical behaviors by visual inspection (see 37). In effect, criteria-based scales like FM have a substantive ceiling effect, ranging from the abilities necessary to ‘fully perform’ the FM tasks up to and through the range of motor skills of neurologically healthy individuals (Figure 1B). As also noted in Figure 1B, two individuals may have different levels of motor skill pre-stroke. The impact of stroke may initially lead to similar levels impairment for these two individuals. In the cases depicted in Figure 1B, a score of 66 may represent a performance closer to the pre-stroke motor skill for Subject 2, but Subject 1 was a highly skilled individual pre-stroke and 66 is a long way from their pre-stroke “normal.”

Figure 1.

(A) Aggregate data for Fugl-Meyer scores for participants with stroke. ¹⁴ Black trace represents initial scores and grey trace represents final scores. (B) Schematic reflecting how the range of possible Fugl-Meyer scores falls below the range of abilities typically observed in healthy individuals. Gaussian distribution denotes variation of skills for healthy individuals. Vertical grey lines denote two individuals’ (S1 and S2) abilities are different pre-stroke, but are similar initially after stroke. (C) Exemplar subject performance on the visually guided reaching (VGR) task. Top traces are hand paths for the most affected arm (or the non-dominant arm for the healthy control participant). The black traces are the trials that correspond to the hand speed shown below in the bottom traces. Participant on the left is a control participant (male, 70 years, right handed) who had no impairments on the task (Z-Task Score = −0.69). The middle participant (male, 71 years old, right affected) was assessed 2 days post-stroke and showed mild impairments on the task (Z-Task Score = 2.82). Participant on the right (male, 69 years, right affected) was assessed 28 days post-stroke and showed more severe impairments on the task (Z-Task Score = 7.12).

The Solution: Kinematic-Based Assessment and Statistical Models of Healthy Performance

Human motor performance spans a spectrum, as emphasized and embraced at the Olympics. Even for a simple task like reaching, some individuals will move faster, straighter, and require minimal corrective movements compared to others. Thus, there is a distribution that characterizes healthy performance, with some better, and some worse.

For situations like stroke, it is not possible to know what was an individual’s pre-stroke perfomance, and thus, it is not possible to measure an individual’s true impairment from pre-stroke baseline. However, pre-stroke performance for a group of individuals should be equivalent to the performance of a group of healthy individuals (assuming no pre-existing neurological issues). Thus, expected mean and range of performance of individuals pre-stroke can be estimated by assessing the performance of a very large sample of healthy individuals.

Demographic factors can also impact performance. For example, reaction times increase with age ³³ so what is atypical for a 40-year-old may be typical for an 80-year-old. Thus, factors such as age, sex, and handedness, need to be considered when estimating performance, just like growth charts estimate height and weight based on age and sex.

Thus, the key to our approach is to characterize the performance of a large cohort of neurologically healthy individuals. We develop mathematical models to capture healthy performance and then use these models to quantify performance of individuals with stroke providing quantitative measures of impairments relative to healthy performance. Reasonable estimates of the mean of a distribution may only require 30 to 50 individuals, but it requires many hundreds of individuals to estimate the shape of a distribution with reasonable precision (i.e., mean, variance and skew). Further, the model requires a good sample of all ages, both sexes, or any other factor that may impact performance. This requires hundreds of healthy individuals to be assessed. The benefit is that impairments can be defined for each individual. That is, if the patient is a 69-year-old, right-handed female, then we can assess their performance based on what is expected for a 69-year-old right-handed female. Once a normative model is established, there is no need to collect age- and sex-matched individuals for each study, as these factors are already considered in the model.

Here, we highlight our approach to quantify performance using a visually-guided reaching task, an upper limb task commonly assessed due to its importance for many daily activities.^34-38 For didactic purposes we use the Kinarm robotic platform, for which the basic paradigm has been described previously^39-43 and used to assess large cohorts of individuals, both neurologically health and those following stroke. Briefly, the Bimanual Kinarm Exoskeleton lab consists of 2 robotic exoskeletons that support the upper arms and forearm/hands to allow free movement of both arms in the horizontal plane. Participants are seated in a height-adjusted chair that provides truncal support, if required. The robot’s shoulder and elbow joints are aligned with the subject’s joints, and the robot and virtual reality systems are then calibrated for each participant. A virtual reality system projects visual targets in the horizontal workspace along with a visual representation of the location of the index fingertips. Details of the Visually-Guided Reaching (VGR) task are described elsewhere. ³⁹ In general, the VGR task required participants to keep their hand at a visual spatial goal in the center of the workspace. When another spatial goal was presented, participants were asked to quickly and accurately reach to and then maintain their hand at this second spatial goal.

We have assessed 321 neurologically healthy individuals and 112 individuals with stroke. The associated demographic information is provided in Table 1. Each participant with stroke was assessed at four times points: Stroke Initial for our data set was collected on average 1–2 weeks post-stroke (0.25–0.5 months, range: 1–28 days). Subsequent time points were collected at ~6 weeks (1.5 months, 4–9 weeks), ~12 weeks (3 months, 10–20 weeks), and ~26 weeks after stroke (Stroke Final, 6 months, 21–38 weeks after stroke). Participants were recruited from the stroke unit at Foothills Medical Centre and the inpatient stroke rehabilitation unit at Dr. Vernon Fanning Care Centre in Calgary, Alberta, Canada. Healthy participants were recruited from the community in the cities of Kingston, Ontario, Canada, and Calgary, Alberta, Canada. All participants provided informed consent prior to participation in the study. This study was approved by the Queen’s University Health Sciences and Affiliated Teaching Hospitals Research Ethics Board (#ANAT042-05), and the University of Calgary’s Conjoint Health Research Ethics Board (#22123).

Table 1.

Participant Demographics

	Healthy Controls (N = 321)	Participants with stroke (N = 112)
Age: mean (range)	48 (18–84)	61 (21–84) years
Sex	156 M/165 F	79 M/33 F
Handedness	300 RH/18 LH/3 mix	105 RH/6 LH/1 mix
Affected arm	–	61 Left/51 Right
Type of Stroke	–	93 ischemic/19 hemorrhagic
Lesion Location* [# of subjects]	–	[C/SC/C+SC/Cb/Br/Cb+Br/Sp/Uk] [24/42/26/2/12/1/1/4]

^*C = Cortical, SC = Subcortical, C + SC = Cortical + Subcortical, Br = Brainstem, Cb = Cerebellum,

Cb+Br = Cerebellar + Brainstem, Sp = Spinal, Uk = Unknown.

Figure 1C highlights performance of one healthy control and two individuals with stroke, one with mild and one with more severe impairments. The healthy individual displays relatively straight reaches, and consistent reaction times and hand speeds. The individuals with stroke show varying amounts of directional errors, more corrective responses and slower reaction times, and peak hand speeds.

Task Parameters

While one could just identify whether or not a reaching movement attained a goal, kinematic-based measures allow us to assess the quality of that movement. This quality of performance can be captured using various spatial and temporal features of movement, such as reaction time, movement time, initial direction angle, number of corrections, etc.^34-36,39 Individuals may have impairments that specifically impact one aspect of performance. For example, increases in reaction time are commonly observed in those with spatial neglect. ⁴⁴ Similarly, impairments may be more pronounced in the initial phase of the reach, or in the final phase of the reach depending on which hemisphere is damaged. ⁴⁵ For example, uncoordinated motion (limb ataxia) is commonly observed in those with cerebellar lesions. ⁴⁶ Thus, it is important to track several measures of performance, defined here as task parameters, to capture the unique patterns of impairments that may be present for an individual. In VGR we collect fourteen different parameters within the domains of postural control of the arm, reaction time, initial movement, corrective response, total movement metrics, and target success. For a full description of each parameter please see Supplementary Material.

Figure 2 illustrates the basic steps in our process to quantify impairment. Over the years we have developed and refined our approach to develop best practices for this type of quantification, but there are other methods (see discussion). Our crucial first step is to develop a model of neurologically healthy performance that transforms values in raw units of time or distance into a standard, normal distribution, while also removing the influence of demographic factors on performance. Box-Cox equations are a common technique to transform distributions from an asymmetrically skewed distribution to one that is normal. ⁴⁷ Linear regression models can then be used to adjust performance for factors such as age, sex, or handedness. These regressions can be weighted appropriately to account for heteroskedasticity in the original distribution. The normal distribution is useful as statistical tests commonly assume data are normally distributed. The standard normal distribution provides a simple approach to quantify performance with a mean and median both equal to zero and the standard deviation of the distribution equal to 1. Thus, performance of 68% of individuals is between 1 and −1, 95% between 1.96 and −1.96, and 99% between 2.58 and −2.58. Measures of healthy performance transformed to a standard normal distribution create a common language where differences and measures are scaled in units of standard deviation (Z-units). One can do the same for each spatial and temporal measure permitting direct comparisons across task parameters. For example, an individual with a reaction time of 300 ms may convert to −1.64 in Z-units (5th percentile), whereas movement time of 850 ms converts to 0 in Z-units (50th percentile).

Figure 2.

Conversion of kinematic measures into standardized units. A) Calculations for task parameters. Step 1 creates a model that converts raw parameter scores to a standard normal distribution for a kinematic measure for a large healthy population. The distribution of values for a measure of performance (shown here is reaction time) is transformed to a standard normal distribution using Box-Cox equations and the effects of age, sex, and handedness are taken into account using weighted linear regression. Step 2 illustrates the calculations for the same task parameter for participants with stroke. For each participant, the outcome measure (i.e. reaction time) is transformed using the individual’s demographic information and the coefficients from the healthy performance model (from Step 1) to generate a task parameter value in Z-units. Vertical dashed line in right panel indicates 95th percentile performance for healthy controls (1.64 Z-units).

Critically, the same model that is used to transform neurologically healthy performance into standard Z-units is then used to convert task parameters for each subject with stroke into Z-units (Figure 2, Step 2). With measures of performance now in Z-units, impairment can be defined as performance beyond some percentage of the healthy population, commonly the 95^th percentile which is 1.64 Z-units. If poor performance could reflect either a large negative or positive value, then the line for impairment should be defined as ±1.96 Z-units (<2.5 and >97.5 percentiles). Correction for factors such as age means that two individuals with the same reaction time in milliseconds (star and filled circle in left panel of Figure 2, Step 2) receive different Z-scores when transformed by the model (right panel).

Figure 3A highlights distributions of reaction time with the affected arm for individuals 0.5 months (mean 11 days, range 1-28 days) after stroke (Stroke Initial) and 6 months (mean 26 weeks, range 21-38 weeks) after stroke (Stroke Final). The distribution of reaction times for Stroke Initial is shifted rightward and spread more broadly compared to the control distribution. Approximately 40% of individuals with stroke are identified as having impaired reaction times (i.e., Z > 1.64). The distribution of reaction times for Stroke Final is much less shifted compared to control and only 17% are identified as impaired.

Figure 3.

Histograms and probability distribution curves for (A) reaction time (B) movement time (C) Z-Task Scores (affected arm) and D) Z-Task Scores (less affected arm) for participants with stroke (N = 112) who completed the Visually Guided Reaching (VGR) task. Black traces are theoretical probability distribution curves for the healthy control subjects (mean/median = 0 and standard deviation = 1). Red curves and histograms represent data collected at the initial time point after stroke (Stroke Initial: ~0.5 months post-stroke). Blue curves and histograms represent the data collected at the final time point (Stroke Final: ~6 months post-stroke). Distribution curves calculated using a probability density estimate using a normal kernel function.

Figure 3B displays the distributions for movement time during reaching. For Stroke Initial the distribution is shifted with an observable bimodal distribution in the scores (Hartigan’s Dip statistic = 0.05; 95% likelihood of bimodality) ⁴⁸ and 70% of individuals were identified as impaired. For Stroke Final, the distribution is shifted back towards the control distribution and is largely unimodal (Dip statistic = 0.023, 1% likelihood of bimodality), with a few individuals that continue to have large values, and only 29% are impaired. The distributions for reaction and movement time clearly highlight that stroke leads to a greater impairment in the latter compared to the former at both time points, although this varies across individuals.

It is important to note that while the use of standardized units permits comparisons across parameters, some caution is necessary. In particular, comparison across parameters (or tasks) may be problematic when values are far from healthy performance. Near the healthy range, it is reasonable to assume that a reaction time of 2 in Z-units is better than a movement time of 3, but a difference of 1 may not be meaningful when the two values are 9 and 10, respectively. Each parameter has limits on the range of potential values dictated by the distribution of performance for healthy individuals (mean, variance, and skew) as well as limits of possible values that can be attained. Reaction time and movement time are both limited by the maximal time for a given trial, whereas initial direction angle ranges from 0 to 180°. Floor effects can still remain in tasks such as reaching for individuals that cannot move the arm to the central target to initiate a trial. This is partially minimized by having the robotic exoskeleton provide weight support so that some individuals with substantial weakness can move the limb to some degree.

Task Scores

The use of many different spatial and temporal parameters provides considerable richness to understand how stroke impacts performance, and potentially, specify targets for a rehabilitation plan. However, many measures of performance can also lead to information overload and difficulties in assessing overall impairment in a given task. Thus, there is also a need for a more general measure of performance. Our tentative strategy is to aggregate task parameters to generate a single Task Score (see Supplemental Figure 1). Distributions that characterize performance in a task parameter are either one-sided or two-sided. One-sided distributions either have poor performance indicated by larger values (e.g., reaction time) or smaller values (e.g., hand speed). In contrast, two-sided distributions have poor performance indicated by both larger and smaller than normal scores (e.g., arm performance asymmetries in bimanual tasks). ⁴⁰ In order to aggregate different task parameters equally, one-sided task parameters are transformed such that best performance is 0 and poor performance is a large positive value. For two-sided parameters, (where poor performance could be either a large positive or a large negative) the absolute value is used. The end result is that all parameters are now aligned such that 0 is best performance and higher values reflect poorer performance. We chose to aggregate the values using root-sum-squares (RSS) and then to transform to standard normal space using Box-Cox equations. This model, developed from the performance of healthy controls, is then used to transform the aggregate measures for each individual with stroke, again into Z-units (termed Z-Task Score).

Figure 3C displays Z-Task Scores for the affected arm of individuals with stroke. The distribution for Stroke Initial is shifted compared to the standard control distribution with 85% of individuals identified as impaired. Task Scores tend to decrease for Stroke Final and shift towards the control distribution, but 49% of individuals are still identified as impaired. Figure 3D displays the Z-Task Scores for the less affected arm. The distribution for Stroke Initial is shifted compared to the standard control distribution and 46% of individuals were identified as impaired. In contrast, the distribution for Stroke Final is similar to the healthy control distribution and only 13% of individuals are identified as impaired.

Estimates of Recovery

Measures of recovery are important as they provide essential information to assess the efficacy of novel therapeutic interventions. Figure 4A displays hand traces of a participant with stroke assessed at four time points: 0.5, 1.5, 3, and 6 months following stroke. Their performance improves at each time point which is reflected by the decreasing Task Scores: 10.68 reflects a severe impairment at Stroke Initial, 4.36 and 2.36 reflect moderate impairments at 1.5 and 3 months post-stroke, respectively. Task Score of 2.31 at Stroke Final, 6 months post-stroke, shows that the participant has improved but still has some mild impairments. Figure 4B displays measures of impairment at the four time points for the stroke cohort. Z-Task Score means are plotted against their Standard Deviations (SD). The affected arm displays both a very large mean (5.43) and SD (3.54) reflecting the broad range in performance for Stroke Initial, as also displayed in Figure 3C. Mean and SD values reduce in magnitude with time, signifying a reduction in impairment and a reduction in the range of performance in the stroke cohort, respectively. The less affected arm also shows an improvement over time reaching a mean near zero and an SD of about 1.5 within 3 months. Individuals with stroke showed statistically significant improvements from initial measures to 1.5 months, and from 1.5 months to 3 months for reaching with both the affected and less affected arms. While there is a small reduction in the means from 3 to 6 months, it is not statistically significant (paired t-test, P > .05).

Figure 4.

(A) Hand traces for the VGR task plotted for an exemplar participant assessed at 0.5, 1.5, 3, and 6 months post-stroke. (B) Mean Z-Task Score plotted vs the Standard Deviation of Z-Task Scores. Black circle represents healthy controls, white circles represent the affected arm of participants with stroke and white triangles represent the less affected arm of participants with stroke. Error bars denote 95% confidence interval for the mean and SD estimated by measuring the mean and SD from repeat samples with replacement of Z-Task Scores from the original distribution and repeating this process 1000 times. (C) Left axis: the percentage of individuals classified as impaired based on Z-Task Score (i.e., Task Score higher than 95% of healthy controls). Right axis: % Recovery plotted across time points. Recovery is calculated as the mean of ((Z-Task Score initial—Z-Task Score T(i))/Z-Task Score initial)*100. Circles represent data collected for reaches with the most affected arm and triangles represent data collected for the less affected arm. Errors bars denote 95% confidence interval for recovery based on repeat samples with replacement of individuals from the stroke cohort.).

As the distribution of healthy performance in Z-units has a mean value of zero, recovery as a population can be estimated simply as the difference between mean initial impairment and mean impairment at a given time point, divided by the mean initial impairment (Stroke Initial). Figure 4C displays recovery (mean and confidence interval) and the percentage of individuals impaired at different time points following stroke. The 95% confidence intervals are calculated using repeat samples with replacement of individuals from the stroke cohort. These estimates suggest that individuals with stroke display ~60% recovery in performing VGR in the affected arm at 6 months (confidence interval 48 to 66%). We can also track the percentage of individuals impaired which decreases over time to approximately 44% at 6 months. The less affected arm displays almost 100% recovery at 6 months, with a confidence interval of 80 to 112%. The percentage of individuals who are impaired at 6 months with the less affected arm approaches 10%, which is just above the 5% of individuals who would be expected to show impairments in a healthy population, based on using a 95% cutoff for defining impairment.

It is important to recognize that the calculation of recovery is for the population and not individual participants. For example, in some cases individuals with stroke can attain Z-Task Scores below zero, denoting performance above average for healthy individuals. However, this does not mean that this individual had greater than 100% recovery. As mentioned above, we do not know an individual’s pre-stroke abilities so we cannot accurately assess impairment or recovery at the individual level. In situations where baseline tests are available, recovery can be estimated for each individual. In contrast, the distribution functions do allow us to estimate the probability that someone will attain some level of recovery. For example, reaching performance with the affected arm will fall within a typical healthy range for ~50% of individuals 5 months after stroke. Further, recovery distributions could be quantified for a sub-set of individuals with a specific level of initial impairment (i.e., Initial Task Score between 7 and 10).”

While we and others have recommended against this approach,^{12,14,15,49,50} we also estimated recovery based on commonly used techniques of regressing initial impairment versus change in impairment for comparison purposes (Figure 5A and B). For the affected arm, the regression slope is 0.55 which is similar to the 60% recovery calculated for the affected arm in Figure 4C. However, for the less affected arm, the slope is 0.63 which is far lower than the ~100% recovery in the less affected arm estimated in Figure 4C. These plots of recovery look different than observed in previous studies that plot initial impairment versus change in impairment based on FM because our Z-Task Scores do not have a ceiling effect. When we add an artificial ceiling to our Z-Task Score data and cap the possible values at 0 (Figure 5C and D) we can create a plot that looks more like the regression plots generated for the FM scale.^14,17,51,52 The calculated slope for the affected arm is now 0.59 and for the less affected arm is 0.73. For the less affected arm regressions still predict far lower than the 100% recovery, as predicted in Figure 4C.

Figure 5.

(A) Recovery plotted for VGR Z-Task Scores for the affected arm. Black lines are the linear regression lines represented by equation in the top left corner and r-values are the Pearson correlation coefficient. (B) Same as (A) but for the less affected arm. (C) The same data plotted for VGR Z-Task Scores for the affected arm but with an artificial ceiling added to the data of Z-Task Score = 0 (median of healthy performance). (D) Same as (C) but for the less affected arm.

Patient-Specific Impairments

As stated earlier, we cannot quantify true impairment or true recovery for each individual following stroke as we do not know their pre-stroke abilities. However, the use of neurologically healthy individuals provides a basis to quantify patient-specific impairments. ⁵³ This is a similar concept to diagnostic blood tests or many other clinical tests, where individual samples are compared to a normalized range based on healthy individuals. Figure 6 displays Z-Task Scores for Stroke Initial and Stroke Final for each individual (A: affected arm, B: less affected arm). The horizontal and vertical dashed lines denote the 95th percentile performance for healthy controls, and thus, values greater than this cutoff signify impairment. The solid diagonal unity line denotes identical performance for both initial and final assessments. The dashed diagonal lines denote the region of equivalence, where there is no significant change in performance between the two time points (open circles). Measures of significant change were identified by repeat testing of healthy individuals to assess performance consistency. ⁵⁴ Below the lower diagonal dashed line denotes significant improvement in the second assessment, whereas above the upper diagonal dashed line denotes significant degradation in performance (both indicated by filled circles). Not surprisingly, many individuals significantly improved reaching performance. In some cases, some residual impairment remained (above horizontal dashed line), whereas in other cases, performance returned to within the range for healthy individuals (below horizontal dashed line). A few individuals that were already within the healthy range for the initial assessment statistically improved performance. Z-Task Scores below zero denote performance better than the average for healthy individuals.

Figure 6.

VGR Z-Task Score in Z-Units plotted for individual participants with stroke for the affected (A) and less affected (B) arms at Stroke Initial (0.5 months post-stroke) vs Stroke Final (6 months post-stroke). Vertical and horizontal dashed lines represent the cutoff line for impairment compared to 95% of healthy controls. Solid black diagonal lines represent the unity line and dashed diagonal lines represent significant change values calculated from repeat testing in a healthy normal population. Circles are filled grey if they surpassed thresholds for significant change from the initial exam to the final exam.

Discussion

Our objective here is to describe how kinematic-based measures can deal with many of the key drawbacks of criteria-based, ordinal scales. Here we propose using the natural distribution of performance observed in healthy individuals as a foundation to assess impairment and recovery. Standardized units create a common language for comparing performance. Age and other factors impact performance, obfuscating interpretation of performance in native units, but can be corrected when transforming into standardized units. We used a visual-guided reaching task to demonstrate our approach, but the basic approach can be applied to any behavioural task or technology (i.e., wearable sensors, motion capture, force plates) where performance can be quantified with some level of resolution, including even simple measures like the 10m walk test.

It is recognized that quantifying the quality of tasks like reaching is important when assessing impairments, ³⁶ although it is less clear which parameters or measures of performance should be considered when assessing this quality. In the VGR task we use 14 parameters related to the motion and position of the hand to quantify performance of postural stability before movement, the speed, and accuracy of the initial motor response as well as measures of secondary movements to attain the spatial goal (for simplicity, here we only presented reaction time and movement time). However, over 150 different kinematic metrics have been used previously to characterize upper limb movements across studies. ⁵ Some of this variety reflects differences in technologies (motion capture versus interactive robotics) or behavioural task (2D or 3D movements). In other cases, studies focus on more detailed measures related to the joints to explore issues related to joint coordination.^55,56 While the plethora of scientific questions examined dictates the need for many different measures to be assessed, some standardization would clearly be useful to facilitate ease of interpretation, and for clinical trials and meta-analyses.

While individual metrics allow us to observe the quality of movement and identify potential therapeutic targets, there is also a need for a singular measure of performance to characterize overall impairment and recovery. This is particularly true for clinical trials, where a global measure of performance is desirable as an outcome measure to determine efficacy of treatment. How many and which measures of performance should be included in an overall performance metric is an open question. Inter-rater reliabilty analysis may be useful in identifying which parameters best separate performance of different individuals, and thus, would be advantageous to include in an overall score. Other ways to reduce the data could include principal component analysis techniques based on extracted parameters^11,57 or raw kinematic data, ⁵⁸ regression models that weight parameters based on some criteria, ⁵⁹ or machine learning techniques.^2,60

Similarly, once parameters are identified, there are many possible ways to aggregate different measures of performance. We chose root-sum-squares which equally weights each parameter, and thus values with larger deviations from healthy performance are weighted more. An alternate approach to the Euclidean distance is the Mahalanobis distance (MD) that removes inter-correlations between parameters. ⁶¹ However, measures of MD can have unexpected outcomes. For example, reaction and movement times tend to co-vary together both in healthy individuals and individuals following stroke. However, MD distance will increase more for an individual if movement time increases without a corresponding increase in reaction time. That is, an individual with only an impairment in movement time and not reaction time can be scored as more impaired as compared to someone that was impaired in both movement and reaction time. It is possible that one can receive an MD that is beyond the healthy range, and yet all task parameters are within the healthy range but the pattern of values is atypical. This is not to say that using the MD is not worthwhile. Where Task Scores using Euclidean distances are useful for describing overall impairment, Mahalanobis distances may be useful to identify when individuals are impaired in unexpected ways. Regardless of the approach, we suggest that transforming the data based on performance of a large cohort of healthy individuals should still be performed to aid in the interpretation of the level of impairment and the amount of recovery.

There has been considerable debate on how to assess recovery following stroke. A popular approach with the FM Scale has been to regress the amount of impairment observed from the first time point (66-FM Initial) versus the difference between the two assessments (FM Final-FM Initial). The resulting slope of the regression tends to be near 0.7 and explains a substantial amount of variance. This has been interpreted as evidence that most strokes recover 70% of function.^13,29 Subsequent studies have found that this 70% proportional recovery rule holds for many other clinical scales, including lower extremity FM,^52,62 aphasia, ⁶³ hemispatial neglect,^28,64 and even the Functional Independence Measure. ¹⁴ The ubiquitous nature of these observations across multiple stroke impairments and scales has drawn criticism. Several recent studies have challenged this regression approach due to mathematical concerns, related to mathematical coupling and variance reduction between time points.^{12,14,15,49,50} It is important to recognize that these mathematical concerns do not mean proportional recovery may not exist. Simply that the use of regression models between initial impairment and change in impairment is flawed and poor evidence, at best, to support a 70% proportional recovery rule.

Our approach for quantifying recovery highlights the clear problem with regressing the amount of initial impairment versus change in impairment. For the impaired arm, our measures are comparable although slightly lower to that predicted with regression, but there are glaring differences when quantifying recovery of reaching with the less affected arm. The regression approach found recovery was 63%, whereas our comparison to the healthy cohort suggests virtually complete recovery of function (Figure 5E). This is obvious when simply looking at the distribution of performance for healthy controls and Stroke Final in Figure 3D, as they are almost identical. This highlights the clear benefits of quantifying impairment as well as recovery based on comparisons to kinematic-based measures of performance observed for healthy neurologically-intact individuals.