Abstract
Background
A common assumption is that changes in upper limb (UL) capacity, or what an individual is capable of doing, translate to improved UL performance in daily life, or what an individual actually does. This assumption should be explicitly tested for individuals with UL paresis post-stroke.
Objective
To examine changes in UL performance after an intensive, individualized, progressive, task-specific UL intervention for individuals at least 6 months post-stroke.
Methods
Secondary analysis on 78 individuals with UL paresis who participated in a Phase II, single-blind, randomized parallel dose-response trial. Participants were enrolled in a task-specific intervention for 8 weeks. Participants were randomized into 1 of 4 treatment groups with each group completing different amounts of UL movement practice. UL performance was assessed with bilateral, wrist-worn accelerometers once a week for 24 hours throughout the duration of the study. The six accelerometer variables were tested for change and the influence of potential modifiers using hierarchical linear modeling.
Results
No changes in UL performance were found on any of the 6 accelerometer variables used to quantify UL performance. Neither changes in UL capacity nor the overall amount of movement practice influenced changes in UL performance. Stroke chronicity, baseline UL capacity, concordance, and ADL status significantly increased the baseline starting points but did not influence the rate of change (slopes) for participants.
Conclusions
Improved motor capacity resulting from an intensive outpatient UL intervention does not appear to translate to increased UL performance outside the clinic.
MeSH terms: stroke, upper limb, task-specific training, accelerometry
Introduction
A large proportion of individuals post-stroke experience significant difficulty incorporating their paretic hand into daily activities and they often identify improved upper limb (UL) function as a top rehabilitation priority.1–4 Despite the resources spent on stroke rehabilitation, individuals continue to experience ongoing barriers with performing activities at home and will ultimately discontinue 57% of their daily activities.5 Given the importance of both upper limbs in daily activity,6 decreased daily performance following stroke is likely influenced by UL paresis. Clinical interventions to address UL paresis are often aimed at improving UL capacity, which describes an individual’s ability to execute a task, or what a person is capable of doing, within the structured environment of a clinic or laboratory.7 A common assumption is that improvement in UL capacity directly translates to improved UL performance. Performance describes what individuals actually do in their current environment, outside of the clinic or laboratory.7 Recent research has emphasized the importance of measuring capacity and performance separately.8,9
Protocol-based UL motor interventions can improve capacity after stroke.10–14 Less clear is whether gains made in UL capacity, measured with standardized assessments (e.g. Action Research Arm Test15 [ARAT] and Wolf Motor Function Test16), translate to improved UL performance, or use, in daily life. Several studies have reported increased UL performance when measured via self-report,10,17–19 and individuals with larger improvements in UL capacity demonstrate a positive trend towards clinically significant changes in self-reported performance.10,17,19 Self-report measures, however, are subject to many report biases including cognitive deficits20,21 and social desirability.22 Self-report measures often rely on an individual’s ability to recall activities completed over a previous week, which may be of particular difficulty for individuals with stroke, given the high prevalence of cognitive impairment.23,24 Pilot studies that have used a more quantitative measure of UL performance, i.e. accelerometry, report little to no improvement in UL performance in daily life, despite gains in UL capacity.25–27 The growing emphasis on efficient, evidence-based rehabilitation services demands an evaluation of the relationship between change in UL capacity and change in UL performance in post-stroke chronic UL paresis.
In this analysis, we examine changes in performance in the community that resulted from an individualized, intensive, progressive, task-specific UL intervention. We explicitly test the assumption that increased UL capacity translates to increased UL performance using data from a recent clinical trial.14 Finally, we examine the effect of dose (i.e. amount) of task-specific movement practice on UL performance in daily life.
Methods
This paper is a secondary analysis from a Phase II, single-blind, randomized, parallel dose-response trial (NCT 01146379).14 Individuals were recruited for this study via the Brain Recovery Core database and the Cognitive Rehabilitative Research Group at Washington University in St. Louis, MO. Inclusion criteria were: 1) ischemic or hemorrhagic stroke as determined by a stroke neurologist and consistent with neuroimaging; 2) time since stroke ≥ 6-months; 3) cognitive skills to actively participate, as indicated by scores of 0–1 on items 1b and 1c of the National Institutes of Health Stroke Scale (NIHSS);28 4) unilateral UL weakness, as indicated by a score of 1–3 on item 5 (arm item) on the NIHSS; and 5) mild-to-moderate functional motor capacity of the paretic UL, as indicated by a score of 10–48 on the ARAT.15,29,30 The lower limit of 10 on the ARAT meant that participants had at least some ability to open the hand, grasp and lift off the table at least 2–3 test items. Exclusion criteria were: 1) participant unavailable for 2-month follow-up; 2) inability to follow-2-step commands; 3) psychiatric diagnoses; 4) current participation in other UL stroke treatments (e.g. Botox); 5) other neurological diagnoses; 6) participants living further than one hour away and were unwilling to travel for assessment and treatment sessions; and 7) pregnancy. The trial was approved by the Washington University Human Research Protection Office and all participants provided informed consent.
Clinical trial details and primary outcomes have been reported.14 Briefly, the primary aim of the trial was to determine the range of doses of UL task-specific practice that produce the largest change in UL functional capacity in individuals with chronic UL paresis. Participants were randomized into one of four groups. Dose was quantified by the total number of repetitions achieved over the course of the intervention. The four dose groups were: 3200 (100 repetitions/session; median = 13.6 hours of active practice), 6400 (200 repetitions/session; 20 hours of active practice), 9600 (300 repetitions/session; 26.3 hours of active practice), and individualized maximum, respectively. Participants completed four treatment sessions per week for 1-hour over eight weeks. The individualized maximum group completed 300 repetitions per session and continued their enrollment past the 8 weeks until specific stopping criteria were met (32.8 hours of active practice).14 The >32 hours of active practice in the individualized maximum group is likely equivalent to the total scheduled therapy time in the constraint induced movement therapy trials (≥ 65 hours),31 given that active practice is often 50% or less of scheduled time.32,33 At least once every two weeks throughout the length of enrollment, participants were asked questions related to UL performance at home (e.g. what new activities have you tried with your arm?) and new activities were identified and discussed to help facilitate increased UL performance at home. The primary outcome for the trial was the ARAT, a valid and reliable measure of UL capacity.34–36
Performance Measures
UL performance was measured with bilateral, wrist-worn accelerometers (w GT3X+, Actigraph, Pensacola, FL). Accelerometers are a well-established, valid, and reliable instrument for quantifying UL performance in both non-disabled adults6,9 and individuals with stroke.9,25,37–40 Accelerometers record accelerations along three axes in activity counts where 1 count = 0.001664 g. Data were sampled at 30 Hz and activity counts were binned into 1-second epochs for each axis using ActiLife 6 (Actigraph Corp., Pensacola, FL) software. Activity counts across each axis were combined to create a single vector magnitude value (√x2+y2+z2) for each second.
Each participant wore bilateral, wrist accelerometers once a week for 26 hours throughout the intervention, at the conclusion of the intervention, and at 2-months follow-up. Using custom-written software in MATLAB (Mathworks Inc., Natick, USA) the first two hours of each recording were removed because this included the time in therapy session plus transportation home afterwards. Data from the remaining 24 hours at home were used for this analysis. The single day wearing period was chosen because previous research has shown this is an adequate representation of performance in non-employed adults6,40,41 and to ensure increased adherence to wearing the accelerometers multiple times over the course of the study. Accelerometers are waterproof so participants could wear them for all activities, including bathing. Accelerometers were worn during the night (sleeping) and those data are part of the 24 hours. Accelerometers were returned the next treatment session and the data were downloaded using ActiLife 6 software.
Six variables were calculated from the data collected over the 24 hours at home, with each variable quantifying different, but related, aspects of UL movement. Upper limb movements associated with walking were included in our calculations. Previous work has established that walking does not influence the accelerometer ratio variables in individuals post-stroke.37 Although inclusion of walking does not change the non-ratio variables for neurologically-intact adults,42 it is possible that the inclusion of walking could result in an overestimation of the non-ratio variables for participants with stroke. We first examined summary variables of UL performance with the use ratio (also called activity ratio) and hours of use. The use ratio is the hours of paretic limb use divided by the hours of non-paretic limb use and quantifies the contribution of the paretic limb relative to the non-paretic limb to activity.6 Healthy, neurologically intact adults (54.3 ± 11.3 years of age, 53% female, and 84% right hand dominant, recruited to match the demographic characteristics of the trial participants)9 have a use ratio of 0.95 ± 0.06, indicating nearly equal amounts of UL use during activities.6 A use ratio value close or equal to 1 indicates nearly equal durations of activity from both limbs while values less than 1 indicate greater non-paretic activity and values greater than 1 would indicate more paretic UL activity.6 The total hours of use is the total amount of time, in hours, the paretic limb was active, as measured by summing the seconds when the activity count was > 2, and is a broad measure of paretic limb activity over the recording period.6,9 Neurologically intact adults use their dominant UL 9.1 ± 1.9 hours and their non-dominant UL 8.6 ± 2.0 hours.6
We then more closely examined, on a second-by-second basis, the contribution of both limbs to activity and the intensity of movement with the magnitude ratio and bilateral magnitude, respectively. The magnitude ratio is the natural log of the vector magnitude of the paretic UL divided by the vector magnitude of the non-paretic UL, and describes the contribution of both limbs to an activity for each second of data.43 A magnitude ratio value of 0 indicates both ULs contributed equally to an activity.9,43 A negative magnitude ratio value indicates greater non-paretic UL activity and positive values indicate greater paretic UL activity.9 Across 74 healthy, non-disabled adults, the median magnitude ratio value (median of all the seconds recorded during 24 hours) averages −0.1 (0.3), indicating that both ULs are used nearly the same amount during activity.9 The bilateral magnitude measures the intensity of UL activity by summing the vector magnitude of the paretic UL and the non-paretic UL.43 The bilateral magnitude distinguishes between high intensity and low intensity movements, for every second of data. Bilateral magnitude values of 0 indicate no movement and increasing values are indicative of more intense UL movement. A referent median value of 136.2 (36.6) has been established in non-disabled adults.9 Higher values are associated with activities requiring larger, faster movements, (e.g. placing boxes on an overhead shelf).43 A low bilateral magnitude value would indicate smaller, less intense movements such as chopping vegetables.44
The final quantification of UL performance examined only the paretic limb performance using the median paretic acceleration magnitude and the acceleration variability. The median paretic acceleration magnitude captures the individual’s median acceleration value over the entire recording period.45 The acceleration variability is the variance of the mean acceleration value over the recording period and explains the average distance of the paretic accelerations from the mean acceleration.45 A higher value for both the median acceleration magnitude and acceleration variability indicates more overall UL movement and greater variability of movement, respectively.
These variables can detect differences between participants with stroke9 and, with the exception of the bilateral magnitude, are responsive to change in UL function following a task-specific intervention in individuals with UL paresis post-stroke.45 While not responsive to change in UL function, the bilateral magnitude may be a valuable variable for quantifying the intensity of bilateral movement, which is of interest to rehabilitation professionals.
Statistical Analysis
All data were analyzed in R, an open source statistical computing program. The primary analysis used hierarchical linear modeling (HLM), also referred to as linear mixed effects regression analysis,14,46 for all six accelerometer variables. HLM is applied to longitudinal data and is an extension of a traditional regression analysis.46 In contrast to repeated measures ANOVA, HLM allows for modeling of individual intercepts and slopes over time in addition to modeling potential moderators of the intercepts and slopes. HLM does not require the same number of assessments across participants and can account for missing data, therefore participants with varying assessment sessions can still be included in the analysis.46 Slopes for each variable were of primary interest for this analysis, as they quantify the amount of change in UL performance over the duration of the study. Preliminary analyses indicated that nonlinear model components were not necessary. The group level intercepts and slopes are derived from the individual intercepts and slopes for each variable.46
We initially analyzed change in UL performance across the entire sample by testing growth curves for all six variables, with individual time trajectories nested within participants (Model 1). This model allowed us to estimate the intercept and slope for the entire sample. Next, we tested the relationship between change in UL capacity (i.e. ARAT score) and change in UL performance with nested models. Participants were stratified into two groups, those who improved ≥ 6 points on the ARAT and those who did not improve at least 6 points, as the 6 point value has been previously described as an estimate of the minimal clinically important difference for individuals with chronic stroke.35 These two groups (dummy coded) were added to Model 1 and time was nested within each participant, with individual intercepts and slopes allowed to vary randomly. We evaluated the potential influence of change in UL capacity on both the intercepts and slopes (via group × time interaction). Nested models were compared using χ2 tests and the final, best fit model for change in UL capacity was identified (Model 2). Additionally, a new series of nested models were created to test for a potential dose effect on UL performance by adding treatment group (dummy coded), and a group by time interaction to Model 1, yielding a final model identified as Model 3. Outliers were identified using Cook’s distance and when necessary, models were re-evaluated with outliers excluded. Across all levels of analysis, no outliers significantly influenced the results. We verified the inferences reported from the models using bootstrapping procedures and no differences were found as a result of this procedure. Finally, we evaluated the potential modifiers of time post-stroke (months), baseline UL capacity (i.e. baseline ARAT score), concordance (dominant side = affected side), and activities of daily living (ADL) status (i.e. requires assistance vs. independent) and their influence on change in UL performance over time for all six variables. Each modifier was added to Model 1 separately, and the effects of the modifier on the intercept and the slope (interaction between modifier and time) were evaluated (Models 4–7). Time post-stroke is widely assumed to be a predictor of UL performance, therefore we tested its influence on both initial baseline intercept and change over time (i.e. slope). Baseline ARAT scores were grand mean centered across all participants. Concordance and ADL status were tested because they have previously been shown to modify UL performance in daily life.44
Visual representations of accelerometer data were examined using density plots which display second-by-second data for the magnitude ratio (x-axis) and bilateral magnitude (y-axis) over the entire recording period, at every assessment time point. Example density plots have been previously published for healthy, neurologically intact adults (Figure 1, Bailey et. al; Supplemental Figure 1, Doman et. al27; Figure 3, Hayward et. al47). There are a few salient characteristics in healthy adults that should be considered when interpreting these density plots. First, in healthy adults, plots are symmetrical, indicating that both ULs are used similarly. The bottom portion is wide, and rounded, indicating that the majority of UL movements in a 24 hour period are low intensity. Additionally, the rounded edges or rims of the bowl-like structure represent movements when one limb is moving while the other is relatively still (e.g. holding a piece of paper with one hand while the other hand writes, holding a container with one hand and opening it with the other). The color bar represents the overall frequency of movement. Warmer colors (i.e. red and/or orange) represent more UL movement overall, and the small color bars on both sides of the density plot are specific to the frequency of unilateral non-paretic UL movement and unilateral paretic UL movement, respectively. While specific UL movements are highly variable across individuals,43 the salient characteristics of these graphs are highly consistent across community-dwelling, neurologically-intact nondisabled adults.9 Six examples of individual patients from the baseline data of this same cohort can be seen in Figure 2 of Bailey et al. NNR 2015. Examples of how individual density plots change over time in persons with stroke can be seen in Figure 4B of Hayward et al.47 and in Figure 1 in Doman et al.27
Figure 1.
UL performance over time for all six accelerometer variables by dose group. Values are group means ± SE for each assessment. Week 1 corresponds to the baseline assessment, and subsequent weeks correspond to the weekly assessment out to the immediate post-intervention assessment and follow-up assessment. Participants in the individualized maximum (IM) group were allowed to continue beyond the 8-week enrollment period until specific stopping criteria were met, observed here by the presence of additional data points.
Figure 2.
Density plots showing second-by-second data from three representative participants. Time points are from baseline (top), week four (middle), and post-intervention assessments (bottom). The y-axis (Bilateral magnitude) represents the intensity of movement, with higher values indicating larger, more intense movements. The x-axis (Magnitude ratio) represents the contribution of each limb to an activity, with 0 indicating equal UL contribution, negative values indicate more non-paretic UL movement and positive values indicate more paretic limb movement. The color scale shows overall frequency of UL movement, with warmer colors indicating more UL movement. The small bars on each side of the plot indicate non-paretic (negative) and paretic (positive) unilateral movement. Overall, participants had a moderate level of UL paresis at baseline (participant 1 = 38 points; participant 2 = 35 points, and participant 3 = 36 points). Participants 1 and 2 demonstrated 10-point and 18-point changes in ARAT score, respectively. Participant 3 increased 3 points on the ARAT. Regardless of UL capacity changes, there was no evidence of sustained changes in performance.
Results
Seventy-eight of the 85 participants in the trial had available data for this analysis. Of the seven excluded participants, four had accelerometer recording errors, two withdrew from the study prior to the intervention, and one did not consistently wear the accelerometers for > 6 hours at each assessment time point. Table 1 presents the demographic characteristics of the 78 participants and baseline values for the six accelerometer variables. The 6400 repetition group had low concordance (i.e. fewer people reporting their dominant side was the affected side).14 Individuals had mild to moderate levels of UL paresis at baseline and most of the participants were independent with basic activities of daily living.
Table 1.
Participant demographics and baseline accelerometer intercepts by treatment group
Total sample (n=78) |
3200 Group (n= 19) |
6400 Group (n=21) |
9600 Group (n=21) |
IMb (n=17) |
|
---|---|---|---|---|---|
Age (years) | 61.9 ± 10.5 | 59.4 ± 12.5 | 62.6 ± 8.5 | 60 ± 8.3 | 62.4 ± 13.1 |
Gender | 27 F, 51 M | 6 F, 13 M | 5 F, 16 M | 10 F, 11 M | 6 F, 11 M |
Race | 40 Caucasian 36 Af American 1 Asian 1 Multi-race |
10 Caucasian 9 Af American |
11 Caucasian 10 Af American |
10 Caucasian 9 Af American 1 Asian 1 Multi-race |
9 Caucasian 8 Af American |
Type of stroke | 56 ischemic 10 hemorrhagic 12 unknown |
14 ischemic | 16 ischemic | 15 ischemic | 11 ischemic |
Months post-stroke | 12 (5, 221) | 11 (6, 180) | 13 (6, 221) | 13 (5, 54) | 12 (6, 144) |
Affected side | 36 L, 42 R | 8 L, 11 R | 11 L, 10 R | 11 L, 10 R | 6 L, 11 R |
% Concordancea | 51% | 58% | 33% | 48% | 71% |
% Independent with ADL | 79% | 89% | 71% | 86% | 71% |
% Completed ≥ 32 treatment sessions | 81% | 89% | 76% | 71% | 88% |
Baseline ARATc score | 32.4 ± 11.2 | 34.1 ± 7.9 | 31.9 ± 13.1 | 32.1 ± 12.3 | 31.7 ± 11.2 |
Post-intervention ARAT scored | 36.9 ± 12.9 | 39.1 ± 8.7 | 36.4 ± 14.5 | 35.6 ± 15.4 | 36.5 ± 13.4 |
Post-intervention #2 ARAT scoree | 35.9 ± 13 | 38 ± 9.4 | 34.4 ± 14.1 | 34 ± 15.2 | 37.1 ± 13.8 |
Baseline Values | |||||
Use Ratio | 0.66 ± 0.23 | 0.67 ± 0.19 | 0.61 ± 0.19 | 0.67 ± 0.21 | 0.73 ± 0.33 |
Hours of Use | 4.73 ± 2.12 | 4.72 ± 2.42 | 4.09 ± 2.23 | 4.82 ± 1.45 | 5.4 ± 2.32 |
Magnitude Ratio | −3.04 ± 2.86 | −2.81 ± 2.68 | −3.52 ± 2.96 | −2.99 ± 2.81 | −2.78 ± 3.21 |
Bilateral Magnitude | 89.29 ± 27.45 | 86.91 ± 34.34 | 83.64 ± 25.45 | 94.76 ± 21.38 | 92.02 ± 28.78 |
Median acceleration | 15.53 ± 17.61 | 14.21 ± 15.38 | 11.67 ± 13.06 | 17.67 ± 19.59 | 19.38 ± 22.28 |
Acceleration variability | 45.56 ± 17.46 | 45.22 ± 15.86 | 41.04 ± 13.07 | 49.61 ± 20.49 | 46.27 ± 19.91 |
Values reported as means ± SD or median (range) as determined by distribution of data
= Concordance = dominant side is paretic side; values here indicate the percentage of the sample who identified their dominant upper limb as the paretic upper limb
= Individualized Maximum
= Action Research Arm Test; scores range from 0–57 points with higher scores indicating more normal movement
= Assessment completed immediately after the final treatment session
= Assessment completed 2-months after conclusion of intervention
Overall, there was no change in UL performance across all 78 participants on any of the six accelerometer variables. The final model for 5 of the 6 variables included the linear effect of time which produced better fitting models for the use ratio (χ2= 15.08, df =3, p = 0.002), hours of paretic limb use (χ2= 15.45, df=3, p = 0.001), the magnitude ratio (χ2= 15.08, df =3, p = 0.03), median acceleration (χ2= 10.84, df = 3, p = 0.01), and the acceleration variability (χ2= 12.24, df=3, p = 0.007). For the sixth variable, the bilateral magnitude, the addition of a linear effect of time was not significant (χ2= 2.76, df =3, p = 0.43), indicating that time did not increase the predictive ability of the model and was not a significant predictor of change. Time was still included in the final model to acquire a slope value for the bilateral magnitude and also test potential modifiers of the slope. Rates of change, quantified as model slopes are reported in the top row of Table 2, in units of change per week. The slopes for each accelerometer variable were not significantly different from zero.
Table 2.
Slopes over time for the entire sample, by group, ARAT change score, and potential modifiers; values are means ± SE.
Use Ratio | Hours of Use | Magnitude Ratio | Bilateral Magnitude | Median Acceleration | Acceleration Variability | |
---|---|---|---|---|---|---|
Entire samplea | −0.0005 ± 0.0009 | −0.027 ± 0.01 | −0.023 ± 0.013 | −0.15 ± 0.09 | −0.03 ± 0.06 | −0.04 ± 0.05 |
Change in ARATb | ||||||
ARAT change ≥ 6 pts | 0.0013 ± 0.002 | −0.006 ± 0.02 | −0.008 ± 0.03 | −0.01 ± 0.18 | 0.032 ± 0.12 | 0.061 ± 0.1 |
Groupc | ||||||
3200 6400 9600 IM |
0.0004 ± 0.002 0.0006 ± 0.002 −0.0028 ± 0.002 −0.0004 ± 0.002 |
−0.015 ± 0.02 −0.037 ± 0.03 −0.022 ± 0.03 −0.034 ± 0.03 |
−0.005 ± 0.024 −0.005 ± 0.03 −0.078 ± 0.03 −0.008 ± 0.03 |
−0.18 ± 0.19 −0.30 ± 0.27 0.005 ± 0.27 −0.12 ± 0.26 |
−0.13 ± 0.12 0.02 ± 0.17 0.01 ± 0.17 0 ± 0.17 |
0.07 ± 0.1 −0.04 ± 0.14 −0.28 ± 0.15 0.08 ± 0.14 |
Modifiers | ||||||
Chronicity (months)d | 0.000008 ± 0.00002 | 0.0001 ± 0.0003 | 0.0004 ± 0.0003 | −0.002 ± 0.002 | 0.0001 ± 0.002 | −0.0006 ± 0.001 |
Baseline ARATe | 0.0001 ± 0.0001 | 0.0003 ± 0.001 | 0.00003 ± 0.001 | −0.006 ± 0.01 | −0.002 ± 0.01 | 0.005 ± 0.005 |
Concordancef | −0.0017 ± 0.002 | −0.025 ± 0.02 | −0.03 ± 0.03 | −0.118 ± 0.19 | −0.081 ± 0.12 | −0.08 ± 0.12 |
ADL independenceg | −0.0013 ± 0.002 | −0.014 ± 0.026 | −0.032 ± 0.03 | −0.173 ± 0.22 | −0.055 ± 0.14 | −0.048 ± 0.13 |
Slope values reported as rate of change per week over the duration of the study.
= Results from Model 1, slope is per week change over time
= Results from Model 2, slope is interaction of time × ARAT change
= Results from Model 3, slope is interaction of time × dose group
= Results from Model 4, slope is the interaction of time × chronicity; Values reported for a 1-unit change in chronicity (i.e. 1 month)
= Results from Model 5, slope is interaction of time × baseline ARAT score; Values reported for 1-point increase in baseline ARAT score (e.g. for every one point increase in baseline ARAT score, the participant’s slope increased by 0.0001 on the use ratio)
= Results from Model 6, slope is interaction of time × concordance; Values reported for individuals who indicated their dominant UL was the paretic UL
= Results from Model 7, slope is interaction of time × ADL independence; Values reported for individuals who were independent with basic activities of daily living (ADL; e.g. bathing, dressing, toileting)
To test the possibility that some changes in performance were masked in the entire sample, we grouped participants based on changes in UL capacity. Seventy-five participants had available data for this portion of the analysis. The three excluded cases withdrew from the study prior to the first assessment after treatment was initiated, and therefore did not have an ARAT change score. Individuals who had larger changes in UL capacity (ARAT change score ≥ 6 points, n= 36) started better (higher baseline intercepts) for the use ratio (p < 0.001), hours of paretic limb use (p = .007), magnitude ratio (p < .001), median acceleration (p < .001), and acceleration variability (p < .001), as would be expected (values not shown). Despite the better starting points, ARAT change did not influence the rates of change between the two groups (p values > .05) and all slopes were still not significantly different from zero (Table 2, second row of data).
The addition of treatment group, and the interaction between treatment group and time also did not change the results. Figure 1 illustrates the lack of group effect on all six variables. No significant intercept and slope differences existed between groups and results from general linear hypothesis tests indicated none of the group slopes were significantly different from zero for any of the six variables (all p-values > 0.05). Results of the group effects are reported in the middle rows of Table 2. Finally, we tested potential modifiers that have previously been shown to influence UL performance.44 Time post-stroke, baseline UL capacity, concordance, and ADL status influenced the starting points (i.e. better intercepts) as expected, but did not influence change over time (model slopes calculated from time × modifier interaction, all p-values > 0.05, slope values reported at the bottom of Table 2). Specifically, time post-stroke significantly influenced the use ratio (p =0.04) and magnitude ratio (p= 0.03) intercepts, respectively, but was not a significant modifier of the remaining four intercept values. Baseline UL capacity and ADL status significantly influenced the intercepts of all six variables, and concordance significantly influenced the use ratio (p < .001), magnitude ratio (p= 0.01), median acceleration (p=0.04), and acceleration variability (p = 0.04) intercepts.
Figure 2 provides representative individual examples of the lack of change in UL daily performance. Despite substantial changes in UL capacity, participants 1 (10 point ARAT change) and 2 (18 point ARAT change) show no change in performance from baseline to post-intervention. The pictures from these two participants are not distinctly different from participant 3 (3 point ARAT change). Compared to healthy, neurologically intact adults, the density plots in Figure 2 are all asymmetrical, with mostly negative magnitude ratio values that indicate increased non-paretic UL activity. An absence of warmer colors indicate less movement overall, with no change in the frequency of movement from baseline to post-intervention. There is a noticeable peak in the center of participant 1’s density plot at week four (i.e. a higher bilateral magnitude value), indicating more intense movements, but this was not sustained by the post-intervention assessment. While some participants showed small fluctuations such as this one, no subjects showed sustained changes over time.
Discussion
We evaluated changes in UL performance in daily life resulting from a task-specific intervention using a quantitative measure of performance. None of the 78 participants increased their UL performance in daily life, as measured by the six accelerometer variables. Dividing the sample into groups based on changes in UL capacity or dividing the sample into groups based on amount of motor practice failed to produce changes in performance over time. Additionally, despite having various effects on the initial intercepts, none of the modifiers influenced change over time (slopes). Thus, UL task-specific training, designed to improve UL capacity in the clinic may be unable to improve UL performance in daily life. This is contrary to the long-standing clinical assumptions that improving UL capacity directly translates to improved UL performance in daily life.
A key reason people are referred to motor rehabilitation services is to improve UL performance in daily life. With the cost of stroke expected to exceed 2.2 trillion dollars by 2050,48 it is striking that not one person changed UL performance after this carefully delivered intervention.14 These results are consistent with a few other studies that have begun to identify a discrepancy between changes observed in the rehabilitation clinic (i.e. capacity) and a failure to increase UL performance in daily life both for adults26,49 and children50 (but see51). When changes seen in the clinic do not carry over to life at home, then it is time to reconsider what is being delivered in the clinic and how it might need to be changed.
One argument against these striking results is that perhaps the accelerometers failed to capture change that really occurred. There are numerous studies of UL interventions post-stroke that have reported a positive increase in UL performance in daily life when measured via self-report10,17–19 and for individuals earlier after stroke, the MAL and use ratio are correlated.38 Self-perception of changes in UL performance is a valuable component of the rehabilitation process, but perhaps not the whole story. We cannot completely rule out the possibility that other variables from the accelerometers would show changes. In deciding on these 6 variables however, we did test a number of others that were not useful (e.g. highly variable across neurologically intact population).39,43 Our accelerometers were on the wrists, so we also cannot completely rule out the possibility that small, dexterous movements of the fingers improved and we did not capture this. This second possibility is also unlikely, since pilot testing with accelerometers picked up most of the hand/finger movements,43,52 particularly the less efficient and uncoordinated movements of the paretic hand and fingers post-stroke.
There are several possible reasons for these striking results. First, this study included those with chronic (≥ 6 months) stroke when habits have likely already formed. Perhaps changes in UL performance would be observed if task-specific training was delivered earlier after stroke. Second, changes in capacity may be insufficient or not enough to improve UL performance in daily life. There may be a specific threshold for UL capacity that must be exceeded to drive changes in UL performance.53,54 Third, UL performance is not solely a function of UL capacity but dependent on other factors such as motivation, health behaviors, and environmental supports. It is likely that these results are a combination of all three proposed reasons. Future studies could examine the timing of intervention to improve UL performance post-stroke, as some pilot studies have reported changes in both UL capacity and UL performance earlier after stroke.27,55 Additional studies could explore other potential factors related to UL performance such as health behaviors and motivation, and explicitly test interventions that target these factors. Indeed, the 36 participants who demonstrated improvements in capacity (i.e. ARAT change ≥ 6 points) may be ideal candidates for interventions targeting UL performance in daily life, given their ability to change at this stage of recovery.
Several limitations influence the interpretation of these data. First, wearing sensors on the upper limbs could potentially cause people to do more with their ULs in daily life. Sensor data were collected weekly with more than 8 assessments. Thus the novelty of wearing the devices likely wore off early. If anything, we may have overestimated UL performance in daily life within the first few assessments. Second, the sensor-based methodology quantifies movement but does not quantify specific activities or movement parameters (e.g. speed, efficiency, accuracy). It is possible that some participants made small improvements in these parameters that went unmeasured. These changes, however, were not sufficient to change the involvement of the paretic limb either in total duration (use ratio) or on a second-by-second basis (magnitude ratio).
Conclusions
We found no evidence of improvement in UL performance in daily life in 78 people with long-standing paresis post-stroke who completed an 8-week individualized, intensive, progressive, task-specific intervention. Neither changes in UL capacity nor the overall dose (i.e. amount) of movement practice influenced changes in UL performance. These results expose an emerging problem in stroke rehabilitation. Rehabilitation services, and the providing clinicians, may be changing what people can do while they are in the rehabilitation clinic, but these benefits do not carry over to improved UL performance at home, when measured with wrist-worn accelerometers. If a primary goal of rehabilitation is to improve performance in daily life for individuals post-stroke, then it is imperative that future research investigate this emerging issue.
Acknowledgments
Funding for this project was provided by NIH R01HD068290. The authors would like to acknowledge Brittany Hill, BA and Keith Lohse, PhD, for assisting with data collection (BH) and preliminary data analysis (KL).
References
- 1.Dobkin BH. Clinical practice. Rehabilitation after stroke. N Engl J Med. 2005 Apr 21;352(16):1677–1684. doi: 10.1056/NEJMcp043511. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Barker RN, Brauer SG. Upper limb recovery after stroke: the stroke survivors' perspective. Disability and rehabilitation. 2005 Oct 30;27(20):1213–1223. doi: 10.1080/09638280500075717. [DOI] [PubMed] [Google Scholar]
- 3.Barker RN, Gill TJ, Brauer SG. Factors contributing to upper limb recovery after stroke: a survey of stroke survivors in Queensland Australia. Disability and rehabilitation. 2007 Jul 15;29(13):981–989. doi: 10.1080/09638280500243570. [DOI] [PubMed] [Google Scholar]
- 4.Kwakkel G, Kollen BJ, van der Grond J, Prevo AJ. Probability of regaining dexterity in the flaccid upper limb: impact of severity of paresis and time since onset in acute stroke. Stroke; a journal of cerebral circulation. 2003 Sep;34(9):2181–2186. doi: 10.1161/01.STR.0000087172.16305.CD. [DOI] [PubMed] [Google Scholar]
- 5.Hartman-Maeir A, Soroker N, Ring H, Avni N, Katz N. Activities, participation and satisfaction one-year post stroke. Disability and rehabilitation. 2007 Apr 15;29(7):559–566. doi: 10.1080/09638280600924996. [DOI] [PubMed] [Google Scholar]
- 6.Bailey RR, Lang CE. Upper-limb activity in adults: referent values using accelerometry. J Rehabil Res Dev. 2013;50(9):1213–1222. doi: 10.1682/JRRD.2012.12.0222. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.International classification of functioning, disability and health. World Health Organization; 2001. [Google Scholar]
- 8.Young NL, Williams JI, Yoshida KK, Bombardier C, Wright JG. The context of measuring disability: does it matter whether capability or performance is measured? Journal of clinical epidemiology. 1996 Oct;49(10):1097–1101. doi: 10.1016/0895-4356(96)00214-4. [DOI] [PubMed] [Google Scholar]
- 9.Bailey RR, Klaesner JW, Lang CE. Quantifying real-world upper limb activity in nondisabled adults and adults with chronic stroke. Neurorehabilitation and neural repair. 2015 doi: 10.1177/1545968315583720. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Wolf SL, Winstein CJ, Miller JP, et al. Effect of constraint-induced movement therapy on upper extremity function 3 to 9 months after stroke: the EXCITE randomized clinical trial. Jama. 2006 Nov 1;296(17):2095–2104. doi: 10.1001/jama.296.17.2095. [DOI] [PubMed] [Google Scholar]
- 11.Birkenmeier RL, Prager EM, Lang CE. Translating animal doses of task-specific training to people with chronic stroke in 1-hour therapy sessions: a proof-of-concept study. Neurorehabilitation and neural repair. 2010 Sep;24(7):620–635. doi: 10.1177/1545968310361957. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Dromerick AW, Lang CE, Birkenmeier RL, et al. Very Early Constraint-Induced Movement during Stroke Rehabilitation (VECTORS): A single-center RCT. Neurology. 2009 Jul 21;73(3):195–201. doi: 10.1212/WNL.0b013e3181ab2b27. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Harris JE, Eng JJ, Miller WC, Dawson AS. A self-administered Graded Repetitive Arm Supplementary Program (GRASP) improves arm function during inpatient stroke rehabilitation: a multi-site randomized controlled trial. Stroke; a journal of cerebral circulation. 2009 Jun;40(6):2123–2128. doi: 10.1161/STROKEAHA.108.544585. [DOI] [PubMed] [Google Scholar]
- 14.Lang CE, Strube MJ, Bland MD, et al. Dose-response of task-specific upper limb training in people at least 6 months post stroke: A Phase II, single-blind, randomized, controlled trial. Annals of Neurology. doi: 10.1002/ana.24734. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Lyle RC. A performance test for assessment of upper limb function in physical rehabilitation treatment and research. Int J Rehabil Res. 1981;4(4):483–492. doi: 10.1097/00004356-198112000-00001. [DOI] [PubMed] [Google Scholar]
- 16.Wolf SL, Catlin PA, Ellis M, Archer AL, Morgan B, Piacentino A. Assessing Wolf motor function test as outcome measure for research in patients after stroke. Stroke; a journal of cerebral circulation. 2001 Jul;32(7):1635–1639. doi: 10.1161/01.str.32.7.1635. [DOI] [PubMed] [Google Scholar]
- 17.van Delden AL, Peper CL, Beek PJ, Kwakkel G. Match and mismatch between objective and subjective improvements in upper limb function after stroke. Disability and rehabilitation. 2013;35(23):1961–1967. doi: 10.3109/09638288.2013.768303. [DOI] [PubMed] [Google Scholar]
- 18.Page SJ, Levine PG, Basobas BA. “Reps” Aren’t Enough: Augmenting Functional Electrical Stimulation With Behavioral Supports Significantly Reduces Impairment in Moderately Impaired Stroke. Arch Phys Med Rehabil. 2016 May;97(5):747–752. doi: 10.1016/j.apmr.2016.01.004. [DOI] [PubMed] [Google Scholar]
- 19.Fleming MK, Newham DJ, Roberts-Lewis SF, Sorinola IO. Self-perceived utilization of the paretic arm in chronic stroke requires high upper limb functional ability. Arch Phys Med Rehabil. 2014 May;95(5):918–924. doi: 10.1016/j.apmr.2014.01.009. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.Bradburn NM, Rips LJ, Shevell SK. Answering autobiographical questions: the impact of memory and inference on surveys. Science. 1987 Apr 10;236(4798):157–161. doi: 10.1126/science.3563494. [DOI] [PubMed] [Google Scholar]
- 21.Tatemichi TK, Desmond DW, Stern Y, Paik M, Sano M, Bagiella E. Cognitive impairment after stroke: frequency, patterns, and relationship to functional abilities. Journal of neurology, neurosurgery, and psychiatry. 1994 Feb;57(2):202–207. doi: 10.1136/jnnp.57.2.202. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.Adams SA, Matthews CE, Ebbeling CB, et al. The effect of social desirability and social approval on self-reports of physical activity. Am J Epidemiol. 2005 Feb 15;161(4):389–398. doi: 10.1093/aje/kwi054. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23.Douiri A, Rudd AG, Wolfe CD. Prevalence of poststroke cognitive impairment: South London Stroke Register 1995–2010. Stroke; a journal of cerebral circulation. 2013 Jan;44(1):138–145. doi: 10.1161/STROKEAHA.112.670844. [DOI] [PubMed] [Google Scholar]
- 24.Sun JH, Tan L, Yu JT. Post-stroke cognitive impairment: epidemiology, mechanisms and management. Annals of translational medicine. 2014 Aug;2(8):80. doi: 10.3978/j.issn.2305-5839.2014.08.05. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 25.Rand D, Eng JJ. Predicting daily use of the affected upper extremity 1 year after stroke. Journal of stroke and cerebrovascular diseases: the official journal of National Stroke Association. 2015 Feb;24(2):274–283. doi: 10.1016/j.jstrokecerebrovasdis.2014.07.039. [DOI] [PubMed] [Google Scholar]
- 26.Lemmens RJ, Timmermans AA, Janssen-Potten YJ, et al. Accelerometry measuring the outcome of robot-supported upper limb training in chronic stroke: a randomized controlled trial. PloS one. 2014;9(5):e96414. doi: 10.1371/journal.pone.0096414. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 27.Doman CA, Waddell KJ, Bailey RR, Moore JL, Lang CE. Changes in Upper-Extremity Functional Capacity and Daily Performance During Outpatient Occupational Therapy for People With Stroke. American Journal of Occupational Therapy. 2016;70(3) doi: 10.5014/ajot.2016.020891. 7003290040p7003290041-7003290040p7003290011. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 28.Brott T, Adams HP, Jr, Olinger CP, et al. Measurements of acute cerebral infarction: a clinical examination scale. Stroke; a journal of cerebral circulation. 1989 Jul;20(7):864–870. doi: 10.1161/01.str.20.7.864. [DOI] [PubMed] [Google Scholar]
- 29.Lang CE, Bland MD, Bailey RR, Schaefer SY, Birkenmeier RL. Assessment of upper extremity impairment, function, and activity after stroke: foundations for clinical decision making. J Hand Ther. 2013 Apr-Jun;26(2):104–115. doi: 10.1016/j.jht.2012.06.005. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 30.Yozbatiran N, Der-Yeghiaian L, Cramer SC. A standardized approach to performing the action research arm test. Neurorehabilitation and neural repair. 2008 Jan-Feb;22(1):78–90. doi: 10.1177/1545968307305353. [DOI] [PubMed] [Google Scholar]
- 31.Kwakkel G, Veerbeek JM, van Wegen EE, Wolf SL. Constraint-induced movement therapy after stroke. Lancet Neurol. 2015 Feb;14(2):224–234. doi: 10.1016/S1474-4422(14)70160-7. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 32.Hayward KS, Brauer SG. Dose of arm activity training during acute and subacute rehabilitation post stroke: a systematic review of the literature. Clinical rehabilitation. 2015 Dec;29(12):1234–1243. doi: 10.1177/0269215514565395. [DOI] [PubMed] [Google Scholar]
- 33.Host HH, Lang CE, Hildebrand MW, et al. Patient Active Time During Therapy Sessions in Postacute Rehabilitation: Development and Validation of a New Measure. Phys Occup Ther Geriatr. 2014 Jun;32(2):169–178. doi: 10.3109/02703181.2014.915282. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 34.Platz T, Pinkowski C, van Wijck F, Kim IH, di Bella P, Johnson G. Reliability and validity of arm function assessment with standardized guidelines for the Fugl-Meyer Test, Action Research Arm Test and Box and Block Test: a multicentre study. Clinical rehabilitation. 2005 Jun;19(4):404–411. doi: 10.1191/0269215505cr832oa. [DOI] [PubMed] [Google Scholar]
- 35.van der Lee JH, De Groot V, Beckerman H, Wagenaar RC, Lankhorst GJ, Bouter LM. The intra- and interrater reliability of the action research arm test: a practical test of upper extremity function in patients with stroke. Archives of physical medicine and rehabilitation. 2001 Jan;82(1):14–19. doi: 10.1053/apmr.2001.18668. [DOI] [PubMed] [Google Scholar]
- 36.Lin JH, Hsu MJ, Sheu CF, et al. Psychometric comparisons of 4 measures for assessing upper-extremity function in people with stroke. Physical therapy. 2009 Aug;89(8):840–850. doi: 10.2522/ptj.20080285. [DOI] [PubMed] [Google Scholar]
- 37.Uswatte G, Foo WL, Olmstead H, Lopez K, Holand A, Simms LB. Ambulatory monitoring of arm movement using accelerometry: an objective measure of upper-extremity rehabilitation in persons with chronic stroke. Archives of physical medicine and rehabilitation. 2005 Jul;86(7):1498–1501. doi: 10.1016/j.apmr.2005.01.010. [DOI] [PubMed] [Google Scholar]
- 38.Uswatte G, Giuliani C, Winstein C, Zeringue A, Hobbs L, Wolf SL. Validity of accelerometry for monitoring real-world arm activity in patients with subacute stroke: evidence from the extremity constraint-induced therapy evaluation trial. Archives of physical medicine and rehabilitation. 2006 Oct;87(10):1340–1345. doi: 10.1016/j.apmr.2006.06.006. [DOI] [PubMed] [Google Scholar]
- 39.Urbin MA, Bailey RR, Lang CE. Validity of body-worn sensor acceleration metrics to index upper extremity function in hemiparetic stroke. Journal of neurologic physical therapy: JNPT. 2015 Apr;39(2):111–118. doi: 10.1097/NPT.0000000000000085. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 40.Lang CE, Wagner JM, Edwards DF, Dromerick AW. Upper Extremity Use in People with Hemiparesis in the First Few Weeks After Stroke. Journal of neurologic physical therapy: JNPT. 2007 Jun;31(2):56–63. doi: 10.1097/NPT.0b013e31806748bd. [DOI] [PubMed] [Google Scholar]
- 41.Michielsen ME, Selles RW, Stam HJ, Ribbers GM, Bussmann JB. Quantifying nonuse in chronic stroke patients: a study into paretic, nonparetic, and bimanual upper-limb use in daily life. Arch Phys Med Rehabil. 2012 Nov;93(11):1975–1981. doi: 10.1016/j.apmr.2012.03.016. [DOI] [PubMed] [Google Scholar]
- 42.Bailey RR. Assessment of Real-World Upper Limb Activity in Adults with Chronic Stroke [Doctoral]: Interdisciplinary Program in Movement Science. Washington University; St. Louis, MO: 2015. [Google Scholar]
- 43.Bailey RR, Klaesner JW, Lang CE. An accelerometry-based methodology for assessment of real-world bilateral upper extremity activity. PloS one. 2014;9(7):e103135. doi: 10.1371/journal.pone.0103135. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 44.Bailey RR, Birkenmeier RL, Lang CE. Real-world affected upper limb activity in chronic stroke: an examination of potential modifying factors. Topics in stroke rehabilitation. 2015 Feb;22(1):26–33. doi: 10.1179/1074935714Z.0000000040. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 45.Urbin MA, Waddell KJ, Lang CE. Acceleration metrics are responsive to change in upper extremity function of stroke survivors. Arch Phys Med Rehabil. 2015 May;96(5):854–861. doi: 10.1016/j.apmr.2014.11.018. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 46.Long JD. Longitudinal Data Analysis for the Behavioral Sciences using R. Vol. 1. California: SAGE Publications, Inc; 2012. [Google Scholar]
- 47.Hayward KS, Eng JJ, Boyd LA, Lakhani B, Bernhardt J, Lang CE. Exploring the Role of Accelerometers in the Measurement of Real World Upper-Limb Use After Stroke. Brain Impairment. 2016;17:16–33. Special Issue 01. [Google Scholar]
- 48.Brown DL, Boden-Albala B, Langa KM, et al. Projected costs of ischemic stroke in the United States. Neurology. 2006 Oct 24;67(8):1390–1395. doi: 10.1212/01.wnl.0000237024.16438.20. [DOI] [PubMed] [Google Scholar]
- 49.Rand D, Eng JJ. Disparity between functional recovery and daily use of the upper and lower extremities during subacute stroke rehabilitation. Neurorehabilitation and neural repair. 2012 Jan;26(1):76–84. doi: 10.1177/1545968311408918. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 50.Mitchell LE, Ziviani J, Boyd RN. A randomized controlled trial of web-based training to increase activity in children with cerebral palsy. Developmental medicine and child neurology. 2016 Feb 15; doi: 10.1111/dmcn.13065. [DOI] [PubMed] [Google Scholar]
- 51.Shim S, Kim H, Jung J. Comparison of upper extremity motor recovery of stroke patients with actual physical activity in their daily lives measured with accelerometers. Journal of physical therapy science. 2014 Jul;26(7):1009–1011. doi: 10.1589/jpts.26.1009. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 52.Rowe JB, Friedman N, Chan V, Cramer SC, Bachman M, Reinkensmeyer DJ. The variable relationship between arm and hand use: a rationale for using finger magnetometry to complement wrist accelerometry when measuring daily use of the upper extremity. Conference proceedings: … Annual International Conference of the IEEE Engineering in Medicine and Biology Society. IEEE Engineering in Medicine and Biology Society. Annual Conference. 2014;2014:4087–4090. doi: 10.1109/EMBC.2014.6944522. [DOI] [PubMed] [Google Scholar]
- 53.Schweighofer N, Han CE, Wolf SL, Arbib MA, Winstein CJ. A functional threshold for long-term use of hand and arm function can be determined: predictions from a computational model and supporting data from the Extremity Constraint-Induced Therapy Evaluation (EXCITE) Trial. Physical therapy. 2009 Dec;89(12):1327–1336. doi: 10.2522/ptj.20080402. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 54.Hidaka Y, Han CE, Wolf SL, Winstein CJ, Schweighofer N. Use it and improve it or lose it: interactions between arm function and use in humans post-stroke. PLoS computational biology. 2012 Feb;8(2):e1002343. doi: 10.1371/journal.pcbi.1002343. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 55.Waddell KJ, Birkenmeier RL, Moore JL, Hornby TG, Lang CE. Feasibility of high-repetition, task-specific training for individuals with upper-extremity paresis. The American journal of occupational therapy: official publication of the American Occupational Therapy Association. 2014 Jul-Aug;68(4):444–453. doi: 10.5014/ajot.2014.011619. [DOI] [PMC free article] [PubMed] [Google Scholar]