Minimal Detectable Change of the Actual Amount of Use Test and the Motor Activity Log: The EXCITE Trial

Abstract

Background

Efficacy of task-oriented training can be reliably trusted only when the inherent measurement variability is determined. The Actual Amount of Use Test (AAUT) and the Motor Activity Log (MAL) have been used together as measures of spontaneous arm use after an intervention; however, the minimal detectable change (MDC) of AAUT and MAL has not been addressed.

Objective

To compare the MDC₉₀ of the AAUT and the MAL in the context of a randomized controlled trial of a neurorehabilitation intervention, the Extremity Constraint-Induced Therapy Evaluation trial.

Methods

A preplanned secondary analysis was conducted using pre–post test data from the control group. Estimated MDC₉₀ were normalized to the maximum value of the scale of the AAUT and the MAL for each subscale: amount of use (AAUTa, MALa) and quality of movement (AAUTq, MALq).

Results

The MDC₉₀ of the AAUTq and the MALq were 14.4% and 15.4%, respectively. However, the MDC₉₀ required greater change for the AAUTa (24.2%) than the MALa (16.8%). The training-induced spontaneous arm use exceeded the MDC₉₀ for the MAL but fell below that for the AAUT immediately after the intervention and at 1-year follow-up visit.

Conclusions

The greater variability and insensitivity to treatment effect for the AAUTa is likely because of the low resolution of its scoring system. As such, there is a considerable need to develop valid and reliable tools that capture purposeful arm use outside the laboratory, perhaps through leveraging new sensing technologies with objective activity monitoring.

Introduction

During the past decade, clinical meaningfulness has increasingly gained in importance for clinicians and consumers, especially in the field of stroke rehabilitation.^1–5 However, the determination and application of the concept of clinical meaningfulness remains vague. To determine and apply measures of clinical meaningfulness to outcomes for which people have interest, both sensitivity of measurements and efficacy of interventions to promote meaningful outcomes are very important.^2,6 For upper limb recovery after stroke, the most significant outcome is the ability to voluntarily use the paretic arm and hand.^7,8 To our knowledge, the accelerometer,^9,10 the Motor Activity Log (MAL),^11,12 and the Actual Amount of Use Test (AAUT)¹³ are the only instruments in current use to capture changes in spontaneous arm use in this population. To regain spontaneous arm use after stroke, Taub et al¹¹ developed a signature intervention, constraint-induced movement therapy (CIMT), based on evidence from animal models to overcome paretic arm nonuse for individuals poststroke. Although evidence shows the efficacy of CIMT,^14,15 an objective scientific method of determining a meaningful change in spontaneous arm use has not been established.¹ Before that, there is a need to understand the inherent noise level of the measure used to capture spontaneous arm use. Determining the “noise” level through estimation of the minimal detectable change can be considered as a calibration process. Once the systematic error associated with the measure is known, then the detected training-induced change can be trusted.

The minimal detectable change (MDC) is used to determine the threshold for whether a pre–post change is true from a measurement perspective.^2,6,16 This is fundamentally different from that used to determine if the change is “meaningful” (ie, minimal clinically important difference) to the patient, family, clinicians, and others.^3,4 MDC is defined as the smallest amount of change that is detectable and not due to inherent variation or noise in the measure itself.^2,5,17–19 In general, to compute MDC, data from 2 repeated tests are used where there is no intervening manipulation. The 90% and 95% confidence level of a reliable difference (MDC₉₀ and MDC₉₅) are 2 common MDC metrics;²⁰ however, MDC₉₀ is regarded as sufficient for decisions pertaining to the efficacy of clinical interventions.¹⁶ Here, a change in the outcome of interest exceeding the MDC indicates the intervention program is effective.^2,6 In circumstances where the consequences of a potentially wrong decision are more severe (eg, the possibility of mortality from surgery), MDC₉₅ would be a better choice given the higher confidence level with respect to test–retest reliability.¹⁶

The first large-scale randomized controlled trial to evaluate the efficacy of CIMT, the Extremity Constraint-Induced Therapy Evaluation (EXCITE) trial, used the AAUT (secondary outcome measure) and the MAL (outcome measure) as convergent measures to detect improvement of spontaneous arm use after CIMT.²¹ The primary outcomes and their results were presented elsewhere¹⁴. The AAUT is a covert performance-based assessment with 17 daily activities used to observe actual arm behavior in real-life situations.¹³ Spontaneous arm use was filmed covertly in the laboratory using real-world scenarios (eg, grabbing and pulling a chair out from under the table, opening a file folder) staged by an investigator and scored by trained and standardized raters masked to group assignment. Contrary to the AAUT, the MAL is a semistructured interview questionnaire used to retrieve participants’ recall of arm behavior during the past few days. The questionnaire consists of questions regarding 30 daily activities (eg, turning on a light switch, opening a drawer).¹¹ Spontaneous arm use was evaluated based on participants’ recall and compared with the remembered status before their stroke (eg, “50% of use for that task before my stroke”). Although there was a significant increase in the amount of paretic arm use measured with the AAUT²² and with the MAL¹⁴ after 2 weeks of CIMT, there has been little study about whether these increases reflected a true change in spontaneous arm use.^1,23,24

The purpose of this study was to determine the MDC of the AAUT and the MAL in the context of the EXCITE trial. We had 3 aims: (a) to determine the MDC for the AAUT and the MAL, (b) to compare the MDC for the AAUT and MAL, and (c) to determine if the CIMT-induced changes in spontaneous arm use exceed the estimated MDC for the AAUT and the MAL.

Methods

This secondary data analysis used a subset of data from the EXCITE trial where 222 participants 3 to 9 months after stroke with mild to moderate impairment were randomized to 1 of 2 treatment arms. Inclusion criteria for the participants were as follows: (a) >10° active wrist extension, (b) >10° active thumb abduction and extension, and (c) at least 2 additional active digit extension movements. On enrollment, participants were assigned randomly to either the CIMT group or the control group (not receiving CIMT) for the first year. The AAUT and the MAL were used as proxies of the outcome of interest, real-life spontaneous arm use. There are 2 subscales for both the AAUT and the MAL: amount of use and quality of movement. For clarity, we abbreviated the subscales as AAUTa, AAUTq, MALa, and MALq for the AAUT amount of use, AAUT quality of movement, MAL amount of use, and MAL quality of movement, respectively (Table 1).

Table 1.

Abbreviations of 2 Subscales for the Actual Amount of Use Test and the Motor Activity Log

	Actual Amount of Use Test	Motor Activity Log
Amount of use	AAUTa	MALa
Quality of movement	AAUTq	MALq

For the AAUTa subscale, each testing item was scored either “0” (did not attempt to use the paretic arm) or “1” (attempted to use the paretic arm). The AAUTa was calculated by dividing the number of items scored as “1” by the total number of items scored and expressed as a percentage. The AAUTq subscale was scored from “0” to “5” with 0 being that the participant did not attempt to use the paretic arm for that activity and 5 for normal performance of the activity. A “1” on the AAUTq rated movement performance as very poor, a “2” poor, “3” fair, and “4” nearly normal. The AAUTq was calculated by dividing the total score by the number of items scored and expressed as an average. For the MALa subscale, each item (eg, used the weaker arm for that activity) was scored from “0” to “5.” Zero indicated that the participant did not use the paretic arm for that activity. “1” occasionally tried to use their weaker arm, “2” rarely used the weaker, “3” used the weaker arm for that activity half as much as before the stroke, “4” 75% as much as before the stroke, and “5” indicated the same amount of use for that activity as before the stroke. The rating score of the MALq was the same as that of the AAUTq. For the purposes of comparison, the average ratings of the AAUTq, MALa, and MALq were normalized to the maximal value of the scale and expressed as a percentage, rounded to the first decimal place.

Minimal detectable change for each of the measures was calculated from the pre-post test data of the control group. As mentioned earlier, the MDC₉₀ was determined to be the appropriate measure of reliability for the intervention studies.¹⁶ It was chosen to determine the measurement variability and then used to verify whether the magnitude of the CIMT-induced improvement was caused by the intervention or by random noise. To calculate MDC₉₀, we first computed the intraclass correlation coefficient—ICC(3,1)—and the standard error of measurement (SEM). Then, the MDC₉₀ was obtained using the following formula, where SD_baseline is the standard deviation at baseline.^18,20

$$\begin{array}{l}\text{SEM}={\text{SD}}_{\text{baseline}}\sqrt{1-\text{ICC}}\\ {\text{MDC}}_{90}=1.64\times \text{SEM}\times \sqrt{2}a\end{array}$$

To investigate the effect of CIMT, the first step was to determine the significance of the CIMT-induced changes, and the second step was to compare these changes to the calculated MDC values. For the first step, an independent-samples t test was used to test the CIMT-induced changes between groups. For the second step, the amount of CIMT-induced change was compared with the MDC values. Statistical analyses were performed using SPSS software package (version 15).

Results

Pre–post test data from 116 participants in the control group were used for this secondary analysis and the data from 106 participants in the CIMT group were used to test the effects of CIMT (Table 2). The ICC, SEM, and MDC₉₀ of the AAUT and the MAL are shown in Table 3. In general, the MDC₉₀ values of both measures were greater in the amount of use than the quality of movement subscale. The MDC₉₀ in the AAUTa and the MALa were 24.2% and 16.8%, respectively. As for the quality of movement subscale, the MDC₉₀ in the AAUTq and the MALq were similar (14.4% and 15.3%, respectively). Whereas the MDC₉₀ of the AAUTq was very close to the MALq, the MDC₉₀ of the AAUTa was 30.6% greater than MALa. In addition, the 95% confidence interval for the MDC AAUTa (24.2%) did not overlap with that for the MDC AAUTq (14.4%), the MDC MALa (16.8%), or the MDC MALq (15.3%). This finding reveals that the difference in measurement variability between the AAUT and the MAL was much greater for the amount of use than for the quality of movement subscale.

Table 2.

Characteristics of Research Participants

	Agea (Range)	Gender (% Male/Female)	Affected Hand (% Left/Right)	Function (% Low/High)	FM (Range)
Sample, n	106	106	106	106	105
CIMT	60.98 ± 13.47 (28.6–84.0)	69/37 (65.1/34.9)	58/48 (54.7/45.3)	23/83 (21.7/78.3)	42.54 ± 11.67 (6–66)
Sample, n	116	116	116	116	116
Control	63.26 ± 12.56 (18.5–89.8)	73/43 (62.9/37.1)	62/54 (53.4/46.6)	22/94 (19.0/81.0)	41.05 ± 12.88 (8–63)

Abbreviations: CIMT, constraint-induced movement therapy; FM, Fugl-Meyer Assessment (score range 0–66).

^aAge to the nearest year.

Table 3.

ICC, SEM, and MDC₉₀ for the AAUT and the MAL (n = 116)

	Meanbaseline, %	SDbaseline, %	ICC (3,1)	SEM, %	MDC90			Original MDC90a
					Value, %	95% CI_ub, %	95% CI_lb, %
AAUTa	40.4	31.2	0.89	10.4	24.2	20.1	29.0	24.2%
AAUTq	22.8	20.7	0.91	6.2	14.4	12.0	17.4	0.72
MALa	27.8	18.3	0.85	7.2	16.8	14.0	20.0	0.84
MALq	29.1	18.2	0.87	6.6	15.3	12.7	18.3	0.77

Abbreviations: AAUTa: amount of use of the Actual Amount of Use Test; AAUTq: quality of movement of the Actual Amount of Use Test; MALa, amount of use of the Motor Activity Log; MALq, quality of movement of the Motor Activity Log; SD_baseline, standard deviation at baseline; ICC, intraclass correlation coefficient; SEM, standard error of measurement; MDC₉₀, minimal detectable change at 90% of confidence; CI_ub, upper bound of confidence interval; CI_lb, lower bound of confidence interval.

^aValues in this column are the MDC₉₀ values before normalization except for AAUTa.

The independent t test showed that the CIMT-induced changes between groups reached a significant difference (P < .001) for AAUTa, AAUTq, MALa, and MALq (Figure 1). Immediately after the intervention for the CIMT group, we found a 24.9% (9.8/39.4) increase in AAUTa, a 34% (0.34/1.00) increase in AAUTq, an 84.6% (1.15/1.36) increase in MALa, and a 72.2% (1.04/1.44) increase in MALq, suggesting noticeable CIMT-induced improvement in spontaneous paretic arm use (Table 4). This effect was maintained at the 1-year follow-up evaluation (Figure 1).

Figure 1.

Spontaneous arm use as measured by the AAUT and the MAL for both groups. The measures were taken at 3 different time points: pre-CIMT, post-CIMT, and 1-year follow-up. (A) Amount of Use and (B) Quality of Movement. Abbreviations: CIMT, constraint-induced movement therapy; AAUTa, amount of use of the Actual Amount of Use Test; AAUTq, quality of movement of the Actual Amount of Use Test; MALa, amount of use of the Motor Activity Log; MALq, quality of movement of the Motor Activity Log.

Table 4.

Comparison Between CIMT-Induced Improvement for Spontaneous Arm Use and the MDC₉₀ Values Measured by the AAUT and the MAL (Mean ± Standard Error) Immediately After CIMT

	Pre-CIMT	Post-CIMT	Δ	Original MDC90	If Δ > MDC90	Number ≥ MDC90
Sample, n	106	95	95	—	—	95
AAUTa, %	39.4 ± 2.6	49.2 ± 2.9	9.8	24.2	No	18
AAUTq (0–5)	1.00 ± 0.08	1.34 ± 0.97	0.34	0.72	No	18
Sample, n	105	98	98	—	—	98
MALa (0–5)	1.36 ± 0.09	2.52 ± 0.10	1.15	0.84	Yes	66
MALq (0–5)	1.44 ± 0.09	2.48 ± 0.10	1.04	0.77	Yes	62

Abbreviations: CIMT, Constraint-Induced Movement Therapy; AAUTa, amount of use of the Actual Amount of Use Test; AAUTq, quality of movement of the Actual Amount of Use Test; MALa, amount of use of the Motor Activity Log; MALq, quality of movement of the Motor Activity Log; Δ, difference between post-CIMT and pre-CIMT; MDC₉₀, minimal detectable change at 90% of confidence.

For the comparison of the CIMT-induced spontaneous arm use and the MDC₉₀, we found that only those measured by the MAL exceeded the MDC₉₀ (Tables 4 and 5). Surprisingly, these significant training effects, when measured for either sub-using the AAUT, did not exceed the MDC₉₀ scale immediately after the CIMT intervention and 1-year follow-up visits (Tables 4 and 5). Figure 2 shows the CIMT-induced change for each of the 4 subscales as a function of the MDC₉₀ thresholds. We found that the numbers of participants whose MALa and MALq exceeded the MDC₉₀ were 67.3% (66/98) and 63.4% (62/98) immediately after the CIMT intervention (Figure 2, Table 4). These percentages were maintained at approximately the same level at the 1-year follow-up (63.8% [51/80] and 70% [56/80], respectively; Figure 2, Table 5). In contrast, for the AAUT, we found that the number of participants whose AAUTa and AAUTq exceeded the MDC were much lower with both 18.9% (18/95) immediately after the CIMT intervention (Figure 2, Table 4) and 16.3% (13/80) and 31.3% (25/80) at the 1-year follow-up, respectively (Figure 2, Table 5).

Table 5.

Comparison Between CIMT-Induced Improvement for Spontaneous Arm Use and the MDC₉₀ Values Measured by the AAUT and the MAL (Mean ± Standard Error) at 1-Year Follow-Up

	Pre-CIMT	1-Year Follow-Up	Δ	Original MDC90	If Δ > MDC90	Number ≥ MDC90
Sample, n	106	80	80	—	—	80
AAUTa, %	39.4 ± 2.6	48.7 ± 3.0	9.3	24.2	No	13
AAUTq (0–5)	1.00 ± 0.08	1.49 ± 0.11	0.49	0.72	No	25
Sample, n	105	80	80	—	—	80
MALa (0–5)	1.36 ± 0.09	2.52 ± 0.12	1.15	0.84	Yes	51
MALq (0–5)	1.44 ± 0.09	2.64 ± 0.12	1.2	0.77	Yes	56

Abbreviations: CIMT, Constraint-Induced Movement Therapy; AAUTa, amount of use of the Actual Amount of Use Test; AAUTq, quality of movement of the Actual Amount of Use Test; MALa, amount of use of the Motor Activity Log; MALq, quality of movement of the Motor Activity Log; Δ, difference between 1-year follow-up and pre-CIMT; MDC₉₀, minimal detectable change at 90% of confidence.

Figure 2.

Magnitude of change in spontaneous arm use immediately post-CIMT intervention and at 1-year follow-up for individuals poststroke in the CIMT group. Spontaneous arm use change for each participant is plotted for (A) AAUTa, (B) AAUTq, (C) MALa, and (D) MALq as a function of the MDC₉₀ threshold (horizontal line). The total number of participants and the number of participants whose scores were equal/greater or less than the MDC₉₀ threshold are indicated for each evaluation time point. Abbreviations: CIMT, constraint-induced movement therapy; AAUTa, amount of use of the Actual Amount of Use Test; AAUTq, quality of movement of the Actual Amount of Use Test; MALa, amount of use of the Motor Activity Log; MALq, quality of movement of the Motor Activity Log; MDC₉₀, minimal detectable change at 90% of confidence.

Discussion

This is the first study to explore “interpretability” in relation to measurement of spontaneous arm use poststroke. Data from a large multisite longitudinal study compute the MDC₉₀. In turn, MDC₉₀ were used to can be used to determine whether the training-induced effects are due to the intervention itself or to noise inherent to the measurement tool. The MDC₉₀ reflects a 90% confidence interval that the magnitude of measurement variability will be less than the MDC values.²⁵ For example, the MDC₉₀ for the AAUTa is 24.2% in the EXCITE trial. This suggests that the amount of affected arm use measured by the AAUTa will be less than 24.2% for 90% of the individuals who have suffered mild to moderate stroke. This finding implies that a change of amount of use greater than 24.2% for an individual is necessary to be 90% certain that the change is not because of measurement error in the context of CIMT as applied in the EXCITE trial.

The MDC₉₀ for the AAUTa and MALa were 24.2% and 16.8%, respectively, whereas the MDC₉₀ for the AAUTq and MALq were 14.4% and 15.3%, respectively. Relatively speaking, the MDC₉₀ of quality of movement is less than that of amount of use for both the AAUT and the MAL, suggesting that the measurement of quality of movement has relatively higher sensitivity compared with that of amount of use when capturing spontaneous arm use for people with mild to moderate stroke. This observation is consistent with previous studies that explored the clinometric properties of the MAL^24,26 where the quality of movement subscale was found to be a more reliable and valid measure of real-world arm use than the amount of use sub-scale. However, this analysis represents the first time the same comparison is reported for the AAUT.

An absolute difference in the MDC₉₀ between the AAUTq and the MALq of 0.9% indicates a similar level of measurement variation for the quality of movement in both measures. In sharp contrast, an absolute difference in the MDC₉₀ between the AAUTa and the MALa of 7.4% suggests the AAUT amount of use subscale is a relatively unstable measurement.¹⁸ One explanation for the relatively large MDC₉₀ of the AAUTa may be the large standard deviation. Recall that the MDC₉₀ calculation is a function of the SD and ICC. Here, the ICC values for all 4 measures were similar and high, ranging from 0.85 (MALa) to 0.91 (AAUTq). However, a closer look at the SD for the 2 sub-scales of the AAUT and the MAL reveals a relatively large SD for the AAUTa (31.2%) compared with that for the AAUTq (20.7%), the MALa (18.3%), and the MALq (18.2%). Another possible explanation is that the large MDC₉₀ of the AAUTa may be because of the lower resolution of the scoring system.²⁷ The AAUTa score is a ratio based on the number of tested items in which the participant “attempted or used the paretic arm.” Each item receives a binary score of 1 or 0, where a “1” is assigned for any action identified in the range of “attempt to use” to “normal use.” Therefore, the measurement of the AAUTa is not as sensitive as that of the MALa, which uses a finer grained scale of 0 to 5.

Although the AAUT consists of a relatively insensitive amount of use subscale, there are at least 2 reasons that it can be used to provide some useful information about post-stroke spontaneous arm use. First, the AAUT is a real-time observational performance-based measure that provides information closer to actual arm use. Second, most tasks in the AAUT are designed to be bimanual in nature, which may provide a more natural and sensitive means to capture task-based paretic arm use.²⁸ The focus on bimanual tasks is thought to be important in daily life based on evidence from the literature. In an observational study, Kilbreath et al²⁹ investigated the frequency of hand use in older adults during daytime hours and found that in the majority of the observations 54% involved bimanual tasks whereas 29% involved unimanual activities. Recently, Johnson et al³⁰ reported an experimental result using force-feedback cueing in a robot-assisted stroke study. They found that if non-use of the affected arm exists, the affected arm will most likely be underused for bimanual steering compared with unimanual steering. Later, Paranjape et al³¹ reported findings in part supporting those of Johnson et al.³⁰ Together, these reports suggest that bimanual activities play an important role in many activities of daily living, and may be more sensitive in the detection of nonuse of the affected arm. An assessment focused on unimanual activities, therefore, may not capture the ‘amount of use’ as accurately as one that includes bimanual activities.

Constraint-induced movement therapy focuses on affected arm training with primarily unimanual tasks; therefore, how much and when the training effect would transfer to bimanual tasks is uncertain. The mean score improvement in spontaneous arm use after CIMT was significant for both the AAUT and the MAL, and this improvement was maintained at the 1-year follow-up. However, the magnitude of the improvement exceeded the MDC₉₀ only for the MAL but not the AAUT. This was the case for both evaluation time points. The inconsistency between the mean score and the MDC results highlights the important differences in the nature of 2 instruments. In fact, more than half of the MAL tasks pertain only to unimanual performance.¹² The AAUT tasks, however, were designed to be performed primarily bimanually.²⁸ With this in mind, we might expect the AAUT to be more sensitive to changes in bimanual limb use. If there is a generalization to bimanual limb use after the CIMT intervention, we would further expect the AAUT to capture this generalization at the later time point. However, the AAUT result at 1-year follow-up suggests that a pattern of more normal bimanual arm use did not emerge.

One possible explanation for the minimal generalization to bimanual tasks is that, for some tasks, participants may feel more confident using the less-affected arm alone than using both.^32–34 In sum, the fact that significant improvement was observed in the MAL but not in the AAUT (when comparing the MDC₉₀ values) may be influenced by the nature of the tasks in each measure and the participants’ confidence to perform the bimanual tasks with the weak limb.

Regardless of the recall bias for the MAL and the reliability concerns for the AAUT, these 2 measures only provide rough estimates of spontaneous arm use. Uswatte et al^9,10,35 performed a number of studies using accelerometers to directly capture the quantity of arm use in the real world for individuals poststroke and to verify the MAL.^12,26 The Bilateral Arm Reaching Test³⁶ was found to have good reliability and acceptable validity as well. Mobile health innovations, such as wireless sensors that use machine-learning algorithms to determine the type, quantity and quality of movements in the laboratory and home, will make better measurements feasible and inexpensive.^37,38

Conclusions

The determination of measurement variability using MDC is important for uncoupling a meaningful intervention-induced change from random measurement noise. Although the AAUT is arguably an objective measure of spontaneous arm use, here, its sensitivity in capturing “amount of use” is relatively low. One reason for the insensitivity may be the low resolution of its scoring system, particularly for the amount of use subscale. As such, there is considerable need to develop valid and reliable tools to capture purposeful arm use outside the laboratory, perhaps through leveraging new sensing technologies with objective activity monitoring capable of determining the actual amount of purposeful arm use in daily living.