Performance Variability as a Predictor of Response to Aphasia Treatment

Abstract

Background

Performance variability in individuals with aphasia is typically regarded as a nuisance factor complicating assessment and treatment.

Objective

We present the alternative hypothesis that intra-individual variability represents a fundamental characteristic of an individual’s functioning and an important biomarker for therapeutic selection and prognosis.

Methods

Nineteen individuals with chronic aphasia participated in a 6-week trial of imitation-based speech therapy. We assessed improvement both on overall language functioning and repetition ability. Further, we determined which pre-treatment variables best predicted improvement on the repetition test.

Results

Significant gains were made on the Western Aphasia Battery (WAB) Aphasia Quotient, Cortical Quotient, and two subtests, as well as on a separate repetition test. Using stepwise regression, we found that pre-treatment intra-individual variability was the only predictor of improvement in performance on the repetition test, with greater pre-treatment variability predicting greater improvement. Furthermore, the degree of reduction in this variability over the course of treatment was positively correlated with the degree of improvement.

Conclusions

Intra-individual variability may be indicative of potential for improvement on a given task, with more uniform performance suggesting functioning at or near peak potential.

INTRODUCTION

Therapeutic research in aphasia typically characterizes baseline and improved language skills in terms of mean scores on a specific task or assessment battery. Whereas this approach succeeds at capturing variability across individuals, it fails to capture such variability within individuals. Performance fluctuations within a single individual (intra-individual variability) are typically perceived as an inconvenient impediment to reaching the desired general conclusions about a new therapy. But treating intra-individual variability as a nuisance parameter or measurement error, e.g., of the same magnitude and significance as inter-individual variability¹, may be giving up important and highly relevant information about the therapy. In fact, short-term performance inconsistency on a particular task may represent a characteristic feature of – and a metric to gauge – an individual’s functional status. Particularly in the context of large variability, mean performance may oversimplify the true nature of behavior and inadequately capture the range of ability², obscuring insight into potential therapeutic benefit and outcome assessment on an individual basis.

Existing limited research on intra-individual variability in cognitive and perceptual-motor function in healthy aging^3–6 and dementia^7–9, suggests a relation between increased intra-individual variability and decreased performance. Yet other, seemingly contradictory, findings suggest that greater intra-individual variability has positive implications for acquiring skills with practice or training. For example, increased intra-individual variability in a cognitive or motor skill during learning precedes (and presages) mastery of that skill during development,¹⁰ and in cognitive training of healthy older adults, the pre-treatment degree of intra-individual variability predicts higher response accuracy and performance improvement¹¹.

These data suggest that performance variability may suggest susceptibility to change and/or the potential for learning. Fluctuations representing adaptive variability¹² may be conceived of not as vulnerability, but as potential. Further, distinguishing between adaptive and maladaptive variability may be key to understanding the significance of these measures in predicting future outcomes.

In the realm of stroke recovery, extensive investigation has addressed differences between individuals, yet little work studying performance variability within individuals (although important work has explored the role of attention in intra-individual variability in aphasia^13,14). In the recovery of language functions after stroke, intra-individual performance variability has not been investigated, either as a correlate of present functioning or as a predictor of post-treatment ability (but see¹⁵ for a theoretical study). The implications are far-reaching. From a research standpoint, our knowledge of language recovery in aphasia is limited to mean scores and effect sizes using pooled standard deviations, thus neglecting individual parameters of variability. Such data may represent fundamentally incomplete metrics, substituting a crude numerical proxy for the more nuanced complexity of performance, and thus profoundly affecting our understanding of recovery. From a clinical standpoint, such omission could have graver consequences, since the most desirable measure of rehabilitation success is a patient’s consistent performance in the real world, not maximal performance in the clinic or the possibility of good performance under ideal conditions.

We hypothesize that intra-individual variability on a language task is predictive of the ability of an individual with aphasia to improve mean performance on that task through training. We investigate this hypothesis in a clinical trial of an intensive, imitation-based aphasia therapy motivated by neurophysiological evidence^16,17. The therapy uses a computer interface to prompt repetition of words and phrases to engage a frontal-parietal motor cortical network involved in both observation and execution of speech¹⁸. Nineteen subjects with ischemic stroke received imitation-based therapy involving repetition of words and phrases. In this paper, we report on an experiment testing the hypothesis that pre-treatment intra-individual variability would predict therapeutic outcome.

METHODS & MATERIALS

Participants

Nineteen native English speakers with aphasia following single, left hemisphere ischemic stroke, confirmed by neurological examination and MRI, were recruited (age range 31–72; mean = 53.5; SD 11.7; 4 female (21%)). All had sustained a single stroke 5 to 130 months prior to enrollment (mean = 41.6; SD 42.9). Demographic and neurological information are in Supplemental Table 1. The Institutional Review Boards of The University of Chicago and University of California, Irvine approved the study. Consent was obtained according to the Declaration of Helsinki.

Experimental Summary

The intensive imitation-based therapy, administered 6 days per week for three 30 minute sessions each day, required participants to listen to words and phrases presented by six different speakers and to repeat them either once or multiple times. Half of the participants also saw a video of the speaker during the presentation. Because there was no statistical difference in any measure between those subjects who saw the speaker and those who did not, all data have been aggregated for this report. Complete details about the IMITATE therapy system can be found in Supplemental Material and in earlier work^16,17.

Over the six-week therapeutic period (Weeks 1–6 of the overall study), participants undertook the specialized speech therapy on a preprogrammed, dedicated laptop. Participants underwent two behavioral assessments (WAB, repetition test) that were administered twice before and twice after therapy, with all evaluations six weeks apart (Weeks −6, 0, 6, and 12). These measures were administered twice pre-therapy to establish a stable baseline, and twice post-therapy to establish immediate changes and maintenance.

Figure S1 in Supplemental Material depicts the timing of these assessments relative to therapy.

Western Aphasia Battery

The Western Aphasia Battery-Revised (WAB)¹⁹ was used as the primary outcome measure, as it was anticipated that benefits of our imitation-based therapy would generalize to other domains of language^16,17,20. The WAB was administered at each of the four main behavioral assessment sessions by a speech-language pathologist (SLP) blind to treatment group. We analyzed the WAB Aphasia Quotient (WAB-AQ), Cortical Quotient (WAB-CQ), and the four subcomponents of the WAB-AQ (Spontaneous Speech, Auditory Verbal Comprehension, Repetition, and Naming and Word Finding). There was no significant difference in any of these measures between the two pre-treatment sessions or between the two post-treatment sessions (p > 0.05 on two-tailed paired t-tests).

Repetition Test

Tests of repetition accuracy were administered during all four pre- and post-treatment behavioral assessments. These were administered by an SLP using words and phrases randomly selected from the pool of IMITATE therapy stimuli. Repetition tests and behavioral assessments were performed by different SLPs, blinded to the other’s findings. One subject (2) was excluded from this analysis due to missing data, leaving 18 subjects.

Repetition test stimuli consisted of words and phrases of high difficulty, based on the level to which the subject was expected to advance. Each block of words contained 10 words. Each block of phrases contained 10 phrases with a varying number of words (2–6) depending on level. For both blocks, each word was scored on a 5-point scale (0 signifying no vocalization; 5 indicating accurate, prompt repetition). Scoring was performed once offline by a single SLP for all subjects, and therefore reliability rates are not reported. Performance on these measures was combined in a single repetition score for each time point (mean score for Words and Phrases). There were no significant differences between Week −6 and Week 0 repetition scores, or between Week 6 and Week 12 repetition scores (p > 0.05 on two-tailed unpaired t-test).

Changes in Behavioral Performance

Seven measures of language performance were studied: WAB-AQ, WAB-CQ, the four subcomponents of the WAB-AQ, and the score from the repetition test. For each measure, the pre-treatment score was taken to be the mean of Week −6 and Week 0 scores (see Supplemental Materials for details of weighting the repetition test), and the post treatment value is the Week 6 score (post-therapy repetition). We did not use scores from Week 12 (fourth behavioral assessment) as two subjects missed this assessment. Therefore, our definition of improvement, for all measures, is the Week 6 score minus the mean for Weeks −6 and 0.

Pre-treatment and post-treatment scores were compared using two-tailed paired t-tests. All significance tests use α = 0.05. Due to the nested nature of the WAB measures (i.e., four subcomponents of the WAB-AQ are used, which also contribute to WAB-CQ), Bonferroni correction with n = 5 was applied for the repetition assessment and the four subcomponents of the WAB-AQ (Spontaneous Speech, Auditory Verbal Comprehension, Repetition, and Naming and Word Finding).

Intra-individual Variability as Predictor of Improvement

The repetition test based on stimuli from the pool of IMITATE items was used to test directly the hypothesis that intra-individual variability in a language task is predictive of the ability of an individual with aphasia to improve performance on that task through training. This single measure was selected for two reasons: (1) there were two days on which the repetition test was administered at least twice (Week 0 and Week 6), allowing a robust assessment of individual variability before and after treatment, and (2) these stimuli were developed to be grossly equivalent in complexity, in contrast with the hierarchical ordering of increasing complexity on the WAB subtests. We chose not to pool data from Weeks −6 and 0 when computing intra-individual variability to avoid confounding variability on different time scales; our intra-individual variability scores measure performance variability within a given day only. We computed a repetition intra-individual variability measure pooling variances of Words and Phrases blocks. Details can be found in Supplemental Materials.

Our specific question was the extent to which pre-treatment repetition intra-individual variability predicted improvement in repetition Mean, which we determined by computing a Pearson correlation coefficient. We used Week 0 and Week 6 repetition Mean scores as the pre-and post-treatment values, ignoring Week −6 scores for consistency with intra-individual variability calculations. There was no significant difference between pretreatment repetition Mean calculated with and without Week −6 repetition test (two-tailed paired t-test, p > 0.05).

We used stepwise linear regression to identify those pre-treatment variables that best predicted improvement. In addition to pre-treatment repetition intra-individual variability, these variables included participant age, months post stroke onset (MPO), number of sessions completed (NSC), aphasia type (fluent vs. nonfluent), and pre-treatment repetition Mean. NSC was tracked by automated video recording of patient participation during each session via the built-in laptop camera, and then verified by review of these recordings. Stepwise regression was performed with the MATLAB stepwise function, using the default settings: a new predictor is selected if its regression coefficient would be significantly nonzero at the 0.05 level, and an existing predictor is removed if its coefficient is not significantly nonzero at the 0.10 level.

RESULTS

Changes in Behavioral Performance

Statistically significant improvements were demonstrated in five of the seven language measures assessed, with correction for multiple comparisons. Results of two tailed t-tests are summarized in Table 1 with uncorrected p values. We used Bonferroni correction (n = 5) for the six WAB-Revised measures vs. repetition test to determine significance. Significant improvement was measured for the repetition test, WAB-AQ, WAB-CQ, and two of the four WAB-AQ subcomponents (Repetition, Naming and Word Finding). The two remaining subcomponents of the WAB-AQ (Spontaneous Speech, Auditory Verbal Comprehension) did not demonstrate significant change.

Table 1.

Performance measures for all subjects

MEASURE	Mean Pre (SD)	Mean Post (SD)	Mean Improvement (SD)	p value (uncorrected)
WAB-AQ	67.72 (20.00)	70.34 (18.33)	2.61 (3.73)	.0068*
WAB-CQ	71.27 (16.50)	73.89 (15.44)	2.62 (2.72)	.0005*
WAB-SS	12.42 (4.49)	12.81 (3.89)	0.47 (1.36)	.1460
WAB-AVC	164.40 (26.94)	166.42 (25.78)	2.02 (7.67)	.5186
WAB-Rep	67.16 (26.64)	72.00 (24.85)	4.84 (6.49)	.0044*
WAB-NWF	64.61 (29.21)	67.00 (29.19)	2.39 (3.13)	.0037*
Repetition Test	79.40 (19.03)	86.19 (19.11)	6.58 (5.90)	.0002*

The asterisk (^*) marks significant values after Bonferroni correction for 2 comparisons (^*p < 0.01/2). SD: standard deviation; WAB: Western Aphasia Battery; AQ: Aphasia Quotient; CQ: Cortical Quotient; SS: Spontaneous Speech; AVC: Auditory Verbal Comprehension; Rep: Repetition; NWF: Naming and Word Finding.

Intra-individual Variability as Predictor of Improvement

In this section, only the repetition test results are used. In contrast to Table 1, pre-treatment results are from Week 0 only, for reasons explained above. Pre-treatment repetition Mean ranged from 20.5% to 99.5% (overall mean 79.4%, SD 18.8%). Improvement in repetition Mean from pre-treatment to post-treatment (Week 0 to Week 6) ranged from −3.8% to 16.5% (median 5.3, mean 6.7, SD 5.7), representing a mean improvement of 0.34 points on the 5-point scale used to rate the repetition performance.

Fig. 1A shows pre-treatment intra-individual variability versus Improvement in repetition Mean performance, and Fig. 1B shows pre-treatment repetition Mean vs. pre-treatment intra-individual variability.

Figure 1.

Relationships between Pre-treatment Intra-Individual Variability, Pre-treatment Mean and Improvement in Mean for the repetition test. Subject 2 was excluded due to missing data, leaving 18 subjects. Subjects marked with black crosses are excluded from further analysis (see main text).

We removed several subjects from further analysis due to outlier status (4) and possible ceiling effects (i.e., subjects near threshold pre- or post-therapy: 10,11,12,16), as detailed in Supplemental Materials. For the remaining subjects, there is a positive correlation (r=0.68, p=0.01 uncorrected) between pre-treatment intra-individual variability and improvement, i.e. higher pre-treatment intra-individual variability is associated with greater improvement. We then considered all of the pre-treatment variables listed earlier (Age, MPO, NSC, aphasia type, pretreatment intra-individual variability and pre-treatment Mean) as possible predictors of improvement in post-therapy repetition accuracy. In a stepwise regression, the optimal regression model found intra-individual variability to be the only predictor of improvement (p = .01, as noted above). With all subjects included, the relationship remains highly significant (p = .0001), with no additional predictors selected. Repeating this stepwise regression without variability included in the model resulted in the selection of no variables, whether for the entire group or with near-threshold subjects excluded.

Finally, we examined intra-individual variability in repetition accuracy immediately post-treatment (Week 6). This decreased significantly over the course of treatment (two-tailed paired t-test, p < 0.05), regardless of whether we consider all subjects or exclude near-threshold subjects (as detailed in Supplemental Materials). Post-treatment intra-individual variability in repetition accuracy is positively correlated with change in repetition Mean when we consider all 18 subjects (Pearson’s r = 0.49, p = 0.04 in a 2-tailed t-test). However, it is no longer significant when we exclude subjects who were near-threshold either before or after therapy (r = 0.25, p = 0.41). The change in intra-individual variability in repetition accuracy over the course of treatment has a significant negative correlation with improvement, whether considering all subjects (r = −0.48, p = 0.053) or excluding near-threshold subjects (r = −0.57, p = 0.04). See Fig. 2. This effect remains if we control for all of the confounding variables (Age, MPO, NSC, aphasia type, pre-treatment Mean, pre-treatment intra-individual variability; r = −0.69, p = 0.018). Put another way, reduction in intra-individual variability is positively correlated with improvement.

Figure 2.

Relationship between change in Intra-Individual variability and change in Mean for the repetition test. Subjects shown are as in Figure 1.

DISCUSSION

The present study reviews outcomes of a clinical trial of imitation-based therapy for chronic aphasia, and explores a new hypothesis about the role of intra-individual variability in predicting benefit from language therapy following stroke.

This study showed positive effects of the IMITATE system of repetition based, computer assisted speech/language therapy for patients with chronic aphasia. In particular, participants undergoing the therapy had statistically significant gains on composite language and cognitive measures on a standard test for aphasia (WAB-Revised), as well as on two repetition accuracy measures. Significant gains were made over a relatively short treatment period (6 weeks) in subjects who, in some cases, were more than a decade removed from their stroke. Future investigation will refine the IMITATE therapeutic protocol in view of the results from the current study, related research²¹, and other theoretical considerations.

Our analysis suggests that subjects demonstrating higher levels of performance variability prior to therapy are likely to experience greater improvement over the course of treatment. Specifically, subjects demonstrating greater intra-individual variability during repetition before therapy demonstrated greater improvement in repetition than those with lower intra-individual variability. Perhaps most interestingly, intra-individual variability declined over the course of treatment, and there was a significant correlation between performance improvement and intra-individual variability reduction.

The finding that intra-individual variability is a positive predictor of language improvement appears to conflict with existing literature on the relation between intra-individual variability and task performance in cognitive and perceptual-motor domains. In healthy aging and dementia, intra-individual variability has generally been negatively correlated with both short-term performance²² and long-term variables⁵, including time until death²³. On the other hand – and perhaps most relevant here – evidence also suggests that increased variability in a particular cognitive or motor domain may be associated with greater potential for change following training specific to that domain¹².

Correlation of task-specific variability with performance improvement has been attributed to influences of learning and strategy use in development²⁴. During skill acquisition, changes occur in execution of strategies, even in the absence of changes in strategy selection²⁵. These subtle changes may result in adaptive variability while learning specific tasks. Thus as an individual achieves maximal potential on a task, variability decreases. In expert motor control, when an individual is performing a highly practiced skill at or near peak level, performance variability is reduced, and this consistency is reflected in precise activation of neural networks during motor planning²⁶. Our finding that intra-individual variability decreased over the course of therapy provides further support for this proposition, especially given the significant correlation between improvement on the repetition test and intra-individual variability reduction. Within the limitations of their language impairment, our participants became more expert at the practiced task, thus demonstrating more consistent and more accurate performance. Although it is impossible to determine from the present study, it would be of great interest to explore whether such variability might continue to play a predictive role in the outcome of further therapy or with the introduction of new or more difficult tasks.

That increased variability has been found at dynamic periods of cognitive decline and development suggests, not surprisingly, that these transitions do not occur uniformly. It seems probable that such variability indicates a lack of system stability that is influenced by opposing tendencies. On one hand, in a progression towards overall decline, increased variability results as the valleys of performance drop more deeply; on the other, in the case of development, or recovery, the heightening of peaks is responsible for the observed fluctuation. In support of this, increasing latency for an individual’s slowest reaction times is related to increasing variability for older adults²⁷. Nevertheless, cognitive enhancement can occur with training and stimulation programs, with functional gains reported in daily life despite increasing age²⁸. As in development and recovery, when older subjects realize increased potential, greater intra-individual variability is correlated with improved learning¹².

In the present study, it is possible that individuals demonstrating less variability are at or near an asymptote of their abilities, given their neurological status, the extent of lesion damage, and the degree to which they have already experienced recovery. While it is generally accepted that individuals with aphasia encounter a plateau within the first year following stroke²⁹. Intra-individual variability may serve as a more sensitive, individualized measure of potential than time post stroke, as well as an immediate and cost efficient means of prediction.

The implications for language rehabilitation are of great significance, as predictors of response to aphasia treatment are presently limited. Specifically, it may be productive for clinicians to target skills in which patients demonstrate high performance variability prior to treatment, rather than areas in which limited variability suggests a reduced capacity for gains with therapy. It may also be productive to periodically re-assess patient performance on a variety of tasks, in order to determine whether cycling through treatment goals, selected on the basis of variability as a proxy for potential, may be beneficial. However, such possibilities should be interpreted with caution, as the present study represents a new avenue of inquiry, and little is yet known about how intra-individual variability changes over the course of recovery. Although the present analysis considered time post onset, subjects participating in this study were all at chronic aphasia stages. Therefore, there is no suggestion that findings would be identical or even similar in acute phases of recovery. Additionally, several measures that may impact variability in task performance were not included in our assessment, such as attention, mood, and fatigue. Future studies would benefit from operationalization and inclusion of these variables.

Further limitations of our study include the potential for practice effects, given the relatively short time over which these tests were administered. However, we believe that the lack of significant differences between the two pre-therapy time points and the two post-therapy time points suggests that this is not a major confound for the present study. While our inclusion of fluent vs. nonfluent aphasia classifications did not indicate significant differences in benefit between these groups, there was not adequate power in our sample to address the differential effects of repetition therapy that may exist for different aphasia types. It is also worth noting that our imitation-based therapy was heavily dependent on motor processes, as was our repetition outcome measure. Therefore, it is not possible to definitively state that intra-individual variability would predict improvement on purely cognitively-based tasks.

While extrapolation from the present study to clinical guidelines would be premature, if the relationship between behavioral intra-individual variability and post treatment performance withstands further exploration, it may suggest that those demonstrating higher levels of baseline variability are good candidates for intervention. Intra-individual variability, in this conception, could represent a measure of plastic potential, the extent to which an individual’s present neurological status is conducive to the kind of recovery or reorganization necessary to manifest improvement with practice and stimulation. However, individuals performing consistently at the same level may require different types of intervention if they are to realize enhanced function, and these patients may be better candidates for referral to clinical trials, pharmacology or more invasive forms of treatment.