Estimating the Accuracy of the Chedoke–McMaster Stroke Assessment Predictive Equations for Stroke Rehabilitation

ABSTRACT

Purpose: To estimate the predictive accuracy and clinical usefulness of the Chedoke–McMaster Stroke Assessment (CMSA) predictive equations.

Method: A longitudinal prognostic study using historical data obtained from 104 patients admitted post cerebrovascular accident was undertaken. Data were abstracted for all patients undergoing rehabilitation post stroke who also had documented admission and discharge CMSA scores. Published predictive equations were used to determine predicted outcomes. To determine the accuracy and clinical usefulness of the predictive model, shrinkage coefficients and predictions with 95% confidence bands were calculated.

Results: Complete data were available for 74 patients with a mean age of 65.3±12.4 years. The shrinkage values for the six Impairment Inventory (II) dimensions varied from −0.05 to 0.09; the shrinkage value for the Activity Inventory (AI) was 0.21. The error associated with predictive values was greater than ±1.5 stages for the II dimensions and greater than ±24 points for the AI.

Conclusions: This study shows that the large error associated with the predictions (as defined by the confidence band) for the CMSA II and AI limits their clinical usefulness as a predictive measure. Further research to establish predictive models using alternative statistical procedures is warranted.

INTRODUCTION

More than 50,000 people have strokes in Canada each year, and there are currently 300,000 Canadians living with the effects of stroke.¹ Specialized interdisciplinary stroke rehabilitation units have been shown to be effective in reducing length of stay for persons recovering from stroke.² A recent systematic review indicated that rehabilitation incorporating high-intensity, repetitive task-specific practice can optimize motor recovery after stroke.³ Nonetheless, the personal and societal costs that arise following stroke are significant.⁴

Measuring clinical outcomes is an integral part of evidence-based rehabilitation.⁵ Kirshner and Guyatt⁶ have noted that measures of health status have three possible purposes: (1) to discriminate among individuals, (2) to evaluate within-person change over time, and (3) to predict outcomes. The ability to predict outcomes following stroke yields several benefits: it can identify patients who would benefit from specific rehabilitation services, facilitate the provision of education on recovery to patients and their families, and offers the potential to improve planning of stroke care and rehabilitation services.⁷ It has been reported that validated, standardized outcome measures should be used to develop these predictive models.⁷

The Chedoke–McMaster Stroke Assessment (CMSA) is a highly reliable and valid outcome measure used by rehabilitation personnel to assess impairment and activity following stroke.⁸ The CMSA can be used to discriminate among individuals by assigning a specific motor stage of recovery using the Impairment Inventory, and the Activity Inventory (formerly called the Disability Inventory) can be used to evaluate change in physical functioning.⁸ Evidence for the reliability and validity of the CMSA has been reported for patients with stroke,^8–11 patients with varied neurological impairments,¹² patients with acquired brain injury,¹³ and elderly clients receiving inpatient rehabilitation.¹⁴ Furthermore, the CMSA is reported to be useful for predicting clinical outcomes in persons with stroke in both acute¹⁵ and rehabilitation settings.^16–18

The use of the CMSA to predict outcomes in patients with stroke undergoing rehabilitation was first reported by Gowland in the 1980s.^16,17 Subsequently, using the current version of the CMSA and a sample of 182 patients, revised predictive equations were generated using multiple regression analysis; these were reported in the CMSA manual, published in 1995.¹⁸ Gowland reported that predicted outcomes, used in conjunction with clinical judgment, could guide clinicians in selecting appropriate physiotherapeutic interventions and could also inform discharge planning.¹⁸ To date there have been no published studies further evaluating the psychometric properties of these predictive equations and their application to clinical practice.

The purpose of this investigation was to estimate the predictive accuracy and clinical usefulness of the CMSA predictive equations for patients with stroke undergoing rehabilitation. In the absence of previous cross-validation studies, determining the accuracy of the predictive equations will provide important information to health professionals working in stroke rehabilitation about the equations' value in a clinical setting, and is in keeping with the mandate of evidence-based practice.

METHODS

Design

This longitudinal prognostic study used historical data obtained from the Rehabilitation Hospital Health Sciences Centre, a tertiary-care institution in Winnipeg, Manitoba. The data were analyzed at McMaster University. The University of Manitoba Health Research Ethics Board approved access to an anonymized database for this investigation.

Participants and Procedure

The database used in this study was originally developed for a previous study that gathered data retrospectively from patient charts. That study did not address predictive accuracy.¹¹ Data were gathered from the charts of all patients who were admitted to the stroke rehabilitation unit between July 1996 and July 1998 and were assessed at admission and discharge with the CMSA. There was no change to usual assessment and intervention care over this period. Physiotherapists and student physiotherapists assessed patients with the CMSA according to the instructions outlined in the CMSA manual,¹⁸ and all patients received conventional stroke rehabilitation during their stay in the rehabilitation unit. The original database consisted of patient demographic information, Impairment Inventory scores, and total Activity Inventory scores for 104 patients. All physiotherapists working in the stroke rehabilitation unit were involved in reviewing the patient charts and extracting the data. The purpose of the original study was to evaluate the functional outcomes of patients participating in a stroke rehabilitation program; specifically, the study reported on the extent to which change occurred during rehabilitation and the proportion of patients who met or exceeded the predicted discharge scores.¹¹

In January 2009, following confirmation of ethical approval for the present study, additional information was extracted from the same patients' charts by staff from the hospital's medical records department and added to the database by the current investigative team. The newly retrieved information consisted of date of stroke, date of admission to and discharge from the rehabilitation unit, and scores on the 15 individual tasks of the Activity Inventory.

For the current study, patients were excluded from the data analysis if they were admitted to the rehabilitation unit more than 45 days post stroke, if their data were incomplete, or if they had experienced a stroke event other than a unilateral stroke. These criteria were chosen in an effort to achieve a more homogeneous sample.

The Chedoke–McMaster Stroke Assessment

The CMSA comprises two components: an Impairment Inventory (II), which is used to assess motor control across six dimensions, namely the arm, the hand, the leg, the foot, postural control, and shoulder pain; and an Activity Inventory (AI), which assesses functional mobility, including gross motor function and walking.⁸ The II quantifies the degree of motor control according to seven stages across the six dimensions. Scores are determined by the quality of movement; a score of 1 indicates severe motor impairment, and a score of 7 indicates normal movement. The AI consists of two indices: the Gross Motor Function Index (GMFI) measures functional mobility across 10 items, including moving in bed, transferring in and out of bed, and getting on and off the floor; and the Walking Index (WI) assesses the patient's ability to walk on smooth surfaces and rough terrain and to climb stairs across five items. Fourteen of the 15 AI items are scored from 1 (needs maximal assistance) to 7 (completely independent), while the fifteenth item, a 2-minute walk test,¹⁹ is scored relative to age-specific walking speed, either 0 (≤70 years of age: walking 96 m) or 2 (>70 years of age: walking 84 m). The maximum total score of the AI is 100; higher scores indicate greater functional independence. The CMSA manual describes the administration and scoring guidelines for the measure.^18,20 Training for use of the CMSA includes a standardized 1-day training workshop,²¹ videoconferencing,²² and use of a bilingual CD-ROM for self-directed learning.²³

Using the predictive equations and applying the patient's CMSA scores at admission, predicted discharge scores can be determined for each II stage and for the total AI and each AI index. Table 1 provides Gowland's predictive equations, which we applied to patient scores in the study database.¹⁸

Table 1.

Chedoke–McMaster Stroke Assessment Predictive Equations for Clients Undergoing Rehabilitation¹⁸

Outcome Variable	R2	Predictive Equation
Total AI	0.73	17.45+(0.88×Gross Motor Function)+(4.30×Leg)
GMFI	0.70	24.94+(0.76×Gross Motor Function)−(0.30×Weeks)
WI	0.71	(0.28×Gross Motor Function)+(1.23×[Postural Control+Leg])*−4.55
II
Shoulder Pain	0.55	2.33+(0.44×Shoulder Pain)+(0.28×Arm)
Postural Control	0.60	2.23+(0.35×Postural Control)+(0.30×Leg)
Arm	0.80	0.82+(1.03×Arm)−(0.03×Weeks)
Hand	0.78	0.53+(0.98×Hand)
Leg	0.69	1.83+(0.77×Leg)−(0.02×Weeks)
Foot	0.73	1.11+(0.90×Foot)−(0.03×Weeks)

AI=Activity Inventory; GMFI=Gross Motor Function Index; WI=Walking Index; II=Impairment Inventory

^*Sum of scores for the stages of Postural Control and Leg

Data Analysis

We calculated means, standard deviations (SD), and quartiles for the CMSA scores (II dimensions, GMFI, WI, total AI). Next, we applied Gowland's predictive equations to patients' admission scores, as reported in the database, to obtain predicted discharge score estimates for the six II dimensions and for the AI and its two indices.¹⁸ For each item, the predicted estimate (independent variable) was regressed against the discharge score reported in the database (dependent variable) using a linear model. We also calculated the percentage of variance explained by the model ( $R_0^2$) and the residual or error variance ( $S_{y/x}^2$).

A predictive model is reliable to the extent that it generalizes to samples other than the one used to create it.²⁴ To examine the extent to which Gowland's models predicted item scores for our sample, we calculated the shrinkage as $R_R^2-R_0^2$, where $R_R^2$ is the squared correlation coefficient reported by Gowland and $R_0^2$ is the corresponding coefficient when Gowland's predictive equations were applied to our data.²⁴ Kleinbaum et al. have stated that shrinkage values <0.10 indicate a reliable model.²⁴ To help ascertain the clinical usefulness of the predictive equations, we calculated 95% prediction bands for each item's predicted values, as follows:

$${\hat{Y}}_0\pm t_{n-2,1-\alpha /2}{S}_{y/x}\sqrt{1+\frac{1}{n}+\frac{{(X_0-\overline{X})}^2}{(n-1)S_x^2}}$$

where ${\hat{Y}}_0$ is the predicted value for the corresponding X₀ obtained by applying the Gowland predictive model, n is the number of patients contributing data, t is the critical value from the t-distribution on n−2 degrees of freedom (corresponding to a 2-tailed α value of 0.05), S_y/x is the square root of the error variance, $\overline{X}$ is the mean predicted value for the item, and $S_x^2$ is the variance of the predicted values.²⁴ The value obtained from this analysis conveys the amount of uncertainty associated with a predicted value for an individual in the units of the original measurement. The amount of uncertainty depends on the X value and is smallest when $X_0=\overline{X}$.

The final aspect of our analysis was to consider whether a linear model—as assumed by Gowland—provided the best fit for the predictive models. This was accomplished in two steps. First we examined scatter plots of the actual discharge (dependent variable) and admission (independent variable) data with the line of best fit superimposed on the graph. When a curvilinear relationship was evident, we examined higher-order polynomials and tested whether the addition of higher-order terms contributed to the predictive ability of the revised model.

Our sample size was one of convenience, determined by the number of eligible patients in our database.

RESULTS

Subjects

Of the 104 patients admitted to the rehabilitation unit between July 1996 and July 1998 who constituted the initial database, 30 were excluded from the present study owing to missing data, because they had had other than a unilateral stroke, or because they were admitted to the rehabilitation unit more than 45 days post stroke.

The other 74 patients were included in the data analysis. Patients had a mean (SD) age of 65.3 (12.4) years. Of these patients, 26 (35%) were female. The five most prevalent comorbidities were hypertension, diabetes mellitus (type I or type II), hyperlipidemia, musculoskeletal disorders, and a previous stroke or transient ischemic attack (TIA). A summary of patient characteristics is given in Table 2.

Table 2.

Patient Characteristics (n=74)

Characteristic	Summary Value
Age	65.3 (12.4)
mean (SD), years
Gender (F/M)	26/48
Days post stroke	15.6 (8.6)
mean (SD)
LOS	44.8 (24.4)
mean (SD)
Type of stroke
Left CVA	41
Right CVA	33

LOS=length of stay; CVA=cerebro-vascular accident

The mean, standard deviation, and quartiles for CMSA admission, discharge, and prediction scores for the study population are provided in Table 3.

Table 3.

Summary Statistics for Chedoke–McMaster Stroke Assessment Values

Outcome Variable	Admission	Discharge	Predicted
	Mean (SD) Quartiles	Mean (SD) Quartiles	Mean (SD) Quartiles
II dimensions
Hand	3.8 (2.1)	4.6 (2.1)	4.2 (2.0)
	2, 4, 6	3, 6, 6	2.5, 4.5, 6.4
Arm	3.7 (2.1)	4.6 (2.1)	4.6 (2.1)
	2, 4, 6	3, 5, 6	2.8, 4.8, 6.9
Leg	4.3 (1.7)	5.2 (1.5)	5.1 (1.3)
	3, 5, 6	4, 6, 6	4.1, 5.6, 6.4
Foot	3.6 (1.8)	4.4 (1.9)	4.3 (1.7)
	2, 4, 5	3, 5, 6	2.8, 4.6, 5.5
Shoulder pain	5.9 (1.2)	5.8 (1.6)	6.0 (1.0)
	6, 6, 7	5, 7, 7	5.2, 6.1, 6.8
Postural control	4.3 (1.1)	5.1 (1.1)	5.0 (0.8)
	4, 4, 5	5, 5, 6	4.5, 5.1, 5.8
AI	56.2 (18.1)	76.4 (17.6)	77.9 (18.7)
	43, 55, 71	65, 79, 90	63.5, 80.4, 92.9
GMFI	47.2 (14.1)	59.4 (10.7)	60.2 (10.8)
	37, 47, 58	52, 62, 69	52.7, 60.4, 68.8
WI	8.7 (4.9)	17.1 (7.2)	19.2 (6.8)
	4, 8, 12	13, 15, 22	13.8, 19.8, 24.7

II=Impairment Inventory; GMFI=Gross Motor Function Index; WI=Walking Index; AI=Activity Inventory

Predictive Ability

Table 4 summarizes the accuracy of the predictive equations. Reported in this table are the shrinkage values and three 95% prediction-band intervals (lowest, middle, and highest scale X values) for each CMSA II dimension and for the AI and its indices. Shrinkage values varied from −0.05 to 0.09 for the II dimensions; shrinkage values for the GMFI, WI, and AI were 0.19, 0.24, and 0.21 respectively.

Table 4.

Reliability and Accuracy Summary for Predictive Equations

Outcome Variable	Reported $$R_R^2$$	Obtained $$R_0^2$$	Shrinkage ( $$R_R^2-R_0^2$$ )	Sy/x*	95% Prediction Bands for Predicted Values of**
					1	4	7
Hand	0.78	0.76	0.02	1.06	±2.16	±2.13	±2.15
Arm	0.80	0.79	0.01	0.96	±1.96	±1.93	±1.94
Shoulder pain	0.55	0.49	0.06	1.13	±2.64	±2.33	±2.23
Leg	0.69	0.74	−0.05	0.75	±1.60	±1.51	±1.53
Foot	0.73	0.78	−0.05	0.91	±1.87	±1.83	±1.86
Postural control	0.60	0.51	0.09	0.77	±1.78	±1.56	±1.60
					1	35	70
GMFI	0.70	0.51	0.19	7.49	±17.83	±15.57	±15.11
					1	15	30
WI	0.71	0.47	0.24	5.30	±11.1	±10.66	±10.81
					10	50	90
Total AI	0.73	0.52	0.21	11.79	±25.73	±24.04	±23.73

GMFI=Gross Motor Function Index; WI=Walking Index; AI=Activity Inventory

^*S_y/x is the square root of the residual variance.

^**These values represent the lowest, middle, and highest possible scores.

For the II, the error associated with predictive values amounts to approximately ±2 stages for each dimension; more error is associated with end-range scores.

Figure 1 displays a scatter plot of the actual and predicted discharge values for the Hand Impairment item. This figure also shows the line of best fit and the 95% prediction bands.

Figure 1.

Scatter plot of actual versus predicted CMSA Hand Impairment values, including 95% prediction bands (shaded area)

Model Evaluation

Our lack-of-fit analysis revealed that a second-degree polynomial provided better predictive models for CMSA hand and arm impairment data. The additional predictive ability of models that included a squared term were as follows: R² hand=0.77, R² hand+hand²=0.81 (F(1,82) hand²|hand=19.80, p<0.001); R² arm=0.79, R² arm+arm²=0.83 (F(1,82) arm²|arm=25.40, p<0.001). Figure 2 is a graphical representation of the curvilinear relationship between actual admission and discharge hand-impairment data.

Figure 2.

Actual CMSA hand-impairment values at admission and discharge, including 95% prediction bands (shaded area), for the second-degree polynomial model

DISCUSSION

The goal of our study was to estimate the predictive accuracy and clinical usefulness of reported CMSA equations. To meet this objective, we applied statistical procedures to examine the relationships between the actual results in our sample of patients undergoing rehabilitation following stroke and the outcomes predicted using Gowland's predictive equations. The shrinkage coefficients for the stages of the II were all less than 0.10, which suggests that the predictive models proposed by Gowland¹⁸ are supported by our data. In contrast, the shrinkage values for the total AI score and for the GMFI and WI scores greatly exceeded the critical value of 0.10, indicating that the predictive models proposed by Gowland are not supported by our data. One possible explanation for the large shrinkage value is that there were differences between the two samples of patients (e.g., the timing of the post-stroke assessments); it is also possible that differences in physiotherapy management contributed to differences in outcomes between Gowland's patients and those in the database used for the present study. Unfortunately, information on the patients in the sample used by Gowland to generate the predictive equations was not available to explore these potential explanations.

While the results of this analysis suggest that the shrinkage for the II scores is acceptable, we further considered the confidence bands of the predicted outcomes to gain additional information. The clinical usefulness of the predictive equations is directly related to the width of the prediction bands: the narrower the width, the greater the confidence in predicting an individual patient's discharge score. The width of prediction bands depends on the variability in the data, the mean score, and the distance of a particular value from the mean score: the greater the distance from the mean score, the greater the width of the prediction band. Table 3 reports the width of the 95% prediction bands for three impairment values (stages 1, 4, and 7); for most impairment values, the 95% prediction bands are approximately ±2 stages. This width is likely too large to provide a confident prediction of a patient's outcome. We offer the following clinical vignette to assist in the interpretation of our findings.

Clinical Vignette

A 64-year-old man was admitted to the rehabilitation unit after a right cerebral vascular accident (CVA) 16 days ago. The physiotherapist assesses the patient using the CMSA within the first week of admission. The CMSA stage for the hand at admission is 3. The physiotherapist calculates expected discharge scores using Gowland's predictive equations. The predicted discharge score for the hand is stage 3.5. By applying the 95% prediction bands (3.5±2.13) from the current study (see Table 3), the physiotherapist finds that the patient's hand stage at discharge could fall anywhere between 1 (i.e., 1.37) and 6 (i.e., 5.63). Stage 1 indicates that the hand will be flaccid and that there will be no evidence of any reflexive or active movement, while Stage 6 indicates that coordination and patterns of movement for the hand will be near normal.¹⁸ The range of the predicted score of the stage of the arm at discharge (5 stages in total) is too large to provide useful information to the clinician.

We also considered whether a more complex model (in this case, higher-degree polynomial) might provide a better fit than the models proposed by Gowland et al.¹⁸ We found that for the measures of hand and arm impairment, a second-degree polynomial fit our data better statistically; however, confidence in the predicted scores, as represented by the 95% prediction bands, did not improve appreciably. This finding is illustrated for the hand in Figure 2, in which the prediction band continues to be approximately ±2 stages wide.

We wish to stress that our study focused on the predictive ability of the CMSA and not on its properties in assessing patient outcomes. Proficient outcome measures must display high levels of discrimination and the ability to detect change, and previous investigations have provided support for these measurement properties as they apply to the CMSA when applied to patients with stroke.^8–11,18 The role of the CMSA as a valuable discriminative and evaluative measure is not altered by the findings of the present study.

LIMITATIONS

Our study has several limitations, many of which arise from the use of a historical database. Despite efforts to achieve a large sample size, the database was missing data for a number of patients, who were therefore excluded from the analysis. Because there was no standard protocol for data collection, patients were assessed at different time points (including time post stroke and length of time between initial and discharge assessment), producing a non-homogenous sample. In addition, at the time of assessment the patients' CMSA scores were recorded on paper score sheets, which were stored in the patient files; for the purpose of the current study, these scores were then recorded on a new data-collection sheet by staff in medical records and subsequently added to the database by members of our research team. The transferring of data on several occasions by different individuals may have resulted in errors in the database used for the study.

Furthermore, we have limited information about the graduate and student physiotherapists who collected data for the original study. For example, there are no data on their experience in working with patients with stroke or on their formal training (or lack thereof) in administering and scoring the CMSA. The accuracy of the data they collected may also have influenced our results.

The results of this study suggest that statistical procedures other than linear regression can be considered to develop new and more accurate prediction models. Future research to develop predictive equations should involve a prospective study with a larger sample. Methodological considerations would include specifying assessment times related to time post stroke, rather than using admission and discharge dates, to increase accuracy of predictions.²⁵ The collection of more comprehensive demographic information about the patients in the sample and about the persons who gather the data, as well as a description of the care received during rehabilitation, would increase the generalizability of the results.

CONCLUSION

For approximately 20 years, clinicians in rehabilitation settings have been using the CMSA predictive equations developed by Gowland to enhance clinical decision making.¹⁸ The results of this study indicate that the shrinkage values for the AI models are too large for the models to be considered reliable. Furthermore, the confidence bands associated with both II and AI scores are too large to be considered clinically useful. Future research to identify models that rehabilitation professionals can use in predicting outcomes for individual clients with stroke would facilitate evidence-based practice. The importance of the CMSA's role as a discriminative and evaluative measure for persons with stroke remains unchanged.

KEY MESSAGES

What Is Already Known on This Topic

The CMSA is a highly valid and reliable measure to discriminate and detect change in persons post stroke. There has been no research to evaluate the accuracy of the predictive equations developed for use with clients undergoing rehabilitation for stroke.

What This Study Adds

The results of this study call into question the clinical usefulness of the predictive equations developed for stroke rehabilitation. Future research using alternative predictive models is warranted.

Dang M, Ramsaran KD, Street ME, Syed SN, Barclay-Goddard R, Stratford P, Miller PA. Estimating the accuracy of the Chedoke–McMaster Stroke Assessment predictive equations for stroke rehabilitation. Physiother Can. 2011;preprint. doi:10.3138/ptc.2010-17