Manual Wheelchair Skills: Objective Testing versus Subjective Questionnaire

Abstract

Objectives

To test the hypothesis that the total scores of the Wheelchair Skills Test (WST) version 4.1, an observer rated scale of wheelchair performance, and the Wheelchair Skills Test Questionnaire (WST-Q) version 4.1, a self-report of wheelchair skills, are highly correlated. We also anticipate the WST-Q will be slightly higher indicating an overestimation of capacity to perform wheelchair skills, as compared to actual capacity.

Design

A cross-sectional, within-subjects comparison design.

Participants

Convenience sample of 89 community-dwelling, experienced manual wheelchair users ranging in age from 21–94 years.

Setting

Three Canadian cities.

Intervention

Not applicable.

Main Outcome Measures

Participants completed the subjective WST-Q version 4.1, followed by the objective WST version 4.1 in one testing session.

Results

The mean ± SD total percentage scores for WST and WST-Q were 79.5% ±14.4 and 83.0% ±12.1 for capacity and 99.4% ±1.5 and 98.9% ±2.5 for safety. The correlations between the WST and WST-Q scores were ρ=0.89 (p=0.000) for capacity and ρ=0.12 (p=0.251) for safety. WST-Q total score mean differences were an average of 3.5%±6.5 higher than WST scores for capacity (p = 0.000) and 0.52%±2.8 lower for safety (p = 0.343). For the 32 individual skills, the percentage agreement between the WST and WST-Q scores ranged from 82–100% for capacity and 90–100% for safety.

Conclusion

WST and WST-Q version 4.1 capacity scores are highly correlated although the WST-Q scores are slightly higher. Decisions on which of these assessments to use can safely be based on the circumstances and objectives of the evaluation.

The World Health Organization has recognized wheelchair-skills assessment and training as important elements of the wheelchair-provision process.¹ Recent systematic reviews have evaluated the assessment of wheelchair skills.^2,3 These reviews ranked the Wheelchair Skills Test (WST)⁴ highly. The WST is an objective test of a set of skills and a questionnaire version (the WST-Q) is also available. Previous research has documented the psychometric properties of the WST ^5–8 and a number of studies have used it as an outcome measure.^9–15 Previous investigations have also examined the WST-Q.^{16, 17}

The objective, performance-based WST and the subjective, self-report WST-Q each have merits and limitations. The main advantage of the WST is that the tester can see how the wheelchair user carries out the skill, permitting problems due to the wheelchair or the technique used to be identified and addressed. However, the WST requires time to perform (~30 minutes), space and equipment, or the availability of the environmental obstacles in the wheelchair users’ home or community environments. By virtue of providing an opportunity to attempt skills, it can cause an unintended training effect that can be seen at subsequent assessments. As well, it captures an individual’s ability to perform wheelchair skills on a particular day at a particular time, which may be influenced by a variety of factors, such as fatigue and anxiety.¹⁷ Measurement of wheelchair skills in this manner represents an individual’s capacity to perform the skill, or what the individual can do.¹⁸

The WST-Q, on the other hand, can be administered in any location. It enables assessment of wheelchair skills in situations where it is not feasible to assess them objectively, such as when the wheelchair user is confined to bed. The WST-Q can also be used as a screening assessment, for example in a busy outpatient clinic to determine the need for referral for wheelchair skills training. From a research perspective, the WST-Q can be used when objective testing is not feasible due to a lack of human or space and equipment resources, or to reduce participant burden. Depending on how the questions are posed, the WST-Q can be used as a measure of performance (what the individual does do) or capacity (what the subject can do).¹⁸ The WST-Q is advantageous in that it requires less time (only ~10 minutes), expense, equipment, and space to administer. However, there are drawbacks to using the WST-Q. For instance, accuracy of responses can be influenced by cognitive or communication deficits. As well, describing rather than executing a task removes the environmental and motor-memory cues that may be relied upon to perform the task.¹⁹

In an exploratory study designed to determine if objective testing of wheelchair skills is necessary or if subjective estimates would suffice, Newton et al¹⁷ found a high correlation between the WST version 2.4 and 21 wheelchair users’ subjective perceptions of their ability to perform wheelchair skills (ρ=0.95), but wheelchair users overestimated their abilities by an average of 18% (P=0.0002). Participants in this study were asked to respond categorically whether they could or could not perform each skill. Mountain et al¹⁶ used a semi-structured interview for WST-Q version 2.4 in a similar study of 20 wheelchair users. The interviewer probed beyond the initial responses to better determine if the evaluation criteria were met. They found a positive correlation between the objective and questionnaire versions (r=0.91), but the wheelchair users only overestimated their objective capacities by 7%. No such comparison has been carried out for the latest version of the WST (Version 4.1), which includes scores for both capacity and safety.

The primary objectives of this study were to test the hypotheses that the total scores of the WST and WST-Q, version 4.1, are highly correlated, but that the WST-Q scores are slightly higher, indicating an overestimation of capacity to perform the wheelchair skills, as compared to actual capacity. Our secondary objective was to assess the agreement for individual test items (skills) between the WST-Q and the WST.

METHODS

Study Design

This study used a cross-sectional, within-subjects comparison design. The data were collected as part of a larger study on wheelchair confidence that has been reported elsewhere.²⁰

Participants

We studied 89 manual wheelchair users, a sample of convenience. This sample size was deemed necessary for the original study.²⁰

Recruitment and Screening

Participants were recruited from 3 Canadian cities: Vancouver, British Columbia; Hamilton, Ontario; and Halifax, Nova Scotia. Letters of information were provided to potential participants on a rehabilitation research volunteer database and to those who attended rehabilitation services at local rehabilitation centers and a university-based rehabilitation gymnasium. Advertisements were posted in facilities that manual wheelchair users frequent (e.g., fitness facilities, community centers), as well as online via e-bulletins and community organization websites, such as the Canadian Paraplegic Association. To be included in the study, participants had to be at least 19 years of age, use a manual wheelchair as their primary means of mobility (at least 4 hours per day), have at least 6 months of manual wheelchair experience, and live in the community. Participants were excluded if they were unable to read and write in English. Folstein’s Mini Mental State Exam (MMSE) was used to aid in explaining outliers using a cut-off score of ≤ 24 to indicate cognitive impairment.²¹

Ethical Issues

Ethical approval was obtained by local university or hospital research ethics boards. Each participant provided informed consent.

Wheelchair Skills Test, Version 4.1

We used the WST and WST-Q, version 4.1, for manual wheelchair users in this study.⁴ Both tests evaluate 32 wheelchair skills (listed later) ranging in difficulty from wheeling forward for 10m to negotiating stairs. The tests were administered according to the WST 4.1 Manual.⁴ All research assistants were trained by the primary investigator (PWR). A spotter strap was used during the WST to prevent acute injury to the wheelchair user while performing skills for which there was a known risk of tips or falls from the wheelchair. The WST-Q 4.1 was administered using a semi-structured interview, whereby the individual was asked whether he/she believed him/herself to be capable of performing each wheelchair skill and, if so, how he/she would perform the skill. In this study, the WST-Q questions were posed to assess capacity (rather than performance),¹⁸ because capacity is what is assessed by the WST. For both the WST and WST-Q, dichotomous response formats were used to score the capacity (pass/fail) and safety (safe/unsafe) of each wheelchair skill using explicit criteria.⁴ Percentage scores were calculated (number of passed or safe skills/number of possible skills X 100%).

Procedure

After recruitment, screening, and informed consent, a demographic questionnaire was completed to determine the demographic, clinical, and wheelchair-use characteristics of the sample. Administration of the WST-Q and WST was completed in one testing session. The WST-Q preceded the WST to ensure that subjective perception of wheelchair skill was not influenced by objective measurement. Participants used their own manual wheelchairs.

Data Analysis

We used the Statistical Package for the Social Sciences (SPSS) version 16.0 for the statistical analysis.^a Descriptive statistics and total percentages were calculated for the total WST and WST-Q capacity and safety scores. Normal distribution of the data was tested with the Kolmogorov-Smirnov test and Bland-Altman limits-of-agreement plots were used to identify bias and outliers. A Spearman correlation coefficient was used for the total WST and WST-Q capacity and safety scores. The paired Wilcoxon Signed Rank test was used to assess the extent of differences between the total WST and WST-Q scores. For the statistical analyses, we used an α level of 0.05. We compared individual wheelchair skills between the WST and WST-Q performance and safety scores using percentage agreements. A percentage agreement of 75% was defined as clinically significant.^{16, 17}

RESULTS

The participants’ demographic, clinical and wheelchair-use characteristics are presented in Table 1. The sample was composed mostly of men who had a mean age of 50.5±14.7 years and who ranged in wheelchair experience from 0.5 to 53.0 years. The primary diagnosis was spinal cord injury.

Table 1.

Participants’ demographic, clinical, and wheelchair-use characteristics.

Characteristic	Value
Age in years, mean (SD)	50.5 (14.7)
Men (%)	68.5
Marital Status (%)
Married/Common-law	39.3
Single	36.0
Separate/Divorced/Widowed	24.7
Education (%)
Less than high school	4.5
High school	38.2
Professional Diploma	27.0
Bachelor degree	23.6
Master degree	5.6
PhD	1.1
Diagnosis (%)
Spinal cord injury	60.7
Lower limb amputation	10.1
Multiple sclerosis	9.0
Stroke	4.5
Other	15.7
Years with diagnosis
Mean (±SD)	20.2 (±14.3)
Median	18.0
Range	0.5–58.0
Years using wheelchair:
Mean (±SD)	16.4 (±13.2)
Median	13.0
Range	0.5–53.0
Method of propulsion (%)
Two hands	91
Two feet	1.1
One hand, one foot	5.6
Two hands, two feet	2.2

Abbreviation: SD, standard deviation.

The mean ± standard deviation (SD), median, and range for the WST and WST-Q capacity and safety total percentage scores are presented in Table 2. Both the WST and WST-Q capacity and safety scores demonstrated negatively skewed distributions, which were confirmed by Kolmogorov-Smirnov tests (p<0.05). Bland-Altman plots (Figures 1 and 2) showed a fairly equal distribution of values above and below the mean difference illustrating an unsystematic variability in scores between both the WST and WST-Q capacity and safety scores. A ceiling effect was found for both capacity and safety. For the capacity scores, there were six outliers, so nearly 95% of the capacity differences fell between ±2 SD. For the safety scores, five outliers were identified, so 95% of the safety differences fell between ±2 SD. Using the MMSE scores, it was determined that none of the outliers had cognitive impairment.

Table 2.

Comparison of the Total Percentage Scores for the WST and WST-Q Capacity and Safety Scores.

Outcome Measure		Mean	Standard Deviation	Median	Range
WST	Capacity (n=89)	79.5	14.4	81.3	22–100
	Safety (n=88)	99.4	1.5	100	91–100
WST-Q	Capacity (n=89)	83.0	12.1	84.0	47–100
	Safety (n=89)	98.9	2.5	100	87–100

Figure 1. Bland-Altman Plot of the Mean Versus the Difference in WST and WST-Q 4.1 Capacity scores.

This figure shows a comparison of the WST and WST-Q 4.1. The mean difference between the WST and WST-Q was 3.5 with an upper limit of agreement of 16.5 (mean difference + 2 standard deviations) and a lower limit of −9.5 (mean difference − 2 standard deviations). There were six outliers, so nearly 95% of the capacity differences fell between the 2 SD.

Figure 2. Bland-Altman Plot of the Mean Versus the Difference in WST and WST-Q 4.1 Safety scores.

This figure shows a comparison of the WST and WST-Q 4.1. The mean difference between the WST and WST-Q was −0.5 with an upper limit of agreement of 5.1 (mean difference + 2 standard deviations) and a lower limit of −6.1 (mean difference − 2 standard deviations). Five outliers were identified for the safety scores, so 95% of the safety differences fell between 2 SD.

The correlation between the WST and WST-Q capacity scores was ρ=0.89 (P=0.000), while the correlation for safety was ρ= 0.12 (P=0.251). Figure 3 illustrates a scatter plot of the WST and WST-Q total percent capacity scores. Thirty out of 89 participants had the same WST and WST-Q capacity score. For the safety scores, the data violated the assumption of linearity. Sixty-three of the 88 participants had the same WST and WST-Q safety score.

Figure 3. Total percentage scores for the WST and WST-Q 4.1 Capacity Scores.

Note: All data points (n=89) are not visible due to 30 of the WST-Q and WST capacity scores being equal, resulting in many data points presenting as overlapping dots.

The mean differences ± SD for the total WST and WST-Q capacity and safety scores were 3.5%±6.5 (WST-Q higher) (P = 0.000) and 0.52%±2.8 (WST-Q lower) (P = 0.343). The median differences for the WST and WST-Q capacity and safety scores were 3.0% and 0.0%.

The data on individual wheelchair skills are shown in Table 3. For the WST-Q, data were missing for one subject for all safety items. Otherwise, the reason for the sample size being less than n=89 or n=88 for certain skills was often a result of the WST scoring instructions. For example, for skills that have pre-requisites, if the pre-requisite for a skill had been failed (e.g., 5cm level change), the skill under consideration (e.g., 15cm level change) was not tested. In such a case, the participant received a ‘fail’ score for capacity and a ‘not tested’ score for safety. Another reason for the sample size being less than n=89 or n=88 involved the tester being unable to test certain skills. For example, at the Vancouver site there was not an alternate access to the top of the stairs. Therefore, if the participant was unable or unwilling to ascend the stairs the tester was not able to test the descend stairs skill.

Table 3 . Percentage Agreement between the WST and WST-Q and Percent Success Rate for Capacity and Safety Scores for Individual Wheelchair Skills.

Skill	N				% Agreement between WST and WST-Q		WST % Success Rate		WST-Q % Success Rate
	WST		WST-Q
	C	S	C	S	C	S	C	S	C	S
Rolls forward 10m	89	88	89	89	100	100	100	100	100	100
Rolls forward 10m in 30s	89	88	89	89	95.5	100	100	100	95.5	100
Rolls backward 5m	89	88	89	89	98.9	100	100	100	98.9	100
Turns 90° while moving forward L&R	89	88	89	89	100	100	100	100	100	100
Turns 90° while moving backward L&R	89	88	89	89	96.6	100	98.9	100	97.8	100
Turns 180° in place L&R	89	88	89	89	98.9	100	100	100	98.9	100
Maneuvers sideways L&R	89	88	89	89	95.5	100	95.5	100	97.8	100
Gets through hinged door in both directions	89	88	89	89	96.6	100	96.6	100	100	100

The percentage agreement between the WST and WST-Q scores for the individual skills ranged from 82–100% for capacity and 90–100% for safety. All skills met the threshold of 75% or greater for a clinically significant agreement. Generally, the skills with the lowest percentage success rates were the more difficult community- and advanced-level skills.

DISCUSSION

We met our objectives of determining the relationship between the WST and WST-Q version 4.1. Our hypothesis that the WST and WST-Q scores would be highly correlated was confirmed for the capacity scores (ρ=0.89), but not for the safety scores (ρ=0.12). As hypothesized, WST-Q total scores for capacity were slightly, but significantly, higher than WST scores, but the safety score difference was not significant.

The high correlation between the total WST and WST-Q capacity scores is similar to that found by Mountain et al (r=0.91)¹⁶ and Newton et al (ρ=0.95).¹⁷ The non-linear relationship between the WST and WST-Q safety scores can be explained by the very high concordance between the scores with 63 of the 88 participants (34.1%) having the same total score for the WST and WST-Q. This lack of dispersion of values resulted in the non-linear relationship and low correlation coefficient. Therefore, it is important to note that, although the correlation coefficient is low, it is as a result of the very high degree of correspondence between the WST and WST-Q safety scores and is not indicative of a poor relationship between the two outcome measures.

The wheelchair users’ overestimation of their capacity to perform wheelchair skills (3.5%) was less than that found by Mountain et al (7%)¹⁶ and Newton et al (18%).¹⁷ The results of the 3 studies were similar, however, in that it was generally the more difficult, community- and advanced-level skills that were overestimated, such as negotiating ramps, potholes, and curbs. The overestimation of capacity to perform skills may have been less in this study as the wheelchair users had more wheelchair experience (16.4 ±13.2 years) when compared to the participants in the Mountain study (1.8 ±4.6 years)¹⁶ or the Newton study (4.9 ±10.1 years).¹⁷ Participants in this study, therefore, may have been better able to judge their ability to perform the skills. Of significance is the wheelchair users’ minimal (0.5%) underestimation of their safety in performing wheelchair skills. This result indicates that, for the most part, the wheelchair users in this study were able to accurately judge their safety in performing the wheelchair skills. This judgement is important because inaccurate judgements in safety may result in serious injury. The significance of this point is emphasized by the lower capacity and safety success rates in the wheelchair skills at the more difficult end of the spectrum.

When using WST-Q results to determine whether or not a wheelchair user requires further skills training, it is important to recognize that some wheelchair users overestimate their ability to perform the more difficult wheelchair skills. Both types of wheelchair skill assessment can yield important information and have potential roles in clinical and research settings. As discussed earlier, the WST and WST-Q each have merits and limitations that need to be considered when selecting a test for clinical or research purposes. The high correlation between the WST and WST-Q capacity scores, the high concordance between the WST and WST-Q total capacity and safety scores, as well as the low percentage of overestimation of capacity and underestimation of safety to perform the wheelchair skills supports the use of the either the WST or the WST-Q when administration of one is preferable over the other.

These results are in contrast to findings from some other studies comparing objective and subjective measures. For example, Gandhi et al²² found a low correlation between preoperative objective Timed-Up-and-Go and the subjective SF-36 scores among patients undergoing hip and knee arthroplasty. Wand et al²³ found a low correlation between the subjective Roland and Morris Disability Questionnaire (RMDQ) and the objective Timed 5 Minute Walk Test and moderate correlations between the RMDQ and the objective Timed Sit to Stand, Timed Up-and-Go, and Timed Lying to Stand Test among a sample of adults with acute low back pain. Suchy et al²⁴ reported that 38% of participants in a sample of independent and healthy community dwelling older adults exhibited a discrepancy between instrumental activities of daily living (IADL) self-report and IADL performance.

Increasingly, it has been reported that subjective, self-report measures are influenced by psychological status (e.g. confidence, anxiety, and depression) to a greater degree than objective, performance-based measures.^22–24 However, our study showed little difference between the WST and WST-Q scores. We know that this sample of manual wheelchair users had high wheelchair confidence and low anxiety and depression.²⁰ Confidence is an important predictor of future behavior despite skill level²⁵ and depression and anxiety have been associated with wheelchair use.^26,27 Therefore, the high level of wheelchair confidence and low level of anxiety and depression in this group of manual wheelchair users may, in part, explain the high correlation between the WST and WST-Q capacity scores. Although it is tempting to assume that objective assessment is more valid than subjective measures, a wheelchair user could perform poorly in the formal test setting due to anxiety, day-to-day variation or by narrowly failing to meet an evaluation criterion.

Study Limitations

There were limitations to this study, some of which have already been noted. The sample was an experienced group of manual wheelchair users. The clustering of scores at the higher score end of the WST and WST-Q demonstrated by the Bland-Altman plot provides some evidence of that. Because our sample did not include new manual wheelchair users, the results should be generalized with caution. Another limitation was that the same tester administered both the WST and WST-Q, but doing the WST-Q first would have the effect of minimizing any resulting bias. The ceiling effect could be overcome by selectively recruiting a participant population that is more diverse in their abilities and safety. Our sample was also literate and cognitively intact (MMSE scores of ≥24), so these results should not be generalized to wheelchair users with greater language or cognitive limitations. Also, we confined ourselves to the assessment of capacity rather than performance, the other important dimension about which the WST-Q can provide insight. Inkpen et al²⁸ have shown a strong correlation between capacity and performance scores on the WST-Q.

Future work is needed to address the study limitations. As well, this work should be extended to power wheelchair users and caregivers to allow us to determine whether the WST and WST-Q can be used interchangeably with these populations.

CONCLUSIONS

WST and WST-Q version 4.1 scores are highly correlated although the WST-Q scores are slightly higher for capacity. Decisions on which of these assessments to use can safely be based on the circumstances and objectives of the evaluation.

LIST OF ABBREVIATIONS

WST

Wheelchair Skills Test

WST-Q

Wheelchair Skills Test Questionnaire