Self-Report Versus Performance Measure in Gauging Level of Function with Multiple Sclerosis

Abstract

Background

Multiple sclerosis (MS) is a debilitating, progressive disease with no known cure. Symptoms vary widely for persons with MS and measuring levels of fine motor, gross motor and cognitive function is a large part of assessing disease progression in both clinical and research settings. While self-report measures of function have advantages in cost and ease of administration, questions remain about the accuracy of such measures and the relationship of self-reports of functioning to performance measures of function.

Objective

The purpose of this study was to compare scores on a self-report measure of functional limitations with MS with a performance-based measure at five different time points.

Methods

Sixty participants in an ongoing longitudinal study completed two measures of function annually over a five-year period - the self-report Incapacity Status Scale and the MS Functional Composite (MSFC), a performance test. Pearson correlations were used to explore the association of self-report and performance scores.

Results

There were moderate to strong correlations among the ISS total (r= −.53 to −.63, p<.01) and subscale scores of gross (r=.79 to .87; p<.01)) and fine (r= .47 to .69; p<.01) motor function and the corresponding MSFC performance measure. The pattern of change over time in most scores on self-report and performance measures was similar.

Conclusion

Findings suggest that the self-report measure examined here, which has advantages in terms of feasibility of administration and patient burden, does relate to performance measurement, particularly in the area of gross motor function, but it may not adequately reflect cognitive function.

Introduction

Multiple sclerosis (MS) is a neurological disease in which the myelin sheaths covering neurons in the brain and spinal cord are destroyed [1,2]. Onset usually occurs in young adulthood. There is no prevention or cure and the focus of care for people with MS is promoting health, autonomy and function as much as possible. MS progression disrupts nerve signaling, leading to cognitive and physical dysfunction, and occurs unpredictably, in terms of timeline and decrements of deterioration. While there are four broad patterns of progression that can be identified retrospectively over time [3], it remains challenging to predict or track clinically significant changes in levels of function to help people with MS and their health care providers to anticipate and manage symptoms.

There is some question among researchers about the optimal way to measure the progressive functional impact of MS. Performance measures have often been considered the preferred method to gauge function. One particular measure, the Expanded Disability Status Scale (EDSS) [4], is a scale against which other performance measures are often compared. The EDSS is a measurement performed by a neurologist of eight functional systems using a 10-point rating scale, with 0 representing no disability and 10 representing death. Different areas of the scale represent differences in loss of function in specific ways – the lower ranges focus on moderate loss of any function, the middle range focuses on loss of ability to walk, and higher ranges focus on loss of upper extremity function. Another more recently developed performance measure is the MS Functional Composite (MSFC), which is composed of three separate measures - a Timed 25-Foot Walk (T25W), the 9-Hole Peg Test (9HPT), and the Paced Auditory Serial Addition Test (PASAT). Scores on individual measures are standardized into Z-scores that are used to calculate a composite score that can be used to compare measurements across time and to compare scores on different measures [5]. The MSFC was introduced as a potential improvement on the EDSS due to its continuous (rather than ordinal) measurement and inclusion of measures of fine motor and cognitive function. MSFC scores correlate well (−.80) with EDSS scores [5]. The MSFC does not require a neurologist, but must be administered by trained staff and requires use of some special equipment.

While performance measures of function have generally been considered to be more objective and accurate than self-report measures, there is not solid evidence to support this superiority [6]. There is some indication that performance tests may not accurately capture function because of differences in effort expended in testing and other influences of testing situations. The EDSS, for example, has been difficult to use because neurologists may differ in their application of the definitions of scale ratings of patients - thus increasing inter-rater variability [7]. As a true ordinal scale the application of parametric statistics to EDSS scores is questionable [5,8]. Self-report measures of function offer several advantages. They are simpler and cheaper to administer– a self-report questionnaire can be completed on a computer or over the phone, requiring less time and travel for both subjects and clinicians/researchers than in-person visits to complete performance assessments. In addition, the person responding to the self-report items can respond about his/her ‘typical’ function independent of the circumstances of a specific day or testing environment. Self-reports have also been known to offer predictive value for clinically meaningful change in function [9,10]. Valid and reliable self-report measures of function have potential value for self-monitoring for persons with MS if they accurately reflect change in function over time.

The purpose of this study was to compare scores on a self-report measure of functional limitations with MS (the Incapacity Status Scale [ISS]) with a performance-based measure (the Multiple Sclerosis Functional Composite [MSFC] Index) at five different time points. Both measures have been widely used in previous studies of functioning among persons with MS and demonstrated high levels of reliability and validity. While other studies have explored cross-sectional relationships among measures of functioning in MS, this study also observed the pattern of change in scores over time. Strong correlations between self-report and performance measures, at specific time points as well as over a series of time points capturing functioning over time, would support the usefulness of the more convenient and less expensive self-report assessments of functioning.

Methods

Data Collection

The sample for this study is a subsample of participants in an ongoing, 16-year longitudinal study of people with multiple sclerosis. The purpose of the larger study is to assess the trajectory of change over time in functioning and health promoting behaviors as they relate to quality of life. Details of the recruitment of the larger sample have been reported elsewhere [11]. Participants in the larger survey study who resided within 70 miles of two metropolitan areas where performance testing would take place (n=218) were mailed a letter inviting them to participate. When interested participants responded to the study coordinator’s toll-free number the details of participation were explained and verbal consent was obtained to participate in annual MSFC performance tests for five years. Participants were then contacted to schedule an appointment for data collection. All data collection visits were completed within a three-week time period at each site and the dates of testing were kept constant (within a two-week interval) across the five years. Seventy-three individuals volunteered for this phase of the study and completed the testing at least once; 60 individuals completed testing at all five time points.

All aspects of this study were reviewed and approved by The University of Texas at Austin Institutional Review Board. Research staff obtained written consent at the first data collection and reviewed consent verbally with participants at subsequent visits. A trained research associate administered the MSFC according to standard protocol [12] after they had completed the Incapacity Status Scale. Data collection appointments took approximately one hour. The testing sessions occurred in accessible, air-conditioned sites that could offer privacy and an unobstructed path for the timed walk. Participants received $70 for each data collection visit.

Instruments

Multiple Sclerosis Functional Composite Index (MSFC)

The MSFC was created by the National MS Society’s Clinical Outcomes Assessment Task Force (1994–1997) as a way to measure function for people with MS. Its sensitivity and ability to predict change in EDSS scores over time are well-established [3]. Three performance tests are included in the MSFC – the Timed 25-Foot Walk (T25FW), the 9-Hole Peg Test (9HPT), and the Paced Auditory Serial Addition Test (PASAT). The T25FW, the average of two trials of the number of seconds it takes to walk 25 feet, provides a measure of leg function and ambulation; a higher score indicates lower function. The 9HPT, scored as the average of four trials of the number of seconds taken to successfully complete the test with dominant and non-dominant hands, provides a measure of arm and hand function; higher scores indicate lower function. The PASAT is scored as the number of correct calculations made from 2-minute and 3-minute trials of listening to a number series and simultaneously performing a mental computation using the series. Scores of the PASAT reflect attention and concentration and serve as the measure of cognitive function; higher scores indicate better function. Scores from the three tests are standardized into z-scores and then used to calculate a composite z-score – part of the composite calculation is reversing the direction of the walk and 9HPT score so that all three components have the same scoring directionality [12]. Higher composite scores reflect greater functional capacity. Test-retest reliability correlations for the T25FW are .95 to .99 and for the 9HPT are .43 and .69 (left and right hands). Split-half reliability for the PASAT is .96 [7]. Evidence for construct and concurrent validity of the MSFC Composite Index includes documented differences in scores for people with different MS disease progressions and correlations of MSFC scores with other functional measures (e.g. EDSS and the Sickness Impact Profile; r= −.62 and −.80 respectively [5].

Incapacity Status Scale (ISS)

The Incapacity Status Scale (ISS) [13] is a measure of self-reported level of disability and functional impairment in people with MS. The 16 items of the ISS assess walking, vision, cognition, activities of daily living, and sexual and bowel and bladder function. The 16 items are scored using a 5-point scale. The scale ranges from a score of 0 indicating no difficulties in functioning to 4 indicating functioning only with great difficulty or with assistance. The total score ranges from 0 to 64 (0 indicates ease in functioning and 64 indicates total inability to function). Internal consistency reliability is high (alpha exceeding .80) [14], and the scale’s content and concurrent validity was supported by the expert consensus of the International Federation of Multiple Sclerosis Societies [15] and high correlations with other established measures of impairment in MS [16].

In addition to the total score, three ISS subscale scores were also calculated to reflect the content domains assessed by the three MSFC performance tests. The “Gross Motor Subscale” consisted of four questions assessing walking on level ground, climbing steps, getting in and out of bed, and bathing. The “Fine Motor Subscale” included three items addressing ability to dress oneself, shave or apply makeup, and feed oneself. Two questions pertaining to cognition (memory and calculation difficulties, and fatigue) were combined to create a cognition subscale. The one item about fatigue level was also examined separately, due to the effect of fatigue on performance.

Data Analysis

Cronbach’s alpha was calculated for each subscale and for the overall ISS. Cronbach’s alpha for the Gross Motor Subscale items was .89 at baseline. The Cronbach’s alpha for the Fine Motor Subscale items was .83 at baseline. Because they were only one- or two-item subscales, we did not compute reliability checks on the cognition and fatigue questions. Internal consistency reliability for the overall ISS was .82. Scale and subscale mean scores and standard deviations were calculated for the ISS and the MSFC at each time point. Pearson’s correlations were used to assess the relationship between self-report subscales and external performance measures. To assess the sensitivity of the results to possible violations of the assumptions underlying parametric statistics, Spearman rank order correlations were also calculated. Z-scores for the MSFC measures and overall composite were calculated following the instructions in the testing manual [12]. The standard formula for Z scores was applied to ISS scores to facilitate the comparison of change in scores across time (Figure 1). SPSS version 18 was used to analyze data.

Figure 1.

Change in Total Scale Mean Score (ISS and MSFC) Across Five Years (N=60)

Results

In order to accurately assess change in scores over time, the analyses presented here are for the 60 participants (82% of the initial sample) who completed the tests at every time point. Dropouts were due to death (n=4), missing one data collection point over the five years (n=3), moving out of area (n=2), becoming lost to follow-up (n=2), no longer wishing to participate (n=1), and being misdiagnosed with MS (n=1).

At baseline, the 60 participants had a mean age of 54.3 (standard deviation [SD] 7.9), and had been diagnosed an average of 18.4 (SD 6.0) years. The majority of the participants were female (77%), married (70%), White (88%), and well-educated (52% bachelor’s degree or higher). As seen in Table 1, 46% of the participants reported some paid employment at baseline. Approximately half of the sample (49%) responded positively to the National Health Interview Survey question indicating that they experienced some limitation in daily activities due to a physical, mental or emotional condition. The demographic characteristics of this sample were similar to those of the larger longitudinal study sample who did not participate in the study (85% female, 88% white and a mean age of 56.15 and diagnosed for an average of 19.09 years).

Table 1.

Sample Demographics (N=60)

Characteristic	Mean +/−SD	Range (y)	n (%)
Age	54.3 +/− 7.9	30–69
Years of school completed	14.9 +/− 3.1	8–22
Years since diagnosis	18.4 +/− 6.0	11–36
Gender
Female			46 (76.7)
Male			14 (23.3)
Marital Status
Married			42 (70.0)
Divorced			10 (16.7)
Other			6(10.0)
Missing			2 (3.3)
Employment status
Part or full-time employment			27 (45.0)
Unemployed due to disaiblity			17 (28.3)
Retired/Laid Off			14 (24.1)
Missing			2 (3.3)
Type of MS (Self-Report)
Benign sensory			4 (6.7)
Relapsing-remitting			22 (36.7)
Primary-progressive			8 (13.3)
Secondary-progressive			12 (20.0)
Progressive-relapsing			4 (6.7)
Don’t know			10 (16.7)

Table 2 includes means for the total scores on the Incapacity Status Scale and the Multiple Sclerosis Functional Composite, as well as for their corresponding gross motor, fine motor and cognitive subscales. Mean scores showed little change over time and there was not a consistent or linear pattern of change. In most cases, however, greater variability of scores (larger standard deviations) occurred at later time points. Figure 1 shows the change across time for the total ISS score and the total MSFC score (shown as z-scores). Except between Years 4 and 5, the changes in mean score for both scales followed similar trajectories, rising from Year 1 to Year 3 and then falling from Year 3 to Year 4. Overall, the scores on both scales remain fairly stable across the five-year time period.

Table 2.

Incapacity Status Scale and MS Functional Composite Scale and Subscale Mean Scores and Standard Deviations (N=60)

	ISS Total* (0–64)	MSFC Total+	ISS Gross Motor*	ISS Fine Motor*	ISS Cognitive*	MSFC T25FW*	MSFC 9HPT*	MSFC PASAT+
Year 1	15.8 (7.6)	.04 (.72)	3.6 (3.4)	.9 (1.5)	3.3 (1.73)	36.4 (60.4)	35.3 (31.4)	39.8 (12.1)
Year 2	14.7 (7.4)	.1 (.8)	3.5 (3.2)	.8 (1.2)	2.9 (1.6)	39.5 (66.0)	35.8 (36.9)	39.7 (11.4)
Year 3	14.9 (8.5)	.1 (.8)	3.8 (3.6)	1.1 (2.0)	2.8 (1.5)	39.6 (65.4)	38.0 (42.6)	41.6 (12.6)
Year 4	15.5 (8.2)	.1 (.8)	3.9 (3.5)	1.0 (1.6)	3.0 (1.5)	41.6 (67.7)	38.1 (40.1)	42.3 (12.6)
Year 5	15.6 (8.9)	.1 (.8)	3.8 (3.8)	1.1 (1.8)	3.0 (1.8)	41.5 (67.5)	37.8 (40.2)	42.2 (12.9)

ISS=Incapacity Status Scale; MSFC=Multiple Sclerosis Functional Composite [z-score used to compute MSFC total scores]; T25FW= Number of seconds to complete a Timed 25-Foot Walk (limited to 180 sec); 9HPT= Number of seconds to complete the 9-Hole Peg Test with a 300 second limit; PASAT=Number of correct answers on Paced Auditory Serial Addition Test, maximum score is 60;

^*Higher scores indicate poorer function;

⁺Higher scores indicate better function

As seen in Table 3, the pattern of correlations among measures is similar at each time point. The relationships between the total scores on the ISS self-report and the performance-based MSFC composite were moderate to strong (r = −.59 to −.74) at all times. The relationship is inverse because high scores on the MSFC total and low scores on the ISS indicate better function. The Gross Motor and Fine Motor Subscales of the ISS had moderate to strong relationships (r= −.51 to −.76) with the MSFC overall score. There was no significant relationship between scores on the ISS cognitive or fatigue items and the overall composite score. Over time, total ISS scores were most strongly related to performance on the Timed 25-Foot Walk (r = .54 to .70, p < 0.01). With the exception of the cognitive items, the correlations between the subscales of the ISS and the related MSFC performance test were stronger than correlations with the total ISS. In particular, the strong (.79–.87) correlation between the T25FW and the ISS gross motor items support the usefulness of these self-report items as a measure of gross motor function. The single-item self-report of fatigue was not significantly related to performance on the Timed 25-Foot Walk or the PASAT at any time point. The analyses of relationships using Spearman’s rank order correlations yielded similar results to the parametric statistical analyses.

Table 3.

Correlations - ISS and MSFC Overall Scales and Subscales, Over 5 Years, N=60

	Year 1	Year 2	Year 3	Year 4	Year 5
ISS & MSFC Total	−.6**	−.6**	−.7**	−.7**	−.6**
ISS Gross Motor & MSFC Total	−.7**	−.8**	−.8**	−.8**	−.7**
ISS Fine Motor & MSFC Total	−.5**	−.5**	−.7**	−.6**	−.6**
ISS Cognitive & MSFC Total	−.0	.1	−.1	.0	−.1
ISS Fatigue & MSFC Total	−.0	−.1	−.0	−.1	−.1
T25FW & ISS Total	.6**	.5**	.7**	.7**	.6**
9HPT & ISS Total	.4**	.4**	.5**	.5**	.5**
PASAT & ISS Total	−.3*	−.3*	−.3*	−.2	−.2
ISS Gross Motor & T25FW	.9**	.8**	.9**	.8**	.8**
ISS Fine Motor & 9HPT	.5**	.5**	.7**	.6**	.6**
ISS Cognitive & PASAT	−.1	−.1	−.3	−.1	−.2

^**Correlation is significant at the 0.01 level (2-tailed).

^*Correlation is significant at the 0.05 level (2-tailed).

ISS = Incapacity Status Scale (high scores = poor function); MSFC = MS Functional Composite Index (low scores = poor function); T25FW = MSFC Timed 25-Foot Walk; 9HPT = MSFC 9-Hole Peg Test; PASAT = MSFC Paced Auditory Serial Addition Test.

Discussion

The aim of this study was to explore correlations between self-reports and performance testing of functional status for people with MS across time. We chose the widely used and reliable Incapacity Status Scale and the MS Functional Composite for this study. Although both measures are clinical outcome measures for persons with MS, they assess related but distinctly different categories of clinical outcomes. The Incapacity Status Scale represents disability as perceived by the person with MS and the MSFC is a quantitative test of neurological function [17]. While the MSFC has strong psychometric properties it requires a trained clinician for proper administration and the clinical relevance of change in scores is less clear. Although multiple studies have been conducted to assess clinician-administered outcome measures in MS, we are not aware of a similar study comparing the assessment of two domains of function over time. The relationships of the two measures examined here were strong and consistent over time. Strongest correlations between the ISS and MSFC performance tests were in the area of gross motor ability. Measures of fine motor function were moderately correlated with self-reported activities requiring fine motor function, while the cognitive function and fatigue self-reported measures were only weakly correlated with the cognitive performance test perhaps because fewer items on the ISS address these domains of functioning.

For the most part, the self-report and performance measures reflected slight change in the same direction (improvement or deterioration) and at the same pace over time. It is possible that individual MSFC performance and ISS self-reports could be affected by unassessed concomitant conditions at any time point. However, the group’s average scores on both measures were noticeably stable over time. The increasing standard deviations suggest that some individuals were increasing their scores while others were experiencing decreases in self-reported and tested functioning. This finding reflects the variability in impairment trajectory commonly observed among individuals with multiple sclerosis.

The differing strength of relationships (strong to weak) between measures of gross motor, fine motor and cognitive function may be due to lack of similarity in what is specifically assessed in each measure as well as the difficulty individuals may have self-assessing a particular aspect of function. The very strong correlations (.79–.87) between the performance and self-report measures of gross-motor function (ISS Gross Motor and T25W) may be due to the similarity between the performance measure (the timed 25-foot walk) and the self-report items assessing walking on level ground, climbing steps, getting in and out of bed and bathing. It may also be that persons with MS find it easier to self-monitor and assess the more obvious functions of walking and movement. Fine motor function was operationalized as self-reported difficulties with feeding, dressing and grooming; fine motor performance testing here was the amount of time taken to complete the 9-Hole Peg Test. Correlations were stronger here than those in the cognitive and fatigue domains possibly because the fine motor self-report and 9-Hole Peg Test are indeed measuring similar abilities – controlling one’s fingers in order to carry out more precise actions.. However, the skills needed for feeding, dressing, and grooming may be broader than the fine motor skills needed to perform on the peg test, which could explain the somewhat lower correlations compared with the correlations between walking and self-reported gross motor functions

The weaker correlations between self-report and performance measures of cognitive function may be due to differences in what aspects of function are assessed. For example, the self-report of cognitive function from the ISS is a question about memory lapses and problems with counting or calculating. The performance test measure of cognitive function is the PASAT, which is a timed test asking the test subject to carry out an ongoing series of calculations, holding a stream of information as the test progresses, and often experienced as frustrating by test subjects. Both questions do speak to problems with calculation, but the PASAT performance measure is actually testing attention and concentration functions. In addition, there is some question about how accurate self-report can be regarding cognitive change [18]– can a person with MS accurately gauge his or her own cognitive changes, when s/he is having cognitive changes? However, even if individuals experiencing changes in cognitive function cannot accurately report or quantify the changes, self-reports offer a complement to performance tests as a measure of how aware a person with MS is of the cognitive changes that are taking place, and if cognitive ability to sense one’s own condition is being affected by the MS.

This study used a small convenience sample of people with MS, who are in two geographic areas near urban centers where the larger parent study was conducted. Increasing the sample size and expanding the geographic scope of the sample will all lend themselves to increasing the power and generalizability of the study and support more complicated statistical analyses. Future studies could also utilize more complex multivariate analyses to examine the moderating effects of demographic and health-related variables on the relationship between self-report and performance-based measures. Finally, this is a study of the correlations between one set of performance measures and one self-report survey; it would be useful to examine correlations among a wider range of measures.

Conclusion

Despite these limitations, this study provides preliminary support for the relationship of self-reported functioning and performance testing in the areas of fine and gross motor skills. The unique contribution of this study is the documentation of a consistent pattern between self-report and performance measures across a five year period of time. Because the evidence for the relationship between self-reported cognitive impairment and a performance test of attention and memory was weak, researchers are encouraged to consider carefully what aspects of functioning they are attempting to assess when choosing cognitive assessment methods. The decision to use self-report versus performance measures must also take into account the purpose of the assessment (ex: diagnosis versus on-going monitoring of functioning) as well as patient characteristics. None the less, self-report measures provide important complementary information to performance assessment as they reflect individuals’ perceptions of their own current function and meet the important practical requirements of clinical outcomes measures related to ease of administration, acceptability to patients and professionals and efficient use of resources [17].