Trade-Offs in Quality-of-Life Assessment Between the Modified Rankin Scale and Neuro-QoL Measures

Abstract

Introduction:

We aimed to describe the physical and cognitive health of patients with differing levels of post-stroke disability, as defined by modified Rankin Scale (mRS) scores. We also compared cross-sectional correlations between the mRS and the Quality of Life in Neurological Disorders (Neuro-QoL) T-scores to longitudinal correlations of change estimates from each measure.

Methods

Mean Neuro-QoL T-scores representing mobility, dexterity, executive function, and cognitive concerns were compared among mRS subgroups. Fixed-effects regression models with robust standard errors estimated correlations among mRS and Neuro-QoL domain scores and correlations among longitudinal change estimates. These change estimates were then compared to distribution-based estimates of minimal clinically important differences.

Results

Seven hundred forty-five patients with ischemic stroke (79%) or transient ischemic attack (21%) were enrolled in this longitudinal observational study of post-stroke outcomes. Larger differences in cognitive function were observed in the severe mRS groups (ie, 4–5) while larger differences in physical function were observed in the mild-moderate mRS groups (ie, 0–2). Cross-sectional correlations among mRS and Neuro-QoL T-scores were high (r = 0.61–0.83), but correlations among longitudinal change estimates were weak (r = 0.14–0.44).

Conclusions:

Findings from this study undermine the validity and utility of the mRS as an outcome measure in longitudinal studies in ischemic stroke patients. Nevertheless, strong correlations indicate that the mRS score, obtained with a single interview, is efficient at capturing important differences in patient-reported quality of life, and is useful for identifying meaningful cross-sectional differences among clinical subgroups.

Introduction

The modified Rankin Scale (mRS) is the most widely used measure of disability and dependence in daily activities in clinical trials involving patients with stroke.¹ In addition to serving as an outcome for treatment effectiveness trials, the mRS is also used to estimate quality-adjusted life-years that are, in turn, used in cost-effectiveness studies.^2–4 Despite the widespread use of the mRS in clinical care and research, the mRS has well-documented limitations.^1,5 Notably, mRS scores have questionable reproducibility (ie, inter-rater reliability), and they omit important content reflecting physical, cognitive, and mental health from the patient perspective during post-stroke recovery.^6,7

These limitations are not unique to the mRS and have been noted for other clinician-assessed functional status scores that have been in use for many years in other patient populations, such as Karnofsky Performance Status scores in oncology and the Functional Independence Measure in rehabilitation medicine.^8–10 To address these limitations, collections of patient-reported outcome measures have been developed, such as the Quality of Life in Neurological Disorders (Neuro-QoL) for patients with neurologic dysfunction and the Patient-Reported Outcomes Measurement Information System (PROMIS) for patients with neurologic and non-neurologic conditions.^11,12 These new quality-of-life (QoL) measurement systems have emerged as tools that (1) enable valid and reliable assessments of health status; (2) provide greater clarity in specific types of impairments, symptoms, and dysfunction; (3) incorporate the patients’ perspective regarding their current level of function or recovery; and (4) enable valid comparisons of the same symptoms in other diseases and conditions and with the larger the US population.^11–19

Although substantial validity and reliability evidence supports the use of Neuro-QoL measures in multiple stroke populations, to date, their use in stroke trials has been limited.^20–22 In fact, the mRS has remained a benchmark for assessing post-stroke disability and recovery, despite the psychometric advantages of Neuro-QoL measures.¹ There are a variety of reasons this may be the case. Although Neuro-QoL measures can be administered to patients in a variety of different formats (eg, paper surveys, cellular phone/tablet, computer adaptive testing), most individual measures require the patient to complete at least 4 to 6 items to obtain adequate precision in measurement.^23,24 The time required for a patient to complete an assessment is compounded by the number of domains/symptoms assessed at each point. In contrast, the mRS is a single-item measure that can be quickly and efficiently completed by a clinician during a brief clinical encounter, and assessments are still possible when the patient is comatose or dead. Moderate to strong cross-sectional correlations between mRS scores and T-scores from Neuro-QoL and PROMIS measures further suggest the mRS is sensitive to important differences in symptoms from the patient perspective, but correlations between changes in mRS and changes in T-scores are weaker.^14,15 Taken together, these findings suggests that the convenience in administration of the mRS comes at the expense of important psycho-metric properties of validity, reliability, and interpretability, and that the cost may be highest in longitudinal designs.

Therefore, one aim of this study was to use Neuro-QoL measures to provide a deeper description of the physical and cognitive health of patients with differing levels of post-stroke disability (ie, defined by mRS levels). The second aim of this study was to evaluate the cross-sectional differences in patient-reported physical and cognitive health domains of Neuro-QoL that were captured by the mRS. The third aim was to assess if change in mRS predicted change in Neuro-QoL T-scores and to opportunistically evaluate whether differences in QoL change scores were consistent with previously reported minimal clinically important differences (MCIDs) for PROMIS and Neuro-QoL measures.

Methods

Participants

A total of 745 adults were included in this study and represent individuals who (1) had a transient ischemic attack or ischemic stroke, (2) were admitted to a large urban stroke center between August 2012 and January 2014, and (3) were enrolled in a longitudinal observational study of post-stroke outcomes. Patients provided informed consent; if the patient was unable to do so then a legally authorized representative provided consent. The institutional review board waived the informed consent requirement when patients could not provide consent (eg, owing to coma) or when the legally authorized representative could not be located.

Measures

Demographic and clinical data, including stroke severity using the National Institutes of Health Stroke Scale²⁵ and stroke subtype using the Trial of Org 10172 in Acute Stroke Treatment classification^26,27 were prospectively captured by a team of vascular neurologists and research coordinators.^20,28 The mRS is a measure of disability status with scores that range from 0 (no symptoms) to 6 (dead) and was obtained through a standardized telephone interview 30 and 90 (±5) days after stroke.^29,30 To assess health-related QoL, 4 domains of the Neuro-QoL measures were also collected (ie, executive function [EF], general cognitive concerns [GC], upper extremity dexterity [UED], and lower extremity mobility [LEM]) at 30 and 90 days after stoke. Scores from Neuro-QoL measures have substantial validity and reliability evidence supporting their use.^{13,14,31–34} Neuro-QoL scores are centered on the estimated US general population mean of 50 (SD = 10), and higher scores indicate better functioning.

Statistical Analysis

Cross-sectional differences

Participants were stratified into groups based on mRS scores at 30 and 90 days, and mean differences in LEM, UED, EF, and GC T-scores were compared. Fixed-effects regression models with robust standard errors were used to test for statistically significant differences among marginal mean QoL scores and to estimate variance explained in QoL scores by the mRS.

Longitudinal differences

Because a 1-point change in mRS has been identified as clinically meaningful,⁵ participants were classified into groups representing improvement, clinical stability, and decline when their mRS score changed by at least 1 point between 30 and 90 days. Fixed-effects regression models with robust standard errors were then used to estimate average change in LEM, UED, EF, and GC scores for each group.

MCIDs

There are a variety of methods available to estimate MCIDs, and we employed several distinct anchor- and distribution-based methods to triangulate a limited range of MCID estimates, according to recommended best practices.^35,36 Anchor-based methods employ a clinically familiar benchmark known to identify clinically meaningful differences among patient subgroups, and in this study, the mRS served as that benchmark. The average change in QoL scores for patients who got worse (ie, increased by at least 1 level of the mRS) was compared to those who improved (ie, decreased by at least 1 level of the mRS) and to those who remained stable from 30 to 90 days post-stroke.

These anchor-based estimates were then compared to estimates from several distribution-based methods, including the standard error of measurement (SEM).^37,38 The SEM represents a margin of (measurement) error for survey scores, and as such, it represents the minimum statistically detectable difference in scores. Score reliability (r), also known as precision, is largely a function of how consistently study participants endorse a set of items on a survey; conversely, measurement error represents how inconsistently items are endorsed (1 – r). The SEM represents this inconsistency in item responses on the original scale of the measure through the following formula: SEM = SD × SQRT(1 – r). Thus defined, the SEM represents the lower bound for any MCID estimate, because group differences that are smaller than the SEM are by definition statistically meaningless.

Empirical studies have demonstrated that MCIDs for self-report measures typically range from one-third to one-half SDs,^39,40 and this is likely because most self-report measures are designed to achieve an internal consistency reliability (r) between 0.7 and 0.9. According to the formula above, when r = 0.7, then SEM z 1/2 SD, and when r = 0.9, SEM ≈ 1/3 SD. We included both the one-third and one-half SD benchmarks in this study to further facilitate MCID estimation.

Results

Table 1 presents demographic and clinical characteristics of 745 patients with transient ischemic attack (21%) or ischemic stroke (79%). The average age of the sample was 65.1 years (SD =15.6). Most were white (58.7%) and male (51.3%), and presented with mild strokes (median National Institutes of Health Stroke Scale: 2; interquartile range: 0–5).

Table 1.

Demographic profile of study participants (n = 745).

	M	SD
Age	65.10	15.60
NIHSS at admission	3.90	5.30
Median NIHSS, IQR	2	(0–5)
	n	%
Sex
Male	382	51.28
Female	363	48.72
Race/ethnicity
NH white	438	58.79
NH black	219	29.40
Hispanic	56	7.51
Other/multiracial	32	4.30
Stroke type
TIA	154	20.67
Ischemic	591	79.33
TOAST
Cardioembolic	155	20.81
Large artery atherosclerosis	126	16.91
Small artery disease	99	13.29
Other determined	113	15.17
Cryptogenic	252	33.83
mRS (premorbid)
0	650	87.25
1	45	6.04
2	7	0.94
3	35	4.70
4	7	0.94
5	1	0.13
mRS (1 month)
0	244	32.75
1	297	39.87
2	44	5.91
3	92	12.35
4	53	7.11
5	15	2.01
mRS (3 months)
0	344	46.17
1	187	25.10
2	20	2.68
3	60	8.05
4	32	4.30
5	6	0.81
Missing	96	12.89
Change mRS (30–90 days)
Better	201	26.98
Stable	410	55.03
Worse	38	5.1
Missing	96	12.89

IQR indicates interquartile range; M, mean; mRS, modified Rankin Scale; NH, non-Hispanic; NIHSS, National Institutes of Health Stroke Scale; TIA, transient ischemic attack; TOAST, Trial of Org 10172 in Acute Stroke Treatment.

QoL Profiles and Cross-Sectional Differences

Figure 1A,B indicates unique QoL profiles for participants with different levels of function defined by mRS at 30 and 90 days post-stroke. At 30 days, participants with an mRS of 0 reported above average QoL in all 4 assessed domains of LEM (M = 53.62, SE =0.37), UED (M = 54.42, SE = 0.15), EF (M = 55.98, SE = 0.27), and GC (M = 58.47, SE = 0.29). Those with an mRS of 1 indicated below average LEM (M = 47.62, SE = 0.39) and UED (M = 48.65, SE = 0.39), but maintained above average EF (M = 54.70, SE = 0.33) and GC (M = 55.79, SE = 0.35). Those with an mRS of 2 reported lower scores in each domain, but a similar pattern of below average LEM (M = 40.59, SE = 0.91) and UED (M = 46.50, SE = 1.27) and above average EF (M = 50.68, SE = 1.22) and GC (M = 52.30, SE = 1.09) scores remained. In fact, EF scores only dropped below the estimated US population mean of 50 for participants with an mRS of 3 (M = 44.08, SE = 1.20), and for GC scores, the same was true for participants with an mRS of 4 (M = 45.53, SE = 1.31).

Figure 1.

Quality-of-life profile of mRS–based clinical subgroups at (A) 30 days post stroke and (B) 90 days post stroke. mRS indicates modified Rankin Scale.

To evaluate discrimination of each QoL domain at different levels of disability, differences in QoL scores were compared and tested between groups at each level of the mRS. At 30 days (Fig. 1A), larger differences in QoL were observed between patients with high disability levels (mRS of 5 vs 4), and the largest differences were observed in EF (Δ = 20.29, SE = 2.41), followed by UED (Δ = 18.09, SE = 2.05), GC (Δ = 16.11, SE = 2.82), and LEM (Δ =9.50, SE = 1.12). In contrast, at the lower end of disability (ie, mRS 1 vs 0), QoL score differences were largest for LEM (Δ = 6.00, SE =0.54), followed by UED (Δ = 5.78, SE = 0.42), GC (Δ = 2.69, SE =0.45), and EF (Δ = 1.28, SE = 0.43). These differences remained consistent over time as evidenced by an almost identical pattern of differences at 90 days post-stroke (Fig. 1B).

Fixed-effects regression models indicated that mRS scores predicted most of the variance in LEM scores at 30 (R² = .69) and 90 (R² = .66) days and UED scores at 30 (R² =.55) and 90 (R² = .57) days. The mRS scores also predicted approximately half of the variance in EF scores at 30 (R² = .53) and 90 (R² = .50) days and nearly 40% of the variance in GC scores at 30 (R² = .41) and 90 (R² = .37) days.

Longitudinal Differences

Regarding patient decline and recovery of function, changes in mRS score predicted approximately one-fifth of the variance (R² = .20) in changes in self-reported LEM scores and approximately 14% of the variance (R² = .14) in UED changes scores (Table 2). Moreover, those who worsened on the mRS reported approximately 7 points of decline in LEM and UED, while those who improved on the mRS reported approximately 5 points’ improvement in these domains, on average. Similarly, changes in mRS levels also predicted statistically significant changes in EF (R² = .03) and GC (R² = .04) but to a lesser extent. Participants who worsened on the mRS reported approximately 4 points’ decline in EF and GC, while those who improved on the mRS reported approximately 2 points’ increase in these domains. For participants who were stable (ie, no change in mRS), no statistically significant changes in patient-reported outcomes were observed (maximum average change of 0.34).

Table 2.

Predicted change in QOL T-scores and robust SEs by change in mRS subgroup.

N = 643	Δ mRS	Δ T-score	Robust SE	t	P	95% CI-LB	95% CI-UB	R2
EF	Better	1.71	0.55	3.10	<.01	0.63	2.79	0.03
	Same	0.03	0.35	0.09	.93	−0.66	0.71
	Worse	−4.32	1.79	−2.41	.02	−7.83	−0.81
All pairwise comparisons statistically significant (ie, ≤0.02)
GC	Better	1.78	0.47	3.84	<.01	0.87	2.70	0.04
	Same	−0.15	0.32	−0.46	.65	−0.77	0.48
	Worse	−3.81	1.53	−2.50	.01	−6.80	−0.82
All pairwise comparisons statistically significant (ie, ≤0.02)
LEM	Better	5.54	0.52	10.67	<.01	4.52	6.55	0.20
	Same	−0.34	0.31	−1.09	.28	−0.94	0.27
	Worse	−7.35	1.33	−5.51	<.01	−9.96	−4.73
All pairwise comparisons statistically significant (ie, <0.01)
UED	Better	4.52	0.55	8.29	<.01	3.45	5.59	0.14
	Same	−0.06	0.34	−0.18	.86	−0.72	0.60
	Worse	−7.12	1.64	−4.34	<.01	−10.33	−3.91
All pairwise comparisons statistically significant (ie, <0.01)

EF indicates executive function; GC, general cognitive concerns; LEM, lower extremity mobility; QOL, quality of life; SE, standard error; UB, upper bound; UED, upper extremity dexterity.

As indicated in Figure 2, upper-bound MCID estimates using one-half SD approximated 5 points, and lower-bound estimates using one-third SD approximated 3 points. This range of 3 to 5 points was above the minimum statistically detectable difference of approximately 2 points indicated by the SEM. The SEM was calculated with an observed reliability coefficient of 0.93, which was consistent across QoL domains (range 0.92–0.94). In general, larger MCID estimates were observed when using the anchor-based methods in physical function domains (ie, LEM and UED), and smaller MCID estimates were observed when using anchor-based methods in cognitive domains (ie, GC and EF). Also, the range of MCID estimates varied somewhat among domains, with the largest variability observed in LEM (SD = 1.87) and the smallest variability observed in GC (SD = 0.96). Nevertheless, across all domains, the mean and median MCID estimates consistently fell between 3 and 5 points.

Figure 2.

MCID estimates for 4 quality-of-life domains by method. * indicates negative number.

Discussion

Although this is not the first study to compare the mRS with patient-reported outcomes,^14,15,17 this is the first study to compare the performance of the mRS as a clinical outcome to multiple Neuro-QoL domains reflecting diverse dimensions of mental and physical health. We observed important differences between distinct functional domains of physical and mental health known to have differential rates of recovery, particularly during the most critical period of patient recovery (ie, first 90 days post-stroke).⁴¹ Moreover, this is the first study to empirically investigate MCID estimates for NeuroQoL measures in stroke populations.

Our findings indicate that the mRS captures important information regarding patient-reported QoL as assessed through multiple Neuro-QoL measures, despite substantial deficits in content validity.⁷ While most of the variance (≥50%) in cross-sectional differences in LEM, UED, and EF were explained by mRS scores at both 30 and 90 days, cross-sectional differences in QoL scores were not consistent among each level of the mRS. As such, incremental changes in mRS scores likely reflect widely different functional changes, which vary by domain. Katzan et al¹⁴ similarly reported differences among symptom domains, with larger differences overall at the higher end of the mRS (ie, mRS 4–5); however, they reported the largest differences in physical function when compared to pain, fatigue, sleep, anxiety, and depression. Although differences in physical function were similar to the findings of the current study in magnitude of effect, we found an even larger effect at the higher end of the mRS in cognitive function, while at the lower end of the scale (ie, mRS 0–2), the mRS discriminated better in motor function. It is possible that findings from these studies would have been aligned with the current study had Katzan et al¹⁴ included measures of EF and GC, and to our knowledge, this is the first study comparing mRS to scores of EF and GC in stroke. This finding of better discrimination of EF and GC at the higher end of the mRS may also explain why individuals with strokes often continue their care with untreated cognitive impairments, as previously reported.^42,43 It is possible that implementation of Neuro-QoL measures of EF and/or GC in clinical contexts may help address this shortcoming in patient care, particularly during the first 90 days of recovery. Moreover, implementation of these measures in stroke trials may help uncover differential effects of distinct treatment approaches that may go undetected when using the mRS alone.

We also found that cross-sectional correlations between the mRS and NeuroQoL measures of LEM and UED were nearly twice as strong (R² vs r²) as previously reported correlations between mRS and PROMIS physical function.^14,15 This likely reflects the more neurologically focused nature of NeuroQoL items (cf., content validity). Given that the mRS was developed to identify clinically relevant differences among stroke patients, these stronger correlations suggest that NeuroQoL measures may be more useful for assessing symptoms specific to stroke populations.

Despite the strong cross-sectional associations observed, a change in Neuro-QoL measures was weakly associated with a change in mRS, which undermines their potential exchangeability in longitudinal trials. In this study, the strongest effect observed was for LEM, which suggests that observed changes in mRS are most likely driven by changes in patient mobility, but even then, only 20% of the variance in LEM change scores was explained by changes in mRS. Although these longitudinal correlations are weaker than the cross-sectional correlations, they are still nearly twice as large as longitudinal correlations between mRS and PROMIS-physical function reported by Katzan et al.¹⁵ The weakest correlations observed between change in mRS and change in cognitive function measures further underscores a critical limitation of the mRS as a clinical monitoring tool and the need for instruments that better capture longitudinal changes in cognitive function during post-stroke recovery.

Nevertheless, because the mRS is an established clinical benchmark, we opportunistically compared these change estimates with previously reported MCID estimates for Neuro-QoL and PROMIS measures in other patient populations, and the observed estimates were in line with prior reports.^17,40,44,45 That being said, the weak correlation between change in mRS and change in Neuro-QoL measures somewhat undermines the utility of mRS as a clinical anchor for MCID estimation, so validation is warranted in future studies that include additional measures that can serve as more robust clinical anchors. Moreover, there are well-known limitations to both distribution and anchor-based methods of estimating MCIDs, including the use of crude change categories of improved, stable, or worse.^46–48 Nevertheless, critical reviews continue to recommend a combination of distribution-and anchor-based methods to triangulate on a limited range of MCID estimates when feasible, as in the current study.^49,50

Limitations and Future Directions

There are several important limitations to this study. Participants were recruited from a single metropolitan stroke center and predominately had mild strokes. This limited our ability to evaluate longitudinal correlations between change in mRS and NeuroQoL measures in those with more severe strokes. Moreover, participants were only assessed at 2 points (ie, 30 and 90 days post stroke), and it is possible that differences between mRS and Neuro-QoL T-scores could change past 90 days. Lastly, most patients (55%) had no change in mRS over time, and only 5% got worse, which limited our ability to further stratify beyond improvement or worsening by baseline mRS scores. As such, the MCID estimates reported in this study should be considered provisional until these findings are validated in a larger sample with more severe strokes.