The Evolution of the Modified Rankin Scale and Its Use in Future Stroke Trials

Who would have guessed that a scale introduced by Dr. John Rankin in 1957 would become the primary outcome scale for almost all acute stroke trials?¹ The Rankin scale was modified to its current form by Charles Warlow and others as part of the UK-TIA trial in the 1980s ² and its reproducibility was first examined by van Swieten, et al., in 1988 (Table 1).³

Table 1.

The Modified Rankin Scale (mRS)

The scale runs from 0–6, running from perfect health without symptoms to death.

0 - No symptoms.

1 - No significant disability. Able to carry out all usual activities, despite some symptoms.

2 - Slight disability. Able to look after own affairs without assistance, but unable to carry out all previous activities.

3 - Moderate disability. Requires some help, but able to walk unassisted.

4 - Moderately severe disability. Unable to attend to own bodily needs without assistance, and unable to walk unassisted.

5 - Severe disability. Requires constant nursing care and attention, bedridden, incontinent.

6 - Dead.

There is no perfect stroke outcome scale. Regardless, the seven-level, modified Rankin Scale (mRS) has several major strengths: it covers the entire range of functional outcomes from no symptoms to death, its categories are intuitive and easily grasped by both clinicians and patients, its concurrent validity is demonstrated by strong correlation with measures of stroke pathology (for example, infarct volumes) and agreement with other stroke scales,⁴ and its use has demarcated effective and ineffective acute stroke therapies in trials with appropriately powered sample sizes. With a limited number of levels, the mRS may be less responsive to change than some other stroke scales; however, a single-point change on the mRS is clinically relevant.⁴

A limitation of the mRS has been the subjective determination between categories and the reproducibility of the score by examiners and patients.⁴ A systematic review and meta-analysis of studies describing interobserver variability of the mRS reports pooled reliability across ten published studies (n = 587 patients) of a kappa = 0.46 and a weighted kappa of 0.90. ⁵ Multimedia training and certification of examiners in the use of the mRS (http://rankinscale.org/), structured interviews and questionnaires,^6–10 and centralized review of videotape assessments¹¹ have sought to address these issues but reproducibility remains a concern.

But the challenge for a trialist designing a new acute stroke trial is not whether to use the mRS as a primary or major secondary outcome measure, but what statistical approach to use to analyze the mRS, what to expect in terms of effect size, and how to communicate the results of a trial. These decisions are critical for determination of sample size, power, and implementation of study results into clinical practice when the trial is completed.

The Evolution of Statistical Approaches to the mRS

The National Institute of Neurologic Diseases and Stroke (NINDS) tissue plasminogen activator (t-PA) Stroke Trials first demonstrated an efficacious treatment for acute ischemic stroke and the mRS was one of four primary endpoints for a global endpoint used in the trials.¹² The proportion of subjects with a mRS of 0–1 (no or minor symptoms but no functional limitations) was chosen as a primary study endpoint since it was easily communicable, understandable, and desirable to patients and physicians. It also had the advantage of being translatable into a number-needed-to-treat to attain this desired outcome. While all four outcomes in the NINDS Trials were positive, investigators used the mRS primarily to communicate the positive outcome of the trials to physicians and patients. The FDA accepted this dichotomous approach for the mRS as the primary outcome measure for subsequent acute stroke trials. Randomized trials of more severe strokes due to large vessel occlusion used dichotomous cutoffs of 0–2 vs 3–6 (Prolyse in Acute Cerebral Thromboembolism (PROACT) II, Interventional Management of Stroke (IMS) III, etc).^13,14 Trials of intracerebral hemorrhage (ICH), that have even poorer outcomes, focused on dichotomous cutoffs at 0–3 vs 4–6. ¹¹ Other statistical approaches have included varying the dichotomous outcome based upon the initial severity of the stroke.¹⁵ The optimal point for dichotomization depends upon the anticipated distribution of mRS outcomes based on the initial severity of illness, which informs the level of the scale at which a treatment effect is most likely to be observed. Unfortunately, investigators may not know this distribution when planning a trial.

The dichotomous statistical approach does not include the entire range of outcomes across the mRS. Several investigators have argued persuasively for use of the entire ordinal distribution of the mRS as the primary outcome measure since it may provide greater power than the dichotomous approach when the treatment effect occurs along the entire range of mRS, and it is inclusive of both positive and negative outcomes, such as death and symptomatic hemorrhage.^16–19 Statisticians have proposed multiple approaches to analysis of the ordinal mRS that depend in part on meeting or not meeting the proportional odds assumption.^16,18 After scientific discussion and debate in the field, investigators designed FDA-approved trials that use the ordinal distribution of the mRS as the primary outcome measure. ^20,21 Although the relative efficiency has not been shown for all possible tests, using the entire distribution of the mRS may have greater statistical power than a dichotomized analyses when the treatment benefit occurs similarly at several levels of the mRS, rather than clustering at just one end,²² although simulations should be conducted to confirm this for any given hypothesized treatment effect. One disadvantage of the ordinal approach is communicating what a change across the distribution on an ordinal scale means to patients and physicians. Further, the severity distribution of enrolled subjects may impact the ability of the ordinal approach to capture transitions across health states.

More recently, the focus has been on patient-centered outcomes or quality of life, and the most widely accepted patient-centered outcome measure is utility – the desirability of a specific health outcome to the patient.²³ A utility of 1 represents excellent health. The Stroke Therapy Academic Industry Roundtable (STAIR) recommended development of a utility-weighted version of the mRS.²⁴ Investigators subsequently calculated utility values for the various levels of the mRS by mapping responses from the European Quality of Life Scale (EQ-5D)²⁵ onto the mRS levels in populations of stroke patients.^22,26,27 In another study, disability weights for mRS levels were derived using the methodology of the World Health Organization Global Burden of Disease Project (WHO-GBD).²⁸ Based upon these approaches, a UW-mRS was accomplished (Table 2) and compared to ordinal and dichotomous approaches in eight prior acute stroke trials.^22,27,28 This analysis demonstrated the potential advantages of both the utility-weighted modified Rankin Scale (UW-mRS) and the ordinal mRS as compared to the dichotomous analyses. Analysis of the UW-mRS is computationally straight-forward, using t tests that compare the mean utility difference between treatment arms, and the UW-mRS can easily be extended to incorporate adjustments of baseline covariates.

Table 2.

Utility Scores for Each Level of mRS

mRS Score	0	1	2	3	4	5	6
Rivero-Arias et al.27	1	0.87	0.73	0.60	0.28	−0.1	0
Hong & Saver28,28	1	0.95	0.79	0.67	0.35	0.1	0
DAWN Trial22	1	0.91	0.76	0.65	0.33	0	0

An additional feature of a utility measure such as the UW-mRS is the ability to generate quality-adjusted-life-years (QALYs) gained or lost by an intervention or treatment.^29–33 A QALY measure assumes that a year of life lived in perfect health is worth 1 QALY (1 Year of Life × 1 Utility value = 1 QALY) and that a year of life lived in a state of less than this perfect health - is worth less than one. To determine the exact QALY value, one multiplies the utility value associated with a given state of health by the years lived in that state. For example, one year lived in perfect health or 2 years lived at ½ of the value of perfect health as judged by patients are both equivalent to 1 QALY.

To illustrate how QALYs are calculated, let us use a very simplified hypothetical example of an acute stroke trial of 1000 subjects in excellent health before the stroke (mRS of 0 and UW-mRS utility of 1) and the mRS distributions observed in the NINDS t-PA Trials. The effect size at 90 days for IV t-PA versus placebo in the NINDS t-PA Trials as measured by the UW-mRS is 0.09 (UW-mRS methodology from the DAWN Trial (DWI or CTP Assessment with Clinical Mismatch in the Triage of Wake-Up and Late Presenting Strokes Undergoing Neurointervention)²² as illustrated in Table 2). The QALY calculation is 0.09 utility difference x 0.25 years = 0.0225 QALYS (or 8.2 quality-of-life days per subject over those 90 days). The benefit for t-PA in the NINDS t-PA Stroke trial persisted over a year of follow-up³⁴ that equates to 0.09 QALYs or a little over a quality-of-life month per subject. If this trial group is projected to live a mean additional 5 years after their stroke,³⁵ and we assume a continuing mean difference in UW-mRS between the two groups, this would equate to a benefit for t-PA of a mean 0.45 QALYs per subject. A 0.03 difference in the means of the UW-mRS in the treatment group versus controls over 5 years in a hypothetical trial with a smaller effect size would be equivalent to 0.15 QALYS per subject.

Because almost all acute stroke trials complete follow-up at 90 days, statistical models make a number of assumptions that extend the differences in utilities between treatment groups to a lifetime horizon. Sensitivity analyses test the assumptions built into the model. Strong arguments for following patients in acute trials for 1 year rather than 3 months are the demonstration of the durability of treatment effect and the reliability of QALY calculations no matter what statistical approach is used.³⁶

Like any stroke outcome, the UW-mRS has limitations. First, it is based on the mRS with its respective strengths and limitations noted previously. Secondly, the utility weighting of the various Rankin levels can vary among surveyed populations of stroke patients from various countries as nicely detailed by Ali and colleagues.²⁶ This is particularly true at the most severe levels of the mRS and is relevant for international stroke trials where choice of the weighting for the levels of the mRS should ideally reflect the entire population under study. While we think that the concept of quality-of-life years, that integrates how long a patient will live in what proportion of excellent health, should be more intuitive to patients than numerical movement upon an ordinal scale such as the mRS, we are unaware of any study that tests this assumption by interviewing patients.

In summary, the UW-mRS and extrapolated QALYs, despite their limitations, can address the “what does this mean to the patient” limitation of the original mRS. The FDA approved the use of the UW-mRS as the primary outcome measure for the DAWN Trial.²² The Data Safety and Monitoring Board halted the DAWN Trial on February 28^th, 2017 for crossing the prespecified threshold for efficacy at a planned interim analysis of 200 enrolled patients (written personal communication, Tudor Jovin and Raul Noguera, 2–28–2017).

The Evolution of Effect Sizes in Acute Stroke Trials

Table 3 and Figure 1 provide a distribution of the mean differences in the UW-mRS for key positive and negative acute stroke trials that have 150 subjects or more since 1995. The list of trials is not all-inclusive but includes the intravenous t-PA and endovascular trials, large recent medical and surgical ICH trials, and several larger neuroprotective trials including the Stroke–Acute Ischemic NXY Treatment SAINT 1 Trial that was positive by its primary endpoint. Most of these trials used dichotomous mRS endpoints as the primary outcome measure.

Table 3.

Key Randomized Acute Stroke Trials with Greater than 150 Subjects, UW-mRS using DAWN Trial Method

Trial	Total N	Difference in proportion of mRS≤3 (treatment- control)	Difference in UW- mRS Means (treatment-control)	Primary analysis +/−	UW-mRS +/−
ESCAPE	311	25.9%	0.19	+	+
SWIFT PRIME	191	19.8%	0.18	+	+
MR CLEAN	500	15.5%	0.10	+	+
NINDS tPA	624	10.3%	0.09	+	+
REVASCAT	206	14.6%	0.10	+	−
TREVO II	172	11.5%	0.07	+	−
THRACE	402	10.9%	0.06	+	−
PROACT II	180	5.4%	0.05	+	−
ECASS-III	821	2.9%	0.03	+	−
SAINT	1699	3.7%	0.02	+	−
INTERACT 2	2794	2.9%	0.03	−	+
IST 3	3035	4.3%	0.03	−	+
STICH 2	579	5.2%	0.05	−	−
IMS III	629	3.2%	0.04	−	−
CLEAR III	491	2.3%	0.02	−	−
ENCHANTED*	3206	1.2%	0.01	−	−
ATACH 2	961	−1.0%	−0.01	−	−
ALIAS 2	804	−1.9%	−0.01	−	−
SAINT II	3195	−2.4%	−0.02	−	−
FAST: High dose FVIIa vs. placebo	557	−3.7%	−0.03	−	−

^*Non-inferiority design. Trial acronyms for trials not already referenced in text: IST 3 (International Stroke Trial 3); CLEAR III (Clot Lysis Evaluation of Accelerated Resolution of Intraventricular Hemorrhage III); ENCHANTED (Enhanced Control of Hypertension and Thrombolysis Stroke Study); ATACH 2 (Antihypertensive Treatment of Acute Cerebral Hemorrhage 2), ALIAS 2 (High-dose Albumin Treatment for Acute Ischaemic Stroke 2), FAST (Factor Seven for Acute Hemorrhagic Stroke). All UW-MRS analyses were unadjusted for baseline covariates. For some trials, the primary analysis was adjusted or stratified for baseline variables: PROACT II, IMS III, MR CLEAN, ESCAPE, REVASCAT, SAINT, SAINT II, IST-3, ALIAS 2, STICH 2, FAST, and ATACH 2. For other trials, the primary analysis was unadjusted: NINDS t-PA Stroke Trials, ECASS III, SWIFT PRIME, THRACE, TREVO II, INTERACT 2, CLEAR III and ENCHANTED.

Figure 1.

Differences in means of the UW- mRS plotted against differences in proportions of mRS ≤ 3. Size of the circles is proportional to sample size of trial. Red circles are negative trials by both primary dichotomous measure and UW-mRS. Blue solid circles are positive by both measures. Blue circles with clear centers are positive by UW-mRS but negative by primary dichotomous endpoint (like IST3 with mRS of 0–1). Green circles with clear centers are positive by dichotomous primary endpoint but negative by UW-mRS (like PROACT II, TREVO II, and THRACE with mRS of 0–2 and ECASS III with mRS of 0–1). For the REVASCT trial, the two-sided p-value UW-mRS t-test is p=0.0502 which is larger than 0.05 and so it was coded as “- UW-mRS” (green circle*). The SAINT Trial analyzed the whole distribution of scores using the Cochran–Mantel–Haenszel test for its primary analysis that was just statistically positive (odds ratio, 1.20; 95 percent confidence interval, 1.01 to 1.42, green circle*). The ENCHANTED Trial was designed as a non-inferiority trial that was not non-inferior by primary endpoint.

The first observation from the Figure is that treatments added to IV t-PA and/or endovascular therapy are unlikely to have a treatment effect as large as the effects noted in the key definitive NINDS t-PA Trials and endovascular trials. The second observation is that trials with large effect sizes and larger sample sizes are positive by both the primary dichotomous analysis and a two-sample t-test of UW-mRS (solid blue circles).^12,37–39 The REVASCAT Trial (Revascularization with Solitaire FR Device versus Best Medical Therapy in the Treatment of Acute Stroke Due to Anterior Circulation Large Vessel Occlusion Presenting within Eight Hours of Symptom Onset)⁴⁰ had a small sample size and was positive using the primary dichotomous endpoint and was borderline positive by the UW-mRS (p = 0.0502). The THRACE (Thrombectomy in Acute Ischemic Stroke) Trial had the smallest effect size of the endovascular trials and was positive only using the primary dichotomous endpoint.⁴¹

The third observation is that trials with smaller effect sizes require larger sample sizes. The choice of which analysis to choose a priori becomes dicey since the ability of the dichotomous or ordinal shift to detect a treatment effect can differ depending upon the where the treatment effect occurs across the range of the mRS. In two trials, the primary dichotomous endpoint is nonsignificant while the t-test of the UW-mRS is positive.^42,43 The only acute stroke trials with a mean UW-mRS difference of 0.03 or more that were not statistically positive by its primary dichotomous measure or by the t-test of UW-mRS are the IMS III and Surgical Trial in Lobar Intracerebral Haemorrhage (STICH) II; both trials had modest sample sizes.^14,15 A post-hoc analysis of CT angiography positive subjects in the IMS III Trial was statistically positive using an ordinal approach.⁴⁴ If the sample size for IMS III had been 1100, the observed UW-mRS effect would have been statistically significant (two-sided p-value<0.05, favoring endovascular therapy) although the primary approach using the dichotomous approach would still be non-significant. Similarly, if the total N for STICH2 had been 1000, the p-value would be < 0.05 (favoring early surgery). These observations highlight the importance of not underestimating the needed sample size.

Conversely, in trials with smaller sample or effect sizes or both, the primary dichotomous endpoints may be statistically positive while the UW-mRS is not. Such trials include PROACT II, TREVO 2 (Thrombectomy REvascularization of Large Vessel Occlusions), REVASCAT, THRACE, and ECASS III (European Cooperative Acute Stroke Study).^{13,40,41,45,46} These trials illustrate that dichotomous approaches to the mRS can be statistically significant when ordinal approaches are not when the biggest shift in outcomes across the mRS occurs at the chosen endpoint (mRS 0–1 for ECASS III and mRS 0–2 for TREVO II, PROACT II, THRACE and REVASCAT).

Finally, there are trials that are statistically negative by both statistical approaches, largely due to small effect sizes and/or moderate sample sizes.^{11,20,47–50}. There have been no statistically positive Phase III acute ICH trials as determined by the primary clinical outcome measure and prespecified statistical approach, even with larger sample sizes, although INTERACT 2 (Intensive Blood Pressure Reduction in Acute Cerebral Hemorrhage) Trial was statistically positive by UW-mRS.^{11,15,42,49,51,52}

Guidance for the Acute Stroke Trialist

One should carefully consider the expected outcomes in active and standard treatment arms when choosing the preferred statistical approach. Ordinal analyses and the UW-mRS approach capture the entire distribution of outcomes (good and bad) as compared with a dichotomous approach. The translation of the UW-mRS into quality-life-years can communicate the efficacy of a treatment to patients and physicians. Thus, it is a reasonable approach for an acute stroke trial.

No matter what the statistical approach, choosing effect sizes for Phase III acute therapies that are equivalent to t-PA (0.09 mean difference UW-mRS) or endovascular therapy (0.062–0.18 mean difference) is hard to justify and would require very strong preliminary Phase II data. A more reasonable mean difference using the UW-mRS would be effect sizes of 0.03–0.04. This is roughly equivalent to a 5% absolute difference in dichotomous endpoint of mRS of 0–3 at 3 months with various mRS distributions. Phase II trials should provide outcome distributions for various treatment arms to guide the choice of statistical approach and sample size of Phase III trials.

Summary

The mRS has evolved as the primary outcome measure for acute stroke trials. Other patient-centered outcome measures not discussed in this article, such as the Euro-QOL 5-D ²⁵ or Neuro-QOL,⁵³ have complementary strengths and often are used as secondary outcomes, although none have yet to be used as the primary outcome measure in a positive acute stroke trial. This could change in the future. At present, attention to training and standardization of mRS assessments, use of UW-mRS or some equivalent patient-centered outcome, and careful selection of appropriate effect size based upon prior trials and expected distribution of outcomes, are important for planning of future acute stroke trials.