Comparing Early National Institutes of Health Stroke Scale Versus 90‐Day Modified Rankin Scale Outcomes in Acute Ischemic Stroke Trials: A Systematic Review and Analysis

Abstract

Background

Early National Institutes of Health Stroke Scale (NIHSS) assessment may provide practical benefits over 90‐day modified Rankin Scale (mRS), but it is unclear how it compares in adjudicating randomized clinical trial (RCT) results in acute ischemic stroke.

Methods and Results

We searched Ovid Medline (inception to April 1, 2023) and included RCTs of acute therapies for acute ischemic stroke with data for both 90‐day mRS and NIHSS within 7 days. Primary outcome was agreement between trial results (classified as positive, negative, or neutral) based on 24‐hour NIHSS and 90‐day mRS scores. We additionally assessed agreement for 2‐hour, 48‐hour, 72‐ to 96‐hour, and 5‐ to 7‐day NIHSS scores. We aimed to validate our findings using individual patient data from the ESCAPE (Randomized Assessment of Rapid Endovascular Treatment of Ischemic Stroke) and ESCAPE‐NA1 (Safety and Efficacy of Nerinetide [NA‐1] in Subjects Undergoing Endovascular Thrombectomy for Stroke) RCTs. We included 116 trials (44 387 patients), contributing 165 NIHSS assessments. The 24‐hour NIHSS scores resulted in the same classification as 90‐day mRS scores in 61/73 (83.6%) trials (Cohen's kappa, 0.64 [95% CI: 0.45–0.83] and Gwet's agreement coefficient 1, 0.79 [95% CI: 0.67–0.90]). Agreement was not statistically different by timing of NIHSS assessments (range 75%–100%, P=0.33). Individual patient data showed higher agreement for assessments between 48 hours and 7 days, varying by NIHSS dichotomization cutoffs (NIHSS score, 0–2; 2 hours, 56.6%; 24 hours, 66.6%; 48 hours, 71.8%; 5–7 days: 76.5%, P<0.01; NIHSS score, 0–7; 2 hours, 72.8%; 24 hours, 80.5%; 48 hours, 83.1%; 5–7 days: 84.7%, P<0.01).

Conclusions

The 24‐hour NIHSS scores aligned with 90‐day mRS scores in 84% of RCT results, indicating intermediate‐to‐good agreement. However, individual patient data showed that early NIHSS risks misclassifying around 1/4 patients. These data contribute to a better understanding of the nuances of early NIHSS score as an outcome in acute ischemic stroke RCTs.

Nonstandard Abbreviations and Acronyms

mRS

modified Rankin Scale

NIHSS

National Institutes of Health Stroke Scale

SMD

standardized mean difference

Clinical Perspective.

What Is New?

• In this analysis that included 44 387 patients with acute ischemic stroke from 116 randomized controlled trials, the agreement in classifying trials as positive versus neutral versus negative between 24‐hour National Institutes of Health Stroke Scale (NIHSS) score and 90‐day modified Rankin Scale score was 83.6% (Cohen's kappa: 0.64 and Gwet's agreement coefficient 1, 0.79); trial‐level agreement did not differ significantly across NIHSS assessment time points between 24 hours and 7 days, although individual patient data of 1420 participants indicated somewhat higher agreement for later NIHSS assessments.

What Are the Clinical Implications?

• These data will help inform the discussion on early NIHSS score as an alternative outcome measure to 90‐day modified Rankin Scale score in randomized controlled trials on acute ischemic stroke interventions.

• Individual patient data showed that NIHSS score within 7 days risks misclassifying up to one fourth of patients with respect to 90‐day modified Rankin Scale score, although agreement improves with later assessments; therefore, when seeking to prognosticate 90‐day functional recovery in clinical practice, it may be prudent to wait at least 1 week to assess the poststroke NIHSS score.

The most accepted outcome for assessing the efficacy of interventions in phase III randomized clinical trials (RCT) of acute ischemic stroke is the functional neurological outcome measured on the modified Rankin Scale (mRS) at 90‐day follow‐up. ¹ The mRS combines motor and cognitive components mixing the concepts of impairment, disability, and handicap. ² The National Institutes of Health Stroke Scale (NIHSS) is a purely impairment‐based scale that determines neurological deficits on examination. The NIHSS is less prone to psychosocial confounding factors, and if assessed well before the 90‐day period (eg, within the first week after stroke onset), it may reflect a purer assessment of a patient's posttreatment neurological status. A previous review of primary outcome measures in stroke trials showed that the NIHSS was in second place after the mRS as the most frequently chosen primary outcome measure in stroke trials. ³

Using the NIHSS assessed within 7 days follow‐up could be a reasonable early surrogate outcome measure to the mRS at 90 days and could additionally decrease the costs associated with conducting stroke trials by reducing the required follow‐up time. ⁴ Some modeling studies have also suggested that using the NIHSS might allow a reduction in the sample size of RCTs. ⁴ , ⁵ Despite these hypothetical advantages, there are a paucity of data examining whether clinical trial results observed when using the NIHSS (measured within 7 days) as an outcome align with those obtained using the 90‐day mRS score. We therefore aimed to determine the agreement between these 2 outcomes using published data from RCTs of acute ischemic stroke, and to further validate our findings using individual patient data from 2 RCTs that measured the NIHSS at multiple time points.

METHODS

Data Availability

The analysis for this study is based on published results from individual studies. All extracted data from the individual studies and the code used for performing the analyses can be made available upon reasonable request to the corresponding author. The ESCAPE‐NA1 (Safety and Efficacy of Nerinetide [NA‐1] in Subjects Undergoing Endovascular Thrombectomy for Stroke) individual patient data are not currently publicly available for distribution.

Ethical Approval

Approval from an institutional or regional review board was not required as all study level data used in this study were extracted from publications. For the ESCAPE (Randomized Assessment of Rapid Endovascular Treatment of Ischemic Stroke and ESCAPE‐NA1 trials, the ethics board at each participating site approved the trial.

Registration and Reporting Standards

We conducted this systematic review according to a protocol that we registered (International Prospective Register of Systematic ReviewsCRD42023445515) and report the results consistent with Preferred Reporting Items for Systematic Reviews and Meta‐Analyses. ⁶

Search Strategy

We used a comprehensive electronic strategy (Table S1) to search Ovid Medline from inception to April 1, 2023, for articles meeting our eligibility criteria. We performed backward citation searching using the bibliographies of included studies.

Eligibility Criteria

We included RCTs published in English, German, Spanish, French, or Dutch (reflecting the language skills of the study team) in peer‐reviewed journals, reporting on adult patients with acute ischemic stroke, examining interventions initiated within 48 hours of stroke onset and completed within 7 days of stroke onset and with available data for both 90‐day mRS score and NIHSS score within 7 days. A detailed list of inclusion and exclusion criteria is provided in Table 1.

Table 1.

Inclusion and Exclusion Criteria

Inclusion criteria
1.	Randomized controlled trial
2.	Reports on both mRS and NIHSS in the main paper or supplement
3.	Minimum follow‐up period of 90‐d for the mRS
4.	NIHSS assessed within 7 d (168 h) follow‐up
5.	Includes acute‐phase stroke patients (within 48 h of stroke onset)
6.	Studies in humans
7.	Reports in English, German, Dutch, French, or Spanish
Exclusion criteria
1.	Nonacute phase interventions (ie, intervention started >48 h after onset or continued for more than 7 d after onset)
2.	Primarily a secondary prevention study (main outcome is recurrent stroke/transient ischemic attack)
3.	Studies not reporting on original data (meta‐analysis and reviews)

mRS indicates modified Rankin Scale; and NIHSS, National Institutes of Health Stroke Scale.

Study Selection

Four reviewers (L.A.R., J.M.O., M.K., A.G.) independently used the systematic review management platform Covidence to screen titles and abstracts for eligibility and assess the full text of potentially eligible studies for final inclusion using rigorous application of the inclusion and exclusion criteria. All studies were assessed by at least 2 authors. In case of disagreement, the authors resolved discrepancies in consensus meetings whenever possible, with A.G. having the final vote for conflict resolution.

Data Extraction

Six reviewers (L.A.R., J.M.O., M.K., A.S., R.V.M., A.G.) independently extracted the characteristics and outcomes of each trial. For publications reporting on multiple trials or phases of trials, data were extracted for each trial or trial phase separately. We extracted aspects of study design as well as patients and treatment characteristics. We aimed to collect data on NIHSS outcomes at all available time points within 7 days post stroke. Thus, studies reporting NIHSS scores on >1follow‐up time‐point could contribute to multiple timepoints. Data on NIHSS scores were collected in all available formats, including median with interquartile range or mean±SD, dichotomized scores or change from baseline NIHSS score. Data on 90‐day mRS score were extracted as the full distribution across all scores of the mRS whenever possible. If this was not reported, we extracted data on the distribution by dichotomized scores or by summary statistics (median with interquartile range or mean±SD). If studies applied dichotomization for mRS, NIHSS, or both we recorded the definitions that were used. In case of disagreement, the authors resolved discrepancies in consensus meetings.

Risk of Bias

We used version 2 of the Cochrane risk‐of‐bias tool for randomized trials for assessing risk of bias in randomized trials in the following domains ⁷ : (1) randomization process, (2) deviations from intended interventions, (3) missing outcome data, (4) measurement of the outcome, and (5) selection of the reported results. In case of disagreement, the authors resolved discrepancies in consensus meetings.

Outcomes

The primary outcome was agreement between results of the trial assessed on the NIHSS at 24‐hour follow‐up and the results of the trial assessed on the mRS at 90‐day follow‐up. Secondary outcomes were agreement between trial results as assessed on the NIHSS at 2‐hour, 48‐hour, 72‐ to 96‐hour, and 5‐ to 7‐day follow‐up and the mRS score at 90‐day follow‐up.

We used the authors' specifications of favorable versus unfavorable outcomes for dichotomizing scores for each RCT for both the 90‐day mRS and early NIHSS scores. We classified trial results on the NIHSS and mRS as positive versus neutral versus negative. A result on the NIHSS was defined as positive if (1) both the point estimate and the lower boundary of the 95% CI of the odds ratio for favorable NIHSS were >1 (indicating statistically significant improvement on the dichotomized NIHSS) or (2) if the point estimate and the upper boundary of the 95% CI of the standardized mean differences (SMDs) were <0 (indicating improvement of continuous NIHSS scores, including scores reported as change in NIHSS score compared with baseline NIHSS score). Results were classified as negative if the point estimate and upper boundary of the 95% CI were <1 for dichotomized scores and >0 for continuous scores. All other results were classified as neutral. Results measured using mRS scores at 90‐day follow‐up were categorized into positive, negative, and neutral results in the same manner.

Statistical Analysis

We quantified the distribution of cohort‐level characteristics with descriptive analyses. We summarized the effect sizes observed for each intervention arm versus control in each trial, for each timing of the NIHSS until 7 days and for the 90‐day mRS. Effect sizes were grouped as either odds ratios (ORs) or SMDs depending on how results were presented in the included studies. To facilitate synthesis of studies reporting mean and median summary statistics for effect sizes, these were transformed into SMD's whenever possible. Unadjusted summary effect measures were calculated in STATA version 17 using the metan command. We separately calculated each summary effect measure (SMDs and ORs) for each outcome (NIHSS and mRS), including each available time point of NIHSS assessment (2‐hour, 24‐hour, 48‐hour, 72‐ to 96‐hour, and 5‐ to 7‐day follow‐up).

We determined the percentage of agreement across trials between NIHSS and 90‐day mRS scores in terms of a trichotomized classification of trial results as positive versus neutral versus negative results. We also examined additional measures of agreement including Cohen's kappa and Gwet's agreement coefficient 1 (AC1). ⁸ Cohen's kappa is a statistical measure that quantifies agreement for categorical ratings by accounting for the agreement occurring by chance. Gwet's AC1 is another reliability coefficient that adjusts for chance agreement; it has the advantage of providing more stable and unbiased estimates than Cohen's kappa, particularly in cases of high or low prevalence of certain categories. When there is unbalanced category prevalence, use of Cohen's kappa can result in misleadingly low kappa values, even in the setting of good agreement, also known as the kappa paradox. ⁹ Using the Gwet's AC1 in addition to Cohen's kappa can provide a more stable coefficient with varying prevalence or marginal probability, or at least provide contrast with the Cohen's kappa estimate to better understand how changing category definitions or prevalence affects agreement.

We classified the Cohen's kappa and Gwet's AC1 values according to the cutoffs proposed by Fleiss et al. ¹⁰ (with values <0.40 considered poor agreement, 0.40–0.74 considered intermediate to good, and 0.75–1.00 considered excellent) and by Landis and Koch ¹¹ (with values of <0.20 considered poor agreement, 0.21–0.40 considered fair, 0.41–0.60 considered moderate, 0.61–0.80 considered substantial, and 0.81–1.00 considered almost perfect). ¹¹ We did this for all NIHSS time points combined as well as for each individual NIHSS time point separately. To determine whether agreement differed between the different time points of NIHSS assessment we performed a chi‐square test and a chi‐square test for trend. We further assessed agreement between NIHSS and 90‐day mRS scores when dichotomizing trials as positive versus neutral/negative and determined the sensitivity and specificity for the dichotomized classification of trials for each NIHSS time point using the 90‐day mRS scoreas reference standard. Sensitivity here meant the proportion of trials for which the NIHSS analysis yielded a positive result, out of all trials for which the mRS scoreyielded a positive result. Specificity meant the proportion of trials for which the NIHSS analysis yielded a negative/neutral result, out of all trials for which the mRS scoreyielded a negative/neutral result.

Individual Patient Data Analysis

To corroborate our findings, we pooled individual patient data from the ESCAPE and ESCAPE‐NA1 trials. ¹² , ¹³ These trials were selected because they were the only trials that assessed the NIHSS on 4 time points in a similar manner. The mRS was dichotomized at 0 to 2, indicating good functional outcome. NIHSS scores were dichotomized at 0 to 2 (indicating good outcome) in accordance with the trial protocols. ¹² , ¹³ Additionally, we assessed NIHSS scoredichotomized at 0 to 7 because previous research indicated this may be a stronger predictor of 90‐day mRS score. ¹⁴ We assessed agreement measures (percentage agreement, sensitivity, specificity, Cohen's kappa, and Gwet's AC1) between 90‐day mRS and NIHSS scores assessed at 2 hours, 24 hours, 48 hours, and 5 to 7 days. For sensitivity, this meant examining the proportion of all patients who had a favorable outcome according to the NIHSS, out of all patients who had a favorable outcome according to the mRS. For specificity, this meant the proportion of patients who had an unfavorable outcome according to the NIHSS, out of all patients who had an unfavorable outcome according to the mRS. Furthermore, we calculated SMDs between the treatment and control arms for the NIHSS at each time point and for the 90‐day mRS scoreusing Cohen's d. They were derived by applying an ANOVA followed by a mean comparison of the estimates in the combined data of the trials. For the NIHSS, we derived the SMDs after ANCOVA including baseline NIHSS score as a covariate as recommended in recent trial simulations. ¹⁵

RESULTS

After screening 1401 references and excluding ineligible studies, we included 116 RCTs reporting on 90‐day mRS and NIHSS scores within 7 days, representing a total of 44 387 patients. ¹² , ¹³ , ¹⁶ , ¹⁷ , ¹⁸ , ¹⁹ , ²⁰ , ²¹ , ²² , ²³ , ²⁴ , ²⁵ , ²⁶ , ²⁷ , ²⁸ , ²⁹ , ³⁰ , ³¹ , ³² , ³³ , ³⁴ , ³⁵ , ³⁶ , ³⁷ , ³⁸ , ³⁹ , ⁴⁰ , ⁴¹ , ⁴² , ⁴³ , ⁴⁴ , ⁴⁵ , ⁴⁶ , ⁴⁷ , ⁴⁸ , ⁴⁹ , ⁵⁰ , ⁵¹ , ⁵² , ⁵³ , ⁵⁴ , ⁵⁵ , ⁵⁶ , ⁵⁷ , ⁵⁸ , ⁵⁹ , ⁶⁰ , ⁶¹ , ⁶² , ⁶³ , ⁶⁴ , ⁶⁵ , ⁶⁶ , ⁶⁷ , ⁶⁸ , ⁶⁹ , ⁷⁰ , ⁷¹ , ⁷² , ⁷³ , ⁷⁴ , ⁷⁵ , ⁷⁶ , ⁷⁷ , ⁷⁸ , ⁷⁹ , ⁸⁰ , ⁸¹ , ⁸² , ⁸³ , ⁸⁴ , ⁸⁵ , ⁸⁶ , ⁸⁷ , ⁸⁸ , ⁸⁹ , ⁹⁰ , ⁹¹ , ⁹² , ⁹³ , ⁹⁴ , ⁹⁵ , ⁹⁶ , ⁹⁷ , ⁹⁸ , ⁹⁹ , ¹⁰⁰ , ¹⁰¹ , ¹⁰² , ¹⁰³ , ¹⁰⁴ , ¹⁰⁵ , ¹⁰⁶ , ¹⁰⁷ , ¹⁰⁸ , ¹⁰⁹ , ¹¹⁰ , ¹¹¹ , ¹¹² , ¹¹³ , ¹¹⁴ , ¹¹⁵ , ¹¹⁶ , ¹¹⁷ , ¹¹⁸ , ¹¹⁹ , ¹²⁰ , ¹²¹ , ¹²² , ¹²³ , ¹²⁴ , ¹²⁵ , ¹²⁶ , ¹²⁷

Nine of 116 RCTs tested more than 1 intervention (Table 2). Thrombolytic/antithrombotic medication was the most frequently tested intervention type (37/116 [31.9%] trials, total of 115 245 patients). The median age of the study population was 68 (interquartile range, 66–72) years, and 25 218 (56.8%) were male. The median time from stroke onset to initiation of treatment was 6 (interquartile range, 5–11) hours and median duration of the intervention was 1 (1–24) hours.

Table 2.

Study Characteristics

Characteristic	Median (IQR) or N (%)	Studies reporting characteristic	Number of patients
Age, y	68 (66–72)	116	44 387
Male sex	25 218 (56.8%)	116	44 387
Baseline NIHSS score	13 (9–16)	113	43 409
Time from onset to start intervention in hours	6 (5–11)	116	44 387
Duration of intervention in hours	1 (1–24)	116	44 387
Number of study arms
2	106 (91.4%)	106	34 431
3	7 (6.0%)	7	9192
4	2 (1.7%)	2	136
6	1 (0.9%)	1	628
Intervention type
Endovascular therapy (including device)	30 (25.9%)	30	8809
Cerebroprotectant	27 (23.3%)	27	14 598
Thrombolytic/antithrombotic	37 (32.8%)	37	15 245
Other	22 (19.0%)	22	5735
Large vessel occlusion	15 044 (69.0%)	60	21 788
Independent prestroke	28 228 (90.9%)	66	31 050
Year of publication	2019 (2014–2021)	116	44 555
Timing NIHSS assessment*
2 h	4 (3.4%)	4	1832
24 h	74 (63.8%)	74	26 908
48 h	25 (21.6%)	25	8106
72–96 h	14 (12.0%)	14	2684
5–7 d	53 (45.7%)	53	24 815
Format of reported NIHSS score*
Mean/median	62 (53.4%)	62	25 538
Change from baseline	24 (20.7%)	24	6069
Dichotomized	57 (49.1%)	57	27 036
Format of reported mRS score*
Mean/median	41 (35.3%)	41	15 054
Dichotomized	110 (94.8%)	110	43 808
Full ordinal range*	76 (65.5%)	76	37 659

^*More than 100% because results may be presented in multiple ways in a single study.

IQR indicates interquartile range; mRS, modified Rankin Scale; and NIHSS, National Institutes of Health Stroke Scale.

In general, there were some concerns regarding overall bias in almost half (53/116, 45.7%) of included studies while 41/116 (35.3%) studies were deemed to be at low risk of overall bias. (Figure, Figure S1). When examining the individual domains, most studies were at low risk of bias arising from the randomization process 79/116 (68.1%), deviations from intended interventions 74/116 (63.8%), missing outcome data 96/116 (82.8%), outcomes measurement 89/116 (76.7%), and selection of the reported results 64/116 (55.2%).

Figure 1. Risk of bias assessment of included studies.

The 116 included RCTs contributed 165 NIHSS assessments performed across the 5 time points. Unadjusted summary effects for NIHSS and mRS outcomes are presented in Figures S2 through S16. When classifying trial results in a trichotomous manner as positive, negative, or neutral and combining results for all NIHSS time points, there was agreement between NIHSS and mRS scores in 138/165 (83.6%) assessments. NIHSS score was assessed at 24 hours in 73/116 included trials and there was agreement between results assessed on 24‐hour NIHSS and mRS scores in 61/73 studies (84%, Table 3; Table S2). For 24‐hour assessment Cohen's kappa was 0.64 (indicating intermediate to good ¹⁰ or substantial agreement ¹¹ depending on the applied cutoffs) and Gwet's AC1 was 0.79 (indicating excellent ¹⁰ or substantial agreement ¹¹ ). Besides assessment at 24 hours, NIHSS score was most frequently reported for 7‐day follow‐up (36/116 trials) with 86% of these trials showing agreement between results assessed on NIHSS and mRS. Agreement for the remaining time points ranged from 75% to 100%. We observed no significant differences for rates of agreement between NIHSS and mRS scores across the different time points (P=0.33). The effect sizes obtained with the NIHSS score in terms of SMDs were generally similar to those obtained with the 90‐day mRS score (Table S2).

Table 3.

Agreement Between NIHSS and mRS (Positive Versus Neutral Versus Negative)

NIHSS timing	Agreement NIHSS/mRS scores	Agreement—good outcome*	Agreement—neutral outcome*	Agreement—poor outcome*	Cohen's kappa (95% CI)	Gwet's AC1 (95% CI)
2 h	3/3 (100%)	0	3/3 (100%)	0	NA	NA
24 h	61/73 (83.6%)	17/21 (81.0%)	43/50 (86.0%)	1/2 (50.0%)	0.64 (0.45–0.83)	0.79 (0.67–0.90)
48 h	18/24 (75.0%)	3/6 (50.0%)	15/17 (88.2%)	0/1 (0.0%)	0.35 (−0.08 to 0.79)	0.69 (0.44–0.94)
72–96 h	12/12 (100%)	2/2 (100%)	10/10 (100%)	0	NA	NA
5–7 d	44/51 (86.2%)	8/12 (66.7%)	34/36 (94.4%)	2/3 (66.7%)	0.67 (0.43–0.90)	0.83 (0.70–0.95)

AC1 indicates agreement coefficient 1; mRS, modified Rankin Scale; and NIHSS, National Institutes of Health Stroke Scale.

^*Cases in which 90‐d mRS agreed with the NIHSS impression (ie, positive, neutral or negative), out of all NIHSS assessments with that impression at that time point.

When considering dichotomized classification of trial results as positive versus negative/neutral, agreement in trial classification was similar compared with the trichotomized results. The 24‐hour NIHSS score had a sensitivity of 81.0% (95% CI, 58.1%–94.6%) and specificity of 86.8% (95% CI, 74.2%–94.5%) for classifying the trials as positive versus negative/neutral with the 90‐day mRS score as the reference standard. The 7‐day NIHSS score had a corresponding sensitivity and specificity of 80.0% (95% CI, 44.4%–97.5%) and 90.2% (95% CI, 76.9%–97.5%), respectively (Table S3).

Agreement in Individual Patient Data

Among 1420 patients (315 in ESCAPE and 1105 in ESCAPE‐NA1), 90‐day mRS score and at least 1 early NIHSS measurement within 7 days post stroke were available for 1409 patients (99.2%).

When classifying trial results trichotomized as positive versus negative versus neutral, there was agreement between NIHSS and mRS in overall trial results (positive for ESCAPE, neutral for ESCAPE‐NA1) for all time points except 2‐hour NIHSS score. When dichotomizing NIHSS scores at 0 to 2 and combining results for all time points, there was agreement with 90‐day mRS outcomes in 3523/5195 (67.8%) assessments. Specifically, among 1564 NIHSS assessments classified as good outcome, 1438 (91.9%) were also classified as good outcome by the 90‐day mRS score; however, among 3631 poor NIHSS outcomes, only 2085 (57.4%) also had poor 90‐day mRS outcomes. Agreement measures increased for later versus earlier NIHSS assessments (2 hours: 56.6%; 24 hours: 66.6%; 48 hours: 71.8%; 5–7 days: 76.5%, P<0.01; Cohen's kappa and Gwet's AC1 rising from 0.21 and 0.19 [ie, poor/slight agreement] respectively at 2 hours to 0.54 and 0.53 [ie, intermediate to good/moderate] at 5–7 days, Table 4, Table S4). For all time points, sensitivity was low (2 hours 26.2% and 5–7 days 67.6%) and specificity was high (2 hours 97.6% and 5–7 days 89.7%). Agreement measures with 90‐day mRS scores were higher when dichotomizing NIHSS scores at 0 to 7 (percentage agreement 4169/5195 [80.3%], with good mRS outcomes in 2409/2860 [84.2%] good NIHSS outcomes and poor mRS outcomes in 1760/2335 [75.4%] bad NIHSS outcomes), with similar improvement over time (2 ‐hours: 72.8%; 24 hours: 80.5%; 48 hours: 83.1%; 5–7 days: 84.7%, P<0.01; Cohen's kappa and Gwet's AC1 rising from 0.47 and 0.46 [ie, intermediate to good/moderate] respectively at 2 hours to 0.67 and 0.61 [ie, intermediate to good/substantial] at 5–7 days, Table 4). Sensitivity increased (2 hours: 62.9%, 5–7 days: 86.2%) and specificity decreased (2 hours: 93.3%, 5–7 days: 72.1%). That being said, the effect sizes obtained with the NIHSS in terms of SMDs were similar at each time point to those obtained with the 90‐day mRS score (Table 5).

Table 4.

Agreement Between NIHSS (Dichotomized at 0–2 and Dichotomized at 0–7) and mRS Using Individual Patient Data

NIHSS timing	Agreement—overall NIHSS/mRS scores	Agreement—good outcome*	Agreement—poor outcome*	Cohen's kappa (95% CI)	Gwet's AC1 (95% CI)	Sensitivity (95% CI)	Specificity (95% CI)
NIHSS score dichotomized at 0–2 d
2 h
ESCAPE	193/296 (65.2)	24/28 (85.7)	169/268 (63.1)	0.19 (0.11–0.28)	0.44 (0.33–0.55)	19.5 (12.9–27.6)	97.7 (94.2–99.4)
ESCAPE‐NA1	539/998 (54.0)	171/181 (94.5)	368/817 (45.0)	0.20 (0.17–0.24)	0.11 (0.05–0.18)	27.6 (24.1–31.3)	97.4 (95.2–98.7)
Combined	732/1294 (56.6)	195/209 (93.3)	537/1085 (49.5)	0.21 (0.18–0.24)	0.19 (0.13–0.25)	26.2 (23.1–29.6)	97.6 (96.0–98.7)
24 h
ESCAPE	214/302 (70.9)	47/54 (87.0)	167/248 (67.3)	0.35 (0.26–0.45)	0.50 (0.39–0.60)	36.7 (28.4–45.7)	96.0 (92.0–98.4)
ESCAPE‐NA1	697/1066 (65.4)	307/328 (93.6)	390/738 (52.9)	0.36 (0.32–0.41)	0.31 (0.25–0.37)	46.9 (43.0–50.8)	94.9 (92.3–96.8)
Combined	911/1368 (66.6)	354/382 (92.7)	557/986 (56.5)	0.37 (0.33–0.41)	0.35 (0.30–0.40)	45.2 (41.7–48.8)	95.2 (93.2–96.8)
48 h
ESCAPE	213/288 (74.0)	56/63 (88.9)	157/225 (69.8)	0.44 (0.34–0.53)	0.54 (0.44–0.64)	45.2 (36.2–54.4)	95.7 (91.4–98.3)
ESCAPE‐NA1	661/930 (71.1)	306/328 (93.3)	355/602 (59.0)	0.45 (0.40–0.50)	0.42 (0.36–0.48)	55.3 (51.1–59.5)	94.2 (91.3–96.3)
Combined	874/1218 (71.8)	362/391 (92.6)	512/827 (61.9)	0.46 (0.41–0.50)	0.44 (0.39–0.49)	53.5 (49.6–57.3)	94.6 (92.4–96.4)
5–7 d
ESCAPE	219/285 (76.8)	76/91 (83.5)	143/194 (73.7)	0.52 (0.42–0.62)	0.56 (0.46–0.66)	59.8 (50.8–68.4)	90.5 (84.8–94.6)
ESCAPE‐NA1	787/1030 (76.4)	451/491 (91.9)	336/539 (62.3)	0.53 (0.49–0.58)	0.53 (0.48–0.59)	69.0 (65.3–72.5)	89.4 (85.8–92.3)
Combined	1006/1315 (76.5)	527/582 (90.6)	479/733 (65.4)	0.54 (0.50–0.58)	0.53 (0.48–0.58)	67.6 (64.1–70.8)	89.7 (86.8–92.2)
NIHSS score dichotomized at 0–7 d
2 h
ESCAPE	222/296 (75.0)	69/89 (77.5)	153/207 (73.9)	0.46 (0.36–0.57)	0.54 (0.44–0.64)	56.1 (46.9–65.0)	88.4 (82.7–92.8)
ESCAPE‐NA1	720/998 (72.1)	398/454 (87.7)	322/544 (59.2)	0.45 (0.40–0.51)	0.45 (0.39–0.50)	64.2 (60.3–68.0)	85.2 (81.2–88.6)
Combined	942/1294 (72.8)	467/543 (86.0)	475/751 (63.3)	0.47 (0.42–0.51)	0.46 (0.41–0.50)	62.9 (59.3–66.3)	86.2 (83.0–89.0)
24 h
ESCAPE	244/302 (80.8)	101/132 (76.5)	143/170 (84.1)	0.61 (0.52–0.70)	0.62 (0.53–0.71)	78.9 (70.8–85.6)	82.2 (75.7–87.6)
ESCAPE‐NA1	857/1066 (80.4)	526/606 (86.8)	331/460 (72.0)	0.60 (0.55–0.64)	0.62 (0.57–0.67)	80.3 (77.1–83.3)	80.5 (76.4–84.3)
Combined	1101/1368 (80.5)	627/738 (85.0)	474/630 (75.2)	0.61 (0.56–0.65)	0.61 (0.57–0.66)	80.1 (77.1–82.8)	81.0 (77.6–84.1)
48 h
ESCAPE	237/288 (82.3)	108/143 (75.5)	129/145 (89.0)	0.65 (0.56–0.73)	0.65 (0.56–0.74)	87.1 (79.9–92.4)	78.7 (71.6–84.7)
ESCAPE‐NA1	775/930 (83.3)	478/558 (85.7)	297/372 (79.8)	0.65 (0.60–0.70)	0.68 (0.63–0.73)	86.4 (83.3–89.2)	78.8 (74.3–82.8)
Combined	1012/1218 (83.1)	586/701 (83.6)	426/517 (82.4)	0.66 (0.61–0.70)	0.67 (0.63–0.71)	86.6 (83.7–89.0)	78.7 (75.1–82.1)
NIHSS score dichotomized at 5–7 d
ESCAPE	240/285 (84.2)	121/160 (75.6)	119/125 (95.2)	0.69 (0.61–0.77)	0.68 (0.60–0.77)	95.3 (90.0–98.3)	75.3 (67.8–81.8)
ESCAPE‐NA1	876/1030 (84.9)	608/718 (84.7)	266/312 (85.3)	0.66 (0.61–0.71)	0.73 (0.69–0.77)	93.0 (90.7–94.8)	70.7 (65.9–75.3)
Combined	1114/1315 (84.7)	729/878 (83.0)	385/437 (88.1)	0.67 (0.63–0.71)	0.71 (0.68–0.75)	93.3 (91.4–95.0)	72.1 (68.1–75.9)

AC1 indicates agreement coefficient 1; ESCAPE, ESCAPE‐NA1, Safety and Efficacy of Nerinetide (NA‐1) in Subjects Undergoing Endovascular Thrombectomy for Stroke; mRS, modified Rankin Scale; and NIHSS, National Institutes of Health Stroke Scale.

^*Cases in which 90‐d mRS agreed with the NIHSS impression (ie, good or bad), out of all NIHSS assessments with that impression at that time point.

Table 5.

Standardized Mean Difference in NIHSS (Intervention Minus Control Arm) at Different Early Time Points Versus 90‐Day mRS Using Individual Patient Data

Measure and timing	SMD (95% CI)
NIHSS score—2 h
ESCAPE	−0.36 (−0.59 to −0.13)
ESCAPE‐NA1	−0.03 (−0.15 to 0.09)
Combined	−0.10 (−0.21 to 0.01)
NIHSS score —24 h
ESCAPE	−0.50 (−0.73 to −0.28)
ESCAPE‐NA1	−0.02 (−0.14 to 0.10)
Combined	−0.12 (−0.22 to −0.01)
NIHSS score —48 h
ESCAPE	−0.69 (−0.93 to −0.46)
ESCAPE‐NA1	−0.02 (−0.15 to 0.11)
Combined	−0.17 (−0.28 to −0.06)
NIHSS—5 to 7 d
ESCAPE	−0.74 (−0.98 to −0.50)
ESCAPE‐NA1	−0.03 (−0.15 to 0.10)
Combined	−0.18 (−0.29 to −0.07)
mRS score—90 d
ESCAPE	−0.52 (−0.75 to −0.30)
ESCAPE‐NA1	−0.03 (−0.14 to 0.09)
Combined	−0.12 (−0.23 to −0.02)

Negative Values Indicate Treatment Benefit. ESCAPE indicates Endovascular Treatment for Small Core and Proximal Occlusion Ischemic Stroke; ESCAPE‐NA1, Safety and Efficacy of Nerinetide (NA‐1) in Subjects Undergoing Endovascular Thrombectomy for Stroke; mRS, modified Rankin Scale; NIHSS, National Institutes of Health Stroke Scale; and SMD, standardized mean difference.

DISCUSSION

In this systematic review and analysis, which included 44 387 patients from 116 trials with information on 170 NIHSS assessments over 5 time points, we observed that trial results as measured on 24‐hour NIHSS score agreed with results as assessed on the 90‐day mRS in almost 9 out of 10 trials. This resulted in a Cohen's kappa of 0.64 and Gwet's AC1 of 0.79. Agreement between NIHSS within 7‐day follow‐up and 90‐day mRS scores was not significantly different between the various time points. Individual patient data did show improvement for agreement over time and suggested that at an individual level NIHSS within 7 days risks misclassifying 20% to 30% of patients with respect to favorable 90‐day mRS depending on the definitions that are used.

Using the NIHSS within the first 7 days after stroke onset, rather than 90‐day mRS, as the primary outcome in acute stroke trials has several advantages. First, it would drastically reduce the required follow‐up period, which in turn would result in lower operating costs for stroke trials. The use of the NIHSS within 7 days also minimizes interference from confounding factors, such as other medical conditions or socioeconomic barriers unrelated to the ischemic stroke or the tested intervention and thus allows for a better estimation of the direct treatment effect of the intervention on outcome. ¹²⁸ , ¹²⁹ Using early NIHSS score as a primary outcome may allow for a smaller sample size; For instance, a simulation study showed that using NIHSS at day 7, rather than 90‐day mRS, could reduce the required sample size by almost a quarter. ⁴

There are also important downsides to the NIHSS to consider. Compared with the mRS, the NIHSS may be more difficult to interpret, especially for patients. Furthermore, it shows a weaker association with quality of life and its statistical characteristics are less optimal. ¹³⁰ , ¹³¹ Through historical precedent, 90‐day mRS has become the standard for stroke trials and it is favored as the primary outcome by key regulators. The 85% agreement between early NIHSS and 90‐day mRS scores in the current study might be insufficient to convince regulators of the role of early NIHSS as a primary outcome. Adoption of early NIHSS as a substitute primary outcome in stroke trials would be limited if regulatory approval cannot be obtained based on such trials.

A previous study showed that NIHSS increasingly predicts functional outcome at 90 days as the time since stroke onset progresses, accounting for one quarter of the variance at 1 to 3 hours after stroke onset, half at 24 hours, and three quarters at 90 days. ¹³² However, our trial‐level results indicated a consistently high level of agreement across all assessed time points within 1 week of follow‐up, whereas our individual patient data showed there was higher agreement for later assessments. These findings hold potential implications for the design of future stroke trials, as relying on the NIHSS measurement at 24 hours may be more feasible and practical when assessing outcome on a study level, considering that patients are often transferred or discharged from the participating hospital, which complicates later in‐person assessment. However, when the goal is to predict outcomes at an individual patient‐level, our results indicate it might be better to wait at least 1 week before assessing the NIHSS. Furthermore, predicting functional outcomes for individual patients is more accurate when combining several radiological and clinical factors, including early NIHSS score, rather than relying solely on early NIHSS score. Several prediction tools have been developed and demonstrated to accurately predict functional outcomes on an individual level, though many have not been externally validated. ¹³³ , ¹³⁴

Although early NIHSS assessment may be a more direct measure of postintervention impairment, it is vulnerable to interrater variabilities, and these factors may also affect how agreement from one time point to another plays out in practice, especially when comparing NIHSS assessments across different trials. In previous studies, most NIHSS items showed moderate to excellent interrater reliability but certain items, such as ataxia assessment, consistently showed poor interrater reliability. ¹³⁵ , ¹³⁶ This is a potential pitfall of introducing early NIHSS score as a primary outcome in stroke trials. However, problems with interrater variability are also well appreciated for the mRS and are harder to narrow down to specific components given the global/holistic nature of the assessment compared with the examination‐based NIHSS. ¹³⁷ , ¹³⁸

The interpretation of the presented kappa and Gwet's AC1 values varies depending on the definitions used for the cutoff thresholds. For instance, the kappa value for NIHSS assessed at 24 hours was classified as intermediate to good according to the criteria proposed by Fleiss et al. ¹⁰ However, using Landis and Koch's criteria it was categorized as substantial, whereas Altman's criteria would classify it as good. ¹¹ , ¹³⁹ These variations highlight that the specific terminology used to describe agreement depends on the chosen framework. Nevertheless, across all definitions, the agreement between NIHSS at 24 hours and 90‐day mRS consistently falls within the intermediate to good range.

By aggregating results from >100 trials, our findings further strengthen and corroborate the use of early NIHSS outcome assessments in stroke trials. Another study assessed the correlation between NIHSS at 5 to 7 days and 90‐day mRS in the MR CLEAN and IMS III (Interventional Management of Stroke III) trials and showed that the treatment effect on the mRS is largely mediated by NIHSS at 24 hours follow‐up. ¹⁴⁰ Other studies also showed that NIHSS score within 1 week of follow‐up is the most important predictor of functional outcome on the 90‐day mRS. ¹⁴¹

This study has several limitations. First, there was considerable heterogeneity in definitions that were used for dichotomization of NIHSS and mRS scores in the included studies. This may have potentially affected agreement between the outcomes, although the extent to which this may have influenced our results is unknown. Second, we used 90‐day mRS as the reference standard although it may be affected by various factors that occur independent of acute ischemic stroke or the assessed intervention. ¹²⁸ , ¹²⁹ This could have led to disagreement between early NIHSS and 90‐day mRS scores resulting in a potential underestimation of the performance of early NIHSS score, whereas in fact early NIHSS score better reflects the neurological injury. There was considerable variability in the way early NIHSS was reported in the included studies, both in terms of timing and the method of presentation (dichotomized, means/median or change compared with baseline). It is unclear how this may have influenced our findings. Although there is consensus that reporting results of a shift analysis of the full ordinal scale is the desirable way to report 90‐day mRS outcomes, no clear agreement exists for assessing early NIHSS score. ¹⁴² Future studies should establish the optimal way of analyzing early NIHSS measurements, both in terms of timing and analytical methods, to standardize its use as an alternative to 90‐day mRS as the primary outcome in stroke trials. Ideally, such studies should look at individual patient data across a larger number of studies.

CONCLUSIONS

NIHSS assessed at 24‐hours aligned with 90‐day mRS scores in 84% of trials for identifying positive versus neutral versus negative RCT results, indicating intermediate to good agreement. However, individual patient data showed that NIHSS score within 7 days risks misclassifying around 1 in 4 patients with respect to 90‐day mRS score, with some improvement in agreements for assessments between 48 hours and 7 days. These data contribute to a better understanding of the nuances of considering early NIHSS score as an outcome measure in acute ischemic stroke RCTs.

Sources of Funding

This work was supported by the Heart and Stroke Foundation of Canada New Investigator Award.

Disclosures

Leon A. Rinkel is supported by the Niels Stensen Fellowship. Johanna M. Ospel received consulting fees from Nicolab. Mayank Goyal has received grants from NoNo, Medtronic, and Cerenovus via the University of Calgary; royalties or licenses from GE Healthcare and Microvention; consulting fees from Microvention, Medtronic, Stryker, and Mentice and has stock options in Circle Neurovascular. Aravind Ganesh reports consulting fees and honoraria from Atheneum, AlphaSights, MD Analytics, MyMedicalPanel, Creative Research Designs, CTC Communications Corp, Alexion, Biogen, and Servier Canada; research support from Alberta Innovates, Campus Alberta Neuroscience, Brain Canada, the Canadian Cardiovascular Society, Panmure House, the Government of Canada INOVAIT and New Frontiers in Research programs, the Heart and Stroke Foundation of Canada and the Canadian Institutes of Health Research; stock/stock options from SnapDx Inc and LetsGetProof; and a patent application for a system for patient monitoring and cuff‐based therapies – all outside the scope of the published work.

Supporting information

Tables S1–S4

Figures S1–S16