Stroke Severity Is a Crucial Predictor of Outcome: An International Prospective Validation Study

Abstract

Background

Stroke is among the leading causes of morbidity and mortality worldwide. Without reliable prediction models and outcome measurements, comparison of care systems is impossible. We analyzed prospectively collected data from 4 countries to explore the importance of stroke severity in outcome prediction.

Methods and Results

For 2 months, all acute ischemic stroke patients from the hospitals participating in the Global Comparators Stroke GOAL (Global Outcomes Accelerated Learning) collaboration received a National Institutes of Health Stroke Scale (NIHSS) score on admission and a modified Rankin Scale score at 30 and 90 days. These data were added to the administrative data set, and risk prediction models including age, sex, comorbidity index, and NIHSS were derived for in‐hospital death within 7 days, all in‐hospital death, and death and good outcome at 30 and 90 days. The relative importance of each variable was assessed using the proportion of explained variation. Of 1034 admissions for acute ischemic stroke, 614 had a full set of NIHSS and both modified Rankin Scale values recorded; of these, 507 patients could be linked to administrative data. The marginal proportion of explained variation was 0.7% to 4.0% for comorbidity index, and 11.3 to 25.0 for NIHSS score. The percentage explained by the model varied by outcome (16.6–29.1%) and was highest for good outcome at 30 and 90 days. There was high agreement between 30‐ and 90‐day modified Rankin Scale scores (weighted κ=0.82).

Conclusions

In this prospective pilot study, the baseline NIHSS score was essential for prediction of acute ischemic stroke outcomes, followed by age; whereas traditional comorbidity index contributed little to the overall model. Future studies of stroke outcomes between different care systems will benefit from including a baseline NIHSS score.

Introduction

Stroke is the second leading cause of death worldwide and a leading cause of adult disability.1 Despite currently available treatment options for stroke, patients often face the prospect of significant poststroke disability.2 The implementation of regional and nationwide stroke systems of care and quality‐improvement initiatives has significant impact on reducing stroke‐related morbidity and improving outcomes.3, 4, 5, 6

The Global Comparators Stroke GOAL collaboration is an international project facilitated by Dr Foster (www.drfoster.com) with participation from >40 hospitals in Europe, the United States, and Australia.7 The participating hospitals are a selected group of mostly academic hospitals that have been collaborating to share data and compare clinical outcomes and performance across international boundaries. The overall aim of the Global Comparators Stroke GOAL program is to compare stroke systems of care and outcomes across regions and countries.

The modified Rankin Scale (mRS) is a validated measure of functional outcome after stroke (ranging from 0, indicating no symptoms at all, to 6, indicating death), commonly used in large‐scale multicenter stroke trials and prospective studies of poststroke disability.8, 9, 10 The utility of the mRS has been tested in modified telephone‐based interviews11, 12 and at different time points after stroke.13, 14 A number of prior poststroke outcome studies have used mRS in their prediction models, including the multinational Virtual International Stroke Trials Archives (VISTA); however, the majority of reports were based on the retrospective analysis of historical data.15, 16 A major motivation for the current study was to design and implement a prospective international pilot project to validate the mRS as a robust stroke outcome measure and the National Institutes of Health Stroke Scale (NIHSS) score as a reliable predictor of outcome for future use in multinational database research.

Methods

Dr Foster and Global Comparators’ Stroke GOAL Initiative

As part of the Global Comparators collaboration facilitated by health care information company Dr Foster,7 the Stroke GOAL program included 13 hospitals from the United States, the United Kingdom, and Europe that contributed prospective data for this analysis at the time of this pilot project in 2012 (participating hospitals are listed in the Acknowledgements).

Data Collection and Variable Definitions

The following variables were available in the administrative database: age, sex, comorbidity index (CMI), and NIHSS score as well as in‐hospital death within 7 days, all in‐hospital death, 30‐ and 90‐day death, and mRS at 30 and 90 days. CMI had been derived previously with weights specific to stroke and in‐hospital death, using all records in the Global Comparators project, with 31 comorbidities taken from the Elixhauser index, plus dementia.7

Prospective Stroke‐Specific Characteristics

The mRS is a reliable and valid functional poststroke outcome assessment scale; furthermore, the mRS is useful for measurement of 30‐ to 90‐day patient outcome, is available in all languages used throughout the Global Comparators network, and can be performed using structured interviews. An mRS performed at 30 days after stroke may prove to be more feasible as a systematic measurement while serving as a reliable proxy for final outcome.17 Because most research studies that validated the mRS after stroke allowed completion of the examination until 100 days after stroke, we included this expanded time window for follow‐up. It is uncertain whether 90‐day mRS is superior to 30‐day measurement.

The NIHSS is a systematic assessment tool that provides a quantitative measure of stroke‐related neurological deficit.18 The NIHSS was originally designed as a research tool to measure baseline data from patients in acute stroke clinical trials. The scale is widely used today as a clinical assessment tool to evaluate severity of stroke, to determine appropriate treatment, and to predict patient outcome.19

Inclusion and Exclusion Criteria

The target population consisted of adult patients with acute ischemic stroke (AIS). To link the included patients to the patients included in the Global Comparators Stroke GOAL project, the following inclusion criteria were used: (1) discharge primary diagnosis of AIS, (2) patient aged ≥18 years, and (3) admission to the hospital with ≥1 overnight stay. Exclusion criteria included (1) discharge diagnosis of transient ischemic attack (except if acute ischemic lesion was present on diffusion weighted imaging), (2) discharge diagnosis of intracerebral hemorrhage, or (3) discharge diagnosis of subarachnoid hemorrhage.

Pilot Implementation: Data Collection and Patient Follow‐up

All AIS patients admitted to the participating hospitals between March 1, 2012, and April 30, 2012 were included in this prospective pilot study. The patients were followed for 90 days after their admission for assessment of final poststroke outcome. With exception of the in‐hospital stroke cases, all AIS patients admitted to the stroke or neurology department by stroke team staff or designees during the pilot timeline were enrolled. All patients were assigned an NIHSS score before intravenous tissue plasminogen activator administration or other acute reperfusion therapies or within 24 hours of admission.

For each patient or patient's surrogate, reliable contact information was obtained to assess outcome in person (eg, clinic follow‐up) or by telephone or telemedicine between 30 and 90 days from the incident stroke. The mRS score was obtained at 30 days (+7 days allowed to establish contact) and then at 90 days (+14 days allowed for conducting the mRS). In patients who were still in the acute care facility at day 90, an in‐hospital mRS score was obtained. Patients who were deceased before day 90 were included and received an mRS score of 6. Patients who were lost to follow‐up received the last recorded mRS or an mRS derived from the last documented neurological evaluation as their final score. The NIHSS score and the mRS scores between days 30 and 90 were transmitted to Dr Foster. The NIHSS and mRS information was linked to the administrative database submitted by each hospital for the larger Global Comparators project using each hospital's pseudoidentifier and admission data, allowing 1 day either way.

Waiver of Consent

All patients admitted to the hospital for diagnosis and management of AIS were entered in the acute stroke quality improvement database at our respective institutions. These data were collected for quality improvement purposes and were deidentified for the purpose of this project. Furthermore, only data available in the medical record were available for abstraction; therefore, each respective institutional review board either had already waived the need for informed consent and approved such type of database analysis or expedited this waiver prior to project commencement on April 1, 2012.

Statistical Analysis

Medians and interquartile ranges (IQRs) for NIHSS scores were derived for the country groups of hospitals and compared using a 2‐sample comparison of medians test. Outcomes were tabulated by center. Because the mRS is ordinal, the association between 30‐ and 90‐day mRS scores was assessed using the γ statistic and polychoric correlation coefficients. To evaluate whether the effect of patient characteristics on outcome varied by length of follow‐up or type of outcome, logistic regression models were fitted for each outcome with NIHSS (continuous or as a category by NIHSS score: mild, 0–6; moderate, 7–16; severe, 17–40). This was done for all records with complete NIHSS and mRS data. The earliest outcome time point considered in the analyses was 7 days. We repeated the regression in the subset that could be linked to the administrative data and included age (continuous), sex, and CMI (continuous).

The performance of the regression models was summarized using the c‐statistic and the Hosmer‐Lemeshow statistic. The c‐statistic assesses the model's discrimination—its ability to predict a higher risk of death for patients who died than for patients who lived—and values >0.8 are often considered to represent good discrimination. The Hosmer‐Lemeshow statistic assesses the model's calibration, or how accurate the predicted risks are when patients are divided into groups of 10; by convention, P>0.05 suggests good discrimination.

The marginal proportion of explained variation was estimated for each variable and outcome with the matched data. This is the contribution of each variable to the variation explained by the model when the variable is considered by itself (ie, without any other variables included). We used bootstrapping to enable significance testing for comparison of proportion of explained variation between predictors. All analyses were run using SAS version 9.2 (SAS Institute). A SAS macro for the proportion of explained variation was obtained from the Internet.20

Results

There were 1034 admissions for AIS captured during the project period. Of these, 614 records (59%) in 13 hospitals had valid NIHSS and 30‐ and 90‐day mRS scores recorded; furthermore, 507 of 614 (82.6%) could be linked to the administrative records to obtain CMI, length of stay, and in‐hospital death status. Table 1 summarizes the total number of AIS patients admitted over the duration of the pilot study by each participating hospital, with outcomes among the sample. Table 2 demonstrates distribution of mRS scores at 30 and 90 days. Comparison of demographic data and comorbidities for the patients with and without missing NIHSS and mRS values showed no significant differences except for those patients with missing NIHSS data, who were, on average, 3.7 years younger than those with no missing NIHSS data (P=0.02).

Table 1.

Per‐Center Enrollment and Outcomes in the Stroke GOAL Pilot Project (n=614)

Patients (N)	Countrya	NIHSS Score†	Death7 (n)	Death (n)	Death Using mRS30 (n)	Death Using mRS90 (n)	mRS30<2 (n)	mRS30≤2 (n)	mRS90<2 (n)	mRS90≤2 (n)
89	US	4 (2–7)	3	5	11	15	28	36	31	41
82	US	6 (3–14)	5	5	10	12	26	36	31	36
81	UK	9 (3–15)	8	16	18	20	23	24	22	26
74	US	4 (2–14)	10	13	16	24	25	34	27	33
61	Other	6 (2–14)	1	4	6	10	18	19	19	20
56	UK	2 (1–9.5)	1	2	2	2	32	37	35	40
40	Other	4 (2–9)	0	0	3	4	15	23	24	26
39	Other	5 (2–13)	1	1	5	6	6	16	6	18
35	Other	6 (1–14)	0	0	9	9	16	18	17	19
27	UK	6 (2–10)	0	2	1	2	14	15	14	17
16	UK	5 (2.5–18)	1	1	1	2	2	4	7	9
12	US	12.5 (6–19.5)	0	0	2	2	4	5	4	5
2	Other	20 (19–21)	0	0	1	1	0	0	0	0

Death indicates overall in‐hospital mortality; death7, mortality within 7 days; mRS30, modified Rankin Scale score at 30 days; mRS90, modified Rankin Scale score at 90 days; NIHSS, National Institutes of Health Stroke Scale score.

^aOther includes Belgium, Italy, and Netherlands.

NIHSS score is expressed as median (interquartile range).

Table 2.

Distribution of 30‐ and 90‐Day Modified Rankin Scale Scores (n=614)

mRS	30‐Day Count	% of 30‐Day Total	90‐Day Count	% of 90‐Day Total
0	95	15.4	129	21.0
1	114	18.5	108	17.6
2	58	9.4	53	8.6
3	85	13.8	83	13.5
4	99	16.1	83	13.5
5	78	12.7	49	8.0
6	85	13.8	109	17.7

mRS indicates modified Rankin Scale score.

Stroke severity was similar between the country groups of hospitals included in this analysis (median NIHSS score: United Kingdom: 5 [IQR 2–13], n=180; United States: 5 [IQR 2–13], n=257; other countries: 5 [IQR 2–13], n=177; 2‐sample comparison of medians P=0.11). The counts of in‐hospital deaths within 7 days were low (by country group: n=10, United Kingdom; n=18, United States; n=2, other countries).

The 30‐ and 90‐day mRS scores showed very high correlation (ρ statistics were at least +0.90, for which 1.0 equals perfect association), both overall and when stratified by severity (using mild, moderate, and severe categories). Correlation was highest (ρ=0.97) for severe strokes. Results were unchanged if death was excluded. Furthermore, there was high agreement between 30‐ and 90‐day mRS scores (weighted κ=0.82).21

NIHSS score dominated the models for all outcomes, particularly for 7‐day in‐hospital death. Table 3 provides the Hosmer‐Lemeshow and c‐statistics and whether each covariate was significant at P<0.05 for each outcome.22

Table 3.

Summary of C‐statistics and Covariate Significance

Outcome	Outcomes (n)	C‐Statistic	Hosmer‐Lemeshow Statistic (P Value)	NIHSS (P value)	Age (P value)	Sex (P value)	CMI (P value)
Death7	30	0.897	7.6 (0.477)	<0.0001	0.152	0.002	0.812
Death	49	0.880	5.5 (0.666)	<0.0001	0.040	0.033	0.168
Death using mRS30	66	0.854	3.8 (0.873)	<0.0001	0.003	0.003	0.639
Death using mRS90	88	0.814	5.6 (0.688)	<0.0001	0.0001	0.042	0.110
mRS30 0 or 1	171	0.819	17.2 (0.029)	<0.0001	0.0003	0.068	0.839
mRS90 0 or 1	190	0.826	15.2 (0.055)	<0.0001	0.001	0.183	0.170

CMI indicates comorbidity index; Death, overall in‐hospital mortality; death7, in‐hospital mortality within 7 days; mRS30, modified Rankin Scale score at 30 days; mRS90, modified Rankin Scale score at 90 days; NIHSS, National Institutes of Health Stroke Scale score.

Calibration was good in each case (Hosmer‐Lemeshow P>0.05), although less so for mRS scores <2. Discrimination was highest for 7‐day deaths. Fitting severity as a category resulted in the loss of an appreciable amount of information compared with including it as raw scores.

The results for proportion of explained variation agreed with the P values from the regression (Table 4). They demonstrate that NIHSS score dominates the variation in all outcomes explained by the model, with age as the second most important of the 4 variables. NIHSS score was significantly more important than comorbidity for all outcomes (P<0.001 for all comparisons) and was more important than the 3 other covariates for all outcomes (P<0.01 for all comparisons). CMI had the greatest proportion of explained variation for 90‐day death. Some of CMI's marginal effect is explained by the other variables when they are added to the model. Overall, age, sex, NIHSS score, and CMI explain up to about a quarter of the variation in the outcome but less for 7‐day in‐hospital death.

Table 4.

Marginal Proportion of Explained Variation Per Covariate for Each Outcome

Outcome	NIHSS (%)	Age (%)	Sex (%)	CMI (%)	Model as a Whole
In‐hospital death by day 7	11.3	1.2	1.4	0.7	16.6
In hospital death all	21.9	3.5	0.3	2.7	25.8
Death at 30 days	21.0	4.6	0.7	1.9	25.3
Death at 90 days	19.9	6.5	0.1	4.0	25.3
Good outcomea at 30 days	22.2	6.1	1.2	1.1	27.2
Good outcomea at 90 days	25.0	5.8	0.8	2.6	29.1

CMI indicates comorbidity index; NIHSS, National Institutes of Health Stroke Scale score.

^aGood outcome equals a modified Rankin Scale score 0 or 1.

Discussion

Results from this prospective multinational hospital‐based collaboration demonstrated (1) that 30‐day mRS score is a valid proxy for long‐term functional outcome (90‐day mRS score) after ischemic stroke, (2) that stroke severity is an essential predictor of poststroke outcomes in comparisons of regional and national stroke systems of care, and (3) that additional predictors are required to explain the remaining variability in stroke outcomes.

The Stroke GOAL program initially analyzed administrative data from all collaborating hospitals. These data relied on outcome measures that were limited to in‐hospital mortality, length of stay, and 30‐day readmission. Due to significant differences in care systems, these outcome measures were not reliable indicators for comparison of poststroke outcomes across multinational samples. Length of stay, for example, varies widely across countries and regions and affects in‐hospital mortality and 30‐day readmission. In England, for example, some rehabilitation is often done in the acute hospital, leading to longer mean stays than in the United States. In addition, differences in referral patterns make in‐hospital mortality unreliable because some centers transfer patients more frequently to outside facilities than others. Because the Global Comparators database includes records only for participating hospitals, it cannot capture posttransfer activity.

We selected the mRS as an outcome measurement in this multinational sample. It is widely available in all languages throughout our network, and it is reliable and validated using face‐to‐face interviews and telephone interviews.12, 23, 24, 25, 26, 27 Since its first report, the scale has become the most widely accepted clinician‐reported measure of global disability for evaluating recovery from stroke28 and is often used as a primary end point in randomized clinical trials of emerging acute stroke treatments.10, 28 Our data demonstrate that the mRS is a valid measurement of poststroke outcome and that measurement of mRS score at 90 days provides only a little additional information above the mRS score at 30 days. This is consistent with previously published study‐specific data14, 17 and indicates that 30‐day mRS score could be used as a reliable proxy for a long‐term functional outcome measure in future studies of poststroke outcomes in large multinational collaborations.

In this analysis, the numbers of in‐hospital deaths were too few over the course of 2‐month enrollment to allow comparisons between hospitals. In contrast, analyzing good outcomes seemed viable at 30 or 90 days using the mRS; therefore, functional poststroke outcome assessment may offer a more reliable measure of hospital‐based stroke outcomes10, 28 versus in‐hospital mortality when using multiple data sets with considerable degrees of variability.

Including stroke severity measured as an NIHSS score into stroke outcome models is becoming a standard statistical approach in planning and implementation of randomized clinical trials of stroke.29, 30 Furthermore, a large‐scale, outcome‐based study of AIS reaffirmed the importance of adding a stroke severity measure such as NIHSS score to a hospital 30‐day model because it considerably improved model discrimination and changed the mortality performance hospital ranking,31 a major practical implication for setting the standard in hospital quality data analysis. Our data further support these advances in applied stroke outcome prediction modeling by demonstrating that inclusion of NIHSS score into statistical modeling uniformly affected prediction of outcomes. Moreover, NIHSS score appears to be a far more important predictor than comorbidity score (ie, CMI), previously thought to be an important contributor to outcomes. Because previously used measures such as mortality, length of stay, and readmission rate for stroke patients cannot be adequately corrected for stroke severity, they have only limited utility for comparison of stroke outcomes.19 This issue has become central in developing properly risk‐adjusted outcome measures for stroke to ensure quality care.32 Our findings firmly support this development and suggest that all future comparisons of the hospital‐level performance and outcomes in stroke must include stroke severity measured by the NIHSS as a critical predictor of functional outcomes.

Study Strengths and Limitations

The strengths of this analysis are (1) the prospective nature of this international collaboration, (2) the prespecified data collection and statistical analysis plan, (3) the inclusion of a large spectrum of hospital‐based clinical data matched with the administrative database, and (4) the use of previously validated tools.

Limitations of this analysis are related to its pilot nature, including a relatively small number of participants and a substantial proportion of missing data. We were unable to compare model performance by country because of a small number of participating centers. We were also not able to account for clustering by hospital because of the limited number of outcomes per hospital. Furthermore, the study was limited by the inability to match the patient data to the administrative sample in all cases and to monitor that all patients at each center were assessed with the NIHSS and the mRS. In addition, the timeline of NIHSS score assessment in acute stroke may be an important correlate of outcome, and in this study, the measure was obtained within 24 hours. Although this is common in data sets using administrative data, it limits generalizability. Nevertheless, the significance of NIHSS score for stroke outcome models is so strong that there is a little doubt that our findings should be replicable in research databases and other large samples.28

Another limitation is that the mRS score was not available for all patients; therefore, selection bias cannot be excluded. Given the available data, imputation of the missing data also was not feasible; however, we found similar results in a large research database (VISTA), indicating that 30‐ and 90‐day mRS scores were very strongly correlated and maybe used interchangeably.14, 17 Using the 30‐day mRS score would make stroke outcomes measurements easier to complete in general stroke populations; small time windows are frequently challenging in complex stroke populations, for which rehabilitation and other medical demands limit accessibility for follow‐up at specific time points.

Disclosures

Dr Rost is funded in part by the National Institutes of Health. Dr Bottle: receives funding through the Dr Foster Unit at Imperial College, which is largely funded by a research grant from Dr Foster, a private health care information company. He also receives grants from the National Institute for Health Research. Dr Jin‐Moo Lee is funded by the National Institutes of Health, the Alzheimer's Association, and the Hope Center for Neurological Disorders. Dr Marc Randall has nothing to disclose. Mr Steven Middleton is an employee of Dr Foster. Dr Gabriel Rinkel has nothing to disclose. Dr Thomas Hemmen is in part funded by the National Institutes of Health and is a Consultant for Merck Co. Dr Louise Shaw has nothing to disclose. Dr Vincent Thijs is funded by FWO Flanders.

(J Am Heart Assoc. 2016;5:e002433 doi: 10.1161/JAHA.115.002433)