Development and validation of a simplified Stroke–Thrombolytic Predictive Instrument

Abstract

Objectives:

The Stroke–Thrombolytic Predictive Instrument (Stroke-TPI) predicts the probability of good and bad outcomes with and without recombinant tissue plasminogen activator (rtPA). We sought to rebuild and externally validate a simpler Stroke-TPI to support implementation in routine clinical care.

Methods:

Using the original derivation cohort of 1,983 patients from a combined database of randomized clinical trials (NINDS [National Institute of Neurological Disorders and Stroke] 1 and 2; ATLANTIS [Alteplase Thrombolysis for Acute Noninterventional Therapy in Ischemic Stroke] A and B; and ECASS [European Cooperative Acute Stroke Study] II), we simplified the Stroke-TPI by reducing variables and interaction terms and by exploring simpler (3- and 8-item) stroke severity scores. External validation was performed in the ECASS III trial (n = 821).

Results:

The following 6 variables were most predictive of good outcomes: age, systolic blood pressure, diabetes, stroke severity, symptom onset to treatment time, and rtPA therapy. Treatment effect modifiers included onset to treatment time and systolic blood pressure. For the models predicting a bad outcome (modified Rankin Scale [mRS] score ≥5), significant variables included age, stroke severity, and serum glucose. rtPA therapy did not change the risk of a poor outcome. Compared with models using the full NIH Stroke Scale, models using the 3-item severity score showed similar discrimination and excellent calibration. External validation on ECASS III showed similar performance (C statistics 0.75 [mRS score ≤1] and 0.80 [mRS score ≤2]).

Conclusion:

A simpler model using a 3-item stroke severity score, instead of the 15-item NIH Stroke Scale, has similar prognostic value and may be easier to use in routine care. Future studies are needed to test whether it can improve process and clinical outcomes.

Thrombolytic therapy improves patients' functional outcomes after acute ischemic stroke (AIS),¹ and is a guideline-endorsed, Class 1A recommendation.^2,3 Despite this, more than 40% of AIS patients ideal for reperfusion therapy remain untreated.⁴ Among the challenges in promptly treating patients is uncertainty regarding whether the risks of treatment outweigh the benefits in any given patient. While multiple prediction models have been developed to individualize prognosis,^5–11 these have not been routinely implemented in clinical care.

Several barriers have limited the use of predictive models in stroke care. For example, most prognostic models for AIS have been derived only on cohorts receiving thrombolytic therapy, and therefore are unable to provide estimates of treatment benefit. In addition, model complexity can pose a practical barrier, particularly when the model requires a detailed neurologic examination (e.g., the NIH Stroke Scale¹² [NIHSS]) in the emergency department.

The Stroke–Thrombolytic Predictive Instrument (Stroke-TPI) is the only model for predicting outcomes with and without rtPA developed using patient-level data from the first 5 randomized clinical trials using standard-dose rtPA.⁶ It has been validated on an external cohort of patients receiving thrombolytic therapy,¹³ but because it was derived on the totality of randomized data testing standard-dose rtPA available at the time, outcome predictions with and without thrombolysis could not be validated. Thus, we sought to develop a simpler model that could be more easily implemented in routine clinical care and to externally validate its predictions on more recent trial data.

METHODS

Conceptual overview for model development.

Defining the outcomes.

The original Stroke-TPI predicted the probability of a normal/near-normal outcome (modified Rankin Scale [mRS] score = 0 or 1) and severe disability/death (mRS score = 5 or 6) both with and without thrombolytic therapy. Before this analysis, a qualitative research study was conducted with numerous patient, caregiver, and provider focus groups to evaluate these outcome measures for informing care decisions (unpublished, 2014). Based on these qualitative studies, we incorporated a new threshold at mRS score = 0–2 to capture the independent outcome, in addition to those thresholds previously included in the original Stroke-TPI.

Defining the optimal measure of stroke severity.

The most common measure of stroke severity is the NIHSS, a 15-item neurologic assessment that reproducibly quantifies the extent of neurologic disability.¹² While excellent training programs exist,¹⁴ the scale is often not used in routine clinical care; more than half (54.9%) of patients within the Get with the Guidelines–Stroke registry do not have an NIHSS score documented.¹⁵ To simplify the implementation of the revised Stroke-TPI tool, we explored 2 simpler, previously developed stroke severity scores, an 8-item and a 3-item measure of stroke severity. The 8-item severity score¹⁶ uses the following neurologic findings from the NIHSS: (1a) level of consciousness; (2) gaze; (3) visual fields; (4) facial paresis; (6a) motor–leg right; (6b) motor–leg left; (9) language; and (10) dysarthria. The 3-item score¹⁷ uses disturbance of consciousness (none = 0, mild = 1, and severe = 2 points), gaze/head deviations (absent = 0, incomplete = 1, and forced gaze/head deviation = 2 points), and hemiparesis (absent = 0, moderate = 1, and severe = 2). These scores were mapped from individual items within the full NIHSS. We assumed that reduced-item severity scores validated in only the prehospital setting would be as reliable in the more controlled setting of the hospital, particularly since the items were drawn from the NIHSS, which is applied within this setting.

Minimizing the number of predictor variables in the model.

For a model to be implemented in routine clinical care, it is important to minimize the required data elements. Model reduction was performed with a combination of clinical and statistical judgment. In addition, treatment interaction factors, the variables that modify the effects of therapy, were carefully evaluated. Treatment interactions are distinguished from prognostic factors, which affect the probability of outcome with or without therapy, but not the proportional effects of therapy. In addition to longer onset to treatment time (OTT), which had been known to modify the effectiveness of therapy, the original Stroke-TPI project identified 3 other variables that appeared to modify the effect of therapy. Specifically, thrombolysis appeared less beneficial when initial systolic blood pressure was higher, the patient was male, and the patient had a prior stroke.

In this iteration of the Stroke-TPI, we assumed a more conservative approach to effect modification, eliminating those variables without strong external evidence.^18,19 We reviewed the literature to assess the consistency of observed interactions with existing evidence, including but not limited to the reported subgroup analyses from both the European Cooperative Acute Stroke Study (ECASS) III²⁰ and the Third International Stroke Trial (IST-3) reports.²¹ Based on these comparisons, as well as statistical and clinical considerations, we included the 2 interaction terms evaluated as being most reliable (OTT and systolic blood pressure) and eliminated 2 others (sex and prior stroke). In addition, while the prior Stroke-TPI included models both with and without the ASPECT (Alberta Stroke Program Early CT) score, this simplified version did not include the variable, given the difficulty of obtaining this information in most settings and the fact that it does not appear to influence treatment effect for IV thrombolysis.^22,23

Data source: The Stroke-TPI database.

We used a combined, patient-level dataset from 5 clinical trials (National Institute of Neurological Disorders and Stroke [NINDS] 1 and 2, Alteplase Thrombolysis for Acute Noninterventional Therapy in Ischemic Stroke [ATLANTIS] A²⁴ and B,²⁵ and ECASS II²⁶) that had previously been used to develop the original Stroke-TPI. This dataset included common clinical characteristics and outcomes, including 90-day mRS scores. The only exception was that item-by-item detail for the NIHSS was not available for the NINDS trials, requiring multiple imputation in this dataset to estimate the scores for the reduced stroke severity scales. Of note, the range of OTT varied substantially across these randomized trials, with patients being treated between 58 and 360 minutes after symptom onset, enabling a broad range of treatment times to be modeled.

Validation dataset: ECASS III.

ECASS III was a double-blind, parallel-group trial that enrolled 821 patients from 130 sites in 19 European countries. Criteria for study entry were similar to other studies in the derivation cohort of the Stroke-TPI, with some notable exceptions. In particular, because thrombolytic therapy was standard of care at the time of the trial when administered before 3 hours from symptom onset, patients in ECASS III were enrolled only if able to receive study drug in the window between 3 and 4.5 hours after the onset of symptoms. In addition, the combination of both prior stroke and diabetes was an exclusion from enrollment.

Statistical analysis.

Unadjusted comparisons of the patients and outcomes across the trials were performed with t tests for continuous data and χ² for categorical data. As described above, 3 binary outcomes were modeled: 90-day mRS score ≤1 (normal/near-normal outcome), 90-day mRS score ≤2 (independent outcome), and 90-day mRS score ≥5 (severe disability/death outcome). Multivariable logistic regression models were used to predict each outcome. Variable selection relied on the previously published Stroke-TPI,⁶ with no additional data exploration, incorporating the changes specified above: use of an alternative independent outcome threshold (mRS score ≤2); removal of 2 treatment interaction terms (tPA × sex and tPA × prior stroke); and testing reduced-item severity scores (including 3-item and 8-item neurologic assessments) against the full 15-item NIHSS on a subset of the data (ATLANTIS A and B and ECASS II). After these changes were made, we removed Stroke-TPI variables that were no longer significantly prognostic. A total of 9 separate models were developed using logistic regression, one for each of 3 outcomes using each of 3 stroke severity scales. The performance of each model was assessed by the C statistic and calibration plots, and was compared between the full and reduced models. Individual patient predictions with the different models were examined for clinical plausibility and compared across models. Model-to-model comparisons were made using Spearman rank correlation and integrated discrimination improvement (IDI)²⁷ indices.

For the 3-item severity score model, bootstrap sampling was used both to estimate 95% prediction intervals and to internally validate the model (i.e., correct for overoptimism). We drew 500 bootstrapped samples across the multiple imputed datasets of 1,983 patients. To compute 95% prediction intervals, new parameter estimates for the model were obtained, and a new C statistic was estimated, for each of the 500 bootstrapped samples. Optimism-corrected C statistics were calculated by applying each of the 500 models to the original sample. All analyses were conducted using SAS version 9.3 TS Level 1M2 in XP_PRO platform (SAS Institute Inc., Cary, NC); IDI was calculated using a program²⁸ run on R Studio (version 0.98.507, 2009–2013; RStudio, Inc., Boston, MA) via R software (version 3.1.0).²⁹

External validation.

Using fixed coefficients, we generated predictions on the ECASS III population for both the normal/near-normal outcome (mRS score 0 or 1) and the independent outcome (mRS score 0–2). Because baseline serum glucose was not available for ECASS III patients, a key patient characteristic associated with a bad outcome, predictions for the severe disability/death outcome (mRS score 5 or 6) could not be externally validated. Thus, a total of 6 models were evaluated (2 outcomes by 3 severity scores). For each model, performance on the validation cohort was assessed using the C statistic and by examining calibration plots, and calculating the mean (absolute) bias across quintiles. While model derivation was performed in Boston, model validation was performed independently in Boehringer, Germany.

Net benefit.

To compare the potential benefits of using individualized predictions against a treatment recommendation based only on time window, we examined the distribution of the absolute benefit (predicted probability of a good outcome with rtPA minus predicted probability of a good outcome without rtPA) in the derivation dataset across each 30-minute OTT period. This analysis describes the proportion of patients expected to achieve benefit of varying magnitudes with treatment at a given time (i.e., to be harmed by treatment, to have an absolute risk reduction of 0% to 5% [i.e., number needed to treat >20] and an absolute risk reduction of >5% [number needed to treat <20]).

RESULTS

There were 1,983 patients in the Stroke-TPI dataset (table 1). Given similar performance of the 3 stroke severity measures, table 2 describes the models predicting normal/near-normal outcomes (mRS score ≤1) and independent outcomes (mRS score ≤2) with the 3-item stroke severity measure. When the prior stroke and sex treatment interaction terms were removed from the models, the main effects of these variables were no longer significant and they were also removed. Thus, only the following 6 variables were included in the models for normal/near-normal outcomes and independent outcomes: age, systolic blood pressure, history of diabetes, stroke severity, symptom onset to treatment time, and thrombolytic therapy. Models using both the full NIHSS and the 8-item severity scale are shown in tables e-1 and e-2 on the Neurology® Web site at Neurology.org. Table 2 shows the models predicting severe disability/death outcome (mRS score ≥5). The 3 variables included in the model were the same as the original Stroke-TPI: age, stroke severity, and serum glucose. Thrombolytic therapy did not increase or decrease the risk of a poor outcome, indicating that the benefits of reperfusion and the harms of intracerebral hemorrhage are approximately balanced for this outcome.

Table 1.

Patient characteristics

Table 2.

Models using 3-item stroke severity score

Model correlations, reclassification, and internal validation.

We compared the predictions of individual patients across models using the 3 different severity scores. These predictions were found to be highly correlated. The Spearman rank correlations between the models using the full NIHSS and the 3-item stroke severity score were 0.80, 0.80, and 0.84 (all with p < 0.0001) for predicting mRS scores ≤1, ≤2, and ≥5, respectively. The IDI²⁷ indices of the models using the full NIHSS compared to that using the 3-item scale were 0.032 (−0.010 to 0.073), 0.013 (−0.025 to 0.051), and 0.026 (−0.019 to 0.071) for predicting mRS scores ≤1, ≤2, and ≥5, respectively, suggesting that the full NIHSS did not significantly improve the classification of patients with good outcomes.

The C statistics for all 9 models are shown in table 3, and all models had similar discrimination (C statistics between 0.75 and 0.80). On bootstrap validation, the models using the 3-item score had the following optimism-corrected C statistics: mRS score ≤1, 0.76 (95% prediction interval [PI]: 0.75–0.79); mRS score ≤2, 0.78 (95% PI: 0.77–0.80); and mRS score ≥5, 0.76 (95% PI: 0.73–0.79). When calibration curves were examined across quintiles, bias of the model on the internal database was minimal, and did not appear to substantially depend on which stroke severity score was used. Figure 1A shows the calibration curve on the derivation dataset for the most parsimonious model using the 3-item severity score.

Table 3.

C statistics for all 9 models

Figure 1. Calibration plots for 3-item severity score model.

These plots show the proportion of patients with each outcome as both predicted by the model and as observed, for equal-sized quintiles in the derivation population (A) and validation population (B). Calibration is not shown for the severe disability/death outcome for the validation population, since ECASS III did not collect values for serum glucose at baseline. ECASS = European Cooperative Acute Stroke Study; mRS = modified Rankin Scale.

External validation.

Discrimination and calibration were evaluated in ECASS III. All 6 models evaluated showed near identical discrimination as compared with that seen in the derivation dataset (table e-3). Calibration was excellent for the model predicting normal or near-normal outcome (mRS score ≤1) when the 15-item score was used. However, observed outcomes in ECASS III were slightly better than predicted for all other models. Chi-square goodness-of-fit tests across the quintiles showed no significant differences between observed or expected for any of the models using the full NIHSS or the 3-item score, although they were significant for the 8-item score (table e-3). Figure 1B shows the calibration curve on the validation dataset for the most parsimonious model using the 3-item severity score.

Net benefit.

The distribution of predicted benefit over time for the 3-item severity model are shown in figure 2 for both the model predicting normal/near-normal outcomes (mRS score ≤1) and the model predicting independent outcomes (mRS score ≤2). As can be seen, there are a small number of patients that are predicted to be harmed (i.e., have treatment unfavorable characteristics), even with an OTT of less than 4.5 hours. Conversely, there are some patients predicted to have substantial benefit beyond this time window.

Figure 2. Distribution of net benefit across different symptom onset to treatment time windows.

These figures illustrate the change in individual predicted net benefit (probability of a good outcome with minus without recombinant tissue plasminogen activator) in patients in the derivation database (N = 1,983) across different treatment onset to symptom times. As can be seen, there is variation in predicted net benefit at all time windows, with predicted harm in some patients treated before the 4.5-hour window expires and predicted benefit in some patients treated after this window. mRS = modified Rankin Scale; OTT = onset to treatment time.

DISCUSSION

To support the use of personalized estimates of the benefits of thrombolytic therapy in the treatment of AIS, we simplified a well-established prediction model, the Stroke-TPI. We found that removing several interaction terms and predictor variables, and using a reduced-item stroke severity measure, did not substantially degrade the discriminatory performance of the Stroke-TPI model. (For comparison, the C statistics in the original Stroke-TPI were 0.788 and 0.775 for the models predicting mRS scores ≤1 and ≥5, respectively.) Furthermore, this study represents the first external validation of the Stroke-TPI on a large randomized dataset. These steps should increase confidence and ease of use of this modified Stroke-TPI, particularly in settings where the full NIHSS is not routinely assessed.

Multiple stroke predictive models have been developed to provide prognostic information, including the Stroke-TPI, iScore, ASTRAL, DRAGON, and others.^5–11 These models share several important features. In particular, prognosis in all models is largely predicted by stroke severity, age, and time to treatment. Additional variables provide further improvement in prognostic performance, yet with diminishing returns. While previous models include the full NIHSS, a simpler score, such as the 3-item version we use, may enable a broader range of clinicians to estimate treatment benefits and support broader adoption of personalized, evidence-based treatment. Also, unlike the other prognostic models, the Stroke-TPI was developed on a combined database of randomized clinical trials and has now been validated in an additional large randomized trial. This is important because the estimates of treatment benefit are unbiased due to the randomization process used in each trial.

A critical element needed to differentiate patients likely or unlikely to benefit is reliably identifying factors that modify the effects of tPA. Four factors have been shown with randomized data to produce statistically significant treatment modification.⁶ The following characteristics have been shown to be associated with less benefit: longer symptom onset to treatment time, male sex, higher systolic blood pressure, and prior stroke. While the original Stroke-TPI included all these variables in treatment interaction terms, it should be recognized that inclusion of such terms has many of the same pitfalls as classic subgroup analysis. As yet, there is little rigorous guidance to help inform when such interaction terms are likely to be credible, although both clinical criteria (based on prior external evidence) and statistical criteria should be used.^30–32 In this iteration of the Stroke-TPI, we removed previously included interaction effects, pending further confirmation.

Our study should be interpreted in the context of several potential limitations. This database does not incorporate all randomized data currently available. After the development of the Stroke-TPI, several additional clinical trials were completed, most notably ECASS III,^20,33 which enrolled patients between 3 and 4.5 hours only, and IST-3,²¹ which included patients in the 0- to 6-hour window. ECASS III was used as a validation dataset, but we did not have access to IST-3 for these analyses. The Stroke-TPI is intended as a prognostic tool that may be of use for shared decision-making and informed consent. It is not intended as a diagnostic tool or to determine eligibility criteria for consideration of tPA. Selection of patients for lytic therapy should be based on the presence of a potentially disabling neurologic deficit evident on the general neurologic examination, and not on any particular score on the full NIHSS or the shorter scale used in this study.

As with all prediction models, prognostic outputs represent “evidential probabilities” conditional on the databases used and the mathematical assumptions of the model. Other models may yield different predictions at the individual patient level.³⁴ While the use of randomized controlled trials minimizes bias in estimating the benefits of rtPA, it is unknown whether similar results would be obtained in the broader population of patients presenting with AIS ineligible for the clinical trials. In addition, our validation dataset included patients treated only in the 3- to 4.5-hour time window. Finally, the clinical impact of this model on treatment rates and outcomes has not been tested.

Nevertheless, because estimates of absolute benefit vary substantially across the population, even among patients treated at the same symptom onset to treatment time, this tool can support decision-making. By having estimates of treatment benefit, clinicians can not only better clarify the benefits of treatment in their own minds, but can also share these estimates with patients as they jointly decide whether or not to treat. The next step is to test the implementation of this new risk model in clinical care to examine its effect on treatment rates, time to treatment, and, most importantly, patient outcomes.

GLOSSARY

AIS

acute ischemic stroke

ATLANTIS

Alteplase Thrombolysis for Acute Noninterventional Therapy in Ischemic Stroke

ECASS

European Cooperative Acute Stroke Study

IDI

integrated discrimination improvement

IST-3

Third International Stroke Trial

mRS

modified Rankin Scale

NIHSS

NIH Stroke Scale

NINDS

National Institute of Neurological Disorders and Stroke

OTT

onset to treatment time

PI

prediction interval

rtPA

recombinant tissue plasminogen activator

TPI

Thrombolytic Predictive Instrument

AUTHOR CONTRIBUTIONS

D.M.K.: drafting/revising the manuscript for content, including medical writing for content, study concept or design, analysis or interpretation of data, statistical analysis, study supervision or coordination, obtaining funding. R.R.: drafting/revising the manuscript for content, including medical writing for content, analysis or interpretation of data, statistical analysis. C.D.: drafting/revising the manuscript for content, including medical writing for content, analysis or interpretation of data. P.G.J.: drafting/revising the manuscript for content, including medical writing for content, analysis or interpretation of data. J.L.S.: drafting/revising the manuscript for content, including medical writing for content, analysis or interpretation of data. E.B.: drafting/revising the manuscript for content, including medical writing for content, analysis or interpretation of data, statistical analysis, study supervision or coordination. J.A.S.: drafting/revising the manuscript for content, including medical writing for content, study concept or design, analysis or interpretation of data, statistical analysis, study supervision or coordination, obtaining funding.

STUDY FUNDING

Dr. Kent has received grant support from Genentech and the NIH (UL1 RR025752, R01 NS062153, and R21 NS079826). Dr. Spertus has received grant support from the American College of Cardiology Foundation, NIH, PCORI, Genentech, Abbott Vascular, Amorcyte, EvaHeart, and Lilly.

DISCLOSURE

D. Kent has consulted for W.L. Gore Associates. R. Ruthazer, C. Decker, and P. Jones report no disclosures relevant to the manuscript. J. Saver is an employee of the University of California. The University of California, Regents, receive funding for Dr. Saver's services as a scientific consultant regarding trial design and conduct to Covidien, CoAxia, Stryker, BrainsGate, and St. Jude Medical. Dr. Saver has served as an unpaid site investigator in multicenter trials run by Lundbeck for which the UC Regents received payments on the basis of clinical trial contracts for the number of participants enrolled. Dr. Saver serves as an unpaid consultant to Genentech advising on the design and conduct of the PRISMS trial; neither the University of California nor Dr. Saver received any payments for this voluntary service. The University of California has patent rights in retrieval devices for stroke. E. Bluhmki reports no disclosures relevant to the manuscript. J. Spertus has consulted for United Healthcare, Janssen, Amgen, and Abbott Vascular. He has an equity interest in Health Outcomes Sciences and owns the copyright to the Seattle Angina Questionnaire, Kansas City Cardiomyopathy Questionnaire, and the Peripheral Artery Questionnaire. Go to Neurology.org for full disclosures.