Investigating the Utility of Previously Developed Prediction Scores in Acute Ischemic Stroke Patients in the Stroke Belt

Abstract

Background

To assess the utility of previously developed scoring systems, we compared SEDAN, named after the components of the score (baseline blood Sugar, Early infarct signs and (hyper) Dense cerebral artery sign on admission computed tomography scan, Age, and National Institutes of Health Stroke Scale on admission), Totaled Health Risks in Vascular Events (THRIVE), Houston Intra-arterial Therapy (HIAT), and HIAT-2 scoring systems among patients receiving systemic (intravenous [IV] tissue plasminogen activator [tPA]) and endovascular (intra-arterial [IA]) treatments.

Methods

We retrospectively reviewed all IV tPA and IA patients presenting to our center from 2008–2011. The scores were assessed in patients who were treated with IV tPA only, IA only, and a combination of IV tPA and IA (IV-IA). We tested the ability of THRIVE to predict discharge modified Rankin scale (mRS) 3–6, HIAT and HIAT-2 discharge mRS 4–6, and SEDAN symptomatic intracerebral hemorrhage (sICH).

Results

Of the 366 patients who were included in this study, 243 had IV tPA only, 89 had IA only, and 34 had IV-IA. THRIVE was predictive of mRS 3–6 in the IV-IA (odds ratio [OR], 1.95; 95% confidence interval [CI], 1.30–2.91) and the IV group (OR, 1.71; 95% CI, 1.43–2.04), but not in the IA group. HIAT was predictive of mRS 4–6 in the IA (OR, 3.55; 95% CI, 1.65–7.25), IV (OR, 3.47; 95% CI, 2.26–5.33), and IV-IA group (OR, 6.48; 95% CI, 1.41–29.71). HIAT-2 was predictive of mRS 4–6 in the IA (OR, 1.39; 95% CI, 1.03–1.87) and IV group (OR, 1.36; 95% CI, 1.18–1.57), but not in the IV-IA group. SEDAN was not predictive of sICH in the IA or the IV-IA group, but was predictive in the IV group (OR, 1.54; 95% CI, 1.01–2.36).

Conclusions

Our study demonstrated that although highly predictive of outcome in the original study design treatment groups, prediction scores may not generalize to all patient samples, highlighting the importance of validating prediction scores in diverse samples.

Introduction

Acute ischemic stroke (AIS) remains one of the leading causes of morbidity and mortality worldwide.¹ Treatment for AIS includes intravenous (IV) tissue plasminogen activator (tPA), intra-arterial (IA) therapy, or a combination of IV tPA and IA (IV-IA).^2,3 Despite IV tPA being the only approved treatment for AIS, utilization remains low at approximately 3%–9% in the United States.^4–7 There are many reasons for the underutilization of tPA including lack of public awareness of stroke signs and symptoms, patients presenting outside of the time window, and fear of adverse events associated with tPA, including hemorrhagic transformation and symptomatic intracerebral hemorrhage (sICH).^8–10 There are similar safety concerns for patients undergoing IA and IV-IA. The severity of the adverse events associated with these reperfusion strategies has driven the need to develop scoring mechanisms to identify patients who may be at risk for experiencing an adverse event or a poor outcome.

Several scoring tools have been developed to predict poor outcomes after IV or IA using readily available clinical variables. The Totaled Health Risks in Vascular Events (THRIVE) score has been used to predict poor functional outcome (defined as modified Rankin scale [mRS] 3–6) in IV, as well as IA-treated patients.^11,12 The Houston Intra-arterial Therapy (HIAT) score and HIAT-2 score were designed to predict poor functional outcome (defined as mRS 4–6) in IA-treated patients.^13,14 The SEDAN score, named after the components of the score (baseline blood Sugar, Early infarct signs and (hyper) Dense cerebral artery sign on admission computed tomography scan, Age, and National Institutes of Health Stroke Scale on admission), was designed to predict sICH in IV-treated patients.¹⁵ As these scoring systems have been designed in different geographic and treatment populations, the utility of these scales in the Stroke Belt population remains largely unknown. We hypothesized that patients in the Stroke Belt would be younger and have more uncontrolled vascular risk factors¹⁶ potentially altering their score on existing scoring systems. The aim of this study was to systematically investigate the utility of each of these scoring systems in our Stroke Belt sample of patients treated with IV, IA, and IV-IA at a tertiary care center located in the United States Stroke Belt to see if the scoring systems remain predictive of outcomes in these different patient populations.

Methods

Study Population and Variable Definition

Using our prospectively collected stroke registry, we retrospectively identified consecutive patients who presented with AIS to our tertiary stroke center from 2008–2011 who were treated with IV tPA, IA or IV-IA. This study was approved by the institutional review board. Patients were excluded if they were transferred to our facility from another institution. Admission demographic and clinical data, as well as outcome measures, were retrospectively extracted from the medical record. Clinical characteristics included information from the clinical examination, stroke severity as measured by the National Institutes of Health Stroke Scale (NIHSS), vital signs, laboratory tests, and imaging information. Trained neurologists who were blinded to other clinical data and the application of the scores did independent review of brain imaging. Patients were classified as being treated with IV tPA, IA, or IV-IA. THRIVE, HIAT, HIAT-2, and SEDAN scores were calculated for each patient (Table 1).

Table 1.

Comparison of scoring mechanisms

Investigator	Scoring system	Designed using patients treated with:	Primary outcome	Points awarded
Flint et al11	THRIVE	IV	mRS 3–6	0 points NIHSS ≤10 2 points NIHSS 11–20 4 points NIHSS ≥21 0 points age ≤59 y 1 point age 60–79 y 2 points age ≥80 y 1 point for history of diabetes 1 point for history of hypertension 1 point for history of atrial fibrillation
Hallevi et al13	HIAT	IA	mRS 4–6	1 point age >75 y 1 point NIHSS >18 1 point admission glucose ≥150
Sarraj et al14	HIAT-2	IA	mRS 4–6	2 point age 60–79 y 4 points age ≥80 y 1 point NIHSS 10–20 2 points NIHSS ≥21 1 point admission glucose ≥150 3 points for ASPECTS score <8
Strbian et al15	SEDAN	IV	sICH	1 point for NIHSS ≥10 1 point for hyperdense cerebral artery sign on admission CT 1 point for early infarct sign on admission head CT 1 point for age >75 y 1 point for admission glucose 145–216 2 points for admission glucose >216

Abbreviations: ASPECTS, Alberta Stroke Program Early Computed Tomography score; CT, computed tomography; HIAT, Houston Intra-arterial Therapy; IA, intra-arterial; IV, intravenous; mRS, modified Rankin scale; NIHSS, National Institutes of Health Stroke Scale; SEDAN, baseline blood Sugar, Early infarct signs and (hyper) Dense cerebral artery sign on admission computed tomography scan, Age, and National Institutes of Health Stroke Scale on admission; sICH, symptomatic intracerebral hemorrhage; THRIVE, Totaled Heath Risks in Vascular Events.

Outcomes were measured at discharge, as long-term data were not available. Outcomes of interest included sICH and poor functional outcome. We used the definitions of poor functional outcome originally used by each scoring system (ie, THRIVE mRS 3–6 and HIAT and HIAT-2 mRS 4–6). THRIVE was originally designed to predict mRS 3–6, and HIAT and HIAT-2 were originally designed to predict mRS 4–6. We used the originally defined outcomes for each score to assess the predictive nature of the scores. The presence of sICH (defined as a type 2 parenchymal hemorrhage with deterioration in NIHSS score of ≥4 points or death¹⁷) was determined by independent review of repeat head computed tomography or magnetic resonance imaging obtained as standard of care after treatment.

Statistics

We compared admission, clinical, and discharge information among patients who received IV, IA, and IV-IA treatments using the chi-square and analysis of variance tests with nonparametric equivalents when appropriate. In addition, we used receiver operating characteristic curves and area under the curve (AUC) values to compare the predictive ability of each scoring system to predict the outcome of interest in the IV, IA, and IV-IA groups. Crude and adjusted logistic regression analyses were conducted to assess the relationship between each score’s primary outcome and the continuous score itself. As this was an exploratory analysis, no adjustments were made for multiple comparisons.¹⁸ An alpha of .05 was set as the level of significance.

Results

Of the 366 patients who met inclusion criteria, 243 were treated with IV, 89 with IA, and 34 with IV-IA. Demographic and baseline clinical information for each of the 3 groups is displayed in Table 2. On an average, patients in the IV group were older than those in the IA and IV-IA groups (65 years vs. 60 years vs. 60 years; P =.0403). The median NIHSS on admission was higher in the IV-IA group than in the IA or IV groups (18 vs. 13 vs. 7; P <.0001). Figure 1 illustrates the performance of THRIVE, HIAT, and HIAT-2 scores in predicting poor functional outcome (AUC curves range from 0.681–0.818) and SEDAN in predicting sICH (AUC curves range from .512–.665) in each of the 3 groups (IV, IA, and IV-IA) in our sample.

Table 2.

Demographic and baseline clinical characteristics of IV, IA, and IV-IA groups

Characteristics	IV (n = 243)	IA (n = 89)	IV-IA (n = 34)	P value
Age, median (range)	65 (20–99)	60 (29–85)	60 (22–93)	.0403
Sex (male), n (%)	110 (45.3)	48 (53.9)	13 (38.2)	.7147
NIHSS on admission, median (range)	7 (0–32)	13 (0–30)	18 (0–30)	<.0001
ASPECT score 8–10, n (%)	129 (76.3)	37 (66.1)	15 (65.2)	.2208
Glucose on admission	112 (73–536)	118 (76–360)	123 (89–216)	.1366
History of diabetes, n (%)	57 (23.5)	21 (25)	8 (23.5)	.9590
History of hypertension, n (%)	183 (75.3)	66 (79.5)	23 (67.7)	.3936
History of atrial fibrillation, n (%)	42 (17.3)	14 (16.9)	9 (26.5)	.4057
Hyperdense MCA sign on initial imaging, n (%)	45 (26.6)	28 (50)	10 (43.5)	.0032
Early ischemic changes on initial imaging, n (%)	44 (26.0)	23 (41.1)	5 (21.7)	.0717
mRS on discharge, median (range)	3 (0–6)	3 (0–6)	3 (0–6)	.7988
mRS 3–6 on discharge, n (%)	135 (57.2)	43 (56.6)	17 (56.7)	.9946
mRS 4–6 on discharge, n (%)	95 (40.3)	33 (43.4)	11 (36.7)	.7967
sICH, n (%)	12 (4.9)	24 (27.6)	11 (32.3)	<.0001
In-hospital death, n (%)	31 (13.1)	10 (13.2)	4 (13.3)	.9995

Abbreviations: ASPECT, Alberta Stroke Program Early Computed Tomography; IA, intra-arterial; IV, intravenous; IV-IA, intravenous tissue plasminogen activator and intra-arterial; MCA, middle cerebral artery; mRS, modified Rankin scale; NIHSS, National Institutes of Health Stroke Scale; sICH, symptomatic intracerebral hemorrhage.

Figure 1.

(A) Comparison of Totaled Heath Risks in Vascular Events (THRIVE) in the 3 groups. (B) Comparison of Houston Intra-arterial Therapy (HIAT) in the 3 groups. (C) Comparison of HIAT-2 in the 3 groups. (D) Comparison of baseline blood Sugar, Early infarct signs and (hyper) Dense cerebral artery sign on admission computed tomography scan, Age, and National Institutes of Health Stroke Scale on admission (SEDAN) in the 3 groups. (Color version of figure is available online.)

Table 3 displays the predictive ability of the continuous scores for each scoring system in each of the 3 groups. Designed for use in IV or IA, we found the THRIVE score to be predictive of mRS 3–6 in the IV-IA group (odds ratio [OR], 1.95; 95% confidence interval [CI], 1.30–2.91; P =.0012) and the IV group (OR, 1.71; 95% CI, 1.43–2.04; P < .0001), but not in the IA group (OR, 1.47; 95% CI, .86–2.53; P =.1593). Designed exclusively for patients undergoing IA therapy, the HIAT score was not only predictive of mRS 4–6 in the IA group (OR, 3.55; 95% CI, 1.65–7.25; P = .0010), but also the IV group (OR, 3.47; 95% CI, 2.26–5.33; P < .0001) and the IV-IA group (OR, 6.48; 95% CI, 1.41–29.71; P =.0162). Also designed to predict poor outcome after IA therapy, we found HIAT-2 to be predictive of mRS 4–6 in the IA group (OR, 1.39; 95% CI, 1.03–1.87; P = .0302) and in the IV group (OR, 1.36; 95% CI, 1.18–1.57; P <.0001), but not in the IV-IA group (OR, 1.58; 95% CI, .90–2.77; P =.1077). The SEDAN score was not predictive of sICH in the IA group (OR, .89; 95% CI, .55–1.45; P =.6515) or the IV-IA group (OR, .66; 95% CI, .25–1.76; P = .4089), but was predictive in the IV group (OR, 1.54; 95% CI, 1.01–2.36; P = .0457). Of the 4 scores, only HIAT-2 used a dichotomized score to predict outcome (HIAT-2 >4). Using this dichotomized score, in our sample, 71.7% of patients with a HIAT-2 score greater than four had a poor functional outcome at discharge (mRS 4–6). The Table S1 in Appendix illustrates how the AUC generated by our sample compares to the AUCs from the original studies.

Table 3.

Odds of the outcome of interest for each scoring system for the IV, IA, and IV-IA groups

Score (primary outcome)	IV, N = 243	IA, N = 74	IV-IA, N = 30
THRIVE (mRS 3–6)	OR, 1.71; 95% CI, 1.43–2.04; P < .0001	OR, 1.47; 95% CI, .86–2.53; P = .1593	OR, 1.95; 95% CI, 1.30–2.91; P = .0012
HIAT (mRS 4–6)	OR, 3.47; 95% CI, 2.26–5.33; P < .0001	OR, 3.55; 95% CI, 1.65–7.25; P = .0010	OR, 6.48; 95% CI, 1.41–29.71; P = .0162
HIAT-2 (mRS 4–6)	OR, 1.36; 95% CI, 1.18–1.57; P < .0001	OR, 1.39; 95% CI, 1.03–1.87; P = .0302	OR, 1.58; 95% CI, .90–2.77; P = .1077
SEDAN (sICH)	OR, 1.54; 95% CI, 1.01–2.36; P = .0457	OR, .89; 95% CI, .55–1.45; P = .6515	OR, .66; 95% CI, .25–1.76; P = .4089

Abbreviations: HIAT, Houston Intra-arterial Therapy; IA, intra-arterial; IV, intravenous; IV-IA, intravenous tissue plasminogen activator and intra-arterial; mRS, modified Rankin scale; SEDAN, baseline blood Sugar, Early infarct signs and (hyper) Dense cerebral artery sign on admission computed tomography scan, Age, and National Institutes of Health Stroke Scale on admission; sICH, symptomatic intracerebral hemorrhage; THRIVE, Totaled Heath Risks in Vascular Events.

Discussion

Thrombolysis rates in the United States remain disappointingly low, partly because of safety concerns of hemorrhagic transformation after treatment. Several prediction scores have been developed to help clinicians identify patients who would be at high risk of hemorrhagic complications and poor outcome. Unfortunately, our data suggest that these scores are not applicable to all geographic patient populations or forms of thrombolysis.

The THRIVE score was originally developed to predict poor functional outcome (mRS 3–6) in IA-treated patients using data from the MERCI (Mechanical Embolus Removal in Cerebral Ischemia) and Multi MERCI trial databases. Patients in these studies were enrolled primarily in the western United States and Canada. The THRIVE score has previously been shown to be predictive of poor outcomes in IV-treated patients in multiple regions of the United States.^{11,12,19–21} Although we hypothesized that THRIVE may not be able to predict poor outcome in Stroke Belt patients because of their young age and their vascular risk factor burden, we found that the THRIVE score was able to predict mRS 3–6 in IV and IV-IA patients in our Stroke Belt sample. It was, however, unable to predict poor outcome in IA-treated patients. This predictive ability of THRIVE in the IV and IV-IA groups could be because of the higher frequency of risk factors (history of diabetes and so forth) in the IV group and the higher baseline NIHSS in the IV-IA group. In addition to a small sample size, the patients in the IA group are younger than those in the IV group and the NIHSS at baseline is lower than that in the IV-IA group. This could contribute to why this score is not predictive of outcomes in the IA group in this sample.

HIAT and HIAT-2 were developed to predict poor functional outcomes (mRS 4–6) in patients undergoing IA therapy. Patients involved in the creation of these scores came from a single center in Houston, Texas. The HIAT score allotted points for age greater than 75, NIHSS greater than 18, and admission glucose greater than 150. We hypothesized that many of our patients would score points for stroke severity and for glucose, potentially making the HIAT score less predictive in our sample. Interestingly, the HIAT score was able to predict mRS 4–6 in all 3 treatment groups. HIAT-2 was the only score that included neuroimaging findings in its prediction model. We did not have a clear hypothesis as to how the Alberta Stroke Program Early Computed Tomography score would affect the ability of HIAT-2 to accurately predict poor outcome. Although HIAT-2 was able to predict mRS 4–6 in patients treated with IV and IA, the HIAT-2 score did not reliably predict poor outcomes in our patients undergoing combined IV-IA therapy, potentially because of the small sample size. Despite our hypothesis, these 2 scores worked well at our center for predicting outcomes. The utility of these scores in this population could be because of the higher than hypothesized stroke severity and glucose of the patient population resulting in worse outcomes. This supports the utility of the HIAT and HIAT-2 score in different patient populations. SEDAN was developed to help clinicians predict the odds of developing sICH after IV tPA administration. As previously stated, the young age and vascular risk factor burden in our patients raised concern that SEDAN may not be able to accurately predict sICH in our sample. Interestingly, SEDAN was able to predict sICH in the IV group, but not in the IA or IV-IA groups.

Our study is limited by its retrospective nature and sample size, particularly in the IV-IA group. These scores need to be tested in a larger study of IV-IA patients to assess the utility of these scores in this group. Furthermore, our sample was taken from a single academic center, and thus, our findings may not be generalizable to other sites. We were unable to examine long-term functional outcomes, as these data were not available. Fortunately, disability status at the time of discharge has been shown to be predictive of 90-day functional outcome.²² As with any observational study, patients in this study were not randomized to treatment group. This likely resulted in baseline imbalances in the IV, IA, and IV-IA groups in measured and unmeasured variables. Prospective studies are needed to further investigate the utility of these scores in differing populations.

Despite its limitations, this study demonstrated that although highly predictive of outcome in the original study design treatment groups, prediction scores may not generalize to all patient samples. Surprisingly, our study found that some scoring systems were able to predict outcomes in treatment groups other than the group for which the score was designed. Our study highlights the importance of validating prediction scores in diverse samples. Given our findings, we urge clinicians to be cognizant of regional differences in stroke patients and to understand how lack generalizability may limit these scores before routinely applying them in clinical practice. Furthermore, scoring systems that are designed to predict poor outcomes in patients should not be used to exclude patients from treatment.