Abstract
Introduction
The Cincinnati Prehospital Stroke Severity Scale (CPSSS) was recently developed to predict large vessel occlusions (LVO) in patients with acute ischemic stroke (AIS). In its derivation study, which consisted of patients enrolled in thrombolysis and endovascular therapy trials, CPSSS had excellent discriminatory performance. We sought to externally validate CPSSS in an independent cohort.
Methods
Using our institution’s prospective stroke registry, we calculated CPSSS scores for all patients diagnosed with AIS at Weill Cornell Medical Center in 2013 and 2014. The primary outcome was presence of large vessel occlusion (LVO) and the secondary outcome was NIHSS ≥15. Harrell’s c-statistic was calculated to determine CPSSS’ discriminatory performance. Using the previously defined cut-point of ≥2 (range 0–4), we evaluated the test properties of CPSSS for predicting study outcomes.
Results
Among 751 patients with AIS, 664 had vessel imaging and were included in the final analysis. Of these, 80 (14.2%) had LVO and 117 (17.6%) had NIHSS ≥15. Median CPSSS was 0 (IQR 0–1) and 133 (20%) had scores of ≥2. C-statistic was 0.85 (95% CI 0.81–0.90) for predicting LVO and 0.94 (95% CI 0.92–0.97) for predicting NIHSS ≥15. Using a cut-point of ≥2, CPSSS was 70.0% sensitive and 86.8% specific for predicting LVO and 87.2% sensitive and 94.3% specific for predicting NIHSS ≥15.
Conclusions
In a cohort of patients with AIS treated at a tertiary-care stroke center, CPSSS had reasonable sensitivity and specificity for predicting LVO and severe stroke. Future studies should aim to prospectively validate the score in emergency responders.
Keywords: stroke, ischemic stroke, large vessel occlusion, clinical prediction score, cerebral infarction, acute stroke therapy
Introduction
Following the recently successful trials of endovascular therapy for acute ischemic stroke (AIS) from large vessel occlusions (LVO), (1–5) multiple clinical scoring systems have been developed to aid clinicians in identifying potential candidates for endovascular therapy. Examples include the 3 Item Stroke (3IS) scale, (6) the Rapid Arterial oCclusion Evaluation (RACE) scale, (7) and the Cincinnati Prehospital Stroke Severity Scale (CPSSS).(8) Such scales could potentially enable emergency medical services to circumvent primary stroke centers in favor of comprehensive stroke centers with endovascular capability and thereby increase endovascular therapy rates and reduce treatment times.
The newly created CPSSS is based on individual National Institutes of Health Stroke Scale (NIHSS) items and was derived using a dataset containing 624 patients from two National Institute of Neurological Disorders and Stroke Tissue Plasminogen Activator Stroke Study trials. In this population, CPSSS was shown to have a sensitivity of 89% and specificity of 78% for identifying patients with an NIHSS ≥15. (9) Using a 303-patient dataset drawn from the Interventional Management of Stroke trial, the CPSSS was shown to have a sensitivity of 83% and specificity of 40% for identifying patients with LVO. (10) Despite adequate sensitivity, the scale was derived and validated in a patient population culled from prospective thrombolysis and endovascular therapy trials, raising the possibility of non-generalizability and selection bias towards patients with good baseline neurological status and higher stroke severity. Considering these limitations in the development of CPSSS, we sought to externally validate CPSSS in an independent cohort of AIS patients.
Methods
Study Design and Population
This was a retrospective cohort study using a prospectively collected registry of patients diagnosed with AIS at New York-Presbyterian Hospital/Weill Cornell Medical College, a tertiary-care, teaching hospital and a certified primary stroke center with 24-7 endovascular capability. Our cohort comprised all patients diagnosed with AIS in 2013 and 2014 at our center in either the inpatient or emergency settings. The institutional review board of Weill Cornell Medical College approved the creation and maintenance of this registry, as well as the analysis for this study. Formal consent was not required for this retrospective study.
Measurements
From the electronic medical record, a single stroke research fellow (G.G.) abstracted each patient’s recorded NIHSS at the time of initial neurological consultation. Both the components of the NIHSS and the total NIHSS had been previously entered into the electronic medical record at the time of initial consultation. Using components of the NIHSS, the CPSSS, which was our predictor of interest, was determined for each patient. The CPSSS is a simple score ranging from 0 to 4, with 2 points assigned for conjugate gaze deviation (≥1 on NIHSS item for gaze), 1 point for abnormal level of consciousness questions and commands (≥1 on NIHSS items for consciousness questions and commands), and 1 point for severe arm weakness (≥2 on NIHSS item for arm weakness). Individual CPSSS components, as well as the total CPSSS, were recorded.
Our primary outcome of interest was presence of LVO, which was determined by abstracting the attending neuroradiologist’s final interpretation of available neuroimaging studies from the electronic medical record. In keeping with the definition of LVO used in the CPSSS derivation study, patients were considered to have LVO only if one of the following occlusions were demonstrated: extracranial or intracranial internal carotid artery (ICA), M1 segment of the middle cerebral artery (MCA), tandem occlusion involving both the extracranial ICA and the M2 segment of the MCA, or basilar artery. LVO was considered present if any of these occlusions were demonstrated on at least one vessel imaging study (magnetic resonance angiography, computed tomographic angiography, or digital subtraction angiography). We recorded whether intravenous thrombolysis or endovascular therapy was performed. Our secondary outcome was an NIHSS ≥15 (i.e., severe stroke).
Statistical Analysis
Descriptive statistics were used to determine clinical characteristics of study subjects, stratified by presence of LVO. Logistic regression was used to evaluate the association between individual components of the CPSSS and the outcomes of interest (e.g., presence of LVO and NIHSS ≥15). Harrell’s c-statistic was used to evaluate the score’s discriminatory performance. Using several different cut-points, including the previously defined cut-point of ≥2, we calculated the score’s sensitivity, specificity, positive likelihood ratio, and negative likelihood ratios for predicting our outcomes of interest.
Based on the results of our multivariate logistic regression analysis of the individual components of the CPSSS, which demonstrated that only conjugate gaze deviation and severe arm weakness were independently associated with LVO, we performed a post-hoc sensitivity analysis examining the discriminatory performance of a modified score that only incorporated those two components. In this modified scale, 2 points were assigned for conjugate gaze deviation and 1 point was assigned for severe arm weakness with a possible score range of 0 to 3. Statistical analysis was performed with Stata (version 12.1; Stata Corp., College Station, TX).
Results
Patient Characteristics
Among 751 patients with AIS, 664 (88%) had vessel imaging and were included in the final analysis. Of these, 80 (14.2%) had LVO and 117 (17.6%) had NIHSS ≥15. Patients with LVO were similar in age, gender, race, and vascular risk factors to patients without LVO, although patients with LVO had higher median NIHSS scores and more often were treated with intravenous thrombolysis and endovascular therapy (Table 1). Median CPSSS was 0 (IQR 0–1) and 133 (20%) had scores of ≥2 with 477 (71.8%) scoring 0, 54 (8.1%) scoring 1, 51 (7.7%) scoring 2, 40 (6.0%) scoring 3, and 42 (6.3%) scoring 4.
Table 1.
Patient Characteristics Stratified by Presence of Large Vessel Occlusion (LVO)
| Characteristic* | LVO (N = 80) | No LVO (N =584) | p value |
|---|---|---|---|
| Age in years, mean (SD) | 68.4 (15.0) | 70.0 (15.3) | 0.38 |
| Female | 43 (53.7) | 265 (45.4) | 0.19 |
| Race | 0.23 | ||
| White | 64 (80.0) | 504 (86.3) | |
| Black | 7 (8.7) | 42 (7.2) | |
| Other | 9 (11.2) | 38(6.5) | |
| Hypertension | 29 (36.2) | 201 (34.4) | 0.80 |
| Hyperlipidemia | 23 (28.7) | 144 (24.7) | 0.41 |
| Diabetes | 12 (15.0) | 74 (12.7) | 0.59 |
| Coronary artery disease | 9 (11.2) | 57 (9.8) | 0.69 |
| Atrial fibrillation | 8 (10.0) | 52 (8.9) | 0.68 |
| Prior stroke | 9 (11.2) | 83 (14.2) | 0.60 |
| Peripheral vascular disease | 1 (1.2) | 21 (3.6) | 0.50 |
| Active tobacco use | 4 (5.0) | 25 (4.3) | 0.77 |
| Received tPA | 31 (38.7) | 77 (13.2) | <0.01 |
| Received endovascular treatment | 28 (35.0) | 5 (0.9) | <0.01 |
| NIHSS score, median (IQR) | 19 (11.5–22.0) | 3 (1.0–6.0) | <0.01 |
Abbreviations: SD, standard deviation; tPA, tissue plasminogen activator; IQR, interquartile range.
Data are presented as number (%) unless otherwise specified.
CPSSS Test Properties
In bivariate logistic regression, all CPSSS items were significantly associated with presence of LVO; however, in multivariate analysis, only the first (e.g., gaze deviation) (OR 4.1, 95% CI 2.2–7.7) and third (e.g., severe arm weakness) (OR 8.5, 95%CI 4.2–16.9) items were (Table 2). In contrast, all three CPSSS items were significantly associated with NIHSS ≥15 in both bivariate and multivariate analyses (Table 3).
Table 2.
Association of Individual CPSSS Items with Large Vessel Occlusion in Logistic Regression Analyses
| Bivariate | Multivariate | |||
|---|---|---|---|---|
| OR | 95% CI | OR | 95% CI | |
| Conjugate gaze deviation | 13.4 | 7.9–22.6 | 4.1 | 2.2–7.7 |
| Abnormal LOC | 5.5 | 3.3–9.2 | 1.0 | 0.5–2.0 |
| Severe arm weakness | 17.2 | 9.9–29.8 | 8.5 | 4.2–17.0 |
Abbreviations: OR, odds ratio; CI, confidence interval; LOC, level of consciousness.
Table 3.
Association of Individual CPSSS Items with NIHSS≥ 15 in Logistic Regression Analyses
| Bivariate | Multivariate | |||
|---|---|---|---|---|
| OR | 95% CI | OR | 95% CI | |
| Conjugate gaze deviation | 33.0 | 19.3–56.5 | 12.3 | 5.6–26.7 |
| Abnormal LOC | 33.2 | 19.3–57.2 | 14.2 | 6.4–31.3 |
| Severe arm weakness | 80.0 | 43.1–148.4 | 22.0 | 10.6–45.7 |
Abbreviations: NIHSS, National Institutes of Health Stroke Scale; OR, odds ratio; CI, confidence interval; LOC, level of consciousness.
The c-statistic was 0.85 (95% CI 0.81–0.90) for predicting LVO and 0.94 (95% CI 0.92–0.97) for predicting NIHSS ≥15. Using a cut-point of ≥2, CPSSS was 70.0% sensitive and 86.8% specific for predicting LVO, and 87.2% sensitive and 94.3% specific for predicting NIHSS ≥15 (Table 4). At this cut-point, the positive likelihood ratio was 5.3 and the negative likelihood ratio was 0.3 for predicting LVO, while the positive likelihood ratio was 15.4 and the negative likelihood ratio was 0.1 for predicting NIHSS ≥15.
Table 4.
Predictive ability of the Cincinnati Pre-Hospital Stroke Severity Score (CPSSS)
| Outcome | CPSSS cut-point | Sensitivity (%) | Specificity (%) | Positive likelihood ratio | Negative likelihood ratio |
|---|---|---|---|---|---|
| LVO | 0 | 100.0 | 0.0 | 1.0 | NC |
| ≥1 | 86.2 | 79.8 | 4.3 | 0.2 | |
| ≥2 | 70.0 | 86.8 | 5.3 | 0.3 | |
| ≥3 | 51.2 | 93.0 | 7.3 | 0.5 | |
| 4 | 25.0 | 96.2 | 6.6 | 0.8 | |
| NIHSS ≥ 15 | 0 | 100.0 | 0.0 | 1.0 | NC |
| ≥1 | 93.2 | 85.7 | 6.5 | 0.1 | |
| ≥2 | 87.2 | 94.3 | 15.4 | 0.1 | |
| ≥3 | 61.5 | 98.2 | 33.7 | 0.4 | |
| 4 | 35.0 | 99.8 | 191.7 | 0.6 |
Abbreviations: LVO, large vessel occlusion; NIHSS, National Institutes of Health Stroke Scale; NC: not calculable.
Using a cut-point of ≥1, the score demonstrated a sensitivity of 86.3% and a specificity of 79.8% for LVO, and a sensitivity of 93.2% and a specificity of 85.7% for NIHSS ≥15. Similarly, the positive likelihood ratio was 4.3 and the negative likelihood ratio was 0.2 for predicting LVO, while the positive likelihood ratio was 6.5 and the negative likelihood ratio was 0.1 for predicting NIHSS ≥15.
Sensitivity Analysis
In a sensitivity analysis examining the discriminatory ability of a simplified prediction score that only included the components of conjugate gaze deviation and severe arm weakness, the c-statistic was 0.84 (95% CI 0.80–0.89) for predicting LVO and 0.92 (95% CI 0.89–0.95) for predicting NIHSS ≥15. Using a cut-point of ≥2, the limited CPSSS was 58.8% sensitive and 90.2% specific for predicting LVO, and 64.1% sensitive and 94.7% specific for predicting NIHSS ≥15 (Table 5). At this cut-point, the positive likelihood ratio was 6.0 and the negative likelihood ratio was 0.5 for predicting LVO, while the positive likelihood ratio was 12.1 and the negative likelihood ratio was 0.4 for predicting NIHSS ≥15.
Table 5.
Predictive Ability of a Simplified CPSSS Limited to the Individual Items of Severe Arm Weakness and Conjugate Gaze Deviation
| Outcome | Score cut-point | Sensitivity (%) | Specificity (%) | Positive likelihood ratio | Negative likelihood ratio |
|---|---|---|---|---|---|
| LVO | 0 | 100.0 | 0.0 | 1.0 | NC |
| ≥1 | 82.5 | 82.5 | 4.7 | 0.2 | |
| ≥2 | 58.8 | 90.2 | 6.0 | 0.5 | |
| 3 | 50.0 | 93.7 | 7.9 | 0.5 | |
| NIHSS ≥ 15 | 0 | 100.0 | 0.0 | 1.0 | NC |
| ≥1 | 90.6 | 88.7 | 8.0 | 0.1 | |
| ≥2 | 64.1 | 94.7 | 12.1 | 0.4 | |
| 3 | 59.8 | 98.7 | 46.8 | 0.4 |
Abbreviations: LVO, large vessel occlusion; NIHSS, National Institutes of Health Stroke Scale; NC: not calculable.
Discussion
In a large cohort of AIS patients treated at a tertiary-care stroke center, the CPSSS score had excellent discriminatory ability in predicting LVO and severe stroke. In fact, the c-statistic for predicting LVO was considerably higher in our study (0.85) than in the original CPSSS derivation study (0.67). In addition, the previously defined cut-point of ≥2 demonstrated acceptable sensitivity and specificity for predicting both outcomes of interest, although it was slightly more sensitive and specific in predicting severe stroke than LVO, similar to the results of the original CPSSS derivation study.
In our validation study, only two of the three CPSSS items, conjugate gaze deviation and severe arm weakness, were significantly associated with LVO in multivariate logistic regression, while the abnormal level of consciousness item was not. Furthermore, in a sensitivity analysis, a modified score limited to the two independently associated items had similar discriminatory ability for predicting LVO as the CPSSS score itself (c-statistic 0.84 versus 0.85). Given these results and the fact that abnormal level of consciousness is common in patients with many non-stroke acute neurological diseases (i.e., seizures, toxic-metabolic encephalopathy, etc.), a simplified form of the CPSSS limited to just conjugate gaze deviation and severe arm weakness could be more clinically useful for first responders and warrants further study.
The CPSSS is a simple, objective score based on three elements of the NIHSS (e.g., gaze, level of consciousness, and arm strength) that are familiar to many emergency medical responders. Our results suggest that CPSSS is highly specific and reasonably sensitive for predicting LVO and severe stroke when applied to an independent cohort of AIS patients treated at an urban, tertiary-care stroke center. However, using a cut-point of 2 as defined in the score’s derivation study would result in approximately one in three patients with LVO being missed, which may not be sufficiently sensitive for a prediction score screening for a highly morbid disease. In addition, the sensitivity of the score when used by emergency medical services personnel among patients with acute neurological disease of unknown cause may be worse and therefore further validation of the score is required before any consideration for clinical use.
This study has several limitations. First, it was a retrospective study performed at a single, urban, tertiary-care stroke center and thus our results may not generalize to other settings. Second, CPSSS scores were calculated based on the initial examinations of neurologists trained in performing the NIHSS score, and not by the examinations of emergency medical responders who would be most expected to use CPSSS in clinical practice. Therefore, before clinical use, the score should be further validated by emergency medical services personnel in a prospective, multicenter study. Third, our study examined the predictive ability of the CPSSS in patients who were diagnosed with AIS rather than in patients with acute neurological disease of unknown cause. As a result, these patients likely had a higher pre-test probability of LVO than most patients typically encountered by emergency medical response teams, which may have artificially inflated the potential accuracy of the CPSSS. Therefore, further validation of the CPSSS in patients presenting with acute neurological complaints of unknown cause is needed before implementation into clinical practice. Fourth, in contemplating the use of CPSSS in the pre-hospital setting, consideration should be given to the variable sensitivity of CPSSS with respect to the prediction of LVO, especially when using higher cut-points. Fifth, 12% of stroke patients at our hospital did not acutely receive vessel imaging and therefore were excluded from analysis, which could have introduced selection bias. However, this is unlikely since the clinical characteristics of these patients were similar to those who did receive vessel imaging (data not shown). Finally, our study was solely focused on validating the CPSSS, and therefore we are unable to comment on the discriminatory ability of other recently developed clinical prediction scores for LVO, or how CPSSS compares to these other scores.
Conclusions
In summary, we found the CPSSS as determined by the examination of neurologists to be a robust predictor of LVO and severe stroke in a diverse cohort of patients with AIS treated at an urban, tertiary-care stroke center. However, before clinical use, future validation studies are needed to evaluate the real-time effectiveness of the score when used by emergency medical personnel in patients with acute neurological disease of unknown cause.
Acknowledgments
Grant support: Florence Gould Endowment for Discovery in Stroke, and National Institutes of Health grant K23NS091395, both awarded to Babak B. Navi
Footnotes
Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final citable form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.
References
- 1.Campbell BC, Mitchell PJ, Kleinig TJ, et al. Endovascular therapy for ischemic stroke with perfusion-imaging selection. N Engl J Med. 2015;372:1009–1018. doi: 10.1056/NEJMoa1414792. [DOI] [PubMed] [Google Scholar]
- 2.Goyal M, Demchuk AM, Menon BK, et al. Randomized assessment of rapid endovascular treatment of ischemic stroke. N Engl J Med. 2015;372:1019–1030. doi: 10.1056/NEJMoa1414905. [DOI] [PubMed] [Google Scholar]
- 3.Saver JL, Goyal M, Bonafe A, et al. Stent-retriever thrombectomy after intravenous t-PA vs. t-PA alone in stroke. N Engl J Med. 2015;372:2285–2295. doi: 10.1056/NEJMoa1415061. [DOI] [PubMed] [Google Scholar]
- 4.Berkhemer OA, Fransen PS, Beumer D, et al. A randomized trial of intraarterial treatment for acute ischemic stroke. N Engl J Med. 2015;372:11–20. doi: 10.1056/NEJMoa1411587. [DOI] [PubMed] [Google Scholar]
- 5.Molina CA, Chamorro A, Rovira A, et al. REVASCAT: a randomized trial of revascularization with SOLITAIRE FR device vs. best medical therapy in the treatment of acute stroke due to anterior circulation large vessel occlusion presenting within eight-hours of symptom onset. Int J Stroke. 2015;10:619–626. doi: 10.1111/ijs.12157. [DOI] [PubMed] [Google Scholar]
- 6.Singer OC, Dvorak F, du Mesnil de Rochemont R, et al. A simple 3-item stroke scale: comparison with the National Institutes of Health Stroke Scale and prediction of middle cerebral artery occlusion. Stroke. 2005;36:773–776. doi: 10.1161/01.STR.0000157591.61322.df. [DOI] [PubMed] [Google Scholar]
- 7.Pérez de la Ossa N, Carrera D, Gorchs M, et al. Design and validation of a prehospital stroke scale to predict large arterial occlusion: the rapid arterial occlusion evaluation scale. Stroke. 2014;45:87–91. doi: 10.1161/STROKEAHA.113.003071. [DOI] [PubMed] [Google Scholar]
- 8.Katz BS, McMullan JT, Sucharew H. Design and Validation of a Prehospital Scale to Predict Stroke Severity: Cincinnati Prehospital Stroke Severity Scale. Stroke. 2015;46:1508–1512. doi: 10.1161/STROKEAHA.115.008804. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Tissue Plasminogen Activator for Acute Ischemic Stroke. NINDS rt-tPA Stroke Study Group. N Engl J Med. 1995;333:1581–1587. doi: 10.1056/NEJM199512143332401. [DOI] [PubMed] [Google Scholar]
- 10.Broderick JP, Palesch YY, Demchuk AM, et al. Endovascular Therapy after Intravenous t-PA versus t-PA Alone for Stroke. N Engl J Med. 2013;368:893–903. doi: 10.1056/NEJMoa1214300. [DOI] [PMC free article] [PubMed] [Google Scholar]