Design and Validation of a Prehospital Scale to Predict Stroke Severity: The Cincinnati Prehospital Stroke Severity Scale

Abstract

Background and Purpose

We derived and validated the Cincinnati Prehospital Stroke Severity Scale (CPSSS) to identify patients with severe strokes and large vessel occlusion (LVO).

Methods

CPSSS was developed with regression tree analysis, objectivity, anticipated ease in administration by EMS personnel, and the presence of cortical signs. We derived and validated the tool using the two NINDS t-PA Stroke Study trials and IMS III Trial cohorts, respectively, to predict severe stroke [NIH stroke scale (NIHSS) ≥15] and LVO. Standard test characteristics were determined and receiver operator curves were generated and summarized by the area under the curve (AUC).

Results

CPSSS score ranges from 0-4; composed and scored by individual NIHSS items: 2 points for presence of conjugate gaze (NIHSS ≥1); 1 point for presence of arm weakness (NIHSS ≥2); and 1 point for presence abnormal level of consciousness (LOC) commands and questions (NIHSS LOC ≥1 each). In the derivation set, CPSSS had an AUC of 0.89; score ≥2 was 89% sensitive and 73% specific in identifying NIHSS ≥15. Validation results were similar with an AUC of 0.83; score ≥2 was 92% sensitive, 51% specific, a positive likelihood ratio (PLR) of 3.3 and a negative likelihood ratio (NLR) of 0.15 in predicting severe stroke. For 222/303 IMS III subjects with LVO, CPSSS had an AUC of 0.67; a score ≥2 was 83% sensitive, 40% specific, PLR of 1.4, and NLR of 0.4 in predicting LVO.

Conclusions

CPSSS can identify stroke patients with NIHSS ≥15 and LVO. Prospective prehospital validation is warranted.

Introduction

Currently, Primary Stroke Centers (PSC) and Comprehensive Stroke Centers (CSC) form a two-tier regionalized system of care for the efficient emergent management of patients with acute stroke.¹ While both PSCs and CSCs provide acute stroke care,¹ CSCs are better equipped to provide state-of-the-art care for patients with severe ischemic and hemorrhagic stroke, including endovascular therapies, advanced diagnostic imaging, continuous in-hospital neurosurgical availability, and neurocritical care. These capabilities have led to improved cost-effectiveness and outcomes for ischemic stroke, intracerebral hemorrhage (ICH), and subarachnoid hemorrhage (SAH) at CSCs compared with non-CSCs.^2-12 Large volume stroke centers provide faster treatment, and improved the use of thrombolysis for acute ischemic stroke (AIS) patients.¹³ With the recent findings that timely endovascular therapy may improve outcomes in AIS patients with confirmed large vessel occlusions (LVO),^14-16 and the greater likelihood of LVO in AIS patients with more severe AIS,^{17, 18} there is now greater clinical need to care for patients with severe stroke at CSCs early in their clinical course.

Emergency Medical Services (EMS) providers are the first medical contact for half of all stroke patients,¹⁹ and are responsible for triaging patients to appropriate hospitals. EMS bypass of non-stroke centers in favor of PSCs has been advocated,^{1, 20, 21} but the concept of bypassing a PSC for a CSC has been recommended for consideration but not yet been widely adopted.²² Bypass could be justified because interfacility transfer of patients introduces significant unnecessary delays in care and patient charges, compared to direct EMS transport to a specialty care center.^23-27 Other prehospital stroke scales^28-30 have been proposed to identify patients experiencing an acute stroke or LVO,^30-32 but have not been widely adopted in clinical practice due to the absence of positive LVO endovascular therapy trials.

We describe the retrospective derivation and validation of a new brief neurologic scale, the Cincinnati Prehospital Stroke Severity Scale (CPSSS), in prediction of severe AIS and LVO.

Methods

Datasets

To derive the CPSSS, we used the two NINDS t-PA Stroke Study trials dataset,³³ which is comprised of a diverse cohort patients suffering mild to severe strokes [NIH Stroke Scale (NIHSS) 1-37] randomized to treatment with t-PA or placebo within 3 hours of onset. The IMS-III dataset served as the validation cohort.³⁴ IMS-III was a prospective randomized trial comparing outcomes of moderate to severe stroke patients (NIHSS 8-40) treated with t-PA with or without endovascular intervention. In both trials, pre-treatment NIHSS scores (component and composite) were collected by NIHSS-certified study physicians at the time of enrollment. Subjects in each dataset with complete pre-treatment NIHSS component scores were included in the derivation or validation cohorts. Because the IMS-III cohort had higher NIHSS scores per the eligibility criteria, we chose to derive the CPSSS from the two NINDS t-PA Stroke Study cohort to study sensitivity and specificity among a broader severity of stroke patients.

Outcome Measures

The primary outcome was the CPSSS ability to identify a severe stroke defined as NIHSS ≥15 because over 95% of AIS patients with NIHSS ≥15 have a LVO by baseline imaging^{17, 18} and is the lower limit of stroke severity for the consideration of hemicraniectomy.^9,10 Secondary outcomes included CPSSS ability to identify moderate stroke severity (NIHSS ≥10) and proximal LVO [extracranial internal carotid artery, intracranial internal carotid artery, M1, tandem cervical ICA plus M2, or basilar artery occlusions (excluding M3, M4, Posterior Cerebral Artery occlusion sites, and isolated M2)] by CT angiography prior to IV t-PA therapy.

Derivation and Validation Methods

CPSSS was designed with a similar approach to that used in the development of the original Cincinnati Prehospital Stroke Scale.³⁵ A classification and regression tree (CART) analysis was used to identify subsets of NIHSS items, as well as item score cut-points, working together to predict severe stroke defined as NIHSS total score ≥15. Next, important candidate items identified by the classification tree were reviewed by the study clinicians (including Board-certified EMS, Emergency Medicine, and Vascular Neurology physicians), and those items that are objective (e.g. correctly answering questions or following commands) were chosen by expert consensus over those that are subjective (i.e. degree of aphasia) to ensure ease of use by EMS personnel. Selected items were given a point score of 1 or 2 based on importance in predicting severe stroke as determined by the CART analysis. Receiver operating characteristic (ROC) curves were generated to determine the accuracy of the CPSSS and to identify an optimal cut point to define an “abnormal/positive” screening assessment.

After derivation, the tool performance was assessed using the IMS III Trial data (validation dataset) where, in addition to predicting AIS severity, the CPSSS was assessed for its accuracy in detecting LVO. Sensitivity, specificity, and positive and negative likelihood ratios were calculated for the optimal cut point.

Results

The two NINDS t-PA Stroke Study trials dataset contained 624 patients, of which there were 447 patients (71.6%) with pre-treatment NIHSS ≥10 and 311 patients (49.8%) with pre-treatment NIHSS ≥15. The presence of pre-treatment proximal LVO was not obtained. CART analysis was employed to identify potential NIHSS items for consideration by scale designers. CART analysis identified 5 NIHSS items that were important factors in predicting severe stroke (NIHSS ≥15). The most important factor, first defining split, was abnormal gaze (best gaze >0). For those with normal gaze (best gaze = 0), motor function (right arm, right leg and left leg) and language were found to be important factors. Of these, abnormal gaze and arm motor function were chosen to be included in the CPSSS, and the presence of abnormal gaze was weighted heavier on the CPSSS based on its predictive value on the CART and due to its suggestion of focal cortical dysfunction. Motor function of arm falling to bed was selected for ease of EMS providers to ascertain. Language was specifically not chosen to be included, despite being an important predictive factor, due to subjectivity of interpretation. Language was replaced with level of consciousness questions and commands due to the objective interpretation for EMS providers, yielding the CPSSS (Figure 1). A score is given for the presence of each clinical feature. The total CPSSS score is calculated as the sum of the individual scores and ranges from 0 to 4, with higher score reflecting higher stroke severity.

Figure 1.

The CPSSS’s individual items and scoring.

In the derivation set, the CPSSS score had an area under the ROC of 0.89 for detecting severe stroke (See Figure 2). A CPSSS score ≥ 2 was 89% sensitive, 73% specific, positive likelihood ratio (PLR) of 3.30, and negative likelihood ratio (NLR) of 0.15 in identifying severe stroke (NIHSS of ≥15). For identification of stroke patients with a moderate stroke, a CPSSS score of ≥ 2 or greater had a sensitivity of 75%, a specificity of 85%, PLR of 5.00, and NLR of 0.29 (Table 1).

Figure 2.

The receiver operating characteristic (ROC) curve for CPSSS in detecting stroke severity in the derivation dataset.

Table 1.

Accuracy of CPSSS ≥ 2 in detecting moderate/severe stroke and large vessel occlusion

Outcome:	Dataset:	Sensitivity	Specificity	PLR	NLR
Stroke Severity
Severe (NIHSS≥15)	Derivation	89%	73%	3.30	0.15
	Validation	92%	51%	1.89	0.16
Moderate (NIHSS≥10)	Derivation	75%	85%	5.00	0.29
	Validation	79%	89%	7.18	0.24
LVO	Validation	83%	40%	1.38	0.42

PLR=positive likelihood ratio.

NLR=negative likelihood ratio.

LVO = Large Vessel Occlusion defined as occlusion sites of ICA, M1, tandem cervical ICA plus M2, or basilar arteries.

The performance of CPSSS was validated using the IMS III dataset, which was composed of 650 patients of which 390 patients (60%) presented with an NIHSS ≥15 and 641 patients (99%) presented with an NIHSS ≥10. In the IMS III dataset, the CPSSS had similar area under the ROC and test characteristics to the derivation set (Table 1, Figure 3).

Figure 3.

The receiver operating characteristic (ROC) curve for CPSS in detecting stroke severity and LVO in the validation dataset.

For identifying LVO, 303 IMS III subjects had data available and were included in the analysis, of which, 222 had LVO. For LVO, CPSSS had an AUC of 0.67 and a score ≥ 2 was 83% sensitive, 40% specific, a PLR of 1.4, and NLR of 0.4 (Table 1).

A comparison of the CPSSS to other prehospital stroke scales is shown in Table 2.

Table 2.

Comparison of CPSSS to other published prehospital stroke severity scales

	CPSSS	LAMS	RACE	I3SS
N in derivation dataset	624	119	654	171
Validated in independent dataset (Y/N)	Y	N	Y	N
Number of items scored	3	3	6	3
Sensitivity/Specificity for severe stroke	89%/72%	NA	NA	86%/95%†
Sensitivity/Specificity for LVO ‡	83%/40%	81%/89%*	85%/67%**	67%/92% §
Evaluated in Prehospital Setting (Y/N)	N	N	Y	N

^†Severe Stroke defined as NIHSS ≥14.

^‡LVO indicates large vessel occlusion.

^*LVO defined as terminal intracranial carotid artery occlusion, M1 segment middle cerebral artery occlusion, M2 segment middle cerebral artery occlusion, and M3/M4 segment middle cerebral artery occlusion using catheter angiography, MRA, CTA, and carotid ultrasound.

^**LVO was defined as occlusion of the terminal intracranial carotid artery, proximal middle cerebral artery (M1 segment), tandem (extracranial carotid artery plus middle cerebral artery) and basilar artery with transcranial duplex (50% of cases), CTA and MRA.

^§LVO defined as T/M1 occlusion all using MRA.

Discussion

We found that the CPSSS has high sensitivity and acceptable specificity in detecting the presence of severe stroke, moderate stroke, and LVO among populations of patients with AIS. Specifically, the CPSSS can identify stroke patients who could be most likely to benefit from rapid triage to a CSC, including those who harbor proximal LVO that are appropriate targets for IV tPA and/or endovascular therapy. These patients may also be eventual candidates for hemicraniectomy, and/or benefit from a dedicated Neurologic Intensive Care Unit (ICU). Given the time-sensitivity of the above therapies, accurate initial triage of patients to hospitals where these therapies are available is key to preventing delays in care and increased costs associated with transfers.

Three screening tools to identify severe ischemic strokes have been previously proposed, but none has been widely adopted into EMS practice (Table 2). The Los Angeles Motor Scale (LAMS)³¹ is limited by the fact that a sizable proportion of the population studied to derive LAMS occurred retrospectively and included patients enrolled in clinical trials over 11 years at a single institution. The RACE scale was developed to predict LVO.³⁰ However, RACE scale was not evaluated in 60% of patients transferred by EMS, and patients not included had less severe strokes and less frequency of LVO than patients included in the analysis. The complexity of RACE (6 domains requiring subjective interpretation in several of the domains) further poses a significant challenge for accurate EMS implementation. It should be noted that we constructed the CPSSS prior to awareness of the recent RACE publication, but there are similarities between the two scales. The major differences are that the CPSSS is shorter, uses dichotomous answers, and rates conjugate gaze deviation more heavily. Finally, the 3-Item Stroke Scale (3ISS)³² is most similar to our prehospital scale given modalities that were tested. The 3ISS was limited by sample size and requires subjective grades such as none, moderate, or severe “Disturbance of Consciousness.”

Using a prehospital scale such as the CPSSS as a triage tool in the field would impact the initial transport of patients in many communities in the U.S. and potentially other countries depending upon the prehospital system. However, since almost 80% of ischemic stroke patients have a baseline NIHSS < 10,³⁶ and nearly 90% have a baseline NIHSS < 15, the actual number of ischemic stroke patients triaged to CSCs would have relatively limited impact on individual community hospitals that are not comprehensive centers. The NIHSS distribution of hemorrhagic stroke within a population is unknown; as such we are unable to reliably estimate the proportion of hemorrhagic stroke patients that may be triaged to CSCs by EMS personnel using the CPSSS. In the Greater Cincinnati/Northern Kentucky region, severe strokes have a mean estimated time difference between direct EMS transport to the CSC and direct EMS transport to the nearby community hospital of 6.7 +/− (6.0) minutes.³⁷ In patients with severe strokes, further investigation is needed to determine the acceptable delay in tPA administration in order for direct EMS transport to a CSC for faster endovascular treatment²³ and Neurologic ICU monitoring. The impact of preferential triage of patients from the closest or requested community hospitals to a CSC on stroke-onset-to-hospital-arrival times depends upon size of the region and location of hospitals.

An effective prehospital triage system for severe strokes would substantially decrease emergent hospital-to-hospital transfers and decrease the time from symptom onset to endovascular therapy, if applicable. Costs of transfers may be up to $1000 by ambulance and $4,000 to $25,000 by helicopter^{25, 26} and are borne by patients and their families since they are not reimbursed by hospital diagnosis-related groups (DRGs).²⁷

This study has some limitations. First, this was a retrospective analysis of two existing ischemic stroke trial cohorts; therefore prospective evaluation by EMS providers is required. Second, there is variability of the NIHSS during the first few hours of AIS onset,^{17, 18} and it is possible the stroke severity at the time of EMS examination will change by the time a treatment decision for ischemic stroke is made by medical providers. Next, isolated M2 lesions were not included in the CPSSS’s LVO prediction analysis; however, only a minority of isolated M2 occlusions (2-8% of patients) was included in recent positive endovascular trials.^14-16 The CPSSS has not been tested in populations of hemorrhagic stroke patients or in a general population of potential stroke patients that are evaluated by EMS personnel in the field. The CPSSS is likely to be less sensitive to subarachnoid hemorrhage in which patients’ presentations are often non-focal, unless the patient presents in coma. However, patients with sudden onset of coma are probably more likely to be preferentially triaged to tertiary centers.

In summary, the CPSSS was designed to be user-friendly and applicable for EMS providers in the field. CPSSS ≥ 2 has promising characteristics in predicting severe strokes and LVO and should be prospectively evaluated to demonstrate clinical utility. This study serves as the foundation for an ongoing study assessing the feasibility of CPSSS by EMS providers in the prehospital setting among potential stroke patients. The eventual goal is a prehospital scale that can be used as a reliable and practical method of prehospital triage of stroke patients in which the large majority of patients are transported to the location where the best therapy can be delivered as rapidly as possible.