Maximizing efficiency and diagnostic accuracy triage of acute stroke patients: A case-control study

Abstract

Background

Recent data have demonstrated that mechanical thrombectomy (MT) is beneficial for patients presenting within zero to six hours of symptom onset after stroke. However, transferring all patients with possible strokes for endovascular therapy and MT would be inefficient and costly. We conducted a case-control study to identify a subset of the National Institutes of Health Stroke Scale (NIHSS) to identify patients with large-vessel occlusion (LVO) to a high degree of specificity, in order to select those patients for whom transfer is most appropriate.

Methods

Acute code stroke alerts presenting to a comprehensive stroke center from 2012 to 2013 (779) and corresponding NIHSS were collected. All patients had vascular imaging and 125 demonstrated LVO (cases) and were compared to 272 small-vessel strokes and stroke mimics (controls). Demographics of both groups and modified receiver operating characteristic (ROC) curves were generated for each combination of three NIHSS items to optimize specificity of LVO for those who would benefit from MT.

Results

The average NIHSS of cases was higher than controls (12.5 vs. 6.5, p < 0.0001). The subset of three NIHSS items with the largest modified AUC (optimized for specificity) was maximum “Arm,” “Sensory,” and “Extinction.” Using a cutoff of seven out of a total 10 possible points, the sum score for these items has 90.2% specificity and 16.0% sensitivity for LVO.

Conclusion

We present a validated three-question subset of the NIHSS for those who would benefit from MT with a high degree of specificity.

Introduction

Ischemic stroke is one of the leading causes of morbidity and mortality in the United States (US), with approximately 795,000 strokes and 130,000 deaths each year.^1,2 Between medical care, rehabilitation needs, and work hours lost, the cost of stroke approached $41 billion in 2007.¹

A major source of poor stroke outcome is timely access to stroke care. A Centers for Disease Control and Prevention (CDC) report cited that approximately 31% of US counties did not have a hospital emergency department (ED), and approximately 77% of counties lacked hospitals with neurological services.³ Additionally, the national usage of tissue plasminogen activator (tPA) is remarkably low, with rates of 2.4% in 2006.⁴ This makes ischemic stroke treatment difficult since therapies must be administered within a short time of symptom onset.^5,6

Several standardized scales exist to objectively evaluate stroke symptoms and severity, including the Canadian Neurological Scale,⁷ Los Angeles Motor Scale (LAMS),⁸ and the Cincinnati Prehospital Stroke Scale (CPSS).⁹ Perhaps the most widely used stroke scale is the National Institutes of Health Stroke Scale (NIHSS), composed of 15 items for a maximum score of 42.^7,10–13

Treatments for ischemic stroke include intravenous IV (tPA) and intra-arterial (IA) thrombolysis and thrombectomy. For patients outside the tPA window, and patients who have received IV tPA, the role of endovascular intervention has been recently confirmed. Given the influx of recent randomized clinical trials showing the overwhelming benefit of acute endovascular stroke therapy, specifically Multicenter Randomized Clinical trial of Endovascular treatment for Acute ischemic stroke in the Netherlands (MR CLEAN) and six other trials that were halted prematurely because of positive results, we are at an exciting time in acute stroke care.^14,15

In the world of stroke, “Time is brain.”^6,14 Transfer of patients for acute endovascular therapy has become the gold standard, yet regionalization of care throughout health systems can be costly with financial disincentives to transfer.^16–18 Early regionalization systems can lead to BOTH unnecessary interfacility transfers as well as delays in transfers for acute patients leading to unintended consequences. These more “sensitive” systems can lead to inappropriate use of resources, which has been the impetus for more specific systems to evolve. An example of this occurred in the 1970s–1980s, when the general trauma construct was for all trauma patients to be cared for at only high-level centers.¹⁹ This led to inappropriate utilization of high-level resources for minor injuries, high-level centers became overburdened, and commitment from local EDs to care for these patients plummeted.²⁰ The trauma community has since recovered from this trend by integrating all regional hospitals and reserving transfer for the most acute, life-threatening injuries.^21,22 Given the positive results of recent trials, the stroke community cannot afford to make this same mistake.

Transferring all patients with a suspicion of stroke would be inefficient, costly, and an inappropriate allocation of resources. The goal of our study was to successfully identify a three-item NIHSS subset analysis that can identify, with a high degree of certainty, patients with large-vessel occlusion (LVO) who would benefit from mechanical thrombectomy (MT). The primary objective was to find a scale that maximizes specificity above all else, to enforce the time and financial efficiency of patient transfers. To clarify, from a medical perspective one would aim for 100% sensitivity; however, from an economical, real-world perspective, high specificity reduces cost and allocation of resources.

Methods

The study was conducted at a 584-bed tertiary care center designated by the Joint Commission as a Comprehensive Stroke Center. The stroke algorithm begins with identification of a patient with a neurologic deficit. A code stroke mass page is sent to all physicians, nurses, and technicians involved, and the patient is immediately mobilized as Neurology goes directly to bedside. A non-contrasted computed tomography (CT) scan is immediately obtained, followed by a CT angiography and perfusion (CTA/CTP). Computed tomography angiography (CTA)/Computed tomography perfusion (CTP) is then obtained. Based on the history and imaging, a treatment decision is rendered by the Neurology Stroke attending physician and possibly the Neurointerventional attending physician. NIHSS is recorded for every code stroke. Institutional review board approval was obtained for the study (IRB# 140895).

Design

This was a retrospective, case-control study conducted from 2012 to 2013. Cases were defined as acute large-vessel ischemic stroke, also termed large vessel occlusions (LVO). LVO was defined as evidence of major vessel occlusion on CTA or magnetic resonance angiography (MRA). The six vessels were: internal cerebral artery (ICA), anterior cerebral artery (ACA), middle cerebral artery (MCA), posterior cerebral artery (PCA), basilar artery (BA), or vertebral artery (VA). M2 or M3 occlusions were categorized as MCA, and this same categorization scheme was followed for all vessels. Though recent trials have focused on proximal anterior circulation, it was the feeling of the senior author that vertebral or proximal M3 occlusions with a significant NIHSS deficit are reasonable candidates for thrombectomy. We therefore included these patients in the effort to be more inclusive along these lines.

Controls were stroke mimics and strokes without LVO. Stroke mimics included seizures, mass lesions, metabolic disturbances, or altered mental status (AMS) of other origin. Strokes without LVO included hemorrhagic strokes and small-vessel infarcts. Strokes labeled controls had CTA or MRA evidence that confirmed the absence of LVO.

Patient selection

From January 1, 2012 to December 31, 2013, there were 779 code strokes. Of those 779 code strokes, 382 patients had no record of NIHSS, and were thus excluded from our analysis. This was likely due to the transition to the new acute stroke algorithm that took place in August 2012, whereas previously NIHSS was not documented for every patient. Even with a transition in stroke algorithm, patients in August 2012 were still included in our analysis if they had the requisite NIHSS documented. Regardless of the institutional protocol, the method of obtaining an NIHSS is the same for every patient. Of the remaining 397 patients, 207 were stroke mimics and 65 were small vessel or hemorrhagic strokes without LVO leaving 125 LVOs. The low number of hemorrhagic infarcts was thought to be due to institutional triage patterns. Sometimes hemorrhagic stroke patients are taken directly to the operating room if decompression is needed, and a documented NIHSS may be missed. Furthermore, patients with a known intraparenchymal hemorrhage are often transferred via helicopter from an outside hospital. Thus, they are not a candidate for thrombolytics or endovascular thrombectomy and may not be triaged as a possible stroke intervention. NIHSS sum scores were calculated for each case (LVO) and control (stroke mimic or stroke without LVO).

Data collection

All data were retrospectively obtained from the electronic medical record (EMR). For each patient, demographic data were recorded. Imaging (CTA or MRA) was reviewed to identify or exclude LVO and to determine sidedness of the stroke. All imaging was read by an attending radiologist. For non-stroke patients, diagnosis of what led to the initial code stroke was recorded. All NIHSS scores were taken at the exam immediately after the code stroke was called, and the time of examination was noted. Despite the change in stroke algorithms, an immediate neurologic exam was always the first step. No patients were given tPA prior to their exam, as the neurologic exam used was precisely at admission.

Statistical analysis

Descriptive statistics were calculated for our case and control groups. Using every possible combination of three items from the 15-item NIHSS, modified receiver operating characteristic (ROC) curves were then generated for each combination in which the y-axis represents the true-negative rate and the x-axis represents false-negative rate. The subsets were then ranked by a modified area under the curve (AUC) so the false-negative rate could not exceed 90%, enabling us to compare the specificity for a stroke of each combination. The NIHSS item of “language deficit” was excluded from the three-item combinations because it is inherently biased toward left-sided strokes. Moreover, “right arm motor deficit” and “left arm motor deficit” were combined into one component defined by the maximum of those two scores so that stroke sidedness was not a factor, and the same was also conducted with “leg motor deficit.”

Results

Demographic data for cases and control groups are seen in Table 1. While the case group had a higher average age (67.0 vs. 60.3 years, p = 0.0001), both groups had a similar number of males and females. The average NIHSS of cases was higher than controls (12.5 vs. 6.5, p < 0.0001). With respect to time to evaluation, NIHSS was obtained at an average of 14 minutes after the patient arrived at our hospital for each group (p = 0.8355).

Table 1.

Demographic characteristics of participants (n = 397).

Characteristic	Cases (n = 125)	Controls (n = 272)	p
Age, mean (SD)	67.0 (14.5%)	60.3 (16.3%)	0.0001
Male, n (%)	60 (48.0%)	67 (55.1%)	–
NIHSS, mean (SD)	12.5 (7.7)	6.5 (7.0%)	<0.0001
NIHSS time (min), mean (SD)	14 (25)	14 (24)	0.8355

NIHSS: National Institutes of Health Stroke Scale.

Details of LVOs are summarized in Table 2, which include sidedness, imaging, and vessel occluded. Control group information is summarized in Table 3. Etiologies of stroke mimics included TIA, seizure, infection, and hypertensive emergency.

Table 2.

Characteristics of patients with large-vessel occlusion (LVO).

Cases	N = 125
Right, n (%)	55 (44.0%)
Left, n (%)	63 (50.4%)
Midline basilar, n (%)	7 (5.6%)
Imaging, n (%)
CTA alone	63 (50.4%)
MRA alone	6 (4.8%)
CTA/MRA	56 (44.8%)
Vessel, n (%)
ICA	23 (18.4%)
ACA	6 (4.8%)
MCA	71 (56.8%)
PCA	9 (7.2%)
Basilar	6 (5.6%)
VA	10 (12.5%)
Multi-vessel	10 (12.5%)

CTA: computed tomography angiography; MRA: magnetic resonance angiography; ICA: internal cerebral artery; ACA: anterior cerebral artery; MCA: middle cerebral artery; PCA: posterior cerebral artery; VA: vertebral artery.

Table 3.

Characteristics of stroke mimics and patients without large-vessel occlusion (LVO).

Controls	N = 272
Stroke	65 (24.0%)
Non-stroke	207 (76.0%)
Stroke, n (24.0%)
Hemorrhagic	26
Non-hemorrhagic	39
Non-stroke, n (76.0%)
Transient ischemic attack (TIA)	115
Seizure	14
Dementia/Encephalopathy	8
Infection	12
Hypertension	13
Metabolic	3
Medication	2
Migraine	8
Neurosurgicala	5
Dissection	2
Catatonia	2
Other	23

^aNeurosurgical denotes neurosurgical problems such as aneurysm rupture or brain tumor that may have presented as a code stroke.

The subset of three NIHSS items with the largest modified AUC (optimized for specificity) was maximum “Arm Motor Deficit,” “Sensory Deficit,” and “Extinction.” Using a cutoff of five out of a total eight possible points, the sum score for these items has a 90.2% specificity and 16.0% sensitivity for anterior circulation stroke.

Discussion

The goal of our retrospective, case-control study was to identify a rapid three-item modified NIHSS that can identify, with high specificity, patients with LVO in the setting of acute stroke. After comparing 125 cases of LVO compared to 272 stroke mimics and non-LVO strokes, the subset of questions with the greatest specificity for LVO and highest AUC included: “Arm Motor Deficit,” “Sensory Deficit,” and “Extinction.” This combination gave us a specificity of 90.2% despite a low sensitivity of 16.0%.

NIHSS is the gold-standard stroke grading scale owing to its reliability and validity for acute and long-term outcomes.^7,10–13 Fischer et al.²³ evaluated 226 patients presenting to the ED with concern for stroke and compared NIHSS scores to arteriography results. An NIHSS ≥ 12 had a positive predictive value (PPV) of 91% and a sensitivity of 81% for predicting an LVO; however, the authors excluded stroke mimics, thus tailoring their cohort to a population not consistent with that seen by first responders and first-line hospitals. Similar studies have looked only at patients with confirmed stroke, thus the accuracy of NIHSS cut-off scores will be overestimated^11,12,24,25 whereas to establish the utility of a triage scoring system, the control population should reflect that which is actually seen, rather than a selected group of patients with known stroke pathology. Moreover, each of these studies used the entire NIHSS, which can be cumbersome. Real-world, rural triage practices often do not provide the opportunity for formal NIHSS evaluation at the point of acute decision making. As a result, a surrogate, simpler scale may be helpful. This has led to the development of a number of more easily applied pre-hospital stroke scales; discussed below.

The LAMS is a brief three-item stroke severity measure designed for prehospital and ED use.⁸ Nazliel et al.²⁶ evaluated 119 patients with anterior circulation stroke and correlated LAMS and NIHSS to evaluate the predictive value for vascular occlusion. Seventy-four (62%) had confirmed arterial occlusion, though the presence or absence of large vessel involvement was not known. The average time from normal to ED was 190 minutes, and time to LAMS and NIHSS was 254 minutes—a mean difference of 64 minutes from presentation to evaluation. The CPSS is a similar three-item scale developed from the NIHSS, first developed in 1997,²⁷ with reproducibility between physicians and prehospital personnel.²⁸ One study showed that stroke was missed more frequently by emergency medical services (EMS) providers when the CPSS was not documented.²⁹ Several studies document the benefit of a shorter stroke scale; however, there are challenges inherent when applying these scales in the context of stroke triage and transfer paradigms.

Gupta and co-authors retrospectively reviewed 442 patients across nine stroke centers to determine if high-volume (HV) centers more efficiently delivered endovascular treatment.³⁰ Multivariate analysis showed that patients treated at HV centers were more likely to have good clinical outcomes (odds ratio (OR) 1.86, 95% confidence interval (CI) 1.11 to 3.10, p < 0.018) and successful reperfusion (OR 1.82, 95% CI 1.16 to 2.86, p < 0.008) compared to those treated at lower-volume centers. These results were confirmed by another multicenter study across three institutions evaluating 91 patients with acute stroke. The authors found that time interval from computed tomography (CT) to microcatheter were significantly longer in non-level I trauma centers.³¹ Reinforcing the need for an efficient way to identify candidates for MT, one study found a delay from admission to groin puncture during weekends and after hours.³²

Despite these data, it is a resource-intensive process to direct all stroke transfers to HV centers. Additionally, whether desired or not, there are significant economic implications for non-MT-enabled hospitals should large numbers of stroke patients be diverted. As a result, for the time being, it may be that an appropriate triage scale would be designed to identify LVO with a high degree of specificity, despite a relatively low sensitivity. This would allow front-line caregivers to be confident that more than 90% criteria-meeting patients, out of a population of all stroke symptom-presenting patients (including mimics), will have an LVO, thereby making transfer or directed EMS delivery of these patients a very reasonable consideration.

It is our hope that utilization of a brief NIHSS subset might significantly improve transfer times and provide a more rapid identification of patients who can benefit from MT with a high degree of specificity. Moreover, the three-item NIHSS score is not meant to be used in all clinical settings. Rather, as cost savings becomes an increasingly important aspect of health care, this score can be used in select clinical settings where economic constraints prevent transfer of all stroke patients to a higher level of care. Our three-item scale is a starting point to detect LVO with high specificity, and should not be used alone. It is our hope that this shortened scale be used in conjunction with other means of stroke triage to provide the highest level of patient care possible to transfer patients in need of acute thrombectomy.

Limitations

This study is a retrospective analysis, and as such, is subject to all potential bias inherent with a retrospective design. However, all data points collected were prospectively documented in the medical record, and no inferences were made from physician histories or exams. There was no use of clinical interpretation by the authors. In addition, our cohort included vertebral and M3 thrombi more than proximal anterior circulation strokes. These strokes extend beyond the most commonly accepted candidates for intervention, which may have biased our sample. Moreover, the presented scale has a relatively low sensitivity. This is by design, as the intent was to develop a score that, when applied to a broad population of patients presenting with stroke symptoms (including mimics), would provide a high degree of certainty that an LVO was present. It is a significant limitation that this tool will not identify many LVO patients; however, for those environments with limited resources, the use of a highly specific scale should help ensure appropriate allocation of those scarce resources when activated.

Figure 1.

The modified ROC curve for this combination of items.

ROC: receiver operating characteristic; NIHSS: National Institutes of Health Stroke Scale.

Conclusion

In a real-world setting of limited health care resources and front-line stroke care expertise, the identification of a simple three-point stroke assessment tool that provides a high level of specificity may provide significant value. While the low sensitivity of this tool will ensure that numerous LVO patients will not be identified, the three-item score will maximize diagnostic accuracy for interhospital transfer and regionalization of care. With more than a 90% specificity, this tool will allow those areas with scarce resources to have a high degree of confidence when deciding to initiate transfer or directed EMS delivery.

Declaration of conflicting interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors received no financial support for the research, authorship, and/or publication of this article.