The Accuracy of Large Vessel Occlusion Recognition Scales in Telestroke Setting

Abstract

Introduction: A significant proportion of acute ischemic stroke (AIS) patients who are evaluated through telestroke consultation are transferred to thrombectomy-capable stroke centers (TSCs) for concern of large vessel occlusion (LVO). Patient triage selection is commonly based on the clinical suspicion of LVO, which lacks specificity and could result in unnecessary transfers. In this study, we aimed to assess the accuracy of the most commonly used LVO recognition scales in telestroke setting.

Methods: AIS patients transferred to TSCs for suspicion of an LVO were included in this retrospective study. Patients were evaluated by a stroke neurologist through a telestroke consult before transfer. The National Institute of Health Stroke Scale (NIHSS) score documented by the stroke neurologist was retrieved from medical records and used to calculate five other LVO recognition scales (Rapid Arterial Occlusion Evaluation Scale [RACE], Field Assessment Stroke Triage for Emergency Destination [FAST-ED], Cincinnati Prehospital Stroke Severity Scale [CPSSS], 3-item stroke scale [3I-SS], and Prehospital Acute Stroke Severity Scale [PASS]). We calculated the sensitivity, specificity, accuracy, positive and negative predictive values, false positive rate (FPR), and false negative rate (FNR) of each score using published cutoffs and then examined all possible cutoff values for each of these scales in addition to the NIHSS.

Results: A total of 439 patients were included in the final analysis. A total of 48.5% of patients had an LVO confirmed on computed tomography angiogram. RACE score had the highest accuracy (78%). Overall, the five derived LVO recognition scores have at least 10% FNR. When examining all possible cutoff values, the NIHSS (cutoff of 6) had a 3% FNR but 73% FPR (false transfer).

Conclusion: The use of the NIHSS and other LVO recognition scores over telestroke may result in unnecessary transfers. Better diagnostic tools that could maximize sensitivity with acceptable specificity are urgently needed.

Introduction

Treatment of acute ischemic stroke patients largely depends on achieving reperfusion of the brain tissue.¹ Mechanical thrombectomy (MT) has recently been shown to have a high success rate in achieving reperfusion and improving functional outcome in patients with acute stroke secondary to large vessel occlusion (LVO).^2–8 MT is primarily available in thrombectomy-capable stroke centers (TSCs), which include large primary stroke centers, and comprehensive stroke centers (CSCs). Under the current system of care, patients presenting to community hospitals are evaluated through a telestroke network for eligibility for tissue plasminogen activator, and are transferred to the nearest TSC for thrombectomy evaluation when an LVO is suspected. Suspicion of LVO largely depends on the National Institute of Health Stroke Scale (NIHSS) and the presence of cortical signs. However, this approach is not unified and often lacks specificity, which results in unnecessary transfers of patients without LVO to TSC (false transfer). In a case series of 157 patients who were transferred for thrombectomy evaluation, only 46% underwent MT,⁹ and the most common reason for false transfer was the absence of LVO. False transfer does not only increase TSC volume workload but also remarkably increases the cost of patients' care and complications secondary to transfer.¹⁰

Multiple efforts have been made to develop clinical scales that could identify patients with LVO in a prehospital setting with promising results.^11,12 However, none of these scales have been validated in telestroke setting. In this study, we aimed to investigate the accuracy of LVO scores in identifying LVO in telestroke patients and compare it with the NIHSS.

Methods

Setting

The Medical University of South Carolina (MUSC) telestroke program was established in 2008 to provide evidence-based stroke care to patients in rural South Carolina. MUSC telestroke network works in a hub (MUSC main hospital–the CSC) and spoke (community hospitals) paradigm.¹³ The number of participating spoke sites covered by the MUSC telestroke network has increased from 5 sites in 2008 to 26 sites in 2017.¹⁴ MUSC telestroke program was detailed in previous report.¹³

Study Population

We performed a retrospective chart review of all consecutive telestroke patients who were transferred to MUSC hub for suspected LVO during study period from May 2014 to March 2018. Exclusion criteria were patients with hemorrhagic stroke on initial computed tomography (CT) scan at spoke center (before transfer), patients who received tissue plasminogen activator (tPA) at spoke before transfer, and patients who did not receive vessel imaging after arrival to MUSC. We excluded tPA-treated patients because tPA could lead to recanalization in 20–50% of LVO and subsequently could have led to a false negative CT angiogram ultimately resulting in result bias.¹⁵ The following data were collected from the charts: age on presentation, gender, race, vascular risk factors, NIHSS before transfer, whether patient had LVO on CT angiography (CTA) after arrival to the TSC, and the location of LVO on cerebral vessel imaging.

LVO was defined as an occlusion of the proximal or distal internal carotid artery, first or second segment of the middle cerebral artery (M1 and M2), first segment of posterior cerebral artery (P1), first segment of anterior cerebral artery (A1), vertebral artery, and basilar artery.

LVO Recognition Scales

We used NIHSS that was calculated and documented by the consulting stroke neurologists during the telestroke consults to calculate the following scales: 3-item stroke scale (3I-SS),¹⁶ Rapid Arterial Occlusion Evaluation Scale (RACE),¹⁷ Cincinnati Prehospital Stroke Severity Scale (CPSSS),¹⁸ Prehospital Acute Stroke Severity Scale (PASS),¹⁹ and the Field Assessment Stroke Triage for Emergency Destination (FAST-ED) stroke scale.²⁰ We used the cutoffs that were suggested by the initial publication for each scale (Table 2).

Table 2.

False Negative Rate, False Positive Rate, and Overall Accuracy of Each Score in Identifying Large Vessel Occlusion

SCALE	CUTOFF POINT	FNR (1-SENSITIVITY)	FPR (1-SPECIFICITY)	OVERALL ACCURACY
NIHSS	≥6	3%	73%	61%
	≥11	14%	40%	72%
RACE score	≥5	11%	33%	78%
FAST-ED	≥4	17%	32%	75%
CPSSS	≥2	17%	40%	71%
PASS	≥2	10%	49%	70%
3I-SS	≥4	53%	15%	66%

3I-SS, 3-item stroke scale; CPSSS, Cincinnati Prehospital Stroke Severity Scale; FAST-ED, Field Assessment Stroke Triage for Emergency Destination; FNR, false negative rate; FPR, false positive rate; LL, likelihood ratio; PASS, Prehospital Acute Stroke Severity; RACE, Rapid Arterial Occlusion Evaluation Scale.

Statistical Analysis

Demographic variables for the study participants were described as mean and standard deviation (SD) for quantitative variables or as frequency and percentage for categorical variables.

We calculated the false-positive (1-specificity) and false-negative (1-sensitivity) rates, sensitivity, specificity, accuracy, positive, and negative predictive values of each score. We used the published cutoffs to first determine these values. Secondarily, we considered all possible cutoff values for a given scale to identify the cutoff that maximized the sum of sensitivity and specificity. This study was approved by the Institutional Review Board of MUSC.

Results

During the study period, a total of 1,130 patients were transferred to the TSC. Of whom, 511 patients fit the inclusion criteria (Supplementary Fig. S1). Seventy-two (14%) patients were excluded due to missing NIHSS in their record, leaving 439 patients to be included. Study flowchart is in the supplemental material. Baseline characteristics of our patients are detailed in Table 1. Mean (SD) age was 66.7 years (15.3), and 47.3% of patients were female. Mean (SD) NIHSS was 14.2 (7.9). Two-hundred thirteen patients (48.5%) had LVO confirmed on CTA upon arrival to the TSC. Among patients with confirmed LVO, 194 (91%) patients had occlusion of the anterior circulation arteries, and 19 (9%) patients had occlusion of the posterior circulation arteries.

Table 1.

Baseline Characteristics of Study Cohort

VARIABLE NAME	VALUE
Ageb	66.7 (15.3)
Femalea	208 (47.3)
Whitea	216 (49.2)
Hypertensiona	284 (64.7)
Afiba	98 (22.3)
CADa	67 (15)
DMa	140 (31.9)
HLDa	8 (1.8)
NIHSSb	14.2 (7.9)
LVOa	213 (48.5)
LVO in the anterior circulationa	194 (91)
LVO in the posterior circulationa	19 (9)

^aValues are n (%).

^bValues are in mean (SD).

Afib, atrial fibrillation; CAD, coronary artery disease; DM, diabetes mellitus; HLD, hyperlipidemia; LVO, large vessel occlusion; NIHSS, National Institute of Health Stroke Scale; SD, standard deviation.

Table 2 presents the overall accuracy, false negative rate (FNR), and false positive rate (FPR) for each of the stroke scales. FNR represents the proportion of patients with LVO that would not have been transferred to the hub despite LVO (missed LVO), in contrast FPR represents the proportion of patients without LVO who could have been transferred to the hub (false transfer).

RACE score (using cutoff of 5) had the highest accuracy (78%). If RACE score was used to triage patients, 11% of patients would not have been transferred despite LVO. In contrast, using NIHSS of 6 would have resulted in only 3% of missed LVO but 73% of false transfer. Table 3 and Figure 1 depict the comparative performance of all included LVO scores using suggested cutoff values.

Table 3.

Performance of Scales in Identifying Large Vessel Occlusion Using Published Cutoff

MODEL	C-STATISTIC	CUT-POINT	SENSITIVITY	SPECIFICITY	PPV	NPV	OVERALL ACCURACY	POSITIVE LL RATIO	NEGATIVE LL RATIO
NIHSS	0.769	≥6	97%	27%	55%	91%	61%	1.32	0.11
		≥11	86%	60%	67%	82%	72%	2.13	0.23
RACE score	0.820	≥5	89%	67%	72%	87%	78%	2.69	0.16
FAST-ED	0.809	≥4	83%	68%	71%	81%	75%	2.61	0.25
CPSSS	0.768	≥2	83%	60%	66%	79%	71%	2.09	0.28
PASS	0.739	≥2	90%	51%	63%	84%	70%	1.84	0.20
3I-SS	0.761	≥4	47%	85%	74%	63%	66%	3.03	0.63

NPV, negative predictive value; PPV, positive predictive value.

Fig. 1.

Comparative analysis of six scales in identifying LVO among patients transferred to TSCs for suspected LVO. LVO, large vessel occlusion; ROC, receiver operating characteristic; TSCs, thrombectomy-capable stroke centers.

In addition, we studied all possible cutoff points for each score to determine the cutoff with the highest sum of sensitivity and specificity. An NIHSS score of 11 had the highest sum of sensitivity and specificity. RACE, CPSS, FAST-ED, and PASS had the highest sum of sensitivity and specificity at the same cutoff values as those reported in the initial publication of each score.^17–20 3I-SS had a higher accuracy and sum of sensitivity and specificity when using cutoff point of ≥3 instead of the suggested cutoff point of ≥4 that was suggested in the initial publication (Supplementary Table S1).¹⁶

Most of the patients transferred had findings consistent with acute stroke on brain imaging. Stroke mimics were seen in 27 patients (6.2%) (Supplementary Fig. S2). The most common stroke mimic seen was postictal paralysis, which was seen in 12 patients. Other stroke mimics noted were sepsis in three patients, migraine with aura in two patients, recrudescence of old stroke in two patients, posterior reversible encephalopathy syndrome in two patients, sedating medications in two patients, hyperglycemia in one patient, hypertensive emergency in one patient, intracerebral hemorrhage that was not seen on CT at the spoke site in one patient, and subarachnoid bleeding in one patient.

All LVO scores had a poor accuracy in identifying stroke mimics (24–68%) (Table 4). 3I-SS had the highest accuracy (68%), however, FPR was 30%.

Table 4.

False Negative Rate, False Positive Rate, and Overall Accuracy of Each Score in Identifying Stroke Mimics

SCALE	CUTOFF POINT	FNR (1-SENSITIVITY)	FPR (1-SPECIFICITY)	OVERALL ACCURACY
NIHSS	>6	12%	81%	24%
	>11	19%	62%	41%
RACE score	5	22%	59%	43%
FAST-ED	4	41%	57%	44%
CPSSS	2	41%	61%	40%
PASS	2	26%	68%	34%
3I-SS	4	59%	30%	68%

Discussion

This is the first study to investigate the accuracy of LVO recognition scales in telestroke patients who were transferred to TSCs for suspected LVO. We included RACE, PASS, CPSS, FAST-ED, and 3I-SS scores, along with NIHSS. We found that using LVO scores to triage telestroke patients could result in missing at least 10% of acute stroke patients with LVO. The only exception was when using NIHSS with a score ≤6, which had FNR of 3% but FPR of 73%. In our study, RACE score (cutoff of 5) had the highest accuracy, which accords with previous studies in the prehospital setting.²¹

One of the most challenging problems in telestroke is making the decision regarding transfer to a TSC versus no transfer. Since the publication of the five pivotal thrombectomy trials in 2015, a remarkable emphasis has been placed on developing tools that could identify LVO in patients with suspected ischemic stroke.²² Los Angeles Motor Scale (LAMS), FAST-ED, 3I-SS, CPSS, PASS, and RACE were the most widely used and studied clinical scales. In a comparative study of LVO scores in tPA patients, accuracy, sensitivity, and specificity were in the range of 0.74–0.78, 50–69%, and 83–89%, respectively.¹⁹ Another comparative study of eight scales in ED patients with cerebral ischemia showed lower accuracy (0.62–0.7) and sensitivity (42–64%) than the previously mentioned study, but higher specificity (81–96%). Discrepancy of the results could be attributed to different inclusion criteria and different LVO definitions.¹² The largest comparative study to date evaluated the performance of 13 different clinical stroke scales in 1,004 ischemic stroke patients. In that study, the author found that using a published cutoff point for the LVO scores was associated with FNR of at least 20%.²¹

It should not be surprising that sensitivity (ranging from 47% to 97%) in our study was higher than that of previously published studies.^11,21 This finding has multiple potential explanations. First, we excluded patients with hemorrhagic stroke and included only patients who were transferred to the hub for suspected LVO. Second, we considered M2, P1, and A1 as LVO due to accumulating evidence, suggesting the benefit of thrombectomy in these groups of patients.^23,24 Third, all scales were derived from NIHSS that was documented by a stroke neurologist.

Identifying stroke mimics is challenging in telestroke setting and relies mainly on the examiner expertise and available brain imaging. In our study, 6.2% of the patients had stroke mimics, which is lower than that reported in the literature (20–40%).^25,26 The low stroke mimics rate in our study stems from the fact that all patients included in our study were evaluated by stroke neurologist through telestroke and were deemed to have possible LVO before transfer. Even though multiple efforts have been made to develop scores that can be used to differentiate stroke mimics for ischemic stroke, to date there is no reliable scale that is routinely utilized.^25,26 In our study, all studied LVO scores did poorly in identifying stroke mimics.

LVO recognition scores generally rely on motor deficit with or without cortical findings to predict LVO. It is not uncommon for non-LVO stroke patients to present with severe hemiparesis or cortical findings mimicking LVO strokes.^27–29 Moreover, up to 7% of patients with LVO could present with NIHSS of 0.³⁰ In a study of 565 ambulance-initiated stroke alerts, 38% of anterior circulation LVO patients presents with atypical presentation.¹¹ LVO scores were able to identify anterior circulation LVO in <20% LVO patients with atypical presentation versus ≥95% of LVO patients with typical presentation.¹¹

After the extension of thrombectomy eligibility with the publications of DAWN and DEFUSE trials,^7,8 patients transfer will likely remarkably increase, likewise, false transfers will probably increase. A study in the United States by Sonig et al. evaluating 1,311,511 inpatient stroke admissions from 2008 to 2010 revealed that transferred patients undergoing MT and tPA had an average expenditure of $27,000 more than nontransferred patients.¹⁰ In addition to increasing cost, false transfer increases workload in the TSC and patient/family discomfort. Thus, a more sensitive and specific tool is urgently needed to improve telestroke triage system and decrease false transfer.

CT angiogram and perfusion are now increasingly incorporated in telestroke.³¹ A simulation study by Boulouis et al. evaluated 508 patients transferred from spoke sites for thrombectomy evaluation and found that doing vascular imaging at spoke sites can be a useful triaging method and can avoid 20% of futile transfers.³² In addition, obtaining vascular imaging at spoke before transfer can help in not only identifying intracranial occlusions but also assessing collateral blood vessels, which may further provide additional information to determine thrombectomy candidacy.^33,34

Several experimental technologies have been proposed for detection of LVO. The volumetric impedance phase shift spectroscopy (VIPS) device is a noninvasive device capable of detecting an asymmetric of bioimpedance between two hemispheres that could result from ischemic brain injury.³⁵ A pilot study investigating the accuracy of VIPS device in detecting LVO in patients transferred from satellite hospitals to CSCs for suspected LVO demonstrated a sensitivity of 86% and a specificity of 77% for LVO.³⁵

Our study has multiple limitations that should be considered while interpreting our results. First, LVO recognition scales were scored based on NIHSS, which did not allow us to evaluate other LVO recognition scores that require elements not included in NIHSS such as grip strength. However, in previous studies evaluating LVO scores, RACE and FAST-ED had the highest accuracy, therefore, including other LVO scores would unlikely alter our study results. Second, retrospective design is a methodological shortcoming. Notably, our data including NIHSS were collected prospectively and updated periodically as part of our telestroke registry, so while the records were retrospectively reviewed, the NIHSS score was collected at the time of patient evaluation. Finally, patients who received tPA were not included in the study, which could be a source of bias given that tPA patients may have higher likelihood of having “real stroke” and, arguably, may have higher likelihood of having LVO. However, tPA is associated with considerable rate of recanalization, which may alter the results of the analysis.

Conclusion

Our study showed that LVO scores should not be used to triage telestroke patients, and until we have better tools, all patients with clinical examination concerning for LVO should be transferred to TSCs.

Statement of Compliance

Authors confirm that the study is an observational minimal risk study and no consent is required per the Medical University of South Carolina Institute policy. Our study was approved by the institutional review board of the Medical University of South Carolina.

Authors' Contributions

M.A. performed study concept and design, data interpretation, and article writing; E.A. was involved in data collection, data interpretation, and article writing; A.H.W. carried out statistical analysis; C.A.H. contributed to critical revision of the article for important intellectual content and study supervision.

Disclosure Statement

No competing financial interests exist.

Supplementary Material

Supplementary Figure S1

Supplementary Figure S2

Supplementary Table S1