Performance of the Vision, Aphasia, Neglect (VAN) assessment within a single large EMS system

Abstract

Background:

There is limited evidence on the performance of emergent large vessel occlusion (LVO) stroke screening tools when used by emergency medical services (EMS) and emergency department (ED) providers. We assessed the validity and predictive value of the Vision, Aphasia, Neglect (VAN) assessment when completed by EMS and in the ED among suspected stroke patients.

Methods:

We conducted a retrospective study of VAN performed by EMS providers and VAN inferred from the National Institutes of Health Stroke Scale (NIHSS) performed by ED nurses at a single hospital. We calculated sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) of VAN by EMS and in the ED for LVO and a combined LVO and intracerebral hemorrhage (ICH) outcome.

Results:

From January 2018 to June 2020, 1,547 eligible patients were identified. Sensitivity and specificity of ED VAN were similar for LVO (72% and 74%, respectively), whereas EMS VAN was more sensitive (84%) than specific (68%). PPVs were low for both EMS VAN (26%) and ED VAN (21%) to detect LVO. Due to several VAN-positive ICHs, PPVs were substantially higher for both EMS VAN (44%) and ED VAN (39%) to detect LVO or ICH. EMS and ED VAN had high NPVs (97% and 96%, respectively).

Conclusions:

Among suspected stroke patients, we found modest sensitivity and specificity of VAN to detect LVO for both EMS and ED providers. Moreover, the low PPV in our study suggests a significant number of patients with non-LVO ischemic stroke or ICH could be over-triaged with VAN.

Introduction

Large vessel occlusions (LVOs) make up roughly one-third of acute ischemic strokes but, given their severity, account for over 60% of stroke-related morbidity and mortality.¹ Endovascular thrombectomy (EVT) was recently demonstrated to significantly improve long-term outcomes for patients with emergent LVO stroke.² Moreover, the benefit of EVT is highly time-dependent; faster reperfusion in even 15-minute increments is associated with less disability.³ Currently, many eligible patients experience delays to EVT treatment prior to hospital arrival and during initial triage and evaluation.⁴ Therefore, early identification and triage of potential LVO is now an essential component of an efficient and effective stroke system of care.^5,6

While emergency medical services (EMS) screening for stroke-like symptoms is an established practice in prehospital management, LVOs are less frequently encountered in the field than non-LVO ischemic strokes, and LVO symptoms can be indistinguishable from other types of stroke, making early identification of LVO a significant challenge. There are several stroke severity scales to rapidly assess suspected stroke patients for emergent LVO. However, no single LVO screen has demonstrated both high sensitivity and high specificity, with sensitivity ranging from 47%−73% and specificity from 78%−90%.⁷ Despite limited accuracy of these tools, their use by EMS to transport potential LVO patients directly to EVT-capable facilities and to activate EVT treatment teams prior to hospital arrival is associated with faster times to EVT.^{8, 9}

One quick and simple LVO screen is the Vision, Aphasia, Neglect (VAN) assessment, which was designed to capture the major cortical signs of an LVO stroke.¹⁰ In a small study of 62 acute ischemic stroke patients from which it was developed, VAN had 100% sensitivity and 90% specificity to identify LVO when completed by emergency department (ED) nurses.¹⁰ However, a recent external validation of VAN used by EMS reported 81% sensitivity and 38% specificity, with the majority of intracerebral hemorrhages (ICH) found to be VAN positive.¹¹ While initial results on the validation of VAN are encouraging, additional investigation is needed on the predictive value of VAN when implemented in a stroke system of care.

The objective of this study was to assess the real-world performance of VAN completed by EMS in the field and in the ED among suspected stroke patients including ischemic and hemorrhagic strokes and stroke mimics. To explore its use in stroke systems, we evaluated the VAN assessment to predict LVO in addition to a combined LVO or ICH outcome.

Methods

Study Design and Setting

We conducted a retrospective study with existing data on suspected stroke patients presenting to a single community hospital, by either EMS or private vehicle, between January 2018 and June 2020. The study setting was a large metropolitan area of about 1 million residents in central North Carolina and located within the Stroke Belt, a multi-state region in the Southeast United States with a disproportionately higher burden of stroke morbidity and mortality than the rest of the country.¹² The study hospital has been a Joint Commission-certified stroke center since 2011, was certified as a Thrombectomy-Capable Stroke Center in September 2019, and had EVT capabilities throughout the study period along with two other stroke centers in the region.

The local EMS system provides sole 9–1-1 response and ambulance transport services to an urban/suburban county located in central North Carolina, with an area of 854 square miles and a population of 1,000,000+ residents and receives about 110,000 calls per year. The system operates with centralized medical oversight at an advanced life support (ALS) level, with at least one paramedic on each ambulance. Since 2009, an EMS triage and destination plan determines the appropriate facility to transport patients screening positive for acute stroke symptoms. In August 2018, the EMS agency added VAN to its prehospital stroke screening protocol. EMS personnel were trained on VAN across July, August, and September 2018 during usual continuing education sessions. The stroke triage and destination plan was updated to route VAN positive patients to the closest EVT-capable stroke center.

The data source for this study was a hospital-based quality improvement database of “Code Stroke” patients, i.e., those experiencing acute onset of stroke-like symptoms within 24 hours that led to an activation of the ED acute stroke clinical pathway and the hospital stroke team. Among acute ischemic stroke patients considered for EVT, computed tomography angiography (CTA) imaging is ordered and completed concurrently with the initial neurologic assessment and intravenous alteplase treatment decision and administration. The stroke center team continually updates this database with Code Stroke patient demographics and medical history, initial stroke evaluation and treatment, ED diagnosis, and discharge disposition. Additionally, prehospital data from EMS records are linked directly to the hospital electronic health record system with the ESO (Austin, TX) Health Data Exchange. The University of North Carolina Institutional Review Board approved this study by expedited review.

Study Population

We included “Code Stroke” patients that were activated from the field by EMS or in the ED. Eligible patients arrived at the hospital by private vehicle or EMS. To represent the real-world performance of VAN, cancelled activations because the patient was determined not a stroke upon arrival were included. Patients were ineligible if less than 18 years old, in-hospital strokes, or transferred from an outside hospital. Patients were excluded from the analysis if emergent LVO could not be determined. For the analysis of VAN assessed in the ED, we used data on patients starting in January 2018. Since VAN was added to the EMS stroke protocol in August 2018, the analysis of VAN assessed by EMS allowed for a ramp-up period and used data starting in September 2018.

Key Measures

The presence of an emergent LVO was determined by initial CTA imaging findings. Specifically, LVO was defined by an acute thromboembolic occlusion of the terminal intracranial carotid, M1 and M2 segments of the middle cerebral, A1 segment of the anterior cerebral, P1 segment of the posterior cerebral, or basilar arteries. Patients with non-ischemic stroke diagnoses were classified as non-LVOs. These diagnoses included transient ischemic attack (TIA), intracerebral hemorrhage (ICH), subarachnoid hemorrhage (SAH), subdural hematoma (SDH), and stroke mimics (e.g., hypoglycemia, seizure, migraine).

EMS VAN results were retrieved directly from linked electronic records of prehospital care. VAN started with an assessment of arm strength. If weakness was present in one or both arms, the assessment proceeded to identify visual disturbance, aphasia, or neglect, in order. A patient was VAN positive if arm weakness and one of the VAN deficits were detected. While ED providers at our stroke center are not required to complete a VAN, the ED nurses complete the NIHSS for all Code Stroke patients upon arrival. Therefore, ED VAN was inferred from corresponding items on the initial NIHSS (i.e., Motor Arm, Vision, Best Language, Best Gaze, and Extinction and Inattention) completed by the ED nurse.

We extracted patient demographics (age, gender, and race), medical history (atrial fibrillation, coronary artery disease, carotid stenosis, diabetes, drugs/alcohol abuse, dyslipidemia, hypertension, obese/overweight, and previous stroke/TIA), time from last known normal to hospital arrival, initial NIHSS score, administration of intravenous alteplase, and final diagnosis. For EMS arrivals, we also categorized patients by EMS provider impression (stroke/TIA, altered mental status, generalized weakness, and other).

Statistical Analysis

The study population was characterized with descriptive statistics. Sensitivity and specificity and positive and negative predictive values (PPV and NPV) were computed for the ED and EMS VAN versus an LVO diagnosis and a combined LVO or ICH outcome. A 95% confidence interval (CI) was calculated for each proportion using normal approximation. The agreement between the ED and EMS VAN findings was assessed with percent agreement and kappa.

The main analysis was conducted after excluding patients with missing assessments. In a sensitivity analysis of incomplete documentation of the VAN assessment by EMS, we used multiple imputation by chained equations to create 100 imputed datasets based on observed data.¹³ The imputation model included covariates that could be predictors of missing EMS VAN (age, gender, race, EMS provider impression, time from last known normal to arrival, and initial NIHSS score). Sensitivity and specificity of EMS VAN to detect LVO were computed for each imputed dataset. Pooled estimates and 95% CIs account for the uncertainty associated with the imputation. All statistical analyses were conducted in SAS 9.4.

Results

We identified 1,708 suspected stroke patients, 161 of whom (9%) did not have an initial NIHSS completed in the ED or initial CTA imaging to determine whether there was an LVO, leaving 1,547 patients for the ED VAN analysis (Figure 1). Patients were on average 70 years old, and 56% were female (Table 1). The study population was predominantly Caucasian (73%) although a substantial proportion were African American (23%). Most patients (75%) arrived by EMS, and almost half (49%) arrived over 6 hours since time last known normal. The mean initial NIHSS score was 7.1, and 62% of patients had a score less than 6. Although all patients were initially under suspicion of acute stroke, 45% had a non-stroke final diagnosis. Starting in September 2018, 873 patients were eligible for the EMS VAN analysis. Of these, 623 (71%) patients had complete documentation of the VAN assessment by EMS. Two hundred fifty patients were excluded because of incomplete VAN documentation. Compared to the patients included in the EMS analysis, excluded patients scored on average 3.5 points higher on the initial NIHSS (Table S1).

Figure 1.

Selection of eligible patients for ED and EMS VAN analyses

Table 1.

Patient characteristics of overall study population (N=1,547)

	No.	%
Age Group
18–49 years	179	12%
50–64 years	360	23%
65–79 years	537	35%
80+ years	468	30%
missing	3
Age, in years (mean, SD)	69.5	15.8
Female Gender	865	56%
Race
African American/Black	353	23%
Caucasian/White	1,095	73%
Other	60	4%
missing	39
Medical History
Atrial Fibrillation	253	16%
CAD/Prior MI	362	23%
Carotid Stenosis	67	4%
Diabetes	374	24%
Drugs/Alcohol Abuse	59	4%
Dyslipidemia	631	41%
Hypertension	1,046	68%
Obesity/Overweight	263	17%
Previous Stroke/TIA	467	30%
Arrival Mode
EMS	1,156	75%
Private vehicle	391	25%
Time from Last Known Normal to Arrival
<3 hours	614	40%
3–6 hours	170	11%
6–24 hours	716	47%
24+ hours1	28	2%
missing	19
Initial NIHSS score
<6	963	62%
≥6	580	38%
missing	4
Initial NIHSS score (mean, SD)	7.1	8.7
Intravenous alteplase treatment	142	9%
Final Diagnosis
Ischemic stroke	549	35%
Transient ischemic attack	190	12%
Intracerebral hemorrhage	98	6%
Subarachnoid hemorrhage	8	1%
Subdural hematoma	21	1%
Non-stroke	681	44%

¹initially treated as <24 hours until time last known normal was obtained

Table 2 summarizes the accuracy and performance of ED and EMS VAN to detect LVO and combined LVO or ICH diagnoses. ED VAN had similar sensitivity (72%) and specificity (75%) for LVO. EMS VAN was more sensitive (84%) than specific (68%) for LVO. PPVs were low for both ED VAN (21%) and EMS VAN (26%) to detect LVO. Almost half of the false VAN positives were diagnosed as non-LVO ischemic stroke or ICH (Table 3). Because of several VAN-positive ICHs, PPVs were substantially higher for both ED VAN (39%) and EMS VAN (44%) to detect LVO or ICH. The NPVs for ED and EMS VAN and both outcomes were greater than 90%.

Table 2.

Validity and predictive values of ED and EMS VAN

	ED VAN (N=1,547)		EMS VAN (N=623)
	LVO Diag.	LVO or ICH Diag.	LVO Diag.	LVO or ICH Diag.
Prevalence of Diag.	9%1	17%	12%2	22%
Sensitivity	0.72 (0.64–0.79)	0.69 (0.63–0.74)	0.84 (0.76–0.92)	0.76 (0.69–0.83)
Specificity	0.74 (0.72–0.76)	0.78 (0.76–0.80)	0.68 (0.64–0.72)	0.72 (0.68–0.76)
Positive Predictive Value	0.21 (0.18–0.25)	0.39 (0.35–0.44)	0.26 (0.21–0.32)	0.44 (0.37–0.50)
Negative Predictive Value	0.96 (0.95–0.98)	0.92 (0.91–0.94)	0.97 (0.95–0.99)	0.91 (0.89–0.94)

¹25% prevalence of LVO diagnosis among ischemic strokes

²29% prevalence of LVO diagnosis among ischemic strokes

Table 3.

Final diagnoses of false positives from ED and EMS VAN

	ED VAN (N=365)	EMS VAN (N=176)
	No. (%)	No. (%)
Non-LVO ischemic stroke	108 (30%)	53 (30%)
Transient ischemic attack	25 (7%)	16 (9%)
Intracerebral hemorrhage	68 (19%)	34 (19%)
Subarachnoid hemorrhage	3 (1%)	0 (0%)
Subdural hematoma	12 (3%)	7 (4%)
Stroke mimic	39 (11%)	25 (14%)
Other	108 (30%)	41 (23%)

Because 29% of eligible patients did not have a documented EMS VAN assessment, multiple imputation with observed covariate data was used to handle missing data. Derived from imputed data, EMS VAN had a sensitivity of 78% (95% CI 70%−87%) and specificity of 67% (95% CI 63%−70%) for LVO (Table S2), which had overlapping 95% CIs with estimates from the main analysis.

Among 623 patients arriving by EMS, the agreement between the EMS and ED VAN results was 73% with a kappa of 0.41 (95% CI 0.33–0.48). Of 171 patients with discordant VAN results, 100 (58%) were VAN positive according to EMS but were VAN negative in the ED whereas 71 (42%) were found to be negative by EMS but positive in the ED. The majority of LVO diagnoses (69%) were both EMS and ED VAN positive. However, 63% of patients who were both EMS and ED VAN positive were non-LVOs.

Discussion

We found modest sensitivity and specificity of VAN to detect LVO in both the prehospital and ED settings. Moreover, PPVs were relatively low in part due to the underlying low prevalence of LVO among our broad population of suspected stroke patients. The high false-positive rate for prehospital LVO screening has important implications for stroke systems implementing EMS triage and routing algorithms that bypass local hospitals for EVT-capable stroke centers. Our findings suggest a significant number of non-LVO patients would be over-triaged with an EMS bypass protocol, which is consistent with prior studies.^14–16 In our study, about one-third of false positives were non-LVO ischemic stroke, suggesting several patients could incur delays to rapid intravenous thrombolysis if unnecessarily transported greater distances.¹⁷ Since VAN was designed to be sensitive to the major cortical symptoms, we found a substantial proportion of false positives were due to ICHs and other stroke mimics that often present with similar signs. As noted by others, patients with ICH may also benefit from preferential transport to advanced stroke centers with neurosurgical capabilities.^18–20 In our study, VAN in the prehospital setting and in the ED had NPVs of greater than 95% for LVO, indicating early screening with VAN would result in substantially fewer false negatives than false positives. Therefore, our findings suggest the under-triage of LVO patients would be a lesser concern when implementing VAN into EMS or ED triage although strategies will still be needed to ensure under-triaged patients do not miss the opportunity for timely receipt of EVT.

Compared to our findings, a recent study by Birnbaum, et al. of EMS use of VAN reported a comparable sensitivity (81%) and lower specificity (38%).¹¹ Incompatibility with our results may be due to differences in the distribution of final diagnoses and the severity of stroke symptoms. Other stroke severity scales like the Rapid Arterial oCclusion Evaluation (RACE)²¹ and the Cincinnati Stroke Triage Assessment Tool (C-STAT)²² have also shown limited accuracy. Pooled C-STAT studies show a sensitivity of 56% and specificity of 82% and RACE show a sensitivity of 69% and specificity of 81% among all suspected stroke patients.⁷ It is expected that scales with more cortical signs like VAN and RACE would have a greater sensitivity but lower specificity than C-STAT and other scales focused on motor deficits.¹³ Our findings suggest VAN is at least as sensitive as RACE but less specific. The RACE scale involves a greater number of items and requires the computation of a score following the completion of each item. Further, it was found to have the lowest feasibility compared to other prehospital scales.²³ Depending on the goals of the stroke system, the advantage of greater specificity provided by RACE needs to be considered in relation to the simplicity and ease of use of the VAN screening tool.

The low PPV of EMS VAN (26%) found in our study is similar to 29% reported by Birnbaum, et al.¹¹ A systematic review of published studies on LVO screens found reported PPVs between 35% and 50% over a wide range of LVO prevalence.⁷ Navalkele, et al. recently showed higher PPV for VAN in the ED (53%) although they studied only confirmed acute ischemic strokes with 31% having an LVO.²⁴ Given the relatively low prevalence of LVO in our broad study population (9%), a lower PPV than those previously reported is not surprising. Conversely, the NPV was higher (97%) than other studies although patients who test negative but have an LVO, even if infrequently, need to be planned for so as not to delay EVT in these potentially eligible patients.

We found the agreement between EMS and ED VAN was moderate although the kappa suggests weak agreement. While the lack of agreement in our study could be due to comparing VAN by EMS to one inferred from the NIHSS in the ED, patients’ symptoms may have also changed between the prehospital and ED settings. Our findings suggest suspected stroke patients could be improving as well as declining over time. Initial evaluation of VAN observed perfect agreement between nurses and physicians,¹⁰ and recently, prehospital providers were found to agree over 90% on the Field Assessment Stroke Triage for Emergency Destination (FAST-ED) stroke severity scale.²⁵ While the inter-rater reliability of VAN and other LVO screening tools may be high within a certain setting, our findings support additional investigation into how evolving symptoms from the field to hospital arrival could influence early LVO screening and triage.

Recent research, in addition to our study, has demonstrated the feasibility of implementing LVO screening tools, such as RACE and VAN, for EMS stroke triage.^11,26 Still, it is critical for a stroke system to understand the magnitude of false positives and false negatives and their potential adverse impact on the utilization of medical resources and patient outcomes within the system’s local and regional context. For well-resourced regions with multiple EVT-capable centers such as our system, over-triage may be less of a concern, and strategies to minimize under-triage would be higher priority. As additional clinical evidence and experience emerges on EVT for acute ischemic stroke due to more distal occlusion sites,²⁷ early triage of EVT candidates based exclusively on symptom severity will likely lead to new challenges. To address the inherent limitations of current symptom-based screening methods, non-invasive brain monitoring technologies, such as ultrasound or electroencephalography devices, may offer novel solutions.^28,29

Our study limitations include data only from a single stroke center and EMS system though a large sample (N=1,547) from a demographically diverse region within the Stroke Belt of the United States. With data on 623 patients arriving by EMS, our study is the largest to date on VAN used in the prehospital setting. Further, our assessment of prehospital VAN was based on a single EMS system with uniform protocols and training. Because we inferred ED VAN from the NIHSS completed by the ED nurse, our results may not be generalizable to EDs that implement the VAN screening tool as it was designed. Also, by inferring VAN from the NIHSS, our evaluation of agreement between EMS and ED providers was not a direct comparison, and the results should be interpreted with caution. A substantial proportion of EMS arrivals did not have complete documentation to determine the VAN assessment, possibly because EMS providers may not have felt VAN was needed to appropriately triage and transport more mild or severe strokes. We observed patients excluded because of missing EMS VAN had on average higher NIHSS scores, suggesting more severe symptoms. Using the NIHSS score and other covariate data, multiple imputation analysis showed similar estimates of sensitivity and specificity. However, this method does not account for non-random missing data.

Conclusions

In this large study of a broad population of suspected stroke patients, VAN had comparable sensitivity and specificity to other stroke severity scales and screening tools for rapid detection of LVO stroke. For a population with a low prevalence of LVO, the modest specificity of VAN may result in a considerable number of false positives, which can have important clinical and economic implications for a stroke system of care. Given the limited accuracy of existing LVO screens, future research is required on the optimal use of these tools within stroke systems accounting for local resources and needs.

Data Availability:

Data are available upon reasonable request. The data are deidentified, retrospective from suspected stroke patients at a single center.