Prehospital stroke scales in urban environments

Abstract

Objective:

To identify and compare the operating characteristics of existing prehospital stroke scales to predict true strokes in the hospital.

Methods:

We searched MEDLINE, EMBASE, and CINAHL databases for articles that evaluated the performance of prehospital stroke scales. Quality of the included studies was assessed using the Quality Assessment of Diagnostic Accuracy Studies–2 tool. We abstracted the operating characteristics of published prehospital stroke scales and compared them statistically and graphically.

Results:

We retrieved 254 articles from MEDLINE, 66 articles from EMBASE, and 32 articles from CINAHL Plus database. Of these, 8 studies met all our inclusion criteria, and they studied Cincinnati Pre-Hospital Stroke Scale (CPSS), Los Angeles Pre-Hospital Stroke Screen (LAPSS), Melbourne Ambulance Stroke Screen (MASS), Medic Prehospital Assessment for Code Stroke (Med PACS), Ontario Prehospital Stroke Screening Tool (OPSS), Recognition of Stroke in the Emergency Room (ROSIER), and Face Arm Speech Test (FAST). Although the point estimates for LAPSS accuracy were better than CPSS, they had overlapping confidence intervals on the symmetric summary receiver operating characteristic curve. OPSS performed similar to LAPSS whereas MASS, Med PACS, ROSIER, and FAST had less favorable overall operating characteristics.

Conclusions:

Prehospital stroke scales varied in their accuracy and missed up to 30% of acute strokes in the field. Inconsistencies in performance may be due to sample size disparity, variability in stroke scale training, and divergent provider educational standards. Although LAPSS performed more consistently, visual comparison of graphical analysis revealed that LAPSS and CPSS had similar diagnostic capabilities.

When recognized in the field, prehospital notification by emergency medical services (EMS) has been associated with improved rates of recombinant tissue plasminogen activator (rtPA) delivery with reduced door-to-needle times.^1,2 Increased use of rtPA and shorter door-to-needle times have both been associated with improved stroke outcomes.^3,4 However, paramedics and emergency medical technicians (EMTs), limited in both time and training, are not able to perform a detailed stroke examination and thus rely on screening tools that are designed to identify potential strokes with minimal assessment.⁵ We conducted a systematic review of the diagnostic accuracy of a variety of prehospital stroke scales. Our primary goal was to identify the prehospital stroke scale with optimal operating characteristics for the diagnosis of stroke.

METHODS

Search strategy.

With the aid of a medical librarian, we searched for studies of prehospital stroke scales in MEDLINE, EMBASE, and CINAHL Plus databases from 1966 until October 2, 2013. We also searched the Cochrane Central Register of Controlled Trials and the bibliographies of the included and relevant articles and reviews. We chose the key words “paramedic,” “stroke,” “transient ischemic attack,” “accuracy,” and “reproducibility” as text words and MeSH terms to identify related studies (table e-1 on the Neurology® Web site at Neurology.org). Two authors (E.S.B., M.S.) reviewed each title for relevance. Titles thought to be relevant by either author were then subjected to further review by the other authors (R.H.S., S.R.L.) to ensure that they met all inclusion/exclusion criteria.

Inclusion and exclusion criteria.

We considered studies in which EMTs or paramedics performed prehospital stroke scales as recommended by the American Heart Association/American Stroke Association.⁶ English articles that studied only adult populations were used. We included studies in which discharge diagnosis of stroke or TIA was used as the standard reference. For this review, we were not concerned with the severity of the stroke; only stroke scales with dichotomous results, i.e., stroke present or absent, were included, because severity indices implied that the diagnosis was already made.

Studies in which physicians were involved in prehospital application of a stroke scale were excluded because physicians are not present in most EMS systems in the United States. All case reports, case reviews, systematic reviews, letters to the editor, and poster presentations were excluded. Studies that did not publish sufficient raw data to calculate operating characteristics were also excluded unless provided by the authors upon request.

Data extraction and quality assessment.

Data from the selected studies were abstracted by 2 authors (E.S.B., M.S.) and were checked for accuracy by 2 other authors (R.H.S., S.R.L.). We used Meta-DiSc⁷ software to calculate the operating characteristics of the various stroke scales as reported in each study. For statistical and visual comparisons, we plotted a series of graphs. The initial graph, receiver operating characteristic (ROC) plane, plotted sensitivity vs false-positive rate for each scale as measured independently in each study. Symmetric summary ROC (SSROC) curves were produced for scales tested in more than 2 studies.

In order to document potential large differences in study methodologies, we used the inconsistency index (I²) and tau squared (τ²) to evaluate between-study heterogeneity, with I² >50% or τ² >1 indicating substantial statistical heterogeneity.^8,9 Fixed-effect models (Mantel-Haenszel) were to be used for comparing statistically homogenous studies and random-effects models (DerSimonian and Laird) were to be used for comparing statistically heterogeneous studies.^8,9 We also generated an ROC ellipse plot to describe the uncertainty of the pairs of sensitivities and false-positive rates.

For studies meeting our inclusion and exclusion criteria, we performed quality assessments using Quality Assessment of Diagnostic Accuracy Studies–2¹⁰ (QUADAS-2), which assesses the quality of studies by identifying sources of bias and concerns regarding applicability. Each of the QUADAS-2 variables was graded by 2 physicians (E.S.B., M.S.) independently and compared for interrater reliability using the kappa coefficient. The QUADAS-2 domains were labeled high, low, or unclear, indicating the degree of bias and concerns regarding applicability. Differences in assessments were adjudicated by consensus and by one senior author (R.H.S.).

RESULTS

Search results.

Our search yielded 254 articles from MEDLINE, 66 titles from EMBASE, and 32 titles from CINAHL Plus. Eight studies^11–18 met all of our inclusion/exclusion criteria (figure 1). Studies by Iguchi et al.,¹⁹ Tirschwell et al.,⁵ and Llanes et al.²⁰ were excluded because their prehospital stroke scales measured stroke severity, not its presence. We excluded a study by Bergs et al.²¹ where emergency physicians and EMTs jointly diagnosed stroke. A study by Frendl et al.²² was excluded because data for 76% of the study patients were missing. We attempted to retrieve raw data from the authors of several studies (Harbison et al.,²³ Nor et al.,²⁴ Frendl et al.,²² and Ramanujam et al.²⁵); however, these data were no longer available to the authors in a usable format. We searched for data on other known prehospital stroke scales including the Miami Emergency Neurological Deficit Scale, the Boston Operation Stroke Scale, and the Birmingham Regional Emergency Medical Services System Scale, but data/articles using these scales could not be found in peer-reviewed journals.

Figure 1. Results of the literature search.

Last updated October 2, 2013.

Description of studies.

We reviewed 8 studies (Kidwell et al.,¹¹ Wojner-Alexandrov et al.,¹² Bray et al.,¹³ Bray et al.,¹⁴ Studnek et al.,¹⁵ Chenkin et al.,¹⁶ Chen et al.,¹⁷ and Fothergill et al.¹⁸) reporting the operating characteristics of 7 different stroke scales: the Cincinnati Prehospital Stroke Scale (CPSS), the Los Angeles Prehospital Stroke Screen (LAPSS), the Melbourne Ambulance Stroke Screen (MASS), Medic Prehospital Assessment for Stroke Code (Med PACS), Ontario Prehospital Stroke Screen Tool (OPSS), Recognition of Stroke in the Emergency Room (ROSIER), and Face Arm Speech Test (FAST). Included studies used stroke scales with overlapping motor elements without any sensory or coordination/cerebellar testing. See figure 2 for a comparison of the various prehospital stroke scales.

Figure 2. Descriptive comparison of different prehospital stroke scales.

Cincinnati Prehospital Stroke Scale (CPSS), Los Angeles Prehospital Stroke Screen (LAPSS), Melbourne Ambulance Stroke Screen (MASS), Medic Prehospital Assessment for Code Stroke (Med PACS), Ontario Prehospital Stroke Screening Tool (OPSS), and Face Arm Speech Test (FAST) are considered positive if any of the physical findings are present after all eligibility criteria (if applicable) are met. Recognition of Stroke in the Emergency Room (ROSIER) scale assigns either a positive or a negative point value to each factor; scale is positive if the sum is ≥1. EMS = emergency medical services.

All included studies used similar methodologies of a retrospective review of a prospectively collected database of EMS-measured stroke scales, which were eventually linked to inpatient discharge diagnosis of stroke or TIA. Table 1 describes the included studies. Sample sizes from the studies were highly variable, ranging from 100¹³ to 11,296¹² subjects. Sample size and prevalence are reported in table 1 with notable variation in both characteristics among the studies. Sex, race, and age were not uniformly reported.

Table 1.

Characteristics of included studies

Studies were conducted in a variety of urban environments and were heterogeneous with respect to patient populations. Patients' ethnicity also varied across study settings, with Melbourne having a larger percentage of Malaysians (5%)²⁶ and Houston with 44% Hispanic/Latino population.²⁷ Similarly, Los Angeles also has a large Hispanic/Latino element (48%),²⁸ while Charlotte has only 12% Hispanic/Latinos.²⁹ The population of the province of Ontario is comprised primarily of persons extracted from the British Isles.³⁰ Beijing has a homogenous population, with 95% comprising Han nationality.³¹ The city of London has a largely white population (60%) with a significant black (13%) and Asian (19%) population.³²

Quality assessment.

Two authors (E.S.B., M.S.) evaluated all studies using the QUADAS-2 tool. Interrater agreement for QUADAS-2 scoring between the authors was almost perfect, kappa 0.89 (95% confidence interval [CI] 0.81–1.0).³³

In all the studies, many patients were excluded post hoc due to incomplete data collection of prehospital stroke scales.^17–24 The reasons for incomplete documentation were unclear in these studies. These excluded patients raise concern over selection bias (table e-2). No significant applicability concerns were noted in the QUADAS-2 assessment.

Performance assessment of prehospital stroke scales.

The scales used in each study are listed in table 2 together with their operating characteristics. The forest plots and ROC plane for sensitivity and specificity are presented for all studies in figure 3. The SSROC and ROC ellipse plots comparing CPSS and LAPSS are shown in figure 4. We could plot SSROC only for CPSS and LAPSS (figure 4A).⁷ Due to considerable heterogeneity (CPSS: I² = 97.8%, τ² = 4.33, LAPSS: I² = 96.8%, τ² = 4.16), we used the DerSimonian and Laird methodology to generate the SSROC. Area under the curve for CPSS was 0.813 ± SE 0.129 and for LAPSS 0.964 ± SE 0.028. Because of high heterogeneity (I² > 50%), we did not report pooled sensitivity and specificity for the various scales under review.

Table 2.

Operating characteristics of prehospital stroke scales

Figure 3. Graphical comparison of 7 different prehospital stroke scales.

(A) Forest plots for all prehospital stroke scales in the included studies. (B) Receiver operating characteristic curve (ROC) plane. Size of the circles indicates relative sample size. CI = confidence interval; CPSS = Cincinnati Prehospital Stroke Scale; FAST = Face Arm Speech Test; LAPSS = Los Angeles Prehospital Stroke Screen; MASS = Melbourne Ambulance Stroke Screen; Med PACS = Medic Prehospital Assessment for Code Stroke; OPSS = Ontario Prehospital Stroke Screening Tool; ROSIER = Recognition of Stroke in the Emergency Room.

Figure 4. Graphical comparison of CPSS and LAPSS.

(A) Symmetric summary receiver operating characteristic (SSROC) curve comparing area under the curve (AUC) for Cincinnati Prehospital Stroke Scale (CPSS) and Los Angeles Prehospital Stroke Screen (LAPSS) performance. Computational method: DerSimonian and Laird model. Circles in the plot are proportional to the weight/sample size. (B) Receiver operating characteristic (ROC) curve ellipse plot. Each point estimate is surrounded by 2D 95% confidence intervals.

DISCUSSION

It would appear from the Kidwell et al.¹¹ and Wojner-Alexandrov et al.¹² studies that LAPSS had the most favorable operating characteristics. Overall, LAPSS with its low negative likelihood ratio appears to be a good screening test, but despite that, when applied to a large population, it still misses up to 22% of strokes.¹⁷ Potential reasons for better performance of LAPSS include the more stringent screening criteria and the lack of a potentially subjective speech assessment.

The ROC plane illustrates a graphical description and visual comparison of different prehospital stroke scales (figure 3B). If a scale has its point estimate close to the diagonal line of uncertainty, the chances of that particular scale picking up a stroke correctly are similar to a coin flip. FAST, ROSIER, Med PACS, and CPSS when studied by Studnek et al.¹⁵ appear to be very close to that line. In contrast, the point estimates for LAPSS, OPSS, MASS, and CPSS when studied by Bray et al.¹⁴ are concentrated on the upper left corner of the graph, indicating better performance. Furthermore, as seen in the ellipse plot (figure 4B), CPSS when studied by Studnek et al.¹⁵ overlaps the line of uncertainty. The ellipses for CPSS do not overlap one another and are spread out on the graph, making us question the reproducibility of CPSS performance. However, the point estimates of LAPSS performance cluster in the upper left hand corner of the graph with confluent ellipses indicating that LAPSS has more consistent performance and perhaps is a more reliable tool. Despite the high between-study heterogeneity, we tried to compare the studies and generate an SSROC using DerSimonian and Laird methodology noting wide CI for CPSS (figure 4A). Although the CIs for CPSS and LAPSS overlap, lower limit of CI for CPSS crosses the line of uncertainty, indicating the scale may not perform better than a coin flip.

Though not included in the present study, an article by Ramanujam et al.²⁵ reported a lower sensitivity (44%) and a low positive predictive value of 40% for CPSS.²⁵ FAST, which has very similar elements to CPSS,²³ screened well, but demonstrated very poor specificity.¹⁸ MASS, a combination of LAPSS and CPSS, offers no significant benefit over LAPSS alone. When studied by Bray et al.,¹³ MASS and LAPSS were compared in the same population of patients with statistically indistinguishable operating characteristics. Med PACS, similarly combining elements of LAPSS, CPSS, and adding gaze and motor leg components, counterintuitively added little to specificity while sacrificing sensitivity. Likewise, even after excluding seizures and syncope cases, which are potential confounders in the diagnosis of stroke,³⁴ the ROSIER scale also has poor specificity. Surprisingly, Med PACS and ROSIER have very different sensitivities despite having similar scale elements.

Chenkin et al.¹⁶ reported lower specificity than either Kidwell et al.¹¹ or Wojner-Alexandrov et al.¹² despite the fact that OPSS excludes on-scene seizure patients. However, Chenkin et al.¹⁶ reported rtPA administration rates among OPSS-positive patients and demonstrated an increase in rtPA administration rate from 5.9% to 10.1% after the implementation of OPSS and, perhaps most importantly, none of the patients excluded by OPSS were later found to be eligible for rtPA. Additional study is required to determine whether this finding is reproducible and whether other scales perform similarly in this regard.

Limitations.

We were limited in our attempt because of the flawed methodologies in all of the studies included in this review. Unresponsive patients were excluded in at least 2 of the studies, threatening the applicability of stroke scales to these patients. Furthermore, all included studies were conducted at urban university centers in different cities and thus may not be generalizable to other environments.

While studying varied patient populations is desirable, sources for unwanted heterogeneity include (1) differences in stroke prevalence and (2) divergent background EMS education standards. In addition, both high stroke prevalence and wide variations in stroke prevalence (2.5%–88%) could introduce selection bias. In general, studies with small sample sizes had higher stroke prevalence, suggesting a selection bias in these studies that would inappropriately inflate diagnostic accuracy. There was also a lack of a prestudy sample size estimate by any of these studies except for Studnek et al.¹⁵ The large degree of heterogeneity between the reviewed studies prevented us from reporting pooled operating characteristics.

Since all studies included TIA as a stroke diagnosis, physical examination findings present in the prehospital environment may have disappeared by the time the patient was examined by the physician making the discharge diagnosis of stroke. As such, stroke scales performed by prehospital providers may influence the ultimate diagnosis of a TIA in the hospital. Prehospital stroke scales thus have the potential to introduce bias because the reference standard (discharge diagnosis) is not independent of the index test (stroke scale) (table e-2). This bias is clearly unavoidable. However, the prehospital tests were conducted without knowledge of the ultimate discharge diagnosis. These issues were inherent in all of the studies under review and similarly bias all results.

Verification bias is inherent in many of the studies under discussion. Falsely increasing the sensitivity is the fact that the primary inclusion criterion in many studies was suspected stroke. These patients are more likely to have the stroke scale performed and to test positive. True negatives may be inappropriately excluded, thereby falsely decreasing specificity.

Furthermore, the primary reason for prehospital identification of stroke is to speed access to rtPA. Given that all the included studies used discharge diagnosis of stroke as the gold standard and not the appropriate identification of patients for rtPA as the important diagnosis, all the studies may inappropriately overestimate the performance of the various scales for this important screening function.

Due to the availability of numerous prehospital stroke scales, it is important to compare them systematically so that EMS medical directors and vascular neurologists involved in prehospital stroke care can choose the scale that performs optimally for their individual systems.

There are several important methodologic issues in the current application of prehospital stroke scales. The high degree of heterogeneity between the studies suggests variability in methodology and nonrandom sampling. As a result, there is a need for more reliable assessments of prehospital scales for the diagnosis of stroke. More study is required to identify the best currently available methodology for prehospital identification of stroke and to find new tools that are easy to perform and may capture stroke more accurately in the field. Nonetheless, LAPSS appears to have the best operating characteristics when assessed both by likelihood ratios and ROC curve.

GLOSSARY

CI

confidence interval

CPSS

Cincinnati Prehospital Stroke Scale

EMS

emergency medical services

EMT

emergency medical technician

FAST

Face Arm Speech Test

LAPSS

Los Angeles Prehospital Stroke Screen

MASS

Melbourne Ambulance Stroke Screen

Med PACS

Medic Prehospital Assessment for Stroke Code

OPSS

Ontario Prehospital Stroke Screen Tool

ROSIER

Recognition of Stroke in the Emergency Room

QUADAS-2

Quality Assessment of Diagnostic Accuracy Studies–2

ROC

receiver operating characteristic

rtPA

recombinant tissue plasminogen activator

SSROC

symmetric summary receiver operating characteristic

AUTHOR CONTRIBUTIONS

E.S.B. conceived the study, designed the study, supervised data collection, performed statistical analyses, drafted the manuscript, and takes responsibility for the study as a whole. M.S. extracted the data, drafted and revised the manuscript, and performed statistical analyses. R.H.S. assisted with study design and conception, assisted with statistical analyses, and revised the manuscript. S.R.L. obtained research funding and revised the manuscript for important scientific content.

STUDY FUNDING

Funded in parts by NIH grants 1U01NS044364, R01 HL096944, 1U10NS077378, and 1U10NS080377.

DISCLOSURE

E. Brandler, M. Sharma, and R. Sinert report no disclosures relevant to the manuscript. S. Levine serves on the Scientific Advisory Boards of Independent Medical/Safety Monitor for National Institute of Neurological Disorders and Stroke–funded IMS 3, FAST MAG, INSTINCT, and CLEAR-ER and Adjudication Committee for National Institute of Neurological Disorders and Stroke–funded WARCEF. He received travel funding or speaker honoraria from Genentech in 2011. He also serves as the Associate Editor of MEDLINK and is the scientific content advisor for the National Stroke Association. He is a consultant for Genentech study on cost-effectiveness of primary stroke centers and receives research support from Genentech, Inc. He was on the Speakers' Bureaus for Medical Education Speakers Network, lecturer, 2008–2012. He receives research support from NIH-NHLBI 1R01 HL096944, principal investigator, 2009–2013; NIH–National Institute of Neurological Disorders and Stroke 1UO1 NS044364, Independent safety monitor, 2003–2012; NIH–National Institute of Neurological Disorders and Stroke 1 U10 NS077378, PI, 2011–2017; NIH–National Institute of Neurological Disorders and Stroke 1 U10 NS080377, PI, 2012–2017; PCORI, Scientific PI, 2012–2014; NIH–National Institute of Neurological Disorders and Stroke 1 R25 NS079211, MPI, 2012–2017; The Patient-Centered Outcomes Research Institute (PCORI) 1IP2PI000781, scientific PI, 2012–2014; NIH-NIA 1 R01 AG040039, Co-I, 2011–2016; NIH–National Institute of Neurological Disorders and Stroke 2 P50 NS044283, safety monitor, 2008–2013; and NIH–National Institute of Neurological Disorders and Stroke 2 U01 NS052220, independent medical monitor, 2005–2013. Go to Neurology.org for full disclosures.