Skip to main content
PMC home page
PLOS ONE logoLink to PLOS ONE
. 2020 Apr 15;15(4):e0231113. doi: 10.1371/journal.pone.0231113

Stepwise stroke recognition through clinical information, vital signs, and initial labs (CIVIL): Electronic health record-based observational cohort study

Sung Eun Lee 1,2,#, Mun Hee Choi 1,#, Hyo Jung Kang 2, Seong-Joon Lee 1, Jin Soo Lee 1, Yunhwan Lee 3, Ji Man Hong 1,*
Editor: Juan Manuel Marquez-Romero4
PMCID: PMC7159200  PMID: 32294085

Abstract

Background

Stroke recognition systems have been developed to reduce time delays, however, a comprehensive triaging score identifying stroke subtypes is needed to guide appropriate management. We aimed to develop a prehospital scoring system for rapid stroke recognition and identify stroke subtype simultaneously.

Methods and findings

In prospective database of regional emergency and stroke center, Clinical Information, Vital signs, and Initial Labs (CIVIL) of 1,599 patients suspected of acute stroke was analyzed from an automatically-stored electronic health record. Final confirmation was performed with neuroimaging. Using multiple regression analyses, we determined independent predictors of tier 1 (true-stroke or not), tier 2 (hemorrhagic stroke or not), and tier 3 (emergent large vessel occlusion [ELVO] or not). The diagnostic performance of the stepwise CIVIL scoring system was investigated using internal validation. A new scoring system characterized by a stepwise clinical assessment has been developed in three tiers. Tier 1: Seven CIVIL-AS3A2P items (total score from –7 to +6) were deduced for true stroke as Age (≥ 60 years); Stroke risks without Seizure or psychiatric disease, extreme Sugar; “any Asymmetry”, “not Ambulating”; abnormal blood Pressure at a cut-off point ≥ 1 with diagnostic sensitivity of 82.1%, specificity of 56.4%. Tier 2: Four items for hemorrhagic stroke were identified as the CIVIL-MAPS indicating Mental change, Age below 60 years, high blood Pressure, no Stroke risks with cut-point ≥ 2 (sensitivity 47.5%, specificity 85.4%). Tier 3: For ELVO diagnosis: we applied with CIVIL-GFAST items (Gaze, Face, Arm, Speech) with cut-point ≥ 3 (sensitivity 66.5%, specificity 79.8%). The main limitation of this study is its retrospective nature and require a prospective validation of the CIVIL scoring system.

Conclusions

The CIVIL score is a comprehensive and versatile system that recognizes strokes and identifies the stroke subtype simultaneously.

Introduction

Stroke remains with a high burden in societies, and improving the recognition of a stroke can help reduce this burden [1]. “Time-is-brain” is a crucial concept in ischemic stroke management [2,3], the importance of early recanalization has been further addressed after the success of endovascular recanalization therapies [4]. In the case of hemorrhagic stroke, early surgical interventions can be beneficial in selected patients [5,6]. Candidates for urgent interventions should be transported to an appropriately-equipped hospital, however, treatment can be delayed due to various reasons [7].

Stroke recognition systems have been developed to reduce time delay in community and hospital settings [8,9]. Various scales and scoring systems have been developed, but there is still no consensus on which scale perform better. [10,11]. Previous systems have issues related to false positives and false negatives, making it difficult for stroke specialists to handle a considerable number of patients [12,13]. In addition, most research has covered only one aspect of stroke, ‘true stroke or not’ or ‘selection of emergent large vessel occlusion (ELVO),’ and clinical parameters can be subjective even after thorough training. Finally, previously published systems have a greater focus on reducing pre-hospital delay [14,15]. Therefore, a comprehensive triaging system for considering next-step treatments should address to reduce the workload at stroke centers with acceptable sensitivity and specificity.

In this context, we have developed a new scoring system using Clinical Information, baseline Vital sign, and Initial Labs (CIVIL). Here, we aimed to evaluate the feasibility of the CIVIL system and compare with previous screening systems in suspicious acute stroke patients.

Methods

Study population

This electronic health record-based observational cohort study was performed in a tertiary referral hospital from January 2012 to December 2015. Care in the tertiary stroke center fulfilled the Brain Attack Coalition’s standardized criteria, and the stroke unit had also obtained certification from the Korean Stroke Association [16]. The regional emergency medical center serves the southern part of Gyeonggi Province of South Korea with a population of approximately four million, and it’s emergency room (ER) has approximately 89,000 patients annually [17]. Previously, we developed the stroke recognition system ‘Cubic S model’ (S1 Fig), which is based on the Electronic Medical Record (EMR) system for suspected stroke patients in the ER [18]. It is based on common signs and symptoms and contains three domains: time, body-part involvement, and symptomatic presentations. When suspicious stroke patients visit the ER, ER physicians check the presence of three domains: sudden onset, one-sided involvement of face/arm/leg, and 6 representative symptoms of stroke.

Each dataset was automatically stored in the database from a prospectively registered critical pathway system for rapid thrombolysis in suspicious stroke patient. The data has been used to improve the quality of registered data through monthly reports. Inclusion criteria for this study were (a) acute neurologic manifestations within 6 hours, (b) acute thrombolysis code activating cases who meets all three domains of EMR-based Cubic S system, and (c) final confirmation with clinical and imaging findings by stroke neurologists. Exclusion criteria were: (a) incorrect activation of an acute thrombolysis code, (b) onset-to-door time > 6 hours, and (c) uncertain final diagnosis due to incomplete study. The study protocol was approved by the Institutional Review Board of Ajou University Hospital (AJIRB-MED-MDB-16-407). Informed consent was waived because of the study’s retrospective nature.

Processing after critical pathway activation

Initial assessment and activation of acute thrombolysis code was performed by ER physicians. Education of ER physicians and nurses was routinely performed every 6 to 12 months by stroke neurologists. Immediately after the activation of acute thrombolysis code, stroke neurologists assessed the patients. All patients except those with contraindications to contrast use underwent computed tomography (CT) scan with angiography. Simultaneously, neurologists meticulously investigated clinical information, baseline vital signs, initial laboratory findings, and stroke images. In cases where recanalization treatments were needed, intravenous recombinant tissue plasminogen activator and/or endovascular therapies were implemented according to critical pathway in our institute. Clinical information was recorded including age, sex, prior medical histories (hypertension, diabetes mellitus, previous stroke occurrence, seizure or syncope, and psychiatric history) from the patient, care-givers, or paramedics. Initial neurological manifestations were assessed with the Cubic S model. Baseline vital signs were comprised of blood pressure, pulse rate, and body temperature, and initial laboratory findings (glucose level and oxygen saturation) were also included.

Confirmation of final diagnosis

The adjudication meeting comprised of stroke specialists was held weekly for final diagnoses of all patients according to the three tiers; as stroke mimic vs. true stroke, ischemic vs. hemorrhagic stroke, non-ELVO vs. ELVO. Final diagnosis was determined after review of ER chart, imaging and laboratory studies for differential diagnoses. True stroke was diagnosed when the neurologic exam was compatible with supportive imaging evidence of CT and/or magnetic resonance imaging including diffusion weighted image. Transient cerebral ischemic attack was classified into the true stroke and ischemic stroke group. Stroke mimics were designated when the clinical details were compatible with non-vascular etiologies. Initial CT angiography confirmed hemorrhagic stroke and large artery occlusion. ELVO was designated as occlusion of the internal carotid artery, M1 or M2 segment of the middle cerebral artery, or basilar artery [19,20].

Development of new scoring system

We intended to develop a new scoring system that is characterized by a “stepwise clinical assessment system which enables rapid discrimination of patients suspected of acute stroke,” potential variables were assessed including Clinical Information, Vital signs, and Initial Labs (CIVIL) used in ER and prehospital settings. There were 23 clinical findings, four vital signs, and two laboratory findings. We analyzed these variables by the three tiers; stroke mimic vs. true stroke, ischemic vs. hemorrhagic stroke, non-ELVO vs. ELVO. At the first tier (stroke mimic vs. true stroke), we assigned +1 to positive variables for true stroke (odds ratio [OR] > 1.0) and -1 to negative variables (OR <1.0). In the second tier, items suggestive of hemorrhagic stroke were derived (only positive scores). Finally, to discriminate ELVO from non-ELVO, we applied the gaze to face-arm-speech-time (GFAST) scoring system [21]. For evaluation of performance, the CIVIL system was compared with three previous recognition systems for acute stroke: Cincinnati Prehospital Stroke Scale (CPSS) [14], Los Angeles Prehospital Stroke Screen (LAPSS) [15], and Recognition Of Stroke In the Emergency Room (ROSIER) system [8].

Statistical analysis

Differences between the two groups at each steps were analyzed using χ2 or Student t-test for categorical and continuous variables, respectively. Significant variables from univariate analyses (p<0.05) were assessed with multivariate logistic regression models for deduction of scoring items (enter method). Associations were presented as odds ratios (OR) with corresponding 95% confidence interval (CI). Internal validation using receiver operating characteristic (ROC) curve analysis was performed to determine the optimal cut off point for each steps. Diagnostic performance including sensitivity, specificity, positive predictive value, negative predictive value, and Youden index were assessed for each cut-off point. We performed all analyses using SPSS 25.0 for Windows (SPSS Inc., Chicago, Ill).

Results

Patient assessment

The flow chart of study population is shown in S2 Fig. A total of 1,621 patients were screened by acute thrombolysis code activation. Sixty-two patients were excluded, and the remaining 1,559 suspected stroke patients were enrolled, of these, true stroke was confirmed in 1,153 (74.0%). Causes of stroke mimicking symptoms were metabolic disease (18.0%), drug intoxication (15.0%), peripheral neuropathy (14.3%), psychogenic disorder (14.3%), seizure (13.8%), infectious disease (7.4%), syncope (6.9%), and tumorous condition (3.4%). True stroke patients comprised of ischemic stroke (n = 894, 77.5%) and hemorrhagic stroke (n = 259, 22.5%), and the number of ischemic stroke patient requiring recanalization therapy was 291 (32.6%).

Clinical information, baseline vital signs, and initial labs (CIVIL)

Tables 1 and 2 summarizes detailed findings according to the final diagnosis. In the first tier (stroke mimic vs. true stroke), true stroke patients were older and male-dominant. History of stroke risk factor were more frequent in the true stroke group, whereas history of seizure or psychiatric disease were less common. From clinical manifestations, “after awakening”, lateralizing symptoms, “not ambulating”, and “not able to grasp” were more prevalent in the true stroke group, while “mental change” was more frequent in the stroke mimic group. From vital-sign and initial laboratory findings, systolic and diastolic BP were higher in the true stroke group. In contrast, the stroke mimic group included more patients with low systolic BP (≤90mmHg) and extreme glucose level (initial glucose <80 or ≥400 mg/dl).

Table 1. Clinical information, vital signs, and initial labs (CIVIL).

Stroke- mimic (n = 406) True stroke (n = 1,153) pa Ischemic stroke (n = 894) Hemorrhagic stroke (n = 259) pb
Clinical information
        Age, years 62.2 ± 15.8 64.5 ± 14.2 0.012 65.6 ± 14.0 60.5 ± 14.5 <0.001
        Age ≥ 60 years, n (%) 230 (56.7) 743 (64.4) 0.005 613 (68.6) 130 (50.2) <0.001
        Age ≤ 40 years, n (%) 38 (9.4) 57 (4.9) 0.001 40 (4.5) 17 (6.6) 0.172
        Male, n (%) 200 (49.3) 699 (60.6) <0.001 561 (62.8) 138 (53.3) 0.004
        Onset-to-door time (minute) 172.6 ± 209.9 182.9 ± 205.2 0.118 197.9 ± 210.0 131.2 ± 178.7 <0.001
        Onset-to-door ≥ 90 min, n (%) 173 (42.6) 466 (40.4) 0.439 314 (35.1) 152 (58.7) <0.001
    Prior history, n (%)
        Hypertension 170 (41.9) 606 (52.6) <0.001 473 (52.9) 133 (51.4) 0.355
        Diabetes 100 (24.6) 250 (21.7) 0.419 214 (23.9) 36 (13.9) <0.001
        Cardiac diseases 69 (17.0) 263 (22.8) <0.001 244 (27.3) 19 (7.3) <0.001
        Previous stroke 78 (19.2) 242 (21.0) 0.120 199 (22.3) 43 (16.6) 0.028
        Seizure or psychiatric history 89 (21.9) 21 (1.8) <0.001 19 (2.1) 2 (0.8) 0.192
    Clinical manifestations
        Time
         “Sudden”, n (%) 365 (89.9) 1038 (90.0) 0.172 786 (87.9) 252 (97.3) <0.001
         “After awakening”, n (%) 32 (7.9) 114 (9.9) 0.021 105 (11.7) 9 (3.5) <0.001
        “As unusual”, n (%) 13 (3.2) 34 (2.9) 0.476 31 (3.5) 3 (1.2) 0.034
    Body-spatial
         “one-side arm”, n (%) 144 (35.5) 821 (71.2) <0.001 642 (71.8) 179 (69.1) 0.221
         “one-side leg”, n (%) 120 (29.6) 720 (62.4) <0.001 557 (62.3) 163 (62.9) 0.457
        “one-side face”, n (%) 72 (17.7) 401 (34.8) <0.001 306 (34.2) 95 (36.7) 0.255
        “any asymmetry’, n (%) 205 (50.5) 958 (83.1) <0.001 763 (85.3) 195 (75.3) <0.001
    Symptoms
        “not ambulating”, n (%) 79 (19.5) 508 (44.1) <0.001 388 (43.4) 120 (46.3) 0.222
        “not able to speak”, n (%) 185 (45.6) 576 (50.0) 0.077 447 (50.0) 129 (49.8) 0.506
        “not able to grasp”, n (%) 37 (9.1) 206 (17.9) <0.001 164 (18.3) 42 (16.2) 0.245
        “mental change” *, n (%) 156 (38.4) 193 (16.7) <0.001 92 (10.3) 101 (39.0) <0.001
        “abnormal sensation”, n (%) 56 (13.8) 162 (14.1) 0.247 138 (15.4) 24 (9.3) 0.006
        “visual disturbance”, n (%) 4 (1.0) 18 (1.6) 0.165 17 (1.9) 1 (0.4) 0.062
Baseline vital signs
    SBP, mmHg 137.7 ± 28.9 151.1 ± 28.6 <0.001 147.7 ± 25.4 163.0 ± 34.9 <0.001
    SBP ≥ 160mmHg, n (%) 100 (24.6) 473 (41.0) <0.001 327 (36.6) 146 (56.4) <0.001
    SBP ≥ 140mmHg, n (%) 202 (49.8) 801 (69.5) <0.001 598 (66.9) 203 (78.4) <0.001
    SBP ≤ 90mmHg, n (%) 12 (3.0) 3 (0.3) <0.001 1 (0.1) 2 (0.8) 0.128
    DBP, mmHg 81.9 ± 34.7 86.8 ± 16.4 0.018 85.6 ± 15.5 90.6 ± 18.7 0.005
    Pulse rate, bpm 84.2 ± 19.1 82.9 ± 15.5 0.454 82.8 ± 15.9 83.2 ± 13.9 1.000
    Body temperature, °C 36.5 ± 0.8 36.5 ± 0.5 1.000 36.5 ± 0.4 36.4 ± 0.6 0.173
Initial laboratory findings
    Glucose, mg/dl 150.0 ± 101.6 146.8 ± 60.1 0.510 144.0 ± 60.5 156.4 ± 57.8 0.051
    Extreme glucose level†, n (%) 24 (5.9) 14 (1.2) <0.001 11 (1.2) 3 (1.2) 1.000
    Oxygen saturation, % 99.3 ± 3.2 99.5 ± 2.5 0.062 99.6 ± 2.2 99.2 ± 3.2 0.082
Open in a new tab

SBP means systolic blood pressure, and DBP indicates diastolic blood pressure. *”mental change” was defined when a decrease in the level of consciousness below drowsiness was observed on initial neurological examination. †initial blood glucose level ≤ 80 or ≥ 400 mg/dl, pa = Stroke mimic vs. true stroke, pb = Ischemic stroke vs. hemorrhagic stroke

Table 2. Clinical information, vital signs, and initial labs (CIVIL) between ELVO and non-ELVO patients.

ELVO stroke (n = 291) Non-ELVO stroke (n = 603) p
Clinical information
        Age, years 68.1 ± 13.6 64.4 ± 14.0 <0.001
        Male, n (%) 171 (58.8) 390 (64.7) 0.087
        Onset-to-door time (minute) 191.8 ± 208.8 200.9 ± 210.7 0.045
    Prior history, n (%)
        Hypertension 160 (55.0) 313 (51.9) 0.388
        Diabetes 59 (20.3) 155 (25.7) 0.075
        Cardiac problems 115 (39.5) 129 (21.4) <0.001
        Previous stroke 57 (19.6) 142 (23.5) 0.182
    Manifestation
    Time domain
        “Sudden”, n (%) 260 (89.3) 526 (87.2) 0.363
         “After awakening”, n (%) 30 (10.3) 75 (12.4) 0.354
         “As unusual”, n (%) 10 (3.4) 21 (3.5) 0.972
    Body-spatial domain
         “one-side arm”, n (%) 241 (82.8) 401 (66.5) <0.001
         “one-side leg”, n (%) 217 (74.6) 340 (56.4) <0.001
         “one-side face”, n (%) 112 (38.5) 194 (32.2) 0.062
         “any asymmetry’, n (%) 257 (88.3) 506 (83.9) 0.081
    Symptom domain
        “not ambulating”, n (%) 159 (54.6) 229 (38.0) <0.001
        “not able to speak”, n (%) 188 (64.6) 259 (43.0) <0.001
        “not able to grasp”, n (%) 53 (18.2) 111 (18.4) 0.944
        “mental change” *, n (%) 59 (20.3) 33 (5.5) <0.001
        “abnormal sensation”, n (%) 14 (4.8) 124 (20.6) <0.001
        “visual disturbance”, n (%) 9 (3.1) 8 (1.3) 0.070
        “gaze deviation”, n (%) 197 (67.7) 48 (8.0) <0.001
Baseline vital signs
    SBP, mmHg 143.1 ± 26.4 149.9 ± 24.7 <0.001
    DBP, mmHg 83.5 ± 16.0 86.7 ± 15.1 0.003
    Pulse rate, bpm 84.2 ± 18.1 82.1 ± 14.7 0.064
    Body temperature, °C 36.4 ± 0.4 36.5 ± 0.4 0.004
Initial laboratory findings
    Glucose, mg/dl 142.4 ± 49.3 144.8 ± 65.2 0.571
    Oxygen saturation, % 99.5 ± 1.7 99.7 ± 2.4 0.202
Stroke characteristics
    NIHSS, median (IQR) 16 (12–20) 3 (1–6) <0.001
    TOAST classification, n (%) <0.001
        Large artery disease 69 (23.7) 105 (17.4)
        Cardioembolism 157 (54.0) 106 (17.6)
        Small artery disease 0 (0.0) 134 (22.2)
        Others 65 (22.3) 258 (42.8)
    Vessel occlusion, n (%)
        ICA 90 (30.9) -
        M1 102 (35.1) -
        M2 7 (16.2) -
        BA 33 (11.3) -
        Others 19 (6.5) -
    tPA use, n (%) 155 (53.3) 83 (13.8) <0.001
    Endovascular treatment, n (%) 150 (51.5) 0 (0.0) <0.001
Open in a new tab

*”mental change” was defined when a decrease in the level of consciousness below drowsiness was observed on initial neurological examination. SBP = systolic blood pressure, DBP = diastolic blood pressure, NIHSS = National Institute of Health Stroke Scale, ICA = internal carotid artery, BA = basilar artery, tPA = tissue plasminogen activator

In the second tier (ischemic vs. hemorrhagic stroke), the ischemic stroke group was older and had a higher proportion of males than the hemorrhagic stroke group. The patients with hemorrhagic stroke had shorter onset-to-door time. History of stroke risk factor was more prevalent in the ischemic stroke group. “Sudden onset” was more frequent in the hemorrhagic stroke, while “after awakening” and “as unusual” were more common in the ischemic stroke. The ischemic stroke group showed more common “any asymmetry. “Mental change” was more prevalent in the hemorrhagic stroke group, while “abnormal sensation” was more frequent in the ischemic stroke group. Systolic and diastolic BP were higher in the hemorrhagic stroke group.

Tier 1: Stroke mimic vs. true stroke: CIVIL-AS3A2P

We determined independently significant factors in the CIVIL system using multiple regression analysis (Fig 1A). Age (≥60 years), Stroke risks (history of cardiac disease), “any Asymmetry”, and “not Ambulating” were positive discriminatory items for diagnosis of true stroke, while younger Age (≤40 years), history of Seizure or psychiatric disease were negative discriminatory items. In vital signs and laboratory data, high BP (systolic BP ≥140mmHg) was a positive discriminatory item, and low BP (systolic BP ≤90mmHg) and extreme Sugar level (≤80 or ≥400 mg/dL) were included as negative discriminatory items. Asymmetric leg weakness, non-lateralizing symptoms, mental change, and initial oxygen saturation were indiscriminate. The CIVIL-AS3A2P score was finally determined as overall 7 items (Fig 2): 5 clinical items, 1 vital sign, and 1 initial laboratory finding. The total score ranged from -5 to +6. Retrospective validation on 1,559 suspected stroke patients determined an optimal cut-off point for stroke diagnosis as ≥ +1. At this cut-off point, the diagnostic performance of CIVIL-AS3A2P score was as follows: sensitivity 82.1%, specificity 56.4%, positive predictive value (PPV) 84.3%, and negative predictive value (NPV) 52.6% (Youden’s index 0.385). We compared the performance of the CIVIL-ASAP score to the CPSS, LAPSS, and ROSIER scales in our data set. The sensitivity and specificity of these established recognition systems were 90.4% and 29.1% in CPSS, 69.7% and 67.7% in LAPSS, 93.8% and 34.0% in ROSIER. In ROC curve analysis, CIVIL-AS3A2P score had a superior diagnostic performance than the other three systems per area under the curve (S3 Fig, 0.767 in CIVIL-ASAP vs. 0.751 in ROSIER vs. 0.687 in LAPSS vs. 0.597 in CPSS). Comparisons among early stroke recognition scales were described in Fig 3.

Fig 1. Results of multiple regression analysis and distribution of patients according to the CIVIL scores.

Fig 1

Open in a new tab

(A) Tier 1: CIVIL-AS3A2P score consisted of Age (≤40 years or ≥60 years), Stroke risk (cardiac disease history) without Seizure or psychiatric history, extreme Sugar level (≤80 or ≥400mg/dl), any Asymmetry, not Ambulating, and Pressure (SBP≤90 mmHg or ≥140mmHg). (B) Tier 2: CIVIL-MAPS included Mental change, Age (≤60 years), Pressure (SBP≥160mmHg), and no Stroke risks (history of diabetes mellitus or cardiac disease). (C) Tier 3: To identify emergent large vessel occlusion patients, GFAST score was incorporated into the CIVIL scoring system (Gaze, Face asymmetry, Arm asymmetry, Speech disturbance).

Fig 2. Summary of items and scoring in the CIVIL system.

Fig 2

Open in a new tab

In tier 1 (CIVIL-AS3A2P), 7 items which included clinical information (white), vital signs (grey), and initial labs (dark grey) were used. Stroke-preferred items were assigned positive points and stroke mimic preferred items were negative points (ranged from ―5 to +6). Tier 2 (CIVIL-MAPS) allocated 4 items with clinical information and vital signs, and the GFAST system was applied in tier 3 (CIVIL-GFAST) for the selection of ELVO patients.

Fig 3. Descriptive comparison of various early stroke recognition scales.

Fig 3

Open in a new tab

CIVIL = Clinical Information, Vital signs, and Initial Labs, CPSS = Cincinnati Prehospital Stroke Scale, LAPSS = Los Angeles Prehospital Stroke Screen, ROSIER = Recognition Of Stroke In the Emergency Room.

Tier 2: Ischemic vs. hemorrhagic stroke: CIVIL-MAPS

To differentiate between ischemic and hemorrhagic stroke, second tier analysis using multiple regression was conducted (Fig 1B). MAPS: Mental change, Age below 60 years, high blood Pressure (systolic BP ≥160mmHg), no Stroke risk (without history of diabetes or cardiac disease) were positive discriminatory items for diagnosis of hemorrhagic stroke (Fig 2). The CIVIL-MAPS score consisted of 3 clinical items and 1 vital sign, and the total score ranged from 0 to +4. Retrospective validation on the 1,153 true stroke patients determined an optimal cut-off point for hemorrhagic stroke diagnosis as ≥ 2. At this cut-off point, the diagnostic performance of CIVIL-MAPS score was as follows: sensitivity of 47.5%, specificity of 85.4%, PPV of 50.6%, NPV of 83.8% (Youden’s index 0.329).

Tier 3: Non-ELVO vs. ELVO: CIVIL-GFAST

In the final tier, we applied the GFAST score to select ELVO patients (Fig 1C). The score was calculated as the sum of positive symptoms: Gaze deviation, Face asymmetry, Arm asymmetry, and Speech disturbance (Fig 2). Retrospective validation on 894 ischemic stroke patients determined the optimal cut-off point for ELVO diagnosis as ≥ 3. At this cut-off point, the diagnostic performance of CIVIL-GFAST score was as follows: sensitivity of 66.5%, specificity of 79.8%, PPV of 54.6%, NPV of 86.7% (Youden’s index 0.463). The CIVIL scoring system is summarized in Fig 2.

Discussion

Our data support that the CIVIL scoring system is feasible for identifying suspicious acute stroke patients in a stepwise fashion: true stroke or not, hemorrhagic stroke or not, and ELVO or not. In addition, step-by-step acronyms (AS3A2P, MAPS, and GFAST) can be used in a wide range of fields of prehospital and ER-based situations to serve as triaging tools for patients to be easily remembered.

The CIVIL scoring system can help us to differentiate different types of stroke at the same time. Acute stroke is an urgent condition that requires rapid evaluation and proper management because the longer a stroke goes untreated, the greater the brain damage (time is brain) [22]. Efficient triaging is important for acute stroke patients to guide proper disposition and early interventions, which may be entirely decisive in some cases [23,24]. Due to limited time window for thrombolytic therapy, numerous prehospital scoring systems for early recognition of ischemic stroke have been developed [8,14,15]. Recently, endovascular recanalization therapy in ELVO patients has been proven as the standard treatment, consequently, several scoring systems to recognize ELVO have been addressed [19–21]. However, current scoring systems have focused only on one aspect of stroke identification: stroke versus stroke mimic, or ELVO discrimination [25]. In addition, little has been elucidated to distinguish hemorrhagic stroke from ischemic stroke especially in situations with limited imaging facilities [26]. To the best of our knowledge, there has been no definitive scoring system that integrates various aspects of stroke diagnosis.

This scoring system features a stepwise approach to triage stroke suspicious patients. When paramedics in emergency medical service (EMS) or ER physicians proceed step by step, they will be able to properly classify stroke patients who require rapid treatment. CIVIL-AS3A2P initially differentiates patients with true stroke from stroke mimics. In this first tier, it is important not to exclude potential patients who need. In this context, as we expected, CIVIL-AS3A2P showed relatively high sensitivity and low specificity for including all possible candidates. Second (CIVIL-MAPS) and third tiers (CIVIL-GFAST) showed low sensitivity and high specificity so that patients in need of urgent treatments (thrombolysis and/or endovascular therapy) could be selected effectively. The CIVIL scoring system enables rapid identification of patients delivered to the ER with high sensitivity to identify the actual stroke, and also enables the recognition of hemorrhagic stroke and ELVO with high specificity.

The CIVIL scoring system included objective vital signs and laboratory findings as well as clinical information. Previous scoring systems consisting of clinical manifestations may be affected by the examiner’s experience and special training is needed to reduce inter-observer variability [27]. Some validation studies on early recognition scoring systems reported high variability in inter-observer reliability ranging from 69% to 90% [14,28]. To compensate for these variations, ROSIER [8] and LAPSS [15] included laboratory finding such as blood glucose levels. For this reason, the CIVIL scoring system contained vital signs in addition to laboratory findings designed to apply more objective parameters. Moreover, the extreme values of vital signs and laboratory findings-blood glucose level ≤ 80 or ≥ 400mg/dl and systolic BP ≥140 mmHg or ≤ 90 mmHg help to discriminate stroke mimic conditions such as sepsis, shock, or syncope. Therefore, our new scoring system could overcome some potential limitations of other previous scoring systems by including objective and quantitative items.

In this study, the CIVIL scoring system was developed for use in both ER and prehospital settings. The selection of acute stroke patients in the prehospital and emergent setting continue to be the subject of research due to the time-dependent nature of stroke [27]. Various early recognition systems have been used in the EMS to properly transport stroke patients to more appropriate centers. However, there have been several limitations including inconvenience, imperfect accuracy, and time-consuming training [19,20]. Moreover, an increase in items adds complexity to the system for rapid evaluation [29]. The CIVIL scoring system applies an intuitive and easy-to-remember acronym for EMS and other medical professionals to be easily used. We applied simple and familiar GFAST to improve accessibility in the third tier for the identification of ELVO patients.

Our study has some limitations. First, it is an observational study with retrospective nature; however, all information has been automatically stored in a prospectively-collecting database at a large regional emergency and stroke center. Second, data with time windows over 6 hours were not covered in the current study. From recent trials, mechanical thrombectomy is indicated up to 24 hours after stroke onset. Nevertheless, most patients with onset to treatment time less than 6 hours need more urgent treatment regardless of core-penumbra mismatch, so that our recognition system can more properly apply to those patients. Third, there are limits in the conclusions that can be drawn regarding the performance of the CIVIL system in patients with posterior circulation acute ischemic stroke. This scoring system was designed to focus on patients with anterior circulation which is supported by the current guideline for endovascular treatment. Finally, the sensitivity of tier 2 and 3 are less than 80%, the results should be interpreted with caution. In the future, prospective validation of the CIVIL scoring system should include a systematic education program for paramedics to improve performance.

In conclusion, the CIVIL scoring system can be used as a comprehensive and versatile tool to recognize true stroke and identify stroke subtypes simultaneously.

Supporting information

S1 STROBE Checklist

(DOC)

Click here for additional data file. (83KB, doc)
S1 Data. Deidentified raw dataset.

(XLSX)

Click here for additional data file. (691.8KB, xlsx)
S1 Fig. The Korean version of EMR based matrix for stroke suspicious patients.

(Cubic S model)

(PDF)

Click here for additional data file. (105.7KB, pdf)
S2 Fig. Flow diagram of 1,621 suspicious stroke patients.

(PDF)

Click here for additional data file. (260.4KB, pdf)
S3 Fig. Receiver-Operating Characteristic (ROC) curve and corresponding area under the curve (AUC) statistics of the CIVIL scoring system.

(PDF)

Click here for additional data file. (189.5KB, pdf)

List of abbreviations

CIVIL

Clinical Information, baseline Vital sign, and Initial Labs

CPSS

Cincinnati Prehospital Stroke Scale

CT

computed tomography

ELVO

emergent large vessel occlusion

EMR

Electronic Medical Record

EMS

emergency medical service

ER

emergency room

GFAST

gaze to face-arm-speech-time

LAPSS

Los Angeles Prehospital Stroke Screen

OR

odds ratio

ROC

receiver operating characteristic

ROSIER

Recognition Of Stroke In the Emergency Room

Data Availability

All relevant data are within the paper and its Supporting Information files.

Funding Statement

This study was supported by the Korea Centers for Disease Control and Prevention (KCDC, M2020-A0258-00012) and the Korean Stroke Society (KSS).

References

  • 1.Feigin VL, Roth GA, Naghavi M, Parmar P, Krishnamurthi R, Chugh S, et al. Global burden of stroke and risk factors in 188 countries, during 1990–2013: a systematic analysis for the Global Burden of Disease Study 2013. Lancet Neurol. 2016; 15(9):913–24. 10.1016/S1474-4422(16)30073-4 [DOI] [PubMed] [Google Scholar]
  • 2.Saver JL. Time is brain—quantified. Stroke. 2006; 37(1):263–6. 10.1161/01.STR.0000196957.55928.ab [DOI] [PubMed] [Google Scholar]
  • 3.Lees KR, Bluhmki E, von Kummer R, Brott TG, Toni D, Grotta JC, et al. Time to treatment with intravenous alteplase and outcome in stroke: an updated pooled analysis of ECASS, ATLANTIS, NINDS, and EPITHET trials. Lancet. 2010; 375(9727):1695–703. 10.1016/S0140-6736(10)60491-6 [DOI] [PubMed] [Google Scholar]
  • 4.Goyal M, Menon BK, van Zwam WH, Dippel DW, Mitchell PJ, Demchuk AM, et al. Endovascular thrombectomy after large-vessel ischaemic stroke: a meta-analysis of individual patient data from five randomised trials. Lancet. 2016; 387(10029):1723–31. 10.1016/S0140-6736(16)00163-X [DOI] [PubMed] [Google Scholar]
  • 5.de Oliveira Manoel AL, Goffi A, Zampieri FG, Turkel-Parrella D, Duggal A, Marotta TR, et al. The critical care management of spontaneous intracranial hemorrhage: a contemporary review. Crit Care. 2016; 20:272 10.1186/s13054-016-1432-0 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 6.Steiner T, Al-Shahi Salman R, Beer R, Christensen H, Cordonnier C, Csiba L, et al. European Stroke Organisation (ESO) guidelines for the management of spontaneous intracerebral hemorrhage. Int J Stroke. 2014; 9(7):840–55. 10.1111/ijs.12309 [DOI] [PubMed] [Google Scholar]
  • 7.Jiang B, Ru X, Sun H, Liu H, Sun D, Liu Y, et al. Pre-hospital delay and its associated factors in first-ever stroke registered in communities from three cities in China. Sci Rep. 2016; 6:29795 10.1038/srep29795 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 8.Nor AM, Davis J, Sen B, Shipsey D, Louw SJ, Dyker AG, et al. The Recognition of Stroke in the Emergency Room (ROSIER) scale: development and validation of a stroke recognition instrument. Lancet Neurol. 2005; 4(11):727–34. 10.1016/S1474-4422(05)70201-5 [DOI] [PubMed] [Google Scholar]
  • 9.Purrucker JC, Hametner C, Engelbrecht A, Bruckner T, Popp E, Poli S. Comparison of stroke recognition and stroke severity scores for stroke detection in a single cohort. J Neurol Neurosurg Psychiatry. 2015; 86(9):1021–8. 10.1136/jnnp-2014-309260 [DOI] [PubMed] [Google Scholar]
  • 10.Walker GB, Zhelev Z, Henschke N, Fridhandler J, Yip S. Prehospital Stroke Scales as Screening Tools for Early Identification of Stroke and Transient Ischemic Attack. Stroke. 2019; 50(10):e285–e286. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 11.Zhelev Z, Walker G, Henschke N, Fridhandler J, Yip S. Prehospital stroke scales as screening tools for early identification of stroke and transient ischemic attack. Cochrane Database Syst Rev. 2019; 4:CD011427 10.1002/14651858.CD011427.pub2 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 12.Nor AM, McAllister C, Louw SJ, Dyker AG, Davis M, Jenkinson D, et al. Agreement between ambulance paramedic- and physician-recorded neurological signs with Face Arm Speech Test (FAST) in acute stroke patients. Stroke. 2004; 35(6):1355–9. 10.1161/01.STR.0000128529.63156.c5 [DOI] [PubMed] [Google Scholar]
  • 13.Oostema JA, Konen J, Chassee T, Nasiri M, Reeves MJ. Clinical predictors of accurate prehospital stroke recognition. Stroke. 2015; 46(6):1513–7. 10.1161/STROKEAHA.115.008650 [DOI] [PubMed] [Google Scholar]
  • 14.Kothari RU, Pancioli A, Liu T, Brott T, Broderick J. Cincinnati Prehospital Stroke Scale: reproducibility and validity. Ann Emerg Med. 1999; 33(4):373–8. 10.1016/s0196-0644(99)70299-4 [DOI] [PubMed] [Google Scholar]
  • 15.Kidwell CS, Starkman S, Eckstein M, Weems K, Saver JL. Identifying stroke in the field. Prospective validation of the Los Angeles prehospital stroke screen (LAPSS). Stroke. 2000; 31(1):71–6. 10.1161/01.str.31.1.71 [DOI] [PubMed] [Google Scholar]
  • 16.Alberts MJ, Latchaw RE, Selman WR, Shephard T, Hadley MN, Brass LM, et al. Recommendations for comprehensive stroke centers: a consensus statement from the Brain Attack Coalition. Stroke. 2005; 36(7):1597–616. 10.1161/01.STR.0000170622.07210.b4 [DOI] [PubMed] [Google Scholar]
  • 17.Yang HJ, Jeon W, Yang HJ, Kwak JR, Seo HY, Lee JS. The Clinical Differences between Urgent Visits and Non-Urgent Visits in Emergency Department During the Neonatal Period. J Korean Med Sci. 2017; 32(11):1870–5. 10.3346/jkms.2017.32.11.1870 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 18.Lee JS, Hong JM, Choi SC, Jung YS. "Cubic S Model": A for Early Recognition of Acute Ischemic Stroke Patients. J Korean Soc Emerg Med. 2010;21(3):313–320. [Google Scholar]
  • 19.Perez de la Ossa N, Carrera D, Gorchs M, Querol M, Millan M, Gomis M, et al. Design and validation of a prehospital stroke scale to predict large arterial occlusion: the rapid arterial occlusion evaluation scale. Stroke. 2014; 45(1):87–91. 10.1161/STROKEAHA.113.003071 [DOI] [PubMed] [Google Scholar]
  • 20.Teleb MS, Ver Hage A, Carter J, Jayaraman MV, McTaggart RA. Stroke vision, aphasia, neglect (VAN) assessment-a novel emergent large vessel occlusion screening tool: pilot study and comparison with current clinical severity indices. J Neurointerv Surg. 2017; 9(2):122–6. 10.1136/neurintsurg-2015-012131 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 21.Scheitz JF, Abdul-Rahim AH, MacIsaac RL, Cooray C, Sucharew H, Kleindorfer D, et al. Clinical Selection Strategies to Identify Ischemic Stroke Patients With Large Anterior Vessel Occlusion: Results From SITS-ISTR (Safe Implementation of Thrombolysis in Stroke International Stroke Thrombolysis Registry). Stroke. 2017; 48(2):290–7. 10.1161/STROKEAHA.116.014431 [DOI] [PubMed] [Google Scholar]
  • 22.Fransen PS, Berkhemer OA, Lingsma HF, Beumer D, van den Berg LA, Yoo AJ, et al. Time to Reperfusion and Treatment Effect for Acute Ischemic Stroke: A Randomized Clinical Trial. JAMA Neurol. 2016; 73(2):190–6. 10.1001/jamaneurol.2015.3886 [DOI] [PubMed] [Google Scholar]
  • 23.Marler JR, Tilley BC, Lu M, Brott TG, Lyden PC, Grotta JC, et al. Early stroke treatment associated with better outcome: the NINDS rt-PA stroke study. Neurology. 2000; 55(11):1649–55. 10.1212/wnl.55.11.1649 [DOI] [PubMed] [Google Scholar]
  • 24.Hacke W, Donnan G, Fieschi C, Kaste M, von Kummer R, Broderick JP, et al. Association of outcome with early stroke treatment: pooled analysis of ATLANTIS, ECASS, and NINDS rt-PA stroke trials. Lancet. 2004; 363(9411):768–74. 10.1016/S0140-6736(04)15692-4 [DOI] [PubMed] [Google Scholar]
  • 25.Brandler ES, Sharma M, Sinert RH, Levine SR. Prehospital stroke scales in urban environments: a systematic review. Neurology. 2014; 82(24):2241–9. 10.1212/WNL.0000000000000523 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 26.Brainin M, Teuschl Y, Kalra L. Acute treatment and long-term management of stroke in developing countries. Lancet Neurol. 2007; 6(6):553–61. 10.1016/S1474-4422(07)70005-4 [DOI] [PubMed] [Google Scholar]
  • 27.Vidale S, Agostoni E. Prehospital stroke scales and large vessel occlusion: A systematic review. Acta Neurol Scand. 2018; 138(1):24–31. 10.1111/ane.12908 [DOI] [PubMed] [Google Scholar]
  • 28.Demeestere J, Garcia-Esperon C, Lin L, Bivard A, Ang T, Smoll NR, et al. Validation of the National Institutes of Health Stroke Scale-8 to Detect Large Vessel Occlusion in Ischemic Stroke. J Stroke Cerebrovasc Dis. 2017; 26(7):1419–26. 10.1016/j.jstrokecerebrovasdis.2017.03.020 [DOI] [PubMed] [Google Scholar]
  • 29.Turc G, Maier B, Naggara O, Seners P, Isabel C, Tisserand M, et al. Clinical Scales Do Not Reliably Identify Acute Ischemic Stroke Patients With Large-Artery Occlusion. Stroke. 2016; 47(6):1466–72. 10.1161/STROKEAHA.116.013144 [DOI] [PubMed] [Google Scholar]

Decision Letter 0

Juan Manuel Marquez-Romero

Transfer Alert

This paper was transferred from another journal. As a result, its full editorial history (including decision letters, peer reviews and author responses) may not be present.

20 Jan 2020

PONE-D-19-32649

Stepwise stroke recognition through Clinical Information, Vital signs, and Initial Labs (CIVIL): Electronic health record-based observational cohort study

PLOS ONE

Dear Dr. Hong,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

We would appreciate receiving your revised manuscript by Mar 05 2020 11:59PM. When you are ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter.

To enhance the reproducibility of your results, we recommend that if applicable you deposit your laboratory protocols in protocols.io, where a protocol can be assigned its own identifier (DOI) such that it can be cited independently in the future. For instructions see: http://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols

Please include the following items when submitting your revised manuscript:

  • A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). This letter should be uploaded as separate file and labeled 'Response to Reviewers'.

  • A marked-up copy of your manuscript that highlights changes made to the original version. This file should be uploaded as separate file and labeled 'Revised Manuscript with Track Changes'.

  • An unmarked version of your revised paper without tracked changes. This file should be uploaded as separate file and labeled 'Manuscript'.

Please note while forming your response, if your article is accepted, you may have the opportunity to make the peer review history publicly available. The record will include editor decision letters (with reviews) and your responses to reviewer comments. If eligible, we will contact you to opt in or out.

We look forward to receiving your revised manuscript.

Kind regards,

Juan Manuel Marquez-Romero, M.D., M.Sc.

Academic Editor

PLOS ONE

Journal requirements:

When submitting your revision, we need you to address these additional requirements.

1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at

http://www.journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and http://www.journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf

2. We note that you have indicated that data from this study are available upon request. PLOS only allows data to be available upon request if there are legal or ethical restrictions on sharing data publicly. For information on unacceptable data access restrictions, please see http://journals.plos.org/plosone/s/data-availability#loc-unacceptable-data-access-restrictions.

In your revised cover letter, please address the following prompts:

a) If there are ethical or legal restrictions on sharing a de-identified data set, please explain them in detail (e.g., data contain potentially identifying or sensitive patient information) and who has imposed them (e.g., an ethics committee). Please also provide contact information for a data access committee, ethics committee, or other institutional body to which data requests may be sent.

b) If there are no restrictions, please upload the minimal anonymized data set necessary to replicate your study findings as either Supporting Information files or to a stable, public repository and provide us with the relevant URLs, DOIs, or accession numbers. Please see http://www.bmj.com/content/340/bmj.c181.long for guidelines on how to de-identify and prepare clinical data for publication. For a list of acceptable repositories, please see http://journals.plos.org/plosone/s/data-availability#loc-recommended-repositories.

We will update your Data Availability statement on your behalf to reflect the information you provide.

Reviewers' comments:

1. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

Reviewer #2: Yes

**********

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: No

Reviewer #2: Yes

3. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: Yes

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: No

Reviewer #2: Yes

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: This paper is interesting, but the sensitivity results are less than 80% for tier 2 and 3. The stroke is a serious disease with potential treatment therefore requires better levels of sensitivity. I recommend mentioning this finding as a weakness of the study.

Reviewer #2: I had the pleasure to read and review the manuscript: "Stepwise stroke recognition through Clinical Information, Vital signs, and Initial Labs (CIVIL): Electronic health record-based observational cohort study", which is a very interesting article addressing the main objective of develop a scoring system for three different scenarios: ischemic stroke, hemorrhagic stroke and LVO, in a pre-hospital stage. Before considering this manuscript for publication, I have some comments to add:

1) Abstract:

Nothing to add

2) Introduction:

One of the main aspects that should be mentioned in this manuscript, is the current availability of scales and scoring systems in the pre-hospital management of possible stroke cases. Two recent articles (Stroke. 2019;50:e285–e286 and Stroke. 2015 Jun;46(6):1513-7) already intended to analyze the complexity of the different systems that are currently part of many emergency and pre-hospital departments around the world. The justification of doing your research, seems that optimal and more valid scoring systems are needed, but I can't see this rationale the way you are presenting the manuscript.

2) Methods:

- Data availability (repository or by request) should be clarified.

- I understand that information to develop this scoring system is based on information from the Emergency Department, mainly from data that already is being recruited from the Cubic S model, but all this information is captured AT the emergency department, and was transferred to the pre-hospital scenario, which seems logic, but also is a different area of work and the possibility of losing information from the real pre-hospital work is plausible.

- Statistical analysis: which variables were included at the multivariate analysis? Were they pre-defined? If so, which cut-off point was decided to include the variable in the multivariate model.

- Youden's Index together with the scoring system performance should be included.

- OR and CI 95% are included in your analysis, but this also should be mentioned at the "statistics" section

- As you are doing only internal validation of your scoring system, did you consider a bootstraping to evaluate a more accurate performance in your population?

3) Results

- The decision on each variable included in the CIVIL ASAP tool, was done based on current, recent and the highest level of bibliography, or only extracted from the dataset.

- How do you define "mental change"? In terms of the MAPS scoring system?

- I can't see any of the results referring to the multivariate model... Do all the OR are un-adjusted or adjusted? If you adjusted, which variables were included at the model?

- Did you perform a ROC-AUC analysis to evaluate the performance of your scoring system compared to the other systems (you mention that at the methods section)? Could you provide a figure of the comparison of each curve according to the pre-hospital score used to recognize a "confirmed stroke case"

4) Discussion:

- Only one comment: current scoring systems are very easy to use; its performance vary, and seems that the CIVIL has a very good opportunity to prove your rationale, but I think that the applicability of the scoring point is very difficult, so, you should try to convince the readers that they should use this system.

**********

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files to be viewed.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email us at figures@plos.org. Please note that Supporting Information files do not need this step.

PLoS One. 2020 Apr 15;15(4):e0231113. doi: 10.1371/journal.pone.0231113.r002

Author response to Decision Letter 0

19 Feb 2020

Reviewer #1:

Comment 1. This paper is interesting, but the sensitivity results are less than 80% for tier 2 and 3. The stroke is a serious disease with potential treatment therefore requires better levels of sensitivity. I recommend mentioning this finding as a weakness of the study.

Response 1. Thank you for the reviewer’s valuable comment. We totally agree with the reviewer’s comment. As the reviewers mentioned, the sensitivity of CIVIL score is limited, and the results should be interpreted with caution. The CIVIL scoring system had a relative high sensitivity in tier 1 (to identify all possible candidates), and a relative high specificity in tier 2 and 3 (to recognize the need for urgent treatments, such as intravenous thrombolysis or endovascular thrombectomy). Therefore, we added limitation of the sensitivity results in discussion section of the revised manuscript (p20-1). Please check this point.

 

Reviewer #2: I had the pleasure to read and review the manuscript: "Stepwise stroke recognition through Clinical Information, Vital signs, and Initial Labs (CIVIL): Electronic health record-based observational cohort study", which is a very interesting article addressing the main objective of develop a scoring system for three different scenarios: ischemic stroke, hemorrhagic stroke and LVO, in a pre-hospital stage. Before considering this manuscript for publication, I have some comments to add:

1) Abstract:

Nothing to add

2) Introduction:

Comment 1. One of the main aspects that should be mentioned in this manuscript, is the current availability of scales and scoring systems in the pre-hospital management of possible stroke cases. Two recent articles (Stroke. 2019;50:e285–e286 and Stroke. 2015 Jun;46(6):1513-7) already intended to analyze the complexity of the different systems that are currently part of many emergency and pre-hospital departments around the world. The justification of doing your research, seems that optimal and more valid scoring systems are needed, but I can't see this rationale the way you are presenting the manuscript.

Response 1. Thank you for the reviewer’s valuable comment. Various scales and scoring systems have been developed and analyzed, but there is still no consensus on which scale work better, especially in large vessel occlusion (LVO).1,2 In addition, the accuracy of EMS stroke recognition is not enough due to issues related to false positives and false negatives.3 As noted by the reviewer, the need to develop a more optimized and valid scoring system should have been more convincing. Therefore, we added this rationale in the introduction section of the revised manuscript (p5). Please consider this point.

1. Walker GB, Zhelev Z, Henschke N, et al. Prehospital Stroke Scales as Screening Tools for Early Identification of Stroke and Transient Ischemic Attack. Stroke. 2019;50(10):e285-e286.

2. Zhelev Z, Walker G, Henschke N, et al. Prehospital stroke scales as screening tools for early identification of stroke and transient ischemic attack. Cochrane Database Syst Rev. 2019;4:CD011427.

3. Oostema JA, Konen J, Chassee T, et al. Clinical predictors of accurate prehospital stroke recognition. Stroke. 2015;46(6):1513-7.

2) Methods:

Comment 2. Data availability (repository or by request) should be clarified.

Response 2. In accordance with journal requirements, we added an unidentified data set as supporting information. Please check this point.

Comment 3. I understand that information to develop this scoring system is based on information from the Emergency Department, mainly from data that already is being recruited from the Cubic S model, but all this information is captured at the emergency department, and was transferred to the pre-hospital scenario, which seems logic, but also is a different area of work and the possibility of losing information from the real pre-hospital work is plausible.

Response 3. We fully understand the reviewer’s concern about the applicability of the CIVIL scoring system to real pre-hospital work. As the reviewer mentioned, our data were collected in the emergency department by ER physicians. In detail, the items consist of clinical information, vital signs, and initial labs. The selected items were age, past history, key clinical manifestations, vital sign, and blood glucose level. In actual prehospital situations, clinical manifestations (asymmetry, not ambulating, gaze deviation, speech disturbance) are familiar and have often been used in other well-known scoring systems (CPSS, LAPSS, ROSIER, etc.). Also, the vital signs and blood sugar levels are always checked in ambulances. Therefore, we believe that the CIVIL scoring system can be easily applied to actual pre-hospital work after systematically educating EMS paramedics. In the discussion section of the revised manuscript (p21), we described that EMS paramedics will require systematic training and verification. Please check this point.

Comment 4. Statistical analysis: which variables were included at the multivariate analysis? Were they pre-defined? If so, which cut-off point was decided to include the variable in the multivariate model.

Response 4. We entered variables with p value < 0.05 in univariate analyses into a multivariate logistic regression model. For confounding factors, more significant or intuitive variables were selected. Please check this point in the revised manuscript (p10).

Comment 5. Youden's Index together with the scoring system performance should be included.

Response 5. Thank you for the reviewer’s comment. Youden’s index was 0.385 at a cut-off point ≥ 1 in tier 1, 0.329 at a cut-off point ≥ 2 in tier 2, and 0.463 at a cut-off point ≥ 3 in tier 3, respectively. We have provided these values in the results section of the revised manuscript (p16-7). Please check this point.

Comment 6. OR and CI 95% are included in your analysis, but this also should be mentioned at the "statistics" section

Response 6. We added OR and CI 95% analysis in the statistics section of the revised manuscript (p10). Please check this point.

Comment 7. As you are doing only internal validation of your scoring system, did you consider a bootstraping to evaluate a more accurate performance in your population?

Response 7. Thank you for the reviewer’s valuable comment. As the reviewer’s comment, we performed additional analysis using bootstrap to evaluate more accurate performance. Bootstrap results showed that the parameter estimates closely agreed with the corresponding values in the final model (Theses results were based on 1,000 bootstrap sample.). Please check the tables below.

Tier 1

Model estimates Bootstrap results

Variables β (SE%) OR with 95% CI P β (SE%) OR with 95% CI P

Age ≥ 60 years 0.32 (15.4) 1.38 (1.02-1.86) 0.04 0.32 (15.1) 1.38 (1.03-0.87) 0.03

Age ≤ 40 years -0.58 (26.6) 0.56 (0.33-0.94) 0.03 -0.58 (28.8) 0.56 (0.32-1.00) 0.04

Hypertension history 0.16 (14.3) 1.18 (0.89-1.56) 0.25 0.16 (14.2) 1.18 (0.89--1.54) 0.24

Cardiac disease history 0.40 (18.0) 1.49 (1.05-2.12) 0.03 0.40 (18.2) 1.49 (1.05-2.13) 0.03

Seizure or psychiatric history -2.79 (27.8) 0.06 (0.04-0.11) <0.01 -2.79 (29.3) 0.06 (0.03-0.10) 0.00

After awakening 0.05 (24.6) 1.05 (0.65-1.70) 0.84 0.05 (25.5) 1.05 (0.63-1.76) 0.84

Any asymmetry 1.20 (16.5) 3.32 (2.40-4.58) <0.01 1.20 (16.9) 3.32 (2.40-4.68) 0.00

Not ambulating 0.76 (16.4) 2.14 (1.55-2.95) <0.01 0.76 (16.6) 2.14 (1.54-2.98) 0.00

No grasping 0.11 (21.7) 1.12 (0.73-1.72) 0.60 0.11 (23.0) 1.12 (0.74-1.83) 0.62

Mental change -0.23 (17.0) 0.79 (0.57-1.11) 0.17 -0.23 (16.8) 0.79 (0.57-1.12) 0.16

SBP ≥ 140 mmgHg 0.66 (14.0) 1.93 (1.47-2.53) <0.01 0.66 (14.6) 1.93 (1.46-2.55) 0.00

SBP ≤ 90 mmHg -1.60 (70.2) 0.20 (0.05-0.80) 0.02 -1.60 (455.3) 0.20 (0.00-0.84) 0.03

Extreme glucose level -1.97 (38.3) 0.14 (0.07-0.30) <0.01 -1.97 (42.7) 0.14 (0.06-0.30) 0.00

Tier 2

Model estimates Bootstrap results

Variables β (SE%) OR with 95% CI P β (SE%) OR with 95% CI P

Age ≤ 60 years 0.68 (16.6) 1.96 (1.42-2.72) <0.01 0.68 (16.9) 1.96 (1.43-2.77) 0.00

Diabetes mellitus history -0.45 (21.7) 0.64 (0.42-0.97) 0.04 -0.45 (22.0) 0.64 (0.41-0.95) 0.03

Cardiac disease history -1.49 (27.0) 0.23 (0.13-0.38) <0.01 -1.49 (27.6) 0.23 (0.11-0.36) 0.00

Sudden 1.00 (60.8) 2.71 (0.82-8.94) 0.10 1.00 (65.6) 2.71 (0.80-9.84) 0.08

After awakening -0.39 (53.5) 0.68 (0.24-1.93) 0.46 -0.39 (57.0) 0.68 (0.18-1.71) 0.45

As unusual -0.66 (70.6) 0.52 (0.13-2.06) 0.35 -0.66 (436.7) 0.52 (0.00-2.30) 0.40

Any asymmetry 0.04 (22.1) 1.04 (0.67-1.60) 0.86 0.04 (22.3) 1.04 (0.69-1.68) 0.85

Mental change 1.84 (20.8) 6.28 (4.18-9.44) <0.01 1.84 (22.1) 6.28 (4.25-10.08) 0.00

Abnormal sensation -0.35 (25.4) 0.71 (0.43-1.16) 0.17 -0.35 (27.2) 0.71 (0.39-1.14) 0.20

SBP ≥ 160 mmgHg 0.76 (15.9) 2.13 (1.56-2.91) <0.01 0.76 (16.6) 2.13 (1.56-3.00) 0.00

Tier 3

Model estimates Bootstrap results

Variables β (SE%) OR with 95% CI P β (SE%) OR with 95% CI P

Gaze deviation 3.08 (20.1) 21.77 (14.68-32.27) <0.01 3.08 (21.4) 21.76 (15.04-34.99) 0.00

Face asymmetry 0.04 (20.0) 1.04 (0.70-1.54) 0.85 0.04 (20.3) 1.04 (0.69-1.55) 0.85

Arm asymmetry 0.78 (22.1) 2.19 (1.42-3.37) <0.01 0.78 (22.2) 2.19 (1.47-3.43) 0.00

Speech disturbance 0.88 (19.6) 2.421 (1.65-3.55) <0.01 0.88 (19.5) 2.421 (1.66-3.59) 0.00

3) Results:

Comment 8. The decision on each variable included in the CIVIL ASAP tool, was done based on current, recent and the highest level of bibliography, or only extracted from the dataset.

Response 8. The decision to include variables is important and critical in developing a scoring system. Various aspects had to be considered, such as previous systems, statistical results, and availability in prehospital and emergency room. In tier 1 (CIVIL-ASAP tool), multivariate analysis indicated that nine variables are independent prognostic indicators: age (≥ 60 years, ≤ 40 years), cardiac disease history, seizure or psychiatric history, any asymmetry, not ambulating, systolic BP (≥ 140 mmHg, ≤ 90 mmHg), and extreme glucose level (≤ 80 or ≥ 400 mg/dl). In tier 2 (CIVIL-MAPS), five parameters were independent variables (age ≤ 60 years, diabetes mellitus, cardiac disease, mental change, SBP ≥ 160 mmHg). We developed CIVIL scoring system based on statistical results. These variables are also well-recognized because they are frequently used in previous scoring systems. Please consider this point.

Comment 9. How do you define "mental change"? In terms of the MAPS scoring system?

Response 9. As the reviewer mentioned, the definition of “mental change” needs to be clarified. After the training of ER physicians, a decrease in the level of consciousness below drowsiness during the initial neurological examination was assigned to “mental change”. It is equivalent to a NIHSS level of consciousness score (1a) ≥ 1. We clarified the definition of mental change in the Table 1 and 2 of the revised manuscript (p14-5). Please check this point.

Comment 10. I can't see any of the results referring to the multivariate model... Do all the OR are un-adjusted or adjusted? If you adjusted, which variables were included at the model?

Response 10. Thank you for the reviewer’s comment. Due to spatial limitations, the results of the multivariate model are shown only in Figure 1. Multivariate logistic regression analyze was performed after adjusting significant variables in univariate analyses (P<0.05). We clarified this point in Figure 1 (adjusted OR). Please check this point.

Comment 11. Did you perform a ROC-AUC analysis to evaluate the performance of your scoring system compared to the other systems (you mention that at the methods section)? Could you provide a figure of the comparison of each curve according to the pre-hospital score used to recognize a "confirmed stroke case"

Response 11. The results of ROC-AUC analysis were shown in the supplementary figure (S3) of supporting information. In Tier 1, CIVIL-ASAP score was compared with CPSS, LAPSS, and ROSIER scores. Please consider this point.

4) Discussion:

Comment 12. Only one comment: current scoring systems are very easy to use; its performance vary, and seems that the CIVIL has a very good opportunity to prove your rationale, but I think that the applicability of the scoring point is very difficult, so, you should try to convince the readers that they should use this system.

Response 12. As the reviewer have mentioned, the CIVIL scoring system appears to be more complex than current scoring systems. Usability is also important, but more detailed items may be required to improve the accuracy of the scoring system. Although the CIVIL scoring system included a significant number of items and had a stepwise approach, included items were frequently used in other well-known scoring systems (Figure 2). We also used an intuitive and easy-to-remember acronyms, and existing GFAST score was applied to Tier 3. These efforts can improve the accessibility of the scoring system. The complexity of the CIVIL scoring system can be overcome with systematic education and practical tools (such as mobile applications, checklist and automatic calculation system). We added need for further investigation to increase applicability of the CIVIL scoring system in the discussion section of the revised manuscript (p21). Please consider this point.

Attachment

Submitted filename: CIVIL_PLOS ONE_response letter.docx

Click here for additional data file. (265.3KB, docx)

Decision Letter 1

Juan Manuel Marquez-Romero

17 Mar 2020

Stepwise stroke recognition through Clinical Information, Vital signs, and Initial Labs (CIVIL): Electronic health record-based observational cohort study

PONE-D-19-32649R1

Dear Dr. Hong,

We are pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it complies with all outstanding technical requirements.

Within one week, you will receive an e-mail containing information on the amendments required prior to publication. When all required modifications have been addressed, you will receive a formal acceptance letter and your manuscript will proceed to our production department and be scheduled for publication.

Shortly after the formal acceptance letter is sent, an invoice for payment will follow. To ensure an efficient production and billing process, please log into Editorial Manager at https://www.editorialmanager.com/pone/, click the "Update My Information" link at the top of the page, and update your user information. If you have any billing related questions, please contact our Author Billing department directly at authorbilling@plos.org.

If your institution or institutions have a press office, please notify them about your upcoming paper to enable them to help maximize its impact. If they will be preparing press materials for this manuscript, you must inform our press team as soon as possible and no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

With kind regards,

Juan Manuel Marquez-Romero, M.D., M.Sc.

Academic Editor

PLOS ONE

Additional Editor Comments (optional):

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation.

Reviewer #2: All comments have been addressed

**********

2. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #2: Yes

**********

3. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #2: Yes

**********

4. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #2: Yes

**********

5. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #2: Yes

**********

6. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #2: No comments, all the suggestions were addressed therefore I consider it is suitable for publication

**********

7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #2: Yes: Miguel A. Barboza

Acceptance letter

Juan Manuel Marquez-Romero

25 Mar 2020

PONE-D-19-32649R1

Stepwise stroke recognition through Clinical Information, Vital signs, and Initial Labs (CIVIL): Electronic health record-based observational cohort study

Dear Dr. Hong:

I am pleased to inform you that your manuscript has been deemed suitable for publication in PLOS ONE. Congratulations! Your manuscript is now with our production department.

If your institution or institutions have a press office, please notify them about your upcoming paper at this point, to enable them to help maximize its impact. If they will be preparing press materials for this manuscript, please inform our press team within the next 48 hours. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information please contact onepress@plos.org.

For any other questions or concerns, please email plosone@plos.org.

Thank you for submitting your work to PLOS ONE.

With kind regards,

PLOS ONE Editorial Office Staff

on behalf of

Dr. Juan Manuel Marquez-Romero

Academic Editor

PLOS ONE

Associated Data

    This section collects any data citations, data availability statements, or supplementary materials included in this article.

    Supplementary Materials

    S1 STROBE Checklist

    (DOC)

    Click here for additional data file. (83KB, doc)
    S1 Data. Deidentified raw dataset.

    (XLSX)

    Click here for additional data file. (691.8KB, xlsx)
    S1 Fig. The Korean version of EMR based matrix for stroke suspicious patients.

    (Cubic S model)

    (PDF)

    Click here for additional data file. (105.7KB, pdf)
    S2 Fig. Flow diagram of 1,621 suspicious stroke patients.

    (PDF)

    Click here for additional data file. (260.4KB, pdf)
    S3 Fig. Receiver-Operating Characteristic (ROC) curve and corresponding area under the curve (AUC) statistics of the CIVIL scoring system.

    (PDF)

    Click here for additional data file. (189.5KB, pdf)
    Attachment

    Submitted filename: CIVIL_PLOS ONE_response letter.docx

    Click here for additional data file. (265.3KB, docx)

    Data Availability Statement

    All relevant data are within the paper and its Supporting Information files.

    Articles from PLoS ONE are provided here courtesy of PLOS

    RESOURCES