Reliability of the revised Swiss Emergency Triage Scale: a computer simulation study

Abstract

Background

The Swiss Emergency Triage Scale (SETS) is a four-level emergency scale that previously showed moderate reliability and high rates of undertriage due to a lack of standardization. It was revised to better standardize the measurement and interpretation of vital signs during the triage process.

Objective

The aim of this study was to explore the inter-rater and test–retest reliability, and the rate of correct triage of the revised SETS.

Patients and methods

Thirty clinical scenarios were evaluated twice at a 3-month interval using an interactive computerized triage simulator by 58 triage nurses at an urban teaching emergency department admitting 60 000 patients a year. Inter-rater and test–retest reliabilities were determined using κ statistics. Triage decisions were compared with a gold standard attributed by an expert panel. Rates of correct triage, undertriage, and overtriage were computed. A logistic regression model was used to identify the predictors of correct triage.

Results

A total of 3387 triage situations were analyzed. Inter-rater reliability showed substantial agreement [mean κ: 0.68; 95% confidence interval (CI): 0.60–0.78] and test–retest almost perfect agreement (mean κ: 0.86; 95% CI: 0.84–0.88). The rate of correct triage was 84.1%, and rates of undertriage and overtriage were 7.2 and 8.7%, respectively. Vital sign measurement was an independent predictor of correct triage (odds ratios for correct triage: 1.29 for each additional vital sign measured, 95% CI: 1.20–1.39).

Conclusion

The revised SETS incorporating standardized vital sign measurement and interpretation during the triage process resulted in high reliability and low rates of mistriage.

Introduction

Triage is a necessity for most overcrowded emergency departments (EDs). Although five-level triage instruments are currently used in North America, in the UK, and in Australia 1–3, few European countries have developed and adopted a universal triage system, especially in French-speaking countries. A four-level triage scale [the Swiss Emergency Triage Scale (SETS)] has been used by many EDs in Switzerland, France, and Belgium since 1997 and it was evaluated using an interactive computerized simulator 4. This evaluation showed only moderate reliability and high rates of undertriage (31%). Similar rates of undertriage have been reported with the Emergency Severity Index (ESI) and the Manchester Triage Scale (MTS) and occurred mainly in intermediate acuity emergencies 5,6. In our previous evaluation, the suboptimal performance resulted mainly from a wide variability in the triage process, especially a lack of standardization of vital sign measurement and interpretation during triage.

Vital sign measurement is part of most modern triage instruments and is used to better categorize patients, especially to identify life-threatening conditions 1,7–9. Although these measurements are frequently incorporated in the triage process, their specific contribution to the triage process has never been evaluated. A revised version of the SETS incorporates a systematic and standardized way of interpreting vital sign results during the triage process (Table 1). We hypothesized that this procedure should improve triage performance.

Table 1.

Criteria used to guide triage decision using the Swiss Emergency Triage Scale

The primary objective of this study was to evaluate the inter-rater and test–retest reliability of the revised SETS incorporating standards for vital sign measurement and interpretation. The secondary objectives were (a) to evaluate the rate of correct triage, undertriage, and overtriage using the revised SETS and (b) to identify factors associated with correct triage.

Patients and methods

Study design

This study undertook a prospective evaluation of triage situations using a computer triage simulator 4. Thirty interactive computerized scenarios describing adult patients presenting at the ED were developed. Each scenario started with a presenting complaint, and triage nurses could type questions and obtain replies, including vital signs, through the computer simulator, before deciding on the triage severity rating. The computer program simulated the triage process in a manner as close as possible to real-life triage. These scenarios were rated twice by all evaluators at a 3-month interval.

Study setting

This study was conducted in the ED of a primary and tertiary urban teaching hospital admitting 60 000 patients/year. All patients admitted into the ED are first triaged by a triage nurse using the SETS. The SETS is a symptom-based four-level triage scale incorporating timeliness objectives 4: SETS level 1 represents life/limb-threatening situations that should be assessed and treated immediately; SETS level 2 represents potentially life-threatening situations in which assessment and treatment should be performed within 20 min; SETS level 3 represents situations whose assessment and treatment are mandatory within 120 min; SETS level 4 represents nonurgent conditions.

To attribute an emergency level, the nurse has to identify the main presenting complaint of the patient from a predetermined list of presenting complaints (Table 2) and measure vital signs. For each presenting complaint one or more emergency levels may be attributed based on the results of vital sign measurement (Table 1). Nurses were trained to measure vital signs when the main complaints could be associated with a choice of two or more emergency levels. The choice of which vital signs were measured was conditioned by the main presenting complaint.

Table 2.

Examples of presenting complaints of the revised Swiss Emergency Triage Scale

In the simulator, three scenarios concerned level 1, 11 considered level 2, 12 considered level 3 and four concerned level 4 emergencies (Table 3).

Table 3.

Summarized description of clinical scenarios played by the computer simulator

Study participants

A convenient sample including all emergency triage nurses of our ED was selected for participation in this study.

Methods of measurement and data analysis

To evaluate the primary outcome, inter-rater and test–retest reliabilities were determined using κ statistics by computing the concordance rates between observers (inter-rater reliability) during the first phase and between both phases for each observer (test–retest reliability). To answer our secondary objectives, triage acuity levels determined by the study participants were compared with the acuity levels attributed by an expert panel, and were considered the gold standard. The expert panel attributed one emergency level to each scenario taking into account the clinical history and physical evaluation at triage, including vital signs. Rates of undertriage and overtriage were calculated. Undertriage and overtriage were defined respectively as underestimation or overestimation of the emergency level by the evaluator compared with the expert-attributed acuity level. To identify predictors of correct triage, undertriage, and overtriage, univariate and multivariate analyses were performed in hierarchical logistic regression models adjusting for nurses’ characteristics (age, sex, qualification, and experience) and clinical scenario characteristics (severity, number of vital parameters measured). Adjusted odds ratios (ORs) with 95% confidence intervals (CIs) were reported. Statistical significance was defined as a P value less than 0.05 (two sided). Analyses were performed using Stata 8.1 (Stat Corp., College Station, Texas, USA) and IBM (Armonk, New York, USA) SPSS statistics for Windows, version 22.

The study was approved by our institutional review board and all participants gave their written informed consent (http://www.ClinicalTrials.gov identifier: NCT00321243).

Results

Participants’ characteristics

Fifty-eight nurses participated in the two phases of the evaluation of the revised scale. Their mean±SD age was 38.3±6.1 years; 90% (n=52) were female, and 74% (n=43) were specialized in emergency medicine (2 years’ specialization after RN diploma in Switzerland). Their employment activity rate was greater than 80 for 72% (42 nurses), their experience in emergency medicine greater than 2 years for 91% (53 nurses), and their experience in triage greater than 2 years for 66% (38 nurses).

Inter-rater and test–retest reliability

To evaluate inter-rater and test–retest reliability, 3387 triage situations were analyzed. The inter-rater reliability of our revised triage scale had a mean κ of 0.68 (95% CI: 0.60–0.78). Test–retest reliability of our revised scale had a mean κ of 0.86 (95% CI: 0.84–0.88).

Triage performance

A perfect concordance between the triage levels attributed by the evaluators and the gold standard was observed in 84.1% of the situations. The rate of undertriage was 7.2% and the rate of overtriage was 8.7%. The rates of undertriage and overtriage varied across triage categories (Table 4).

Table 4.

Percentages of overtriage and undertriage according to the Swiss Emergency Triage Scale emergency levels

Triage process and predictors of correct triage

The evaluation of vital parameters varied across evaluators and scenarios. The mean±SD number of measured vital parameters varied from 0.90±0.99 to 5.57±2.33 across nurses and from 1.28±1.23 to 5.24±1.65 across scenarios (Fig. 1a and b). At least one vital parameter was measured in 92.1% of triage situations.

Fig. 1.

(a, b) Variability in the triage process. (a) Variability in the number of vital parameters requested by each evaluator (nurse ID number). (b) Variability in the number of vital parameters gathered for each simulated scenario (vignette number).

In our multivariate models, two variables were statistically associated with higher rates of correct triage (Table 5). First, lower acuity scenarios (SETS emergency levels 3 and 4) were more frequently correctly triaged (OR 2.27; 95% CI: 1.72–3.00). Second, vital sign measurement was positively associated with correct triage (OR 1.29 for each additional vital parameter measured during triage, 95% CI: 1.20–1.39). In mirror, measurement of vital signs during triage was associated with reduced risk for overtriage and undertriage. No nurse-related factor (sex, age, experience, activity rate) was associated with correct triage.

Table 5.

Predictors of correct triage, undertriage, and overtriage in hierarchical multivariate analyses

Discussion

Achieving the best reliability should be a goal for any triage instrument. The introduction of a standardized process for the evaluation and interpretation of vital sign measurement and simple bedside tests (glycemia and peak flow) during the triage process improved the reliability of the SETS. Its inter-rater reliability showed substantial agreement and its test–retest reliability almost perfect agreement, which is comparable to the best validated triage instruments 8,10–14. Indeed, recent meta-analyses showed that pooled κ coefficients were 0.67, 0.79, 0.75, and 0.42, respectively, for the Canadian Emergency Department Triage, ESI, MTS, and Australasian Triage Scale 11–13,15. The results obtained with the revised SETS were better than those obtained with the former version of our triage scale that did not standardize vital signs 4. Furthermore, we achieved rates of undertriage and overtriage similar or lower than those achieved with other triage scales 8,10,16. Rates of undertriage observed with the MTS varied from 11 to 25% and those observed with the ESI were as high as 17% 5,6,17. Again, the results of the revised SETS outperformed our unrevised scale (58% of correct triage and 11 and 31% of overtriage and undertriage, respectively) 4.

There is little published evidence on the impact of vital sign measurement and bedside tests on triage decisions, and the added value of their standardized evaluation in the triage process has not been specifically addressed. Among the most widely validated triage tools, the ESI integrates the measurement of heart rate (HR), respiratory rate (RR), and oxygen saturation (SpO₂) and uses these parameters to upgrade patients from ESI 3 to 2, but it does not integrate other clinical signs 7. How these measurements impact reliability has not been assessed. Our study suggests that vital sign measurement is a cornerstone of the triage process and that other signs than HR, RR, or SpO₂ may be important for triage decisions. Each additional vital parameter measured at triage was associated with an increased chance of correct triage (OR 1.29; 95% CI: 1.20–1.39). Other studies support our results: Cooper et al. 18 showed that the knowledge of BP, HR, temperature, RR, and SpO₂ affected the triage decisions in 7.9% of the triage situations, although the specific role of each parameter was not explored. In this study, the knowledge of vital signs resulted in an upgrade of the emergency level in the majority (70%) of situations. In another study, Nakagawa et al. 19 showed that knowledge of vital signs modified triage destinations in 5.7% of the situations, with an upgrade in 1.2% and downgrade in 4.5%. In addition, even in patients with low to moderate urgency, abnormal vital signs have been shown to be associated with critical care admission 20. These results are in concordance with our study showing that a better use of vital signs at triage reduced the risk of mistriage.

Our study has some limitations. First, we used simulated scenarios instead of real-life patients. As the evaluation of a triage scale should explore the triage process, some authors suggest that triage scales should optimally be evaluated using real-life patients’ triage conditions. This has been only rarely and partly realized 9,21. In contrast, most published evaluations of triage instruments have been performed using written vignettes displaying all the information required for triage decisions, which does not reflect the triage complexity 3,7,8. Our triage computerized simulator allows an interactive evaluation of patients’ complaints in a way very close to real-life conditions. The evaluation of the same scenario by numerous evaluators can be performed, which is not possible with real patients. Our triage computerized simulator could also be a tool to compare different triage scales in different settings. Therefore, we think that this simulator is a good alternative to real-life conditions, not only to evaluate triage decisions but also the triage process. Second, the gold standard used to determine correct triage, overtriage, and undertriage was based on expert agreement. As the experts were aware of the SETS criteria, this may potentially lead to an overestimation of triage performance.

Conclusion

This study showed that the revised SETS integrating standardized vital sign measurements had high reliability and low rates of undertriage and overtriage, comparable to those of the best validated triage instruments.