Reliability of the Emergency Severity Index

Abstract

Objectives:

Although triage systems based on the Emergency Severity Index (ESI) have many advantages in terms of simplicity and clarity, previous research has questioned their reliability in practice. Therefore, the aim of this meta-analysis was to determine the reliability of ESI triage scales.

Methods:

This meta-analysis was performed in March 2014. Electronic research databases were searched and articles conforming to the Guidelines for Reporting Reliability and Agreement Studies were selected. Two researchers independently examined selected abstracts. Data were extracted in the following categories: version of scale (latest/older), participants (adult/paediatric), raters (nurse, physician or expert), method of reliability (intra/inter-rater), reliability statistics (weighted/unweighted kappa) and the origin and publication year of the study. The effect size was obtained by the Z-transformation of reliability coefficients. Data were pooled with random-effects models and a meta-regression was performed based on the method of moments estimator.

Results:

A total of 19 studies from six countries were included in the analysis. The pooled coefficient for the ESI triage scales was substantial at 0.791 (95% confidence interval: 0.787–0.795). Agreement was higher with the latest and adult versions of the scale and among expert raters, compared to agreement with older and paediatric versions of the scales and with other groups of raters, respectively.

Conclusion:

ESI triage scales showed an acceptable level of overall reliability. However, ESI scales require more development in order to see full agreement from all rater groups. Further studies concentrating on other aspects of reliability assessment are needed.

Advances in Knowledge

• - It is important to determine the reliability of triage scales as this reveals the consistency of patient prioritisation within emergency departments.

• - The results of this meta-analysis demonstrate that the Emergency Severity Index (ESI) triage scale has substantial agreement and shows acceptable overall reliability when implemented in emergency departments within and outside of the USA.

• - However, the ESI was also found to have a tendency to allocate patients to level 2.

Application to Patient Care

• - A triage scale enables emergency departments to allocate resources to the most critically ill patients. It is therefore important for medical personnel to be aware of the reliability of the scale as inconsistent triage decisions may result in under-triage and put patients’ health in danger.

In emergency departments (EDs), patients are categorised based on their clinical acuity; thus, the more critically ill the patient, the sooner treatment is delivered and care needs are addressed.1 The Emergency Severity Index (ESI) is a five-level ED triage algorithm designed to stratify patients into groups based on clinical need. The ESI is continuously developed by physicians in the USA and has been adopted by several other developed countries.2,3 It has also been endorsed by the American College of Emergency Physicians and the Emergency Nurses Association.4

Many studies have investigated the validity and reliability of the ESI triage scale in both adult and paediatric populations.2,3,5–10 However, the extent to which the ESI triage scale is used in triage nurses’ decision-making outside of the USA is still unclear, especially considering the wide variety of healthcare systems currently in existence around the world.11 With regards to this, Andersson et al. addressed contextual influences on the triage decision-making process in rural Sweden.12 The reliability of triage scales outside of the USA should therefore be assessed for internal consistency, repeatability and inter-rater agreement.1,11 While the kappa statistic is most commonly used to measure inter-rater agreement, this statistic can be influenced by incidence, bias and levels of scale, potentially generating misleading results.13–15 Additionally, weighted kappa statistics have been reported to reveal deceivingly high reliability coefficients.11 Therefore, a pooled estimate of a reliability coefficient is more practical in the identification of significant differences among reliability methods.

A meta-analysis is a systematic approach for the introduction, evaluation, synthesis and unification of results with reference to a specific research question. It also produces the strongest evidence for intervention and is therefore an appropriate method to gain insight regarding the reliability of triage scales.16 A review by Christ et al. on the reliability of the ESI scale demonstrated kappa statistics ranging from 0.46 (moderate) to 0.98 (almost perfect).17 This considerable variation in kappa statistics indicates a discernable gap in the reliability of the triage scale. Thus, the aim of this meta-analysis was to review the reliability of the ESI triage scale in a variety of contexts.

Methods

This study was performed in March 2014. The first phase consisted of a literature search using the Cumulative Index to Nursing and Allied Health Literature (CINAHL), Scopus, MEDLINE^®/PubMed, Google Scholar and Cochrane Library databases. Studies published before 1 March 2014 in these databases and found using the following search terms were included: reliability; triage; system; scale; agreement; emergency, and emergency severity index. All studies in English identified by the database search were examined by two researchers to identify eligible articles regarding the reliability of the ESI. Irrelevant or duplicate results were eliminated. Reference lists of acceptable publications were also examined to identify further articles for inclusion in the study.

Articles were chosen for inclusion according to the Guidelines for Reporting Reliability and Agreement Studies.18 According to these guidelines, studies were only included in the analysis if they reported more than six of the following eight items in sufficient detail: sample size; number of raters; number of subjects; sampling method; rating process; statistical analysis, and reliability coefficient. Disagreements were resolved by consensus with a third researcher. Articles that did not report the type of reliability (either inter-rater reliability, intra-rater reliability or internal consistency) were excluded from the analysis. Researchers also recorded moderator variables relating to participants, raters and the origin and year of publication of the study as well as studies which were conducted based on the latest version (2012) of the ESI triage scale.

In the next phase, further information was retrieved from the articles, including: age group of participants and size of cohort; raters’ professions and overall number of raters; instruments used (e.g. live or scenario-based cases); country of origin and year of publication of study; reliability coefficient, and type of reliability. Reliability was determined by inter-rater reliability (weighted or unweighted kappa coefficients), intra-rater reliability (reliability statistics including intraclass correlation coefficient, Pearson correlation coefficient or Spearman’s rank correlation coefficient) and internal consistency (alpha coefficients) statistics. Authors of research articles were contacted for supplementary information if necessary.

In the meta-regression, each rater sample was considered to be a unit of analysis. If the same sample was reported in two or more articles, it was included only once in the analysis. In contrast, if several samples with different populations were reported in one study, each sample was included as a separate unit of analysis.

Data were pooled for all three types of reliability. Many articles reported a reliability coefficient using the kappa statistic; it could be considered an r-type coefficient ranging from −1.00 to +1.00. Standard agreement was categorised as poor (κ = 0.00–0.20), fair (κ = 0.21–0.40), moderate (κ = 0.41–0.60), substantial (κ = 0.61–0.80) or almost perfect (κ = 0.81–1.00).13 The kappa statistic can be treated as a correlation coefficient in meta-analyses.19 In order to obtain the correct interpretation, back-transformation (Z to r transformation) of pooled effect sizes to the level of primary coefficients was performed.20,21 Fixedand random-effect models were applied. Data were analysed using Comprehensive Meta-Analysis (CMA) Software, Version 2.2.050 (Biostat Inc., Englewood, New Jersey, USA).

A simple meta-regression analysis was performed according to the method of moments estimator.22 In the meta-regression model, effect size was a dependent variable while studies and subject characteristics were considered independent variables in order to discover potential predictors of reliability coefficients. Z-transformed reliability coefficients were regressed based on variables of country origin and year of publication as well as studies based on the latest version of triage scale versus those of a prior version. Distance was defined as the distance from each study’s country of origin to the city of Boston, Massachusetts, in the USA (where the ESI triage scale originated). Meta-regression was performed using a random-effects model due to the presence of significant inter-study variation.23

This study received ethical approval from the Research & Ethics Committee of Mashhad University of Medical Science in Mashhad, Iran.

Results

A total of 260 primary citations relevant to the reliability of the ESI triage scale were identified during the literature search. However, only 19 unique citations (7.3%) met the inclusion criteria.2,3,5–10,24–34 The studies were organised into subgroups according to participants (adult/paediatric); raters (nurses/physicians/experts); method of reliability (intra-/inter-rater); reliability statistics (weighted/unweighted kappa statistics), and by country of origin and publication year. The level of agreement among the researchers regarding the final selection of articles for the meta-analysis was almost perfect (κ = 1.0).

A total of 40,579 cases (both paper-based case scenarios and live triage cases) were included in the initial analysis. Among the 19 studies meeting the required criteria, the reliability of the ESI triage scale had been assessed in six different countries with publication years ranging from 2000–2013 (median year of publication: 2009). In addition, 70% had been conducted using the latest version of the triage scale. The inter-rater reliability method was used in all studies except for one, which used intra-rater reliability. None of the studies in the analysis had used the alpha coefficient to report internal consistency. The weighted kappa coefficient was the most commonly utilised statistic [Table 1].

Table 1:

Studies on the reliability of the Emergency Severity Index triage scale included in the meta-analysis (N = 19)

Author and year of study	Participant	Raters	Instrument	Method	Statistic	CO
Wuerz et al.5 2000	Adult	EE	Scenario	Inter	Kw	USA
	Adult	NP	Live cases	Inter
Wuerz et al.25 2001	Adult	NN	Scenario	Inter	Kw	USA
	Adult	NP	Scenario	Inter
Travers et al.26 2002	Adult	NE	Scenario	Inter	Kw	USA
Eitel et al.27 2003	Adult	NE	Scenario	Inter	Kw	USA
	Adult	NN	Live cases	Inter
Tanabe et al.24 2004	Adult	NE	Scenario	Inter	Kw	USA
	Adult	NE	Scenario	Inter
Worster et al.6 2004	Adult	NN	Scenario	Inter	Kw	Canada
Travers et al.28 2002	Adult	NN	Scenario	Inter	Kw	USA
	Paediatric	NN	Scenario	Inter
Choi et al.29 2009	Adult	NE	Live cases	Inter	Kw	Korea
	Adult	NP	Live cases	Inter	Kw
Storm-Versloot et al.3 2009	Adult	NN	Scenario	Intra	Kuw	Netherlands
	Adult	NN	Scenario	Inter	Kuw
	Adult	NN	Scenario	Intra	Kw
	Adult	NN	Scenario	Inter	Kw
Grossman et al.2 2011	Adult	EE	Scenario	Inter	r	Switzerland
	Adult	EE	Scenario	Inter	Kw
	Adult	NE	Scenario	Inter	r
	Adult	NE	Scenario	Inter	Kw
Platts-Mills et al.30 2010	Adult	NE	Live cases	Inter	Kw	USA
	Adult	NP	Live cases	Inter
Grossmann et al.31 2012	Adult	EE	Scenario	Inter	Kw	Switzerland
Baumann et al.7 2005	Paediatric	NE	Live cases	Inter	Kw	USA
	Paediatric	NE	Scenario	Inter
	Paediatric	NN	Scenario	Inter
	Paediatric	NP	Scenario	Inter
	Paediatric	PE	Scenario	Inter
Durani et al.32 2007	Paediatric	NN	Scenario	Inter	Kw	USA
	Paediatric	NP	Scenario
	Paediatric	PP	Scenario
Durani et al.8 2009	Paediatric	NN	Scenario	Inter	Kuw	USA
	Paediatric	NN	Scenario		Kw
	Paediatric	NP	Scenario		Kuw
	Paediatric	NP	Scenario		Kw
	Paediatric	PP	Scenario		Kuw
	Paediatric	PP	Scenario		Kw
Travers et al.9 2009	Paediatric	NE	Live cases	Inter	Kw	USA
	Paediatric	NN	Scenario
Green et al.10 2012	Paediatric	NN	Live cases	Inter	ICC	USA
	Paediatric	NN	Live cases		Kuw
	Paediatric	NP	Live cases		ICC
	Paediatric	NP	Live cases		Kuw
Jafari-Rouhi et al.33 2013	Paediatric	NN	Scenario	Inter	Kuw	Iran
	Paediatric	NP	Scenario		r
	Paediatric	NP	Scenario		Kuw
Ho et al.34 2007	Adult	EE	Live cases	Inter	Kw	USA

CO = country of origin; Inter = inter-rater reliability; Kw = weighted kappa; NP = nurse-physician; NN = nurse-nurse; NE = nurse-expert; Intra = intra-rater reliability; Kuw = unweighted kappa; EE = expert-expert; r = correlation coefficient; PE = physician-expert; PP = physician-physician.

The overall pooled coefficient for the ESI triage scale was substantial, at 0.791 (95% confidence interval [CI]: 0.752–0.825). Participants’ pooled coefficients ranged from substantial (0.732; 95% CI: 0.625–0.812) for nurse-expert agreement to almost perfect (0.900; 95% CI: 0.570–0.980) for expert-expert agreement [Figures 1 and 2]. Agreement regarding adult/paediatric versions of the ESI triage scale was almost perfect for both adult (0.815; 95% CI: 0.753–0.862) and paediatric patients (0.769; 95% CI: 0.747–0.837). Additionally, almost perfect agreement was noted for paper-based scenario assessments (0.824; 95% CI: 0.778–0.861) and substantial agreement was observed for live case assessments (0.694; 95% CI: 0.575–0.784). Inter- and intra-rater reliability agreement was 0.786 (95% CI: 0.745–0.821) and 0.873 (95% CI: 0.801–0.921), respectively. Substantial agreement was found for both weighted (0.796; 95% CI: 0.751–0.834) and unweighted kappa statistics (0.770; 95% CI%: 0.674–0.841). Agreement for the latest version of the ESI was 0.833 (95% CI: 0.774–0.878), whereas it was 0.808 (95% CI: 0.762–0.846) for previous versions.

Figure 1:

Fisher’s Z-transformation showing pooled estimates of participants’ reliability (random-effect model) among studies relating to the reliability of the Emergency Severity Index.2,3,5–10,24–34

EE = expert-expert; NE = nurse-expert; NN = nurse-nurse; NP = nurse-physician; PE = physician-expert; PP = physician-physician.

Figure 2:

Pooled estimates of measures of raters’ reliability (random-effect model) using weighted kappa statistics among studies relating to the reliability of the Emergency Severity Index.2,3,5–10,24–34

CI = confidence interval.

Only six studies presented a 5 × 5 contingency table to show the frequency distribution of triage decisions for each ESI level between two raters [Table 2].2,3,5,7,9,24 The overall agreement in these studies was 78.55%. Agreement for each ESI level was 1.12%, 23.40%, 19.55%, 18.81% and 15.67% for levels 1 to 5, respectively, while disagreement was 0.25%, 4.07%, 6.10%, 6.90% and 4.12%, respectively. A total of 80% of all disagreements concerned levels 3 to 5 (17.12% out of 21.44%). Only ESI level 2 decisions showed a wide distribution across all levels [Table 2].

Table 2:

Distribution of triage decision-making relating to each Emergency Severity Index triage category among emergency department raters in six studies2,3,5,7,9,24

Rater 2 ESI category	Rater 1 ESI category n (%)
	1	2	3	4	5	Total
1	35 (1.12)	9	0	0	0	44
2	7	730 (23.39)	102	26	3	868
3	0	82	610 (19.55)	91	12	795
4	0	18	77	587 (18.81)	85	767
5	0	7	16	134	489 (15.67)	646
Total	42	846	805	838	589	3,120 (100.00)

ESI = Emergency Severity Index.

Table 3 shows the meta-regression analysis based on the method of moments for moderators (distance from ESI origin, year of publication and ESI version). Studies using the latest version of the ESI scale and which had been published more recently showed significantly higher pooled coefficients. However, higher pooled coefficients were not indicated for studies conducted closest in geographical distance to Boston, USA [Figure 3].

Table 3:

Meta-regression of Fisher’s Z-transformed kappa coefficients on predictor variables^*

Independent variable	B	SEb	P
Latest ESI version	0.302	0.018	0.00
Distance from ESI origin**	−0.00	0.000	0.53
Publication year	0.015	0.002	0.00

SEb = simultaneous equation bias; ESI = Emergency Severity Index.

^*Using studies relating to ESI reliability with weighted kappa coefficients.2,3,5–10,24–34

^**The ESI triage scale originated in Boston, Massachusetts, USA.

Figure 3:

Fisher’s Z-transformation of kappa coefficients regarding the year of publication among studies relating to the reliability of Emergency Severity Index.2,3,5–10,24–34

Discussion

The results of this study indicate that the overall reliability of the ESI triage scale is substantial. The ESI showed an acceptable level of reliability which guarantees consistent decisions regarding the allocation of patients to appropriate categories; thereby supporting evidence-based practice in EDs.17,35

In the ESI, levels 3 to 5 are generally defined by the requirement and availability of resources; because this varies from one setting to another, most disagreements are to do with these levels. This was reflected in the current study. Fortunately, these levels indicate semi- to non-urgent patients which means that misclassifications rarely occur among those who are critically ill. Another interesting observation of the current meta-analysis was the strong tendency towards level 2 categorisation. Although this may prevent the under-triaging of certain patients, it could also create a significant disturbance in the patient flow and interfere with the overall functioning of the ED.

Diverse pooled reliability coefficients were observed regarding participants, patients, raters, reliability methods and statistics among the studies. The results of the meta-analysis found that agreement with the latest and the adult versions of the ESI and among experts was higher than those with previous or paediatric versions and among the other groups of raters. The different results determined by these moderator variables could lead to further studies to explore these variables in more depth. The reliability and consistency of the ESI across EDs in different countries has been documented and supported by scientific evidence.35 This could be due to the fact that the simplicity and objectivity of the ESI algorithm plays a pivotal role in helping clinicians reach optimal agreement.36 In the current study, the analysis of reliability in studies of non-American origin show that the ESI triage scale can be adopted successfully in countries outside of the USA in spite of cultural differences.

Studies using the latest version of the ESI scale and those which were published more recently showed higher agreement. As the ESI triage scale has been updated several times and its reliability has improved over the years, this indicates that revisions have been effective. Additionally, this emphasises the need for EDs to update their triage systems according to the latest versions of the chosen triage scale.

In general, intra-rater reliability is more satisfactory than inter-rater reliability.37 This was highlighted in the current study, which revealed almost perfect agreement in the former as compared to the substantial agreement yielded by the latter. While intra- and inter-rater reliability are intended to report the degree to which measurements taken by the same and different observers are similar, other methods of examining reliability have remained uncommon in studies regarding triage reliability.38

The current study’s analysis demonstrated that the weighted kappa coefficient showed substantial agreement. In fact, the weighted kappa coefficient reveals higher reliability than the unweighted kappa coefficient because it places more emphasis on the larger differences between ratings than on the smaller ones.39 In practical terms, the misallocation of critically-ill patients by even a single ESI level can endanger their clinical outcomes; unweighted kappa statistics therefore provide a more realistic estimation of triage scale reliability.11

A number of limitations of this study must be noted. While the ESI showed an acceptable level of reliability, it is important to remember that there is a considerable gap between research and clinical practice even at the best of times.40 In addition, almost all of the studies used weighted kappa statistics to report reliability coefficients. As weighted kappa statistics generally overestimate the reliability of a triage scale, it is necessary to interpret these results with caution.11 Therefore, it is likely that the ESI is in fact only moderately reliable. Furthermore, none of the studies in the analysis reported raw agreement for each individual ESI level and only a few studies presented a contingency table for inter-rater agreement. Since this study was limited to overall reliability, some inconsistencies may exist across each ESI level. Finally, research has indicated that raters’ experiences could affect the reliability and validity of triage decision-making; these experiences were not reported among any of the studies in this analysis.41

Conclusion

Overall, the ESI triage scale was shown to display an acceptable level of reliability in this meta-analysis. However, there is a need for further development of the scale in order to reach almost perfect agreement. The reliability of triage scales requires a more comprehensive approach, including a thorough assessment of all aspects of reliability. In light of this, further studies should concentrate on the reliability of triage scales in terms of specific moderator variables, such as the version of the ESI used.