Validity and Reliability of the Emergency Severity Index and Australasian Triage System in Pediatric Emergency Care of Mofid Children's Hospital in Iran

Abstract

Objective:

To evaluate the validity, reliability, sensitivity, and specificity of the Emergency Severity Index (ESI) and Australasian Triage System (ATS) for children visiting admitted to the emergency department (ED).

Methods:

This was a prospective study occurred in the Mofid children's Hospital in Iran from August 2017 to November 2018 and children had aged ≤14 years and presented at the ED with a medical symptom were considered eligible for participation. This study was divided into two phases: in the first phase, we determined the inter-rater reliability of ESI version 4 and ATS by triage nurses and pediatric residents. In the second phase, to analyze the validity, sensitivity, and specificity of each triage system. Reliability and agreement rates were measured using kappa statistics.

Results:

ESI showed inter-rater reliability with kappa of 0.65–0.92 (P<0.001) and ATS showed inter-rater reliability with kappa of 0.51–0.87 ESI had sensitivity ranged from 81% to 95% and specificity ranged from 73% to 86%. In addition, sensitivity ranged of the ATS were 80% to 95% and specificity ranged from 74% to 87%. Under triage and over triage occurred in 12% and 15% of patients respectively in ESI and 13% and 15% of patients respectively in ATS.

Conclusion:

The ESI and ATS both valid to triage children in the ED section of Mofid children's Hospital paediatric. Reliability of the ESI is good, moderate to good for the ATS.

Introduction

Patients with non-urgent problems increasingly call emergency departments (ED) in hospital [1]. This leads to overcrowded of waiting rooms and then patients demanding care urgently may not be treated on time [1]. Therefore, ED needs a valid triage system to classify patients in some category related to certain waiting time including need immediate attention, wait for a longer time or not need emergency care [2, 3]. An advanced and evaluative triage system is important because personal triage by nurses has low competence without using a good system [4].

Different triage systems have been worked in the past years. Among them, the Emergency Severity Index (ESI) and the Australian triage scale (ATS) are the most widely used [5, 6]. The sensitivity and specificity of ATS and ESI system has been reported in specific subgroups of adults and was shown to be validating in classify patients [5, 7, 8]. Triage of pediatric patients is challengeable compared to adults as performing signs and symptoms of children and final determine vary from adult patients [9]. Children are more vulnerable to a dehydration and viral infections, and due to the need for special communication skills, assessing the level of urgency of children’s clinical symptoms is known to be challenging [10].

The reliability and validity of both triage system, ESI and ATS, have been assessed in adult and pediatric populations in many developed countries [11], however, not adequately information on the ESI and ATS reliability and validity in ED of Iran especially in pediatric patients are available. The aim of this study therefore was to assess and compared the reliability and validity of the ESI version 4 and ATS in a prospective cohort of pediatric patients at emergency department of. Mofid children's Hospital in IRAN.

Materials and Methods

 Study design This study was conducted as observational prospective study in single-center, to assed the reliability and validity of ESI and ATS triage systems. The study was performed at the ED of Mofid children's Hospital affiliated with Shahid Beheshti University of Medical Sciences, Tehran. Iran with approximately 45,000 per year visits to ED. Data were collected in all children aged under 14 years old presenting to the ED in Mofid children's Hospital. All patients visiting the ED for a 1-year period (between August. 2017 to November 2018) were evaluated. Cases were selected from all patients received on 3 random days of every week using a computer-based randomization system. In this study, patients presented with aged upper 14 years old and patients who could not be followed and the ones who left the ED before being visited by a physician were excluded from the study. In this study informed consent was obtained from all the participants and the medical ethics committee of in Mofid children's hospital in Iran approved this study.

Triage system

 Each triage system used physiological parameters including body temperature, heart rate, respiratory rate, oxygen saturation, blood pressure, and work of breathing in prediction of illness's severity. ESI developed in the United States and in the fourth version of ESI, fever added to assessment urgency of children triage system. According to Table1, subject added in level 1 to level 5 in ESI and ATS triage system.

Table 1.

Five level of ESI and ATS triage systems in children according to requiring time interventions

	ESI	ATS
Level 1	requiring immediate life-saving interventions	requiring immediately interventions
Level 2	high risk conditions, who are confused, lethargic, disoriented, having severe pain, distress and highly abnormal vital signs	requiring 10 minutes interventions
Level 3	requiring two or more resources	requiring 30 minutes interventions
Level 4	requiring one resource	requiring 60 minutes interventions
Level 5	no resources are expected to be required	requiring 120 minutes interventions

Study protocol

 This study was conducted in two phases. The first phase of the study was to assessment the inter-rater reliability between ESI and ATS triage system. Inter-rater reliability between pediatric triage nurses and pediatric residents, among pediatric residents and between the nurses were measured using Kappa statistics. Raters were members of an experienced team including PEM physicians and PT nurses who had been working together for at ED. All pediatric triage nurses and pediatric residents in the study have been formally trained how to apply all triage systems. The triage nurses had at least 2 years of experience in the ED. The second phase of the study was to determine the validity sensitivity, and specificity of ESI and ATS triage system. Validity was measured using area under the receiver operating characteristics ROC curves.

Data analysis

 Demographic information including age and were documented. All statistical analyses were performed using SPSS version 19.0 (IBM corporation, Armonk, New York). Frequency and percentages were used for quantitative variables, while mean and standard deviation were used for qualitative data. Pearson correlation coefficient and kappa statistics were used for the inter-rater reliability. The inter-rater reliability was characterized using interclass correlation coefficients, between PT nurses and PEM physicians. The relationship between each ESI level and admission site was evaluated using the chi-square test and the relationship between the admission status and related triage level was assessed using Spearman’s correlation coefficient. Sensitivity, the ability of triage nurse to correctly assign patients to the appropriate ESI designation and specificity, the ability to correctly not assign to an ESI level, were measured.

Results

A total of 1350 patients were enrolled during the study period. We excluded 50 patients due to incomplete record and exclusion criteria. Thus, 1300 participants with age of fewer than 14 years were included for validation analyses, of which 47.9% were male and 51.9% were female. ESI showed inter-rater reliability with kappa of 0.65–0.92 (p<0.01) and ATS showed inter-rater reliability with kappa of 0.51–0.87. (p<0.01). These results were shown in Table 2 and 3.

Table 2.

Inter-rater reliability of. Australian triage scale (ATS)

	Internal consistency spearman's r coefficient	Test-Retest Intra-class correlation	Internal-Rater Agreement kappa Cohen's
Level 1	0.870	0.88	0.833SEM: 0.050
Level 2	0.857	0.85	0.777SEM: 0.064
Level 3	0.66	0.67	0.520SEM: 0.105
Level 4	0.93	0.76	0.850SEM: 0.048
Level 5	0.94	0.89	0.883SEM: 0.043

Table 3.

Inter-rater reliability of Emergency severity index. (ESI)

	Internal consistency spearman's r coefficient	Test-Retest Intra-class correlation	Internal-Rater Agreement kappa Cohen's
Level 1	0.89	0.879	0.833SEM: 0.050
Level 2	0.851	0.84	0.777SEM: 0.064
Level 3	0.650	0.682	0.650SEM: 0.105
Level 4	0920	0.751	0.850SEM: 0.048
Level 5	0.941	0.889	0.923SEM: 0.043

The result of this study showed that ESI illustrated the most appropriate predicting ability for admission, i.e., AUC 0.88 (95%CI 0.64–0.94), ESI had sensitivity ranged from 81% to 95% and specificity ranged from 73% to 86%. In addition, ATS illustrated AUC 0.77 (95%CI 0.65–0.84), sensitivity ranged of the ATS were 80% to 95% and specificity ranged from 74% to 87%. The second phase determined the predicting ability of each triage system using area under the receiver operating characteristics curves (AUC), sensitivity and specificity as shown in Table 4.

Table 4.

Sensitivity and specificity of ESI and ATS

	ESI		ATS
	Sensitivity	Specificity	Sensitivity	Specificity
Level 1	95. 92%(85.85% to 98.94%)	73.77 %(60.93% to 84.20%)	95.12%(84.85% to 97.14%)	84.85 %(64.94% to 81.21%)
Level 2	90.83%(79.17% to 96.18%)	83.61 %(71.91% to 91.85%)	91.53%(47.18% to 90.15%)	74.51 %(61.61% to 81.65%)
Level 3	82.90%(63.06% to 89.15%)	89.83 %(79.17% to 96.18%)	83.50%(32.01% to 57.12%)	79.63 %(59.57% to 76.11%)
Level 4	81.28%(55.00% to 75.89%)	85.50 %(55.36% to 77.31%)	80.21%(54.00% to 78.69%)	77.80 %(54.36% to 79.38%)
Level 5	90.51%(61.32% to 87.19%)	86.87 %(67.15% to 80.83%)	91.53%(81.32% to 77.19%)	87.67 % (67.15% to 88.83%)

Discussion

The aim of this study was to establish the reliability validity of the ESI v4 and ATS in our pediatric emergency department when used by an experienced team of PEM physicians and PT nurses. In the first phase, the performance of PT nurses was acceptable; in the second phase, the ESI v4 and ATS were found to be a reliable and valid tool in the ED triage section.

Among all triage systems, 4 triage systems including ESI, ATS, MTS and CATS are largely used in developed and developing countries to triage children in ED [5]. The reliability and validity of these systems in children were performed by previous studies [8, 12, 13]. The best triage was the one that could precisely distinguish urgency of treatment, caused to decline morbidity and mortality of patients as well as decreases overcrowded of ED in hospital [8, 12, 13]. The psychological aspects defined in detail in previous studies report that the use of these triage tools could not be approved in a computerized way when it comes to decision-making in serious condition such as the ones encountered in the pediatric emergency department [10]. Triage of children in the ED challenges unlike physiologic variations that make communication, assessment and identification of serious illnesses or injury quite different compared with adult triage [14].

The results of this study showed the ESI (kappa 0.65–0.92) had well and ATS (kappa 0.51–0.87) had moderate reliability for children triage in ED. The ESI v4 is a simple triage tool but proved to be more reliable than previous three-level ones. The first version of ESI only used for adults, while pediatric vital signs such as fever were added in ESI version 4, which is designed to include triage patients of any age group [15-17]. Study on reliability of triage systems in children using inter-rater agreement analyzed by Cohen's kappa which intended poor, fair, moderate, good and very good if K<0.20, 0.21to 0.40, 0.40 to 0.60, 0.61 to 0.80 and >0.80 respectively [18]. According to mentioned range of kappa value, Travers et al., [19], Aeimchanbanjong et al., [13], and Green et al., [15] reported that ESI has very good reliability. In addition, Aeimchanbanjong et al., [13] and Sanjay et al., [20] reported that ATS has poor to moderate reliability. In addition, Mirjam van Veen showed that ESI has moderate to good reliability and ATS had poor reliability [5]. Several other studies have also showed moderate to high reliability using the ESI v.4 in children [15, 21, 22]. Results from pervious study showed that agreement level between nurses for use the triage system to children demonstrations was only moderate to poor and appears to be lower than the reliability with which it is useful to adult presentations [23, 24].

ESI had sensitivity ranged from 81% to 95% and specificity ranged from 73% to 86%. In addition, ATS had sensitivity ranged of80% to 95% and specificity ranged from 74% to 87%. The aim of triage is to identify high urgent patients. Triage systems which had high percentage of under triage or low sensitivity (real high urgent patients are triaged as low urgent) are unsafe [5]. In triage system reach to 100% sensitivity and specificity are difficult, but a good balance between over and under-triage in triage system is important. A high sensitivity decreases specificity of triage system (real low urgent patients are triaged as high urgent) resulting long waiting times for real high urgent patients [5] Storm-Versloot et al. reported that ESI had low sensitivity and high specificity and then showed highest percentage of under triage [25].

The parameters used for validity of children triage in ED are different and including hospitalization, ICU admission, length of stay in ED and the cost of ED consultation [15, 26]. Therefore, a comparison between children triage systems in ED could not be made on how they anticipate the good urgency [15, 26]. There is association between urgency and admission and previous studies reported that triage system could predict the admission [13]. Green et al., reported that admission rate used in ESI triage system for measurement of validity [15] in addition, Sittichanbuncha et al., [27] and Aeimchanbanjong et al., [13] showed the other aspect of outcome measurement including vital signs, diagnosis, resource use, admission rate, and follow-up by comparison of triage systems such as ESI, ATS, MTS, RTS and CTAS and found that triage systems had equally valid.

In overall, the results of our study showed that ESI expressed the best validity followed by ATS, based on outcome measurement by using triage urgency for prediction of admission respectively.Considering this study and previous research works, we can conclude that the ESI and ATS has proved to be reliable in pediatric patients.

Conflict of Interest:

None declared.