Inter-rater Reliability of Triages Performed by the Electronic Triage System

Abstract

Objective:

To examine the inter-rater reliability of triages performed by the Electronic Triage System (ETS) which has recently developed and used in hospital emergency department (ED).

Methods:

This cross-sectional study was conducted prospectively and studied 408 visitors of Tabriz Imam Reza hospital’s ED. The variables of interest were age, sex, nurse-assigned triage category, physician-assigned triage category, disease type (trauma, non-trauma), and the referred room within the ED. Cohen’s un-weighted kappa, linear weighted kappa, and quadratic weighted kappa were used to describe the reliability.

Results:

Un-weighted kappa observed to be 0.186 (95% CI: 0.123-0.249). Linear weighted kappa observed as 0.317 (95% CI: 0.251-0.384) and quadratic weighted kappa as 0.462 (95% CI: 0.336-0.589). In general, low agreement was seen between the triage nurses and ED physicians. For trauma patients and for those who were referred to the cardiopulmonary resuscitation room (CPR), all three types of kappa were higher than other visitors of the ED.

Conclusion:

Inter-rater reliability of the triages performed by the ETS observed as ranging from poor to moderate. Implementing interventions that would create a common language between nurses and physicians about the triage of the ED visitors seems necessary. The more agreement on the triage of trauma and CPR patients might be due to their condition and the more attention to them.

Introduction

At the emergency department (ED) of the hospital the triage is used to prioritize the visitors. The primary purpose of the triage is to identify those who their waiting for receiving care may lead to damage, loss of organ or even death [1]. Thus reliability of the triage is important for patient safety [2]. In an ideal situation it’s expected that if different people perform triage on same patients with same triage system, they would arrive at the same triage category [3]. But the situation is not always like this and sometimes there seems disagreement about the triage category of the patient [4].

The Persian translation of the Emergency Severity Index- ESI triage system [5] is in use in the hospital EDs of Iran [6] and its electronic version has recently been developed and is called the Electronic Triage System (ETS) [7]. Validity of the triages performed by the ETS is proved in predicting patient outcomes [8] and the reliability of them needs to be investigated. This study had been conducted with the aim of examining the inter-rater reliability of those triage ratings. The reliability was calculated by comparison of triage categories assigned by the triage nurses and ED physicians and reported by the Kappa statistics.

Materials and Methods

Study population

This cross-sectional study was conducted prospectively during a one-week period in March 2013 in Tabriz Imam Reza hospital. The institutional review board (IRB), which is the Ethics Committee of Tabriz University of Medical Sciences, reviewed and approved the study protocol. The understudied ED is located in a tertiary care educational hospital and the triage is performed by the nurses for all who seek care from the ED. The ESI triage system was at work in this ED since 5 years ago and the triage nurses attended a two-hour retraining course two months prior to the study. They did not attend any other educational program on ESI triage during the last year before the study. In this teaching hospital patients at the ED are first visited by the Emergency Medicine residents (the first year, second year, and third year residents).Attends visit patients if it is needed and also in the periodic rounds during the work shift. Since the specialty hospitals of obstetrics, pediatrics, heart, psychiatry, and burnt exist in the city, these case are not seen in this ED except in complicated cases.

 Study protocol

The gathered data included the triage category that nurses were assigned to the patients by the ETS, physician-assigned triage category, type of complaint(trauma, non-trauma), the treatment room within the ED (CPR room, trauma room, internal medicine room, and the fast track room), age and gender of the patients. To assess the reliability we used the inter-rater reliability and the nurse-assigned triage categories were compared against the physician-assigned triage category. To achieve this purpose immediately after the physicians visited the patients, their opinion on the triage category of the patients were asked by the observers located in different rooms of the ED. All the observers were trained how to perform their duty and were present at the ED 24 hours a day during the study period.

All the visitors to the ED were potentially eligible to be included in the study. Of the 491 patients that were triaged by the ETS, 83 were excluded due to unavailability of physician-assigned triage category leaving 408 individuals for the analysis. Lack of physician-assigned triage category was due to that some physicians refused to participate in the study or the patient left the ED without being seen.

 Statistical Analysis

To analyze the data we used Cohen’s un-weighted kappa, linear weighted kappa, and quadratic weighted kappa. The analysis was performed using Stata 10 software and 95% confidence interval. Independent sample t-test was used to compare the mean age between included and excluded patients. Chi-square test was used to compare the proportions. Kappa can range from -1 to 1 of which the negative values show disagreement and positive values show agreement [9]. If kappa equals 1, it shows the perfect agreement. If it is in the range of 0 to 0.2 is considered as poor, 0.2 to 0.4 is considered as fair, 0.4 to 0.6 is considered as moderate, 0.6 to 0.8 as good, and if it is in the range of 0.8 to 1 is shows very good agreement between the raters [10]. A 2-sided p-value of less than 0.05 was considered statistically significant.

Results

Of the 408 ED visitors who included in this study, 162 (40%) were women. The mean age of the patients was 42.6±21.4 years. The average age of the excluded patients was 44.5±21.4and 40% of them were women. There was no statistically significant difference in age and gender of the included and excluded patients. Comparison of the triage category assigned by the nurses and the physicians is presented in Table1. As it can be seen, there is little agreement between the nurses and the physicians on the triage category of the patients.

Table 1.

Comparison of the nurse-assigned and the physician assigned ESI triage category in triages performed by the Electronic Triage System

Nurse-assigned triage	Physician-assigned triage category
	1	2	3	4	5	Total
1	8	7	4	0	0	19
2	2	15	31	7	2	57
3	0	27	64	25	2	118
4	2	19	72	89	32	214
5	0	0	0	0	0	0
Total	12	68	171	121	36	408

The un-weighted kappa was 0.186 (95% CI: 0.123-0.249) (Standard Error: 0.032). Linear weighted kappa was 0.317 (95% CI: 0.251-0.384) (Standard Error: 0.034), and the quadratic weighted kappa was 0.462 (95% CI: 0.336-0.589) (Standard Error: 0.064). The un-weighted, linear weighted and quadratic weighted kappa was also calculated separately for the trauma and non-trauma patients and for different treatment rooms within the ED. The results of this calculation are presented in Table 2. As it can be seen in the table, all three types of kappa were higher for the trauma patients than non-trauma ones. The agreement between nurses and physicians was higher for those patients who were referred to the CPR and trauma rooms.

Table 2.

Inter-rater reliability of triages performed by the ETS^a, divided by trauma and non-trauma patients and the treatment area within the ED^b

Group		Kapp a	LWK c	QWK d
Trauma		0.231(95% CIe: 0.11-0.35)	0.372(95% CIe: 0.25-0.48)	0.509(95% CIe: 0.26-0.75)
Non-trauma		0.163(95% CIe: 0.08-0.24)	0.288(95% CIe: 0.20-0.36)	0.435(95% CIe: 0.29-0.57)
Treatment Area	CPR f room	0.250(95% CIe: 0-0.61)	0.261(95% CIe: 0-0.54)	0.232(95% CIe: ncg)
	Internal Medicine	0.069(95% CIe: 0-0.22)	0.113(95% CIe: 0-0.25)	0.173(95% CIe: ncg)
	Trauma room	0.209(95% CIe: 0-0.42)	0.286(95% CIe: 0.07-0.49)	0.352(95% CIe: ncg)
	Fast track	0.012(95% CIe: 0-0.11)	0.060(95% CIe: 0-0.13)	0.119(95% CIe: ncg)

^a ETS: Electronic Triage System;

^bED: emergency department;

^cLWK: linear weighted kappa;

^dQWK: quadratic weighted kappa;

^eCI: confidence interval;

^fCPR: cardiopulmonary resuscitation;

^gNC: not calculated

Discussion

Reliability is one of the most important pre-requisites of every tool. In case of triage it is the reproducibility of the results in assessment by different people. Findings of this study showed that there is little agreement between the triage nurses and ED physicians on the triage category of the patients. Kappa, which is used as a measure for inter-rater reliability of triages performed by the ETS, was in the range of poor to moderate agreement.

In this study the un-weighted kappa was in the range of poor to fair agreement, the linear weighted kappa was in the range of fair agreement and the quadratic weighted kappa was in the range of poor to moderate. Previous studies also reported the Weighted Kappa higher than the un-weighted one [3] which are due to the weighting in the calculation of the kappa. A similar study on the ESI triage in Tehran had reported the linear weighted kappa as 0.87 (very good agreement). While in that study the triage nurses were attended a retraining course on the ESI triage [11].Attending such retraining course could elevate the reliability of the triage [12].We studied the reliability of the triages that were performed by the ETS, which is based on the ESI triage system, and found poor to moderate reliability. A study on the computerized version of the Canadian Triage and Acuity Scale (CTAS) reported a linear weighted kappa of 0.52 and quadratic weighted kappa of 0.66 (moderate to good agreement) [13]. Another study on an electronic triage system reported a weighted kappa of 0.87 [14]. In both of these mentioned studies the value of kappa is bigger than what we observed in our study. Yet we should keep in mind that the reliability of the ETS had not considerable difference with the routine ESI triage at the under-studied ED [15]. So it is hard to attribute the low reliability of the triages to the ETS. It seems that a common language is missing about the triage category of the ED patients between the nurses and the physicians. Thus interventions such as retraining courses for both physicians and nurses can be helpful.

We found all three types of studied kappa to be higher for the traumatic patients than the non-trauma ones. This may be due to their clinical condition that necessitates more attention of the triage nurses. The analysis by the treatment area within ED showed greater reliability for the CPR and trauma rooms. This finding enforces the hypothesis of more attention to traumatic and acute patients. Yet more studies are needed to examine this hypothesis.

A limitation of this study was exclusion of nearly 17% of the ED visitors from the study which was either due to non-participation of the physicians or the patients left the ED without being seen. Although this rate seems high, it is general conditions which exist in many other hospitals and studies have shown a discharge against medical advice from the ED of 20% [16].

In conclusion, the inter-rater reliability of the triages performed by the ETS was observed as poor to moderate. It shows a need for attaining a common language between the triage nurses and ED physicians on the triage category of the patients. The agreement on the triage of traumatic patients and those who were referred to the trauma room and CPR room was higher than other patients. This might be due to their clinical condition which requires more attention from the triage nurses.

Conflict of interest: None declared.