Effect of a triage course on quality of rating triage codes in a group of university nursing students:a before-after observational study

Abstract

BACKGROUND:

Most current triage tools have been tested among hospital nurses groups but there are not similar studies in university setting. In this study we analyzed if a course on a new four-level triage model, triage emergency method (TEM), could improve the quality of rating in a group of nursing students.

METHODS:

This observational study was conducted with paper scenarios at the University of Parma, Italy. Fifty students were assigned a triage level to 105 paper scenarios before and after a course on triage and TEM. We used weighted kappa statistics to measure the inter-rater reliability of TEM and assessed its validity by comparing the students’ predictions with the triage code rating of a reference standard (a panel of five experts in the new triage method).

RESULTS:

Inter-rater reliability was K=0.42 (95%CI: 0.37–0.46) before the course on TEM, and K=0.61 (95%CI: 0.56–0.67) after. The accuracy of students’ triage rating for the reference standard triage code was good: 81% (95%CI: 71–90). After the TEM course, the proportion of cases assigned to each acuity triage level was similar for the student group and the panel of experts.

CONCLUSION:

Among the group of nursing students, a brief course on triage and on a new in-hospital triage method seems to improve the quality of rating codes. The new triage method shows good inter-rater reliability for rating triage acuity and good accuracy in predicting the triage code rating of the reference standard.

INTRODUCTION

Triage is the first assessment and sorting process used to prioritize patients arriving in the emergency department (ED). The most common triage systems are traffic director, spot-check, and comprehensive triage.[1] Most current triage tools are based on a categorical measurement acuity scale and are three-, four-, or five-level systems. The Australasian Triage Scale,[2] the Canadian Triage and Acuity Scale (CTAS),[3] the Manchester Triage System (MTS)[4] and the Emergency Severity Index (ESI)[5–8] are all five-level triage tools. Because Italian guidelines require a four-level in-hospital triage based on an acuity scale measurement,[9] we cannot use five-level triage systems in our hospitals. Consequently, we devised a new four-level triage system, the triage emergency method (TEM)[10] to fulfill Italian requirements.

In a previous pilot study, TEM had shown good inter- and intra-rater reliability for rating triage acuity and for accuracy in patient admission prediction.[10] The TEM contains a flowchart (Figure 1) and a table (Table 1). As shown in Figure 1, this new system is based on an acuity scale, resources used, the time the patient waits to be evaluated, and nursing procedures. It has four urgency categories (UC): 1 (Red), immediate assessment; 2 (Yellow), assessment within 20 minutes; 3 (Green), assessment within 60 minutes; and 4 (White), assessment within 120 minutes.

Figure 1.

Triage emergency method algorithm (TEM).

Table 1.

Triage emergency method (TEM) patient’s chief complaints

To our knowledge, there are many quality studies on triage methods in hospital setting[2–8] but there are not similar studies in university setting. This study aimed to test if a course on triage and on a new four-level triage model, triage emergency method (TEM), could improve the quality of rating triage code in a group of nursing students.

METHODS

Study design and setting

This is an observational study performed using triage scenarios from June to September in 2010. Fundamental concepts of triage and the new triage emergency method (Figure 1) were shown, during a five-hour course, to 50 students enrolled in the third year of the nursing program at the University of Parma, Italy.

Data collection

We used 105 triage scenarios from the database used in previous studies.[10–12] The paper triage scenarios were created using the medical records of patients admitted to the emergency department of Imola Hospital, Italy.

We recorded the following data: demographic and clinical characteristics, original nurse’s triage category, admission status and site, and the data on triage forms completed by the nurse, namely, presenting complaint, mode and time of arrival, past diseases, vital signs, and pain score. Each case given to the study participants included the patient’s age and gender, presenting complaint, a brief case scenario with mode and time of arrival, past diseases, vital signs and pain score.

Study participants

Seventy students of the third year of the nursing program at the University of Parma were assigned to undergo a five-hour training in triage and TEM. Twenty were excluded for missing data in their assignment. of them Four senior triage nurses and one doctor who had emergency nursing and teaching triage certification, more than 15 years of emergency nursing experience, and a prolonged training in the new triage method (TEM) comprised the panel of triage experts. The panel independently assigned, using the TEM, triage scores to the 105 scenarios. Their triage codes were the reference standard (RS) for the triage level in this study. They were blinded to the triage category assigned both by the original triage nurse and by the students involved in this study.

The students enrolled in the study completed a questionnaire about their demographics, education, and work experience.

Study protocol

In June 2010, a five-hour course on triage (one hour) and TEM (four hours) was performed at University of Parma. Each participant independently assigned triage scores to the 105 scenarios before the course, at time zero (T0) and three months after the course (T1). The students, at time zero, did not use any triage method; at T1, after the course, they used the TEM. To prevent communication between participants, the students assigned triage codes in different rooms and in the presence of two investigators. The triage scenarios were given randomly to the participants. During the second test, three months after course, students could consult the TEM flowchart (Figure 1 and Table 1). The data were collected and entered on a spreadsheet by an investigator who was blinded to the aim of the study. The student group remained concealed during data entry and analysis.

Being a quality assurance investigation, our institution exempted the study from formal review. Anyway the protocol for the research project has been approved by a committee of professors of the institution within which the work was undertaken, and it conforms to the provisions of the Declaration of Helsinki.

The patients and nurses involved in the study gave informed consent and permission to access their data.

Data analysis

To test the quality of rating triage code in the group of participants, we choose the inter-rater reliability before and after the course and we assessed the validity of TEM.

Reliability was measured with weighted kappa (K) by comparing the triage nurses’ rating (inter-rater) at T0 and T1. We also measured the inter-rater reliability between the group and its reference standard by measuring the weighted kappa (K) against the urgency category assigned by the panel of triage experts. According to the literature,[13] we considered poor agreement a K value between 0.00 to 0.20, fair-moderate agreement a K value between 0.20 and 0.60, good agreement a K value between 0.60 and 0.80, and very good agreement a K value between 0.80 and 1.

We evaluated the validity of TEM by calculating sensitivity and specificity for prediction of the reference standard’s triage score and for prediction of the patient admission.

To analyze the predictive validity for patient admission and for reference standards triage score, for each scenario we considered the mode of the urgency category assigned by the nurses and we used this code in all validity calculations. We evaluated the validity by calculating sensitivity and specificity for prediction of patient admission and reference standards triage score, using the following cut-offs: true codes 1 and 2: patient sick and likely to be admitted; true codes 3 and 4: less urgent and patient likely to be discharged.

We calculated participant and scenarios sample size according to Worster et al,[14] anticipating a K value of approximately 0.80 from previous studies and a standard error of 0.05. Statistical significance was tested at an alpha level of 0.05. We used the STATA v9.2 software (Statacorp, Texas, USA) for statistical analysis. As a quality assurance investigation, the study was exempt from formal reviews. The patients and nurses involved in the study gave permission to access their data.

RESULTS

Of the 105 patients included in triage scenarios, 46 (44%) were women and their mean age was 39.2 years (27.5±SD). The most frequent symptoms were minor trauma (19%) and pain (14%). Twelve hospital admissions were recorded: eleven in non-intensive wards and one in intensive care units. The group of fifty students had a media age of 24 years (SD=3.04). There were 29 (58%) women. The knowledge of triage models was limited: two out of the fifty (4%) had attended a brief triage course and nine (18%) had previous triage training in ED.

The rate of urgency categories 1 and 3 assigned to each scenario was similar between the triage panel experts and the students, but the students assigned using TEM more UC 2 and fewer UC 4 compared to the triage panel experts (Figure 2).

Figure 2.

Urgency category assigned by students and experted nurses.

After the TEM course, the nursing students had overtriage and undertriage rates of 6.7% and 7%, respectively, (7/105 and 8/105 scenarios) compared with the reference standard rating.

The rate of disagreement in assigning UC among the students was different before and after the TEM course. Complete disagreement (when nurses of the same group assigned to the same scenario triage codes that differed in more than two priority levels, e.g., one nurse assigned “white” (level 4), and another, “red” (level 1) occurred in 98% and 64% of scenarios evaluated before and after the TEM course, respectively. There no complete agreement (when all the fifty students enrolled the same triage code was assigned) before or after the course (Table 2).

Table 2.

Comparison of inter-rater reliability of the students before and after the TEM course

The inter-rater reliability among the students was improved using TEM: K=0.42 (95%CI: 0.37–0.46), before the course, and K=0.61 (95%CI: 0.56–0.67), after the course.

The inter-rater reliability among the students using the new triage model and the triage score of their reference standard was K=0.64 (95%CI: 0.51–0.77). The sensitivity and specificity of triage rating of the students for prediction of reference standard’s triage code and admission status were good (Table 3). There were no in-hospital deaths in the patients of the triage scenarios.

Table 3.

The sensitivity and specificity of the students using TEM to predict the reference standard’s triage code and admission status

DISCUSSION

Among the group of nursing students, a brief course on triage and a new in-hospital triage method seems to improve the quality of rating triage code.

The new in-hospital triage method, TEM, shows good inter-rater reliability for rating triage acuity and good accuracy in predicting the triage code rating of the reference standard. Its performance was also good despite students’ lack of experience in triage practice and in using the new triage model. In addition, unlike other Italian triage models, TEM has the advantage of predicting resource utilization in the emergency department. Moreover, it could be easy to teach, learn and consult for a group of inexpert nursing students: a brief course seems to improve its performance.

Many studies have evaluated the reliability and validity of acuity ratings by triage nurses,[5–7,10–14] probably because a triage scale should meet at least these two criteria to perform accurately as intended.[13] But to our knowledge there are few studies on quality indicators (as reliability and validity) of the main triage in-hospital methods among a group of nursing students at an university.

The new triage tool, TEM, is a four-level acuity triage system which showed good inter-rater and intra-rater reliability of triage assessments in a previous pilot study.[10] The new triage tool, like ESI v4, is used to predict resource needs. However, the TEM algorithm has a different structure: the first step includes “modification of vital signs” and “very high-risk situation”; the second step includes “modified vital signs” (Figure 1). TEM also has a more detailed list of clinical situations set out in a table linked to the algorithm (Table 1). Finally, the assignment of a lower acuity level is based not only on the prediction of resources but also on the prediction of the time the patient can wait to be evaluated (step 3 of the algorithm in Figure 1).

In our study, the inter-rater reliability of triage assessments of fifty students with limited knowledge of triage models using TEM after only five hours of training was as good as that of the a panel of triage experts (K=0.61 and K=0.74, respectively). The inter-rater reliability of the students was fair-moderate before the TEM course. TEM also has a good inter-rater reliability with the reference standard’s reliability: K=0.64 (95%CI: 0.51–0.77).

To our knowledge, ours is the first study that measures the inter-rater reliability of a four-level triage system in a group of nursing students. Moreover, there are limited data on triage-tool performance among nursing students.

The group of students who used TEM proved accurate in predicting the reference standard’s triage code and the admission status: accuracy=81% (95%CI: 71–90) and 87% (95%CI: 78–95) respectively (Table 3). Few previous studies used a reference standard to test the validity of a triage system.[12,15,16] However, it’s very difficult to establish validity criteria for triage acuity classification in the absence of a clear reference standard. For this reason, we tried to develop a surrogate “gold standard” based on a panel consensus, and we tested the predictive validity of our triage system against this gold standard.

It is difficult to compare our results on validity with previous studies because of the differences in the setting and in the type of triage system (five levels compared to four levels). Nevertheless, our results on validity and reliability of TEM are similar to previous studies on ESI v4.[6,7]

The main limitation of our study is that it was conducted with paper scenarios, not with patients; however this procedure has been validated in other studies on the inter-rater reliability of triage tools.[2,4,7,8]

Another limitation of the study is that the findings on TEM prediction of the admission status are not sure because in this study there were very few patients admitted and a sensitivity analysis based on few data could be wrong.

Lastly, we evaluated the validity of the triage system based on the accuracy in predicting hospital admission, and hospitalization rates may vary due to factors other than patients’ acuity. The hospital admission rate is not the best outcome to test predictive validity of triage tools because it is a surrogate outcome and there are many confounding variables that could affect it.[17]

In conclusion, this is the first study that measures the reliability and the predictive validity of a four-level acuity triage system in a group of nursing students at an university. It has been interesting to test these outcomes in an university for almost two reasons: 1) a triage system should use sample to understand and to consult by young nursing students without practice experience; 2) all teaching programs should include a practical quality verification. Our data suggest that a brief course on triage and a triage method could improve the quality of rating triage code using triage scenarios. The new TEM seems to have good inter-rater reliability for rating triage acuity in both the nursing students and the panel of triage experts. It is also accurate in predicting a reference standard’s triage code.