Predictive criterion validity requires strong correlation between triage levels and outcomes

Dear Editor,

The manuscript titled “Evaluation of Emergency Severity Index (ESI) triage quality by nurses and associated factors in Iran” by Razavian et al.[1] was reviewed with great interest. This study investigated the quality of nurses’ triage using the Emergency Severity Index (ESI) method. Nurses determined the patient’s priority level in the software. Then, each case was reviewed by an emergency medicine specialist using the recorded information. In this study, the criteria for assessing triage accuracy were not specified, which necessitates further clarification. It was reported that 82% of triage decisions were accurate (for both nurses and emergency medicine specialist), with a triage error rate of approximately 17%. However, the evaluation criteria for the accuracy of triage decisions and the method used to calculate triage errors were not clearly defined. Therefore, it is essential to provide a detailed explanation regarding the validity of the triage decision-making.

In studies examining triage validity, predictive criterion validity is commonly used. This form of validity refers to the ability of a scale to predict a future outcome or characteristic. Identifying an appropriate outcome—commonly referred to as the gold standard—is crucial. A lack of empirical correlation between triage levels and outcomes can introduce bias, thereby compromising the study’s validity. The primary purpose of triage is to prioritize patient access to care and treatment based on the acuity of their condition. Thus, clinical acuity should serve as the gold standard in triage validation. It is critical that selected outcomes be closely related to this acuity. Outcomes, such as mortality, ICU admission, or coronary care unit (CCU) admission, are strongly associated with clinical severity and are therefore appropriate indicators.[2] Triage should identify this group of critically ill patients in crowded emergency departments (ED). However, many studies are limited by sample size and cannot adequately assess mortality as an outcome. Commonly used indicators for validating levels 1 and 2 include ICU admission, CCU admission, and life-saving interventions. For example, it is expected that patients triaged as levels 3, 4, or 5 should not die in the ED or require intensive care, and any such occurrences would be classified as mistriage (under-triage or false negative). In addition, many studies utilize alternative outcomes, such as hospital admission or length of stay, which can moderately reflect clinical acuity.[3]

The primary aim of triage is typically to identify critically ill patients. However, some patients who are not critically ill may still require hospitalization, which introduces potential bias in outcome assessment. It is expected that patients classified as triage levels 1 through 3 will require hospital admission. Conversely, if patients categorized as levels 4 or 5 are admitted, it may be considered mistriage (under-triage), while discharging patients from levels 1 to 3 may indicate mistriage (over-triage or false positive).[3] Other outcomes, such as resource utilization in the ED, associated costs, and expert opinion, have also been used as gold standards in triage studies. These metrics typically have an inverse relationship with triage level.

Another important consideration in determining valid outcomes is selecting an appropriate prediction time frame. In triage research, this timeframe is usually defined as 6 to 24 hours.[3] Long-term outcomes are often affected by multiple confounding factors, which may compromise the predictive validity of triage assessments. For instance, 7-day or 30-day mortality may be influenced by variables, such as the quality of care in the ED, thereby threatening triage validity. Likewise, a patient’s response to treatment from admission through discharge can act as an intervening variable if long-term outcomes are used.

Agreement between nurses and physicians has been thoroughly reported in numerous studies.[4] The pooled coefficient for the level of agreement between physicians and nurses was substantial, with a value of 0.756 (confidence interval [CI] 95%: 0.659–0.828). However, it is important that this agreement be presented using a contingency table and quantified using the appropriate statistics. The choice of statistical test depends on the type of variable (nominal, ordinal, interval, and ratio) and the number of raters (two or more than two). Since the triage level variable is considered an ordinal variable, kappa is used to assess the reliability of decision-making between two raters, and for more than two raters, kappa designating agreement on relevance (k*)[5] or interclass coefficient (ICC) is used.[6] Therefore, it is imperative that triage validity be reported in a comprehensive and systematic manner in research studies.

In addition, a few points are worth mentioning. First of all, live patient assessment demonstrates a higher inter-rater reliability 0.9 (95% CI = 0.87 to 93) than patients’ records review of 0.76 (95% CI = 0.73 to 0.79).[7] Therefore, it should be noted in studies that review patient records. Second, reliability is a crucial aspect that must be reported in triage studies.[4] It is essential to assess how consistently a triage system assigns the same urgency level to patients, whether by different nurses (inter-rater reliability) or by the same nurse at different times (intra-rater reliability). The intraclass correlation coefficient (ICC) is a statistical measure used to assess the reliability of measurements made by multiple raters, essentially quantifying the degree of agreement between them.[4] The third refers to the nature of retrospective studies, which can be associated with information bias. It is not clear to what extent the triage nurse followed the ESI triage criteria while a triage level has been allocated to a patient in the triage room.

Declaration section:

Ethical approval

Not applicable.

Consent for publication

Not applicable.

Availability of data and Materials

Not applicable.

Competing interest

None declare.

Funding

No funding of research.

Author’s contributions

AM contributed to the design concept of the study, performed data collection, contributed to the statistical analysis, wrote the manuscript draft, and critically revised and approved the manuscript.

Conflicts of interest

There are no conflicts of interest.

Funding Statement

Nil.