Reply to “Why should more than two reliability coefficients be reported in triage studies?”

Dear Editor,

We thank Amir Mirhaghi for his valuable comments [1], which provide an opportunity to clarify and further discuss the methodological considerations underlying our study, “An inter-rater reliability study of a modified version of SATS as a prehospital triage tool” [2]. Comparisons between triage systems are challenging [3, 4], and methodological differences between reliability studies may further complicate comparisons across studies [4, 5]. Our study was reported in accordance with the Guidelines for Reporting Reliability and Agreement Studies (GRRAS), which aim to improve the completeness and transparency of reporting in reliability and agreement research [6]. The choice of reliability coefficient was based on statistical literature concerning agreement in ordinal data, whereas GRRAS guided the reporting of the study design, analyses, and results.

In our study, the four triage levels were considered ordinal rather than nominal, as they represent an ordered sequence of clinical priority. Consequently, disagreement between adjacent triage levels represents a smaller ordinal discrepancy than disagreement spanning several levels. Krippendorff’s alpha was therefore calculated using an ordinal distance metric, whereby disagreements between adjacent triage levels contributed less to overall disagreement than disagreements spanning multiple levels. Kappa is commonly used in reliability studies, but previous triage research has demonstrated that reliability estimates may vary depending on the statistical approach and weighting method applied [5].

Our study included 34 raters and 10 missing ratings among 1,020 possible assessments. Krippendorff’s alpha was selected primarily because it accommodates the ordinal measurement level and multiple-rater design within a single framework; its ability to handle missing observations without imputation was an additional practical advantage [7].

Mirhaghi [1] suggests the intraclass correlation coefficient (ICC) or Fleiss’ kappa as complementary coefficients. Although ICC may be useful as a complementary measure when ordinal categories are assumed to approximate a continuous scale, we considered Krippendorff’s alpha preferable because it directly accommodates ordinal measurement without assuming equal intervals between categories [7]. Jakobsson and Westergren [8] further note that ICC estimates may be affected by the number of categories and suggest that ICC is most appropriate for continuous data. Krippendorff’s alpha allowed the ordinal structure of the triage categories to be incorporated directly while accommodating the multiple-rater design.

We agree that reporting a kappa statistic may facilitate comparison with previous triage reliability studies. For this reason, we calculated Fleiss’ kappa as a post hoc supplementary nominal-scale analysis for comparability (Table 1) using R version 4.4.3 (R Core Team, Vienna, Austria). To obtain a complete rating matrix for the supplementary Fleiss’ kappa analysis, the 10 missing ratings were imputed using the modal triage category for each vignette. However, standard Fleiss’ kappa treats the four triage categories as nominal and therefor does not incorporate their ordinal structure [9]. For this reason, it was not considered the primary coefficient.

Table 1.

Agreement coefficients according to number of triage categories and level of measurement. The Fleiss’ kappa analysis was performed after imputing 10 missing ratings (0.98%) using the modal triage category for each vignette. The nominal four-level Krippendorff’s alpha was an additional post hoc explanatory analysis; Krippendorff’s alpha does not require imputation of missing ratings

Analysis	Measurement level	Coefficient	Estimate	Purpose
Four triage levels	Ordinal	Krippendorff’s alpha	0.602	Primary analysis
High/low acuity	Nominal	Krippendorff’s alpha	0.475	Secondary analysis
Four triage levels	Nominal	Krippendorff’s alpha	0.428	Post hoc explanatory analysis
Four triage levels	Nominal	Fleiss’ kappa	0.428	Supplementary post hoc comparability analysis

In the supplementary analyses, Fleiss’ kappa was 0.428. We also performed an additional post hoc explanatory analysis in which the original four-level scale was treated as nominal using Krippendorff´s alpha; this yielded α = 0.428 (Table 1). The supplementary Fleiss’ kappa analysis did thereby not alter the substantive interpretation regarding imperfect inter-rater reliability. As noted by Mirhaghi [1], Zapf et al. [9] found Fleiss’ kappa and Krippendorff’s alpha to provide comparable estimates for complete nominal data. Importantly, however, Zapf et al. [9] recommend Krippendorff’s alpha when the measurement scale is higher than nominal and/or missing observations are present. This corresponds closely to the characteristics of our dataset, in which the four triage levels were treated as ordinal and a small number of ratings were missing.

Mirhaghi [1] also notes the apparent decrease in reliability when the four triage levels were dichotomized into high and low acuity (α = 0.602 vs. 0.475). These estimates were calculated using different measurement metrics: ordinal alpha for the four-level scale and nominal alpha for the dichotomized outcome. When the four triage categories were treated as nominal using the K-Alpha Calculator [10], Krippendorff´s alpha was 0.428. Thus, when the measurement metric was held constant, dichotomization resulted in an increase rather than a decrease in alpha (0.428 to 0.475). This post hoc analysis was performed solely to clarify the apparent discrepancy raised in the commentary and should not be interpreted as replacing the primary ordinal analysis. We thank Mirhaghi for drawing attention to this distinction. This result is consistent with the findings of Jakobsson and Westergren [8], who highlight the challenges of comparing agreement estimates across scales with different numbers of categories.

Comparability of reliability studies is not determined by the choice of statistical coefficient alone. The overall study design, including vignette construction, the type of clinical information provided, characteristics of the raters, and the components of the triage system, may all influence the conditions under which agreement is assessed. In our study, the vignettes were deliberately detailed to allow mobility, the trauma factor, and manual uptriage to be incorporated into the triage assessments [2]. The complete vignettes were published as supplementary material to facilitate interpretation of the results and comparison with future studies [2].

Mirhaghi [1] further suggests that presenting the raw inter-rater classifications would facilitate calculation of alternative reliability coefficients. Although the ratings were presented graphically rather than as a numerical contingency table, all individual ratings included in the study are displayed in the graphical presentation provided in our original article [2]. As each individual rating can be identified from this presentation, the underlying rating data can be reconstructed and used to calculate alternative agreement coefficients.

We thank Amir Mirhaghi for his comments and welcome the broader discussion on the methodological challenges of comparing reliability studies in triage. Reporting additional agreement coefficients may facilitate comparison with previous studies and future evidence syntheses; however, the choice of coefficient represents only one aspect of comparability. Study design, vignette construction, clinical context, rater characteristics, the structure and distribution of triage categories, and statistical methodology should all be considered when interpreting and comparing reliability estimates across studies.

Abbreviations

GRRAS

Guidelines for Reporting Reliability and Agreement Studies

ICC

Intraclass correlation coefficient

SATS

South African Triage Scale

Author contributions

LD drafted the manuscript and performed the additional analyses. MAH reviewed the manuscript and contributed to revisions. DB reviewed and approved the manuscript. All authors approved of the final manuscript.

Funding

Open access funding was provided by Linköpings University. This work was supported by the Centre for Research and Development (CFUG), Uppsala University/Gävleborg Region, Gävle, Sweden.

Data availability

The post hoc analyses reported in this article were conducted using data from the original study. The dataset is available from the corresponding author upon reasonable request.

Declarations

Ethical approval

Not applicable.

Consent for publication

Not applicable.

Competing interests

The authors declare no competing interests.