Concordance between an artificial intelligence self-triage programme and physical triage

Abstract

Objectives

We evaluated the concordance between the Dutch National Triage Standard (NTS) triage classification and the triage of an artificial intelligence (AI) self-triage programme.

Methods

This observational comparative study was performed in the emergency departments (EDs) of two OLVG hospital locations in Amsterdam in October and November 2023. Adult patients who entered the ED without ambulance transport and were triaged using NTS (usual care), were asked to complete the digital AI triage programme. Cohen’s kappa (ĸ), per cent agreement and the distribution across urgency categories were used to evaluate the concordance between the triage methods.

Results

Of the 264 patients approached, 203 consented to participation and were included in analysis and follow-up. Agreement between the triage methods was none to slight, ĸ=0.092 (95% CI −0.196 to 0.380). The AI triage programme overtriaged in 12.8% of cases and undertriaged in 5.4% compared with the NTS. However, the AI triage programme classified more patients with serious clinical sequelae into S1 (ie, emergency care with ambulance) and S2 (ie, emergency care) compared with NTS U1 and U2. The AI triage programme predicted the final diagnosis at ED discharge in 27.1% of the cases. There were significantly more clinical sequelae (eg, death, ward admission or follow-up consultation) for general practitioner referred patients compared with self-referrals.

Conclusion

The AI triage appeared more distinctive in urgency classification than classic triage when considering clinical sequelae for the patient. However, refinements and better validation of AI triage programmes are needed before these can be implemented in healthcare systems.

WHAT IS ALREADY KNOWN ON THIS TOPIC.

WHAT THIS STUDY ADDS

• This study evaluates the possibility of AI self-triage to match the urgency scores of a classic triage method for emergency department triage.

HOW THIS STUDY MIGHT AFFECT RESEARCH, PRACTICE OR POLICY

• Currently, AI self-triage is not suitable for widespread implementation and cannot yet replace physical triage.

Introduction

Artificial intelligence (AI) is increasingly applied to clinical triage, with potential to reduce workload and healthcare costs, improve patient flow and outcomes, and support timely decision-making.^1–4 As AI programmes become more integrated into triage processes, minimising errors and bias is essential to ensure safety and build trust.^3–6 Demonstrating that AI can perform at least as reliably as conventional methods will be key to broader acceptance.

Classic emergency triage is under scrutiny for various reasons. First, physical triage is influenced by individual clinical judgement, the patient’s context and organisational issues.⁷ Moreover, different triage systems—such as the Emergency Severity Index (ESI), the Manchester Triage System (MTS) and the Netherlands Triage Standard (NTS)—employ fundamentally different strategies. However, they are widely used in clinical practice and deemed interchangeable, as all classify patients into five urgency categories. Second, patients with highly urgent conditions can present with normal vital signs and incorrectly receive lower urgency classifications in triage.^8–11 Third, triage is an unpopular task among emergency department (ED) triage personnel who are in short supply.

An AI triage programme could standardise triage and reduce the workload for ED personnel.^{4 5 9–11} Moreover, the AI triage programme follows a dynamic algorithm, enabling it to detect hidden correlations in individual patient data. These may not be identified using medical expertise or fixed algorithms alone.^4–6 In the planned reorganisation of the Dutch acute care system a key role is envisioned for digital innovations to improve efficacy.¹² This may include AI triage programmes for patients. A limited number of studies have been conducted, but these have failed to show conclusive benefit of AI triage.^{4 5 9–11 13–15} Since developments in this field are evolving rapidly, further investigation is warranted.

We hypothesised that a digital AI self-triage programme could effectively guide patients to the appropriate care setting. Therefore, we compared the urgency classifications of the classic Dutch NTS for physical triage to those of AI triage using Symptomate (Infermedica, Wroclaw, Poland¹⁶).

Methods

Classic physical triage

In Dutch ED triage and general practitioner (GP) triage the Dutch NTS is widely used, even though the system—like other widely used triage systems—is not validated and is criticised by healthcare personnel.^{5 8} The NTS is similar to the MTS. The NTS triage recommendations (ie, urgency classifications) range from U0 (resuscitation needed) to U5, which does not require same-day treatment. The classifications are specified in table 1.⁷

Table 1. The NTS urgency classifications specified by clinical status and timeframe of treatment.

NTS	Clinical status	Treatment timeframe
U0	Failed ABCD	Cardiopulmonary resuscitation
U1	Unstable ABCD: immediate danger to life	Immediate treatment
U2	Imminent ABCD instability or organ damage	Treatment as soon as possible
U3	Real risk of damage	Treatment within a few hours
U4	Negligible chance of damage	Treatment within 24 hours
U5	No chance of damage	Treatment next working day

NTS, National Triage Standard.

The AI triage programme

Symptomate is a medical symptom checker designed to evaluate the urgency of the users’ symptoms and advise possible care pathways. The system uses an inference engine, and its outputs are continuously reviewed and validated by medical and technical experts to ensure clinical accuracy. It is used globally, certified as a class I medical device and is compliant with the General Data Protection Regulation. Symptomate leverages AI, machine learning, natural language processing and Bayesian modelling to assess patient-reported outcomes. Symptomate is a proprietary, closed source and fine-tuned chatbot. The system dynamically guides questions without predefined pathways. The programme output classifies patients into five urgency classifications—similar to the NTS—and reports probable conditions with their degree of evidence, leading to tailored triage recommendations. Urgency classifications of the AI triage programme can be one of the following: S1 = Emergency care with ambulance; S2 = Emergency care at the ED without ambulance; S3 = Consultation with a GP or medical specialist within 24 hours; S4 = Consultation with a GP; S5 = Self-care advice, in the triage report a specific self-care advice is given.^{16 17}

Study design, setting and participants

The data collection for this prospective, observational, comparative study was conducted at the OLVG hospital locations East and West, Amsterdam, the Netherlands in October and November 2023. The OLVG has one of the largest EDs in the Netherlands, with approximately 70 000 presenting patients per year. Adult patients presenting at the ED, for whom an NTS Score was determined during physical triage (usual care), were approached by the researcher (MW) for study participation. All patients who did not meet the exclusion criteria were invited to provide consent and complete the web interface AI self-triage on a tablet. Consecutive sampling was used for all eligible patients during the inclusion period. Exclusion criteria were arrival at the ED by ambulance transport, no familiarity with the Dutch or English language, and age under 18 years. The researcher was available for technical questions regarding the AI programme and the tablet, or to read the questions aloud to the patient when asked. To avoid bias, the researcher did not provide advice or information regarding the triage questions.

Outcomes

The primary outcome was the concordance between the urgency classifications of the NTS triage and AI triage programme. Secondary outcomes include the risk of overtriage and undertriage; the correlation of either method to clinical outcomes (‘sequelae’) 30 days after the initial ED visit; the concordance between the differential diagnosis of the AI triage programme and the diagnosis at ED discharge; and patient satisfaction with the AI triage programme.

Clinical data and definitions

The urgency classification and differential diagnosis of the AI triage report (Symptomate V.2.5) were recorded. Additional data collected from the Electronic Health Record were age, gender, presenting complaint, NTS Triage Score (update 10.1, as actual in the study period), and final diagnosis on ED discharge. Overtriage and undertriage were defined by experienced clinicians based on the urgency classification definitions of the NTS and Symptomate. Overtriage was defined as S1 and S2 classifications in U4 and U5 patients. Undertriage was defined as S3, S4 and S5 classifications for U1 patients and S4 and S5 classifications for U2 patients. For the undertriaged patients the presence of clinical sequelae was used to determine the actual rate of undertriage. The clinical sequelae after 30 days that were recorded out of the electronic patient dossier (EPD) by the researcher were: hospital admission (cardiac care unit (CCU) or ward), follow-up (remote) consultation, ED revisits, operation and death. Those sequelae were predefined based on standardised outcome measures commonly used in previous research. The urgency classifications and the clinical sequelae were also separated for GP patients and self-referred patients, to investigate for confounding by the way of arrival at the ED. The concordance between the diagnosis on ED discharge and the possible diagnosis according to the AI triage programme was assessed individually by two experienced clinicians (MG, consultant in emergency medicine and FH, consultant in acute internal medicine). These clinicians were unblinded to the NTS triage and AI triage diagnosis. The agreement was categorised as follows: most likely diagnosis matched, final diagnosis in the top 3, final diagnosis mentioned, final diagnosis not specifically mentioned but reasonable alternative provided or final diagnosis not mentioned. Disagreements between the two clinicians’ individual judgements were resolved in discussion by the authors. The satisfaction with (very satisfied to very dissatisfied) and the clarity of the triage programme were scored by the patients after completing the triage programme using a 0–10 Likert Scale.¹⁷ Since there is no standard for urgency classification in current care,⁸ the triage by NTS was the ‘gold standard’ in this study, given its established role in Dutch clinical practice. Data for all participants, with no missing values in the primary outcome, were entered in a Castor Electronic Data Capture database (Castor, Amsterdam, the Netherlands).

Statistical analyses

The population characteristics are presented with descriptive statistics. Categorical variables are presented with frequency counts and percentages. Continuous variables are presented with mean and SD. The agreement between triage classifications was tested using weighted and unweighted Cohen’s kappa (ĸ), χ² tests, exact per cent agreements and agreement within ±1 category (Clopper-Pearson method). Two-tailed tests were used, with significance at a value of p<0.05. Both classifications were stratified for GP patients and self-referred patients. Since the assumptions of the Yates’ and Pearson’s χ² tests were not met (ie, low cell counts), these analyses are not presented. The concordance between the diagnoses of both triage methods was presented with counts and percentages. The Net Promoter Score (NPS; range –100 to 100) was calculated by subtracting the percentage detractors (score ≤6) from the percentage promoters (score 9 or 10) of the Likert Scale.¹⁸ Statistical analyses were performed using the Statistical Package for Social Sciences V.28.0.¹⁹

Ethics

Patient and public involvement was limited to their involvement as study participants. Patients were not involved in study design or in dissemination of results.

Enhancing the QUAlity and Transparency Of health Research (EQUATOR)²⁰ and Chatbot Assessment Reporting Tool (CHART)²¹ guidelines were followed for transparent and standardised reporting.

Results

Demographics

We approached 264 patients of whom 203 participated (see figure 1). Characteristics of participants are summarised in table 2.

Figure 1. Flow chart. AI, artificial intelligence.

Table 2. Baseline characteristics.

Characteristics	Total population
	203 (100.0%)
Male	108 (53.2%)
Age, years (mean (SD))	50 (20)
Native Dutch speaker	164 (80.7%)
Arrival via
GP referral	120 (59.1%)
Self-referred	83 (40.9%)
Presenting complaint
Trauma	59 (29.1%)
Abdominal complaints	38 (18.7%)
Wounds and abscesses	33 (16.3%)
General malaise	22 (10.8%)
Respiratory complaints	17 (8.4%)
Neurological complaints	14 (6.9%)
Thoracic complaints	10 (4.9%)
Urinary tract complaints	6 (3.0%)
Other complaints	4 (2.0%)

GP, general practitioner.

Primary outcome

Table 3 presents the absolute numbers and the concordance of the triage classifications. The most frequent recommendation following physical NTS triage was U3. Categories U4 and U1 were less frequently assigned. Triage recommendations were more evenly distributed in the AI triage programme, with S5 advised less frequently than S1–4. There was no to slight statistical agreement between the triage methods, as indicated by unweighted Cohen’s ĸ (ĸ=0.018; 95% CI −0.053 to .089) and quadratically weighted Cohen’s ĸ (ĸ=0.092; 95% CI −0.196 to 0.380). Agreement within ±1 category occurred in 68.9% (95% CI 62.1% to 75.3%) of the cases. Exact agreement between urgency classifications was observed in 21.7% (95% CI 16.3% to 27.8%) of the patients. Agreement was highest in category 3 (11.8%; 95% CI 7.7% to 17.1%), followed by category 2 (5.4%; 95% CI 2.7% to 9.5%). Agreement in categories 1, 4 and 5 was rare, each accounting for ≤2.0% of patients.

Table 3. Cross table of combined NTS triage and AI triage classifications.

		NTS triage
		U5	U4	U3	U2	U1	Total
AI triage	S5	1	1	5	.	.	7 (3.4%)
	S4	6	4	32	7	2	51 (25.1%)
	S3	5	3	24	10	2	44 (21.7%)
	S2	16	3	27	11	.	57 (28.1%)
	S1	6	1	16	17	4	44 (21.7%)
	Total	34 (16.7%)	12 (5.9%)	104 (51.2%)	45 (22.2%)	8 (3.9%)	203 (100.0%)

AI, artificial intelligence; NTS, National Triage Standard.

Secondary outcomes

Overall, 26 (12.8%) patients were overtriaged and 11 (5.4%) were undertriaged using the AI triage programme. Of the 11 undertriaged patients, 3 were admitted to a ward, 4 had follow-up consultation and 4 had no follow-up. Among these, one ward-admitted patient and one patient with a follow-up consultation underwent an operation within 30 days.

Within 30 days after the initial ED visit, 2 patients died (1.0%), 30 (14.8%) revisited the ED, 14 (6.9%) underwent an operation, 45 (22.2%) were hospitalised and 74 had a follow-up consultation (36.5%). One patient had a CCU admission and died afterwards. As can be seen in figure 2, NTS U1 and U2 had the highest percentage of ward admission. U5 patients had more and relatively more serious clinical sequelae than S5 patients. The two patients who died were classified as U2 in NTS and S1 by the AI triage programme. In the AI triage recommendations, the ward admission percentages were higher for S1–3 than S4–5.

Figure 2. (A) NTS classifications and 30-day clinical sequelae in percentages and counts. (B) AI triage classifications and 30-day clinical sequelae in percentages and counts. AI, artificial intelligence; NTS, National Triage Standard.

For both GP referred and self-referred patients, the AI triage programme assigned higher urgency categories to cases compared with NTS (more S1–S2 vs U1–U2 and less S5 vs U5), as can be seen in figure 3. Within both NTS triage and AI triage, there was little difference in urgency classification between GP-referred patients and self-referred patients. However, there were more clinically relevant sequelae for patients who were referred by a GP compared with self-referrals. Significant differences between GP-referred and self-referred patients were found for the clinical sequelae ward admission (37 GP-referred vs 7 self-referred; χ²=14.501, p<0.001), ED revisit (23 GP-referred vs 7 self-referred; χ²=4.488, p=0.034) and no follow-up (38 GP-referred vs 46 self-referred; χ²=11.414, p=0.001). No significant relation for follow-up consultation (44 GP-referred vs 30 self-referred) and death (2 GP-referred) was found.

Figure 3. (A) Percentages of NTS triage classifications for GP referrals and self-referrals. (B) Percentages of AI triage classifications for GP referrals and self-referrals. AI, artificial intelligence; GP, general practitioner; NTS, National Triage Standard.

The predicted diagnosis in the AI triage report was in good concordance with the diagnosis at ED discharge in 27.1% of the cases (ie, in the top three possible conditions). Moreover, for 32.0% of the population, a reasonable alternative diagnosis was provided. For example, the AI triage programme predicted gallstones, whereas clinical investigation revealed kidney stones.

The satisfaction with AI self-triage via Symptomate was scored by 162 of the 203 patients. The scores were: very satisfied 19.7%, satisfied 46.8%, neutral 10.8%, dissatisfied 2.5%, very dissatisfied 0.0%. The clarity of the Symptomate programme was 8.1 (SD 1.3) on a 0–10 Likert Scale, the NPS is 29.0.

Discussion

Key results

The concordance in urgency classifications between the Dutch NTS and the AI self-triage programme Symptomate was low. However, 68.9% cases agreed within ±1 category, which suggests that large clinically relevant discrepancies were uncommon. The AI triage programme assigned more patients to a higher urgency category than the NTS, resulting in overtriage in 12.8% of cases and undertriage in 5.4%. The actual rate of undertriage was probably lower, since 4 of the 11 patients had no clinical sequelae. Overall, the AI programme was better at predicting clinical sequelae by higher urgency scores than the NTS. The AI programme had good prediction of the final diagnosis in 27.1% of the cases and additionally a reasonable prediction of the diagnosis in 32.0%.

Limitations of this study

This study is the first to compare the traditional NTS triage method to an AI programme. It has several limitations that should be considered when interpreting the results.

Although both triage systems use five urgency categories and are widely applied, their urgency definitions differ: the NTS focusing on time to care and AI on type of care with the corresponding timeframe. To address this, overtriage and undertriage were defined for the urgency categories based on expert consensus. Nevertheless, these inherent differences may have influenced agreement levels. Adding to that, the NTS, used as the reference standard due to its common use in Dutch clinical practice and its comparability to MTS and ESI, has not been independently validated. The lack of a true gold standard for triage limits the conclusiveness of the comparisons. Moreover, Symptomate was designed for use earlier in the care process for a generally less urgent population; thus, observed overtriage in this more acute ED cohort may underestimate overtriage in lower-acuity patients. Patients arriving by ambulance, not fluent in Dutch or English, and those under 18 years were excluded for practical and ethical reasons. This limits the inclusion of vulnerable or high-acuity patients, which may have affected the AI tool’s performance and limits generalisability. The researcher gave some patients technical assistance with the AI tool without intending to influence responses, but observer bias could have been introduced by her presence in itself. Additionally, the clinicians who assessed the concordance between discharge and AI diagnoses were not fully blinded (reasonably given the study design) to the triage classifications, which potentially introduces assessment bias. Lastly, the observed low to slight agreement should be interpreted with caution. Small subgroup sizes and the skewed distribution across urgency categories likely impacted the low κ value and χ² statistics. This potentially underestimates true concordance and may have contributed to the ‘kappa paradox’. To provide additional context, overall and category-specific per cent agreement was reported. This additional analysis allows readers to interpret the concordance more transparently, particularly given the imperfect nature of NTS as a reference standard.

Potential advantages of the AI triage

Despite its limitations, this study shows the promise of AI triage. Importantly, patients were generally satisfied and reported positive NPS for clarity. Also, when assessed in terms of clinical sequelae, potentially dangerous undertriage was rare.

The AI triage appeared more evenly distributed than the NTS in this study. The most frequent recorded NTS classification is U3 (51.2%), which is in concordance with numbers found in the general population²² and in another study in which the concordance in NTS classifications and Modified Early Warning Scores was examined.²³ However, a quirk of the NTS appears to be that U4 has an unexpectedly low frequency, while a normal distribution is expected. Lower counts of U1 are generally expected and can be further explained by the study criteria as participants did not arrive by ambulance. Importantly, NTS has a subjective character as triage personnel can adjust the NTS triage urgency classifications based on their clinical judgement. Clinical context, patient presentation and referrals for specialty services (eg, X-ray, ultrasound, check-up) can thus influence the urgency classification.^8–11 A machine learning AI algorithm would need access to these data to incorporate and standardise this.

The AI programme seemed more accurate in relation to clinical sequelae than the NTS and made a fair prediction of the eventual diagnosis even without clinical and advanced investigation. The more urgent classifications in the AI triage can be explained by different facts. First, overclassification can be due to the self-triage of patients. Some studies show that higher urgency scores are seen in patients doing self-triage versus triage by another person interpreting the patients’ complaints.^24–26 Second, the AI triage asks for the patients’ intended use of care to estimate the patient’s perceived urgency, however this indication was biased as patients were already present at the ED. In general, any AI triage programme needs to be fit to the population in which it is going to be used.²⁷ Lastly, the AI triage is dependent on the built-in algorithm. For example, the clinicians who scored the concordance of the diagnosis noted that specific or rare differential diagnoses were mentioned surprisingly often, while more obvious/common diagnoses were ignored. The AI triage programme appeared to give greater weight to rarer but potentially dangerous diagnoses. These algorithms are crucial when considering AI implementation in healthcare. However, given AI’s adaptive and self-learning capabilities and the rapid evolution of AI, the very use of AI systems may remedy such imbalances.

GP consultation prior to ED visit was prognostic for clinical sequelae (eg, follow-up consultation, ward admission and death) in 30 days. However, the NTS triage and AI triage urgency scores for GP-referred versus self-referred patients did not differ within the triage methods. This suggests that factors beyond the scope of the triage tools, such as conversation with the GP, give crucial additional information for the eventual clinical course of patients. Incorporating these factors could improve triage tool algorithms.

Conclusion

AI self-triage programmes have the potential to support or even replace (ED) triage, given their simplicity, potentially even at home. The AI programme appeared more accurate than NTS when considering clinical sequelae and predicted diagnoses. However, low concordance was observed between the AI self-triage programme and physical NTS triage. It remains unclear whether this discrepancy is primarily due to limitations in the AI triage, the NTS triage or both, as each triage method has inherent imperfections. Large multicentre studies integrating richer patient data and assessing concordance and unique strengths of triage systems are essential for safe and effective integration of AI triage programmes into clinical practice.

Data availability statement

Data are available upon reasonable request. All data relevant to the study are included in the article or uploaded as supplementary information.