Communication quality in telephone triage conducted by general practitioners, nurses or physicians: a quasi-experimental study using the AQTT to assess audio-recorded telephone calls to out-of-hours primary care in Denmark

Abstract

Objectives

To compare the quality of communication in out-of-hours (OOH) telephone triage conducted by general practitioners (GPs), nurses using a computerised decision support system and physicians with different medical specialities, and to explore the association between communication quality and efficiency, length of call and the accuracy of telephone triage.

Design

Natural quasi-experimental cross-sectional study.

Setting

Two Danish OOH services using different telephone triage models: a GP cooperative and the medical helpline 1813.

Participants

1294 audio-recorded randomly selected OOH telephone triage calls from 2016 conducted by GPs (n=423), nurses using CDSS (n=430) and physicians with different medical specialities (n=441).

Main outcome measures

Twenty-four physicians assessed the calls. The panel used a validated assessment tool (Assessment of Quality in Telephone Triage, AQTT) to measure nine aspects of communication, overall perceived communication quality, efficiency and length of call.

Results

The risk of poor quality was significantly higher in calls triaged by GPs compared with calls triaged by nurses regarding ‘allowing the caller to describe the situation’ (GP: 13.5% nurse: 9.8%), ‘mastering questioning techniques’ (GP: 27.4% nurse: 21.1%), ‘summarising’ (GP: 33.0% nurse: 21.0%) and ‘paying attention to caller’s experience’ (GP: 25.7% nurse: 17.0%). The risk of poor quality was significantly higher in calls triaged by physicians compared with calls triaged by GPs in five out of nine items. GP calls were significantly shorter (2 min 57 s) than nurse calls (4 min 44 s) and physician calls (4 min 1 s). Undertriaged calls were rated lower than optimally triaged calls for overall quality of communication (p<0.001) and all specific items.

Conclusions

Compared with telephone triage by GPs, the communication quality was higher in calls triaged by nurses and lower in calls triaged by physicians with different medical specialities. However, calls triaged by nurses and physicians were longer and perceived less efficient. Quality of communication was associated with accurate triage.

Strengths and limitations of this study.

• The natural quasi-experimental study design using randomly selected telephone calls from out-of-hours (OOH) primary care explored quality in real-life settings increasing generalisability.

• This study compared quality of communication conducted by general practitioners, nurses using a computerised decision support system and physicians with different medical specialities that were already experienced with OOH telephone triage.

• The thorough assessments of 1294 calls were conducted using a comprehensively developed and tested assessment tool, Assessment of Quality in Telephone Triage (AQTT), with an explicit rating manual minimising rater subjectivity.

• Calls were assessed by only one assessor and all assessors were physicians.

• Our study cannot determine whether the differences in quality of communication should be attributed to use/non-use of computerised decision support system, differences in educational background, differences in personality and/or different working and organisational conditions.

Introduction

Telephone triage plays a pivotal role in the management of the increasing workload in the out-of-hours (OOH) services.^1–3 While aiming for a safe and efficient delivery of healthcare, telephone triage must also avoid undertriage and minimise overtriage.⁴ Accurate telephone triage is often a challenge due to lack of visual cues, little knowledge of the patient, higher likelihood of severe illness and time pressure.^5–7

Good communication has been shown to benefit the patient’s health.⁸ Moreover, training in communication has been associated with more accurate diagnosing⁹ and better stress relief in patients.¹⁰ Patient-centred communication focusing on the patient’s perspective and shared decision-making¹¹ is the gold standard for good quality in face-to-face consultations. Patient-centred communication is associated with higher patient satisfaction,^12–14 better adherence¹⁴ and fewer referrals.¹⁵ However, telephone triage differs from face-to-face consultations by focusing on the gathering of medical information and less on the patient’s perspective.^16–18 Previous studies have identified communication failure as a reason for malpractice complaints in OOH telephone triage.^{19 20} In telephone consultations ending with malpractice complaints, triage professionals have been shown to use few open-ended questions and little attentive listening with back-channel responses.¹⁹ Moreover, triage professionals should be alerted when the patient expresses concern.^{20 21}

Internationally, different healthcare professionals, such as general practitioners (GPs) or nurses, conduct the OOH telephone triage using different decision-making strategies.^{22 23} Nurses are mostly obliged to use a computerised decision support system (CDSS). The CDSS aids the information gathering,²⁴ but it may also lead to fixation on only one symptom.^24–26 Studies comparing the quality of the communication in OOH telephone triage conducted by GPs and nurses using CDSS are sparse.^{27 28} Most studies are non-comparative and explore the quality of the communication in nurse-led or physician-led triage,¹⁶ the length of the call^{29 30} or the degree to which the patient is allowed to describe the situation.^31–33 Thus, to our knowledge, there is no study comparing the quality of communication in telephone triage conducted by GPs and nurses or by GPs and physicians with different medical specialities. The Danish OOH telephone triage has been reorganised, and currently we have a window of time where different triage models exist in Denmark. Thus, we aim to compare the quality of the communication in OOH telephone triage conducted by GPs, nurses using a CDSS and physicians with different medical specialities. We also aimed to explore the association between communication quality and efficiency, length of call and the accuracy of telephone triage.

Method

Design and setting

We conducted a natural quasi-experimental study in two OOH services in Denmark: the general practitioner cooperatives (GPC) in the Central Denmark Region and the MH-1813 in the Capital Region of Denmark (see table 1). In 1992, a national reform introduced large-scale GPCs with GP-led telephone triage.³ In 2014, the Capital Region of Denmark reorganised the OOH service and formed the MH-1813 with nurse-led telephone triage guided by CDSS and with the option to redirect calls to physicians.^{34 35} All triage nurses are registered nurses with different medical specialisation and complete a 6-week introductory course. In the following referred to as ‘nurses’. The physicians at MH-1813 answer approximately one-third of all calls and additionally triage the calls redirected from nurses.³⁴ The physicians at MH-1813 have different medical specialities and varying experience, but they are not obliged to use CDSS.^{34 35}

Table 1.

Description of the two out-of-hours organisations and their telephone triage models

Region	GP cooperative (GPC)	Medical helpline 1813 (MH-1813)
	Central Denmark region	Capital region of Denmark
Population44	1.3 million citizens	1.8 million citizens
Telephone calls in 201445	697 000	911 000
Organiser	GPs in the region	Regional administration
Organisation and services	Telephone triage, home visits and face-to-face consultations at the GPC GPs are obliged to take part in the service	Telephone triage and home visits run by MH-1813 Face-to-face consultations are located in hospital facilities and managed by EDs
Remuneration	Fee for service	Payment by the hour
Triage professional	GPs or GP trainees in their final year of specialty; no CDSS available	Nurses who are obliged to use a CDSS and option to redirect calls to a physician Physicians with different medical specialities (a minority being GPs)

CDSS, computerised decision support system; ED, emergency department; GP, general practitioner.

The GPC and the MH-1813 are open outside office hours, that is, from 16:00 to 08:00 on weekdays, all day on weekends, and all day on national holidays and during OOH they provide telephone triage for inhabitants of the entire region.³⁵ MH-1813 can also be reached during daytime. Both types of OOH services offer telephone consultations, clinic consultations and home visits.

Inclusion and exclusion of patient calls

We included calls directly answered by GPs at the GPC or answered by nurses and physicians at MH-1813 outside office hours during the inclusion period (MH-1813: 23 November to 8 December 2016, GPC: 23 November to 7 December 2016). For calls redirected by a nurse to a physician at MH-1813, we only included the part conducted by the nurse. We selected randomly using STATA from all eligible calls while ensuring equal distribution of time of day (ie, day, evening, night) and day of week (ie, weekend/not weekend). Due to an unforeseen 3-day system outage of the IT system at MH-1813, we were unable to get the audio-recordings of 194 selected calls. Thus, we substituted these with randomly selected calls matched on day of week and time of day from the following week.

We defined a number of exclusion criteria (table 2), which were checked in a two-step procedure (figure 1). The first step was based on register information of all eligible calls. The second step was performed on the basis of the actual audio-recordings. Three medical students identified calls eligible for exclusion, which were approved by the first author (DSG). Moreover, the students anonymised the audio-recordings using beep tones to mask triage profession, organisation and patient identification information.

Table 2.

Exclusion criteria for telephone calls

Type	Definition (clarification)
Step 1	Identified through register-based information
Frequent callers	Defined as patients with ≥7 calls during the 2-week inclusion period (assessment of the triage quality could be difficult as the patient’s medical record from the OOH service could include important information on these patients that was available only to the triage professional and not to the assessor)
No answer	Calls with no caller answering the triage professional
Daytime calls	Calls performed during daytime (the TT service at MH-1813 was available during daytime)
Step 2	Identified by listening to the triage contact
Other health professionals	The caller was another healthcare professional, for example, from a nursing home
Administrative calls	The reason for calling was administrative, for example, calling to get the number for the acute dentist
Simple drug prescriptions	The patient called for renewal of a prescription that required little information sharing
Preterm termination	Calls that were ended too early, for example, calls made by error, no sound on call or sound interrupted in the middle of call
Other localisation	Calls from a caller who was not in the same location as the patient, for example, parent on the way to pick up a sick child from day care
Poor sound quality	Calls with poor sound quality (making assessment difficult)
Language issues	Calls in which language issues challenged the triage, that is, caller did not speak Danish or English
Unable to identify call	Random calls in which an exact linkage to the corresponding audio-recorded call could not be established

OOH, out-of-hours.

Figure 1.

Flowchart of inclusion and exclusion of calls from the general practitioner cooperative (GPC) and MH-1813. For definition of exclusion criteria, see table 2. £Due to higher expected exclusion rate of calls triaged by a health professional, more calls were selected from the GPC.

Power calculation

We aimed to include 435 audio-recorded calls per group of triage professionals. This number was based on a power calculation of sample size assuming an undertriage rate of approximately 9.5%, and we aimed to detect a 5% difference between triage professionals with a power of 0.8 and an alpha of 0.05. A distinctive feature of MH-1813 is that staff at nursing homes use another telephone number than the public one. Consequently, as we expected more GPC calls from nursing homes to be eligible for exclusion, we selected extra calls from the GPC (n=525 vs 500).

Assessment tool

The rating of the quality of the communication was performed using the tool ‘Assessment of Quality in Telephone Triage’ (AQTT) and the accompanying rating manual (online supplementary appendix 1). The AQTT has been comprehensively developed and tested.³⁶ The AQTT comprises 24 items: 9 specific items assessing the communication, 11 specific items assessing the health-related quality and 4 items enabling the assessor to give an overall rating of their general perception (ie, the gut feeling) of the quality of communication, health-related quality, patient safety and efficiency. The majority of items were rated on a 5-point Likert scale, and an additional category (‘not applicable’) could be applied if an item was correctly found not to be relevant or if the available information was insufficient for assessment. In item 11, the accuracy of the triage was rated on a 7-point scale to differentiate between levels of undertriage and overtriage. The rating ‘4’ was assigned in calls that were considered to be optimally triaged, whereas increasing and decreasing ratings were used according to the degree of potential undertriage (towards 1 if patient safety was impaired) or overtriage (towards 7 in case of overuse of health resources).³⁶ In this paper, we present results on the nine specific items assessing the communication, the overall perceived communication and efficiency measured on a 10-point visual analogue scale, and item 11 assessing the accuracy of the triage (table 3).

Table 3.

Overview of AQTT items used for assessment of communication quality

Items assessing specific health-related aspects
Item 12:	Gives the caller sufficient time and space to describe the situation
Item 13:	Conducts the conversation in understandable language adapted to the caller’s situation
Item 14:	Ensures that triage decision and advice given are understandable and feasible
Item 15:	Ensures that the caller agrees on the triage decision and advice given and is accommodating in case of disagreement
Item 16:	Structures the conversation
Item 17:	Masters suitable questioning techniques (including suitable use of open-ended, closed-ended and non-leading questions)
Item 18:	Summarises (if relevant), verifies and adjusts if needed
Item 19:	Pays attention to the caller’s experience and situation
Item 20:	Conducts the conversation in an accommodating and friendly tone
Items assessing overall quality
Item 21:	How would you assess the overall quality of the communication?
Item 24:	How would you assess the overall efficiency?

AQTT, Assessment of Quality in Telephone Triage.

Assessment panel

We recruited an assessment panel consisting of 24 physicians from the GPC and the MH-1813 using two inclusion criteria: >1 year of experience and currently active in OOH triage. An email invitation was sent to all GPs and physicians by their organisers. Using STATA, we randomly selected 16 GPs from 56 interested GPs at the GPC while considering age and sex distribution. At the MH-1813, we included all physicians fulfilling our inclusion criteria (n=8) from 10 interested physicians. The assessment panel participated in a 2-day training course, which provided them with knowledge on telephone triage and related communication and also familiarised them with assessing triage calls by using the AQTT. This was done through both individual and plenary training sessions.

Assessment process

Every audio-recorded call was distributed at random to one assessor. Thus, each assessor assessed calls triaged by varying triage professionals. Information on age and sex of the patient as well as day of week and time of day of the call was available for the assessor. Assessments were conducted at home; each assessor assessed a median of 53 (range: 48–61) calls during a median period of 111 days. Assessors were payed an hourly fee.

Statistical analyses

The AQTT is not a summative scale and the score of each AQTT item is handled individually. We explored two different categorisations of the specific communicative items while excluding calls rated ‘not applicable’: poor quality (rated ‘1’ or ‘2’) and good quality (rated ‘4’ or ‘5’). The accuracy of the triage decision (item 11) was recoded into cumulative undertriage (rated ‘1’, ‘2’ or ‘3’), optimal triage (rated ‘4’) and cumulative overtriage (rated ‘5’, ‘6’ or ‘7’). We used descriptive analyses to describe the baseline characteristics of both the patients and the calls. We used descriptive analyses to characterise the specific communicative items, while calculating the risk of poor and good quality excluding the rating ‘n/a’ from our analyses. For the overall perceived items, we used descriptive analyses to calculate medians for the different triage professionals.

In comparative analyses, generally overall comparison of the three groups of triage professionals was conducted using χ² tests and Kruskal-Wallis test. Second, for categorical and continuous outcomes, the χ² tests and Mann-Whitney U-test was used, respectively, to analyse differences between the compared groups (ie, pairwise comparisons of GPs vs nurses and GPs vs physicians). We calculated the relative risk (RR) of having poor quality on the specific communicative items related to type of triage professional using binomial regression. In all pairwise comparative analyses, we used calls triaged by GPs as the reference group. However, to explore the association between the quality of communication, efficiency and length of call and the accuracy of telephone triage, calls handled by nurses, physicians and GPs were pooled and in pairwise comparisons, optimally triaged calls were used as reference (ie, calls perceived as undertriaged vs optimally triaged and overtriaged vs optimally triaged).

We noticed a tendency for assessors from GPC (ie, GPs) to rate the quality of GP-led triage more favourable compared with assessors from MH-1813 assessing GP-led triage as well as the reverse: assessors from MH-1813 rated the quality of physician-led triage more favourable compared with GP assessors assessing physician-led triage. We concluded that a ‘similar-to-me’ bias was present in the data, that is, assessors giving a slight bonus to triage led by triage professionals similar to themselves.³⁷ Since the dataset is unbalanced concerning the origin of our assessors (GPC: 16 vs MH-1813: 8) and, more importantly, since nurses could never receive such favourable assessment, we decided to adjust for whether or not assessor had the same professional background as the triage professional. All analyses were performed in Stata V.14.2 (StataCorp. 2015. Stata Statistical Software: Release 14.2. College Station, Texas, USA: StataCorp LP).

Public and patient involvement statement

The patients’ perspective was explored in a focus group interview, and the input was incorporated in the rating manual in item 12, 13, 15, 16 and 19 in the AQTT.³⁶ All participants who contributed to the development of the AQTT will be informed of the knowledge gained in the study.

Results

Characteristics of calls

A total of 1294 calls were assessed (GP=423, nurse=430, physician=441). No differences were identified between the compared groups concerning patient’s age, patient’s sex and time of call (table 4). However, the distribution of patient’s sex differed between calls triaged by nurses and physicians (p<0.001). The length of calls differed; both nurses and physicians had significantly longer calls than GPs.

Table 4.

Baseline description of study sample by patient and call characteristics

Triage professional		GP (n=423)	Nurse (n=430)	Physician (n=441)
Patient characteristics
Sex, % (n)*	Male	42.8 (181)	37.9 (163)	47.2 (208)
	Female	57.2 (242)	62.1 (267)	52.8 (233)
Age in years, % (n)	0–4	20.3 (86)	23.6 (101)	21.9 (96)
	5–17	15.8 (67)	13.3 (57)	14.8 (65)
	18–39	29.6 (125)	31.5 (135)	30.6 (134)
	40–64	21.8 (92)	20.6 (88)	20.1 (88)
	≥65	12.5 (53)	11.0 (47)	12.6 (55)
Call characteristics
Time of call†, % (n)	Weekend	51.6 (218)	51.2 (220)	50.3 (222)
	Not weekend	48.5 (205)	48.8 (210)	49.7 (219
	Day	22.2 (94)	22.6 (97)	21.1 (93)
	Evening	61.5 (260)	60.9 (262)	61.5 (271)
	Night	16.3 (69)	16.5 (71)	17.5 (77)
Length of call, min and s  (95% CI in seconds) *	Mean	2 min 57 s (167 to 188)	4 min 44 s (268 to 300)‡	4 min 1 s (227 to 254)‡

*Indicates a significant difference (p<0.05) between all three groups of triage professional, using χ² tests for categorical variables and Kruskal-Wallis for length of call.

†Time of call: weekend=Friday 16:00–Sunday midnight; not weekend=Monday 00:00–Friday 08:00.

‡Significant difference between nurses or physicians compared with the GP reference group (p<0.05) (pairwise) identified using χ² test and Mann-Whitney U-test.

GP, general practitioner.

Distribution of ratings

Figure 2 reveals that four specific items were rated as having poor quality in over 20% of calls: ‘structuring of conversation’, ‘questioning techniques’, ‘summarising’ and ‘paying attention to caller’s experience’.

Figure 2.

Assessment of the quality of communication using the specific communicative items for different triage professionals (percentage, scale 1–5). GP, general practitioner.

Quality of specific communicative items

As shown in table 5, the RR of poor quality in nurse triage was significantly lower compared with GP triage for four items: ‘allowing the caller sufficient time to describe the situation’ (GP: 14.2%, nurse: 9.8%), ‘mastering suitable questioning technique’ (GP: 27.3%, nurse: 21.1), ‘summarising appropriately’ (GP: 33.0%, nurse: 21.0%) and ‘paying attention to the caller’s experience’ (GP: 25.7%, nurse: 17.0%). The RR of poor quality in physician-led triage was significantly higher compared with GP triage for five items: ‘conducting the conversation in understandable language’ (GP: 5.5%, physician: 13.2%), ‘ensuring that the caller agrees on triage decision and advice’ (GP: 7.8%, physician: 13.3%), ‘structuring the conversation’ (GP: 21.2%, physician: 31.7%), ‘mastering suitable questioning techniques’ (GP: 27.3%, physician: 34.5%) and ‘leading the conversation in an accommodating and friendly tone’ (GP: 2.8%, physician: 8.0%). Table 5 additionally shows the RR estimates adjusted for evaluator background (GPC, MH-1813) (ie, similar-to-me) and the uneven constitution of assessors (assessors from GPC: MH-1813 – 16:8).

Table 5.

Assessment of specific communicative items and relative risk (RR) of poor quality for different triage professionals

Communicative items (AQTT)	Triage professional	N/A (%)	Poor quality (1 or 2)			Good quality (4 or 5)
			% (n)	RR (95% CI)	RR* (95% CI)	% (n)
12: Gives the caller sufficient time and space to describe the situation†	All	0.1	13.5 (174)			59.1 (764)
	GP	0.2	14.2 (60)	1	1	60.2 (254)
	Nurse	0.0	9.8 (42)	0.69 (0.47 to 1.00)‡	0.67 (0.43 to 1.06)	63.5 (273)
	Physician	0.0	16.3 (72)	1.15 (0.84 to 1.57)	1.12 (0.79 to 1.62)	53.7 (237)
13: Conducts the conversation in understandable language adapted to the caller’s situation†	All	0.1	8.0 (103)			74.3 (961)
	GP	0.2	5.5 (23)	1	1	78.0 (329)
	Nurse	0.0	5.1 (22)	0.94 (0.53 to 1.66)	0.82 (0.44 to 1.55)	79.8 (343)
	Physician	0.0	13.2 (58)	2.41 (1.52 to 3.84)‡	2.26 (1.40 to 3.67)‡	65.5 (289)
14: Ensures that the triage decision and the advice given are understandable and feasible	All	2.0	7.4 (94)			52.4 (665)
	GP	1.4	7.4 (31)	1	1	53.5 (223)
	Nurse	3.7	6.0 (25)	0.81 (0.49 to 1.35)	0.83 (0.44 to 1.58)	53.4 (221)
	Physician	0.9	8.7 (38)	1.17 (0.74 to 1.84)	1.18 (0.70 to 1.99)	50.6 (221)
15: Ensures that the caller agrees on the triage decision and advice given and is accommodating in case of disagreement†	All	7.0	10.1 (121)			48.9 (588)
	GP	8.0	7.8 (30)	1	1	49.6 (193)
	Nurse	8.4	8.9 (35)	1.15 (0.72 to 1.84)	0.97 (0.58 to 1.65)	51.3 (202)
	Physician	4.8	13.3 (56)	1.73 (1.13 to 2.64)‡	1.59 (1.03 to 2.45)‡	46.0 (193)
16: Structures the conversation†	All	4.2	25.0 (310)			44.4 (551)
	GP	6.2	21.2 (84)	1	1	48.4 (192)
	Nurse	2.3	21.9 (92)	1.04 (0.80 to 1.35)	0.87 (0.66 to 1.15)	49.3 (207)
	Physician	4.1	31.7 (134)	1.50 (1.18 to 1.90)‡	1.40 (1.11 to 1.78)‡	35.9 (152)
17: Masters suitable questioning techniques (including suitable use of open-ended, closed-ended and non-leading questions)†	All	0.8	27.7 (355)			35.5 (455)
	GP	1.2	27.3 (114)	1	1	34.0 (142)
	Nurse	0.7	21.1 (90)	0.77 (0.61 to 0.98)‡	0.69 (0.53 to 0.90)‡	41.5 (177)
	Physician	0.7	34.5 (151)	1.26 (1.03 to 1.55)‡	1.21 (0.98 to 1.48)	31.1 (136)
18: Summarises (if relevant), verifies and adjusts if needed†	All	11.8	30.5 (348)			28.7 (328)
	GP	16.1	33.0 (117)	1	1	25.4 (90)
	Nurse	9.3	21.0 (82)	0.64 (0.50 to 0.81)‡	0.62 (0.47 to 0.82)‡	35.9 (140)
	Physician	10.0	37.5 (149)	1.14 (0.94 to 1.38)	1.13 (0.92 to 1.38)	24.7 (98)
19: Pays attention to the caller’s experience and situation†	All	12.2	24.4 (277)			38.0 (432)
	GP	13.5	25.7 (94)	1	1	40.4 (148)
	Nurse	11.2	17.0 (65)	0.66 (0.50 to 0.88)‡	0.61 (0.44 to 0.84)‡	40.6 (155)
	Physician	12.0	30.4 (118)	1.18 (0.94 to 1.49)	1.13 (0.88 to 1.45)	33.3 (129)
20: Conducts the conversation in an accommodating and friendly tone†	All	0.2	5.2 (67)			70.6 (912)
	GP	0.0	2.8 (12)	1	1	73.3 (310)
	Nurse	0.2	4.7 (20)	1.64 (0.81 to 3.32)	1.30 (0.51 to 3.31)	74.1 (318)
	Physician	0.5	8.0 (35)	2.81 (1.48 to 5.34)‡	2.31 (1.02 to 5.24)‡	64.7 (284)

Percent rated as poor quality (ie, rated ‘1’ or ‘2’) and ‘good quality’ (ie, rated ‘4’ or ‘5’) are percentages of these rates of all calls in which the item was relevant (ie, ‘not applicable’ excluded)

*RR of poor quality adjusted for evaluator background (GPC, MH-1813) (ie, if call is assessed by an assessor with the same professional background and organisation (similar-to-me)) and the uneven constitution of assessors (assessors from GPC:MH-1813 – 16:8).

†Indicates a significant difference (p<0.05) in the risk of poor quality between all three groups of triage professional, using χ² test.

‡Significant difference (p<0.05) from reference group GP, using binomial regression.

GP, general practitioner; RR, relative risk of poor outcome (95% CI) compared with GP triage.

Overall perceived quality of communication

The overall perceived quality of communication (scale 0–10) was assessed to be similar for GP-led and nurse-led triage (p=0.63), but the quality was significantly lower for triage conducted by physicians with different medical specialties compared with GPs (p<0.001) (table 6). The overall perceived efficiency was significantly lower for both nurses (p<0.001) and physicians (p<0.001) compared with GPs.

Table 6.

Overall perceived quality of communication and efficiency for different triage professionals

Overall assessed quality (AQTT)*	Triage professional	Median (10th to 90th percentile)
How would you rate the overall quality of the communication in the telephone triage?†	GP	7 (3 to 9)
	Nurse 1813	7 (3 to 9)
	Physician 1813	6 (2 to 9)**
How would you rate the overall efficiency in the telephone triage?†	GP	8 (4 to 10)
	Nurse 1813	6 (2 to 9)**
	Physician 1813	7 (2 to 10)**

Median (10th to 90th percentile): Differences in rank sum from GP triage using Mann-Whitney U-test indicating a significant difference from GP triage, **p<0.001.

*Items were rated on a scale from 0 to 10 (0=very low quality, 10=optimal quality).

†Indicates a significant difference in rank sum between all three groups of triage professional, using Kruskal-Wallis, p<0.05.

AQTT, Assessment of Quality in Telephone Triage.

Quality of communication and accuracy of triage

Compared with optimally triaged calls, undertriaged calls were assessed to have significantly lower quality for overall perceived communication and efficiency; this was seen for all specific items (table 7). Moreover, undertriaged calls were significantly longer compared with optimally triaged calls (median length: 230 and 197 s, respectively). Compared with optimally triaged calls, the quality of overtriaged calls was assessed as significantly lower for overall perceived communication and efficiency. Moreover, in overtriaged calls, the quality of all specific items (but not in item 18) were assessed significantly higher compared with optimally triaged calls, but with lower levels of associations.

Table 7.

The quality of communication, length of calls and perceived efficiency for all calls comparing undertriaged and overtriaged calls with optimally triaged calls for all health professionals together

Communicative items (AQTT)	Undertriaged 1–3	Optimally triaged 4	Overtriaged 5–7
	Median (10th and 90th percentile)	Median (10th and 90th percentile)	Median (10th and 90th percentile)
Overall perceived communication†	5 (2 to 8)**	7 (4 to 9)	6 (3 to 9)**
Overall perceived efficiency†	5 (1 to 8)**	8 (3 to 10)	6 (2 to 9)**
Length of call†	230 (98 to 471)*	197 (90 to 426)	198 (81.2 to 427)
12: Gives the caller sufficient time and space to describe the situation†	3 (2 to 4)**	4 (3 to 5)	4 (2 to 5)*
13: Conducts the conversation in understandable language adapted to the caller’s situation†	4 (2 to 5)**	4 (3 to 5)	4 (3 to 5)*
14: Ensures that the triage decision and the advice given are understandable and feasible†	3 (2 to 5)**	4 (3 to 5)	3 (3 to 5)*
15: Ensures that the caller agrees on the triage decision and advice given and is accommodating in case of disagreement†	3 (2 to 5)**	4 (3 to 5)	3 (2 to 5)*
16: Structures the conversation†	3 (2 to 4)**	4 (2 to 5)	3 (2 to 4)**
17: Masters suitable questioning techniques (including suitable use of open-ended, closed-ended and non-leading questions)†	3 (1 to 4)**	3 (2 to 4)	3 (2 to 4)**
18: Summarises (if relevant), verifies and adjusts if needed†	3 (2 to 4)**	3 (2 to 4)	3 (2 to 4)*
19: Pays attention to the caller’s experience and situation†	3 (2 to 4)**	3 (2 to 5)	3 (2 to 5)*
20: Conducts the conversation in an accommodating and friendly tone†	4 (3 to 5)**	4 (3 to 5)	4 (3 to 5)**

Rating scale for item 11 assessing accuracy of triage decision with definitions of each rating: 1. Severe undertriage: the call is undertriaged with risk of severe consequences; 2. Moderate undertriage: the call is undertriaged, but unlikely with risk of severe consequences; 3. Mild undertriage: the call is undertriaged, but could have been triaged ‘somewhat higher’; 4. Optimal triage: the call is optimally triaged; 5. Mild overtriage: the call is overtriaged, but could have been triaged ‘somewhat lower’; 6. Moderate overtriage: the call is overtriaged, it would have been sufficient with a ‘less burdensome service’; 7. Severe overtriage: the call is overtriaged; it seems completely irrelevant to choose this triage outcome.

Perceived accuracy of triage for all health professionals (n=1294): cumulated undertriage (ie, ‘1’, ‘2’ and ‘3’) (n=189), optimal triage (ie, ‘4’) (n=820), overtriage (ie, ‘5’, ‘6’ and ‘7’) (n=223), ‘not applicable’ (n=62).

Median (10th to 90th percentile): Mann-Whitney U-test was used to pairwise compare rank sum between undertriaged versus optimally triage and overtriaged versus optimally triaged, significance level *p<0.05, **p<0.001.

†Indicates a significant difference (p<0.05) in rank sum between calls that are perceived as undertriaged, optimally triaged and undertriaged, using Kruskal-Wallis

Discussion

Principal findings

Our study found that in four items, 20% or more of telephone calls were assessed as having poor communication quality (ie, ‘structuring of the conversation’, ‘mastering questioning techniques’, ‘summarising’ and ‘paying attention to caller’s experience’). Compared with calls triaged by GPs, calls triaged by nurses had significantly lower probability of having poor quality when assessed by the items regarding ‘allowing the caller to describe the situation (10% poor among nurses vs 14% poor among GPs)’, ‘mastering questioning techniques (21% vs 27%)’, ‘summarising’ (21% vs 33%) and ‘paying attention to caller’s experience (17% vs 26%)’. Compared with calls triaged by GPs, calls triaged by physicians with different medical specialities had significantly higher probability of being rated as having poor quality on the five items regarding ‘using understandable language (13% poor among physicians vs 6% poor among GPs)’, ‘ensuring that the caller agrees with the triage decision (13% vs 8%)’, ‘structuring the conversation (32% vs 21%)’, ‘mastering suitable questioning techniques (35% vs 27%)’ and ‘conducting the telephone call in a friendly tone (8% vs 3%)’. Compared with optimally triaged calls, undertriaged calls were longer and assessed as having significantly lower quality on all specific communication items, overall perceived communication and triage efficiency. A similar trend was seen for overtriaged calls.

Strengths and limitations of the study

This study quantitatively compared the quality of important communicative aspects of OOH telephone triage. A major strength was the quasi-experimental design using randomly selected calls in real-life settings. Moreover, the 1294 patient calls were assessed thoroughly for nine communicative aspects by the AQTT, which has high face and content validity.³⁶

This study also had some limitations. Each call was assessed by only one assessor, which could have imposed bias due to misclassification. Two or more assessors, optimally a physician and a registered nurse, were not feasible as the thorough assessment process exceeded >700 hours, but we took several precautions to minimise bias. All assessors followed a thorough training process, and an explicit rating manual was used. Testing of the AQTT revealed satisfactory test–retest reliability, but revealed disagreements when assessing using the entire scale (ie, from 1 to 5). However, when distinguishing poor from sufficient quality the inter-rater agreement was satisfactory. This suggests that (at least) comparisons on the likelihood of poor quality were valid.³⁶

Assessments were made by only GPs and physicians, and our post hoc sensitivity analyses suggested a similar-to-me bias,³⁷ indicating that assessors disproportionately favoured triage professionals who were similar to themselves. In this study, the assessment panel both assessed the quality of communication and the health-related quality for each call.³⁶ Different approaches have been chosen in regard to who assesses the quality. Although nurses have been chosen to assess quality of communication,³⁸ we chose to only include physicians, as this correlates with the strategy of prior studies assessing the health-related quality.^39–41 However, this may be problematic as nurses would never have this favourable assessment and as the distribution of assessors from each setting was uneven (16 GPs from GPC; 8 physicians from MH-1813). Consequently, we adjusted for the similar-to-me bias and for the uneven distribution of assessors. The adjusted RR were generally comparable with the crude estimates, but generally tended to favour the quality of nurse triage (ie, lower RR for all items except item 14) and points towards smaller difference between GPs and physicians (ie, RR closer to 1 for all items except item 14). The use of non-parametric rank sum for the overall perceived quality items did not allow these adjustments. As these items encompass a high level of subjectivity, we assume that adjustment for these factors may have increased differences between the triage professionals. In the study design, we did not have identification of the individual triage professionals. Hence, we cannot reject a clustering effect may be present if the same triage professional appear more than once. However, as both the GPC and Helpline 1813 are large-scale organisations with approximately 765 GPs, 96 nurses and 160 physicians³⁴ conducting the telephone triage, it seems of minor importance.

The observed differences in the quality of communication relating to perceived accuracy of triage could be related to reverse causality. The risk that an assessor who considers a call to be optimally triaged would be more likely to assess the quality as high (and vice versa if undertriaged or overtriaged). Moreover, it is important to notice that the assessed accuracy is based on the perception of the assessor with extensive expertise with telephone triage after listening to the call but without knowing what will happen to the patient after the call. However, we aimed to limit the potential influence as assessments followed the explicit rating descriptions. Additionally, a third variable problem cannot be ruled out. For instance, a reason for encounter, which is difficult to triage, may increase the risk of both inaccurate triage and poor communication. As the reasons for encounters according to ICPC-2 coding did not differ (p=0.46) (see online supplementary file), the risk of this third variable problem seems rather unlikely but cannot be excluded since other factors than reason for encounter have the potential to influence the difficulty of the triage. The proportion of male patients differed between calls triaged by nurses and physicians, but we do not suspect this to impact the quality of communication.

Interpretation and comparison of results

The communication in calls triaged by nurses had lower likelihood of poor quality on four items when compared with calls triaged by GPs, but the quality seemed similar between these two groups when comparing the overall perceived quality of communication. In contrast to the specific items, which follow an explicit rating manual, the overall perceived communication allows subjectivity on a more general level. Thus, the overall perceived quality should be interpreted with caution. A previous study suggested great variation for aspects of communication in GPs.¹⁸ This could also have influenced our results. Owing to the systematic setup for nurse triage with CDSS and strict guidelines,^{25 26} the risk of poor quality may be lower in nurse-led triage compared with GP-led triage as GPs triage without CDSS and thus with little quality assurance. Furthermore, the fee-for-service remuneration of GPs may stimulate faster decision-making and shorter calls in GPs compared with nurses, which may increase the risk of poor quality of communication.¹⁶

The items measuring aspects of a patient-centred approach were generally assessed as more positive (higher scores) than items mainly considering the gathering of information (ie, ‘structure’, ‘questioning techniques’ and ‘summarising’). This suggests that telephone triage is patient-centred to some degree, although prior studies have reported that telephone calls are less oriented towards the patient’s perspective compared with face-to-face contacts.^16–18For two key items assessing patient-centred aspects (‘allowing sufficient time to describe the situation’ and ‘paying attention to the caller’s experience’), nurses had significantly lower risk of poor quality than GPs, which indicates that nurses are more patient-centred. This was supported in a substudy involving 200 of the calls included in this study that nurses allowed the patient to speak uninterrupted in the beginning for significantly longer time than GPs (GP median spontaneous talking time: 17.9 s vs nurse: 23.4 s).²⁸ Previous findings that have found CDSS use constraining and explores mostly medical information.^{26 27} Moreover, indicators of efficiency could influence the patient-centredness of the communication negatively.^{16 18} Although interpreting aspects of efficiency seems difficult and should be done with caution, calls triaged by nurses were longer and perceived as less efficient, which could be because nurses tend to allocate more time to describe the situation or pay more attention to the caller’s experience.¹⁸ However, as physicians with different medical specialities had longer calls and yet lower quality of communication compared with GPs, the length of a call does not appear to be the only factor that influences the quality of communication. The lower quality of communication in calls triaged by physicians with other medical specialities suggests that the type of medical specialty influences the level of communication skills. Family medicine is a medical specialty that focuses on communication, and compared with physicians in internal medicine, physicians in family medicine may engage more in psychosocial discussions and prioritise emotionally supportive exchanges, such as empathy and reassurance.^{42 43} Moreover, it cannot be ruled out that personal characteristics of the triage professional, such as experience, prior communication skill training, age, sex and ability to act emphatic, could also influence the quality of communication.^{18 42} Regrettably, we did not have access to the characteristics of the triage professionals.

Regarding questioning, nurses performed significantly better than GPs, and physicians performed significantly poorer than GPs. CDSS’s have been criticised because it facilitates use of primarily closed-ended questions,^{26 27 38} and limits independent thinking,^{25 26} which holds the risk of missing important information.²⁶ Furthermore, a study by Murdoch et al found that nurses using a CDSS predominantly used closed-ended statements requesting declarative confirmation, whereas GPs used more interrogative statements.²⁷ Our study assessed if the triage professional was perceived to master suitable questioning techniques; this included an appropriate balance of open-ended and closed-ended questions and limited use of leading questions. The nurses in the study by Murdoch et al had an average of only 8 weeks of training for the purpose of the study.²⁷ The nurses from MH-1813 all had a minimum of 6 weeks of introduction and up to 3 years of triage experience from the implementation of MH-1813. This is likely to have made the nurses in our study more experienced, which could account for the differences found between our study and the study by Murdoch et al²⁷ as the use of CDSS might change with increasing experience.²⁴ As experience increases, the triage professional may follow the CDSS less rigorously, which could account for the better questioning techniques among nurses in our study. Moreover, nurses were better at giving the caller sufficient time to describe the situation and providing more relevant summaries than GPs. Prior studies have indicated that allowing the patient to describe their concerns uninterruptedly is related to optimal information gathering.^31–33

Implications for future research and practical implications

Our study revealed that differences exist in the quality of communication depending on whether the telephone triage is conducted by GPs, nurses using a CDSS or physicians with different medical specialities. As we found better communication in calls triaged accurately compared with undertriaged and overtriaged calls, ensuring high quality of communication may reduce the risk of inaccurate triage. However, the causality should be explored in future studies. Emphasis on communication skills could be a way to enhance safety and efficiency. Despite longer calls, nurse-led triage seems a feasible choice if aiming to increase the quality of communication, but efficiency-related aspects must also be considered when organising future OOH telephone triage. Physicians seem to play an inevitable role in OOH telephone triage, but it also seems important to provide sufficient training on telephone triage and appropriate communication for both GPs and physicians with different medical specialities. Future studies could explore whether organisational factors, use of CDSS or efficiency aspects contribute to differences in the quality of communication. Moreover, future studies could explore if factors in the triage professional (eg, empathy, age and sex) or characteristics of the call (eg, time of day, waiting time and reason for encounters) influence the quality of communication.

Conclusion

The quality of the communication in OOH calls triaged by nurses using CDSS was higher than in calls triaged by GPs, but calls triaged by nurses were longer and perceived as less efficient. The quality of the communication in calls triaged by physicians with different medical specialities seemed to be of lower quality and was perceived as less efficient compared with calls triaged by GPs. The quality of the communication may be of importance when aiming to avoid undertriage and reduce overtriage. The knowledge gained on differences in the quality of communication should be considered when organising future OOH telephone triage and could be used in the teaching of future triage professionals to ensure better communication skills.