Optimisation of telephone triage of callers with symptoms suggestive of acute cardiovascular disease in out-of-hours primary care: observational design of the Safety First study

Abstract

Introduction

In the Netherlands, the ‘Netherlands Triage Standard’ (NTS) is frequently used as digital decision support system for telephone triage at out-of-hours services in primary care (OHS-PC). The aim of the NTS is to guarantee accessible, efficient and safe care. However, there are indications that current triage is inefficient, with overestimation of urgency, notably in suspected acute cardiovascular disease. In addition, in primary care settings the NTS has only been validated against surrogate markers, and diagnostic accuracy with clinical outcomes as the reference is unknown. In the Safety First study, we address this gap in knowledge by describing, understanding and improving the diagnostic process and urgency allocation in callers with symptoms suggestive of acute cardiovascular disease, in order to improve both efficiency and safety of telephone triage in this domain.

Methods and analysis

An observational study in which 3000 telephone triage recordings (period 2014–2016) will be analysed. Information is collected from the recordings including caller and symptom characteristics and urgency allocation. The callers’ own general practitioners are contacted for the final diagnosis of each contact. We included recordings of callers with symptoms suggestive of acute coronary syndrome (ACS) or transient ischaemic attack (TIA)/stroke. With univariable and multivariable logistic regression analyses the diagnostic accuracy of caller and symptom characteristics will be analysed in terms of predictive values with urgency level, and ACS and TIA/stroke as outcomes, respectively. To further improve our understanding of the triage process at OHS-PC, we will carry out additional studies applying both quantitative and qualitative methods: (i) case-control study on serious adverse events (SAE), (ii) conversation analysis study and (iii) interview study with triage nurses.

Ethics and dissemination

The Medical Ethics Committee Utrecht, the Netherlands endorsed this study (National Trial Register identification: NTR7331). Results will be disseminated at scientific conferences, regional educational sessions and publication in peer-reviewed journals.

Strengths and limitations of this study.

• We will assess the accuracy of the telephone triage in the out-of-hours services in primary care setting in the domain of patients with symptoms suggestive of acute cardiovascular disease.

• We will apply the actual clinical outcome as the reference, not just ‘expert opinion’.

• New insights retrieved by our ‘holistic approach’ will help improve both patient safety and efficiency.

• Use of routine care data and therefore missing data on some variables of interest.

• Routine care diagnoses and therefore the risk of missing some cases with an unrecognised acute cardiovascular disease.

Introduction

Digital decision support systems for telephone triage are widely used, also in primary care out-of-hours settings. For these settings, telephone triage systems were derived from triage systems applied in emergency department (ED) settings. An important difference, however, is that telephone triage systems do not include items related to face-to-face contact and physical examination.¹ In the Netherlands, a digital decision support system for telephone triage called the ‘Netherlands Triage Standard’ (NTS) was introduced in 2011.² The NTS was constructed by an expert panel using (i) the Dutch national telephone guidelines for office hours in family practice (‘NHG triage index’) and (ii) a modified version of the Manchester Triage System (MTS) developed in the ED setting.³ Since its introduction in 2011, the NTS has been used by most out-of-hours services in primary care (OHS-PC), and by approximately half of the ambulance dispatch centres in the Netherlands.⁴ An important argument for using the same system within the ‘critical care chain’ was that it would facilitate communication during care transitions, and potentially improve transitional safety.⁵

Many studies assessed the diagnostic accuracy of triage systems in ED settings.^6–9 Yet, comparable studies on validity of telephone triage systems in primary care settings are limited.⁵ The validity of the five-level triage system for physical triage in ED settings was evaluated in a systematic review of 57 studies.¹ Various reference standards for the assessment of validity were used; researchers’ own creation of a reference standard or combining severity with resource measures, for example, hospital admissions.¹ Only one of the included studies in this review assessed the validity of the NTS, for both physical triage in the ED and for telephone triage in the OHS-PC.⁵ For the assessment of telephone triage in OHS-PC, 6668 patients were evaluated. Referrals to the ED and self-care advices were the two surrogate markers used as best proxies for adequate urgency allocation. Consequently, patients referred to the ED were considered as being correctly labelled as high urgency and those receiving a self-care advice as correctly low urgency, independent of the final diagnosis related to the episode. Not surprisingly, the researchers found an association between high urgencies and ED referrals and between low urgency level allocation by the triage nurse and self-care advices. Information on whether the urgency allocation was in line with the clinical outcomes is lacking.⁵ There are no other studies available that assessed the validity of the NTS for telephone triage in OHS-PC settings or validated the NTS to the final diagnosis instead of to surrogate markers.^{5 10} Our study addresses this current knowledge gap.

Over the last decade, out-of-hours primary care in the Netherlands has been reorganised from small practices into larger OHS-PC.⁴ Patients primarily contact the OHS-PC when they need medical help outside office hours. In case of life-threatening symptoms, however, patients can also directly contact the emergency number of the ambulance dispatch centres or directly visit the ED. At the OHS-PC, triage nurses trained in using the NTS handle all incoming calls, under supervision of a general practitioner (GP).⁴ The goal of this telephone triage is to identify callers with the most urgent need of care.¹¹ With urgent clinical situations in mind, the experts formulated the NTS questions in a hierarchically ordered algorithm; questions focusing on the symptoms and circumstances considered most critical first, followed by questions about less critical symptoms.² The triage nurse fills out the caller’s responses in the semi-automatic NTS system, which then automatically generates urgency allocations. Within the NTS, six possible urgency levels can be distinguished that are linked to the response time within which a caller should receive medical help (table 1). The urgency level generated by the NTS can be adjusted by the triage nurse if the nurse disagrees with the NTS allocation (upgrading or downgrading).¹¹ Most often such adjustments are preceded by consultation of the supervising GP, who has the final responsibility for the urgency level allocation.

Table 1.

NTS levels of urgency^{2 5}

NTS urgency level	Definition	Response time	Medical help
U0—Resuscitation	Loss of vital functions	Immediately	Ambulance
U1—Life threatening	Unstable vital functions	Within 15 minutes	Ambulance
U2—Emergent	Vital functions in danger or organ damage	As soon as possible, within one hour	Home visit by GP or appointment at OHS-PC
U3—Urgent	Possible risk of damage, humane reasons	A few hours (within three hours)	Home visit by GP or appointment at OHS-PC
U4—Non-urgent	Marginal risk of damage	Within 24 hours	Appointment at OHS-PC or telephone advice
U5—Advice	No risk of damage	Advice, no time related	Telephone advice

GP, general practitioner; NTS, Netherlands Triage Standard; OHS-PC, out-of-hours services in primary care.

The aim of the NTS is to guarantee both efficient and safe care. Questions have been raised, however, about the current efficiency.⁴ A national report showed an increase in high urgency allocations after the implementation of the NTS at the Dutch OHS-PC, notably U2 urgencies.¹² This is in contrast to the unchanged distribution of diseases, and suggests an increase of a more defensive triage strategy. In addition, a survey in 2016 among GPs revealed that the large majority (83.9%) believed that telephone triage with NTS resulted in (i) many unnecessary consultations and home visits in patients with non-urgent problems, and (ii) a high workload at the OHS-PC.¹³ The authors of this survey speculated that overconsumption of care at the OHS-PC was attributable to certain characteristics of the users of the NTS.¹¹ Previous studies on OHS-PC telephone triage in the UK described that the clinical background of triage nurses, the range of their experience, their gender and their attitudes to risk did not affect the triage decisions made.^{14 15} More insight in the users of the NTS (triage nurse) could help improve the system. Yet, it is currently unknown how triage nurses actually use the NTS in their clinical reasoning process and how this affects the performance of the triage system. Also, media coverage of (fatal) adverse events has grown, and adverse event reports and investigation by the Dutch Health and Youth Care Inspectorate have been made publicly available.¹⁶ The increasing attention for (fatal) adverse events might possibly be related to more defensive triage to avoid such (fatal) adverse events, which could potentially worsen efficiency. Thus, thorough research on the accuracy and use of the NTS is needed in order to assess whether improvements in the balance between efficiency and safety are required.

Callers with symptoms suggestive of acute cardiovascular diseases, for example, acute coronary syndrome (ACS), or transient ischaemic attack (TIA)/stroke generally receive high urgency allocations.¹⁷ Chest pain is the most common reason for dispatching an ambulance (U1) with 60.7% of all ambulance rides deployed from the OHS-PC.¹⁸ However, the incidence of ACS among patients experiencing chest discomfort in Dutch primary care settings and who are referred to the hospital is low with around 10%.^19–21 The trade-off between efficiency and patient safety of telephone triage is particularly challenging in these cases. On the one hand, prompt detection of ACS or TIA/stroke is crucial to initiate interventions as early as possible and prevent myocardial or brain necrosis, and improve prognosis.^22–24 On the other hand, overestimation of urgency results in unnecessary high workload, high referral rates, high costs and potentially iatrogenic damage of the caller.²⁵ The discussion about balancing efficiency and safety is especially complicated because missing an acute myocardial infarction (AMI) is the most common reason for malpractice claims worldwide.^26–28

Our ultimate goal is to improve the efficiency and safety of the triage of callers with symptoms suggestive of ACS or TIA/stroke who contact the OHS-PC, and to improve diagnosis. In the Safety First study, we will evaluate real-life telephone triage recordings with corresponding urgency allocations, and patients’ final clinical outcomes. Our research questions are (i) what is the diagnostic accuracy of the urgency allocation in callers with symptoms suggestive of ACS or TIA/stroke, (ii) which (clinical) determinants are predictive of an ACS or TIA/stroke diagnosis and (iii) how is the NTS triage system used and how can it be improved in view of balancing efficiency and safety.

Methods and analysis

Design

An observational study in which telephone triage recordings from nine OHS-PC will be evaluated against the clinical outcomes. The researchers will listen to 3000 real-life recordings, and collect information on the medical history (eg, patient’s history of heart failure, hypertension, ACS, TIA/stroke, diabetes, etc), caller and symptom characteristics (eg, onset, duration and severity of symptom(s), additional symptoms (eg, transpiration, nausea, vomiting, pale skin), pain score in case of chest discomfort, etc) discussed between the caller and triage nurse during the telephone triage conversations. We will also collect call characteristics (ie, time of calling, duration of the call and who calls) and triage information such as the urgency allocation, available from the registered notes of the OHS-PC (ie, notes from both triage nurses and the supervising GP). All information will be collected on an electronic case record form. Because there is no national Dutch database containing information about diagnoses, we will ask the callers’ own GPs to provide the final diagnosis related to the OHS-PC contact, which is based on the patient’s medical record, and includes medical specialist’s and hospital discharge letters.

Setting

The foundation ‘Primair Huisartsenposten’ is a collaboration of nine different OHS-PC locations in the central part of the Netherlands that provides out-of-hours primary care for approximately 1.5 million residents. It is one of the largest OHS-PC collaborations in the Netherlands and it covers both rural and urban areas. Every OHS-PC location has its own telephone triage centre.

Data collection

All telephone calls are routinely recorded and archived for training, quality control and research purposes in a computer program called ‘Callmanager’ (ie, electronic patient record of the OHS-PC). Archiving is based on the International Classification of Primary Care (ICPC) code.²⁹ ICPC codes allow for classification of symptoms and diagnoses in the electronic patient records. The standardisation of the coding facilitates (inter)national comparison of primary care diagnoses. At Dutch OHS-PC, both triage nurses and GPs assign ICPC codes to patient contacts manually. The study sample will be derived retrospectively from all triage recordings in the years 2014–2016 and will contain calls linked to the ICPC codes reflecting our study domain: symptoms suggestive of ACS (ie, K01, K02, K03, K24, K74, K75, K76, K77, K93, L04, P74, R02, R98) and symptoms suggestive of TIA/stroke (ie, K89, K90, N17, N18, N19, N29, N89, N91). The distribution of sampling of ICPC codes within our study was based on the actual distribution of ICPC codes at the OHS-PC in 2014, and thus best reflects current practice. Per ICPC code a random sample will be taken.

Inclusion and exclusion criteria

Telephone triage recordings of callers of the OHS-PC with complaints that lead to suspicion of ACS or TIA/stroke will be included. The selection of recordings is based on (i) specific ICPC codes and (ii) a selection of keywords used in Callmanager (figure 1). As the specific ICPC codes may not cover all eligible callers due to inter-rater variability as a result of ICPC coding by different triage nurses and GPs,^{30 31} we choose to select cases on ICPC codes, and on keywords in the free text, in order to include a broad but nevertheless clinically realistic range of patients with possibly typical and less typical symptoms that could be considered to be caused by underlying ACS or TIA/stroke. A major exclusion criterion is other recordings than triage conversations, for example, calls between colleagues. All exclusion criteria are displayed in table 2.

Figure 1.

Flow chart of the Safety First study. ACS, acute coronary syndrome; GP, general practitioner; ICPC, International Classification of Primary Care; OHS-PC, out-of-hours services in primary care; TIA, transient ischaemic attack.

Table 2.

Inclusion and exclusion criteria for participation in the Safety First study

Inclusion criteria

Exclusion criteria

Recordings of triage conversations of callers with symptoms suggestive of ACS based on (i) the following ICPC codes in ‘Callmanager’: K01, K02, K03, K24, K74, K75, K76, K77, K93, L04, P74, R02, R98 and (ii) keyword selection: thoracic pain, chest pain, myocardial infarction, heart attack and their common abbreviations. Recordings of triage conversations of callers with symptoms suggestive of TIA or stroke based on (i) the following ICPC codes in ‘Callmanager’: K89, K90, N17, N18, N19, N29, N89, N91 and (ii) keyword selection: TIA, stroke, cerebral or brain bleeding or haemorrhage or infarction, neurological deficit, arm or leg weakness, face or mouth drooping, speech or visual problems or sensory disturbances.

No triage conversation, but consultation between colleagues or questions about medication use. Poor quality recordings. Callers younger than 18 years. Callers not living in the catchment area of the OHS-PC. Callers enlisted with a GP who refused to provide information on the final clinical outcome.

ACS, acute coronary syndrome; GP, general practitioner; ICPC, International Classification of Primary Care; OHS-PC, out-of-hours services in primary care; TIA, transient ischaemic attack.

Data analysis

1. The diagnostic accuracy of the urgency allocation on two separate outcomes: ACS, or TIA/stroke, will be calculated in terms of sensitivity, specificity, positive and negative predictive values. In addition, we will perform sensitivity analyses: (a) also considering other emergency cases with chest discomfort, eg, pulmonary embolism, acute heart failure and thoracic aortic dissection, or in the domain symptoms suggestive of TIA/stroke; (b) other emergency cases with neurological symptoms, eg, subarachnoidal haemorrhage, epilepsy.

2. Multivariable logistic regression analysis will be applied to identify history items or caller characteristics that independently predict either ACS or TIA/stroke. In addition, multivariable models will be developed that predict the presence of respectively ACS, or TIA/stroke, and the C-statistic with the 95% CI will be calculated after bootstrapping to correct for overoptimism.³² Before multivariable analysis, multiple imputation techniques will be applied for missing values. Variables with >50% missings will not be considered. Variable selection will be based on literature review (known diagnostic predictors in both domains) and on univariable analysis (inclusion of variables with p<0.15). Special attention will be paid to gender and age, either as interaction term or by calculating gender and/or age-specific prediction models. All data analyses will be performed using SPSS statistical software program V.25.0.

Blinding of researchers

All researchers listening to the recordings, and filling out the electronic case record forms or who are otherwise involved in the Safety First study, for example, the expert panel of one of our additional studies, will be blinded to the callers’ clinical outcomes.

Power calculation

For the power calculation, we used the recommendations that apply to diagnostic studies.²¹ Records of 2014–2016 were used and after application of the exclusion criteria two random samples of 2000 recordings with ‘symptoms suggestive of ACS’ and 1000 recordings with ‘symptoms suggestive of TIA/stroke’ will be drawn. A prior pilot study showed that one out of nine callers with symptoms suggestive of ACS is, after diagnostic workup, diagnosed with ACS and one out of two callers with symptoms suggestive for TIA/stroke is diagnosed with TIA or stroke. Based on Harrell’s rule of thumb, for ACS 2000 audio-tapes allow for the analysis of 22 variables and for TIA/stroke 1000 audio-tapes allow for the analysis of 50 diagnostic items. We speculated that more variables would be needed to evaluate suspected TIA/stroke cases than patients in the domain ‘suspicion of ACS’ because the former patients may present themselves with a larger variety of symptoms.³³

Additional studies

We explicitly chose to describe the observational cross-sectional study in more detail and not the other three studies because it is the largest study within the Safety First project, and for readability. As part of the Safety First project, three additional but smaller studies will be carried out to further expand the knowledge about the triage process at OHS-PC.

A case-control study on serious adverse events (SAE)

A SAE is defined as an unintended or unexpected event related to the quality of care and resulting in death or a severe harmful event for the patient.³⁴ By Dutch law, root cause analysis (RCA) of every single SAE is required. RCA often points at flaws in the triage process as root causes for such SAEs,³⁵ resulting in an array of improvement measures for daily practice. However, it is unknown whether cases that result in a SAE really differ from those not evolving into a SAE, as all only have been evaluated as individual cases by experts knowing they evaluated a SAE and their assessment is thus prone to hindsight bias.^{36 37} In OHS-PC settings, SAEs are very rare (0.006%) and the majority is of cardiovascular origin, eg, missed AMI.³⁵ The aim of this additional study is to provide a more thorough view on missed or too late diagnosed AMIs that resulted in SAE by comparing these cases with matched controls in the domain of callers with chest pain, either due to ACS or not. The main question is whether the SAE cases actually differ from controls with respect to call and patient characteristics. For this additional study, we will collect new data (SAE cases) and use data from the larger observational study described earlier for sampling our controls. Archived triage conversations from the period January 2013 to December 2017 of the cSAE cases (defined as a missed diagnosis of AMI) will be compared with triage conversations of controls not meeting the SAE definition and contacting the OHS-PC with symptoms suggestive of ACS. Per SAE case, eight controls will be matched to the case according to age, gender and symptom presentation. A researcher blinded to the outcome will extract information regarding call characteristics, caller and symptom characteristics (including medical history) and urgency allocation, by relistening the archived triage calls and by using the registered information. All data will be inserted in a database. An expert panel of experienced GPs blinded to the final outcome (SAE or no SAE), thus without case-specific hindsight bias, will evaluate the quality and safety of triage conversations, and the diagnostic accuracy (sensitivity, specificity, predictive values) of safe triage handling according to the expert panel will be calculated against the final outcome (SAE or no SAE).

Conversation analysis study

In this additional study, we will assess how conversation aspects may affect the urgency allocation during telephone triage. The ways in which obligatory NTS questions and other questions are formulated by the triage nurse (ie, suggestive questions), and responded to by the conversation partner may provide more thorough insight into question-answer sequences at the OHS-PC that can be helpful to improve the (use of) the NTS. We will transcribe a sample of the conversations from the case-control study following the Jefferson conventions. These transcripts that can be considered as new data will then be qualitatively analysed with established conversation analysis techniques.^38–40 We will specifically focus on either/or questions and their various formats (eg, two-choice, multiple-choice, open-ended or tag format) in the urgency decision phase of the triage calls.

Interview study with triage nurses

Finally, we will perform a qualitative interview study of triage nurses with the aim to gain more knowledge on triage nurses’ reasoning during telephone triage of callers with symptoms suggestive of ACS or TIA/stroke, and how they use the NTS for decision making in these two domains. We will conduct semi-structured audio-stimulated recall interviews.⁴¹ Per interview one or two recordings from a recent shift at the OHS-PC will be used to stimulate triage nurses’ reflections. The interviews will be transcribed and the transcripts will be qualitatively analysed according to the principles of the grounded theory approach.⁴²

Patient and public involvement

Patients were not directly involved in the design of this study, but representatives participate in the stakeholders meetings every six months together with other representatives, that is, of the local and national OHS-PC organisation and the Dutch Health and Youth Care Inspectorate. They provide crucial input that helps us formulating additional, clinically relevant research questions.

Ethics and dissemination

All personal data, as well as all research data were stored anonymised according to the European General Data Protection Regulation. Data collection started in January 2016 and will last until December 2020. Communications and publications on the results will not enable identification of individual callers. Results will be disseminated to relevant primary care communities in peer-reviewed journals, and at scientific conferences. In addition, this research project will also result in two PhD theses.

Discussion

The Safety First study will provide diagnostic accuracy data on ACS and TIA/stroke in primary care settings. The efficiency and safety of the NTS in the OHS-PC will be evaluated for callers with symptoms suggestive of ACS or TIA/stroke. In a case-control study, we will explore SAEs (missed AMIs) and assess whether they differ from controls (other patients with symptoms suggestive of ACS but not fulfilling the SAE criteria) with respect to call and patient characteristics. With qualitative studies, we will generate more thorough knowledge on how triage nurses work and how the NTS is used.

Efficiency and patient safety

A systematic review about patient safety in OHS-PC settings concluded that 10% of all OHS-PC calls could be considered as ‘unsafe’.⁴³ However, the claim of ‘unsafe’ calls was based on surrogate markers, and not on patients’ clinical outcomes, and clearly contrasts with the very low incidence of SAEs(0.006%).³⁵ By using the patients’ final clinical outcomes as the reference, the Safety First study will address an important knowledge gap on diagnostic accuracy of telephone triage. In addition, our study will provide a more complete view on the patient safety of the NTS than can be achieved with RCA of individual cases only, often resulting in the conclusion that the urgency level is underestimated.^{35 44} It is difficult to define when telephone triage at the OHS-PC is safe. Our case-control study will provide deeper knowledge on telephone conversations that resulted in SAEs and their matched controls. This knowledge might put the general view on analysing SAEs and the weight that is assigned to improvement measures in a different perspective.

Users of the NTS

A questionnaire among Dutch GPs working at the OHS-PCs reported that a majority of GPs (83.9%) believe that nowadays telephone triage contributes to overconsumption of care at OHS-PC. About 85% of responders consider this to be at least partly attributable to the level of experience and education, and the attitude and personality of the triage nurses.¹³ As the performance of the NTS is inextricably linked to its users, we want to include conversation analysis and interview studies with triage nurses. The quality of telephone triage is highly dependent on the quality of communication between the triage nurse and the caller, and therefore the communication skills of triage nurses.^{27 45} Moreover, they have to deal with callers who greatly vary in their ability to communicate effectively and coherently.^{46 47} Additionally, the telephone triage is necessarily only based on oral information provided by the caller, lacking any visual cues or diagnostic information from physical examination. Our additional study on conversation analysis will add knowledge on the ways in which questions and answers are formulated by triage nurses and callers, and how this affects the triage conversation.

Second, we will also focus on the obligatory question structure within the NTS, and how this affects the users. The NTS is a semi-automatic support system that is heavily dependent on the input of the triage nurses themselves; which ‘main presenting problem’ he/she chooses is based on the initial information from the caller, and each of the total 57 ‘presenting problems’ are linked to a separate algorithm of hierarchically organised, prespecified questions. These questions need to be filled out before the NTS generates an urgency allocation.⁴⁸ Previous studies suggested that triage nurses may experience interactional dilemmas while working with a computer decision support.⁴⁹ Our interview study will provide more insight into triage nurses’ clinical reasoning process during telephone triage, and how they use the NTS for decision making within the domains of callers with symptoms suggestive of ACS or TIA/stroke. This information can help improve the workability of NTS in daily practice. Furthermore, in these domains the knowledge that ‘time is muscle’ or ‘time is brain’ puts extra strain on the triage nurse, because unnecessary delays directly affect morbidity and mortality of patients who actually show to have ACS or TIA/stroke.^{22 23} Beside this stress factor, other factors such as shiftwork, fatigue, multitasking and understaffing can attribute to the risk of errors during the triage process.²⁷ In our interview study we want to include these risk factors, and their possible influence on the reasoning process of triage nurses.

Conclusion

Our Safety First study will provide diagnostic accuracy data on ACS and TIA/stroke, and new insights on efficiency, patient safety and the use of the NTS by triage nurses during telephone triage of callers with symptoms suggestive of ACS or TIA/stroke at the OHS-PC. Results from our study will help to improve the telephone triage in these important domains and improve both efficiency and safety for callers.