Optimising telephone triage of patients calling for acute shortness of breath during out-of-hours primary care: protocol of a multiple methods study (Opticall)

Abstract

Introduction

Callers with acute shortness of breath (SOB) are a challenge for telephone triage at out-of-hours primary care (OHS-PC) as SOB could be the sign of a potentially life-threatening disease, yet mostly is a symptom of a broad range of self-limiting disorders. Current telephone triage practice is mainly expert based and clear evidence on accuracy, safety and efficiency of the use of the Netherlands Triage Standard (NTS) by triage nurses based on the eventual clinical outcome is lacking for this domain.

Methods and data analysis

Multiple methods study in five OHS-PC services in the Utrecht region, the Netherlands. Data will be collected from OHS-PC electronic health records (EHR) and backed up tapes of telephone triage conversations, which will be linked to routine primary care EHR data. In cross-sectional studies, we will (1) validate the NTS urgency classification for adults with SOB against final diagnoses and (2) develop diagnostic prediction models for urgent diagnoses (eg, composite endpoint of urgent diagnoses, pulmonary embolism, acute coronary syndrome, acute heart failure and pneumonia). We will develop improvement measures for the use of the NTS by triage nurses through practice observations and semistructured interviews with patients, triage nurses and general practitioners (GPs). In an action research approach, we will, in collaboration with these stakeholders, implement and evaluate our findings in both GP and triage nurse educational programmes as well as in OHS-PC services.

Ethics and dissemination

The Medical Ethics Review Committee Utrecht, the Netherlands, approved the study protocol (protocol 21/361). We will take into account the ‘code of conduct for responsible research’ of the WHO, the EU General Data Protection Regulation and the ‘Dutch Medical Treatment Contracts Act’. Results will be disseminated in peer-reviewed publications and at (inter)national meetings.

Trial registration number

NL9682.

Strengths and limitations of this study.

• We will use multiple methods to assess both the Netherlands Triage Standard (NTS) itself as well as the use of the NTS by triage nurses during clinical practice, which will provide information about the quality of triage of adult callers with acute shortness of breath during out-of-hours primary care (OHS-PC) with the callers’ final diagnosis as the meaningful outcome.

• The sample size of this observational study design (>2500 patients) and event rate will be large enough to validate the existing NTS rules and to develop and validate a new prediction model.

• The risk of work-up bias will be minimised because researchers will be blinded for patients’ OHS-PC electronic health record outcome data.

• Inherent to an observational study design embedded within routine care, missing data incur a risk of bias. We will apply state-of-the-art methods for handling missing data, for example, using multiple imputation techniques.

• By conducting observation studies, interviews and action research in collaboration with professionals and patients, we will optimise implementation in and thereby impact on the clinical practice of OHS-PC.

Introduction

Outside regular working hours, out-of-hours primary care (OHS-PC) provide urgent primary care to ensure 24/7 medical access. In the Netherlands, as in many other European countries, OHS-PC is organised in large-scale cooperatives.¹ Under supervision of a general practitioner (GP), triage nurses initially assess the urgency of the patients’ health problems by telephone and decide whether the patient should be seen by a GP or another medical professional, within which time frame, and which type of contact is needed (immediate ambulance, home visit, face-to-face consultation or telephone advice).² A major challenge is to achieve an adequate balance between safety and efficiency, while dependant on the information acquired in a conversation with the caller. Based on presented signs and symptoms, triage nurses have to discriminate highly urgent medical conditions from non-urgent conditions, and they should come to a decision within a few minutes.^{3 4} ‘Undertriage’ would increase the risk of patient harm by missing acute diseases or delaying urgently necessary treatment, while ‘overtriage’ may jeopardise capacity of medical care, which in turn may also harm patients.⁵

Since 2011, the Netherlands Triage Standard (NTS) has been implemented in the Dutch OHS-PC setting to assist triage nurses in this crucial and critical triage process.^{1 6} The NTS is a semiautomatic computer algorithm that is hierarchically ordered, and the triage nurse should decide based on presented symptoms which of the 56 entrance complaints he/she has to choose. After filling out on average five questions, the NTS automatically gives a recommendation for an urgency level allocation with a corresponding (maximum) response time (see table 1).⁶

Table 1.

NTS urgency levels

NTS urgency level	Definition	Response time	Medical help
U0—resuscitation	Loss of vital functions	Immediately	Ambulance
U1—life threatening	Unstable vital functions	Immediately, within 15 min	Ambulance
U2—emergent	Vital functions in danger or organ damage	As soon as possible, within 1 hour	Home visit by GP or consultation at OHS-PC
U3—urgent	Possible risk of damage, human reasons	A few hours (<3 hours)	Home visit by GP or consultation at OHS-PC
U4—non-urgent	Marginal risk of damage	24 hours	Consultation at OHS-PC or telephone advice
U5—advice	No risk of damage	Advice, no time related	Telephone advice

GP, general practitioner; NTS, Netherlands Triage Standard; OHS-PC, out-of-hours primary care.

The NTS is based on the Manchester triage system developed in and for the emergency department setting,^{7 8} and it may be over-ruled by the triage nurse, with or without consulting the supervising GP.^{2 7 9} However, the accuracy of the NTS has solely, and only recently, been validated against clinically relevant outcomes in the domains’ chest discomfort and neurological deficit.^9–12 So, for other domains, it is still unknown if the urgency allocation is in line with clinical outcomes.

Interestingly, Dutch GPs expressed their concerns about the applicability of the NTS and the quality of triage with the NTS in OHS-PC.^{13 14} Their opinion was that the NTS was too defensive, inflexible, not patient centred and thereby even potentially harmful for their patients.¹⁴ Furthermore, interviews with triage nurses revealed that the NTS may hinder triage, especially in very urgent calls where emotions of both triage nurses and patients are prominent during the telephone conversation.¹⁵ Previous studies from our research group showed that the accuracy of NTS was at the best moderate for patients with chest discomfort or suspected neurological deficits, and that both the NTS itself as working with the tool could be improved in these important domains.11 12 15 16 Thus, it is important to analyse the use of the NTS by triage nurses as well beside the NTS itself. More information about the use of the NTS by triage nurses was obtained by an interview study from our research group. However, patients’ perspectives about the NTS were not taken into account in this study and earlier studies. This could be important since triage calls are routine practice for triage nurses but not for patients or patient representatives, resulting in other experiences during the triage call. Furthermore, all information about the use of the NTS was obtained from interviews, but observations of how triage conversations proceed have never been done. Altogether, studies on the validity, practical use and potential improvements of telephone triage are needed, most urgently in patients with complaints with potentially life-threatening underlying diseases.

Shortness of breath (SOB) has such a broad range of disorders that could underlie patients’ conditions, from life-threatening (eg, pulmonary embolism or acute heart failure) to self-limiting (eg, a mild upper respiratory tract infection).^{17 18} Moreover, it is among the top five of entrance complaints at the OHS-PC and the prime reason for home visits by the GPs generating important workload, while workforce capacity in OHS-PCs is limited.¹⁹ Therefore, adequate telephone triage of SOB is highly relevant in OHS-PC practice.

Telephone triage of SOB, however, is a serious challenge. Patients and their relatives typically use an array of different ways of expressing SOB, such as ‘cramped’, ‘tight feeling on the chest’, ‘not able to breathe properly’, ‘obstructive feeling in the throat’, ‘wheezing’ and ‘not getting enough oxygen/air’.²⁰ Importantly, the perceived severity of SOB is rather subjective, and comparable respiratory distress may be experienced and expressed differently by each individual.^{21 22} These perceived severity of SOB could be influenced by concerns or emotions of fear and anxiety. On the other hand, SOB could cause these feelings as well by arousal of the sympathetic nervous system, leading to concerns or emotions of fear and anxiety.²³

These emotions may complicate the conversation with the triage nurse and thus, subsequently the urgency allocation process.^{24 25} Expressions of the emotions during triage calls may vary from verbalising such as pleading till paralinguistic behaviour such as sobbing or panting. Both could influence interaction during triage calls. In conversations with patients who use verbalised emotions triage nurses often deviate from the usual conversation structure with the potential consequence of missing the question of the caller.²⁶ Simultaneously, triage nurses often try to interrupt these patients with the potential consequence of missing patients’ expressions.²⁶ Patients who express emotions as paralinguistic behaviour do not deviate from the structure. However, these patients are, most of the times, not able to answer on a question asked by the triage nurse.²⁷ Yet, further information on the influence of these concerns on the urgency allocation during triage calls is lacking.

Therefore, there is an urgent need for in-depth analyses of the telephone triage in patients with acute SOB who call the OHS-PC service. In our research project entitled ‘Opticall’, we will provide key answers by conducting multiple quantitative and qualitative studies to assess the performance of the NTS for SOB and the use of the NTS by triage nurses if the patients’ final diagnosis is considered. A translation of the NTS algorithm for SOB in adults is shown in online supplemental file 1. In our analysis, we will pay special attention to callers with concerns or emotions of fear or anxiety. We aim to improve the algorithm’s performance by adding new or removing redundant items as well as the refining the use of the algorithm in practice.²⁸ Also, it is important to test and evaluate these potential improvements in the context of the OHS-PC to make sure that these indeed improve clinical practice before implementing them at all OHS-PC services. Therefore, we will apply an action research approach for short cyclic testing and adjusting improvements in practice on a small scale before implementing these on a larger scale.²⁹ Triage nurses, GPs and patients will participate to ensure that our research findings indeed improve both safety and efficiency of clinical practice. Adopting the action research approach of Andriessen, we will develop generalisable knowledge (‘knowledge stream’) and work on potential improvement measures for triage practice (‘practice stream’) simultaneously, to keep the research questions and the actual problems and feasibility of potential solutions in practice aligned (see figure 1).³⁰

Figure 1.

Flowchart showing the design of Opticall.

In summary, there are several knowledge gaps about telephone triage in patients with SOB, including lack of knowledge about the performance of the NTS algorithm itself, as well as how triagists use this semiautomatic tool. Also, the patient’s perspective in telephone triage is insufficiently considered in previous research. We will assess the NTS and its workability by both quantitative and qualitative studies, and use action research for implementation in clinical practice of our findings.

We have five research objectives:

1. To assess how triage nurses actually use the NTS during telephone triage of adults with acute SOB in OHS-PC.

2. To determine the perspectives of adults with acute SOB regarding telephone triage using the NTS.

3. To validate the NTS urgency allocation against clinically relevant outcomes (pulmonary embolism (PE), acute coronary syndrome (ACS), acute heart failure (AHF), pneumonia, exacerbation of asthma or chronic obstructive pulmonary disease (COPD)).

4. To build new diagnostic algorithms based on the clinical relevant medical outcomes.

5. To implement and evaluate knowledge from (1), (2) and (3) in clinical practice and in both the education of GP trainees and triage nurses.

Methods and data analysis

Design

Observational multiple methods study design, including cross-sectional diagnostic studies, qualitative studies with practice observations and interviews and action research to implement and evaluate our findings (see figure 1).

Setting

The studies will be conducted in five OHS-PC services in the vicinity of Utrecht, the Netherlands (Huisartsenpost Eemland, Huisartsenpost Gelderse Vallei, Spoedpost Utrecht Stad, SpoedzorgNU and Unicum Huisartsenspoedzorg).³¹

Study population

Based on a computer-generated random sequence list, we will include a random sample of 2500 adult patients (aged 18 years and older) for whom the OHS-PC triage nurse selected SOB as entrance complaint in the NTS system in the period 1 September 2020 to 1 September 2022. Our study requires accessibility of recordings of triage conversations as well as follow-up data on final diagnosis of included participants from the electronic health record (EHR) of their own GP. Participants who are enlisted with GPs who do not want to provide follow-up information on the final diagnosis will be excluded from the analyses. A sensitivity analysis will be done to ensure that the participants of these GPs are random and not a specific group of patients.

Data collection and management

Quantitative data

Data collection will take place in both the OHS-PC and routine primary care setting (see figure 2). From OHS-PC, we will extract demographics, medical history, signs and symptoms and triage data from the patients’ EHR and data from backed up telephone tapes. These data are routinely and safely stored in digital environments called Topicus (OHS-PC EHR) and VCare (backup tapes), respectively. Using research-specific accounts, we will capture relevant, pseudonymised data via standardised electronic case report forms (eCRFs) in ResearchOnline, a CE certified data-management system. It meets all Good Clinical Practice guidelines for electronic data collection in terms of protecting data integrity and securing the information collected.

Figure 2.

Flowchart showing the data collection of an included patient in the Opticall Study. GP, general practitioner; OHS-PC, out-of-hours primary care.

These pseudonymised data will be linked to follow-up EHR data retrieved from the patients’ own GP. From the patients’ own GP, we will capture data about the final diagnosis, interventions, hospitalisation and mortality within 30 days of the OHS-PC index contact, by using a standardised eCRF.

Pseudonymised data will be stored within a secured folder of the data management department of the Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht. On completion of the study, data will be stored for at least 15 years at a central drive of the data-management department of the Julius Center and will be made available for the use by third parties on every reasonable request and approval of the research team.

Qualitative data

We will study the actual use of NTS by (1) observing the work processes of triage nurses handling calls about SOB during around 20–40 OHS-PC shifts at two participating OHS-PC centres and (2) interviewing a random sample of patients or relatives who called the OHS-PC for SOB, preferably in an audio stimulated recall design, that is, relistening the triage tape with the patient or relative during the interview.^32–35

For the observation study, data will be collected according to Spradley’s nine dimensions: space, actors, activities, objects, acts, events, time, goals and feelings.³⁶ Therefore, researchers will observe these triage conversations for acute SOB during OHS-PC shifts and describe their observations on field notes during the observations. We try to fill out a field note for every triage conversation for SOB.

For the interview study, we will use purposeful sampling in which we select triage conversations, which stand out for strong emotions, confusing language or moments of miscommunication determined while relistening the backup tapes by researchers.³⁷

We try to interview callers, triage nurses and eventually the involved GP separately to obtain information from several perspectives. Therefore, we will ask participating OHS-PC services to ask callers for SOB and the corresponding triage nurses if they are interested in reflecting on the triage call for research purposes. If callers and triage nurses agree to participate in our study, we will contact them separately for a face-to-face appointment for a 1-hour interview within 2 weeks. So, we interview both callers and triage nurses separately about the same triage conversation. We will start the interview by relistening their triage call via VCare.

For the action research, we will closely collaborate with the stakeholders in daily practice, that is, triage nurses, GPs and patients who were once or more often involved in triage conversations for SOB in a project working group. The ‘knowledge stream’ will contribute to the literature about the organisation of (acute) healthcare, where front-office telephone services and decision support tools play a predominant role.^38–41 The ‘practice stream’ will contribute to tools and educational material for NTS users. Andriessen distinguished different phases in action research that occur in short cycles during the research period, for example, ‘diagnose and action planning’ and ‘action taking and evaluation’.³⁰ In the ‘diagnose and action planning’ phase, available results from our quantitative and/or qualitative studies will be discussed with patients, GPs and triage nurses and interpreted in order to design potential practice improvements. In the ‘action taking and evaluation’ phase, potential improvements from our quantitative and qualitative studies will be iteratively tested and evaluated in short plan–do–study–act cycles up until general transferability to everyday OHS-PC practice.⁴²

In this last phase, improvements from the ‘diagnosis and action planning’ phase will be implemented on a small scale in daily practice. First in a simulation setting and then in a real-life setting. By interviewing NTS users and triagist when this last phase is incorporated, we are able to further fine-tune the adjustments in the NTS and the workability with it. We repeat this step until the most optimal situation is reached according to the patients, GPs, triage nurses and researchers involved.

Sample size considerations

Quantitative data

The prevalence of medically urgent diagnoses in patients who call the OHS-PC service with SOB is not exactly known. A study from the USA showed that these diagnoses occur in approximately 41% of adult patients with SOB who were seen at the emergency department.¹⁸ Assuming a more conservative 35% (event fraction 0.35) in our domain, a minimal sample size of 1523 patients (with 533 events and an events per predictor parameter number of 26.65) for an estimated area under the curve of 0.70 would allow for inclusion of 20 predictors, including interaction terms in our multivariable model with a composite endpoint of medical urgencies.⁴³ Therefore, we have decided to include at least 2500 patients in our study, to allow for multivariable prediction modelling of single urgent medical outcomes, for example, pulmonary embolism.

Qualitative data

For our qualitative studies, we will include sufficient patients to reach data saturation.^{37 44} Based on an earlier interview study in OHS-PC, we expect that we have to include 20–25 patients or their relatives to reach data saturation.¹⁵ In addition, we will perform semistructured interviews with around 20–25 triage nurses about triage calls for SOB in an audio recall design. For our observation study, we will observe clinical practice during 20–40 OHS-PC shifts, depending on the amount of triage calls for SOB.

Data analysis

Quantitative analysis

First, we will determine the diagnostic accuracy (sensitivity, specificity, positive predictive value and negative predictive value) of the NTS urgency levels in patients who call the OHS-PC service with SOB against medically urgent diagnoses (pulmonary embolism, acute coronary syndrome, acute heart failure, pneumonia, asthma or COPD exacerbation) as the reference standard. For pulmonary embolism, acute coronary syndrome and acute heart failure U1 and U2 are deemed correct and U3 to U5 as incorrect. For pneumonia, asthma or COPD exacerbation U2 and U3 are deemed correct and U1, U4 and U5 as incorrect. A general exception independent of the final diagnosis is that a U1 is deemed adequate in patients with severe SOB, that is, impossible to complete sentences without taking breaths in between, with or without signs of autonomous nervous system related symptoms such as transpiration, pale face, nausea or vomiting. In addition, we will assess the diagnostic accuracy of the ‘final’ urgency allocations; triage nurses may overrule the NTS. We will assess whether there is an association between over-ruled cases and expressed concerns by the caller, and whether the expression of concerns is indeed associated with more severe medical outcomes.

Next, we will use univariable logistic regression analyses to assess the association between demographics, medical history, disease-specific and triage data and (1) the actual urgency allocation and (2) composite of severe medical outcomes. Disease-specific data will include items from the NTS as well as other relevant items for severe medical outcomes according to the literature or clinical practice. Next, multivariable logistic regression analysis will be fitted to derive a diagnostic prediction model with a composite of medically urgent diagnoses as reference standard.⁴⁵ We will—depending on whether the number of outcomes is sufficient—consider exploratory and more descriptive multivariable analyses for separate severe medical outcomes (eg, pulmonary embolism, acute coronary syndrome, acute heart failure, pneumonia, exacerbation of asthma, exacerbation of COPD). We will use predefined predictors based on the literature, current NTS triage criteria and clinical reasoning in addition to age and gender. Continuous variables will not be dichotomised, and non-linearity of predictor values will be explored through restricted cubic splines. The effect of differences between males and females will be assessed by forcing sex into the model (combined and interacted with other covariables, eg, age). The overall ability of the model to discriminate between patients with and without medically urgent diagnoses will be quantified using the c-statistic. Calibration of the model will be assessed visually with calibration plots.⁴⁶

Missing data will be studied and appropriate methods for handling them, such as multiple imputation, will be considered. We will exclude variables with more than 50% missings from our analysis. Data will be analysed using SPSS V.25 and R (R Foundation for Statistical Computing, Vienna, Austria).

Qualitative analysis

During analysis of the interviews and field notes of observations, a grounded theory approach is used, which does not use a predetermined framework.⁴⁷ For the analysis of the practice observations and interviews, a tool for qualitative analysis, NVivo, will be used for open coding of both interviews and field notes of practice observations. Two researchers will do this independently from each other. Both researchers will discuss their findings together with a third researcher to reach consensus about the coding of the transcripts. Observation data and interview data will be iteratively analysed in the research team.

During the action research phase, results of qualitative and quantitative studies will be iteratively discussed, interpreted and integrated in the project working group in order to develop improvement plans. These improvement plans will be incrementally implemented and evaluated in the next phase of ‘action taking and evaluation’ at the work places of the participating OHS-PC centres during shifts, or, if necessary for safety or organisational reasons of the participating OHS-PC centres, first in experimental simulation settings.⁴⁸ For the design and development of educational material, crucial cases and triage situations for learning will be identified during interviews before and during improvement activities, but also may be found during relistening of backed up tapes. In close collaboration with GP, speciality training and triage nurse educators of participating OHS-PC centres these cases together with the other results of this study will be used for innovating telephone triage education.

Patient and public involvement

Patients were not directly involved in the design of the Opticall Study. However, they will be included in the interview study and they will participate in the action research working group. In addition, we will organise advisory board meetings for relevant stakeholders, including patient representatives two times per year. Among others, also the NTS editorial board and representatives of the Association of Dutch OHS-PC services will participate in these meetings. During these meetings, the stakeholders are invited to provide input on the conduct of this study and communication plans. This advisory board will be actively involved in all stages of our quantitative and qualitative studies. In addition, they will be involved in reporting our results and will play a key role during action research to implement the evidence in clinical practice.

Ethics and dissemination

Our study protocol has been reviewed by the medical research ethics committee (MREC) (protocol 21/361). If a study concerns medical scientific research and participants are subject to procedures or are required to follow rules of behaviour, it is subject to the Medical Research Involving Human Subjects Act (WMO) and it must undergo a full review by an accredited MREC or the Central Committee On Research Involving Human Subjects (CCMO). The MREC concluded that this study does not fall under the scope of the WMO. Obviously, there are still rules and legislations that have to be taken into account including the ‘code of conduct for responsible research’ of the WHO, the EU General Data Protection Regulation and the ‘Dutch Medical Treatment Contracts Act’. Data will be kept confidential and anonymity will be guaranteed for patient data in our final database for quantitative studies. For the qualitative studies, confidentiality will be guaranteed, and participation is voluntary. Also, written informed consent will be obtained from all participants.

We will share our results with the triage nurses and GPs of the participating OHS-PC services to help interpret our findings. This will also facilitate dissemination of our findings towards a wider audience. Furthermore, research findings will be published in international peer-reviewed journals and will be presented at national and international scientific conferences.

Discussion

In the Opticall Study, we aim to improve both the decision support tool NTS as well as the workability of this tool by triage nurses in the domain of patients who call the OHS-PC service for acute SOB. This multiple method study helps to investigate the triage process from different perspectives. The results of the Opticall Study will help to improve the telephone triage process in this important domain with several potential (urgent) underlying diseases and herewith patients’ outcomes for community people with acute SOB that seek contact with the OHS-PC.

We aim to include sufficient patients to answer both quantitative and qualitative research questions. Because of the close alignment of the study design and execution with everyday OHS-PC practice, we are able to maximise external validity (generalisability) and optimise valorisation of our findings.

Limitations

Inherent to an observational study design embedded within routine care, missing data may occur and these incur a risk of bias. Including patients for which the triage nurse decided to use SOB as the entrance complaint is helpful in defining the study domain, but some callers with SOB maybe missed because another entrance complaint was chosen by the triage nurse.⁴⁹ Importantly, however, triage nurses are trained to choose for the most urgent symptom, such as SOB. Furthermore, missing data could occur when triage calls or follow-up data are not available for some patients. We expect that these cases will be missing at random and therefore will not influence our results. Lastly, missing data could occur because triage nurses did not ask all questions in which we are interested. If this is the case, we will use state-of-the-art methods to handle these missing data. Notably, a strength of the current study is the use of several data sources. Besides using EHR data, which is usually the case in an observational study design, we also use backup tapes of first-contact telephone conversations with patients, resulting in more detailed and accurate information on patients’ signs and symptoms.

In summary, we will provide a multiperspective view on the triage of community people with acute SOP that contact the OHS-PC, and with these results the triage process in this important domain may be improved, both the decision support tool NTS itself and also the use of the NTS by the triage nurse.

Ethics statements

Patient consent for publication

Not applicable.