Limited evidence of benefits of patient operated intelligent primary care triage tools: findings of a literature review

Abstract

Introduction

There is consistent evidence that the workload in general practices is substantially increasing. The digitalisation of healthcare including the use of artificial intelligence has been suggested as a solution to this problem. We wanted to explore the features of intelligent online triage tools in primary care by conducting a literature review.

Method

A systematic literature search strategy was formulated and conducted in the PubMed database and the Cochrane Library. Articles were selected according to inclusion/exclusion criteria. Results and data were systematically extracted and thematically analysed. 17 articles of that reported large multimethod studies or smaller diagnostic accuracy tests on clinical vignettes were included. Reviews and expert opinions were also considered.

Results

There was limited evidence on the actual effects and performance of triage tools in primary care. Several aspects can guide further development: concepts of system design, system implementation and diagnostic performance. The most important findings were: a need to formulate evaluation guidelines and regulations; their assumed potential has not yet been met; a risk of increased or redistribution of workloads and the available symptom checker systems seem overly risk averse and should be tested in real-life settings.

Conclusion

This review identified several features associated with the design and implementation of intelligent online triage tools in a primary care context, although most of the investigated systems seemed underdeveloped and offered limited benefits. Current online triage systems should not be used by individuals who have reasonable access to traditional healthcare. Systems used should be strictly evaluated and regulated like other medical products.

Background

There is consistent evidence that workloads in general practices are substantially increasing and will potentially reach saturation point. There is a clear trend towards increasing consultation rates, consultation durations and patient-facing clinical workloads.¹ In parallel, the workforce is declining.^2–4 This situation is causing major negative effects such as decreasing accessibility of patients to general practices and decreasing patient satisfaction⁵ as well as increasing stress and burn-out for doctors.⁶ Digitalisation has been suggested as a solution to this problem. Increased digitalisation is expected to enable more effective use of health resources and enhanced patient self-management, and is a central part of national health strategies in several countries.^{7 8}

To date, digitalisation has been mainly concerned with the documentation of health visits but it is increasingly being used for communication and consultation between patients and healthcare personnel. There are also great expectations to benefits of self diagnostic or self monitoring devices.

Digitalisation include the possible implementation of Artificial Intelligence (AI) potentially transforming clinical practice. The development and implementation of an easily accessible, online patient-operated triage tool could potentially contribute to easing the pressure on the health system.⁹ We wanted to study the features of intelligent online triage tools in a primary care context by conducting a literature review.

Method

This literature review explored total populations of study practices seeking initial healthcare contact in the context of primary healthcare. A literature search strategy was formulated after further definition of the investigated population, the intervention and the context. A broad spectrum of digital tools currently exists. The field was narrowed down for this review to digital tools that can be accessed online and operated by the patient. Furthermore, the main component of the tools was to be triaging to enhance the patient’s decision on choosing between such outcomes as self-management, seeking acute medical attention or seeking a planned general practitioner (GP) consultation. In addition, the digital tools were to feature at least some level of AI, defined as the theory and development of computer systems able to perform tasks normally requiring human intelligence.¹⁰ Studies dealing with simple triage decision trees were excluded since they did not employ a level of AI along with studies based on tools with a limited scope, such as only skin symptoms. It was anticipated that assessment of the digital tools would involve an unselected population seeking a conventional triage/health professional response or investigation of the diagnostic accuracy of the tool compared with the performance of a real-life doctor. Our aims were to identify the currently available evidence for the efficacy of digital triage tools, the types of tools that have been evaluated, the factors indicating successful use and implementation of AI-powered triage tools and any potential problems or risk factors. Inclusion and exclusion criteria were developed and applied systematically to select articles within this overall frame.

The inclusion criteria were: all studies which investigated an unselected population seeking initial medical advice and using a digital triage tool with some level of AI involvement. The triage tools were to be patient operated. The excluded studies: were published in a language other than English; investigated patients with specific symptoms (eg, chest pain); investigated conventional telephone triage or video-call triage; included the initial decisions being made by health professionals or were carried out in the context of acute or emergency departments.

We conducted a systematic literature search¹¹ to identify the current evidence on the advantages and disadvantages of designing and implementing an intelligent online triage tool in a primary care context. The search strategy is described in detail in online supplementary appendix 1, figure 1 and online supplementary table S1.

Figure 1.

Flow chart article selection.

Results

The search strategy yielded 17 included articles; these varied widely in terms of quality, type, size of study population, methods and conflicts of interest (online supplementary table S2). It was difficult to distinguish clearly between online triage and other features like e-consultation since systems like ‘eConsult’ also contain a built-in triage function where initial decisions are taken by a combination of the user/patient, doctor/nurse and algorithms.¹² The larger observational studies were all multifunctional, with the triage function only one feature among others such as help for the self-management of various conditions and communication platforms with health professionals.^{9 13–15} These articles were all included as long as they included some kind of digital triage tool as defined earlier.

Characteristics of the articles

Five articles described studies based on mixed methods that combined quantitative and qualitative data as well as retrospective and prospective data. Three articles were considered expert opinions. One article described a case report. Four articles described studies of accuracy outcomes. Two articles described observational studies. Two articles were reviews (online supplementary table S2). The articles originated from the USA (n=5), the UK (n=7), Australia (n=4), New Zealand (n=1) and The Netherlands (n=3) and were published between 2001 and 2018. Four articles described the use of clinical vignettes to test the triage tools. Six articles described studies that enrolled real patients. One article described the methods very poorly, and it was not possible to draw conclusions; this article was considered an expert opinion. The studies enrolling real patients had very few subjects who actually used and evaluated the digital tools. One study reported only two e-consultations per 1000 patients per month.¹³ Another study enrolled 13 133 potential online patients and ended up with only 35 patients going through the complete follow-up, and only 20 patients who actually complied with the advice.¹⁵ In a third study that enrolled 80 546 patients, only 6.5% completed the evaluation during a 6-month period.¹⁴ At least three articles had clear conflict of interests, since the authors had invested in the AI tool they were evaluating.^16–18

Categorising the data

The included articles covered the three main aspects of the overall scope: (1) how to design a digital triage tool, (2) how to implement an existing tool and (3) evaluation of diagnostic accuracy. Three articles contributed critical views on the topic.^19–21

The design

Four articles explored the optimal design of an implementable digital AI triage tool.^{16 22–24} One article pointed out that triage tools should be evaluated in realistic situations on a broad set of randomised cases, in contrast to testing symptom checkers using clinical vignettes which do not reflect real-life complexity or the everyday language of a typical patient. Investigation of the symptom checkers with respect to the balance between correctly identifying a disease and risk of missing a critical diagnosis should be of particular focus.²² Symptom checkers were thought to have great potential for improving diagnosis, quality of care, and health system performance worldwide. However, poorly designed systems could potentially put patients at risk and could even increase the load on health systems if they are too risk averse.²² Implementation of evaluation guidelines specific to each symptom checker was found to be very important for facilitating the development and wide-scale use of the system.²²

Another article found that implementing the system ‘Tele-Doc’ resulted in redistribution of work from the doctors to the administrative staff and patients. There was little evidence of any efficiency gains.¹⁶ This system appeared to implement a very low level of AI.

A third article focused on the design of an AI-powered decision support system for patients. The main finding was that much thought should be put into customising the delivery of the system, based on close consultation with the target users and an iterative development process, until the system is accessible and useful. The design of system content should go beyond the traditional emphasis on scientific evidence to establish patients’ perspectives of options.²³

Other findings were that an intelligent triage system must be able to handle uncertainty and gaps in data, as well as subjective descriptions and perceptions of symptoms since data are filled in by patients. It was stated that, in order to work efficiently, it is more important that the correct interpretation is made than that the correct diagnosis is set, initially.²⁴

Implementation

The main focus of six of the articles was on large scale implementing of an existing online digital tool with some level of AI-powered triage for a real life population. The digital tools appeared to be multifunctional with a low level of AI and with access to online GP consultation.9 13–15 25 26 These studies were mostly multimethod observational studies and were designed to explore multiple facets of the overall scopes.

A large multimethod study on the implementation of the digital tool eConsult in 11 practices in Scotland suggested that the workload was not decreased but, in general, that patients who used eConsult felt that they benefited from the service.¹⁴ Factors that would facilitate the implementation included: the presence of a superuser; the inclusion of innovative methods for promoting appropriate use of the tool and the engagement of staff in all areas of the practice. Barriers to the implementation included: delays in system start-ups; marketing not being aligned with practice expectations; challenges in integrating eConsult with existing systems and low numbers of eConsultations. Patients’ perceptions of eConsult were generally positive, particularly because of the ability to use it anytime and the option of having an alternative way of communicating with their GP.¹⁴

A similar study on eConsult in 36 practices in England found that its use was actually very low, particularly at weekends, with little effect on reducing staff workload. Additionally, e-consultations may be associated with increased costs and workloads in primary care. Patterns of use suggested that the design could be improved by channelling administrative requests and revisits separately.¹³

Another UK study evaluated the implementation of WebGP in six practices. During the evaluation period, the actual use of the system was limited, and there was no noticeable impact on practice workloads. Introducing webGP appeared to be associated with shifts in responsibility and workloads between practice staff and between practices and patients. Patients using E-consultations were somewhat younger and more likely to be employed than face-to-face respondents. The motivation for using webGP mostly concerned saving time.²⁵

A large observational study from the Netherlands on a population level evaluated the effect on healthcare usage of providing evidence-based online health information. The study showed that, 2 years after the launch of an evidence-based health website, nationwide primary care usage decreased by 12%. This effect was most prominent for phone consultations and was seen in all subgroups (sorted by sex, socioeconomic status and age) except for the youngest age group. This suggests that eHealth can be effective in improving self-management and reducing healthcare usage in times of increasing healthcare costs.⁹

Another study from The Netherlands concluded that their web-based triage contributed to a more efficient primary care system, because it facilitated the gatekeeper function.¹⁵ Over a period of 15 months, 13 133 individuals used the web-based triage system and 3812 patients followed the triage process to the end. Most commonly (85%), the system advised contacting a doctor but in 15% of the cases the system provided fully automated, problem-tailored, self-care advice.¹⁵

The author had earlier reported that less well educated patients, elderly patients and chronic users of medication were especially motivated to use e-consultation, but these patients also reported more barriers to using the system.²⁶

Accuracy

In four articles, the main focus was exploring the accuracy of AI-powered digital triage tools in diagnosing disorders from clinical vignettes (not real patients) compared with the diagnoses of real doctors or the known correct diagnosis.17 18 21 27 28

The development of an Australian online symptom checker ‘Quro’²⁹ was described in a small study that used 30 clinical vignettes. The accuracy ranged between 83.3% and 66.6%, and 100% of the vignettes requiring emergency care were appropriately recalled.¹⁷ It was concluded that the chatbots could be greatly improved by adding support for more medical features, such as location, adverse events and recognition of more commonly used medical terms.¹⁷

An article from 2016 (USA) described a direct comparison of diagnostic accuracy between 234 physicians and 23 digital symptom checkers. The physicians significantly outperformed the computer algorithms in diagnostic accuracy: 72.1% vs 34.0% (p < 0.001) put the correct diagnosis first and 84.3% vs 51.2% (p<0.001) put the correct diagnosis in the top three listed. In particular, the physicians were more likely to list the correct diagnosis first for high-acuity and uncommon vignettes. Symptom checkers were more likely to list the correct diagnosis first for low-acuity vignettes and common vignettes.²⁸

The same USA-based author reported in 2015 that 23 digital symptom checkers clearly had deficits in both triage and diagnosis interpretations of clinical vignettes. The triage advice from the symptom checkers was generally more risk averse than necessary; users were encouraged to seek professional care for conditions where self-care was reasonable.²⁷ The 23 symptom checkers provided the correct diagnosis first for 34% of the vignettes (95% CI (CI) 31% to 37%).²⁷ Triage performance varied with the urgency of the condition, with appropriate triage advice provided in 80% of emergency cases, 55% of non-emergency cases and 33% of self-care cases. There were wide variations in performance between the algorithms.²⁷

Analysis and thematic synthesis

We identified several advantages and disadvantages associated with the design and implementation of intelligent online triage tools in a primary care context. The results presented above were used to identify key areas of concern.

Features of an intelligent online triage tool

When designing systems for intelligent online triage tools, it is necessary to have a realistic setting for tests and to use an iterative process of development involving trial and adaptation, with the focus on customised delivery of the service.²³ In order to enhance self-help and reduce the strain on the health system, the tool should not be overly risk averse.²² It would also be a major advantage if evaluation guidelines were formulated and implemented, since this would enhance the further development and evaluation of the tools.²² An intelligent triage system must be able to handle uncertainty and gaps in the data as well as subjective descriptions and perceptions of symptoms, since data are provided by patients. Also, for the system to work efficiently, it is more important that the correct interpretation is made than that a correct diagnosis is made, initially.²⁴

Large scale implementation of existing online triage tools

The studies investigating the large-scale implementation of existing online tools found that some major expected advantages were not clearly realised. Several studies found that workloads were not decreased^{14 25} and sometimes costs and workloads were increased.^{13 16} There were limited numbers of users of the online tools but, for some groups like patients with daytime work, access to primary care was improved.²⁵ The nationwide introduction of ‘eHealth’ in The Netherlands reduced primary care usage after 2 years,⁹ and improved self-help by patients. However, 85% of users were advised to seek help from a doctor, even for common symptoms,¹⁵ which could have increased pressure on primary healthcare systems. The main hindrances to use were delays and technical integration problems, which could lead to loss of engagement. The presence of a superuser and innovative methods for promoting its appropriate use would facilitate implementation of the system.¹⁴ Elderly patients and patients with a low level of education could find that a lack of internet skills are a barrier to the use of online systems.²⁶

Diagnostic accuracy of the triage tool

The four studies exploring diagnostic accuracy all used clinical vignettes, thus limiting conclusions on diagnostic accuracy in a real life setting. The chatbot ‘Quro’ was more accurate in suggesting the correct response for emergency cases than in making correct diagnoses.¹⁷ The direct comparison of physicians and digital symptom checkers found that physicians outperformed algorithms in diagnostic accuracy. The symptom checkers had difficulty in interpreting the vignettes with respect to both triage and diagnosis. Triage advice from the symptom checkers was generally risk averse and inappropriate for many of the vignettes. It was suggested that physicians should be aware that patients may be using online symptom checkers and that algorithms could be improved by adding more features such as locations, adverse effects and recognition of more commonly used medical entities.¹⁷ The evidence on diagnostic accuracy was considered sparse since the studies were vulnerable to bias.^{19 22}

Discussion

This review identified several advantages and disadvantages associated with the design and implementation of intelligent online triage tools in a primary care context, although most of the investigated systems were under-developed and offered limited benefits.

Techno-optimism?

In general, most articles were very optimistic about designing and implementing intelligent online triage tools, predicting potential for substantial advantages even when the systems were performing badly. The disadvantages, such as overly risk-adverse systems, poor diagnostic accuracy and increased workloads, were seen as obstacles that could be overcome, which were being identified in order to improve the next-level algorithms. This may reflect a common techno-optimistic point of view which can bias investigations. On the other hand, it is evident that there is consensus on many levels that ongoing technical evolution is a central aspect of the solutions to current challenges of the healthcare system.^{4 26}

Limited evidence

The identified articles did not provide substantial evidence of the efficacy of digital triage systems since there were only a limited number that evaluated real life implementation and diagnostic accuracy. Furthermore, several weaknesses were found. The multimethod studies included very few patients who actually used and evaluated the systems, making it hard to generalise the results and to gain an idea of the actual effects of using an integrated system. It appears that the concepts investigated were immature and possible effects could have been underestimated. The only articles focusing on system design were expert opinions. Studies on diagnostic accuracy only included clinical vignettes and were therefore not directly comparable to the complexity of advising a real-life patient using natural language. These findings are comparable to those of other studies.²⁰ A recent review on the topic, including patients with specific symptoms, found that research examining the accuracy of triage platforms is limited and that extensive research is needed, keeping in mind that some platforms are designed for a wide range of conditions and others are more specialised.³¹

Risk aversion and lack of regulations

Evaluations of triage systems should of course focus on safety but, in order to actually improve self help and ease pressure on primary healthcare systems, it is essential that the systems are not too risk averse. Exaggerated risk aversion appears to be a common occurrence for many automated symptom checkers. Users were frequently asked to seek emergency care, sometimes regardless of the type of reported symptoms.^{18 27} It is a concern that the combination of patients’ confusion, cyberchondria and risk-averse triage advice might result in patients unnecessarily seeking physical care. It has been suggested that future research should focus on understanding how patients interpret and use the advice from digital tools and on understanding the impact of digital tools on care seeking.²⁷ Existing online symptom checkers are performing badly, in terms of both poor diagnostic accuracy and overly risk-averse advice. They are therefore considered to be unsafe and to promote a demand for unnecessary physical care. The results of this review accentuate the current lack of regulations and evaluation guidelines which could facilitate the development of next-level systems and enable users to choose between platforms. This is in agreement with a similar study, which suggested that the interaction of users with technology should be investigated.³¹

Increase in workload

There was no clear evidence that the implementation of digital triage tools had the proposed effect of decreasing workloads in the primary care setting. Instead, the results indicated that the current tools had a tendency to increase the workload and to shift it to other staff members. A possible explanation for this outcome was the very low number of patients who have actually used the systems.

There were some indications of positive outcomes, however, with some users experiencing increased accessibility to advice¹⁴ and another study⁹ finding a substantial decrease in the primary care workload on a national level that was possibly the result of introducing a national online advice system. This study concluded that online advice systems can be effective in improving self-management and reducing healthcare usage in times of increasing healthcare costs.

Strengths and weaknesses of this review

This review was based on a systematic study of two major databases. Data were transparently and thoroughly extracted from different kinds of studies aiming to provide an overview of the current evidence on the use of online triage tools. In addition to the included studies, numerous studies focused on specific medical conditions or on patients only in an emergency department context; additional information could have been added if these studies had been included. Literature studies are limited in terms of discovering new evidence since they mainly considerer what has already been described. We might have failed to find key evidence. This paper treats primary care contexts in Western countries homogeneously although they might differ in many aspects. Transferability should be considered when evaluating health interventions in different contexts.³²

Recommendation for the use of online triage

These results indicate that current online triage systems should not be used by members of the public who have reasonable access to traditional healthcare. The risk of misdiagnosis or inadequate advice is considered too high. Current digital systems integrated in primary practices also seem undeveloped with respect to decreased workloads, and implementation of current systems including additional resources as required should be considered only a part of the process of developing tomorrow’s digital systems. Nonetheless, there is some evidence that easy online access to high quality intelligent medical information could decrease healthcare consumption overall.

Suggestions for improvement and future research

Digital medical triage tools should be strictly evaluated and regulated like other medical products, since consumers are unable to judge the quality of the tools and the potential for harm is great. Because evidence is sparse and could be conflicted by economic interests, research should be independent. Official evaluation guidelines should be formulated and used to regulate the market. Resources should be allocated for exploring and implementing new systems in order to uncover areas where AI-powered triage could lift the accessibility and quality of healthcare and decrease the associated workloads. Commitment to rigorous initial and ongoing evaluation will be critical to ensuring the safe and effective integration of AI in complex sociotechnical settings.³³

The diagnostic accuracy of and adequacy of advice on digital triage tools should be independently evaluated on real-life patients. Multimethod studies on real-life implementation of triage systems should be conducted in a context where the tools are actually used by the majority of patients. Reviews, including data from emergency departments and limited-range symptom checkers, should be conducted.

Conclusion

This review has identified several advantages and disadvantages of designing and implementing intelligent online triage tools in a primary care context, although most of the investigated systems appeared to be underdeveloped and to offer limited benefits. It appears that only limited evidence is available on the actual efficacy and performance of existing tools, but studies have uncovered several aspects that could guide further development. These aspects can be categorised into concepts of system design, system implementation and diagnostic performance. The most important findings included the need to formulate evaluation guidelines and regulations, the evidence that the assumed potential of the tools has not yet been met, the risk of actually increasing workloads, the apparent redistribution of existing workloads, and the overly risk-averse symptom checker systems that require tested in real-life settings.