Reducing diagnostic errors in primary care. A systematic meta-review of computerized diagnostic decision support systems by the LINNEAUS collaboration on patient safety in primary care

ABSTRACT

Background: Computerized diagnostic decision support systems (CDDSS) have the potential to support the cognitive task of diagnosis, which is one of the areas where general practitioners have greatest difficulty and which accounts for a significant proportion of adverse events recorded in the primary care setting.

Objective: To determine the extent to which CDDSS may meet the requirements of supporting the cognitive task of diagnosis, and the currently perceived barriers that prevent the integration of CDDSS with electronic health record (EHR) systems.

Methods: We conducted a meta-review of existing systematic reviews published in English, searching MEDLINE, Embase, PsycINFO and Web of Knowledge for articles on the features and effectiveness of CDDSS for medical diagnosis published since 2004. Eligibility criteria included systematic reviews where individual clinicians were primary end users. Outcomes we were interested in were the effectiveness and identification of specific features of CDDSS on diagnostic performance.

Results: We identified 1970 studies and excluded 1938 because they did not fit our inclusion criteria. A total of 45 articles were identified and 12 were found suitable for meta-review. Extraction of high-level requirements identified that a more standardized computable approach is needed to knowledge representation, one that can be readily updated as new knowledge is gained. In addition, a deep integration with the EHR is needed in order to trigger at appropriate points in cognitive workflow.

Conclusion: Developing a CDDSS that is able to utilize dynamic vocabulary tools to quickly capture and code relevant diagnostic findings, and coupling these with individualized diagnostic suggestions based on the best-available evidence has the potential to improve diagnostic accuracy, but requires evaluation.

KEY MESSAGE:

• Diagnostic error is the largest threat to patient safety in primary care.

• Little research has been conducted on primary care computerized diagnostic decision support systems (CDDSS).

• Better use of diagnostic coding vocabulary in CDDSS and integrating these with the electronic health record has the potential to improve diagnostic decision making.

INTRODUCTION

Diagnosis is one of the ‘principal’ tasks of primary care, yet incident-reporting systems typically receive very low rates of diagnostic errors (1). More targeted techniques, such as record screening or interviewing GPs about their most memorable errors have been more successful at identifying diagnostic error but have been retrospective in their approach (2,3). They cannot recreate the diagnostic process that resulted in an error and cannot reliably determine patient features at presentation, information elicited by the clinician (due to incomplete records and selective memory), other diagnoses considered, how clinical information was interpreted and what inferences were made at the time of the diagnosis (4). A US study of closed malpractice claims in the ambulatory setting estimated that cognitive factors (e.g. judgment errors, vigilance and memory lapses, lack of knowledge) were implicated in virtually all diagnostic errors, either alone (in 55% of errors) or in association with patient—and/or system-related factors (5).

A recent study of difficult diagnoses in general practice, funded by the UK Department of Health, identified information gathering as the most important determinant of diagnostic accuracy (6). In agreement with other studies, it found no effect of experience (years in practice) on diagnostic accuracy or the gathering of critical information, but did find that there was a reduction in the overall amount of information elicited with increasing experience (7,8).

Computerized diagnostic decision support systems (CDDSS) may be able to play an important role in assisting information gathering, yet previously, CDDSS required clinicians to either enter the clinical features observed or tick them off on a list in order to receive diagnostic suggestions. This was done completely independently from the patient's health record and without considering ‘when’ (i.e. type of diagnostic problem, timing during the consultation) and ‘what’ type of support would be useful. Suggesting differential diagnoses at the start of the consultation (on the basis of the patient's age/sex/risk factors and presenting complaint) is a fairly easy type of support to provide from a technical point of view, would not require clinicians to change their current practice nor the way that they record information, and could represent an easy means of implementing decision support. The aim of this type of support, which we termed ‘suggesting’, would be to draw attention to important diagnoses that might otherwise be ignored. There is encouraging evidence that asking clinicians to engage in the systematic generation and testing of hypotheses can increase diagnostic accuracy in challenging problems without degrading performance in simple problems (9). A second type of support can be triggered at the end of the consultation, only when serious diagnoses have not been excluded. We term this ‘alerting’. ‘Alerting’ has the advantage of being selective, and therefore more likely to be attended to by the clinician. The disadvantage is that it would be provided after the clinician has gone through the problem and made a diagnosis (i.e. after the problem has been mentally represented in a certain way) and this is known to be very resistant to restructuring and change (10). Alerting is more resource-intensive in its development, as it will require GPs to change the way that they record information collected during the consultation. Complete and accurate recording of information would be needed for the support to be triggered effectively and be useful to the clinician.

In this paper, we aimed to conduct a review of existing systematic reviews to assess the current consensus on how CDDSS can meet the requirements of supporting the cognitive task of diagnosis, and the currently perceived barriers that prevent the integration of CDDSS with electronic health record (EHR) systems.

METHODS

Since several systematic reviews of the effect of electronic decision support systems have been conducted, and the purpose of this study was not to perform another review, but to synthesize the requirements for CDDSS, we chose to conduct a ‘meta-review’ of existing systematic reviews and meta-reviews. An electronic search for journal articles was conducted in June 2012, spanning MEDLINE, EMBASE, PsycINFO and Web of Knowledge. A summary is available in the eligibility criteria (Box 1). The reference lists of meta-reviews were searched. One researcher (MN) screened study titles and abstracts for relevance. Two researchers (MN and BD) then independently reviewed the selected full text articles, to agree a final list. Two researchers (BD and OK) then analysed the full text of shortlisted articles. The search was repeated in December 2014 to identify any new papers.

Box 1. Eligibility criteria.

	Included	Excluded
Publication features	1. Peer-reviewed journal articles	1. Books
	2. Post-2004	2. Grey literature
	3. English
Methods	Systematic reviews	1. Primary research
		2. Narrative reviews
Participants	Individual clinicians included among primary end-users	1. Psychiatrists, psychologists, and counsellors only
		2. Allied health professionals only
		3. Non-clinical participants only
Tasks	Medical diagnostic tasks involving computerized decision support system (DSS)	1. Non-computerized DSS only (e.g. protocols, guidelines, triage systems)
		2. Non-diagnostic DSS only
Outcome(s) reviewed	1. Effectiveness of computerized DSS (on diagnostic performance)
	2. Impact of specific features of computerized DSS (on diagnostic performance)
Search terms	1. Term for ‘computerized’ (e.g. computer*/electronic)
	AND
	2. Term for ‘decision support’ (e.g. decision support/decision aid)
	AND
	3. Term for ‘medical’ (e.g. health*/medic*)
	AND
	4. Term for ‘diagnosis’ (e.g. diagnos*/reasoning)

RESULTS

Sample characteristics

The search identified 1970 studies (following removal of 713 duplicates). Title and abstract screening resulted in the exclusion of 1938 papers. Common reasons for exclusion included non-systematic reviews of computerized DSS effectiveness and reviews of CDDSS effectiveness in other areas of medical decision making (e.g., prescription). The resulting shortlist contained 32 articles (31 identified by electronic search) which were read for relevance. Subsequently, 11 were retained for meta- review. Of these 11 reviews, none was exclusively about primary care or CDDSS. The repeat search in December 2014 identified a further 13 potentially eligible papers, of which one was included (Figure 1). We, therefore, were unable to comment exclusively from either a diagnostic or a primary care perspective, but have where possible identified in Table 1 the number of CDDSS studies as opposed to other types of decision support (order entry, guidelines, etc.).

Figure 1. PRISMA flow chart.

Table 1.

Overview of included studies on diagnostic decision support.

Ref	Studies reviewed	Setting	Intervention	Outcomes	Principal findings
11	77	End-users: mostly (86%) physicians. Care setting: primary care (44%), secondary inpatient (29%) and secondary outpatient(25%). Multicentre setting: 51%	Various CDSS (3 CDDSS).	Impact of standardized data on: practitioner performance, patient outcomes, system features	Systems that used standards were more likely to be integrated and to provide an automatic alerting system
24	10	End-users: health professionals. Case type: abdominal pain only	CDSS (unspecified)	Diagnostic accuracy.	80% of included studies showed a clinically significant improvement after introduction of CDSS
16	64	End-users: usually physicians. Care setting: one-third inpatient.	Various CDSS (10 CDDSS): Part of EHR or computerized physician order entry: 47% Point of care: 88% Automatic (vs requested): 60% Suggest new orders (vs critique existing orders): 91%	Practitioner performance (process of care). Patient outcomes	Improved practitioner performance in 4 of 10 studies of diagnostic aids. Five studies assessed patient outcomes for diagnostic aids. No improvement reported
18	17	End-users: health professionals. Care setting: hospital (inpatient and outpatient)	Various CDSS (including CPOE with DS)	Practitioner performance. Patient outcomes	Practitioner performance: 12 of 16 systematic reviews found significant improvement with CDSS. Patient outcomes: 3 of 16 systematic reviews found strong evidence for impact of CDSS
17	70	End-users: health professionals. Care setting: mostly outpatient (77%) and academic (59%). Many (43%) multisite studies	Various CDDSS (mostly not point of care, and unclear if any were diagnostic)	Process outcomes. Patient outcomes	15 potential factors examined in a regression. Automatic reminders, EHR integration. Reason needed for bypass. Provide direct recommendation, all positive factors. Overall, 90% (80–99%) improved decision making
25	27	Setting: unclear. Case type: cancer cases only.	Artificial neural network systems as a decision tool.	Effect on clinical performance: diagnosis, prognosis or image analysis. Accuracy: sensitivity and specificity.	21 studies showed added benefit to healthcare provision
26	34	Care setting: primary and secondary care. Case type: abdominal pain only	CDDSS (unspecified)	Cost-effectiveness: paper checklist vs computerized decision tools	CDDSS have greater specificity and lower false-positive rates than unaided doctors. They are thus ‘… potentially useful in confirming a diagnosis of acute appendicitis, but not in ruling it out’ Paper checklists more cost-effective than a CDDSS
27	35	End-users: healthcare professionals. Care setting: mostly primary care (74%). Case type: various	Various CDDSS providing direct recommendations for ordering/performing diagnostic tests	Increase/decrease in diagnostic testing measures	Improved test-ordering behaviour in 5 out of 6 papers addressing diagnosis (as opposed to disease/treatment monitoring)
28	36	End-users: healthcare professionals (mostly physicians). Case type: acute care	Various CDDSS (incl. medication dosing assistants, management assistants, diagnostic assistants)	Process outcomes. Patient outcomes	Improved process of care in 2 out of 3 papers addressing diagnosis
29	28	End-users: physicians or physician trainees. Care setting: various	Onscreen, point of care reminders		Median absolute improvement in process adherence: 4.2% n = 32.28 studies reported 32 comparisons
15	24	End-users: health professionals. Care setting: all studies on diagnosis = hospital setting. Case type: more chronic (n = 16) than acute (n = 8) conditions	Various CDDSS. Diagnostic DSS provided: point-of-care access to diagnostic guidelines risk assessment condition-specific alerts	Process outcomes (practitioner performance). Patient outcomes	3 out of 4 studies on diagnosis found positive effects in acute conditions rather than chronic disease
30	32 (4 RCTs)	End-users: physicians or physician trainees. Care setting: various	Abdominal pain diagnosis, mental health, laboratory ordering, resuscitation	Process and accuracy	Only one study evaluated accuracy, and its results were inconclusive

Our results suggest that there are four significant challenges to be met if a fully integrated CDDSS is to be realized. First, a more standardized computable approach to knowledge representation is needed, and second, one that can be readily updated as new knowledge is gained. Third, a deep integration with the EHR is needed in order to trigger at appropriate points in cognitive workflow, this latter being the final challenge. The current barriers are a failure to use dynamic vocabulary tools, which are able to capture and code relevant diagnostic findings, integrate these with the EHR and coupling these with an individualized diagnosis made by the physician.

Knowledge representation of sufficient flexibility and depth to represent rules. Ahmadian et al. have examined the role of standardized terminologies in DSS (11). They found that systems that used standard terminologies (such as ICD, SNOMED-CT) were more likely to be integrated with an EHR system than stand-alone systems. The terminology used needs to be sufficiently fine-grained and unambiguous as to adequately represent the decision rule and allow reasoning with the clinical data. This is potentially much more complex than for clinical workflow applications. In particular, the uncertainty and challenges posed by the representation and management of a variety of complex diagnostic problems sit at the intersection of DSS and computational intelligence provided by agent-based technology (12). A demonstration of that potential has been successfully shown in the EU FP6 IST HealthAgents project (IST–2004–27214) (13). This focused on the development of a distributed, agent-based DSS, which implemented a series of automated classifiers based on pattern recognition methodologies for the diagnosis and prognosis of brain tumours.

Sustainability and upgradeability. The implication of maintaining ontology for computerized diagnostic DSS is that work will need to be done to add new problems. Reviews by Shojania et al. (14) and Sintchenko et al. (15), both make the point that not only are CDDSS complex interventions where the net effect is the product not just of the technology but of its uptake and use, but also that the effectiveness of the CDDSS will vary according to the condition studied.

Semantic interoperability with the HER—standards and granularity issues. To achieve the full potential of CDDSS, it is necessary to integrate quite closely with the EHR system itself. The reason for this is that double entry of data is a complete barrier to the uptake of CDDSS. Several reviews by Garg et al., Kawamoto et al., and Jaspers et al., have all shown this effect (16–18). Ahmadian et al. also found that CDDSS that used standards for the representation of semantic concepts were more likely to be integrated (11).

Understanding of cognitive workflow and action points. Kawamoto et al. carried out a regression analysis of predictors of effectiveness in CDDSS studies, where significant factors included automatic triggering of reminders at the point of care, integration with the EHR, a direct recommendation for care and requisite reason to bypass (17). This finding was replicated by Garg et al., who also found that suggesting was more effective than critiquing (16).

Table 1 summarizes the articles that were reviewed.

DISCUSSION

The literature on computerized CDDSS for medical diagnosis is sparse compared with the evidence-base that supports guideline reminders, prompts, prescribing support and other aspects of decision support. Our literature review was a meta-review of existing systematic reviews, the focus of which was to elicit requirements for the future development of CDDSS, and is not reported as a new systematic review. However, we were able to identify four key requirements for the development of future CDDSS. These findings must be tempered by the fact that very few of the decision support systems in the reviews were diagnostic in nature, and, given the nature of the reviews, it is impossible to separate out solely primary care CDDSS. We did not sift out the different original papers from the source systematic reviews, but they were a very heterogeneous group, consisting of a variety of clinical problems and different specialist settings. Risk of bias assessment is difficult in a meta-review, and must rely on appropriate methods in the source reviews. Unfortunately, the extreme heterogeneity of source studies precluded narrative synthesis in these systematic reviews, and we are unable to make further comment on the potential for bias. There is an urgent requirement to study primary care CDDSS in carefully designed randomized studies with diagnostic accuracy and patient outcomes.

We take the view that for computerized diagnostic decision support systems to be effective they need to be automatic; linked seamlessly to the EHR and to its knowledge engine, and to give prompts and reminders at ‘timely’ points in the decision-making process. The system needs to offer an advantage to clinicians over and above the ability to generate safety-critical diagnostic alerts. With increasing litigation against general practitioners, and more than half of all claims being related to diagnostic error, there would be considerable enthusiasm for a system that was able to utilize the dynamic vocabulary tools to quickly capture and code relevant diagnostic findings, and couple these with an individualized diagnosis based on the best-available evidence (19). This information would be inserted into the EHR far faster and more accurately than via most EHR interfaces. In addition, the more richly coded information would be ideal material for enrichment of clinical prediction rules via knowledge mining techniques. A primary care focussed DSS would then enable either suggesting or alerting functions, or both to be built into the EHR system. A recent study has shown that, with simulated cases, suggesting can improve diagnostic accuracy over unaided decision making, whilst alerting had no benefit. Few GPs changed their diagnosis after the alert (20).

There are significant opportunities offered by technical and informatics advances in the past few years that may finally allow for the development of effective computerized diagnostic decision support. These include the development of the semantic web and standards for representing knowledge in a computable way (21). The pervasive availability of information in a machine-processable format, and the ability to link to devices wirelessly, means that we should no longer consider the EHR as a static, desk-bound system but rather as a knowledge-manager (22). This is important as it will eventually lead to a shift in professional and market place expectations of the role of EHR systems (23).