Skip to main content
The BMJ logoLink to The BMJ
. 2003 Feb 15;326(7385):394.

Effect of computerised evidence based guidelines

Computer support is complex intervention

Jon D Emery 1
PMCID: PMC1125249  PMID: 12586680

Editor—Eccles et al's rigorous approach to the evaluation of a computerised decision support system for the management of angina and asthma accounted for many of the flaws in previous trials of computer support.1 They were no doubt disappointed that no effect was seen, probably due to low usage of the system.

Although not discussed in the paper, a possible explanation for this is that, given the comparatively high use of computers required for inclusion in the trial, the practices already used simpler computerised templates to promote collection of process of care data. Practitioners may therefore have perceived little further to be gained by using the more detailed decision support system, particularly if it did not allow easy switching between the guideline and the clinical system.

The study by Eccles et al shows the complexity of interventions in primary care that incorporate computerised decision support systems. This complexity needs to be fully accounted for in designing and evaluating such interventions.2 Even with an apparently well developed piece of software, the trial assumed that offering brief training to a minority of practitioners in each practice would be sufficient for it to be incorporated into the increasingly complex care provided in routine general practice consultations.

Trials of computer support in primary care need to acknowledge this complexity by embedding use of the software in a carefully specified model of care. For the high quality management of chronic disease, this model will probably require subspecialisation within a general practice, as proposed in the new general practitioner contract.3

Providing focused training to key people in a practice and supporting subspecialisation through computer decision support may be a more appropriate approach to chronic disease management in primary care. Future trials of computer support must consider not only the technical features of the software but also the model of service it is supporting and hence the training requirements of potential users. Theoretically derived measures that predict use of the software by practitioners in these trials could provide further important data on the potential role of decision support in clinical practice. Only then can one truly give computer decision support a fair trial.

Footnotes

Competing interests: None declared.

References

  • 1.Eccles M, McColl E, Steen N, Rousseau N, Grimshaw J, Parkin D, et al. Effect of computerised evidence based guidelines on management of asthma and angina in adults in primary care: cluster randomised controlled trial. BMJ. 2002;325:941–944. doi: 10.1136/bmj.325.7370.941. . (26 October.) [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 2.Campbell M, Fitzpatrick R, Haines A, Kinmonth AL, Sandercock P, Spiegelhalter D, et al. Framework for design and evaluation of complex interventions to improve health. BMJ. 2000;321:694–696. doi: 10.1136/bmj.321.7262.694. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 3.Marshall M, Roland M. The new contract: renaissance or requiem for general practice? Br J Gen Pract. 2002;52:531–532. [PMC free article] [PubMed] [Google Scholar]
BMJ. 2003 Feb 15;326(7385):394.

Challenge should not be abandoned

Ian N Purves 1

Editor—As a coauthor of the trial of COGENT, a clinical decision support system, I would like to correct any misunderstanding this paper may have caused.1-1,1-2 I head a large centre for health informatics in the United Kingdom and lead the development of the Prodigy clinical decision support system.

The COGENT trial of two computerised guidelines found no differences in a range of measures of the process and outcomes of care, primarily because the system was not used. But these findings should not be extrapolated to other decision support systems.

Readers to whom I have talked have assumed that COGENT guidance and software was based on the current Prodigy system. COGENT used evidence based guidelines from the north of England on the management of asthma and angina and software based on ideas from early Prodigy software. Constraints in the COGENT trial did not allow the software to be tested in practice before the intervention period or the guidance to be reworded for easier comprehension. Major shortcomings were soon apparent, but these problems could not be addressed because the trial method did not accommodate the usual process of software development and guidance formatting. With hindsight, a randomised controlled trial of a new technology (such as a clinical decision support system) should not be have been undertaken until the technology had been shown to be usable and to be regularly used.1-3

Which way forward? In an increasingly complex world, clinicians overloaded with information need computerised decision support systems if their practice is to be evidence based. The challenge of developing and integrating such systems into clinical workflow should not be abandoned. Not to invest in such systems would be as inappropriate as suggesting that the British army should give up its rifles because of their current technical problems.

Footnotes

Competing interests: INP is a grant holder, Prodigy contract (Department of Health).

References

  • 1-1.Eccles M, Grimshaw J, Steen N, Parkin D, Purves I, McColl E, et al. The design and analysis of a randomised controlled trial to evaluate computerised decision support in primary care: the COGENT Study. Fam Pract. 2000;17:180–186. doi: 10.1093/fampra/17.2.180. [DOI] [PubMed] [Google Scholar]
  • 1-2.Eccles M, McColl E, Steen N, Rousseau N, Grimshaw J, Parkin D, et al. Effect of computerised evidence based guidelines on management of asthma and angina in adults in primary care: cluster randomised controlled trial. BMJ. 2002;325:941–944. doi: 10.1136/bmj.325.7370.941. . (26 October.) [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 1-3.Heathfield H, Pitty D, Hanka R. Evaluating information technology in health care: barriers and challenges. BMJ. 1998;316:1959–1961. doi: 10.1136/bmj.316.7149.1959. [DOI] [PMC free article] [PubMed] [Google Scholar]
BMJ. 2003 Feb 15;326(7385):394.

It is good to be honest and say that systems were not used

Robin Beaumont 1

Editor—The paper by Eccles et al possesses academic integrity, which is widely lacking in computing research.2-1

I was the main researcher for the first two phases of the Prodigy project and believe that this project has much to teach the Prodigy team. One of the first detailed reports I wrote on Prodigy in 1998 indicated that Prodigy was actually used very little, about seven times a week, and most of the time (88%) users requested to bypass the system (www.robinbt2.free-online.co.uk/virtualclassroom/chap13/report1.pdf). I am very heartened to see that this type of information is being disseminated rather than suppressed, as was the case with the report I produced.

Footnotes

Competing interests: None declared.

References

  • 2-1.Eccles M, McColl E, Steen N, Rousseau N, Grimshaw J, Parkin D, et al. Effect of computerised evidence based guidelines on management of asthma and angina in adults in primary care: cluster randomised controlled trial. BMJ. 2002;325:941–944. doi: 10.1136/bmj.325.7370.941. . (26 October.) [DOI] [PMC free article] [PubMed] [Google Scholar]
BMJ. 2003 Feb 15;326(7385):394.

Opportunity was missed

Tom Fahey 1,2,3,4, Liz Mitchell 1,2,3,4, Frank Sullivan 1,2,3,4, Alan Montgomery 1,2,3,4, Peter Gregor 1,2,3,4

Editor—Eccles et al performed a methodologically sound study of a poorly developed intervention.3-1 They define a computerised support system as a system that compares patients' characteristics with a knowledge base and then guides a health provider by offering patient specific and situation specific advice.3-1

The intervention developed and tested in their study does not seem to meet these criteria. It did not depend on patient specific information but entry of a more general Read code. It did not contain a reminder to initiate review of patient care or arrange follow up. How far treatment recommendations depended on the patient's individual clinical review rather than issuing more generic recommendations for treatment is also unclear.3-2,3-3

The Prodigy system, the intervention around which this study was based, is an electronic version of a paper guideline that is triggered by entry of a prespecified Read code. By making this the only way in which to enter the computerised guideline the investigators ensured a low level of use during the study. General practitioners are unlikely to continue to enter the same Read code at every consultation as it would mean that each participating patient would have multiple duplicate entries of the same Read code in their electronic record.

By excluding any sort of reminder function in their system,3-2 the investigators have not accounted for a barrier in managing chronic diseases—registration, recall, and regular review of patients. Analysis of factors that operate in managing angina and asthma should have uncovered such barriers before the start of this study.3-4

Other details about the use of the computerised guideline require clarification. What is the definitive number of patients randomised and followed up in each practice for each intervention? What is the number (percentage) of patients in whom the computer guideline went past the first screen? What is the number (percentage) for whom a complete record entry was made? The authors make no comment on the differential use of the electronic guidelines between the two computer suppliers.

This study reinforces the fact that passive diffusion of guidelines, in electronic or paper format, is an ineffective way to implement best practice.3-4 Paying insufficient attention to how a computer interface operates has produced low levels of usage and made the evaluation less useful than it might have been.3-2,3-5 Future studies should take into account the different functions of computer based clinical decision support systems,3-5 rather than simply generate suggestions to alter prescribing practice.3-2

Footnotes

Competing interests: None declared.

References

  • 3-1.Eccles M, McColl E, Steen N, Rousseau N, Grimshaw J, Parkin D, et al. Effect of computerised evidence based guidelines on management of asthma and angina in adults in primary care: cluster randomised controlled trial. BMJ. 2002;325:941–944. doi: 10.1136/bmj.325.7370.941. . (26 October.) [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 3-2.Randolph A, Haynes RB, Wyatt JC, Cook DJ, Guyatt GH. Users' guides to the medical literature. XVIII. How to use an article evaluating the clinical impact of a computer-based clinical decision support system. JAMA. 1999;282:67–74. doi: 10.1001/jama.282.1.67. [DOI] [PubMed] [Google Scholar]
  • 3-3.Hunt D, Haynes RB, Hanna S, Smith K. Effects of computer-based clinical decision support systems on physician performance and patient outcomes. JAMA. 1998;280:1339–1346. doi: 10.1001/jama.280.15.1339. [DOI] [PubMed] [Google Scholar]
  • 3-4.Anonymous. Getting evidence into practice. Effect Health Care Bull. 1999;5:1–16. [Google Scholar]
  • 3-5.Tai S, Nazareth I, Donegan C, Haines A. Evaluation of general practice computer templates: lessons from a pilot randomized controlled trial. Methods Inf Med. 1999;38:177–181. [PubMed] [Google Scholar]
BMJ. 2003 Feb 15;326(7385):394.

Effect may be function of incentive

Jacobus T van Wyk 1, Peter W Moorman 1, Marc A M van Wijk 1

Editor—In their paper Grimshaw et al showed that having guidelines available does not result in people using them.4-1 Analogously, Eccles et al showed that having a decision support system available does not lead to people using it.4-2 Benson advocated incentives are needed before healthcare workers start using computers.4-3

In contrast to Eccles et al, van Wijk et al showed effects from a guideline decision support system.4-4,4-5 The general practitioners in these studies had incentives to use the tool, whereas such incentives were missing in Eccles et al's design. We believe that authors of papers describing an evaluation of a decision support system should in the future explicitly discuss incentives for and barriers to using these systems.

References

  • 4-1.Grimshaw J, Russell I. Effect of clinical guidelines on medical practice: a systematic review of rigorous evaluations. Lancet. 1993;342:1317–1322. doi: 10.1016/0140-6736(93)92244-n. [DOI] [PubMed] [Google Scholar]
  • 4-2.Eccles M, McColl E, Steen N, Rousseau N, Grimshaw J, Parkin D, et al. Effect of computerised evidence based guidelines on management of asthma and angina in adults in primary care: cluster randomised controlled trial. BMJ. 2002;325:941–944. doi: 10.1136/bmj.325.7370.941. . (26 October.) [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 4-3.Benson T. Why general practitioners use computers and hospital doctors do not—part 2: scalability. BMJ. 2002;325:1090–1093. doi: 10.1136/bmj.325.7372.1090. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 4-4.van Wijk MA, van der Lei J, Mosseveld M, Bohnen AM, van Bemmel JH. Assessment of decision support for blood test ordering in primary care. A randomized trial. Ann Intern Med. 2001;134:274–281. doi: 10.7326/0003-4819-134-4-200102200-00010. [DOI] [PubMed] [Google Scholar]
  • 4-5.van Wijk MA, der Lei J, Mosseveld M, Bohnen AM, van Bemmel JH. Compliance of general practitioners with a guideline-based decision support system for ordering blood tests. Clin Chem. 2002;48:55–60. [PubMed] [Google Scholar]
BMJ. 2003 Feb 15;326(7385):394.

Authors' reply

Martin P Eccles 1,2,3,4, Elaine McColl 1,2,3,4, Nick Steen 1,2,3,4, Nikki Rousseau 1,2,3,4, Jeremy Grimshaw 1,2,3,4, David Parkin 1,2,3,4

Editor—We agree that complex interventions should ideally be developed through an iterative process.5-1 Exceptions to this include evaluating a preformed intervention that would not otherwise be rigorously evaluated. This applied at the outset of our study, although our intervention drew heavily on the iterative development of Prodigy software. We conducted an integrated process evaluation to understand the results better. This will appear in the BMJ shortly.

The NHS has invested large amounts of money in information technology, sometimes for little or no benefit. The evaluation of information technology is complex and multifaceted, but a computerised decision support system can be evaluated as a health technology. Although formative evaluation may be an important element of software development, until someone comes up with better methods of producing unbiased estimates of effectiveness and efficiency we maintain that all health technologies should be considered evaluable in randomised controlled trials.

Important methodological issues exist about the timing and duration of such evaluations, and we agree with Purves that they should be performed on stable systems. Given the cyclical nature of software development and the self belief and enthusiasm of developers, such points must be prespecified and enacted to avoid self perpetuating iterative cycles of development with the constant promise of jam tomorrow.

Our description of the system that we evaluated is accurate, and none of the authors dissented from it up to the point of publication.

Data were collected from November 1997 to September 2000, with the intervention running during the last 15 months of this time period. The trial was paused for six months while the software team worked on improvements. The rates of presentation of patients we reported equated to opportunities for the system to be used between twice a day and every other day. Moreover, by the start of the intervention period, Prodigy software had become available and was delivered to trial practices alongside the study software. Our feedback from practices indicated that at least some asked for the Prodigy software to be turned off. This echoes Beaumont's letter and implies that increasing the number of guidelines offered may not be the remedy that Purves suggests.

Two correspondents identified the importance of the issue of training. Contrary to Purves's letter, two people from each practice were invited to a one day training session and the software was installed within 10 weeks by the computer supplier of two thirds of the trial practices. For the second supplier this interval was almost double, owing to unforeseeable commercial considerations in the company. We acknowledged the importance of training while suggesting that what happened was representative of the real world of primary care. We still believe this to be true but support Emery's and Purves's call for better training in service settings.

Fahey et al say that the low levels of use of the system were partly due to requiring the entry of a single Read code and lack of responsiveness to patient specific information. Initially the system could be triggered automatically by a range of specified Read codes in the patient record. It could also be triggered by a clinician entering Read codes selected by the practice and was therefore not a passive method of dissemination. But this was changed in response to requests from the study practices. The automatic triggering was removed and a customisable Read code entry method was used for the final eight months of the intervention. Thus the system did rely on patient specific information.

Emery said that we may have had a ceiling effect due to practices currently using computerised templates. This seems unlikely because only 26% of practices already had computerised guidelines or protocols for angina and 46% for asthma.

Within Emery's suggestion of specified models of care we see the risk that clinicians and patients in primary care will be constrained to consult in ways that computers can cope with, rather than addressing the challenge of the integration of computers into patient centered consultations.

Footnotes

Competing interests: None declared.

References

  • 5-1.Medical Research Council. A framework for development and evaluation of RCTs for complex interventions to improve health. London: MRC; 2000. [Google Scholar]

Articles from BMJ : British Medical Journal are provided here courtesy of BMJ Publishing Group

RESOURCES