Abstract
Objective
To evaluate the use of a computerised support system for decision making for implementing evidence based clinical guidelines for the management of asthma and angina in adults in primary care.
Design
A before and after pragmatic cluster randomised controlled trial utilising a two by two incomplete block design.
Setting
60 general practices in north east England.
Participants
General practitioners and practice nurses in the study practices and their patients aged 18 or over with angina or asthma.
Main outcome measures
Adherence to the guidelines, based on review of case notes and patient reported generic and condition specific outcome measures.
Results
The computerised decision support system had no significant effect on consultation rates, process of care measures (including prescribing), or any patient reported outcomes for either condition. Levels of use of the software were low.
Conclusions
No effect was found of computerised evidence based guidelines on the management of asthma or angina in adults in primary care. This was probably due to low levels of use of the software, despite the system being optimised as far as was technically possible. Even if the technical problems of producing a system that fully supports the management of chronic disease were solved, there remains the challenge of integrating the systems into clinical encounters where busy practitioners manage patients with complex, multiple conditions.
What is already known on this topic
Computerised decision support systems produce improvements in patient care across a range of conditions and settings
Previous evaluations have been undermined by flaws in study design
Few studies have evaluated complex decision support systems for the management of chronic disease
What this study adds
No impact was found of a computerised decision support system delivering evidence based guidelines for chronic diseases on either the process or outcomes of care
It is unclear whether there are benefits from integrating such systems into clinical encounters where busy practitioners manage patients with complex and multiple conditions
Introduction
Despite the current interest in clinical guidelines there remains uncertainty about how best to introduce them into routine practice. A systematic review suggested the use of patient specific prompts at the time of consultation, which can be achieved with a computerised support system for decision making—that is, “a system that compares patient characteristics with a knowledge base and then guides a health provider by offering patient specific and situation specific advice.”1,2 A recent systematic review of 68 controlled trials examined the effectiveness of such systems.3 They were shown to be beneficial: nine of 15 trials of systems to improve drug dosing; one of five trials evaluating diagnostic aids; 14 of 19 trials evaluating systems to improve preventive care; and 19 of 26 trials evaluating “other” medical care such as the management of disease in hospital and ordering tests. Improvements were found in six of the 14 studies measuring patient outcomes. However, the authors reported that most of the studies had flaws in design or analysis so that the findings should be interpreted with caution. Moreover, no studies were identified in the management of chronic disease in primary care or in computerised decision support systems integrated into routine computer systems in primary care.
We undertook a pragmatic cluster randomised controlled trial of a computerised decision support system to implement clinical guidelines for the management of asthma and angina in adults in primary care.
Methods
Our study methods are reported in detail elsewhere.4,5 We chose as chronic illnesses angina and asthma in adults; these diseases are predominantly cared for in primary care and are important because of their morbidity and mortality. We developed evidence based guidelines for the two conditions.6,7 Our study was approved by the appropriate multicentre research ethics committee.
Study general practices
We chose the study practices because their computer systems were extensively used. General practices in north east England were eligible to participate if at least 50% of the doctors reported using one of two computer systems to view clinical data and to issue prescriptions during consultations. We excluded singlehanded practices as they had too few patients for the required sample size.
Study patients
Study patients were those registered with the participating practices, aged 18 or over with angina or asthma. They were identified from a computerised search for relevant codes for the conditions, their management, and drug treatment.
Study design and power
Our study was a before and after, cluster randomised controlled trial utilising a two by two incomplete block design. We allocated general practices to one of two groups by a computerised randomisation process, stratified by computer system and by the vocational training status of the general practices (as a proxy for being innovative).8 One group received computerised guidelines for the management of angina and provided intervention patients for the management of angina and control patients for the management of asthma. The other received computerised guidelines for the management of asthma and provided intervention patients for the management of asthma and control patients for the management of angina.
We regarded our design as two embedded trials, and we determined the sample size for each separately. Each trial required 80% power to detect a 10% difference in adherence to guideline recommendations (for example, between 45% and 55%) with a significance level of 5%. Adherence to the guidelines was defined by measures of process as recorded in the patients' records. Because the intraclass correlation coefficients for measures of process were estimated to be around 0.05, we collected data from 40 patients with each condition in each of 60 practices.9 We assessed changes in patient outcome with summated Likert scales that could be considered as continuous variables with a normal distribution. Again, we assumed an intraclass correlation coefficient of 0.05.9 Application of standard methods indicated that if we collected data from 40 patients from each of 60 practices we would have 80% power to detect an effect size of 0.2 standard deviations with a significance of 5%.10
Data collection
We collected process data from patients' records at the end of the intervention period for the 24 months from 12 months before to 12 months after the introduction of the computerised decision support system. Prescribing data were abstracted electronically from computerised clinical records. Trained data collectors, blinded to practice allocation, manually abstracted non-prescribing data from the patients' written and computerised records.
We gathered patient reported outcomes and economic data from three rounds of postal questionnaires. We posted questionnaires 12 months before the intervention, immediately before the intervention, and 12 months after the intervention. We used a combination of generic (SF-36 and the EQ-5D) and condition specific measures of quality of life (the Seattle angina questionnaire, the Newcastle asthma symptoms questionnaire, and the asthma quality of life questionnaire).11–16
We collected data on three overlapping non-identical samples of patients. To obtain 40 patients with each condition per practice after three rounds of questionnaires, we drew a random sample of 80 eligible patients with each condition per practice for the first postal outcome survey. The dataset for patient reported outcomes comprised all patients completing all three questionnaires. The dataset for prescribing comprised all patients who responded to the first outcome survey. We gathered process data based on clinical records for a random sample of 40 patients who responded to the first outcome survey. A usage log within the practice computers recorded when the guidelines were used and by whom.
Analysis
We analysed data from the three patient samples independently, and we analysed each condition separately. We analysed each dataset in Stata, with generalised linear modelling procedures appropriate for hierarchical data. We analysed binary variables, data in the form of counts, and continuous outcome variables with binary, Poisson, and normal error structures, respectively. We modelled as random effects variation between practices and variation between patients (nested within practices). We included as a fixed effect the effect of decision support software.
We analysed each variable for clinical record based data and prescribing data twice. We based a pragmatic intention to treat analysis on all patients, whether or not they had consulted. We restricted an explanatory analysis to those patients who consulted after implementation of the system and for whom there was thus an opportunity for the intervention to influence their management.
Intervention
Computerised decision support system
The computerised decision support system was based on pre-existing software that was currently available to support prescribing for acute conditions.17 The two suppliers of the practice computer systems integrated the software into their products. The system anticipated clinicians' requirements by using information contained within a patient's computerised record to trigger the guideline and present patient scenarios—for example, for asthma, review of stable patient or acute exacerbation. Based on the chosen scenario, the system offered suggestions for management (including prescribing) informed by the content of the patient's record and requested the entry of relevant information, which was then stored in the patient's record. The guideline was, however, a separate path within the clinical system, and it was not possible to access all other parts of the clinical system from within the guideline. If the guideline was exited, it was only possible to return to it at the beginning of the pathway. The system was triggered either automatically by entering the electronic record of a patient previously identified as eligible or by entering a relevant morbidity code. After 4 months (based on feedback from practices) the system was altered so that triggering occurred only in response to a relevant morbidity code.
Training
Immediately before the intervention period each practice was invited to send two members to a one day workshop on using the system (training materials were supplied). Each doctor or practice nurse in the study received a paper copy of the summary of both guidelines, and each practice received a paper copy of the full version of both guidelines.
Results
Figure 1 shows the flow of practices through the trial. Six eligible practices were selected for an embedded case study, which is reported elsewhere.5 One practice (randomised to the asthma guideline) upgraded their system software and was unable to provide data on prescribing.
Figure 1.
Flow of practices through trial
Participating general practices and patient samples
Table 1 shows the characteristics of the participating practices. The randomised groups were of similar size and vocational training status, and there were no differences in any aspect of practice organisation or computer usage relevant to the care of patients with angina or asthma.
Table 1.
Characteristics of general practices randomised to receive guidelines on management of angina or asthma. Values are means (standard deviations)
| Randomised to angina guideline
|
Randomised to asthma guideline
|
Overall
|
|
|---|---|---|---|
| Whole time equivalent general practitioner | 4.5 (1.6) | 4.6 (1.8) | 4.5 (1.7) |
| No of patients registered with practice | 7837 (3221) | 8875 (4460) | 8365 (3902) |
| No of patients per whole time equivalent general practitioner | 1760 (299) | 1839 (344) | 1800 (322) |
| No of patients booked per hour with general practitioner | 7.7 (1.9) | 8.6 (2.8) | 8.1 (2.4) |
| General practitioner consultation time (min) | 8.2 (1.8) | 7.6 (2.1) | 7.9 (1.9) |
| Extra patients booked per general practitioner per surgery | 2.7 (1.8) | 3.1 (3.8) | 2.9 (2.9) |
| No of patients booked per hour with practice nurse | 6.1 (1.9) | 6.4 (3.1) | 6.2 (2.5) |
| Practice nurse consultation time (min) | 11 (4.5) | 11.1 (4.3) | 11.1 (4.4) |
Figure 2 summarises the data collected. Across all three samples there were no significant differences in the mean number of patients per practice between the two study groups.
Figure 2.
Data collected
Clinical record data and prescribing data
Reliability coefficients of 0.73 to 1.00 were obtained across the three data collectors for collection of 15 variables of process of care. No significant effects were found of the computerised decision support system on consultation rates, any aspect of the process of care for patients with angina or asthma (tables 2 and 3), or prescription of any category of drugs (tables 4 and 5). For all the reported variables from the clinical records the guidelines suggested that the action should be performed, although at varying intervals ranging from yearly to once only. For angina the guidelines recommended the first line use of β blockers, with other drugs being used in the case of intolerance or inadequate control of symptoms. For asthma the guidelines recommended the appropriate use of all reported drugs. As data collection related to two 12 month periods the results will potentially underestimate less frequently performed actions. It is also possible that actions were performed but not recorded in the record. Intraclass correlation coefficients were calculated for each condition and are reported elsewhere.5
Table 2.
Process of care for patients with angina based on clinical records before and after introduction of computerised decision support system
| Computerised system (n=1117)
|
Controls (n=1218)
|
Odds ratio (95% CI)
|
|
|---|---|---|---|
| No of patients consulting during intervention period | 1084 | 1192 | |
| Mean (SD) No of consultations during intervention period | 8.5 (6.4) | 8.6 (6.2) | 1.01* (0.91 to 1.11) |
| Mean (SD) No of consultations for angina | 1.6 (2.4) | 1.6 (2.3) | 1.05* (0.83 to 1.33) |
| No (%) of patients consulting before and after intervention period | |||
| Blood pressure recorded: | |||
| All patients | 859 (77); 889(80) | 935 (77); 969 (80) | 1.01 (0.74 to 1.39) |
| Patients who consulted | 859 (79); 889 (82) | 935 (79); 969 (82) | 1.05 (0.75 to 1.46) |
| Exercise recorded or advised: | |||
| All patients | 99 (9); 113 (10) | 156 (13); 153 (13) | 0.91 (0.55 to 1.50) |
| Patients who consulted | 99 (9); 113 (10) | 156 (13); 153 (13) | 0.90 (0.54 to 1.48) |
| Weight recorded or advised: | |||
| All patients | 253 (23); 282 (26) | 288 (24); 362 (30) | 0.86 (0.54 to 1.35) |
| Patients who consulted | 253 (23); 282 (26) | 288 (24); 362 (30) | 0.87 (0.55 to 1.37) |
| Smoking status known: | |||
| All patients | 222 (20); 243 (22) | 261 (22); 378 (32) | 0.68 (0.42 to 1.11) |
| Patients who consulted | 222 (20); 243 (22) | 261 (22); 378 (32) | 0.68 (0.41 to 1.13) |
| Smoking education given: | |||
| All patients | 33 (3); 47 (4) | 41 (3); 48 (4) | 1.08 (0.86 to 1.77) |
| Patients who consulted | 33 (3); 47 (4) | 41 (3); 48 (4) | 1.09 (0.66 to 1.78) |
| 12 lead electrocardiogram recorded: | |||
| All patients | 162 (15); 154 (14) | 197 (16); 164 (14) | 1.01 (0.68 to 1.52) |
| Patients who consulted† | 84 (9) | 93 (8) | 0.94 (0.58 to 1.53) |
| Exercise electrocardiogram recorded: | |||
| All patients | 46 (4); 28 (3) | 46 (4); 30 (3) | 1.01 (0.56 to 1.80) |
| Patients who consulted† | 23 (2) | 24 (2) | 1.05 (0.56 to 1.98) |
| Haemoglobin concentration recorded: | |||
| All patients | 322 (29); 371 (33) | 355 (29); 400 (33) | 1.01 (0.72 to 1.42) |
| Patients who consulted† | 201 (29) | 204 (26) | 1.08 (0.74 to 1.56) |
| Thyroid function recorded: | |||
| All patients | 192 (17); 214 (19) | 215 (18); 264 (22) | 0.83 (0.62 to 1.12) |
| Patients who consulted† | 136 (16) | 151 (16) | 0.94 (0.67 to 1.33) |
| Cholesterol or other lipid concentrations recorded: | |||
| All patients | 395 (35); 482 (43) | 427 (35); 574 (47) | 0.85 (0.65 to 1.12) |
| Patients who consulted† | 482 (45) | 572 (48) | 0.87 (0.66 to 1.14) |
| Blood glucose or HbA1c concentrations recorded: | |||
| All patients | 221 (20); 300 (27) | 267 (22); 334 (27) | 0.96 (0.67 to 1.39) |
| Patients who consulted† | 300 (28) | 334 (28) | 0.97 (0.67 to 1.41) |
Effect sizes; estimates of ratio of mean number of consultations for participants in practices that received computerised decision support system to mean number of consultations for participants in control practices based on Poisson regression model.
Patients who consulted after intervention and had not had variable recorded anywhere in their clinical record during period before intervention.
Table 3.
Process of care for patients with asthma based on clinical records before and after introduction of computerised decision support system
| Computerised system (n=1200)
|
Controls (n=1163)
|
Odds ratio (95% CI)
|
|
|---|---|---|---|
| No of patients consulting during intervention period | 1129 | 1101 | |
| Mean (SD) No of consultations during intervention period | 6.7 (6.3) | 6.8 (5.8) | 1.01* (0.92 to 1.11) |
| Mean (SD) No of consultations for asthma | 1.5 (2.3) | 1.6 (2.2) | 0.94* (0.81 to 1.08) |
| No (%) of patients consulting before and after intervention period | |||
| Lung function assessed: | |||
| All patients | 516 (43); 511 (43) | 492 (42); 517 (45) | 0.94 (0.67 to 1.33) |
| Patients who consulted | 515 (45); 510 (45) | 491 (45); 516 (47) | 0.94 (0.66 to 1.34) |
| Compliance checked: | |||
| All patients | 426 (36); 442 (37) | 446 (38); 471 (41) | 0.82 (0.58 to 1.15) |
| Patients who consulted | 425 (37); 441 (39) | 445 (40); 470 (43) | 0.82 (0.58 to 1.16) |
| Inhaler technique assessed: | |||
| All patients | 203 (17); 224 (19) | 234 (20); 262 (23) | 0.8 (0.5 to 1.28) |
| Patients who consulted | 203 (18); 224 (20) | 233 (21); 262 (24) | 0.81 (0.5 to 1.28) |
| Asthma education, action plan, or both: | |||
| All patients | 79 (7); 60 (5) | 108 (9); 78 (7) | 0.84 (0.4 to 1.74) |
| Patients who consulted | 79 (7); 60 (5) | 108 (10); 78 (7) | 0.81 (0.39 to 1.67) |
| Smoking status known: | |||
| All patients | 285 (24); 370 (32) | 305 (26); 367 (32) | 0.97 (0.65 to 1.45) |
| Patients who consulted | 285 (25); 369 (33) | 303 (28); 367 (33) | 0.98 (0.66 to 1.46) |
| Smoking cessation advice or nicotine replacement therapy: | |||
| All patients | 57 (5); 81 (7) | 68 (6); 103 (9) | 0.75 (0.45 to 1.26) |
| Patients who consulted | 57 (5); 81 (8) | 68 (6); 103 (9) | 0.76 (0.46 to 1.27) |
Effect size; see footnote to table 2.
Table 4.
Drugs prescribed for patients with angina before and after introduction of computerised decision support system. Values are numbers (percentages) of patients unless stated otherwise
| Computerised system (n=1415)
|
Controls (n=1466)
|
Odds ratio (95% CI)
|
|
|---|---|---|---|
| Short acting glyceryl trinitrate | 813 (58); 806 (57) | 831 (57); 802 (55) | 1.11 (0.87 to 1.41) |
| β Blockers | 666 (47); 683 (48) | 712 (49); 718 (49) | 0.99 (0.73 to 1.33) |
| Verapamil | 21 (2); 24 (2) | 18 (1); 21 (1) | 1.02 (0.57 to 1.82) |
| Modified release glyceryl trinitrate | 46 (3); 37 (3) | 46 (3); 43 (3) | 0.97 (0.50 to 1.54) |
| Transdermal glyceryl trinitrate | 19 (1); 14 (1) | 34 (2); 27 (2) | 1.03 (0.54 to 1.98) |
| Isosorbide dinitrate (short acting and modified release) | 64 (5); 56 (4) | 86 (6); 79 (5) | 0.91 (0.63 to 1.31) |
| Isosorbide mononitrate (short acting and modified release) | 522 (37); 524 (37) | 556 (38); 547 (37) | 1.11 (0.79 to 1.56) |
| Diltiazem | 270 (19); 269 (19) | 310 (21); 287 (20) | 1.43 (0.87 to 2.34) |
| Calcium channel blockers | 396 (28); 380 (27) | 377 (26); 368 (25) | 1.12 (0.80 to 1.58) |
| Statins | 409 (29); 498 (35) | 435 (30); 550 (38) | 0.92 (0.67 to 1.25) |
| β Blocker and dinitrate* | 20 (1); 20 (1) | 34 (2); 30 (2) | 1.24 (0.66 to 2.33) |
| Calcium blocker and dinitrate* | 22 (2); 24 (2) | 41 (3); 37 (3) | 1.15 (0.68 to 1.95) |
| Nitrate, calcium blocker, and β blocker* | 113 (8); 102 (7) | 115 (8); 120 (8) | 0.75 (0.46 to 1.22) |
Guideline specifically recommended not using these combinations.
Table 5.
Drugs prescribed for patients with asthma before and after introduction of computerised decision support system. Values are numbers (percentages) unless stated otherwise
| Computerised system (n=1391)
|
Controls (n=1385)
|
Odds ratio (95% CI)
|
|
|---|---|---|---|
| Short acting β2 agonists | 1138 (82); 1112 (80) | 1160 (84); 1108 (80) | 1.04 (0.83 to 1.31) |
| Inhaled corticosteroids | 1065 (77); 1001 (72) | 1004 (73); 975 (70) | 0.95 (0.78 to 1.16) |
| Long acting β2 agonists | 181 (13); 198 (14) | 164 (12); 183 (13) | 0.84 (0.59 to 1.20) |
| Oral steroids | 317 (23); 316 (23) | 286 (21); 297 (21) | 1.0 (0.82 to 1.22) |
| Oral bronchodilators | 100 (7); 95 (7) | 120 (9); 119 (9) | 1.38 (0.56 to 3.39) |
Patient reported outcomes
Overall, 4851 patients with angina and 4960 patients with asthma were identified for the first survey of patient outcomes, and 2241 (46%) patients with angina and 1760 (35%) patients with asthma completed questionnaires in all three rounds (fig 2). Response rates to the surveys were similar between the randomised groups. The computerised guidelines showed no effect on any patient reported outcome. For each condition intraclass correlation coefficients were calculated for each measure and each round and are reported elsewhere.5
Usage log files
Figure 3 shows the number of times the guidelines were triggered for each practice and the proportion of active interactions that involved going beyond the first screen. All data are standardised per whole time equivalent general practitioner. For both suppliers, the median number of active interactions was zero for much of the study.
Figure 3.
Box plot of number of times guidelines were triggered per whole time equivalent general practitioner, and proportion of active interactions that involved going beyond first screen, by month of study and supplier. Lower and upper edges of each box represent the 25th and 75th centile of observed data. The line partitioning box corresponds to median observation. In several instances median observation was zero and is coincident with lower edge of box. Whiskers give range of data
Discussion
We found no effect of a computerised decision support system as a vehicle for implementing evidence based guidelines for the management of two chronic diseases in primary care. We addressed the complexities of such management where clinicians provide ongoing care for patients with complex conditions and for extended periods. At the time of its deployment the system was embedded in two of the most widely used practice computer systems in the United Kingdom and was more sophisticated than any other commercially available to primary care facilities in the United Kingdom. We were, however, unable to show any incremental effect over the distribution of paper versions of the guidelines across a comprehensive range of measures, almost certainly due to the low levels of use.
A systematic review suggested caution in interpreting the results of identified studies of computerised decision support systems because of flaws in their design or analysis, a common situation in studies on changing professional behaviour.3,18 However, we addressed all of the important issues for design, conduct, and analysis.4,19
Asthma and angina are common chronic diseases in primary care. Both are of low incidence so dimensions of care such as initial investigations would be infrequent and less likely to be affected by any method of guideline implementation. Much of both guidelines dealt with the ongoing management of established cases, and the low incidence of both conditions should not have affected this. None the less, there are areas of care that an interactive computerised decision support system may be less able to influence, such as the issuing of routine repeat prescriptions by administrative staff. However, good clinical practice suggests that most patients with angina or asthma should receive an annual review, and within the trial almost all the patients consulted sufficiently frequently for this to have been considered. Although the guidelines reflected much of current practice—and performance of some actions may have been close to optimal (for example, use of short acting β2 agonists)—they both made recommendations for management that were not routine at the time of the study. Therefore it is unlikely that the guidelines merely enshrined all of current practice, and the data on process of care before the intervention show that this was not the case.
Although the study practices were selected on the basis of the extensive use of their computer systems (and thus were most likely to use a computerised decision support system), the staff had limited training in the functioning and use of the system. Limiting the amount of training in our study to one day was a pragmatic decision based on resources but was not that far removed from what is routine computer training within primary care in the United Kingdom.
For most general practitioners in the study the computerised system functioned in the context of routine surgeries (as opposed to settings such as clinics dedicated to disease management). Patients could present with any clinical problem such as arthritis or depression and, despite having asthma or angina, might not wish to discuss this, even though the computerised system might suggest this was appropriate. Given the range and complexity of problems that patients present to general practitioners it is demanding for any system to function in an unobtrusive yet helpful manner. The negative findings and low levels of use in our trial are similar to those observed by Hetlevik et al, who evaluated a computerised decision support system for the management of patients with hypertension or diabetes in primary care.20,21 They found that the guideline was used in the management of only 12% of patients with diabetes.21
Assuming that the technical challenges of producing a system that truly supports the management of complex disease can be overcome there remains the problem of how such systems function within clinical encounters where patients with complex conditions are managed. “To be widely accepted by practising clinicians, computerised support systems for decision making must be integrated into the clinical workflow. They must present the right information, in the right format, at the right time, without requiring special effort.”22 It is at least possible that for some or all of the reasons discussed above the low levels of use are all that can be reasonably expected of a computerised decision support system for the management of chronic disease.21 Certainly, in terms of implementing evidence based care, computerisation seems unlikely ever to be the “magic bullet” that answers all questions, and the current system could not be recommended.23 Although an increasing number of studies show that computerised decision support systems can function in a variety of circumstances, the challenge still remains to show how far this is possible, desirable, and efficient.3
Acknowledgments
We thank the participants, David Stables (EMIS Computing), Jon Rogers (Torex Meditel), Nick Booth, Neil Jones, and Bob Sugden (Sowerby Centre for Health Informatics), and members of the trial steering group.
Footnotes
Funding: The study was funded by the UK NHS research and development programme “Methods to promote the uptake of research findings” (£800 497; $1 257 661; €1 273 055) with additional funding from EMIS Computing (£45 500; $71 485; €72 360) and the Department of Health for England and Wales (£85 156; $133 789; €135 426). The Health Services Research Unit, University of Aberdeen, is funded by the Chief Scientist Office of the Scottish Executive Health Department. EMcC is funded by the UK NHS Primary Care Development Programme. The Centre for Health Services Research, University of Newcastle and the Health Services Research Unit, University of Aberdeen are part of the UK Medical Research Council Health Services Research Collaboration. The views expressed are those of the authors and not necessarily those of the funding bodies.
Competing interests: None declared.
References
- 1.Grimshaw J, Freemantle N, Wallace S, Russell I, Hurwitz B, Watt I, et al. Developing and implementing clinical practice guidelines. QHC. 1995;4:55–64. doi: 10.1136/qshc.4.1.55. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Friedman CP, Wyatt JC. Evaluation methods in medical informatics. New York: Springer-Verlag; 1997. [Google Scholar]
- 3.Hunt DL, Haynes RB, Hanna SE, Smith K. Effects of computer-based clinical decision support systems on physician performance and patient outcomes. JAMA. 1998;280:1339–1346. doi: 10.1001/jama.280.15.1339. [DOI] [PubMed] [Google Scholar]
- 4.Eccles M, Grimshaw J, Steen N, Parkin D, Purves I, McColl E, et al. The design and analysis of a randomised controlled trial to evaluate computerised decision support in primary care: the COGENT Study. Fam Pract. 2000;17:180–186. doi: 10.1093/fampra/17.2.180. [DOI] [PubMed] [Google Scholar]
- 5.Eccles M, McColl E, Steen N, Rousseau N, Grimshaw J, Parkin D. An evaluation of computerised guidelines for the management of two chronic conditions. University of Newcastle: Centre for Health Services Research; 2002. [Google Scholar]
- 6.Eccles M, Rousseau N, Adams P, Thomas L. Evidence-based guideline for the primary care management of stable angina. Fam Pract. 2001;18:217–222. doi: 10.1093/fampra/18.2.217. [DOI] [PubMed] [Google Scholar]
- 7.Eccles M, Rousseau N, Higgins B, Thomas L. Evidence-based guideline on the primary care management of asthma. Fam Pract. 2001;18:223–229. doi: 10.1093/fampra/18.2.223. [DOI] [PubMed] [Google Scholar]
- 8.Baker R. General practice in Gloucestershire, Avon and Somerset: explaining variations in standards. Br J Gen Pract. 1992;42:415–418. [PMC free article] [PubMed] [Google Scholar]
- 9.Campbell M, Grimshaw J, Steen N. Sample size calculations for cluster randomised trials. J Health Serv Res Pol. 2000;5:12–16. doi: 10.1177/135581960000500105. [DOI] [PubMed] [Google Scholar]
- 10.Donner A. Some aspects of the design and analysis of cluster randomization trials. Appl Stats. 1998;47:95–113. [Google Scholar]
- 11.Ware JE, Jr, Sherbourne CD. The MOS 36-item short-form health survey (SF-36). I. Conceptual framework and item selection. Med Care. 1992;30:473–483. [PubMed] [Google Scholar]
- 12.Brooks R. Euroqol—the current state of play. Health Policy. 1996;37:53–72. doi: 10.1016/0168-8510(96)00822-6. [DOI] [PubMed] [Google Scholar]
- 13.Spertus JA, Winder JA, Dewhurst TA, Deyo RA, Prodzinski J, McDonnell M, et al. Development and evaluation of the Seattle Angina Questionnaire: a new functional status measure for coronary artery disease. J Am Coll Cardiol. 1999;25:333–341. doi: 10.1016/0735-1097(94)00397-9. [DOI] [PubMed] [Google Scholar]
- 14.Steen IN, Hutchinson A, McColl E, Eccles MP, Hewison J, Meadows KA, et al. Development of a symptom-based outcome measures for asthma. BMJ. 1994;309:1065–1069. doi: 10.1136/bmj.309.6961.1065. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Juniper EF, Guyatt GH, Ferrie PJ, Griffith LE. Measuring quality of life in asthma 832-838. Am Rev Respir Dis. 1993;147:832–838. doi: 10.1164/ajrccm/147.4.832. [DOI] [PubMed] [Google Scholar]
- 16.Juniper EF, Buist AS, Cox FM, Ferrie PJ, King DR. Validation of a standardized version of the Asthma Quality of Life Questionnaire. Chest. 1999;115:1265–1270. doi: 10.1378/chest.115.5.1265. [DOI] [PubMed] [Google Scholar]
- 17.Purves IN. PRODIGY: implementing clinical guidance using computers. Br J Gen Pract. 1998;48:1552–1553. [PMC free article] [PubMed] [Google Scholar]
- 18.Bero L, Grilli R, Grimshaw JM, Harvey E, Oxman AD, Thomson MA. Closing the gap between research and practice: an overview of systematic reviews of interventions to promote implementation of research findings by health care professionals. BMJ. 1998;317:465–468. doi: 10.1136/bmj.317.7156.465. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Campbell MK, Steen N, Grimshaw JM, Eccles M, Mollison J, Lombard C. Design and statistical issues in implementation research. In: Thorsen T, Makela M, editors. Changing professional practice: theory and practice of clinical guidelines implementation, pp 57-76. Copenhagen: DSI - Danish Institute for Health Services Research and Development; 1999. [Google Scholar]
- 20.Hetlevik I, Holmen J, Krüger Ø, Kristensen P, Iversen H. Implementing clinical guidelines in the treatment of hypertension in general practice. Blood Pressure. 1998;7:270–276. doi: 10.1080/080370598437114. [DOI] [PubMed] [Google Scholar]
- 21.Hetlevik I, Holmen J, Krüger Ø, Kristensen P, Iversen H, Furuseth K. Implementing clinical guidelines in the treatment of diabetes mellitus in general practice. Evaluation of effort, process and patient outcome related to implementation of a computer-based decision support system. Int J Tech Assess Health Care. 2000;16:210–227. doi: 10.1017/s0266462300161185. [DOI] [PubMed] [Google Scholar]
- 22.James BC. Making it easy to do right. N Engl J Med. 2001;345:991–992. doi: 10.1056/NEJM200109273451311. [DOI] [PubMed] [Google Scholar]
- 23.Foy R, Eccles M, Grimshaw J. Why does primary care need more implementation research? Fam Pract. 2001;18:353–355. doi: 10.1093/fampra/18.4.353. [DOI] [PubMed] [Google Scholar]