Elizabeth Mitchell (BA); Research Fellow
Frank Sullivan (PhD); Professor of R&D in General Practice and Primary Care
Department of General Practice, University of Glasgow
Tayside Centre for General Practice, University of Dundee
Correspondence to:
Elizabeth Mitchell
Department of General Practice
4 Lancaster Crescent
Glasgow G12 0RR
E-mail: edm1a@clinmed.gla.ac.uk
Funded by: Chief Scientist Office of the Scottish Office Home and Health Department
2300 words
Abstract
Objectives: To appraise findings from studies examining the impact of computers on primary care consultations.
Design: An updated and extended systematic review of the world literature from 1980 until 1997.
Subjects: Studies were identified from electronic databases (Medline, Science Citation Index, Social Sciences Citation Index, Index of Scientific and Technical Proceedings, EMBASE, OCLC FirstSearch Proceedings), bibliographies, books, identified articles and by authors active in the field.
Interventions: Methodological adequacy of experimental studies was assessed using a system based on a previously developed method. A system for scoring non-experimental studies was developed.
Main outcome measures: Effect on clinician performance, patient outcomes or attitudes towards computerisation.
Results: Of the 5475 papers identified, 89 met the inclusion criteria. Computer use during consultations lengthened the consultation. Reminder systems for preventive tasks and disease management produced improvements in process rates, although some returned to pre-intervention levels when reminders were stopped. Generic prescribing increased and using computers for test ordering led to cost savings and fewer unnecessary tests. There were no negative effects on those patient outcomes evaluated. Impact on privacy, the doctor-patient relationship, cost, time and training needs were of concern to patients and clinicians.
Conclusions: Primary care computing systems can improve practitioner performance, particularly for health promotion interventions. This may be at the expense of patient-initiated activities, making many practitioners suspicious of the negative impact on relationships with patients. There remains a dearth of evidence evaluating effects on patient outcomes.
Introduction
Information technology has rapidly become a significant component of primary carea. Its application to the administrative tasks required of a busy practice has already demonstrated a variety of benefits such as patient registration and production of practice profilesb. Its potential contributions to patient management through access to reference informationc and the provision of decision support are more recent developmentsd. Computers are moving into the consultation itself and the government’s pledge to create an "NHS information superhighway"e by the end of 2002 means that this technology will continue to play an increasingly important role.
The development of primary care computing requires rigorous evaluation of existing and emergent information and communication technologiesf. In the past, criticisms have been made of the focus and methodology of studies in this field. Scoring systems have tended to concentrate on randomised controlled trial methodologies, which are often inappropriate in such a fast changing environment where multiple approaches to evaluation may be requiredg. We have extended a systematic review reported in 1995h and evaluated primary care computing systems from 1980 to 1997.
Methods
Systematic review of the literature 1980-1997
We conducted a worldwide review of the literature, updating an earlier systematic review described in detail elsewhereh. In addition to the methods previously outlined, we searched the databases of EMBASE, the Index of Scientific and Technical Proceedings and OCLC FirstSearch Proceedings. Non-English language journals were included in the search.
The Cochrane Group for Effective Practice and Organisation of Care (EPOC) provided references of articles containing the term ‘computer’ and authors active in the field were asked about studies-in-progress and unpublished work. Criteria for selection of studies were identical to those used in our previous review. However, in this review we were interested in identifying potential barriers to effective implementation and use of computers. We included studies which determined physician or patient attitudes towards computerisation. Studies on validation of data or administrative use were excluded.
Appraisal of studies
The methodological adequacy of randomised controlled trials, clinical trials and controlled before and after studies was assessed using a scoring system based on that proposed by Johnson et al.i [Table I]. The nature of this constantly evolving technology has resulted in a lack of trials in this field and strict adherence to Cochrane standards would exclude less rigorous but nonetheless useful studies. We developed a method of scoring non-experimental studies using a 2-round Delphi survey to reach a consensus on the methodological criteria to include [Table II]. This is a well-established decision-making mechanism, which involves recruiting a panel of ‘experts’ to agree about a particular set of hypothetical questionsj. The ten experts chosen were selected because of their experience in primary care research and/or medical informatics and/or critical appraisal.
Results
The search methods identified 5475 references. Editorials, dentistry or veterinary medicine studies and duplicates were excluded. The remaining 1892 abstracts were independently reviewed by both authors to determine their suitability for inclusion.
Full reprints were obtained for 214 which met the criteria and these were evaluated further. Kappa co-efficient for inter-rater agreement beyond chance was 0.7. Differences were discussed and a consensus reached. Eighty-nine studies were included in the review, an increase of 62 on the 1994 total.
Sixty-one studies examined effects on practitioners’ performance [Table III], 17 evaluated impact on patient outcome [Table IV] and 20 determined practitioner or patient attitudes [Table V]. Nine studies examined more than one area and are included in all relevant tables. Fifty-five studies were given an identical score by both reviewers. An agreed score was reached for the others.
Methodological quality was originally assessed for 28 papers in our 1994 review. One paper was excluded from this updated review on the basis that the computer was not specifically the interventionk; a second was re-categorised and included under attitudes45. This left 26 papers for comparison. We assessed a further 46 papers for this update. The remaining 17 papers identified dealt with attitudes towards computers only and were not assessed. The median score in 1994 was 7/10 [inter-quartile range 5-8] and 6/10 [4-8] in 1997 (p=0.240). Although not significantly different, the scores are more widespread in this updated review.
Impact on practitioner performance
This was the most predominant topic and one in which the majority of studies employed a controlled trial design [41/61 studies]. Prominence was given to immunisation and prevention [30/61 vs. 14/27 in 1994]. Other areas covered were management of disease [11/61 vs. 2/27], consultation content [9/61 vs. 6/27] and prescribing [7/61 vs. 4/27].
Studies on consultation content focused mainly on consultation length and doctor-patient interaction [Table III]. Consultation length increased by 48-130 seconds 52,79-81,83 in five of the studies, although this increase reduced after variable time periods. In one study, doctors worked an average of 30 days before their consultation rates returned to baseline levels52. Two studies found that doctors spent 11%-100% more time on computerised records than on conventional records29,79. This is principally due to the increased administrative tasks and preventive issues prompted by computer use29,80,83. Computer use created an increase in doctor-centred speech and medical topics raised, often at the expense of patient-centred activity81,82. Practitioners were also less likely to continue interacting with patients when using computerised records than when using paper records29,79, an occurrence which did not diminish with increased familiarity35. In an attempt to minimise this, patients in one study synchronised their speech with perceived pauses in practitioners’ keyboard use35.
Immunisation rates improved by 8-34%5,13,14,21,24,39,42,44,58 [Table III]. Eight studies involved the use of reminder systems with the greatest increases being seen for patient only reminders5,24 or reminders to both physician and patient13. McDowell et al found that immunisation rates fell to levels similar to that of control practices when the reminders were stopped24.
Performance of preventive tasks, such as blood pressure screening and cervical smears, improved by as much as 47% 2,4,6,7,8,9,11,12,19,20,25,38,47,48,49,54,55,65,73,75,83,85 [Table III]. The greatest increases occurred when practitioners were prompted as part of the consultation4,8,11,49,65. One study showed a 5% increase in performance from no prompt to a nurse initiated prompting system and a further 10% increase with a computerised prompting system49.
However, like immunisations, increased preventive activities fell to pre-intervention levels when reminders were no longer provided7. Studies evaluating reminders to patients also found increases of between 2-30%6,9,54. However, two studies which assessed the effects of both physician and patient reminders on mammography screening found no difference in uptake rates when only the patient was sent a reminder19,73.
Computers were also found to have positive effects on disease management. Four studies evaluated diabetes care standards and found improvements of between 5-69%10,22,51,89. The greatest increase occurred when physicians used an electronic protocol, although this produced a 10-minute increase in encounter length22. Studies evaluating hypertension management found 18-53% improvements in examinations26,70. Again, the largest increase occurred with an electronic protocol and consultation length increased by 35%26. Computerised alerts and reminders to physicians for HIV management produced faster response times than those of control physicians28. However, computerised decision support for lipid management produced no real differences and system usage was less than expected63. The introduction of a computer algorithm for paediatrics increased recording and compliance with management plans, but physicians found it ‘too tedious to use during routine care’ and the study was abandoned after 5 weeks31.
Prescribing studies showed an increase in generic prescribing1 and reductions in prescribing costs53,86,87. One study found significant reductions in doctors’ and receptionists’ time when computerised prescriptions were issued3. These elements combined illustrate why this remains the most commonly utilised feature of general practice computing. Computer use for test ordering led to a reduction of 6-75% as well as cost savings of 8-14%23,41,43,76.
Impact on patient outcomes
Patient outcome studies were less common [2/27 in 1994; 17/89 in 1997]. Computer use for hypertension management significantly increased the number of patients with reduced diastolic pressure15,64,70. However, their usefulness for anticoagulation management was not as readily determined. One study found improvements of 32-66%16 while another found no difference74. The introduction of computers to the consultation did not lead to any increase in service use either in primary care visits or in referrals to secondary care17,28,41,89. One study did however report a slight shift in activity where patients in computerised practices were managed more in the community89. An evaluation of computerised decision support for lipid management found a 55% reduction in the number of expected referrals63. This reduction in service use was evident where computers were used specifically for disease management or appointment scheduling, as numbers of visits to practitioners and no-show rates could be reduced15,62 and recall periods extended16. Four studies on patient satisfaction detected no significant changes when computers were introduced16,57,88,89.
Practitioner and patient Attitudes
Most practitioners willingly accepted computers as part of their working environment and were positive about their use30,32,33. Many thought that computerised records were more accurate than conventional records56 or that patient care improved46,61. Patients said that their doctors had better access to records36,45,61 and that consultations were unaffected36,40,61,72. However, there were five emergent themes which could prove major barriers to successful implementation: privacy, the doctor-patient relationship, cost, time and training.
Loss of privacy and confidentiality was the most common concern to patients. Many thought that computerised notes posed a greater threat to privacy, were more vulnerable to unauthorised access than conventional records36,37,45,56,61,69,71 and as such should be restricted to non-sensitive information37. Some were unwilling to be completely frank about their problems and would consider changing to another doctor68. There was suspicion on the part of both practitioners and patients about the negative impact of computers on the doctor-patient relationship32,33,36,67,69,71,79. This was partly due to the logistics of incorporating a computer in the consultation67,79 and partly to the perception that computers would take over the doctor’s role36.
Cost was viewed as being prohibitive to computerisation, both by practitioners30,33,50,56 and by patients61,69. Many doctors said that the time commitment involved in learning and using computers was too great33,50 and was more than was expected46, resulting in additional stress79. Finally, existing training in computer use was perceived as being poor56 and it was thought that this should be made a component of doctors’ medical career50.
Discussion
The growth of computerisation in the health service and in primary care in particular is set to continue, yet there remains a dearth of published evaluations into the impact of this technology. The most significant shortfall is in research on the impact of computers on patient outcomes and we identified only 17 studies in this area. Although this is a considerable increase on our 1984-94 total, it is not sufficient for what is almost certainly the most contentious issue surrounding computerisation in any field of medicine: whether computers provide real benefits for patients.
The majority of outcome studies used a rigorous methodology and rated highly for sample formation, adjusting for differences between the groups and the objectivity of the outcomes being measured. However, they rarely considered the full implications of using the patient as the unit of analysis rather than randomising work units (e.g. practices). This may be a reflection of the difficulties involved in researching a technology with which we are striving to keep up. Allocation by practice reduces the confounding effect of participating in research on those researched. However, computerisation in primary care is so widespread that finding practices which do not have the specific system feature to be evaluated as well as adequate controls is virtually impossible. Randomising practices to receive particular systems is also problematic. Not only is this expensive, but it often seems inconsequential; no sooner has the system been evaluated than it has been modified or updated and requires further evaluation.
The most fruitful areas of current research are preventive care, prescribing support, chronic disease monitoring, test ordering and hospital referral. Few studies have dealt with nursing research and little has been published on the impact of computer systems on other members of the primary health care team.
The majority of studies in this review found positive effects of computerisation, showing amongst other things, increases in immunisations and preventive care and reductions in prescribing costs and unnecessary tests. Practitioners and patients alike expressed positive opinions about computers, particularly in terms of access, accuracy and the time saving properties of electronic patient records. However, little has been done to help alleviate fears over the interference of computers in the process of the consultation and on the relationship between doctor and patient. We identified three new studies on consultation content for this latest review and again found that consultation length increased.
The proportion of non-interaction time also increased in one case by as much as 28%29 and this did not alter with improved proficiency. Another cause of anxiety for clinicians, and particularly for patients, is the issue of privacy and confidentiality of computerised records. Patients are not always made aware of the uses of information technology in primary care71 and this may account for their ongoing concern over this. The way forward may be a programme of research on the best ways of integrating the computer into the consultation, beginning with examples of current best practice and refining this based upon principles of effective communication.
It is over three decades since information technology was first introduced to primary care. In the sixties its use centred on collating patient data and in the seventies the possibility of electronically linking primary and secondary care emerged. In the eighties computers were introduced to the consulting room and in the nineties the Internet provided the potential to obtain and review useful information during the consultation. Perhaps after 30 years of analysing the ‘potential’ benefits of computers, we should allow primary care information technology to mature. In the 21st century we should accept that the computer is a useful tool. Rather than continually focusing on describing its capabilities, research must move forward to evaluate key outcomes for patients, practices and the health service as a wholel.
References
a. Department of Health. Computerisation in GP practices 1993 survey. Leeds: NHS Management Executive, 1993.
b. Barnett GO. The application of computer-based medical record systems in ambulatory practice. N Engl J Med 1984;310(25):1643-50.
c. Bradley P. Computers in primary health care. Comp Bull 1993;49:151-60.
d. van Bemmel JH, Musen MA, eds. Handbook of medical informatics. Heidelberg: Springer-Verlag, 1997.
e. Department of Health. The new NHS. London: HMSO, 1997.
f. Wyatt JC. Clinical data systems part III: development and evaluation. Lancet 1994;344:1682-88
g. Mowatt G, Bower DJ, Brebner JA, Cairns JA, Grant ASM, McKee L. When and how to assess fast changing technologies: a comparative study of medical applications of 4 generic technologies. Health Tech Assess 1997;1(14).
h. Sullivan F, Mitchell E. Has general practitioner computing made a difference to patient care? A systematic review of published reports. BMJ 1995;311:848-52.
i. Johnson ME, Langton KB, Haynes B, Mathieu A. Effects of computer-based clinical decision support systems on clinician performance and patient outcome. Ann Intern Med 1994;120:135-42.
j. van de Van A, Delbecq A. The effectiveness of Delphi and Interacting Group Decision Making Processes. Acad Man J 1974;17:605-21.
k. Gilliland AEW, Mills KA, Steele K. General practitioner records on computer-handle with care. Fam Pract 1992;9:441-50.
l. van Bemmel JH. Yearbook of medical informatics 1999: the promise of informatics. Stuttgart: Schatteur, 1999.
1. Gehlbach SH, Wilkinson WE, Hammond WE, Clapp NE, Finn AL, Taylor WJ, Rodell M. Improving drug prescribing in a primary care practice. Med Care 1984;22(3):193-201.
2. McDonald CJ, Hui SL, Smith DM, Tierney WM, Cohen SJ, Weinberger M, McCabe GP. Reminders to physicians from an introspective computer medical record. A two year randomized trial. Ann Intern Med 1984;100:130-8.
3. Roland MO, Zander LI, Evans M, Morris R, Savage RA. Evaluation of a computer assisted repeat prescribing programme in a general practice. BMJ 1985;291:456-8.
4. Tierney WM, Hui SL, McDonald CJ. Delayed feedback of physician performance versus immediate reminders to perform preventive care. Effects on physician compliance. Med Care 1986;24(8):659-66.
5. McDowell I, Newell C, Rosser W. Comparison of three methods of recalling patients for influenza vaccination. CMAJ 1986;135:991-7.
6. McDowell I, Newell C, Rosser W. A randomized trial of computerized reminders for blood pressure screening in primary care. Med Care 1989;27(3):297-305.
7. Chambers CV, Balaban DJ, Carlson BL, Ungemack JA, Grasberger DM. Microcomputer-generated reminders. Improving the compliance of primary care physicians with mammography screening guidelines. J Fam Pract 1989;29(3):273-80.
8. McPhee SJ, Bird JA, Jenkins CNH, Fordham D. Promoting cancer screening. A randomized, controlled trial of three interventions. Arch Intern Med 1989;149:1866-72.
9. McDowell I, Newell C, Rosser W. Computerized reminders to encourage cervical screening in family practice. J Fam Pract 1989;28(4):420-4.
10. Mazzuca SA, Vinicor F, Einterz RM, Tierney WM, Norton JA, Kalasinski LA. Effects of the clinical environment on physicians' response to postgraduate medical education. Am Educ Res J 1990;27(3):473-88.
11. McPhee SJ, Bird JA, Fordham D, Rodnick JE, Osborn EH. Promoting cancer prevention activities by primary care physicians. Results of a randomized, controlled trial. JAMA 1991;266(4):538-44.
12. Ornstein SM, Garr DR, Jenkins RG, Rust PF, Arnon A. Computer-generated physician and patient reminders. Tools to improve population adherence to selected preventive services. J Fam Pract 1991;32(1):82-90.
13. Rosser W, Hutchison BG, McDowell I, Newell C. Use of reminders to increase compliance with tetanus booster vaccination. CMAJ 1992;146(6):911-7.
14. McDonald CJ, Hui SL, Tierney WM. Effects of computer reminders for influenza vaccination on morbidity during influenza epidemics. MD Comp 1992;9(5):304-12.
15. McAlister NH, Covvey HD, Tong C, Lee A, Wigle ED. Randomized controlled trial of computer assisted management of hypertension in primary care.BMJ 1986;293:670-4.
16. Fitzmaurice DA, Hobbs FDR, Murray ET, Bradley CP, Holder R. Evaluation of computerized decision support for oral anticoagulation management based in primary care. Br J Gen Pract 1996;46:533-5.
17. Lewis G, Sharp D, Bartholomew J, Pelosi AJ. Computerized assessment of common mental disorders in primary care: effect on clinical outcome. Fam Pract 1996;13(2):120-6.
18. Rubenstein LV, McCoy JM, Cope DW, Barrett PA, Hirsch SH, Messer KS, Young RT. Improving patient quality of life with feedback to physicians about functional status. J Gen Intern Med 1995;10:607-14.
19. Burack RC, Gimotty PA, George J, Stengle W, Warbasse L, Moncrease A. Promoting screening mammography in inner-city settings: a randomized controlled trial of computerized reminders as a component of a program to facilitate mammography. Med Care 1994;32(6):609-24.
20. Litzelman DK, Dittus RS, Miller ME, Tierney WM. Requiring physicians to respond to computerized reminders improves their compliance with preventive care protocols. J Gen Intern Med 1993;8:311-7.
21. Moran WP, Nelson K, Wofford JL, Velez R. Computer-generated mailed reminders for influenza immunization: a clinical trial. J Gen Intern Med 1992;7:535-7.
22. Lobach DF, Hammond WE. Development and evaluation of a computer-assisted management protocol (CAMP): improved compliance with care guidelines for diabetes mellitus. JAMIA 1994;787-91.
23. Tierney WM, McDonald CJ, Martin DK, Hui SL, Rogers MP. Computerized display of past test results. Effect on outpatient testing. Ann Intern Med 1987:107;569-74.
24. McDowell I, Newell C, Rosser W. A follow-up study of patients advised to obtain influenza immunizations. Fam Med 1990;22(4):303-6.
25.Robson J, Boomla K, Fitzpatrick S, Jewell AJ, Taylor J, Self J, Colyer M. Using nurses for preventive activities with computer assisted follow up: a randomised controlled trial. BMJ 1989;298:433-6.
26. Brownbridge G, Evans A, Fitter M, Platts M. An interactive computerized protocol for the management of hypertension: effects on the general practitioner’s clinical behaviour. J Roy Coll Gen Pract 1986;36:198-202.
27. Liaw T, Lawrence M, Rendell J. The effect of a computer-generated patient-held medical record summary and/or a written personal health record on patients’ attitudes, knowledge and behaviour concerning health promotion. Fam Pract 1996;13(3):289-93.
28. Safran C, Rind DM, Davis RB, Ives D, Sands DZ, Currier J, Slack WV, Cotton DJ, Makadon HJ. Effects of a knowledge-based electronic patient record on adherence to practice guidelines. MD Comp 1996;13(1):55-63.
29. Warshawsky SS, Pliskin JS, Urkin J, Cohen N, Sharon A, Binztok M, Margolis CZ. Physician use of a computerized medical record system during the patient encounter: a descriptive study. Comp Meth Prog Biomed 1994;43:269-73.
30. Liaw ST. Information management in primary medical care in South Australia. Fam Pract 1994;11(1):44-50.
31. Margolis CZ, Warshawsky SS, Goldman L, Dagan O, Wirtschafter D, Pliskin JS. Computerized algorithms and pediatricians’ management of common problems in a community clinic. Acad Med 1992;67(4):282-4.
32. Moidu K, Wigertz O, Trell E. Multi center systems analysis study of primary health care: a study of socio-organizational and human factors. Int J Biomed Comp 1992;30:27-42.
33. Thakurdas P, Coster G, Gurr E, Arroll B. New Zealand general practice computerization; attitudes and reported behaviour. NZ Med J 1996;109:419-22.
34. Campbell SM, Roland MO, Gorman B. Evaluation of a computerized appointment system in general practice. Br J Gen Pract 1996;46:477-8.
35. Greatbatch D, Heath C, Campion P, Luff P. How do desk-top computers affect the doctor-patient interaction?. Family Practice 1995;12(1):32-36.
36. Ridsdale L, Hudd S. Computers in the consultation: the patient’s view. Br J Gen Pract 1994;44:367-9.
37. Carman D, Britten N. Confidentiality of medical records: the patient's perspective. Br J Gen Pract 1995;45:485-8.
38. DeVore PA. Computer-assisted comprehensive geriatric assessment in a family physician's office. South Med J 1991;84(8):953-5.
39. Singh AK, Moidu K, Trell E, Wigertz O. Impact on the management and delivery of primary health care by a computer-based information system. Comp Meth Prog Biomed 1992;37:55-64.
40. Brownbridge G, Herzmark GA, Wall TD. Patient reactions to doctors’ computer use in general practice consultations. Soc Sci Med 1985;20(1):47-52.
41. Tierney WM, Miller ME, McDonald CJ. The effect on test ordering of informing physicians of the charges for outpatient diagnostic tests. N Engl J Med 1990;322(21):1499-504.
42. Barton MB, Schoenbaum SC. Improving influenza vaccination performance in an HMO setting: the use of computer-generated reminders and peer comparison feedback. Am J Pub Health 1990;80(5):534-6.
43. Christensen-Szalanski JJJ, Diehr PH, Wood RW, Tompkins RK. Phased trial of a proven algorithm at a new primary care clinic. Am J Pub Health 1982;72(1):16-21.
44. Hutchison BG. Effect of computer-generated nurse/physician reminders on influenza immunization among seniors. Fam Med 1989;21(6):433-7.
45. Rethans JJ, Hoppener P, Wolfs G, Diederiks J. Do personal computers make doctors less personal?. BMJ 1988;296:1446-8.
46. Baldwin DW. Experience with microcomputers in general practice. Practitioner 1985;229:643-6.
47. Turner RC, Peden JG Jr, O'Brien K. Patient-carried card prompts vs computer-generated prompts to remind private practice physicians to perform health maintenance measures. Arch Intern Med 1994;154(17):1957-60.
48. Tape TG, Campbell JR. Computerized medical records and preventive health care: success depends on many factors. Am J Med 1993;94(6):619-25.
49. Harris RP, O'Malley MS, Fletcher SW, Knight BP. Prompting physicians for preventive procedures: a five-year study of manual and computer reminders. Am J Prev Med 1990;6(3):145-52.
50. Rowe BH, Ryan DT, Therrien S, Mulloy JV. First-year family medicine residents' use of computers: knowledge, skills and attitudes. CMAJ 1995;153(3):267-72.
51. Lobach DF. Electronically distributed, computer-generated, individualised feedback enhances the use of a computerized practice guideline. Proc AMIA Ann Fall Symp 1996:493-7.
52. Krall MA. Acceptance and performance by clinicians using an ambulatory electronic medical record in an HMO. Proc Ann Symp Comp App Med Care 1995:708-11.
53. Jones DL, Kroenke K, Landry FJ, Tomich DJ, Ferrel RJ. Cost savings using a stepped-care prescribing protocol for nonsteroidal anti-inflammatory drugs. JAMA 1996;275(12):926-30.
54. Rosser WW, McDowell I, Newell C. Use of reminders for preventive procedures in family medicine. CMAJ 1991;145(7):807-12.
55. Frame PS, Zimmer JG, Werth PL, Hall WJ, Eberly SW. Computer-based vs manual health maintenance tracking. A controlled trial. Arch Fam Med, 1994;3(7):581-8.
56. Bolton P, Gay R. Review of computer usage among RACGP members. Aust Fam Phys 1995;24(10):1882-5.
57. Solomon GL, Dechter M. Are patients pleased with computer use in the examination room?. J Fam Pract 1995;41(3):241-4.
58. Chambers CV, Balaban DJ, Carlson BL, Grasberger DM. The effect of microcomputer-generated reminders on influenza vaccination rates in a university-based family practice center. J Am Board Fam Pract 1991;4(1):19-26.
59. Valenta AL, Wigger U. Early results of user profiles: physicians' opinions on the use of information technology. Proc AMIA Ann Fall Symp 1996:65-69.
60. Strecher VJ, Kreuter M, Den Boer DJ, Kobrin S, Hospers HJ, Skinner CS. The effects of computer-tailored smoking cessation messages in family practice settings. J Fam Pract 1994;39(3):262-70.
61. Ornstein S, Bearden A. Patient perspectives on computer-based medical records. J Fam Pract 1994;38(6):606-10.
62. Quattlebaum TG, Darden PM, Sperry JB. Effectiveness of computer-generated appointment reminders. Pediatrics 1991;88(4):801-5.
63. Hobbs FDR, Delaney BC, Carson A, Kenkre JE. A prospective controlled trial of computerized decision support for lipid management in primary care. Fam Pract 1996;13(2):133-7.
64. van den Hoogen JPH, van Ree JW. Preventive cardiology in general practice: computer-assisted hypertension care. J Hum Hyp 1990;4(4):365-7.
65. Turner BJ, Day SC, Borenstein B. A controlled trial to improve delivery of preventive care: physician or patient reminders?. J Gen Intern Med 1989;4(5):403-9.
66. Davidson KW, Kahn A, Price RD. Reduction of adverse drug reactions by computerized drug interaction screening. J Fam Pract 1987;25(4):371-5.
67. Cruickshank PJ. Computers in medicine: patients' attitudes. J Roy Coll Gen Pract 1984;34:77-80.
68. Potter AR. Computers in general practice: the patient's voice. J Roy Coll Gen Pract 1981;31:683-5.
69. Pringle M, Robins S, Brown G. Computers in the surgery. The patient's view BMJ;1984:288, 289-91.
70. Barnett GO, Winickoff RN, Morgan MM, Zielstorff RD. A computer-based monitoring system for follow-up of elevated blood pressure. Med Care 1983;21(4):400-9.
71. Als AB. The desk-top computer as a magic box: patterns of behaviour connected with the desk-top computer; GPs' and patients' perceptions. Fam Pract 1997;14(1):17-23.
72. Bright S. Nearest and dearest. Br J Health Comp 1991:58-9.
73. Burack RC, Gimotty PA, George J, Simon MS, Dews P, Moncrease A. The effect of patient and physician reminders on use of screening mammography in a health maintenance organization. Results of a randomized controlled trial. Cancer, 1996:78(8):1708-21.
74. Fihn SD, McDonnell MB, Vermes D, Henikoff JG, Martin DC, Callahan CM, Kent DL, White RH. A computerized intervention to improve timing of out-patient follow-up: a multicenter randomized trial in patients treated with warfarin. J Gen Intern Med 1994;9(3):131-9.
75. Landis SE, Hulkower SD, Pierson S. Enhancing adherence with mammography through patient letters and physician prompts. A pilot study. N Carolina Med J, 1992:53(11):575-8.
76. Tierney WM, McDonald CJ, Hui SL, Martin DK. Computer predictions of abnormal test results. Effects on outpatient testing. JAMA, 1988:259(8):1194-8.
77. McDonald CJ, Wilson GA, McCabe GP. Physician response to computer reminders. JAMA 1980;244(14):1579-81.
78. Lobach DF, Hammond WE. Computerized decision support based on a clinical practice guideline improves compliance with care standards. Am J Med 1997;102:89-98.
79. Herzmark G, Brownbridge G, Fitter M, Evans A. Consultation use of a computer by general practitioners. J Roy Coll Gen Pract 1984;34:649-54.
80. Pringle M, Robins S, Brown G. Computer assisted screening: effect on the patient and his consultation. BMJ 1985;290:1709-12.
81. Pringle M, Robins S, Brown G. Timer: a new objective measure of consultation content and its application to computer assisted consultations. BMJ 1986;293:20-2.
82. Pringle M, Robins S, Brown G. Topic analysis: an objective measure of the consultation and its application to computer assisted consultations. BMJ 1985;290:1789-91.
83. Weingarten MA, Bazel D, Shannon HS. Computerized protocol for preventive medicine: a controlled self-audit in family practice. Fam Pract 1989;6(2):120-4.
84. Brownbridge G, Evans A, Wall T. Effect of computer use in the consultation on the delivery of care. BMJ 1985;291:639-42.
85. Garr DR, Ornstein SM, Jenkins RG, Zemp LD. The effect of routine use of computer-generated preventive reminders in a clinical practice. Am J Prev Med 1993;9(1):55-61.
86. Donald JB. On line prescribing by computer. BMJ 1986;292:937-9.
87. Donald JB. Prescribing costs when computers are used to issue all prescriptions. BMJ 1989;299:28-30.
88. Sullivan FM, Manchip A, Hussain S. Does the arrival of a desk-top computer reduce patients' satisfaction with consultations in general practice?. Theor Surg 1992;7:161.
89. Mitchell ED, McConnachie A, Sullivan FM, Howie JGR. The impact of information technology on general practitioner consultations in Scotland. Final report for the Clinical Resource and Audit Group of the Scottish Office Home and Health Department. December 1996.
Acknowledgements
We are grateful to everyone who provided us with information for this review. We are grateful for the input from the expert panel who participated in our Delphi study; we know that this was time consuming and are very appreciative of their help. We would also like to thank Dr Sue Ross for her advice and Michere Beaumont for secretarial support.
The results of this systematic review are also available in a MS Access database. This can be obtained on disk from E Mitchell.
What is already known on this subject?