Computer-generated Patient Education Materials: Do They Affect Professional Practice?

Abstract

A systematic search of seven electronic databases was done to identify randomized controlled trials that assessed the effect of computer-generated patient education material (PEM) on professional practice. Three studies met the authors' criteria.

All three studies involved preventive care. All used a complex intervention of which computer-generated PEM was a major component. Improvements in practice were seen in all studies, although these gains were generally modest. One study showed improvement in patient outcomes. Mann-Whitney statistics calculated for the studies' outcome measures ranged from 0.48 to 0.66, equivalent to risk differences of −4 to 32 percent.

Computer-generated PEM seems to have a small, positive effect on professional practice. The small number of included studies and the complex nature of the interventions makes it difficult to draw conclusions about the ability of computer-generated PEM to change professional practice. Future work should involve well-defined interventions that can be clearly evaluated in terms of effect and cost.

The traditional consultation between a health professional and a patient suffers from the flawed memories of both participants. A study by Kitching ¹ found that, on average, patients forget half of what they were told by a doctor within 5 min of leaving the consultation room. In addition, health professionals can easily forget to pass on important information to the patient. There is, therefore, scope for improvement in communication between professionals and patients.

This is especially true if patients are to be more active participants in their own care. One solution proposed by Coulter et al. ² is to ensure that patients have access to written or audiovisual material that supplements the material discussed during the consultation, which can also be used to guide questioning during later consultations.

This material could be delivered in several ways. Pre-printed booklets or leaflets are the most widely used method, ³ but some authors ^{4, 5} suggest that computerized clinical information systems should routinely include education materials. These materials could also be tailored to individual patients. Another method would be to use interactive educational software packages that allow the patient to direct the delivery of information. These packages have been used to combine audio, video, text, and graphics to provide information on a wide range of topics. ⁶ The Internet is also an increasingly important source of health information, although the quality of this information is currently variable. ⁷

Are health professionals willing to make use of education materials and packages in their practice? Other reviews have focused on patient behavior and outcomes. ^{8, 9} The health professional is, however, often the point of access to these materials, and an education package should, at the very least, be good enough for a professional to want to make it available to patients.

Two potential barriers to increased use of education materials by professionals are storage and access problems (booklets and leaflets take up space and are not always to hand) and the need to keep the materials up to date. ¹⁰ Wilson ¹⁰ suggests computer-generated materials as a solution, because these materials are not subject to storage limitations and can be easily updated. Moreover, these materials are immediately available to the health care professional during a consultation, which is often not the case for preprinted leaflets. The computer system can also be used to remind the clinician about the patient information. ¹¹ However, use of computer-based materials may be limited by financial, administrative, and attitudinal barriers in a health care organization or among its health care professionals.

As Mayberry and Mayberry ¹² point out, it is important that the effect of patient education material (PEM) should be assessed with the same rigor as other interventions. In this review, we aim to systematically determine the effects of computer-generated PEM on professional practice (our primary outcome) and on patient outcomes.

Objectives

To examine the effect of computer-generated PEM as a way of changing the practice of health care professionals and patient outcomes. Three primary comparisons will be considered:

• Comparison 1—providing computer-generated material compared to alternative material (preprinted leaflets, booklets, etc.).

• Comparison 2—providing computer-generated material compared to no material.

• Comparison 3—providing patient-specific computer-generated material compared to generic computer-generated material.

Methods

Our review is based on searching the following databases, from their start date to the end of 2000:

• Medline (1966–2000)

• Embase (1980–2000)

• Cinahl (1982–2000)

• Best Evidence (1991–2000)

• Cochrane Collaboration EPOC specialized register (1966–2000)

• Cochrane Controlled Trials Register (1947–2000)

• Science Citation Index (1987–2000)

Individual search strategies were developed for each database (Appendix A). The search strategies were very liberal and designed to pick up as many potential articles as possible. Publications in any language were considered. The reference lists of review articles identified through the searches were also checked. The initial search identified 1,147 articles. A further 14 articles were identified by checking the reference lists of the review articles, giving a total of 1,161 articles.

Study Selection

Studies were considered eligible for inclusion in the review if they met inclusion criteria in four categories—types of intervention, types of studies, types of participants, and types of outcome measures.

Types of Intervention

The intervention must be PEM that involves the use of a computer in its storage and delivery, including interactive packages. The information in such material may be general or specific to individual patients. Education materials used with and without the health professional being present were considered eligible.

To be included in this review, education interventions must be delivered in one of three ways—before a consultation (e.g., information available on computers in the waiting room), during a consultation (e.g., a leaflet printed out and handed to the patient), or after a consultation (e.g., a computer-generated tailored leaflet, based on the consultation, that is sent to the patient)

Interventions that were simple patient reminders (e.g., a simple computer-generated letter suggesting that a patient come in for a health check) were excluded, since we considered these to have no education content. Moreover, a number of reviews have already considered these interventions ^{13– 15} or are in progress. ¹⁶

Types of Studies

Studies must be randomized controlled trials or controlled trials. The unit of allocation must be the health professional or practice, since we are primarily interested in professional practice. ¹⁷

Types of Participants

A study must involve health care professionals who are responsible for patient care. Studies involving only research staff without clinical responsibility for patients were excluded.

Types of Outcome Measures

All objective measures of health care professional practice and patient outcome were considered.

The primary outcome measure for this review is effect on professional practice, including delivery of information, cost, time spent with patients, prescribing, referrals, and other clinical activities. Studies with no measure of effect on professional practice were excluded.

Abstracts for the 1,161 articles were scanned for relevance by at least two reviewers. Full-text copies of all potentially relevant studies were obtained. These articles were then considered for inclusion in the review by at least two reviewers. Any discrepancies between reviewers arising from the inclusion assessment were resolved by discussion.

Analysis

The Mann-Whitney statistic was used to compare outcomes for intervention and control groups. This statistic estimates the probability that a subject chosen at random and given the intervention would have a better outcome than a subject similarly chosen and given the control. The statistic ranges from 0 to 1, with 0.5 meaning that both groups perform or respond equally well.

For dichotomous data, the Mann-Whitney statistic was calculated using:

where p _i and p _c are the proportions of intervention “successes” (e.g., a screening test was done) and control successes, respectively. The quantity p _i−p _c is also called the risk difference.

For continuous data, a Z statistic was calculated, and normal tables were then used to find the Mann-Whitney statistic. The Z statistic was calculated using the equation:

where _i− _c is the difference between the intervention and control means, and SD _i and SD _c are the standard deviations for the intervention and control means, respectively. More details on the use of the Mann-Whitney statistic to compare the results of intervention studies can be found in Colditz et al. ¹⁸ and Moses et al. ¹⁹

Results

General Information

Of the 1,161 articles identified in our search, only three met all our inclusion criteria—Williams et al. ²⁰ (referred to here as “Williams”), Lowensteyn et al., ²¹ (“Lowensteyn”), and McPhee et al. ²² (“McPhee”). Most articles were excluded because they discussed studies that were randomized by patient and not by professional, even if the study aimed to effect a change in professional practice. Articles that appeared relevant but were rejected after consideration of the full article text are listed in Appendix B.

Table 1▶ gives an overview of the Williams, Lowensteyn, and McPhee studies. All three are based in primary care, Williams and McPhee aiming to improve use of cancer screening, Lowensteyn aiming to support primary prevention of coronary heart disease. The Williams and McPhee studies were done in the United States, the Lowensteyn study in Canada.

Table 1 .

Overview of the Three Included Studies

Study	No. in Study	Length of Study	Study Design	Age of Patients	Disease or Event and Intervention/Control	Main Process Effect Measured
Lowensteyn et al., 1998	253 general practitioners	1 yr	Randomized controlled trial by general practice, blocking for presence or absence ofmedical school in community	30–74 yr	Prevention of coronary heart generated CHD risk profile PEM given to patient 2 wk after first consultation vs. profile given at 3-mo follow-up.	Ratio of high-risk to disease (CHD). Computer-low-risk patients seen at 3-mo follow-up
McPhee et al., 1991	40 general practitioners	2 yr	Randomized controlled trial with two groups	64% >50yr	Prevention of cancer. Computer-generated screening reminder for booklet-based PEM vs. current practice	Average change in the proportion of eligible patients patient and doctor plus screening tests
Williams et al., 1998	58 primary care practices, average 2.3 physicians per practice	1 yr	Randomized controlled trail by general practice, stratified by type (e.g., high or low link to medical college); total of five groups.	>18 yr	Prevention of cancers of the rectum, and oral cavity. PEM and chart reminders generated by a touch-sensitive computer system plus nurse liaison vs. current practice	Average change in breast, cervix, colon andthe proportion of eligible patients having cancer screening tests

Abbreviation: PEM indicates patient education material.

Physicians participating in the McPhee and Lowensteyn studies were not paid for their participation. Physicians in the Williams study who already subscribed to the Virginia Insurance Reciprocal malpractice insurance scheme received a one-time 6 percent reduction in their annual premium in return for their participation. Physicians involved in the McPhee study were all based at fee-for-service practices. Reimbursement arrangements at practices involved in the two other studies were not clear, although one of the five strata in the Williams study involved federally funded practices. None of the studies adequately described the randomization process.

Interventions

No study used an intervention involving only computer-generated PEM. Williams asked patients to use a touch-sensitive computer system to complete a questionnaire on personal and family medical history and lifestyle. The computer then produced PEM, chart organizers, order sheets, and patient-specific reminders for the physician, the PEM being given to the patient during the consultation. A nurse was also available to help practices during implementation of the system. Control practices did not receive the computer system until the end of the 1-year study period.

Lowensteyn collected risk factor information from the patient, and this information was then mailed to a central collection point. This center returned two risk profiles to the practice, one of which was given to the patient at a return visit about 2 weeks after the initial visit. Control practices did not receive the two risk profiles unless the patient was clinically re-evaluated at a follow-up visit following a minimum 3-month delay.

McPhee combined computer-generated PEM with booklet-based PEM and a physician reminder system. The physician reminder system generated information on appropriate screening, assessment, and counseling and other test information. Two copies of this information were generated, one each for the physician and patient. Control practices received nothing.

All three studies considered Comparison 2 (computer-generated PEM vs. no material) and provided estimates of the effects of computer-generated PEM on professional practice. One study (Lowensteyn) also provided estimates of effect on patient outcomes.

Williams reported a significant increase in the completion of screening mammography (8.8 percent difference between intervention and control) and clinical breast examinations (8.3 percent) in women 50 years of age and older. These results were significant at the p≤0.05 level. There were no significant changes for the five remaining screening tests.

Among patients who did not have a health maintenance examination (HME), use of the touch-screen system showed a significant increase (p≤0.05) in the number of patients who had fecal occult blood tests (3.9 percent increase). Patients who did have an HME were associated with higher (p≤0.01) proportions of screening mammographies (30.3 percent increase) and clinical breast examinations (32.4 percent). Williams also reported that patients who used the touch-screen system had a significantly higher rate of completion of screening tests than did non-users for all seven screening tests. However, not all non-users were in the control group, which may lead to a biased result in favor of the intervention (see Discussion).

Lowensteyn found that the coronary heart disease risk profile given to both patients and physicians led to a significant increase in the proportion of high-risk patients being reassessed at 3 months, compared with low-risk patients. The difference in the likelihood ratio (ratio of high-risk patients to low-risk patients returning for follow-up) between the risk profile group and the control group was 0.46 (95% confidence interval, 0.08–0.87). Lowensteyn had a difference in the unit of allocation and analysis, which is a methodological weakness, although the analysis of patient outcome does take this into account. This was not done for the risk likelihood calculation, making it unclear whether statistical significance was actually achieved for this result.

McPhee developed a percentage “performance score” for completion of screening tests and assessments, and the intervention package led to increases of between 8 and 34 percent in completion. These increases were significant for five tests or assessments (rectal examination, Papanicolaou smear, smoking assessment, smoking counseling, and diet assessment) at the p≤0.05 level and for three (stool occult-blood test, pelvic examination, and diet counseling) at the p≤0.01 level. The results for sigmoidoscopy, breast examination, and mammography were not significant.

The results were then adjusted to remove the effects of patient characteristics, using patients as the unit of analysis. As mentioned above, using patients as the unit of analysis when allocation was by physician is a methodological weakness. The adjusted results were broadly similar to the unadjusted results, but with higher levels of significance generally being claimed.

Neither Williams nor McPhee reported patient outcomes. Lowensteyn found that patients in the intervention group demonstrated greater reductions (p<0.05) in total cholesterol, low-density lipoprotein cholesterol, and the total cholesterol/high-density lipoprotein cholesterol ratio. This resulted in a significant improvement (p<0.01) in both cardiovascular age and predicted 8-year coronary risk, compared with the control group. As mentioned earlier, the units of allocation and analysis are different in this study. The authors did, however, include physician as a variable in an analysis of variance model when considering patient outcomes.

Table 2 ▶summarizes the Mann-Whitney statistics for the three studies, together with the published p values for each outcome. A Mann-Whitney statistic has not been calculated for the Lowensteyn “high-risk likelihood ratio,” because different units of allocation and analysis were used for this outcome. The spreadsheet used to calculate the Mann-Whitney statistics is available on request from the authors of this article.

Table 2 .

Summary of the Mann-Whitney Statistics for Each of the Three Included Studies

Study	Outcome Measures	Mann-Whitney Statistic	Published p Values
Lowensteyn et al., 1988	High-risk likelihood ratio	Not possible	95%CI, 0.08–0.87
	Total C (mmol/l)	0.61	p=0.05
	HDL C	0.48	NS
	LDL C	0.61	p=0.05
	Total C/HDL C ratio	0.57	p=0.05
	Systolic BP	0.52	NS
	Diastolic BP	0.53	NS
	Body mass index	0.50	NS
	Smokers	0.50	NS
	8-yr coronary risk	0.54	p<0.01
	Cardiovascular age	0.51	p<0.01
McPhee et al., 1991	Stool occult blook test	0.56	p<0.01
	Digital rectal examination	0.55	p<0.05
	Flexible sigmoidoscopy	0.57	NS
	Papanicolaou smear	0.66	p<0.05
	Pelvic examination	0.56	p<0.01
	Breast examination	0.54	NS
	Mammography	0.52	NS
	Smoking assessment	0.54	p<0.05
	Smoking counseling	0.61	p<0.05
	Diet assessment	0.56	p<0.05
	Diet counseling	0.57	p<0.01
Williams et al., 1998	Screening mammography	0.54	p≤0.05
	Clinical breast examination	0.54	p≤0.05
	Digital rectal examination	0.51	NS
	Fecal occult blood test	0.51	NS
	Flexible sigmoidoscopy	0.51	NS
	Papanicolaou smear	0.51	NS
	Oral cavity examination	0.51	NS

Abbreviations: 95%CI indicates 95% confidence interval; C, cholesterol; HDL C, high-density lipoprotein; LDL, low-density lipoprotein; BP, blood pressure; NS, not significant.

Figure 1▶ shows a plot of the Mann-Whitney statistics for each study's major outcome measures. The point size is proportional to the sample size (actually 1/variance; see Figure 1▶). To make differentiation between the points easier, the x-axis has been started at 0.4, not 0.

Figure 1 .

Mann-Whitney statistics for each of the three included studies. The size of each point is proportional to 1/variance, where variance is given by

and where n _i and n _c are the sample sizes for the intervention and control groups, respectively. The crosshair represents the mean Mann-Whitney statistic for each study.

The Mann-Whitney statistics were largely in favor of the intervention (i.e., greater than 0.50), ranging from 0.48 (equivalent to a risk difference [RD] of −4 percent) to 0.66 (RD, 32 percent). The mean Mann-Whitney statistics were all in favor of the intervention: Williams had a mean of 0.52 (RD, 4 percent), McPhee 0.57 (RD, 14 percent), and Lowensteyn 0.54 (RD, 8 percent).

Discussion

This review has examined the effects of computer-generated PEM on the practice of health professionals. The small number of studies included in this review is a clear indication that the effect of computer-generated PEM on professional practice is an aspect of PEM that is rarely evaluated. In addition, several studies that did assess the effect of computer generated PEM on professional practice failed to randomize by professional or practice, a methodological error that tends to overestimate any effect. ^{17, 23} One included study that was randomized by practice used the patient as the unit of analysis, which can lead to over-narrow confidence intervals and spuriously significant findings. ²³

It is, therefore, difficult to draw conclusions about the effects of computer-generated PEM on professional practice. The Williams, Lowensteyn, and McPhee studies report positive effects, and most of the Mann-Whitney statistics given in Table 2▶ favor the intervention. Table 2▶ and Figure 1▶ suggest that for these three North American studies, involving similar groups of patients and health professionals and using broadly similar interventions, PEM can contribute to a small improvement in professional and patient outcomes. Taking the Mann-Whitney statistics together, the risk difference between PEM intervention and control is around 8 to 10 percent.

Williams found a significant increase (about 8 percent) in the proportion of women older than 50 years who received screening mammography and clinical breast examinations. These authors also compared patients who had used the touch-screen system with patients who had not. Patients who used the system showed a significant increase in the completion of all seven cancer screening tests. However, this result is likely to be optimistic, since not all non-users were in the control group.

At practices in which the touch-screen system was placed in the waiting area (20 of 29), patients who used the system were self-selected. At the remaining nine practices, the system was used by a group of patients selected by the practice. Both methods are likely to result in bias regarding use of the system, and non-users at intervention practices should be considered when assessing the intervention's effect. The intervention also involved a physician reminder, and it is difficult to disentangle the effect of the reminder from the computer-generated PEM. The positive effect of physician reminders is well documented. ^{13, 14} This problem is perhaps more acute with the McPhee study.

The Lowensteyn study, which combined computer-generated PEM with a physician reminder system for managing coronary risk, achieved a significant increase in the ratio of high-risk to low-risk patients returning for follow-up. However, the unit of allocation was different from the unit of analysis. It is likely that the result would have been less favorable had the data been analyzed (correctly) using the 129 physicians who actively participated instead of the 958 patients. An alternative for the original authors would have been to use an intra-cluster correlation coefficient, which would describe the extent to which patients within a cluster (a practice) are truly independent of each other.

We were interested primarily in the effect of PEM on professional practice. Eight studies were therefore excluded from our review because patients rather than professionals were randomized. For studies that aim to measure this effect, randomization by professional is appropriate because randomizing by patient would lead to the same professionals being in both the experimental and the control groups.

Randomizing by patient would be appropriate only if the PEM was delivered independent of the professional and the aim of a study was to measure changes in patient rather than professional behavior. Internet-based PEM obtained without professional input would ensure delivery independent of professionals and prevent contamination in that respect, but the same professionals would still be exposed (via the patients) to both the experimental and control interventions. Thus, randomization at the level of the professional is the only way to reliably measure effects on professional practice.

To illustrate other computer-based PEM systems, however, we thought it would be interesting to consider these eight articles in the Discussion; they are listed in Table 3▶. Of these, four considered Comparison 1, ^{24– 27} five considered Comparison 2, ^{27– 31} and two considered Comparison 3. ^{25, 27} Four studies reported a significant positive change in an outcome related to professional practice, ^{24, 26– 28} whereas the four remaining studies reported no significant change in these outcomes. None of the eight studies reported objective measures of patient outcome; patients' knowledge of their condition was the most common measure reported.

Table 3 .

Overview of the Eight Studies that Did Not Meet the Randomization Criteria but Did Meet the Three Other Inclusion Criteria

Study	No. in Study	Length of Study	Study Design	Age of Patients (yr)	Disease or Event and Intervention/Control	Main Process Effect Measured	Features of System
Campbell et al., 1997	579, selected by staff	6 mo	Randomized controlled trial with two groups	18–70	Cervical cancer. Touch screen survey in waiting room generates PEM vs. use of touch screen system without PEM	Uptake of Papanicolaou screening tests	Touchscreen system located in waiting room, patients given a printed summary of their responses.
Graham et al., 2000	875, self-selected	9 mo	Randomized controlled trial with two groups	30 (mean)	Prenatal tests. Touch screen information system plus information leaflet vs. leaflet only	Uptake of prenatal tests	Touch screen system with video clips and voice-overs. system located in waiting area. Privacy enhanced by the availability of microphone headsets.
Homer et al., 2000	137, self selected (by parents)	1 yr	Randomized controlled trial with two groups	3–12	Asthma. Educational asthma-specific computer game vs. printed materials	Emergency department	Video game that incorporated visits and acute office information on the management of visits for asthma.asthma.
Jones et al., 1999	525, self-selected	3 mo	Randomized controlled trial with three groups	Range not given	Cancer. Computer-generated PEM that was general, personalized, or both vs. information booklet	Cost of providing information	Touch screen system in two selected variants, personalized and general. Personalized system linked to electronic medical record. Navigation system similar to that used on Web pages. Patients were sent printed summaries of information they had viewed.
Liaw et al., 1996	364, 261 recruited by mail, 103 by opportunistic contact at clinic	6 mo	Randomized controlled trial with four groups	25–65	General health promotion. Four interventions—HEA personal health record (PHR), summary of computer-based medical record (CMR), both PHR and CMR, or neither.	Attendance at a health check	Used health promotion template on recruited by mail, 103 by opportunistic contact at clinic a standard electronic medical record system. Patients given a printed summary of their medical record.
Morgan et al., 2000	240, self-selected	6 mo	Randomized controlled trial with two groups	60 (mean)	Ischemic heart disease. interactive video disk decision aid plus brochure vs. current practice	Uptake of revascularization procedures	Interactive video system providing risk and benefit information regarding three treatment alternatives. Information tailored to patient's medical and personal circumstances. Included film clips of patients who took the various treatment alternatives.
Osman et al. 1994	801, selected by staff	1 yr	Pragmatic randomized	?16	Asthma. Four computer-generated PEM booklets plus normal verbal advice vs. normal verbal advice alone.	Number of hospital admissions, consultations with general practitioner for asthma and prescrip-tion of bronchodilators and inhaled steroids	Used simple spreadsheet and mail merge software to generate personalized information for patients.
Rubin et al., 1986	65, self-selected (by parents)	1 yr	Randomized controlled trial with two groups	7–12	Asthma. Educational asthma-specific computer game vs. normal computer game.	Number of acute visits	Video game that incorporated information on the management of asthma.

Abbreviations: PEM indicates patient education materials; HEA, Health Education Authority.

It must be emphasized that these studies are not included to provide quantitative evidence to support or refute the use of particular techniques. As mentioned earlier, studies that assess the effect of an intervention on professional practice should be randomized by professional or practice, not by patient. ¹⁷ Even if the focus is patient outcome, if patients are drawn from several clinical sites, then clustering effects must be considered. These eight studies do, however, provide further (qualitative) food for thought.

None of the three studies meeting our full inclusion criteria considered cost. It is therefore uncertain whether the effects of the interventions are worth the costs involved. One of the studies that did not meet our randomization criteria, that by Jones et al., ²⁵ did mention cost. These authors found that, in the absence of an electronic medical record system, their personalized computer-based PEM system would cost more than nine times that of their general computer-based PEM. With an electronic record, the cost of the personalized system was similar to that of the general system. The cost of the general computer-based PEM system was about 40 percent the cost of full access to booklets and would be less than the cost of booklets within the first year. This suggests that the cost of computer-based PEM may be modest and that even small positive effects may be worth the investment, particularly in the presence of an electronic medical record.

Personalized systems were used in all three of the studies that met our full inclusion criteria. In addition, four of the six studies that did not meet our randomization inclusion criteria ^{24, 25, 27, 28} used personalized systems. Only two studies ^{25, 27} compared these with general computer-based PEM, and both studies reported benefits from use of the personalized system. As Jones et al. ²⁵ point out, in the presence of an electronic medical record, such systems are of similar cost to general systems, and a personalized approach may be worth considering in future computer-based PEM systems. The evidence supporting a positive effect on professional practice is, however, currently weak.

Making computer-based PEM part of an intervention package, rather than the sole intervention, may lead to a greater effect on practice. Combining PEM systems with reminder systems may be especially useful, since reminders have been found to have a significant positive effect on professional practice. ^{13– 15} This approach was taken in all three studies that met our full inclusion criteria, although it is difficult to wholeheartedly endorse the approach on the basis of the results of these three studies.

Conclusions

Implications for Practice

Computer-generated PEM appears to have a small, positive effect on professional practice. However, there is currently scant evidence to support this conclusion. Combination with other interventions, particularly patient and physician reminder systems may also be promising, but further work is required to ascertain which interventions should be combined and in which circumstances. The cost effectiveness of computer-generated PEM has been inadequately assessed.

Until further evidence becomes available, purchasers should be cautious about using computer-generated PEM as a vehicle for large-scale change in professional practice. Other considerations, however, such as improved patient participation in decision making and increased patient satisfaction, may make such investment worthwhile. In decisions about whether investment in computer-generated PEM meets a provider's health care strategy, the current review should be considered with reviews that take a more patient-centered focus. ^{8, 9}

Implications for Future Research

It is surprising that only three studies can be said to adequately address the effects of computer-generated PEM on professional practice. More and better PEM is a feature of many national health policies, and the attitudes of health professionals toward this, and the potential effect of PEM on their practices, is still largely unknown. Future work should address these issues, should be of higher methodological quality, should have a well-defined intervention package that can be clearly evaluated, and should consider cost. How such systems can be implemented into routine care should also be considered.

Internet-based services are already an important source of PEM, and the importance of the Internet as a source of PEM is likely to increase. The effect of these systems on professional practice and patient outcomes should be evaluated. Since computer-generated PEM is likely to have, at best, a modest effect on professional practice and patient outcome, developers should aim to make their systems cheap, preferably linked to existing electronic medical record systems.

Appendix A: Search Strategies for the Review

Effective Practice and Organization of Care (EPOC) specialized register search strategy:

comput* [text term]

Cochrane Controlled Trials Register (CCTR) search strategy:

1. comput* [text term]

2. Patient-education [exploded MeSH term]

3. 1 and 2

MeSH terms for medline search strategy (through Internet Grateful Med):

1. Patient Education

2. Health Education

3. Delivery of Health Care

4. Patient Participation

5. Health Promotion

6. Preventive Medicine

7. Professional-Patient relations

8. Consumer Participation

9. Computers

10. comput* [text term]

11. Computer-Assisted Instruction

12. Therapy, Computer-Assisted

13. Medical Informatics Applications

14. Medical Informatics

15. Clinical Trial [exploded Publication Type]

16. (1 or 2 or 3 or 4 or 5 or 6 or 7 or 8) and (9 or 10 or 11 or 12 or 13 or 14) and 15

MeSH terms for EMBASE search strategy (through Ovid):

1. exp Patient Education/

2. exp Health Promotion/

3. exp Patient counseling/ or Patient guidance/ or Patient information/ or Patient education

4. exp ‘automation, computers and computer applications’/ or Computer/

5. exp Clinical trial/

6. (1 or 2 or 3) and 4 and 5

Best Evidence search strategy (through Ovid):

Comput$ [text terms]

MeSH terms for Cinahl search strategy (through Ovid):

1. exp Patient education/

2. exp ‘Computers and computerization’/

3. exp Experimental studies/

4. 1 and 2 and 3

Science Citation Index search strategy (through BibSys):

1. computer (text word)

2. patient (text word)

3. education (text word)

4. 1 and 2 and 3