Effect of point-of-care computer reminders on physician behaviour: a systematic review

Abstract

Background

The opportunity to improve care using computer reminders is one of the main incentives for implementing sophisticated clinical information systems. We conducted a systematic review to quantify the expected magnitude of improvements in processes of care from computer reminders delivered to clinicians during their routine activities.

Methods

We searched the MEDLINE, Embase and CINAHL databases (to July 2008) and scanned the bibliographies of retrieved articles. We included studies in our review if they used a randomized or quasi-randomized design to evaluate improvements in processes or outcomes of care from computer reminders delivered to physicians during routine electronic ordering or charting activities.

Results

Among the 28 trials (reporting 32 comparisons) included in our study, we found that computer reminders improved adherence to processes of care by a median of 4.2% (interquartile range [IQR] 0.8%–18.8%). Using the best outcome from each study, we found that the median improvement was 5.6% (IQR 2.0%–19.2%). A minority of studies reported larger effects; however, no study characteristic or reminder feature significantly predicted the magnitude of effect except in one institution, where a well-developed, “homegrown” clinical information system achieved larger improvements than in all other studies (median 16.8% [IQR 8.7%–26.0%] v. 3.0% [IQR 0.5%–11.5%]; p = 0.04). A trend toward larger improvements was seen for reminders that required users to enter a response (median 12.9% [IQR 2.7%–22.8%] v. 2.7% [IQR 0.6%–5.6%]; p = 0.09).

Interpretation

Computer reminders produced much smaller improvements than those generally expected from the implementation of computerized order entry and electronic medical record systems. Further research is required to identify features of reminder systems consistently associated with clinically worthwhile improvements.

Computerized systems for entering orders and electronic medical records represent two of the most widely recommended improvements in health care.¹ These systems offer the opportunity to improve practice by delivering reminders to clinicians at the point of care. Such reminders range from simple prescribing alerts to more sophisticated support for decision-making.

Previous reviews have classified all computer reminders together, including computer-generated paper reminders and email alerts sent to providers, along with reminders generated at the point of care.^2–5 They have also typically reported the proportion of studies with results that were on balance “positive.”^2–4 We conducted a systematic review to quantify the expected magnitude of improvements in processes of care from computer reminders delivered to physicians during their routine electronic ordering or charting activities.

Methods

Data sources

We searched the MEDLINE database (1950 to July 2008) using relevant Medical Subject Headings and combinations of text words such as “computer” or “electronic” with terms such as “reminder,” “prompt,” “alert” and “support.” A methodologic filter identified all potential clinical trials. We similarly searched the Embase and CINAHL databases (both to July 2008). We also retrieved all articles that mentioned computers, reminder systems or decision support from the Cochrane Effective Practice and Organisation of Care registry (www.epoc.cochrane.org/welcome), which covers multiple bibliographic databases. Finally, we scanned reference lists of all included studies and review articles. For non-English-language articles, we screened English translations of titles and abstracts, pursuing a full-text translation as needed to determine inclusion or exclusion of the study.

Study selection

Eligible studies evaluated the effects of computer reminders on processes or outcomes of care using a randomized or quasi-randomized controlled design (allocation on the basis of an arbitrary but not truly random process, such as even or odd patient identification numbers). We required that clinicians encounter the reminder during routine performance of the activities of interest, such as prescribing medications or documenting clinical information. Reminders that required clinicians to deviate from their usual activities (e.g., to use a special program without any prompt from the main clinical information system) were excluded because relying on users to remember to call up such resources undermined the core notion of a reminder.

Outcomes

We focused primarily on improvements in processes of care rather than on clinical outcomes, because we wished to determine the degree to which computer reminders achieved their main goal, namely changing provider behaviour. The degree to which such changes ultimately improve patient outcomes will vary depending on the strength of the relation between targeted processes and clinical outcomes. Consequently, if computer reminders do not improve patient outcomes, this may reflect inadequate connections between the targeted processes and outcomes of care rather than a failure to change physician behaviour. Nonetheless, we did capture clinical out-comes, including intermediate outcomes such as control of blood pressure. We excluded outcomes primarily related to resource use, such as length of hospital stay.

We standardized all outcomes so that increases always corresponded to improvements in care. For instance, if a study reported the proportion of patients who received inappropriate medications, we would record the complementary proportion of patients who received appropriate care.

Data extraction

For any given article, two of three investigators (K.S., A.J. or A.M.) independently screened the citation for inclusion. They abstracted the following data from included articles: clinical setting, number of participants, methodologic details, characteristics of the computer reminder, the presence of cointerventions, and the results for eligible outcomes. Discrepancies between the two reviewers were resolved by discussion, involving the third reviewer if necessary to achieve consensus.

Statistical analysis

We anticipated that many studies would assign intervention status at the provider level but would not account for “cluster effects” when analyzing patient-level data.^6,7 Correcting for clustering effects can sometimes be achieved by estimating the intraclass correlation coefficients, especially if the primary studies all report the same outcome and a minority provide relevant data upon which to base imputations.⁸ In this case, however, few studies contained the necessary data, and studies tended to report multiple outcomes, which required an additional assumption that correlations within clusters do not vary across different outcomes.

To preserve the goal of quantifying the effects of computer reminders without resorting to numerous assumptions and conveying a misleading degree of precision, we focused on the median and interquartile range (IQR) for improvements reported by eligible studies. This method, first used in a large review of strategies for implementing guidelines,⁹ has since been applied in Cochrane reviews of interventions to improve practice^10–14 and other systematic reviews of quality improvement interventions.^15–18

Quantifying the median improvement involves two distinct uses of “median.” First, to handle multiple outcomes within individual studies, we identified the median improvement across each study’s eligible outcomes. If a study reported 10 adherence-related outcomes, we calculated the median absolute difference in adherence between the intervention and control groups. With each study represented by its median outcome, we then calculated the median effect and IQR across all included studies. For the purposes of sensitivity analyses, we repeated this calculation using the best outcome from each study.

The median and IQR convey the magnitudes of improvement achieved in the majority of studies. This method avoids skewing by a few outlying studies with highly positive results and 95% confidence intervals inappropriately narrowed by ignoring important clustering effects. It also permits nonparametric analyses of potential associations between study features and effect size in order to examine subgroups of studies with larger or smaller magnitudes of effect. For instance, we looked for associations between magnitude of effect and study size, markers of methodologic quality, features of the study context (e.g., ambulatory v. inpatient) and characteristics of the reminders (e.g., requiring users to enter a response before continuing with their work). We performed all such comparisons using a nonparametric Mann–Whitney rank-sum test.

Results

Of 2036 citations identified, we excluded 1662 at the initial stage of screening and an additional 374 after review of the full-text articles. A total of 28 articles (reporting 32 comparisons) met all of our inclusion criteria (Figure 1).^19–46 The full review has recently been published in The Cochrane Library.⁴⁷

Figure 1.

Results of literature search. *Excluded topics included expert systems (e.g., artificial intelligence or neural network applications) for facilitating diagnosis or for estimating prognosis; decision support not directly related to patient care (e.g., coding medical records); and reminders directed primarily at nonphysicians.

Of the 32 comparisons, 19 were in the United States and 8 occurred in inpatient settings (Table 1, located at the end of the article). Only six comparisons involved a quasi-randomized design, typically allocating intervention status on the basis of even or odd provider identification numbers. Twenty-six comparisons allocated intervention status to providers or provider groups (cluster trials); 12 of these comparisons accounted for clustering effects in the analysis. Seventeen trials reported a power calculation that included a target effect size. Twelve trials reported a target improvement in adherence to processes of care; 10 of these trials specified an absolute increase of at least 10% (Table 1).

Table 1.

Description of 28 studies (32 comparisons) included in a systematic review of the effects of point-of-care computer reminders on physician behaviour

Study	Study design (sample size)	Setting	Intervention	Additional interventions in intervention and control groups	Study groups balanced at baseline	Follow-up complete for ≥ 80% of providers and patients	Effect size used in power calculation
Bates et al.,19 1999	RCT (7090 patients)	Inpatient medical and surgical services at teaching hospital in United States	CPOE-based alert to clinicians regarding potentially redundant orders for laboratory tests	None	Yes	Yes	NR
Christakis et al.,20 2001	Cluster RCT (1339 episodes of care, 38 providers)	Outpatient pediatric teaching clinic in United States	CPOE-based display of evidence for use and duration of antibiotics for otitis media	None	No	Yes for providers; not clear for patients	NR
Dexter et al.,21 2001	Cluster RCT (6371 patients, 8 provider teams)	General medicine inpatient service at teaching hospital in United States	CPOE-based note to providers of inpatients’ eligibility for 4 preventive care measures	None	NR*	Not clear	NR
Eccles et al.,22 2002	Cluster RCT (2335 patients, 60 practices)	Ambulatory general practices in United Kingdom	Decision support for management of outpatients with angina	Distribution of educational materials to providers	Yes	NR	10% absolute improvement in adherence to guideline for care
Eccles et al.,22 2002	Cluster RCT (2363 patients, 60 practices)	Ambulatory general practices, United Kingdom	Decision support for management of outpatients with asthma	Distribution of educational materials to providers	Yes	NR	10% absolute improvement in adherence to guideline for care
Filippi et al.,23 2003	Cluster RCT (15 343 patients, 300 providers)	Ambulatory general practices in Italy	Alert to clinicians to diabetic patients who would benefit from antiplatelet agents to reduce risk of cardiovascular disease	Distribution of educational materials to providers	Yes	Yes	10% absolute improvement in prescription rates
Flottorp et al.,24 2002	Cluster RCT (9887 episodes of care, 57 practices)	Ambulatory general practices in Norway	Display of guidelines for appropriate use of antibiotics and laboratory tests in women with suspected urinary tract infection	Educational materials for providers and patients, educational workshops for providers, financial incentives for providers	Yes	No for providers; NR for patients	15% absolute improvement in recommended processes of care
Flottorp et al.,24 2002	Cluster RCT (16 939 episodes of care, 56 practices)	Ambulatory general practices in Norway	Display of guidelines for appropriate use of antibiotics and laboratory tests for patients with sore throat	Educational materials for providers and patients, educational workshops for providers, financial incentives for providers	Yes	No for providers; NR for patients	15% absolute improvement in recommended processes of care
Frank et al.,25 2004	quasi-RCT (10507 patients)	Urban ambulatory practice (academic status not reported) in Australia	Note to providers of outpatients’ eligibility for various preventive care measures	None	Yes	NR	NR
Hicks et al.,26 2008	Cluster RCT (1834 patients, 12 clinics)	Community-and hospital-based primary care clinics affiliated with teaching hospital in United States	Display of guideline-based suggestions for management of patients with hypertension	None	No†	No	10% absolute improvement in blood pressure control and rates of adherence to guideline
Judge et al.,27 2006	Cluster RCT (3843 episodes of care, 7 wards)	Long-term care facility affiliated with teaching hospital in Canada	Alert to providers to various potential adverse drug events (e.g., severe drug interactions, out-of-recommended-range doses for elderly patients)	None	NR	NR	NR
Kenealy et al.,28 2005	Cluster RCT (2814 patients, 33 practices)	Outpatient general practices in New Zealand	Note to providers suggesting that they screen patients over 50 years of age for diabetes	Distribution of educational materials to providers, educational outreach	Yes	Yes	15% absolute improvement in screening eligible patients for diabetes
Kenealy et al.,28 2005	Cluster RCT (2814 patients, 33 practices)	Outpatient general practices in New Zealand	Note to providers suggesting that they screen patients over 50 years of age for diabetes	None	Yes	Yes	15% absolute improvement in screening eligible patients for diabetes
Kralj et al.,29 2003	Cluster quasi-RCT (2170 episodes of care, 2 practices)	Two community-based outpatient oncology practices in United States	Prompt for providers to order erythropoietin for patients with hemoglobin level < 120 g/dL	None	No†	NR	NR
Krall et al.,30 2004	Cluster RCT (1076 patients, 100 providers)	Ambulatory family and internal medicine practices in United States	Note to providers of patients’ eligibility to receive ASA to reduce risk of cardiovascular disease	None	No	Yes for providers; not clear for patients	NR
Kucher et al.,31 2005	Quasi-RCT (2506 patients)	Major teaching hospital in United States	Alert to clinicians to inpatients at increased risk of venous thromboembolism	None	Yes	Yes	50% reduction in odds of developing venous thromboembolism
McCowan et al.,32 2001	Cluster RCT (477 patients, 46 practices)	Outpatient general practices in United Kingdom	Decision support for management of outpatients with asthma	None	Yes	No	7% absolute reduction in asthma exacerbation rates
Meigs et al.,33 2003	Cluster RCT (598 patients, 2 provider teams)	Internal medicine ambulatory clinic at teaching hospital in United States	Display of recommended target goals of care, last known values of relevant lab tests (e.g., HbA1c, creatinine, lipids)	None	Yes	No	0.5%–1.0% reduction in serum hemoglobin A1c
Overhage et al.,34 1996	Cluster RCT (1622 patients, 24 providers)	Inpatient internal medicine service at academic medical center in United States	Note to providers suggesting orders for various preventive care measures for eligible patients	None	Yes	Yes	NR
Overhage et al.,35 1997	Cluster RCT (2181 patients, 6 services)	Medicine service at teaching hospital in United States	Prompt for providers about “corollary orders”	Drug utilization review program	Yes	NR	NR
Peterson et al.,36 2007	quasi-RCT (2981 patients)	Academic medical centre in United States	Decision support for drug therapy in elderly inpatients (to avoid certain drugs and modify dosing of others)	None	Yes	NR	NR
Rothschild et al.,37 2007	Cluster RCT (350 episodes of care, 453 providers)	Academic medical centre in United States	Display of guidelines regarding indications for transfusion of red blood cells, platelets and frozen plasma	Provider education (printed materials, workshops)	Yes	Yes	NR
Roumie et al.,38 2006	Cluster RCT (871 patients, 116 providers)	8 ambulatory clinics and 2 hospitals in United States	Alert in electronic medical record displaying recent blood pressure value and outlining national recommendations for hypertension treatment and blood pressure goals	Provider education (printed materials delivered via email)	Yes	Yes for providers; no for patients	1.8 increase in odds of achieving target blood pressure
Safran et al.,39 1995	Cluster RCT (349 patients, 2 teams)	Academic primary care clinic in United States	Alert to providers about eligibility of HIV-positive patients for various recommended processes of care	None	Yes	No	NR
Sequist et al.,40 2005	Cluster RCT (6243 patients, 20 clinics)	Outpatient primary clinics (academic and community) in United States	Display of guidelines for recommended management of patients with diabetes and coronary artery disease	Paper reminders to providers	No	NR	10% absolute increase in adherence
Tamblyn et al.,41 2003	Cluster RCT (12 560 encounters, 107 providers)	Primary care practices in Canada	Alert to providers to various potential adverse drug events (e.g., based on drug-drug interactions, and drug-disease or drug-age contraindications)	None	Yes	Yes	30% relative reduction in inappropriate prescriptions (~6% absolute reduction)
Tape et al.,42 1993	Cluster quasi-RCT (1809 patients, 2 clinics)	Internal medicine teaching clinic in United States	Alert to clinicians to deficiencies in preventive care measures for a given patient	Provider education (conferences); paper reminders to providers	NR	NR	50% relative increase in adherence (~10% absolute increase using control adherence rates)
Tierney et al.,43 2003	Cluster RCT (378 encounters, 32 practice sessions)	Academic primary care group practice in United States	Display of guideline-based suggestions for management of heart failure and coronary artery disease	Provider education (printed materials, workshops, outreach visits), use of local opinion leaders	Yes	Yes	1-unit change in standard error of measurement for each subscale of Chronic Heart Failure Questionnaire
Tierney et al.,44 2005	RCT (363 episodes of care)	General medicine practice at teaching hospital in United States	Display of guideline-based suggestions for management of asthma and chronic obstructive pulmonary disease	Provider education (printed materials and workshops)	Yes	Yes	1-unit change in standard error of measurement for health-related quality of life using SF-36
Van Wyk et al.,45 2008	Cluster RCT (3955 patients, 24 clinics)	General practice clinics in the Netherlands	Automatic display of patient-specific guidelines for screening and treatment of dyslipidemia	None	No	Yes	NR
Van Wyk et al.,45 2008	Cluster RCT (3876 patients, 23 clinics)	General practice clinics in the Netherlands	Display on demand of patient-specific guidelines for screening and treatment of dyslipidemia	None	No	Yes	NR
Zanetti et al.,46 2003	Quasi-RCT (449 operations)	Cardiac surgery service at academic medical centre in United States	Written reminder supplemented by auditory cue on computer screen in operating room alerting operating room staff that patient should receive second dose of antibiotic prophylaxis because of prolonged time in surgery	None	Yes	Yes	NR

Note: ASA = acetylsalicylic acid, CPOE = computerized provider order entry, EMR = electronic medical record, NR = not reported, RCT = randomized controlled trial.

^*Analysis adjusted for key demographic variables.

^†Adjustments made in analyses for observed differences.

Figure 2 displays the median improvements in adherence to processes of care for each included study (for details about the results from each study, see Appendix 1, available at www.cmaj.ca/cgi/content/full/cmaj.090578/DC1). Pooling data across studies (Table 2), we found that the median improvement in adherence associated with computer reminders was 4.2% (IQR 0.8%–18.8%). Prescribing behaviours improved by a median of 3.3% (IQR 0.5%–10.6% [21 trials]), adherence to target vaccinations by 3.8% (IQR 0.5%–6.6% [6 trials]) and test-ordering behaviours by 3.8% (IQR 0.4%–16.3% [13 trials]). Table 2 also shows the results obtained when we used the best outcome from each study instead of the median improvement.

Figure 2.

Median absolute improvements in adherence to processes of care between intervention and control groups in each study. Each study is represented by the median and interquartile range for its reported outcomes; studies with single data points reported only one eligible outcome.

Table 2.

Improvements in adherence to processes of care across the 28 studies (32 comparisons) included in the review

Process of care (no. of comparisons)	Median absolute improvement, % (IQR)
	Using median outcome from each study	Using best outcome from each study
All (32)	4.2 (0.8 to 18.8)	5.6 (2.0 to 19.2)
Prescription of medications (21)	3.3 (0.5 to 10.6)	6.2 (3.0 to 28.0)
Prescription of recommended vaccines (6)	3.8 (0.5 to 6.6)	4.8 (0.5 to 7.8)
Ordering of tests (13)	3.8 (0.4 to 16.3)	9.6 (0.6 to 24.0)
Recommended elements of clinical documentation (3)	0.0 (−1.0 to 1.3)	2.0 (2.0 to 4.0)
Other (7)	1.0 (0.8 to 8.5)	4.0 (0.8 to 8.5)

Note: IQR = interquartile range.

Across eight comparisons that reported dichotomous clinical outcomes (e.g., achievement of target treatment goals), patients in the intervention groups experienced a median absolute improvement of 2.5% (IQR 1.3%–4.2%). For blood pressure control, the single most commonly reported outcome, patients in the intervention groups experienced a median reduction in systolic blood pressure of 1.0 mm Hg (IQR 2.3 mm Hg reduction to 2.0 mm Hg increase) and a median reduction in diastolic blood pressure of 0.2 mm Hg (IQR 0.8 mm Hg reduction to 1.0 mm Hg increase).

Study features and effect size

We found no significant correlation between effect size and the following study features: publication year, country (United States v. other), study design (randomized v. quasi-randomized) or sample size (whether calculated on the basis of patients or providers) (Figure 3). We considered that studies with high adherence rates in control groups (a marker for baseline adherence) might achieve smaller improvements in care, because they had smaller opportunities for improvement. Surprisingly, studies with control-group adherence rates that were higher than the median across all studies showed larger effect sizes (Figure 3). When we analyzed the potential impact of baseline adherence in various other ways (e.g., focusing on the highest and lowest quartiles of baseline adherence), we found no evidence that small improvements reflected high baseline quality of care.

Figure 3.

Median effects for adherence to processes of care by study feature. *Kruskall–Wallis test; all other p values reflect Mann–Whitney test. †Quasi-RCT refers to randomized controlled trials in which intervention status was assigned on the basis of an arbitrary but not truly random process, such as even or odd patient (or provider) identification numbers. ‡The total number of comparisons for the analysis of sample size is 31 because one study did not report the number of patients. §Studies classified as having no cointervention were those in which a computer reminder alone was compared with usual care; studies classified as having co-interventions were those in which the intervention group received a computer reminder plus one or more other quality improvement interventions, while the control group received those same quality improvement interventions but no computer reminder.

We observed a trend toward larger improvements with inpatient interventions than with outpatient interventions (median 8.7% [IQR 2.7%–22.7%] v. 3.0% [IQR 0.6%–11.5%]; p = 0.34). All inpatient interventions occurred at two institutions that had well-developed, “homegrown” computerized systems for order entry by providers. Moreover, the recipients of computer reminders from these institutions consisted primarily of physician trainees.

Our grouping of studies on the basis of track records in clinical informatics did not result in significant differences, except that the studies from Brigham and Women’s Hospital in Boston, USA, reported a median improvement of 16.8% (IQR 8.7%–26.0%),^{26,31,37,40,46} compared with 3.0% (IQR 0.5%–11.5%) for studies from the other institutions (p = 0.04).

Features of computer reminders and effect size

We analyzed a number of reminder characteristics to look for associations with effect size (Figure 4). Only the requirement for providers to enter a response to the reminder showed a trend toward larger improvements (median 12.9% [IQR 2.7%–22.7%] v. 2.7% [IQR 0.6%–5.6%] for no response required; p = 0.09). No trends toward larger effect sizes existed based on the type of targeted problem (underuse v. overuse of a targeted process of care), inclusion of patient-specific information, provision of an explanation for the alert, inclusion of a specific recommendation with the alert, development of the reminder by the study authors, or the type of system used to deliver the reminder (CPOE [computerized provider order entry] v. electronic medical records).

Figure 4.

Median effects for adherence to processes of care by reminder feature. *Underuse = targeting improvements to increase the percentage of patients who receive targeted process of care (e.g., increasing the percentage of patients receiving the influenza vaccine); overuse = targeting improvements to reduce the percentage of patients receiving inappropriate care (e.g., reducing the percentage of patients who receive antibiotics for viral upper respiratory tract infections). †Reminders with no patient-specific information were those triggered on the basis of demographic characteristics (e.g., age) or the intent to order a medication or investigation irrespective of any features of the patient involved or patient-specific laboratory results. The sample size is reduced because of the inability to accurately assess the presence or absence of the feature. ‡Active delivery refers to reminders that appeared automatically when triggering conditions were met, as opposed to passive reminders, where, for instance, users might be presented with the option to click on a link to receive decision support related to their current task. §CPOE = computerized order entry system; reminder systems without CPOE were typically electronic medical record systems.

Reminders that were “pushed” onto users (i.e., users automatically received the reminder) did not achieve larger effects than reminders that required users to perform some action to receive them (i.e., users had to “pull” the reminders); only 4 of the 32 comparisons involved “pull” reminders. A three-armed cluster randomized controlled trial of reminders for screening and treatment of hyperlipidemia⁴⁵ directly compared these two modes of delivering reminders. Patients cared for at practices randomly assigned to deliver automatic alerts were more likely to undergo testing for hyperlidemia and receive treatment than were patients at clinics where reminders were delivered to clinicians only “on demand.”

Sensitivity analyses

We re-analyzed the potential predictors of effect size (study features and characteristics of reminders) using a variety of choices for the representative outcome from each study, including the outcome with the middle value (rather than a calculated median) and the best outcome (the outcome associated with the largest improvement in adherence to the process). None of these analyses substantially altered the main findings.

Interpretation

Across the 32 comparisons, computer reminders achieved small to modest improvements in care, with a median improvement of 4.2% (IQR 0.8%–18.8%). Even using the best out-come from each trial, the median improvement was only 5.6% (IQR 2.0%–19.2%). These changes fall below the thresholds for clinically significant improvements specified in most trials, and they are certainly smaller than the improvements generally expected from computerized order entry and electronic medical record systems. Interestingly, these improvements are also no larger than those observed for paper-based reminders.^5,48

With the upper quartile of reported improvements beginning at an almost 20% increase in adherence to processes of care, some studies in our review clearly did show larger effects. However, we were unable to identify any study characteristic or reminder feature that predicted larger effect sizes, except for a statistically significant increase in magnitude of effect seen in studies involving a well-developed, homegrown computer order entry system at Brigham and Women’s Hospital.^{26,31,37,40,46} A trend toward larger effects was also seen for reminders that required users to enter a response in order to proceed; however, this finding may have been confounded by the uneven distribution of studies from Brigham and Women’s Hospital. Thus, we do not know if the success of computer reminders at this institution reflects the design of reminders requiring user responses, other features of the computer system or perhaps institutional culture.

Included studies often provided limited descriptions of key features of the reminders and the systems through which they were delivered. We attempted to overcome this problem by abstracting basic features, such as whether user responses were required and whether the reminder displayed a justification for its content. But heterogeneity within even these apparently straightforward categories could mask important differences in effect. Important differences in effect may also reflect characteristics that we found difficult to operationalize (e.g., the “complexity” of the reminder) or that were inadequately reported. This problem of limited descriptive detail of complex interventions and the resulting potential for heterogeneity among included interventions in systematic reviews has been consistently encountered in the quality-improvement literature.^49,50

Conventional meta-analyses estimate mean effects and 95% confidence intervals by calculating weighted averages across study results. The individual weights derive from study precision such that larger studies contribute greater weight to the meta-analytic result. However, more than half of the studies included in our review reported spuriously high precision, and most of the studies did not report the data required to adjust for this problem. For example, of the 26 clustered trials, only 9 provided a single value for the intra-cluster correlation coefficient, and only 3 reported values for all outcomes. Because we could not accurately weight studies based on precision, we focused on the median and interquartile range for study effects, a method that has found increasing application in systematic reviews of interventions for quality improvement.^{9,13–15,17,18,51}

The main potential drawback of this method is that we assigned equal weight to all of the studies. However, for our results to have substantially misrepresented the true impacts of computer reminders, the minority of studies with large magnitudes of effect would also have to be the larger studies (and thus deserving of greater weight in a meta-analysis). Not only is this unlikely in general, we specifically showed that study size bore no relation to effect size, using various definitions of study and effect size.

Conclusion

Computer reminders typically increased adherence to target processes of care by amounts below thresholds for clinically significant improvements. A minority of studies showed more substantial improvements, consistent with the expectations of those who advocate widespread adoption of computerized order entry and electronic medical record systems. However, until further research identifies study design and reminder features that reliably predict clinically worthwhile improvements in care, implementing these expensive technologies will constitute an expensive exercise in trial and error.