Changing Clinical Practice Through Patient Specific Reminders Available at the Time of the Clinical Encounter: Systematic Review and Meta-Analysis

Abstract

OBJECTIVE

To synthesise current evidence for the influence on clinical behaviour of patient-specific electronically generated reminders available at the time of the clinical encounter.

DATA SOURCES

PubMed, Cochrane library of systematic reviews; Science Citation Index Expanded; Social Sciences Citation Index; ASSIA; EMBASE; CINAHL; DARE; HMIC were searched for relevant articles.

STUDY ELIGIBILITY CRITERIA, PARTICIPANTS AND INTERVENTIONS

We included controlled trials of reminder interventions if the intervention was: directed at clinician behaviour; available during the clinical encounter; computer generated (including computer generated paper-based reminders); and generated by patient-specific (rather than condition specific or drug specific) data.

STUDY APPRAISAL AND SYNTHESIS METHODS

Systematic review and meta-analysis of controlled trials published since 1970. A random effects model was used to derive a pooled odds ratio for adherence to recommended care or achievement of target outcome. Subgroups were examined based on area of care and study design. Odds ratios were derived for each sub-group. We examined the designs, settings and other features of reminders looking for factors associated with a consistent effect.

RESULTS

Altogether, 42 papers met the inclusion criteria. The studies were of variable quality and some were affected by unit of analysis errors due to a failure to account for clustering. An overall odds ratio of 1.79 [95% confidence interval 1.56, 2.05] in favour of reminders was derived. Heterogeneity was high and factors predicting effect size were difficult to identify.

LIMITATIONS

Methodological diversity added to statistical heterogeneity as an obstacle to meta-analysis. The quality of included studies was variable and in some reports procedural details were lacking.

CONCLUSIONS AND IMPLICATIONS OF KEY FINDINGS

The analysis suggests a moderate effect of electronically generated, individually tailored reminders on clinician behaviour during the clinical encounter. Future research should concentrate on identifying the features of reminder interventions most likely to result in the target behaviour.

BACKGROUND

Computer generated reminder systems are commonly used to support routine health care. They utilise electronic data to identify clinical errors and opportunities for screening, preventive interventions, improved prescribing, and both diagnostic and monitoring tests. Previous studies have found that the response of clinicians to such reminders is variable, and a number of reviews have described existing tools, where possible measured their impact, and in some cases attempted to identify factors influencing effect size.^1–11 Reminder systems are diverse in their design. Some are used to support specific clinical areas of care (e.g. diabetes), presenting current recommendations or evidence, and do not require patient specific data. Others are triggered simply by an attempt to prescribe a specific drug therapy, for instance reminding the prescriber of lithium that blood monitoring is required. Shojania et al. studied the impact of ‘on-screen’ reminders as a Cochrane systematic review,¹⁰ and excluded computer generated paper-based reminders and email alerts occurring outside clinical encounters. They hypothesised that this approach would identify a more consistent effect, in contrast to the variable results reported in previous reviews. This group derived a median absolute change in adherence of 4.2% with IQR 0.8-18.8%, suggesting significant variation in response, and factors predicting effect size were difficult to identify.

A subset of reminder system draws on patient specific data in the electronic record and is therefore tailored to the individual. For this review we were interested specifically in individually tailored reminders and in the impact of these tools on decision making. We concurred with Shojania et al. over the importance of the ‘point of care’ setting, but hypothesised that tailored reminders might provide a more consistently positive effect. Although the reminders that we studied were exclusively clinician directed, individual tailoring might conceivably carry greater impact, as the resulting behaviour often requires patient involvement for completion (e.g. uptake of screening).

METHODS

We chose to study both on-screen and paper-based reminders provided that they were generated by electronic information specific to the individual in a health record and available at the clinical encounter. In contrast to the Shojania review, we chose the odds ratio technique to estimate effect size as we were interested in the relative likelihood of achieving the desired outcome in the presence of a reminder rather than the absolute change in outcome. This approach may be more appropriate where baseline activity varies significantly between different trial settings, as relative benefit tends to be more stable across risk groups than absolute benefit.¹² We were also interested in detecting any variation in response according to clinical area and in changes in responsiveness over the past 40 years, during which the use of electronic records has become widespread. A review protocol was written but not published.

Literature Search

We systematically examined the literature from 1970 to February 2011 describing controlled trials of computer generated reminder interventions that draw on patient specific information and are available to clinicians during clinical encounters. We searched the following databases for relevant articles: PubMed, Cochrane library of systematic reviews; Science Citation Index Expanded; Social Sciences Citation Index; ASSIA; EMBASE; CINAHL; DARE; HMIC. The following search strategy (or adaptations of it) was used in each database:

• Reminder systems [MeSH] AND (Health OR Medic* OR Clinical) AND (Computer*

• [text word] OR Electronic* [text word])

We looked at reference lists of retrieved articles and past systematic reviews of similar interventions. We included non-randomised controlled trials, provided data collection from both arms was contemporaneous. We did not consider ‘before/after’ studies to be sufficiently valuable, given the potential for secular trends (including health policy changes) to confound the influence of the effect, and such studies were excluded.

Selection of Articles

The inclusion criteria were applied to each paper by two reviewers, with disagreements resolved by the third reviewer.

Extraction of Data

For each identified paper, two reviewers assessed methodological quality and extracted the outcome data using a formatted extraction sheet. Where necessary, study authors were contacted for clarification. We assessed risk of bias according to inadequate random sequence generation (at study level); and incomplete outcome data, selective reporting, and unit of analysis error (at the outcome level). The latter was used as a basis for a correction for clustering in the meta-analysis.

Outcome Measures

Changes in process or clinical outcome included rates of screening, vaccination, diagnostic tests, blood pressure measurement, blood pressure control, rate of venous thrombo-embolism, and measures of prescribing quality.

Analysis

Odds ratios were derived for all binary outcomes where available. We used a random effects model with the Mantel–Haenszel method in RevMan version 5.2 to combine the data. Where multiple outcomes were reported, we derived a pooled outcome measure for each study. Heterogeneity was measured using the Tau² and I² statistics. Tau² is a measure of between study variance appropriate for a random effects meta-analysis.¹² I² gives the proportion of the variability that is attributable to heterogeneity rather than chance.¹²

Trials of reminder interventions may be affected by ‘unit of analysis errors,¹⁰ through failure to correct for clustering. For instance, a trial may use as its outcome the proportion of patients achieving a clinical target at the end of the study, but it was the clinicians, clinical teams or clinics (not the patients individually) that had been randomised to use or not to use the reminders. If uncorrected, the precision of effect size measurement will be over-estimated by this error.

We tested the effect of introducing a correction factor where clustering had not been accounted for, using a recommended technique.¹² An assumed intra-class correlation co-efficient of 0.03 was identified as appropriate from a published source.¹³ This was used to derive a design effect estimate for each study based on its mean cluster size, and the numerator and denominator values for each trial arm were divided by this factor. The pooled odds ratio was then re-estimated to account for clustering. Recognising the risk of applying a single ICC to many studies, we undertook an analysis to measure the sensitivity of the pooled odds ratio and its confidence interval to a range of assumed ICC values.

We also examined subgroups of reminder intervention according to pre-specified clinical areas and distinguished articles according to whether the trial was ‘explanatory’ or ‘pragmatic’ in design. ‘Explanatory’ studies were those in which the denominator was the reminder opportunity, i.e. the clinical encounter in which the reminder was triggered. The outcome was the proportion of all examples in which a clinician actually encountering a patient and presented with a reminder, responded to it. ‘Pragmatic’ studies used as their outcome the proportion of a population of patients whose clinicians were potentially exposed to a reminder intervention in whom the recommended care occurred. Some of the outcome denominator population might not have presented to the clinician during the study period, whilst others might have presented a number of times. Whilst some studies were difficult to categorise, we considered these groups to represent methodologically distinct designs worthy of separate analysis.

Finally, we sub-grouped studies according to the decade of publication, looking for a secular trend in the responsiveness of clinicians to such reminders, and assessed risk of publication bias using a funnel plot.

Results

Selection of Articles

We initially identified 683 articles following removal of duplicates. Abstracts were examined to remove obviously irrelevant papers, leaving 234 for full text examination. Of these, 192 articles were excluded by at least two reviewers (Fig. 1). This left 42 trial reports in the final group.^14–55 Forty-one of these used binary outcomes. The other⁴⁶ used length of hospital stay. Two papers reported clinical outcomes (control of blood pressure²⁴ and rate of venous thrombo-embolism)²⁹ and all the rest involved process outcomes. One paper¹⁹ reported three different intervention arms and one control arm. This study was entered as three separate comparisons and the numbers in the control arm were divided by three to avoid over-weighting. A further two papers^47,53 reported two equally important forms of reminder that were both included as separate comparisons. Where possible we aggregated separately reported subgroups of outcome within the same trial to provide an estimate of overall effect. For instance, a single reminder intervention might promote screening tests, clinical measurement and immunisations, with each outcome reported separately. One paper²¹ reported multiple outcomes with no primary outcome and was not included in the meta-analysis as it was not possible using this method to aggregate the outcome data from this paper reliably. There were therefore 44 comparisons using a binary outcome available for the meta-analysis. There were examples in which the desired effect of the reminder was to reduce rather than increase the outcome measure^{15,29,32,43,47,48,50}. In such cases we used the method described by Shojania et al.¹⁰ to impute a corrected numerator in order that the effect was measured in the same direction as for the other studies. There was only one example⁴⁹ of a trial that was controlled but not randomised.

Figure 1.

PRISMA flow diagram for systematic review.

Meta-Analysis

For the 44 binary outcome comparisons an overall odds ratio of 1.79 [95% confidence interval 1.56, 2.05] was derived in favour of the reminders. Heterogeneity was high presumably due to clinical and methodological diversity, with an overall Tau² = 0.18, Chi² = 1530.40, p < 0.00001, I² = 97%. The one study using a continuous outcome⁴⁶ reported a non-significant difference in length of hospital stay. The study that was excluded on the basis of multiple outcomes reported no effect of the reminder system on clinical care.²¹ For our included studies, 32 out of 44 comparisons showed a significant positive effect and 11 showed no significant effect. One study⁴⁸ appeared to show a significant negative effect but this was dependent on the definitions of intervention and control in a study comparing two different reminder systems.

To reduce clinical diversity we attempted subgroup analyses based on area of care (although there was much overlap). There was evidence (of borderline significance, Chi² = 11.47, p = 0.04) of subgroup differences in effect size. Odds ratios ranged from 1.24 [95%CI 1.01–1.52] for condition specific but multiple reminders to 4.69 [95%CI 1.25–17.53] for vaccination reminders (Fig. 2). The condition specific but multiple reminders subgroup had a relatively low Tau² score of 0.03 with Chi² = 7.84, p = 0.05. The odds ratios in favour of the intervention for the explanatory and pragmatic sub-groups were 1.90 and 1.71, respectively, and there was no significant improvement in heterogeneity scores.

Figure 2.

Forest plot of all studies (44 comparisons) reporting binary outcomes, grouped by area of care. These are based on raw extracted data prior to our adjustment for clustering.

There was no evidence that odds ratios were different between the 1980s, 1990s and 2000s, and only one study from the 1970s was included.

Characteristics of the reminder interventions were examined to look for factors likely to influence the effect size, including clinical priority, remunerative factors, and ease of use. These are explored in the Discussion section below (Table 1).

Table 1.

Characteristics of Included Studies

Study	Country	Setting	Participants (in addition to the attending clinicians)	Intervention: Paper or Screen reminder?	Comparators	Outcomes used in our analysis
Bates 1999	USA	Tertiary care hospital inpatients	Hospitalised patients undergoing laboratory investigations	Screen	No reminders	Proportion of all reminders resulting in the target behaviour
Burack 1996	USA	Large Health Maintenance Organisation in Detroit	Women due screening mammography	Paper	No reminders	Mammography rates
Burack 1998	USA	Large Health Maintenance Organisation in Detroit	Women aged 18-40 years due a Pap smear	Paper	No reminders	Completion of Pap smear during the study year
Chambers 1991	USA	University-based family practice centre	People over 65 years eligible for influenza vaccination	Paper	No reminders	Rate of influenza vaccine during the study period
Dexter 1998	USA	Academic primary care practice affiliated to an urban teaching hospital	People over 75 years of age or over 50 years with serious health condition	Paper	No reminders	Discussion over advanced directives
Dexter 2001	USA	Inpatient wards of teaching hospital	Hospitalised patients requiring preventive interventions (pneumonia or influenza vaccination, subcutaneous heparin, or aspirin at discharge)	Screen	No reminders	Completion of preventive therapies
Eccles 2002	UK	General practice	Patients over 18 years with asthma or angina	Screen	No reminders	A range of clinical and process outcomes. Unable to extract data as no primary outcome identified so not included in the quantitative meta-analysis
Fillipi 2003	Italy	General practice	Patients over 30 years with diabetes and at least one risk factor for cardiovascular disease and their attending clinicians	Screen	No reminders	Proportion of eligible patients prescribed aspirin
Frank 2004	Australia	General practice	People see in general practice with a range of preventive health needs	Screen	No reminders	Proportion of alerts producing the recommended care
Hicks 2007	USA	Primary care practices	Adults with hypertension	Screen	No reminders	Proportion of patients with blood pressure in target
Holt 2010	UK	Primary care	Individuals registered with general practice	Screen	No reminders	Proportion of patients identifiably at risk of cardiovascular disease
Judge 2006	Canada	Academically affiliated long term care facility	Residents of a long term care facility requiring prescribed medication	Screen	No reminders	Proportion of reminders producing the target behaviour
Kenealy 2005	New Zealand	Primary care practices	Patients over 50 years with no diabetes diagnosis and no blood glucose level recorded in the past 3 years	Screen	No reminders	Proportion of eligible patients who were screened for diabetes
Kralj 2003	USA	Community oncology practices	Patients with cancer and a haemoglobin level <12 g/dL having not already received erythropoetin	Screen	No reminders	Proportion of eligible patients receiving erythropoetin
Krall 2004	USA	Kaiser Permanante Northwest	Patients requiring low dose aspirin	Screen	No reminders	Proportion of patients still eligible for an aspirin reminder
Kucher 2005	USA	Inpatients on medical and surgical wards	Hospital inpatients at risk of venous thrombo-embolism	Screen	No reminders	Proportion of patients diagnosed with venous thromboembolism
Litzelman 1993	USA	Academic primary care internal medicine practice	Patients requiring screening with faecal occult blood, mammography or Pap smear	Screen	Reminder present but requirement to indicate response	Overall compliance with reminder recommendation
Lo 2009	USA	Academic teaching hospitals, community hospitals and outpatient clinics	Patients commencing new medication requiring baseline laboratory tests	Screen	No reminders	Proportion of relevant new medications in which appropriate laboratory test was requested
Matheny 2008	USA	Academic teaching hospitals, community hospitals and outpatient clinics	Patients requiring laboratory monitoring tests related to medication prescribing	Screen	No reminders	Proportion of reminders followed by the ordering of a laboratory test within 14 days
McCowan 2001	UK	General practice	Patients with asthma randomly selected from practices' asthma registers	Screen	No reminders	Patient initiated primary care consultations
McDonald 1976	USA	Hospital diabetes clinic	Patients with diabetes in need of blood tests or medication adjustment	Paper	No reminders	Proportion of reminders resulting in the target behaviour
McDonald 1980	USA	Hospital General Medicine clinic	Patients requiring tests, changes to medication, or recording of information	Paper	No reminders	Proportion of reminders resulting in the target behaviour
McDowell 1989a	Canada	Family Medicine Centre at civic hospital	People over 18 years with no blood pressure recorded in the past year	Paper	No reminders	Proportion of eligible patients with a blood pressure check during the trial year
McDowell 1989b	Canada	Family Medicine Centre at civic hospital	Women aged 18-35 years overdue a Pap smear	Paper	No reminders	Proportion of eligible people that had been screened
Murray 2004	USA	Academic primary care Internal medicine practice	Patients with uncomplicated hypertension	Screen	No reminders	Proportion of patients complying with treatment suggestions
Overhage 1996	USA	Inpatient general medical ward	Inpatients requiring preventive interventions	Both	No reminders	Proportion of alerts producing the recommended care
Overhage 1997	USA	Inpatient general medical ward	Patients requiring investigations due to prescribed medication	Screen	No reminders	Proportion of alerts producing the recommended care
Rosser 1991	Canada	Family Medicine Centre at a civic hospital	People requiring preventive interventions	Paper	No reminders	Proportion of alerts producing the recommended care
Rosser 1992	Canada	Family Medicine Centre at a civic hospital	People requiring tetanus vaccination	Paper	No reminders	Proportion of patients with record of tetanus vaccination at the end of the study period
Rossi 1997	USA	Primary care providers	People with hypertension taking calcium channel blockers	Paper	No reminders	Proportion of patients taking calcium channel blocker changed to alternative drug
Rothschild 2007	USA	Academic medical centre with emergency department and inpatient beds	Patients prescribed transfusion products	Screen	No reminders	Proportion of orders adhering to guidelines
Safran 1995	USA	Hospital-basedoutpatient clinic	People with HIV infection in primary care	Screen	No reminders	Proportion of users taking the recommended action or giving a reason why not
Sequist 2005	USA	Network of outpatient clinics, community and teaching hospitals	Patients with diabetes or coronary heart disease	Screen	No diabetes or CHD reminders, but other non-specific reminders continued to operate	Proportion of alerts producing the recommended care
Shea 1995	USA	Large urban hospital	Patients admitted to a hospital	Screen	No reminders	Length of hospital stay
Tamblyn 2003	Canada	Primary care	People attending primary care and receiving prescribed medication	Screen	No reminders	a) Proportion of alerts resulting in an appropriate prescription, and b) proportion of alerts followed by discontinuation of inappropriate medication
Tamblyn 2008	Canada	Primary care	People receiving prescribed medication and at risk of dosing errors, drug interactions, drug allergy, or other prescribing problems	Screen	Reminders available by active request by physician to initiate it via the screen	Proportion of alerts seen that were acted upon
Tape 1993	USA	Academic Internal medicine clinic	Patients requiring health maintenance procedures	Initially paper, then screen	No reminders and paper-based record	Adherence to reminder recommendation
Tierney 1987	USA	Academic primary care general medicine clinic affiliated to urban hospital	Primary care patients requiring diagnostic tests	Screen	No reminders	Number of tests ordered per visit
Tierney 2003	USA	Academic primary care group practice	Outpatients with heart failure and/or ischaemic heart disease	Both	No reminders	Adherence to reminder recommendation
Tierney 2005	USA	Inner city academic General Medicine clinic	Patients with asthma or COPD	Both	No reminders	Adherence to reminder recommendation
van Wyk 2008	Netherlands	General practice	Patients requiring screening for lipid abnormalities, and those requiring treatment	Screen	No reminders	Rates of screening and treatment for dyslipidaemia
White 1984	USA	Inpatients in a teaching hospital	Patients receiving digoxin and at risk of toxicity	Paper	No reminders	Proportion of 'alert days' resulting in physician action

Methodological Quality

For many studies procedural details such as randomisation techniques were unreported. Trials of reminder interventions sometimes randomise at the level of the clinician or clinical team, but analyse using patient level outcome data. Some form of unit of analysis issue potentially affected 28 studies^{14,18–22,24–28,31,32,35,37–39,41–45,47–49,51–53} and 32 comparisons. In sixteen cases^{14,19–21,24,26,31,37–39,45,47,48,51–53} this was discussed and corrective action taken to adjust confidence intervals or p values appropriately. However, the raw data that we extracted had not undergone this correction and we therefore applied our own adjustment as described above. For the 32 comparisons affected, the initial odds ratio in favour of reminders was 1.87 [95% CI 1.54, 2.28]. Following our adjustments the odds ratio for these studies had changed to 1.90 and the confidence interval had widened slightly to [1.54, 2.33]. There was no change in the overall pooled odds ratio of all studies combined (1.79, [1.58, 2.02]). The results of our adjustment for clustering are given in Fig. 3. Table 2 gives the results based on a range of assumed ICC values, suggesting that the analysis was not sensitive to this assumed value over a 100 fold scale. Figure 4 shows risk of bias tables (a) for each study (b) aggregated.

Figure 3.

Forest plot of all studies reporting binary outcomes, grouped according to presence or absence of a unit of analysis (UoA) issue, with correction to account for clustering in the first group. In the published source papers a similar correction had been applied to some but not all of these by the authors.

Table 2.

Pooled Odds Ratios for the Subgroup of Comparisons Requiring Correction for Clustering, Using a Range of Assumed ICC Values

Assumed intra-class correlation coefficient (ICC) value	Pooled odds ratio (OR) in the meta-analysis corrected for clustering	Confidence interval for the OR
0.003	1.87	[1.55, 2.27]
0.03	1.90	[1.54, 2.33]
0.3	1.95	[1.50, 2.53]

The value of 0.03 was identified as the most relevant for this type of study and is used in the main analysis

Figure 4.

Risk of bias tables (a) for each study (b) aggregated.

Publication Bias

We derived a funnel plot which was broadly symmetrical with no evidence of substantial publication bias.

DISCUSSION

Summary of Findings

The majority of interventions in our review produced significant changes in measured outcomes, but there were numerous examples of no effect and it appears that reminders are often ignored. There is no evidence that such tools were more effective in the 2000s than in the 1980s or 1990s, and our effect size estimate is very similar to a previously published value from 1996⁷, albeit using different inclusion criteria.

Features Influencing Effect Size

Characteristics of individual studies are given in Table 1. We examined these to see whether specific features associated with a more consistent effect could be identified. Kawamoto et al.⁴ have reported four features believed to be relevant in clinical decision support systems: automatic provision of decision support as part of clinical workflow; provisions of recommendations rather than just assessments; provision of decision support at the time and location of decision making; and computer-based decision support. Whilst all our trials involved computer generated reminders, some of these were paper-based. We looked at whether this feature influenced success, and also considered a number of other potentially relevant issues suggested by other investigators.^56–62 These included clinical priority and relevance, cost-effectiveness considerations, accessibility, intrusiveness, and the time required to respond.

Computer generated but paper-based reminders were involved in 12 of our 44 comparisons.^{16–19,33–36,40–42,54} The remainder were displayed either exclusively on a computer screen or in both formats. There was no significant difference in the odds ratios obtained between these subgroups.

It is difficult to judge which issues physicians are likely to consider most important clinically. Vaccination reminders might in most situations be considered less urgent than immediate prescribing safety or laboratory monitoring issues, but in fact were associated with a stronger effect, albeit based on a small number of studies. However the one trial reporting a significantly positive result for a clinical (rather than a process) outcome involved the prevention of venous thrombo-embolism in hospitalised patients identified and flagged as ‘at risk’ of this serious condition.²⁹

None of our included trials specifically reported ‘payment by result’ as a direct consequence of responding to a reminder, but this may have been an unreported factor in settings where remuneration is partly based on quality or efficiency of care. In some cases the electronic record itself had been established at least partly for the purpose of gathering billing information. Shea et al.⁴⁶ mention financial pressures relevant to their length of hospital stay outcome. Others mentioned the health economic benefits of cost-effective monitoring and prescribing, promoted by reminders, and Tierney 1987⁵⁰ included charges per visit as a secondary outcome.

It is difficult to interpret from a published study exactly how much time clinicians had available and how onerous the recommended action might have been. In a large study based in Canada, the reminder requiring activation by the clinician was in fact more effective than the one appearing spontaneously.⁴⁸ Van Wyk et al. arrived at the opposite conclusion in their trial.⁵³ They included an ‘on-demand’ arm that required the user to actively seek the recommendation by accessing an overview screen in the patient’s record. In this arm responsiveness was significantly lower than in the ‘alerting’ arm which required no positive action. Eccles et al. reported a similar finding that highlights the difficulties in successfully embedding the reminder into the workflow.²¹ The negative results in this study were attributed by the authors to low usage of the system, despite its integration into the clinical software.

Other interesting phenomena were reported in the studies we examined. Chambers et al.¹⁸ included an arm in which the reminders only appeared ‘sometimes’ (in addition to the ‘always reminded’ arm whose data were used in our meta-analysis). The clinicians reminded ‘sometimes’ had a lower adherence than those reminded ‘never’ (i.e. controls), suggesting that they had become dependent on the alerts to remember to arrange influenza immunisation for eligible patients.

Strengths and Limitations

Our study is limited partly due to heterogeneity of effect sizes and by difficulties in synthesising data from diverse trial designs. The effect under investigation is likely to depend on the health care setting, the detailed design of the reminder, and the priorities of both clinician and patient. Attempts to substantially reduce heterogeneity through subgroup analyses were unsuccessful but our measurement of effect size is nevertheless meaningful. We focussed specifically on ‘reminder’ interventions and may have missed some studies of more generalised decision support systems in which reminders were a minor element. A further limitation is the lack of detail given in some trial reports over how the system actually operated in practice and what was required of the user in practical terms.

Our review provides data specific to tailored reminders available during clinical encounters, and is the only recently published example of a meta-analysis using a relative (odds ratio) technique rather than an absolute change method in this area of care. This technique provides a more consistent measure of effect across diverse studies, but is more sensitive to outliers than the median absolute benefit technique.¹¹ Trial reports accounted for clustering effects in some cases, risking unit of analysis errors in others. We applied our own correction for clustering in the analysis of the raw trial data to estimate the effect of clustering on our pooled odds ratio.

Future Research

Most individual reminder trials are designed to find out whether a system works rather than why it works. Mayo-Smith and Agrawal used a mixed method to investigate this area, conducting an observational study of reminder completion rates followed by a questionnaire survey of users.⁶³ They also reviewed literature reporting this issue specifically, and included studies using qualitative methods. They reported a number of possible features of reminders, settings and users that appear to facilitate or obstruct response, and such clues might become the basis for a more extensive programme of investigation.

CONCLUSIONS

Individually tailored, computer generated reminders generally produce positive but modest effects on clinicians’ behaviour. Such interventions are inexpensive, widely available, and offer the potential both to improve clinical care and to impact health outcomes. There is now an extensive literature demonstrating these benefits. The specific features of such tools and the particular settings that determine their effect are still unclear but should become the focus of future research in this area.