Abstract
Background
Health professionals sometimes do not use the best evidence to treat their patients, in part due to unconscious acts of omission and information overload. Reminders help clinicians overcome these problems by prompting them to recall information that they already know, or by presenting information in a different and more accessible format. Manually‐generated reminders delivered on paper are defined as information given to the health professional with each patient or encounter, provided on paper, in which no computer is involved in the production or delivery of the reminder. Manually‐generated reminders delivered on paper are relatively cheap interventions, and are especially relevant in settings where electronic clinical records are not widely available and affordable. This review is one of three Cochrane Reviews focused on the effectiveness of reminders in health care.
Objectives
1. To determine the effectiveness of manually‐generated reminders delivered on paper in changing professional practice and improving patient outcomes. 2. To explore whether a number of potential effect modifiers influence the effectiveness of manually‐generated reminders delivered on paper.
Search methods
We searched CENTRAL, MEDLINE, Embase, CINAHL and two trials registers on 5 December 2018. We searched grey literature, screened individual journals, conference proceedings and relevant systematic reviews, and reviewed reference lists and cited references of included studies.
Selection criteria
We included randomised and non‐randomised trials assessing the impact of manually‐generated reminders delivered on paper as a single intervention (compared with usual care) or added to one or more co‐interventions as a multicomponent intervention (compared with the co‐intervention(s) without the reminder component) on professional practice or patients' outcomes. We also included randomised and non‐randomised trials comparing manually‐generated reminders with other quality improvement (QI) interventions.
Data collection and analysis
Two review authors screened studies for eligibility and abstracted data independently. We extracted the primary outcome as defined by the authors or calculated the median effect size across all reported outcomes in each study. We then calculated the median percentage improvement and interquartile range across the included studies that reported improvement related outcomes, and assessed the certainty of the evidence using the GRADE approach.
Main results
We identified 63 studies (41 cluster‐randomised trials, 18 individual randomised trials, and four non‐randomised trials) that met all inclusion criteria. Fifty‐seven studies reported usable data (64 comparisons). The studies were mainly located in North America (42 studies) and the UK (eight studies). Fifty‐four studies took place in outpatient/ambulatory settings. The clinical areas most commonly targeted were cardiovascular disease management (11 studies), cancer screening (10 studies) and preventive care (10 studies), and most studies had physicians as their target population (57 studies). General management of a clinical condition (17 studies), test‐ordering (14 studies) and prescription (10 studies) were the behaviours more commonly targeted by the intervention.
Forty‐eight studies reported changes in professional practice measured as dichotomous process adherence outcomes (e.g. compliance with guidelines recommendations), 16 reported those changes measured as continuous process‐of‐care outcomes (e.g. number of days with catheters), eight reported dichotomous patient outcomes (e.g. mortality rates) and five reported continuous patient outcomes (e.g. mean systolic blood pressure).
Manually‐generated reminders delivered on paper probably improve professional practice measured as dichotomous process adherence outcomes) compared with usual care (median improvement 8.45% (IQR 2.54% to 20.58%); 39 comparisons, 40,346 participants; moderate certainty of evidence) and may make little or no difference to continuous process‐of‐care outcomes (8 comparisons, 3263 participants; low certainty of evidence). Adding manually‐generated paper reminders to one or more QI co‐interventions may slightly improve professional practice measured as dichotomous process adherence outcomes (median improvement 4.24% (IQR −1.09% to 5.50%); 12 comparisons, 25,359 participants; low certainty of evidence) and probably slightly improve professional practice measured as continuous outcomes (median improvement 0.28 (IQR 0.04 to 0.51); 2 comparisons, 12,372 participants; moderate certainty of evidence). Compared with other QI interventions, manually‐generated reminders may slightly decrease professional practice measured as process adherence outcomes (median decrease 7.9% (IQR −0.7% to 11%); 14 comparisons, 21,274 participants; low certainty of evidence).
We are uncertain whether manually‐generated reminders delivered on paper, compared with usual care or with other QI intervention, lead to better or worse patient outcomes (dichotomous or continuous), as the certainty of the evidence is very low (10 studies, 13 comparisons). Reminders added to other QI interventions may make little or no difference to patient outcomes (dichotomous or continuous) compared with the QI alone (2 studies, 2 comparisons).
Regarding resource use, studies reported additional costs per additional point of effectiveness gained, but because of the different currencies and years used the relevance of those figures is uncertain.
None of the included studies reported outcomes related to harms or adverse effects.
Authors' conclusions
Manually‐generated reminders delivered on paper as a single intervention probably lead to small to moderate increases in outcomes related to adherence to clinical recommendations, and they could be used as a single QI intervention. It is uncertain whether reminders should be added to other QI intervention already in place in the health system, although the effects may be positive. If other QI interventions, such as patient or computerised reminders, are available, they should be preferred over manually‐generated reminders, but under close evaluation in order to decrease uncertainty about their potential effect.
Plain language summary
The effect of manually‐generated reminders delivered to providers on paper on professional practice and patient outcomes
What is the aim of this review?
In this Cochrane Review we aimed to find out if health workers who are given reminders on paper give better health care. The reminders contained information about the patients, for instance recommendation to measure blood pressure. We collected and analysed all relevant studies and found 63 studies.
Key messages
It seems likely that providing reminders to health workers probably leads to small‐to‐moderate improvements in their practice measured as adherence to clinical recommendations. It is uncertain whether providing reminders has an effect on patient outcomes.
What was studied in the review?
Health workers do not always provide care that is recommended by clinical guidelines or standards, because of too much information or unconscious forgetfulness. One possible solution is to give them paper reminders that were not created by a computer. These are particularly important in countries where electronic records are not widely available. Reminders may help health workers overcome those problems by prompting them to follow clinical recommendations in guidelines or by providing information in a simple and timely way. In this review we evaluated the effects of reminders generated manually and delivered on paper on professional practice and patient outcomes.
What are the main results of the review?
We identified 63 studies and included 57 in our analysis. The studies evaluated reminders aimed at ordering screening tests, providing vaccinations, prescribing specific medications, or discussing care with patients. The studies show that:
‐ reminders alone (single‐component intervention) probably improve professional practice, measured as compliance with recommendations, compared with usual care;
‐ reminders added to one or more co‐interventions (multicomponent intervention) may slightly improve professional practice, measured as compliance with recommendations, compared with the co‐intervention(s) without the reminder component;
‐ reminders may lead to slightly worse professional practice than other interventions for quality improvement, such as patients reminders;
‐ it is uncertain whether reminders compared with usual care or other quality‐improvement interventions improve patient outcomes;
‐ reminders added to other quality‐improvement interventions may make little or no difference to patient outcomes compared with the quality intervention alone;
‐ there were additional costs to obtain the effects described above, but the relevance of the figures presented was uncertain;
‐ none of the included studies reported outcomes related to harms or adverse effects.
How up‐to‐date is this review?
The review authors searched for studies published up to December 2018.
Summary of findings
Summary of findings for the main comparison. Manual paper reminders compared with usual care.
| Manual paper reminders compared with usual care | ||||
|
Patient or population: Healthcare professionals (physicians, nurses and dentists) Settings: Ambulatory and hospital care in USA, Canada, UK, France, Switzerland, Taiwan, Australia, Germany, Hong Kong, India, Israel, Spain and Thailand Intervention: Manual paper reminders focused on improving compliance with preventive guidelines (e.g. cancer screening, vaccination) and disease management guidelines (e.g. annual follow‐ups, test‐ordering, medication adjustment, counselling) Comparison: Control/usual care | ||||
| Outcomes | Median improvement (IQR) | Nº of studies (Nº participants) | Certainty of the evidence (GRADE) | Comments |
| Changes in professional practice (measured as dichotomous process adherence outcomes) | 8.45% (2.54% to 20.58%) | 39 studies (40,346) | ⊕⊕⊕⊝ moderatea | Studies assessed outcomes such as proportion of patients receiving colonoscopy or compliance with 13 preventive health manoeuvres |
| Changes in professional practice (continuous) | −0.002 (−0.02 to 0.01) | 8 studies (3263) | ⊕⊕⊝⊝ lowb | The effect estimate is a median of the standardised mean differences in the studies assessing outcomes such as number of office visits, number of transfusion units or number of missed opportunities per patient per year |
| Patient outcomes (dichotomous) | 3.24% (2.31% to 4.12%) | 7 studies (8390) | ⊕⊝⊝⊝ very lowc | Studies assessed outcomes such as mortality rates, or smokers quitting rates |
| Patient outcomes (continuous) | 0.001 (−0.002 to 0.11) | 4 studies (1222) | ⊕⊝⊝⊝ very lowc | Studies assessed outcomes such as mean catheter days or systolic and diastolic blood pressure |
| Resource use | Additional health service costs of GBP 65 and between EUR 41 and EUR 59 | 2 studies (2570) | ⊕⊕⊝⊝ lowd | The additional costs are per additional point of effectiveness gained (additional attendance for breast cancer screening and additional point of Guideline Conformity Rate) |
| Adverse effects | Not reported | ‐ | ‐ | None of the included studies reported outcomes related to adverse effects of reminders |
| GRADE Working Group grades of evidence High quality: Further research is very unlikely to change our confidence in the estimate of effect. Moderate quality: Further research is likely to have an important impact on our confidence in the estimate of effect and may change the estimate. Low quality: Further research is very likely to have an important impact on our confidence in the estimate of effect and is likely to change the estimate. Very low quality: We are very uncertain about the estimate. | ||||
aWe downgraded the certainty of the evidence because of methodological limitations in the included studies (17 of them were rated at high risk of bias for this outcome). bWe downgraded the certainty of the evidence because of methodological limitations in the included studies, imprecision (very wide range of effects), and some inconsistency in the results. cWe downgraded the certainty of the evidence because of methodological limitations in the included studies and very serious inconsistency (very diverse clinical areas). dWe downgraded the certainty of the evidence because of inconsistency (very different health systems and currencies) and imprecision in the results.
Summary of findings 2. Manual paper reminders added to another QI intervention compared with the same QI intervention.
| Manual paper reminders added to another QI intervention compared with the same QI intervention without reminders | ||||
|
Patient or population: Healthcare professionals (physicians, nurses and dentists) Settings: Ambulatory and hospital care in USA, Canada, UK, France, Switzerland, Taiwan, Australia, Germany, Hong Kong, India, Israel, Spain and Thailand Intervention: Manual paper reminders added to another QI intervention (feedback, patient reminders, educational meetings, educational materials, test request forms) Comparison: Another QI intervention (feedback, patient reminders, educational meetings, educational materials, test request forms) | ||||
| Outcomes | Median improvement (IQR) | No of studies (Nº participants) | Certainty of the evidence (GRADE) | Comments |
| Changes in professional practice (measured as dichotomous process adherence outcomes) | 4.24% ( −1.09% to 5.50%) | 12 studies (25,359) | ⊕⊕⊝⊝ lowa | Studies assessed outcomes such as attendance for breast cancer screening or coverage of faecal occult blood test screening for colorectal cancer |
| Changes in professional practice (continuous) | 0.28 (0.04 to 0.51) | 2 studies (12,372) |
⊕⊕⊕⊝ moderateb | The effect estimate is a median of the standardised mean differences in the studies assessing outcomes such as scores of compliance with guidelines and mean practice mammography referral and completion rates |
| Patient outcomes (dichotomous) | −3.16% (−8.51% to 2.18%) | 2 studies (1883) |
⊕⊕⊝⊝ lowc | Studies assessed outcomes such as patient smoking quitting rates and proportion of patients with no periods Negative figures indicate a median decrease |
| Patient outcomes (continuous) | 0.001 (−0.003 to 0.003) | 1 study (946) | ⊕⊕⊕⊝ moderated | The effect estimate is a median of 10 outcomes (condition‐specific outcomes and four domains of SF‐36 in women with menorrhagia or urinary incontinence) assessed in a single study Negative figures indicate a median decrease |
| Resource use | Additional health service costs of GBP 30 and between EUR 16.5 and EUR 67 | 2 studies (2570) | ⊕⊕⊝⊝ lowe | The additional costs are per additional point of effectiveness gained (additional attendance for breast cancer screening and additional point of Guideline Conformity Rate) |
| Adverse effects | Not reported | ‐ | ‐ | None of the included studies reported outcomes related to adverse effects of reminders |
| GRADE Working Group grades of evidence High quality: Further research is very unlikely to change our confidence in the estimate of effect. Moderate quality: Further research is likely to have an important impact on our confidence in the estimate of effect and may change the estimate. Low quality: Further research is very likely to have an important impact on our confidence in the estimate of effect and is likely to change the estimate. Very low quality: We are very uncertain about the estimate. | ||||
aWe downgraded the certainty of the evidence because of methodological limitations in the included studies (two studies were at high risk of bias and seven were at unclear risk), inconsistency of the results and some imprecision in the effect estimates. bWe downgraded the certainty of the evidence because of important methodological limitations of the included studies (one at high risk of bias and in the other at unclear risk), imprecision of the effect estimates and inconsistency in the results. cWe downgraded the certainty of evidence because of inconsistency and imprecision in the results. dWe downgraded the certainty of evidence because of imprecision in the results for the different outcomes measured in the study. eWe downgraded the certainty of the evidence because of inconsistency (very different health systems and currencies) and imprecision in the results.
Summary of findings 3. Manual paper reminders compared with other quality‐improvement (QI) interventions.
| Manual paper reminders compared with other quality‐improvement (QI) interventions | ||||
|
Patient or population: Healthcare professionals (physicians, nurses and dentists) Settings: Ambulatory and hospital care in USA, Canada, UK, France, Switzerland, Taiwan, Australia, Germany, Hong Kong, India, Israel, Spain and Thailand Intervention: Manual paper reminders focused on improving compliance with preventive guidelines (e.g. cancer screening, vaccination) and disease management guidelines (e.g. annual follow‐ups, test‐ordering, medication adjustment, counselling) Comparison: Other QI interventions (patient reminders, computerised reminders, educational meeting or multifaceted interventions) | ||||
| Outcomes | Median improvement (IQR) | No of studies (Nº participants) | Certainty of the evidence (GRADE) | Comments |
| Changes in professional practice (measured as dichotomous process adherence outcomes) | −7.89% (−10.98% to 0.70%) | 14 studies (21,274) | ⊕⊕⊝⊝ lowa | Studies assessed outcomes such as compliance with post‐partum screening of diabetes mellitus and adherence to guideline recommendations on cardiovascular prevention Negative figures indicate a median decrease |
| Changes in professional practice (continuous) | Not reported | ‐ | ‐ | None of the included studies reported changes in professional practice measured as continuous outcomes |
| Patient outcomes (dichotomous) | −2.08% (−17.95% to 2.28%) | 3 studies (2305) | ⊕⊝⊝⊝ very lowb | Studies assessed outcomes such as stopping smoking or seizure‐free rates in patients with epilepsy Negative figures indicate a median decrease |
| Patient outcomes (continuous) | Not reported | ‐ | ‐ | None of the included studies reported patient outcomes measured as continuous variables |
| Resource use | Additional health service costs of GBP 30 and between EUR 17 and EUR 55 euros. The additional costs of maintenance were 78 cents (USD 1991) per patient per year. | 3 studies (4235) |
⊕⊕⊝⊝ low | The additional costs are per additional point of effectiveness gained (additional attendance to breast cancer screening and additional point of Guideline Conformity Rate) or the additional costs of maintaining the computer system that generates the computer reminders |
| Adverse effects | Not reported | ‐ | ‐ | None of the included studies reported outcomes related to adverse effects of reminders |
| GRADE Working Group grades of evidence High quality: Further research is very unlikely to change our confidence in the estimate of effect. Moderate quality: Further research is likely to have an important impact on our confidence in the estimate of effect and may change the estimate. Low quality: Further research is very likely to have an important impact on our confidence in the estimate of effect and is likely to change the estimate. Very low quality: We are very uncertain about the estimate. | ||||
aWe downgraded the certainty of the evidence because of methodological limitations in the included studies (two studies were at high risk of bias and nine were at unclear risk), and imprecision in the effect estimates. bWe downgraded the certainty of the evidence because of methodological limitations of the included studies (one was at high risk of bias and two were at unclear risk), imprecision of the effect estimates and inconsistency in the results.
Background
Description of the condition
Clinical practice does not always reflect best evidence, and high proportions of inappropriate care have been reported in different healthcare systems and settings (Grol 2003; McGlynn 2003; Mangione‐Smith 2007). This has an impact on patient outcomes, and healthcare costs. Passive dissemination of the evidence is often not enough to promote uptake of research, so specific strategies to encourage implementation of research‐based recommendations and to ensure changes in practice have been advocated (Bero 1998; CRD 1999; Grol 2003; Shojania 2004; Shojania 2005; Grimshaw 2012). These strategies fall into a number of different categories: educational interventions (directed at clinicians or at patients), audit and feedback of performance data, case management, financial incentives and reminders, to name a few (Grol 2005; Shojania 2005). However, none of them have consistently conferred large improvements in care, especially when evaluated rigorously. In fact, they have been shown to produce only small‐to‐moderate improvements in processes of care, and smaller improvements in patient outcomes (Grimshaw 2012; Johnson 2015).
Description of the intervention
Provider reminders are a common approach to help clinicians remember to perform specific clinical tasks such as prescribing drugs, reviewing blood test results, ordering investigations or recommending preventive services (e.g. vaccination) (Buntinx 1993; Austin 1994; Balas 2000; Dexheimer 2008). They aim to prompt healthcare professionals to recall information that they may already know but could easily forget in the midst of performing other activities of care. Reminders can take many forms, depending on the way in which computers are involved in their production and delivery. Thus, we can distinguish the following three major groups of reminders.
Manually‐generated paper reminders are those where no computer is involved in the production or delivery of the reminder or in selecting target patients. They could range from simple notes attached to the front of every chart ('static' prompts) to more sophisticated reminders given under specific conditions for specific types of patients ('dynamic' prompts).
Computer‐generated paper reminders are those where a computer is used either to generate paper reminders or to identify patients for whom health professionals should receive a paper reminder.
Point‐of‐care computer reminders are those where the computer reminder is delivered to the health professional at the time they are engaged in the target activity of interest through a computer screen.
This review is focuses on the first group of reminders. The effectiveness of the other two groups has been reviewed by related Cochrane Reviews (Shojania 2009; Arditi 2017).
How the intervention might work
Health professionals sometimes do not use the best evidence to treat their patients, in part due to unconscious acts of omission, information overload or a number of practical issues including lack of time (McDonald 1976; Carlsen 2007). Reminders help clinicians overcome some of these problems by prompting them to recall information that they already know or would be expected to know, by presenting information in a different and more accessible or relevant format. In that sense, reminders work as a feedback of information intended to modify clinical practice, but in contrast with audit and feedback the information is presented close to the time of decision‐making, increasing the probable effect of intervention (Mugford 1991; Ivers 2012a).
The effects of reminders on professional practice may be influenced by a number of factors such as: how the reminders are delivered (e.g. checklist, coloured stickers); whether they provide generic or patient‐specific information; whether they provide an explanation of their content or not; whether they are explicitly supported by an influential source (Flodgren 2019); whether they require the healthcare professional to record a response (Litzelman 1993); whether targeted clinicians were involved in their development (Cohen 1994); and the behaviour targeted by them (Carlsen 2007; Arditi 2017).
Why it is important to do this review
When implementing a reminder system, the decision to use manual methods or a computer to produce or deliver reminders has major resource implications. Many health systems around the world, especially those in low‐income countries, cannot afford electronic clinical records currently or in the foreseeable future (Bosch‐Capblanch 2017). Relatively cheap interventions to change clinical practice, such as paper reminders, are therefore especially relevant in many settings, provided that they are effective.
Although a number of systematic reviews have indicated that reminders to healthcare professionals can be effective in promoting change in professional practice across a variety of clinical areas and settings (Johnston 1994; Wensing 1994; Mandelblatt 1995; Oxman 1995; Bero 1998; Balas 2000; Szilagyi 2000; Grimshaw 2001; Grol 2003; Dexheimer 2008; Reckmann 2009; Schedlbauer 2009; Grimshaw 2012), they have focused on reminders as one of a wide range of interventions aimed at improving professional practice (Wensing 1994; Mandelblatt 1995; Oxman 1995; Grimshaw 2001; Grol 2003; Grimshaw 2012), or focused on computer reminders (Johnston 1994; Reckmann 2009; Schedlbauer 2009). In addition, factors that may modify the effectiveness of reminders have not been systematically considered, except for the Cochrane Reviews assessing computer‐generated or point‐of‐care computer reminders (Shojania 2009; Arditi 2017). This review specifically assesses the effects of manual paper reminders on professional practice and patient outcomes, and assesses the extent to which different features of manual paper reminders modify their effectiveness.
Objectives
To determine the effectiveness of manually‐generated reminders delivered on paper in changing professional practice and improving patient outcomes.
To explore whether a number of potential effect modifiers influence the effectiveness of manually‐generated reminders delivered on paper.
Methods
Criteria for considering studies for this review
Types of studies
We have included the following study designs in this review:
-
Randomised trial:
a trial where the allocation of patients, encounters or groups of patients (grouped by practitioners, firms, teams, hospitals) to the reminder intervention is stated as being randomised;
-
Non‐randomised trial:
a trial where non‐random but probably balanced methods of allocation such as alternation by day of the week, odd/even patient case‐note numbers etc. are used.
The minimum methodological inclusion criteria across all study designs are (EPOC 2017a):
the objective measurement of performance/provider behaviour of health/patient outcome(s) in a clinical situation; and
relevant and interpretable data presented or obtainable.
We eliminated duplicate reports of studies by comparing authors' names and the type and location of the study. We included reports published in non‐English languages.
Types of participants
Any qualified healthcare professional, where qualified healthcare professionals form the majority (more than 50%) of the study population. We excluded studies that primarily target healthcare professional trainees.
Types of interventions
Manually‐generated reminders delivered on paper are defined as information given to the health professional with each patient or encounter (e.g. blood pressure measurement recommendation for annual health examination), provided on paper, in which no computer is involved in the production or delivery of the reminder. The reminder is designed or intended to prompt healthcare professionals to recall information usually encountered through their general education, in the medical records or through interaction with peers, and remind them to perform or avoid some action to aid individual patient care. They could range from simple notes attached to the front of every chart ('static' prompts) to more sophisticated reminders given under specific conditions for specific types of patients ('dynamic' prompts).
The comparisons assessed in this review are:
manually‐generated reminders delivered on paper as a single‐component intervention compared with no intervention (control/usual care);
manually‐generated reminders deliver on paper added to other quality improvement (QI) co‐interventions (multicomponent intervention) compared with the co‐intervention(s) without the reminder component;
manually‐generated reminders delivered on paper compared with other QI interventions.
The QI interventions are those described in the Cochrane Effective Practice and Organisation of Care (EPOC) taxonomy under the implementation strategies category 'Intervention targeted at healthcare workers' (EPOC 2015).
Types of outcome measures
Primary outcomes
We include studies which report analysable data for any of the following outcomes:
Professional practice as measured by:
Dichotomous process adherence outcomes: the percentage of patients receiving a target process of care (e.g. prescription of a specific medication, documentation of performance of a specific task, such as referral to a consultant) or whose care was in compliance with a guideline recommendation;
Continuous process outcomes: any continuous measure of how providers delivered care (e.g. duration of antibiotic therapy, time to respond to a critical lab value).
Patient outcomes as measured by:
Dichotomous clinical outcomes: patient‐important endpoints (such as death or development of a pulmonary embolism), as well as surrogate or intermediate endpoints, such as achievement of a target blood pressure or serum cholesterol level;
Continuous clinical outcomes: various markers of disease or health status (e.g. mean blood pressure or cholesterol level).
Although we have included studies reporting the effect of manually‐generated reminders delivered on paper on patient outcomes, the primary goal of our analysis was to evaluate the impact of reminders on adherence to targeted processes of care or guideline recommendations (dichotomous process adherence outcomes). We recognise that improving patient outcomes represents the ultimate goal of any quality improvement activity. However, we focus on process improvement for this review because we want to capture the degree to which reminders achieve their main immediate goal: changing provider behaviour (Mason 1999; Grol 2005).
Secondary outcomes
We also consider the following relevant outcomes:
Adverse effects: for instance, if reminders are intended to promote use of an underused service (e.g. cervical cancer screening services), but they decrease the use of the service.
Resource use: the amount of resources used for implementing the intervention in the trial setting, and any costs or savings attributable to implementation of the intervention.
We did not specify a priori any anticipated differential effects of the intervention on disadvantaged groups. The potential for differential effects depends on the health issue being addressed by the intervention and on the setting. We considered relevant differential effects, if sufficient data were reported in post hoc subgroup analyses.
Search methods for identification of studies
Electronic searches
For this review, the Cochrane EPOC Information Specialist rewrote search strategies in consultation with the authors. We searched the following databases for primary studies on 5 December 2018:
Cochrane Central Register of Controlled Trials (CENTRAL; 2017 Issue 5) in the Cochrane Library;
MEDLINE (OVID) (including Epub Ahead of Print, In‐Process & Other Non‐Indexed Citations and Versions);
Embase (OVID);
Cumulative Index to Nursing and Allied Health Literature (CINAHL (EBSCO)).
We searched all databases from database start date to date of search. We translated the MEDLINE search strategy for other databases using appropriate syntax and vocabulary for those databases. We present full search strategies in Appendix 1. We limited the results by two methodological filters: a modified version of the 'Cochrane Highly Sensitive Search Strategy to identify randomised trials in MEDLINE: sensitivity‐ and precision‐maximising version (2008 revision)' (Higgins 2017), and a Cochrane EPOC Group search filter to identify other study designs.
Searching other resources
Grey literature
We conducted a grey literature search on OpenGrey (www.opengrey.eu/) to identify studies not indexed in the databases listed above. We documented additional sources, if any, in the review.
Trial Registries
International Clinical Trials Registry Platform (ICTRP), World Health Organization (WHO) www.who.int/ictrp/en/ (searched 5 December 2018)
ClinicalTrials.gov, US National Institutes of Health (NIH) clinicaltrials.gov/ (searched 5 December 2018)
We also:
screened individual journals and conference proceedings (e.g. handsearching);
reviewed reference lists of all included studies, relevant systematic reviews/primary studies/other publications;
contacted authors of relevant studies or reviews to clarify reported published information/seek unpublished results/data;
contacted researchers with expertise relevant to the review topic/EPOC interventions;
conducted cited reference searches for all included studies in citations indexes.
Data collection and analysis
Selection of studies
Two review authors (from TP, NC, JL), working independently in pairs, screened all titles and abstracts to assess which studies met the inclusion criteria. We retrieved full‐text copies of all papers that were potentially relevant and two review authors (from TP, NC, JL, CC), working in pairs, assessed them independently. We resolved any disagreement by discussion or by consulting an arbitrator (JG) in case of persistent disagreement. We kept a log of the selection process to complete a PRISMA flow diagram (Moher 2009) and a Characteristics of excluded studies table (See Figure 1).
Data extraction and management
Two review authors (from TP, NC, JL, CC) undertook data abstraction independently, using a modified version of the Cochrane EPOC Group data collection checklist (EPOC 2017b) in Covidence. We resolved any disagreements by discussion or by consulting an arbitrator (JG) if necessary.
Assessment of risk of bias in included studies
Two review authors (from TP, JL, CC), working independently in pairs, assessed the risks of bias of all eligible studies, using the criteria suggested in the Cochrane EPOC Resources for review authors (EPOC 2017c), resolving discrepancies by discussion and by involving an arbitrator (JG) where necessary.
We summarised the overall risk of bias for each study (across outcomes) and for each outcome or class of similar outcomes (across studies) using the following criteria (Higgins 2017).
-
Within each study across domains:
we considered studies with low risk of bias for all key domains or where it seems unlikely for bias to seriously alter the results to have a low risk of bias;
we considered studies where risk of bias in at least one domain was unclear or judged to have some bias that could plausible raise doubts about the conclusions to have an unclear risk of bias;
we considered studies with a high risk of bias in at least one domain or judged to have serious bias that decrease the certainty of the conclusions to have a high risk of bias.
-
Across studies, we defined:
each outcome (or class of outcomes) as having a low risk of bias if most information was from studies at low risk of bias;
as high risk of bias if the proportion of information from studies at high risk of bias was sufficient to affect the interpretation of the results;
an unclear risk of bias if most information was from studies at low or unclear risk of bias.
We present our findings in a 'Risk of bias' table, and use graphs and figures to summarise our assessments across studies.
Measures of treatment effect
Measures of effect depended on the type of outcome data presented in the individual studies. For dichotomous outcomes we calculated differences in proportions between the intervention and comparison groups before and after the intervention. For continuous outcomes we calculated standardised effect sizes by dividing the difference in mean scores between the intervention and comparison group in each study by an estimate of the (pooled) standard deviation. We reported a single effect size for each type of outcome in each comparison in each study. If more than one measure of treatment effect was reported for a type of outcome within the same study, we used the primary outcome as defined by the study author. If there was not a clear primary outcome reported, we computed and used a median from all available outcomes of the same type.
Direction of improvement
Some studies targeted quality problems that involve 'underuse', so that improvements in quality correspond to increases in the percentage of patients who receive a target process of care (for example, increasing the percentage of patients who receive the influenza vaccine). However, other studies targeted 'overuse', so that improvements correspond to reductions in the percentage of patients receiving inappropriate or unnecessary processes of care (for example, reducing the percentage of patients who receive antibiotics for viral upper respiratory tract infections). In order to standardise the direction of effects, we defined all process adherence outcomes so that higher values represented an improvement. For example, data from a study aimed at reducing the percentage of patients receiving inappropriate medications would be captured as the complementary percentage of patients who did not receive inappropriate medications. Increasing this percentage of patients for whom providers did not prescribe the medications would thus represent an improvement. Each outcome can then be interpreted as compliance with desired practice.
Adjustment for baseline differences
Unit of analysis issues
We expected that many eligible studies would be cluster designs (studies in which the unit of allocation is not a person, but is instead a group of people). We determined whether the data were correctly analysed: comparisons that allocate groups of participants (for example, primary care centres) should account for clustering during analysis, in order to prevent 'unit of analysis' errors, resulting in artificially extreme P values and over‐narrow confidence intervals (CIs) (Ukoumunne 1999).
In cluster‐randomised trials, we considered data to have been analysed correctly if:
the analysis was conducted at the same level as the allocation (i.e. at the cluster level);
the usual analysis was used but the sample size was reduced to its 'effective sample size', or the variance was inflated by the design effect; or
the analysis was conducted at the level of the individual, but appropriate statistical correction for the clustering was performed (such as generalised estimating equations (GEE), mixed models, or multilevel models).
When we detected 'unit of analysis' errors we did not attempt to re‐analyse these data and we reported the results of the study as point estimates of the intervention effect, without using any statistical measure of precision (P values or CIs).
Dealing with missing data
We attempted to contact authors to obtain important missing information for studies that were published within the last 10 years. For studies published before 2008, we had planned to use our best judgement to determine the missing information from the available publication (e.g. obtaining the numbers corresponding to outcomes only presented in graphs). However, we were able to obtain all the data needed to compute effect estimates from the available publications.
Assessment of heterogeneity
We assessed heterogeneity visually by preparing box plots (displaying medians, interquartile ranges, and ranges) to explore the size of the observed effects in relation to a number of explanatory factors (Subgroup analysis and investigation of heterogeneity). We considered each of them one at a time by looking for patterns in the distribution of the effect sizes. As we anticipated that there would not be enough studies for each relevant comparison, we did not consider formal tests of homogeneity, and we did not plan to use meta‐regression to see whether the effect sizes could be predicted by study characteristics.
Assessment of reporting biases
Because the number of health professionals/practices in most of the studies was not reported in a reliable way, we were unable to use a funnel plot to visually explore the risk of publication bias (see Differences between protocol and review).
Data synthesis
We based our primary analyses upon consideration of dichotomous process adherence measures (for example, the proportion of patients managed according to evidence‐based recommendations). In order to provide a quantitative assessment of the effects associated with reminders without resorting to numerous assumptions or conveying a misleading degree of confidence in the results, we used the median improvement in dichotomous process adherence measures across studies. As mentioned before, where studies report more than one measure for each outcome, we extracted the primary measure (as defined by the authors of the study) or the median measure identified. For example, when the comparison reported five dichotomous process adherence outcomes and none of them was denoted as the primary outcome, then we ranked the effect sizes for the five outcomes and took the median value. This median would then contribute the single effect size for that study. With each study represented by a single median outcome, we calculated the median effect size and interquartile range across all included studies for that comparison.
Our secondary analyses explored consistency of primary analyses with other types of endpoints (for example, continuous process‐of‐care measures, dichotomous outcome of care measures and continuous outcome of care measures) using the same approach as for dichotomous process adherence outcomes.
'Summary of findings'
One author (TP) assessed the degree of confidence in the estimates of effect across studies for each outcome in each comparison using the GRADE approach (Guyatt 2008). A second author (JL) participated in discussion of unclear assessments. The evaluation included study design, risk of bias, inconsistency of effect size, indirectness, imprecision, and other considerations, including publication bias (EPOC 2017d). We present the certainty of evidence assessment results in GRADE evidence profiles (Appendix 2; Appendix 3; Appendix 4). We prepared 'Summary of findings' tables to summarise the effect estimates for changes in professional practice, patient outcomes, adverse effects and resource use for the three comparisons assessed in this review.
Subgroup analysis and investigation of heterogeneity
We assessed how the effectiveness of reminders was affected by the following modifiers, using subgroup analyses:
-
Delivery of reminder:
reminders that indicate a response should be recorded or given versus reminders that do not.
-
Type of reminder:
reminders that include some individual patient‐specific information versus generic reminders.
-
Content of reminder:
reminders that include an explanation of their content or advice versus reminders that do not include this; and
reminders that are explicitly from or justified by reference to an influential source versus reminders from another source (an influential source is a person or body likely to be perceived as credible by the target clinician).
-
Development of reminder:
reminders developed by the same person/people conducting the study; and
reminders developed with the involvement of target clinicians versus reminders developed without their participation.
-
Behaviour targeted by intervention:
reminders targeting different types of clinical activity (test ordering, medication prescribing); and
reminders targeting underuse of a process of care versus those targeting overuse of a process of care.
Based on other reminders reviews and the evidence from other literature on interventions to change professional behaviour (Shojania 2009; Arditi 2017), we hypothesised that greater effects would be associated with the following characteristics of reminders:
where a response should be recorded (Litzelman 1993);
including patient‐specific information;
including an explanation of their content/advice;
explicitly from or justified by reference to an influential source (Flodgren 2019);
developed by the same person/people conducting the study;
developed with the involvement of target clinicians (Cohen 1994);
targeting different types of clinical activity (test ordering, medication prescribing) (Arditi 2017);
targeting underuse of a process of care (Carlsen 2007).
We explored the effect of the following study features in the effect of reminders:
Publication year: older studies (before 1997) versus those carried out between 1998 and 2002 versus newer ones (after 2002);
Study design: randomised versus non‐randomised trials;
Country: studies carried out in the USA versus those performed in other countries;
Setting: reminders implemented in an outpatient setting versus those implemented in inpatient settings (Ivers 2012a);
Sample size: studies with large (> median) sample sizes versus those with small (< median) sample sizes;
Co‐interventions: studies assessing multifaceted interventions (reminders + other interventions) versus those assessing single interventions (reminders only);
Level of analysis: studies analysed at the level of group of professionals versus at the individual professional/patient level).
We used univariate statistical analyses using a non‐parametric rank‐sum test (Mann‐Whitney or Kruskal‐Wallis) for comparisons of the median effect sizes across studies in those different subgroups. We ran all of these analyses on Stata Statistical Software: Release 14 (StataCorp 2015). The figures displaying those analyses were also produced in Stata Statistical Software. We had planned to use meta‐regression to examine how the effect size was related to the potential explanatory variables previously detailed, but it was not possible to obtain or impute variances of effect sizes (without unit of analysis errors) for most of the included studies.
Sensitivity analysis
We performed sensitivity analyses using, instead of the primary outcome as defined by studies' authors, the median outcome for each study. We calculated a summary effect measure using the adjusted (for baseline differences) difference for dichotomous outcomes, in order to include information presented in studies with baseline compliance measures. We also recalculated the summary effect size for the main comparison, excluding those studies at high risk of bias.
Results
Description of studies
See: Characteristics of included studies; Characteristics of excluded studies
Results of the search
We identified 10,717 records through our search strategy, of which we removed 503 as duplicates and excluded 9972 after screening the titles and abstracts. After assessing full texts of the remaining 242 articles, we selected 63 studies (65 articles) that met all our inclusion criteria. We were unable to obtain useable data for a quantitative analysis for six studies (Cheney 1987; Cohen 1989a; Cohen 1989b; Saitz 2003; Beaulieu 2004; Roetzheim 2004), resulting in 57 studies (64 comparisons) included in our quantitative synthesis (Figure 1).
1.
Study flow diagram (Moher 2009)
Included studies
Participants and settings
Of the 63 included studies, 42 came from North America (34 from the USA and eight from Canada) and eight from the UK. The remaining studies took place in various countries (two each from France, Switzerland and Taiwan and one each from Australia, Germany, Hong Kong, India, Israel, Spain and Thailand). Most studies (54) took place in outpatient/ambulatory settings, with nine focused on hospital care. Thirty‐four studies were carried out in non‐academic institutions. Cardiovascular disease management (11 studies), cancer screening (10 studies) and preventive care (10 studies) were the clinical areas most frequently targeted. General management of a clinical condition (17 studies), test ordering (14 studies) and prescription (10 studies) were the behaviours more commonly targeted by the intervention. Fifty‐seven studies targeted physicians, four targeted multiple cadres of health professionals, and one study each targeted dentists and nurses. In 35 out of the 50 studies that reported funding sources, this came from government agencies. Other sources of funding were academic institutions (seven studies), private not‐for‐profit organisations (four studies), and the pharmaceutical industry (four studies).
Design
Fifty‐nine of the studies were randomised trials, including one cross‐over trial (Chan 2002), and four studies were non‐randomised trials (Wigton 1981; Gonzalez 1989; Burns 2002; Cibere 2002), including one cross‐over trial (Cibere 2002). In 41 of the 59 randomised trials the allocation of the study groups was by cluster (providers or provider groups, cluster‐randomised trials) and in 18 the allocation was by individual (randomised trials).
Intervention (reminder features)
There was a variety of reminder features in the included studies. For illustrative purposes we present below the description of the intervention from selected studies:
"The reminder was red in colour, written in mix Chinese and English. Beneath the reminder, the current policy of statins reimbursement issued by National Health Insurance (NHI) of Taiwan was attached." "In the study group, a reminder was stamped on the next blank page of the medical chart." "The statement stamped on the paper chart read: ‘Statin can be beneficial to the patients with documented coronary artery disease regardless of their LDL level'.'' (Hung 2008; Figure 2).
2.
A reminder to improve the adherence to lipid guidelines (Hung 2008)
"The chart reminder was a sticky note following National Osteoporosis Foundation guidelines that practices could place on the charts wherever they though it would be most effective." "...office personnel where to place these reminders on the charts of all women aged 65 years and older who were scheduled to come in for annual exams, not just those women in the study." (Levy 2009; Figure 3).
The reminder was "a self‐inking paper stamp memory aid tool for use by primary care physicians when they examine their asthmatic patients. The stamp provides a checklist for the physicians that summarizes the eight CPG criteria for asthma control..." "the self‐inking stamp that was given to physicians was designed to be stamped onto the patient's chart at each visit." (Renzi 2006; Figure 4).
3.
A reminder to improve osteoporosis screening (Levy 2009)
4.
A reminder to increase asthma guidelines knowledge and implementation by primary care physicians (Renzi 2006)
In 59 studies the intervention was 'pushed' on the target health professional. In 45 studies the intervention did not use patient‐specific information (generic reminders). For other reminder features, in 23 studies a response to the reminder was required, in 19 studies the reminder was supported by an explanation, in 13 studies there was mention of an influential source in the reminder, in 45 studies the reminder was clearly developed by authors, and in only four studies was there a clear involvement of recipients in the development of the reminder.
Outcome measures
There were large variations in the outcomes measured by the included studies. Forty‐eight studies reported changes in professional practice measured as dichotomous process adherence outcomes (e.g. compliance with guidelines recommendations), with 16 reporting those changes as continuous process‐of‐care outcomes (e.g. number of days with catheters), eight reporting patient outcomes measured in a dichotomous way (e.g. mortality rates, incidence of nosocomial infections) and five reporting patient outcomes measured on a continuous scale (e.g. mean systolic blood pressure, mean cardiovascular risk). In 16 studies, it was not possible to identify the time at which outcomes were measured. For those in which this temporal information was available the range was from immediately after consultation to 18 months after the introduction of the intervention. However, in most of them (38 studies) the outcomes were measured between six and 12 months after the intervention.
Excluded studies
We excluded 174 studies (177 articles) (Figure 1). Seventy‐three studies were excluded because they assessed a multifaceted intervention in which reminders played only a minor role. Twenty‐seven studies assessed the effect of a non‐reminder intervention (in most cases audit and feedback), and 25 studies evaluated patient‐reminder interventions. In 34 studies the reminders were computer‐generated, in seven studies the reminders were presented on a screen, and in three studies the intervention was a mix of computer and paper reminders. The other four studies were excluded because of a non‐eligible comparison group (1), non‐eligible participants (2), and insufficient information to make an inclusion/exclusion judgement (1).
Risk of bias in included studies
See Figure 5 and Figure 6 for summaries of risk of bias, and Characteristics of included studies for details of risks of bias in each study.
5.
Risk of bias summary: review authors' judgements about each risk of bias item for each included study.
6.
Risk of bias graph: review authors' judgements about each risk of bias item presented as percentages across all included studies.
Allocation
Four of the included trials were clearly non‐randomised and therefore their risk of bias for this domain was high (Wigton 1981; Gonzalez 1989; Burns 2002; Cibere 2002). In 39 of the 59 randomised trials the sequence generation procedure was not clearly described, and in only seven trials (Bankhead 2001; Richards 2001; Boekeloo 2003; Saitz 2003; Halterman 2005; Kunz 2007; Yazdany 2011) the process of allocation concealment clearly took place.
Blinding
Because of the type of intervention, in most cases personnel and patients were unblinded, but the impact of these on effect estimates was unclear. Sixteen studies reported that outcomes assessors were blinded (Wigton 1981; Cowan 1992; Headrick 1992; Lilford 1992; Szilagyi 1996; Zenni 1996; Somkin 1997; Shaw 2000; Burns 2002; Chan 2002; Saitz 2003; Halterman 2005; Boltri 2007; Ayanian 2008; Clark 2009; Ely 2015) and in another seven they were clearly unblinded, with a high risk of detection bias (Gonzalez 1989; Baker 1997; Etter 2000; Cibere 2002; Boekeloo 2003; Levine 2003; Kunz 2007).
Incomplete outcome data
Incomplete outcome data were considered when less than 80% of the patients/practices/providers randomised were included in the analysis or when reasons for attrition were different across groups. We considered outcome data to be incomplete in 11 studies (Wigton 1981; Cohen 1989a; Cohen 1989b; Pritchard 1995; Etter 2000; Montgomery 2000; Cibere 2002; McIsaac 2002; Beaulieu 2004; Kunz 2007; Yazdany 2011), and complete in 35 studies. In 17 studies it was uncertain.
Selective reporting
We judged only one study (Etter 2000) to be at high risk of reporting bias, with 41 studies assessed as low risk. We rated the remaining 21 studies at unclear risk of selective reporting (Wigton 1981; Strecher 1991; Cowan 1992; Headrick 1992; Lilford 1992; Buchsbaum 1993; Wang 1994; Szilagyi 1996; Zenni 1996; Montgomery 2000; Shaw 2000; Bray 2002; McIsaac 2002; Daucourt 2003; Saitz 2003; Bouza 2004; Bishop 2006; Renzi 2006; Boltri 2007; Levy 2009; Nendaz 2010).
Other potential sources of bias
Baseline outcome measurements were reported in 23 studies, and population (patients/practices) characteristics at baseline were reported in 47 studies. Only 13 studies (Seto 1989; Chadha 2000; Richards 2001; Bray 2002; Cibere 2002; Daucourt 2003; Beaulieu 2004; Roetzheim 2004; Dubey 2006; Renzi 2006; Boltri 2007; Clark 2009; Levy 2009) reported relevant baseline differences in outcomes or patient characteristics potentially related to a risk of selection bias.
Lack of protection against contamination is a potential source of bias in this type of interventions, because of the risk of physicians receiving reminders for some patients and no reminders for others. We identified this potential source of bias in 22 studies, in which the allocation was by individual patients (and there were no specific procedures to protect from contamination) or when allocation was by cluster but the study was carried out in small organisations where information exchange among health professionals was highly probable.
Effects of interventions
See: Table 1; Table 2; Table 3
See: Table 1; Table 2; Table 3.
Twenty‐three out of the 57 studies with useable data reported baseline outcome measurements, with the primary analysis based on post‐intervention differences between study groups (See Differences between protocol and review).
1. Reminders versus no intervention (control/usual care)
Forty‐seven studies assessed the effect of reminders compared with control or usual care. Forty‐five of them reported effect estimates for professional practice outcomes: in 39 studies as dichotomous process adherence outcomes (23 of them reporting as the primary outcome) and in eight studies (two reporting as the primary outcome) as continuous process outcomes (e.g. number of office visits, number of transfusion units, number of missed opportunities per patient per year). Two studies (Bouza 2004; Halterman 2005) assessed both types of outcomes. In the studies assessing process adherence outcomes, 11 of the effect estimates involve test‐ordering, five were related to prescription, 11 to general management of different clinical conditions, three to professional‐patient relationship (e.g. counselling), three to recommended vaccination practices, five to multiple clinical behaviours, and one effect estimates related to hospital infection control. The absolute post‐intervention improvements (differences) in these outcomes for each of the studies are displayed in Figure 7. Manually‐generated paper reminders probably improve professional practice measured as dichotomous process adherence outcomes compared with control/usual care (median improvement 8.45% (IQR 2.54% to 20.58%); 39 comparisons, 40,346 participants; moderate certainty of evidence). On the other hand, reminders may make little or no difference to continuous process outcomes compared with control/usual care (median of standardised mean differences 0.0 (IQR −0.02 to 0.01); 8 comparisons, 3263 participants; low certainty of evidence).
7.
Reminders versus control/usual care: absolute improvements in processes of care outcomes by study using the primary outcome as defined by authors (red dot) and median improvement (blue dot)
Ten studies reported patient outcome effect estimates (Strecher 1991; Wang 1994; Zenni 1996; Moore 1997; Montgomery 2000; Chan 2002; Thapar 2002; Bouza 2004; Hung 2008; Rattanaumpawan 2016): in seven studies (8390 participants, five reported as the primary outcome) as dichotomous clinical outcomes (e.g. mortality rates, smokers quitting rates; median improvement 3.24%, IQR 2.31% to 4.12%) and in four studies (1222 participants, none of them reported as primary outcome) as continuous clinical outcomes (e.g. systolic and diastolic blood pressure; median of standardised mean differences 0.0, IQR 0.0 to 0.11). One study (Montgomery 2000) assessed both types of outcomes. We were uncertain whether reminders led to better or worse patient outcomes compared with control/usual care, because the certainty of the evidence was very low.
Two studies (Bankhead 2001; Saillour‐Glénisson 2005) (2570 participants) reported outcomes related to resource use. Bankhead 2001 reported additional health services costs of GBP 65 (1998 ‐ 1999) per additional attendance for breast cancer screening compared with no intervention. Simiarly, Saillour‐Glénisson 2005 reported additional costs of between EUR 41 and EUR 59 per point of efficacy gained (guidelines conformity rate) when comparing a memorandum pocket card with control.
None of the included studies reported outcomes related to adverse effects/harms of reminders compared with control/usual care.
Subgroup analyses: impact of reminder features on quality‐of‐care effect size
We examined the impact of a number of reminder features on the magnitude of the effect, with a similar approach as in other reminders reviews (Shojania 2009; Arditi 2017). We did not find an association between effect size and any of the features assessed: overuse/underuse, response required, generic/specific, supported by an explanation, reference to influential source, type of behaviour targeted by the reminder (general management of a condition, prescription, vaccination, test‐ordering, professional‐patient relationship, other), or consultation with recipients in the reminder development process (Figure 8).
8.
Reminders versus control/usual care: median effects for process of care (adherence) outcomes by reminder feature (p values are from Kruskall‐Wallis and Mann‐Whitney tests)
Subgroup analyses: impact of study features on quality‐of‐care effect size
When we explored the impact of a number of study/setting features on the magnitude of the effect, we found an association only with the type of academic setting (teaching/non‐teaching) in which the study was conducted (Figure 9). Studies conducted in teaching settings (e.g. University hospitals and health centres) achieved larger improvement than studies conducted in non‐teaching settings (median improvement 13.53% versus 4.11%; P = 0.0108).
9.
Reminders versus control/usual care: median effects for process of care (adherence) outcomes by study feature care (p‐values are from Mann‐Whitney test)
2. Reminders added to other QI intervention(s) compared with the QI intervention without the reminder
Thirteen studies assessed the effect of reminders added to one or more QI co‐interventions compared with the same QI intervention without the reminder component, such as feedback (Baker 1997), patient reminders (Somkin 1997; Bankhead 2001; Richards 2001; Boekeloo 2003; Clark 2009), educational meetings (Strecher 1991; Grady 1997; Chadha 2000; Laprise 2009), educational materials (Renzi 2006), and test‐ordering forms (Daucourt 2003; Saillour‐Glénisson 2005). All of them reported effect estimates for professional practice outcomes: in 12 studies as dichotomous process adherence outcomes (eight of them reported as the primary outcome) and in two studies as continuous process outcomes (none of them reported as the primary outcome) such as mean scores of compliance with guidelines (Chadha 2000) or mean practice mammography referral rates (Grady 1997). One study (Chadha 2000) assessed both types of outcomes. The absolute post‐intervention improvements (differences) for the studies assessing process adherence outcomes are displayed in Figure 10. Manually‐generated reminders added to one or more QI co‐interventions may slightly improve professional practice measured as process adherence outcomes (median improvement 4.24% (IQR −1.09% to 5.50%); 12 comparisons, 25,359 participants; low certainty of evidence). Reminders added to one or more QI co‐interventions probably slightly improve practice measured as continuous process outcomes (median of standardised mean differences 0.28 (IQR 0.04 to 0.51); 2 comparisons, 12,372 participants; moderate certainty of evidence).
10.
Reminders plus another QI intervention versus another QI intervention: absolute improvements in processes of care outcomes by study using the primary outcome as defined by authors (red dot) and median improvement (blue dot)
Two studies (1883 participants) reported patient outcome effect estimates (Chadha 2000; Strecher 1991), in both of them as dichotomous outcomes (e.g. smoking cessation rates; median improvement −3.2%, IQR −8.5% to 2.2%) and in one study (Chadha 2000) as a continuous outcome (e.g. scores in specific or generic quality‐of‐life scales; median of standardised mean differences 0.0, IQR 0.0 to 0.0). Manually‐generated reminders delivered on paper added to other QI interventions (such as reminders for other clinical conditions, or tutorials) may make little or no difference to health outcomes compared with the QI intervention alone (low and moderate certainty of evidence, respectively).
Two studies (Bankhead 2001; Saillour‐Glénisson 2005) (2570 participants) reported outcomes related to resource use. Bankhead 2001 reported additional health service costs of GBP 30 (1998 ‐ 1999) per additional attendance for breast cancer screening when comparing a patient letter plus a flag with the letter intervention alone. Saillour‐Glénisson 2005 reported additional costs of between EUR 16.5 and EUR 67 per point of efficacy gained (guidelines conformity rate) when comparing a memorandum pocket card added to a test request form versus the form alone.
None of the included studies reported outcomes related to adverse effects/harms for this comparison.
Subgroup analyses: impact of reminder features on quality‐of‐care effect size
We did not find an association between effect size and any of the reminder features assessed, except when the reminder was supported by an explanation (Figure 11). Studies where the reminder was supported by an explanation achieved larger improvements than studies where the reminders were not supported or where this was unclear (median improvement 12.8% versus 4.6% versus 1.9%; P = 0.0373).
11.
Reminders plus another QI intervention versus another QI intervention:median effects for process adherence outcomes by reminder feature (p values are from Kruskall‐Wallis and Mann‐Whitney tests)
Subgroup analyses: impact of study features on quality of care effect size
We did not find an association between effect size and any of the study features assessed (Figure 12).
12.
Reminders plus another QI intervention versus another QI intervention: median effects for process adherence outcomes by study feature (p values are from Kruskall‐Wallis and Mann‐Whitney tests)
3. Reminders versus other QI intervention
Fifteen studies assessed the effect of reminders compared with other QI interventions such as patient reminders (Pritchard 1995; Bankhead 2001; Richards 2001; Thapar 2002; Vinker 2002; Clark 2009), on‐screen or computerised reminders (Lilford 1992; Frame 1994; Cannon 2000; Nendaz 2010), educational meetings (Strecher 1991; Wang 1994), test request forms (Daucourt 2003; Saillour‐Glénisson 2005), and multifaceted interventions (Levine 2003). Fourteen of them present effect estimates for changes in professional practice measured as dichotomous process adherence outcomes (in nine of them reported as the primary outcome). There were no studies assessing continuous professional practice outcomes. The absolute post‐intervention improvements (differences) for each of those studies are displayed in Figure 13. Compared with other QI interventions, manually generated reminders may slightly decrease professional practice measured as process adherence outcomes (median decrease 7.9% (IQR −0.7% to 11%); 14 comparisons, 21,274 participants; low certainty of evidence). There were two studies (Cannon 2000; Levine 2003) with a large effect favouring the other QI groups, but with an unusual 100% adherence to recommendations in most of the outcomes measured in those groups.
13.
Reminders versus another QI interventions: absolute improvements in processes of care outcomes by study using the primary outcome as defined by authors (red dot) and median improvement (blue dot)
Three studies reported patient outcomes for this comparison (Strecher 1991; Thapar 2002; Wang 1994) (2305 participants) measured as dichotomous outcomes and reported as the primary outcome. We were uncertain whether reminders led to better or worse patient outcomes (such as smokers quit rates or seizure‐free rates for patients with epilepsy) compared with educational meetings (Strecher 1991; Wang 1994), or patient reminders (Thapar 2002), because the certainty of this evidence was very low (median improvement −2.08%, IQR −17.95% to 2.28%).
Three studies (Frame 1994; Bankhead 2001; Saillour‐Glénisson 2005) (4235 participants) reported outcomes related to resource use. Bankhead 2001 reported additional health service costs of GBP 30 (1998 ‐ 1999) per additional attendance for breast cancer screening when comparing the flag with the letter intervention. Saillour‐Glénisson 2005 reported additional costs of between EUR 17 and EUR 55 per point of efficacy gained (guidelines conformity rate) when comparing a memorandum pocket card with a test request form. Frame 1994 only reported that the additional cost of maintaining the computer system that generated the computer reminders was 78 cents (USD 1991) per patient per year.
None of the included studies reported outcomes related to adverse effects/harms for this comparison.
Subgroup analyses: impact of reminder features on quality‐of‐care effect size
We did not find an association between effect size and any of the reminders features assessed (Figure 14).
14.
Reminders versus another QI interventions: median effects for process adherence outcomes by reminder feature (p values are from Kruskall‐Wallis and Mann‐Whitney tests)
Subgroup analyses: impact of study features on quality‐of‐care effect size
We did not find an association between effect size and any of the study features assessed, except for the country where the study was carried out (Figure 15). Studies carried out in non‐USA settings showed smaller absolute differences than those conducted in the USA (median differences 1.41% versus 36.44%; P = 0.0072).
15.
Reminders versus another QI interventions: median effects for process adherence outcomes by study feature (p values are from Kruskall‐Wallis and Mann‐Whitney tests)
Disadvantaged populations
Because of limitations in the reporting of specific characteristics of populations in the included studies, we were unable to identify any differential effect of reminders in settings serving disadvantaged populations.
Sensitivity analysis
We found similar median improvements in professional practice measured as process‐of‐care adherence outcomes for each of the three main comparisons assessed when:
we used the median outcome for each study instead of the primary outcome as defined by studies' authors;
we calculated the effect estimate for each study using the adjusted (for baseline differences) difference when available, instead of the unadjusted post‐intervention difference;
we excluded the studies at high risk of bias.
Discussion
Summary of main results
Evidence from 39 studies (40,346 participants) showed that manually‐generated reminders delivered on paper to healthcare professionals (as a single‐component intervention) probably improve professional practice compared with control/usual care. Likewise evidence from 12 studies (25,359 participants) showed that adding these type of reminders to one or more QI co‐interventions (multicomponent intervention) may slightly improve professional practice. On the other hand, evidence from 14 studies (21,274 participants) showed that, compared with other QI interventions, manually generated reminders may slightly decrease professional practice.
It was uncertain whether manually‐generated reminders delivered on paper, compared with usual care or with other quality improvement interventions, led to better or worse patient outcomes (10 studies, 13 comparisons). Likewise, reminders added to other QI interventions may make little or no difference to patient outcomes compared with the QI intervention alone (2 studies, 2 comparisons). None of the included studies reported outcomes related to harms or adverse effects.
The lower median improvement for multicomponent compared with single‐component interventions mirrored what was found by the other Cochrane reminders reviews (Shojania 2009; Arditi 2017). One possible explanation suggested for this finding is that the improvement achieved by the other components in multifaceted interventions leaves less room for improvement by reminders. In the same way as Arditi 2017, our analyses support this explanation as post‐intervention adherence rates in the control groups of multicomponent intervention comparisons were higher than in the control groups of single‐component interventions (49.4%; IQR 22.8 to 61.8; 12 comparisons versus 31.7%; IQR 12.9 to 51.5; 39 comparisons). Another explanation mentioned by the other reviews is that multicomponent interventions target more complex and difficult‐to‐change behaviours than single‐component interventions. However, our analyses did not find differences in effect sizes among different behaviours, and we were not able to classify behaviours targeted in each study by degree of complexity or difficulty to change.
Only one reminder feature was associated with larger effect sizes: providing an explanation for the reminder. This was only found in the comparison between multicomponent interventions and other QI intervention. Although it could seem that providing an explanation may allow health professionals to better understand why they are receiving a reminder and then to act on it, and it is an association also reported by Arditi 2017, it is not possible to draw definitive conclusions, because it was an isolate finding and we did not find a similar association in the comparison between reminders as a single intervention and usual care.
Two study features were associated with effect sizes, although in different directions. When we compared reminders as a single intervention with usual care, studies conducted in academic settings showed larger effect sizes than those conducted in non‐academic settings (median improvement 13.5% versus 4.1%). A possible explanation for this could be related to the efforts spent on implementing the intervention in academic compared with non‐academic settings, and then on different degrees of implementation fidelity. However, there was not enough information in the reports about implementation fidelity to test this hypothesis. On the other hand, when we compared reminders as a single intervention with other QI intervention (such as patient reminders, or computerised reminders), studies conducted in the USA showed larger effect sizes (favouring the other QI intervention) than those carried out in non‐USA settings (median improvement 36.4% versus 1.41%). We could not identify any compelling explanation for this finding, as it was led mainly by huge effect sizes in two studies comparing computerised reminders (Cannon 2000) and a multifaceted intervention (Levine 2003) with paper reminders.
Overall completeness and applicability of evidence
The studies included in this review were conducted over the last 37 years in a variety of countries with different health system structures and organisations. Although computerised clinical record systems are currently widespread in most high‐income countries, paper records remain an important source of clinical information and documentation in health systems of many low‐income countries (Bosch‐Capblanch 2017). In this context paper‐based reminders systems are still a frequently‐used quality improvement strategy that co‐exists with computerised reminder systems. In our review most of the studies were conducted during the 1990s and 2000s (similar to what was reported by Arditi 2017). Taking into account that manual paper reminders should have been developed prior to computerised reminders, it is noteworthy that only six studies published in the 1980s were included in our review. A possible explanation for this could be that early published studies assessed the effect of reminders using designs not considered in our review (such as uncontrolled trials) and were then excluded at the titles and abstracts screening stage. However, it is not possible to discard publication bias as contributing to this finding.
Although most of the included studies were conducted in North America (USA and Canada) and the UK, studies carried out in a number of other countries with different healthcare delivery systems were also included. Additionally, studies were carried out in both ambulatory and hospital care, and interventions targeted various clinical areas for preventive and for chronic and acute care. We therefore consider that the studies included in this review represent a relatively complete body of evidence that could reasonably be applied to many health systems searching for ways to improve their quality of care through the use of manually‐generated reminders delivered on paper in their organisations. An additional aspect to consider is that most of the studies included in the review were assessing the effects of this intervention in health systems that were substantially different in technological terms from what they are currently (health systems' technology in the 1980s and 1990s compared with current health systems), which could limit their applicability. We were uncertain if reminders may have a different effect in settings serving disadvantaged populations, because of the lack of specific information about the characteristics of the populations in the included studies.
Certainty of the evidence
Overall, we rated the certainty of the evidence about the effects of manually‐generated reminders as a single intervention compared with usual care or added to other QI interventions compared with the same QI intervention on process‐of‐care adherence outcomes as moderate and low respectively. In both comparisons we downgraded the level of certainty because of methodological limitations of the included studies, mainly related to the lack of reporting of the procedures used for allocation concealment and for blinding of outcome assessors, and the high risk of contamination in trials where providers were able to take care of patients in both the intervention and control arms, and in cluster‐randomised trials where the unit of randomisation did not avoid the possibility of contact between providers in both arms of the trial. Furthermore, in both cases there was a wide range of effect sizes within individual studies (imprecision), and in the comparison between reminders as part of a multicomponent intervention the effect sizes were inconsistently spread over a range of positive and negative effects.
We rated the certainty of the evidence about the effects of manual paper reminders on patient outcomes as very low for all the comparisons included in this review. We downgraded the level of certainty because of methodological limitations, the imprecision of results and unexplained inconsistency.
Potential biases in the review process
Although we conducted extensive literature searches of multiple databases to avoid publication bias, it is possible that we have missed some studies published in the 1970s and 1980s because of problems with their indexing in different databases at an early stage of their development. It is noteworthy that we only identified six studies published in the 1980s and none before that decade. In some cases we thoroughly discussed the inclusion of some studies because of disagreements about the type of reminders being assessed or the relative importance of the reminder in a multifaceted intervention, or both. Although the use of the median effect size as an analytical approach allowed us to avoid unit‐of‐analysis errors in unadjusted cluster‐randomised trials, it limited the interpretability of the results, as we only have a range of effects for relatively heterogeneous process‐of‐care adherence outcomes, without a specific distribution that allows us to make inferences.
Agreements and disagreements with other studies or reviews
There are a number of previous reviews assessing the effectiveness of reminders as a single intervention or as part of a multifaceted intervention (Buntinx 1993; Balas 2000; Dexheimer 2008; Baron 2010). This body of evidence has recently been summarised by overviews focusing specifically on reminders (Cheung 2012) or on broader professional behaviour‐change interventions (Grimshaw 2012; Johnson 2015). In both cases, the more credible (high‐quality) reviews were the Cochrane Reviews assessing the effects of on‐screen, point‐of‐care reminders (Shojania 2009) and computer‐generated reminders delivered on paper to healthcare professionals (Arditi 2017). It is worth noting that although our review was related to those reviews, we did not have any studies that overlapped with them. When comparing reminders as a single intervention with usual care the median improvement in dichotomous process adherence outcomes in this review was slightly higher than in the review of on‐screen point‐of‐care computer reminders (8.4% versus 4.2%), and slightly lower than in the review on computer‐generated reminders delivered on paper (8.4% versus 11.0%). The median improvement in the same type of outcomes (dichotomous process adherence) by adding a reminder to another QI intervention was similar to that found by Arditi 2017 for computer‐generated reminders delivered on paper (4.2% versus 4.0%). Although Arditi 2017 found the largest improvement in process adherence outcomes for studies focusing on vaccination (median improvement 13.1%), we did not find differences among studies targeting different behaviours (similar finding to Shojania 2009) with an improvement of only 5.1% for studies focused on vaccination. Our findings on effect modifiers agree partially with Arditi 2017 in observing differences in effect sizes when reminders are supported by an explanation. However, unlike the review on computer‐generated reminders we did not find an association between effect sizes and providing space for a response or reference to an influential source. On the other hand, we did find an association with the type of academic setting that was not found in Arditi 2017. Other reviews (Shojania 2009; Baron 2010) have not found specific associations between any reminders or study features and effect sizes.
Authors' conclusions
Implications for practice.
Although the effect of manually‐generated reminders delivered on paper on patients outcomes is uncertain, the evidence supports their use when no other QI intervention is available in order to increase adherence to clinical recommendation by healthcare professionals. It is more uncertain whether this type of reminder should be added to other QI interventions already in place in the health system, although the effects may be positive, especially when the reminder added is supported by an explanation. If other QI interventions, such as patient or computerised reminders, are feasible, the evidence does not support the use of this kind of reminder.
Implications for research.
In order to improve the body of evidence available in this area, and mirroring the conclusions previously offered by Arditi 2017, we suggest that researchers should:
improve the reporting of methods used in their studies (randomisation, allocation concealment, blinding, etc.) following existing reporting standards in the field such as the CONSORT statement (Schulz 2010).
improve the description of quality‐improvement interventions (including reminders) in order to improve identification of studies, assessment of intervention complexity and comparison of reminder features (Ogrinc 2016).
improve reporting of patient outcomes and information about adverse effects (harms produced by reminders incorrectly phrased or with major information gaps) and cost of interventions.
improve reporting of baseline data in order to obtain adjusted outcome measures.
use rigorous statistical methods for the analysis of cluster designs (Higgins 2011).
carry out studies in a wider range of settings in low‐ and middle‐income countries.
assess the role of manually‐generated reminders in health systems transitioning to digital health records (at the national or local level).
assess the modifying effects of different characteristics of reminders (variations in format, content and delivery), using subgroup analysis.
History
Protocol first published: Issue 3, 1998 Review first published: Issue 12, 2019
| Date | Event | Description |
|---|---|---|
| 15 July 2014 | New search has been performed | The methods have been modified in accordance with updated EPOC guidance (EPOC 2013); the authorship has also changed. |
| 4 June 2008 | Amended | Converted to new review format. |
Acknowledgements
We acknowledge the significant contributions from authors of the previous version of the protocol: Alberto Romero, Flora Haaijer Ruskamp, Jeremy Wyatt, Michael E Green, Nicholas Hicks, Paramjit Gill, Petra Denig, Pierre Durieux, Rachel Rowe, Richard Gordon.
We acknowledge the following Cochrane EPOC editors for their contribution to the review: Julia Worswick, managing editor; Paul Miller, Information Scientist; Sofia Massa, statistical editor; Noah Ivers, internal editor; and Xavier Bosch‐Capblanch, contact editor. We also acknowledge the external peer referees: Carolina Danovaro and David Brown, specialist referees, and Jonathan Fuchs, Sheila Page and Vijayluxmi Bose. consumer referees.
We also acknowledge the contribution of the National Institute for Health Research, via Cochrane Infrastructure funding to the Effective Practice and Organisation of Care Group. The views and opinions expressed therein are those of the authors and do not necessarily reflect those of the Systematic Reviews Programme, NIHR, NHS or the Department of Health.
Appendices
Appendix 1. Search strategies
Medline (OVID)
Ovid MEDLINE(R) Epub Ahead of Print, In‐Process & Other Non‐Indexed Citations, Ovid MEDLINE(R) Daily, Ovid MEDLINE and Versions(R)
| No. | Search terms | Results |
| 1 | reminder systems/ | 3032 |
| 2 | (booklet? or chart? or checklist? or check‐list? or flowchart? or flow sheet? or flowsheet? or form? or hard copy or hard copies or insert? or leaflet* or postal? or postcard? or post‐card? or poster? or print* or sheet? or written or handwritten or chart? or checklist? or check‐list? or handwritten or hard copy or hard copies or insert* or leaflet* or manual or mail* or pamphlet* or paper or sticker?).ti,ab. | 2358209 |
| 3 | 1 and 2 | 931 |
| 4 | (reminder? adj3 (chart? or checklist? or check‐list? or handwritten or hard copy or hard copies or insert* or leaflet* or manual or mail* or pamphlet* or paper or paper‐based or postal? or postcard? or post‐card? or poster? or print* or sheet? or written)).ti,ab. | 918 |
| 5 | ((prompt? or physician prompt?) adj5 (booklet? or chart? or checklist? or check‐list? or display? or flowchart? or flow sheet? or flowsheet? or form? or hard copy or hard copies or insert? or leaflet* or manual or mail* or pamphlet* or paper or paper‐based or postal? or postcard? or post‐card? or poster? or print* or sheet? or written or handwritten)).ab. | 454 |
| 6 | (alert? adj3 (chart? or checklist? or check‐list? or handwritten or hard copy or hard copies or insert* or leaflet* or manual or mail* or pamphlet* or paper or paper‐based or postal? or postcard? or post‐card? or poster? or print* or sheet? or written)).ab. | 202 |
| 7 | ((chart? or medical record?) adj2 insert*).ti,ab. | 16 |
| 8 | ((chart? or record?) adj4 (stamp* or sticker?)).ti,ab. | 73 |
| 9 | ((alert? or prompt*) adj4 (stamp* or sticker?)).ti,ab. | 26 |
| 10 | ((alert? or prompt?) adj3 (record? or chart? or progress note?)).ti,ab. | 209 |
| 11 | ((alert? or prompt*) adj (physician? or provider? or practitioner?)).ti,ab. | 906 |
| 12 | ((prompt* or alert? or sticker? or stamp*) adj5 (patient? profile? or cue sheet? or check list? or checklist? or patient‐specific or gener* information)).ti,ab. | 104 |
| 13 | ((memory aid? or memory aide?) adj10 (physician? or professional? or provider? or doctor? or nurse?)).ti,ab. | 16 |
| 14 | ((prompt* or alert? or sticker? or stamp*) adj5 (patient? profile? or cue sheet? or check list? or checklist? or patient‐specific)).ti,ab. | 100 |
| 15 | (medical records/ or documentation/) and (alert? or prompt? or stamp* or sticker?).ti,ab. | 444 |
| 16 | or/3‐15 | 3757 |
| 17 | randomized controlled trial.pt. | 467294 |
| 18 | controlled clinical trial.pt. | 94249 |
| 19 | multicenter study.pt. | 230927 |
| 20 | pragmatic clinical trial.pt. | 599 |
| 21 | (randomis* or randomiz* or randomly).ti,ab. | 760365 |
| 22 | groups.ab. | 1748551 |
| 23 | (trial or multicenter or multi center or multicentre or multi centre).ti. | 215683 |
| 24 | (intervention? or effect? or impact? or controlled or control group? or (before adj5 after) or (pre adj5 post) or ((pretest or pre test) and (posttest or post test)) or quasiexperiment* or quasi experiment* or pseudo experiment* or pseudoexperiment* or evaluat* or time series or time point? or repeated measur*).ti,ab. | 8229357 |
| 25 | non‐randomized controlled trials as topic/ | 170 |
| 26 | interrupted time series analysis/ | 300 |
| 27 | controlled before‐after studies/ | 259 |
| 28 | or/17‐27 | 9194643 |
| 29 | exp animals/ | 2.1E+07 |
| 30 | humans/ | 1.7E+07 |
| 31 | 29 not (29 and 30) | 4423325 |
| 32 | review.pt. | 2320414 |
| 33 | meta analysis.pt. | 82107 |
| 34 | news.pt. | 183702 |
| 35 | comment.pt. | 693374 |
| 36 | editorial.pt. | 443417 |
| 37 | cochrane database of systematic reviews.jn. | 13481 |
| 38 | comment on.cm. | 693373 |
| 39 | (systematic review or literature review).ti. | 99257 |
| 40 | or/31‐39 | 7761099 |
| 41 | 28 not 40 | 6430508 |
| 42 | 16 and 41 | 2142 |
Embase (OVID)
Embase <1974 to 2017 June 26>
| No. | Search terms | Results |
| 1 | reminder system/ | 2110 |
| 2 | (booklet? or chart? or checklist? or check‐list? or flowchart? or flow sheet? or flowsheet? or form? or hard copy or hard copies or insert? or leaflet* or postal? or postcard? or post‐card? or poster? or print* or sheet? or written or handwritten or chart? or checklist? or check‐list? or handwritten or hard copy or hard copies or insert* or leaflet* or manual or mail* or pamphlet* or paper or sticker?).ti,ab. | 2783099 |
| 3 | 1 and 2 | 606 |
| 4 | (reminder? adj3 (chart? or checklist? or check‐list? or handwritten or hard copy or hard copies or insert* or leaflet* or manual or mail* or pamphlet* or paper or paper‐based or postal? or postcard? or post‐card? or poster? or print* or sheet? or written)).ti,ab. | 1208 |
| 5 | ((prompt? or physician prompt?) adj5 (booklet? or chart? or checklist? or check‐list? or display? or flowchart? or flow sheet? or flowsheet? or form? or hard copy or hard copies or insert? or leaflet* or manual or mail* or pamphlet* or paper or paper‐based or postal? or postcard? or post‐card? poster? or print* or sheet? or written or handwritten)).ab. | 601 |
| 6 | (alert? adj3 (chart? or checklist? or check‐list? or handwritten or hard copy or hard copies or insert* or leaflet* or manual or mail* or pamphlet* or paper or paper‐based or postal? or postcard? or post‐card? or poster? or print* or sheet? or written)).ab. | 277 |
| 7 | ((chart? or medical record?) adj2 insert*).ti,ab. | 24 |
| 8 | ((chart? or record?) adj4 (stamp* or sticker?)).ti,ab. | 138 |
| 9 | ((alert? or prompt*) adj4 (stamp* or sticker?)).ti,ab. | 53 |
| 10 | ((alert? or prompt?) adj3 (record? or chart? or progress note?)).ti,ab. | 321 |
| 11 | ((alert? or prompt*) adj (physician? or provider? or practitioner?)).ti,ab. | 1244 |
| 12 | ((prompt* or alert? or sticker? or stamp*) adj5 (patient? profile? or cue sheet? or check list? or checklist? or patient‐specific or gener* information)).ti,ab. | 181 |
| 13 | ((memory aid? or memory aide?) adj10 (physician? or professional? or provider? or doctor? or nurse?)).ti,ab. | 15 |
| 14 | ((prompt* or alert? or sticker? or stamp*) adj5 (patient? profile? or cue sheet? or check list? or checklist? or patient‐specific)).ti,ab. | 177 |
| 15 | (*medical record/ or *medical documentation/) and (alert? or prompt? or stamp* or sticker?).ti,ab. | 302 |
| 16 | or/3‐15 | 4461 |
| 17 | randomized controlled trial/ | 458556 |
| 18 | controlled clinical trial/ | 437367 |
| 19 | quasi experimental study/ | 3850 |
| 20 | pretest posttest control group design/ | 306 |
| 21 | time series analysis/ | 19594 |
| 22 | experimental design/ | 14408 |
| 23 | multicenter study/ | 158354 |
| 24 | (randomis* or randomiz* or randomly).ti,ab. | 988182 |
| 25 | groups.ab. | 2287653 |
| 26 | (trial or multicentre or multicenter or multi centre or multi center).ti. | 275145 |
| 27 | (intervention? or effect? or impact? or controlled or control group? or (before adj5 after) or (pre adj5 post) or ((pretest or pre test) and (posttest or post test)) or quasiexperiment* or quasi experiment* or pseudo experiment* or pseudoexperiment* or evaluat* or time series or time point? or repeated measur*).ti,ab. | 10174246 |
| 28 | or/17‐27 | 11350937 |
| 29 | (systematic review or literature review).ti. | 114934 |
| 30 | "cochrane database of systematic reviews".jn. | 6165 |
| 31 | exp animals/ or exp invertebrate/ or animal experiment/ or animal model/ or animal tissue/ or animal cell/ or nonhuman/ | 24820539 |
| 32 | human/ or normal human/ or human cell/ | 18661356 |
| 33 | 31 not (31 and 32) | 6205960 |
| 34 | 29 or 30 or 33 | 6325956 |
| 35 | 28 not 34 | 8617895 |
| 36 | 16 and 35 | 2811 |
The Cochrane Library (Wiley)
| No. | Search terms | Results |
| #1 | [mh "reminder systems"] | 799 |
| #2 | (booklet? or chart? or checklist? or check‐list? or flowchart? or flow sheet? or flowsheet? or form? or hard copy or hard copies or insert? or leaflet* or postal? or postcard? or post‐card? or poster? or print* or sheet? or written or handwritten or chart? or checklist? or check‐list? or handwritten or hard copy or hard copies or insert* or leaflet* or manual or mail* or pamphlet* or paper or sticker?):ti,ab | 54151 |
| #3 | #1 and #2 | 260 |
| #4 | (reminder near/3 (chart? or checklist? or check‐list? or handwritten or hard copy or hard copies or insert* or leaflet* or manual or mail* or pamphlet* or paper or paper‐based or postal? or postcard? or post‐card? or poster? or print* or sheet? or written)):ti,ab | 155 |
| #5 | ((prompt or (physician prompt)) near/5 (booklet or chart or checklist or check‐list or display or flowchart or (flow sheet) or flowsheet or form or "hard copy" or "hard copies" or insert or leaflet* or manual or mail* or pamphlet* or paper or paper‐based or postal or postcard or post‐card or poster or print* or sheet or written or handwritten)):ab | 46 |
| #6 | (alert near/3 (chart or checklist or check‐list or handwritten or "hard copy" or "hard copies" or insert* or leaflet* or manual or mail* or pamphlet* or paper or paper‐based or postal or postcard or post‐card or poster or print* or sheet or written)):ab | 10 |
| #7 | ((chart or (medical record)) near/2 insert*):ti,ab | 0 |
| #8 | ((chart or record) near/4 (stamp* or sticker)):ti,ab | 3 |
| #9 | ((alert or prompt*) near/4 (stamp* or sticker)):ti,ab | 2 |
| #10 | ((alert or prompt) near/3 (record or chart or (progress note))):ti,ab | 22 |
| #11 | ((alert or prompt*) near (physician or provider or practitioner)):ti,ab | 109 |
| #12 | ((prompt* or alert or sticker or stamp*) near/5 ((patient profile) or (cue sheet) or (check list) or checklist or patient‐specific or (gener* information))):ti,ab | 15 |
| #13 | (((memory aid) or (memory aide)) near/10 (physician or professional or provider or doctor or nurse)):ti,ab | 0 |
| #14 | ((prompt* or alert or sticker or stamp*) near/5 ((patient profile) or (cue sheet) or (check list) or checklist or patient‐specific)):ti,ab | 13 |
| #15 | ([mh "medical records"] or [mh documentation]) and (alert or prompt or stamp* or sticker):ti,ab | 80 |
| #16 | {or #3‐#15} | 582 |
Cinahl (EBSCO)
| No. | Search terms | Results |
| S1 | MH “reminder systems” | 1,558 |
| S2 | (booklet? or chart? or checklist? or check‐list? or flowchart? or flow sheet? or flowsheet? or form? or hard copy or hard copies or insert? or leaflet* or postal? or postcard? or post‐card? or poster? or print* or sheet? or written or handwritten or chart? or checklist? or check‐list? or handwritten or hard copy or hard copies or insert* or leaflet* or manual or mail* or pamphlet* or paper or sticker?) | 206,561 |
| S3 | S1 AND S2 | 327 |
| S4 | (reminder? N3 (chart? or checklist? or check‐list? or handwritten or hard copy or hard copies or insert* or leaflet* or manual or mail* or pamphlet* or paper or paper‐based or postal? or postcard? or post‐card? or poster? or print* or sheet? or written)) | 129 |
| S5 | ((prompt? or physician prompt?) N5 (booklet? or chart? or checklist? or check‐list? or display? or flowchart? or flow sheet? or flowsheet? or form? or hard copy or hard copies or insert? or leaflet* or manual or mail* or pamphlet* or paper or paper‐based or postal? or postcard? or post‐card? or poster? or print* or sheet? or written or handwritten)) | 50 |
| S6 | (alert? N3 (chart? or checklist? or check‐list? or handwritten or hard copy or hard copies or insert* or leaflet* or manual or mail* or pamphlet* or paper or paper‐based or postal? or postcard? or post‐card? or poster? or print* or sheet? or written)) | 49 |
| S7 | ((chart? or medical record?) N2 insert*) | 4 |
| S8 | ((chart? or record?) N4 (stamp* or sticker?)) | 3 |
| S9 | ((alert? or prompt*) N4 (stamp* or sticker?)) | 3 |
| S10 | ((alert? or prompt?) N3 (record? or chart? or progress note?)) | 16 |
| S11 | ((alert? or prompt*) N0 (physician? or provider? or practitioner?)) | 83 |
| S12 | ((prompt* or alert? or sticker? or stamp*) N5 (patient? profile? or cue sheet? or check list? or checklist? or patient‐specific or gener* information)) | 26 |
| S13 | ((memory aid? or memory aide?) N10 (physician? or professional? or provider? or doctor? or nurse?)) | 2 |
| S14 | ((prompt* or alert? or sticker? or stamp*) N5 (patient? profile? or cue sheet? or check list? or checklist? or patient‐specific)) | 12 |
| S15 | ((MH "Medical Records") OR (MH "Documentation")) AND (alert? or prompt? or stamp* or sticker?) | 90 |
| S16 | S3 OR S4 OR S5 OR S6 OR S7 OR S8 OR S9 OR S10 OR S11 OR S12 OR S13 OR S14 OR S15 | 695 |
ClinicalTrials.gov
reminder AND (booklet OR chart OR checklist OR flowchart OR leaflet OR postal OR postcard OR poster OR print OR sheet OR written OR handwritten OR leaflet OR manual OR mail OR pamphlet OR paper OR sticker) in Intervention / treatment Limited to Interventional Studies
WHO International Clinical Trials Registry Platform (ICTRP)
reminder* [intervention]
Appendix 2. GRADE evidence profile: Reminders versus usual care
|
Patient or population: Healthcare professionals (physicians, nurses and dentists) Settings: Ambulatory and hospital care in USA, Canada, UK, France, Switzerland, Taiwan, Australia, Germany, Hong Kong, India, Israel, Spain and Thailand Intervention: Manual paper reminders focused on improving compliance with preventive guidelines (e.g. cancer screening, vaccination) and disease management guidelines (e.g. annual follow‐ups, test‐ordering, medication adjustment, counseling) Comparison: Control/usual care | |||||||||
| Quality assessment | Effect | Certainty of the evidence (GRADE) | Importance | ||||||
| Number of studies | Design | Risk of bias | Inconsistency | Indirectness | Imprecision | Other considerations | |||
| Professional practice: dichotomous process adherence outcomes | |||||||||
| 39 studies | 23 cluster‐randomised trials, 12 individual randomised trials, 4 non‐randomised trials | Serious (−1) | No serious inconsistency | No serious indirectness | No serious imprecision | None | Median improvement (IQR): 8.45% (2.54% to 20.58%) | ⊕⊕⊕⊝ moderate | IMPORTANT |
| Professional practice: continuous outcomes | |||||||||
| 8 studies | 5 cluster‐randomised trials, 3 individual randomised trials | Serious (−1) | Some inconsistency (−0.5) | No serious indirectness | Some imprecision (−0.5) | None | Median standardised mean difference (IQR): −0.002 (−0.02 to 0.01) | ⊕⊕⊝⊝ low | IMPORTANT |
| Patient outcomes dichotomous | |||||||||
| 7 studies | 6 cluster‐randomised trials, 1 individually‐randomised trial | Serious (−1) | Very serious (−2) |
No serious indirectness | No serious imprecision | None | Median improvement (IQR): 3.24% (2.31% to 4.12%) | ⊕⊝⊝⊝ very low | CRITICAL |
| Patient outcomes continuous | |||||||||
| 4 studies | 3 cluster‐randomised trials, 1 individually‐randomised trial | Some limitations | Very serious (‐2) | No serious indirectness | Serious imprecision (−1) | None | Median standardised mean difference (IQR): 0.001 (−0.002 to 0.11) | ⊕⊝⊝⊝ very low | CRITICAL |
| Adverse effects | |||||||||
| ‐ | ‐ | ‐ | ‐ | ‐ | ‐ | ‐ | Not reported | ‐ | IMPORTANT |
| Resource use | |||||||||
| 2 studies | 1 cluster‐randomised trial, 1 individually‐randomised trial | No serious | No serious inconsistency | Serious indirectness (−1) | Serious imprecision (−1) | None | Additional health service costs of GBP 65 and between EUR 41 and EUR 59 | ⊕⊕⊝⊝ low | IMPORTANT |
| GRADE Working Group grades of evidence High quality: Further research is very unlikely to change our confidence in the estimate of effect. Moderate quality: Further research is likely to have an important impact on our confidence in the estimate of effect and may change the estimate. Low quality: Further research is very likely to have an important impact on our confidence in the estimate of effect and is likely to change the estimate. Very low quality: We are very uncertain about the estimate. | |||||||||
Appendix 3. GRADE evidence profile: Reminders plus other QI intervention versus the same QI intervention
|
Patient or population: Healthcare professionals (physicians, nurses and dentists) Settings: Ambulatory and hospital care in USA, Canada, UK, France, Switzerland, Taiwan, Australia, Germany, Hong Kong, India, Israel, Spain and Thailand Intervention: Manual paper reminders added to other QI intervention (feedback, patient reminders, educational meetings, educational materials, test request forms) Comparison: other QI intervention ((feedback, patient reminders, educational meetings, educational materials, test request forms) | |||||||||
| Quality assessment | Effect | Certainty of the evidence (GRADE) | Importance | ||||||
| Number of studies | Design | Risk of bias | Inconsistency | Indirectness | Imprecision | Other considerations | |||
| Professional practice: dichotomous process adherence outcomes | |||||||||
| 12 studies | 7 cluster‐randomised trials, 5 individually‐randomised trials | Serious limitations (−1) | Some inconsistency (−0.5) | No serious indirectness | Some imprecision (−0.5) | None | Median improvement (IQR): 4.2% (IQR −1.1% to 5.5%) | ⊕⊕⊝⊝ low | IMPORTANT |
| Professional practice: continuous outcomes | |||||||||
| 2 studies | 1 cluster‐randomised trial, 1 individually‐randomised trial | No serious limitations | No serious inconsistency | No serious indirectness | Serious imprecision (−1) |
None | Median standardised mean difference (IQR): 0.28 (0.04 to 0.51) | ⊕⊕⊕⊝ moderate | IMPORTANT |
| Patient outcomes: dichotomous | |||||||||
| 2 studies | 1 cluster‐randomised trial, 1 individually‐randomised trial | No serious limitations | Serious inconsistency (−1) |
No serious indirectness | Serious imprecision (−1) |
None | Median improvement (IQR): −3.2% (−8.5% to 2.2%) | ⊕⊕⊝⊝ low | CRITICAL |
| Patient outcomes: continuous | |||||||||
| 1 study | 1 individually‐randomised trial | No serious limitations | Not applicable | No serious indirectness | Serious imprecision | None | Median standardised mean difference (IQR): 0.001 (−0.003 to 0.003) | ⊕⊕⊕⊝ moderate | CRITICAL |
| Adverse effects | |||||||||
| ‐ | ‐ | ‐ | ‐ | ‐ | ‐ | ‐ | Not reported | ‐ | IMPORTANT |
| Resource use | |||||||||
| 2 studies | 1 cluster‐randomised trial, 1 individually‐randomised trial | No serious | No serious inconsistency | Serious indirectness (−1) |
Serious imprecision (−1) |
None | Additional health service costs of GBP 30 and between EUR 16.5 and EUR 67 | ⊕⊕⊝⊝ low | IMPORTANT |
| GRADE Working Group grades of evidence High quality: Further research is very unlikely to change our confidence in the estimate of effect. Moderate quality: Further research is likely to have an important impact on our confidence in the estimate of effect and may change the estimate. Low quality: Further research is very likely to have an important impact on our confidence in the estimate of effect and is likely to change the estimate. Very low quality: We are very uncertain about the estimate. | |||||||||
Footnotes
Appendix 4. GRADE evidence profile: Reminders versus other QI intervention
|
Patient or population: Healthcare professionals (physicians, nurses and dentists) Settings: Ambulatory and hospital care in USA, Canada, UK, France, Switzerland, Taiwan, Australia, Germany, Hong Kong, India, Israel, Spain and Thailand Intervention: Manual paper reminders focused on improving compliance with preventive guidelines (e.g. cancer screening, vaccination) and disease management guidelines (e.g. annual follow‐ups, test‐ordering, medication adjustment, counseling) Comparison: Other QI interventions (patient reminders, computerised reminders, educational meeting or multifaceted interventions) | |||||||||
| Quality assessment | Effect | Certainty of the evidence (GRADE) | Importance | ||||||
| Number of studies | Design | Risk of bias | Inconsistency | Indirectness | Imprecision | Other considerations | |||
| Professional practice: dichotomous process adherence outcomes | |||||||||
| 14 studies | 7 cluster‐randomised trials, 7 individually randomised trials | serious limitations (−1) | No serious inconsistency | No serious indirectness | Serious imprecision (−1) | None | Median improvement (IQR): −7.9% (−11.0% to 0.7%) | ⊕⊕⊝⊝ low | IMPORTANT |
| Professional practice: continuous outcomes | |||||||||
| ‐ | ‐ | ‐ | ‐ | ‐ | ‐ | ‐ | Not reported | ‐ | IMPORTANT |
| Patient outcomes: dichotomous | |||||||||
| 3 studies | 3 cluster‐randomised trials | Serious limitations (−1) | Serious inconsistency (−1) | No serious indirectness | Serious imprecision (−1) | None | Median improvement (IQR): −2.08% (−17.95% to 2.28%) | ⊕⊝⊝⊝ very low | CRITICAL |
| Patient outcomes: continuous | |||||||||
| ‐ | ‐ | ‐ | ‐ | ‐ | ‐ | ‐ | Not reported | CRITICAL | |
| Adverse effects | |||||||||
| ‐ | ‐ | ‐ | ‐ | ‐ | ‐ | ‐ | Not reported | ‐ | IMPORTANT |
| Resource use | |||||||||
| 3 studies | 1 cluster‐randomised trial, 2 individually‐randomised trials | No serious limitations | No serious inconsistency | Serious indirectness (−1) | Serious imprecision (−1) | None | Additional health service costs of GBP 30 and between EUR 17 and EUR 55. The additional costs of maintenance were 78 cents (USD 1991) per patient per year | ⊕⊕⊝⊝ low | IMPORTANT |
| GRADE Working Group grades of evidence High quality: Further research is very unlikely to change our confidence in the estimate of effect. Moderate quality: Further research is likely to have an important impact on our confidence in the estimate of effect and may change the estimate. Low quality: Further research is very likely to have an important impact on our confidence in the estimate of effect and is likely to change the estimate. Very low quality: We are very uncertain about the estimate. | |||||||||
Characteristics of studies
Characteristics of included studies [ordered by study ID]
Ayanian 2008.
| Methods | Randomised trial | |
| Participants | Participants were 717 patients who had colorectal adenomas removed and their primary care physicians in 2 Massachusetts (USA) primary care networks | |
| Interventions | Patient‐specific reminders | |
| Outcomes | Primary: Proportion of patients receiving colonoscopy during the 6‐month observation period Secondary: Proportion of patients with a new adenoma or cancer detected Time: 6 months |
|
| Notes | 1 relevant comparison (reminders versus control) Funding: This study was supported by a grant (R21‐CA112365) from the National Cancer Institute |
|
| Risk of bias | ||
| Bias | Authors' judgement | Support for judgement |
| Random sequence generation (selection bias) | Low risk | Physicians were randomised through their patients |
| Allocation concealment (selection bias) | Unclear risk | No mention of this issue |
| Blinding of participants and personnel (performance bias) All outcomes | High risk | The randomisation was at patient level, so it was possible that a physician provided care for patients in both the control and intervention groups |
| Blinding of outcome assessment (detection bias) All outcomes | Low risk | Although the outcomes were objective, outcome assessors were blinded to the physician and patient's allocation |
| Incomplete outcome data (attrition bias) All outcomes | Low risk | No losses to follow‐up reported |
| Selective reporting (reporting bias) | Low risk | Unlikely |
| Other bias | High risk | Possible contamination in the case of physician providing care to patients in both the control and intervention groups |
Baker 1997.
| Methods | Cluster‐randomised trial | |
| Participants | 18 general practices in Leicestershire (UK). Participants were 1731 patients who had been taking a benzodiazepine anxiolytic or hypnotic drug for 4 weeks or longer | |
| Interventions | Intervention group: Reminders plus feedback Control group: Feedback alone |
|
| Outcomes | Entries in medical records indicating compliance with 5 criteria of care: assessment of suitability for withdrawal; being told about dependency; withdrawal being recommended; withdrawal or continuing medication; and a consultation with the general practitioner in the past year Assessed at: 12 months |
|
| Notes | 1 relevant comparison (reminders + another QI versus another QI alone) Funding: The study was funded by the Department of Health and Trent Regional Health Authority Directorate of Research and Development |
|
| Risk of bias | ||
| Bias | Authors' judgement | Support for judgement |
| Random sequence generation (selection bias) | Low risk | Quote: "After the first data collection, practices were randomly allocated, with a table of random numbers and without stratification..." |
| Allocation concealment (selection bias) | Unclear risk | No specific details given |
| Blinding of participants and personnel (performance bias) All outcomes | Low risk | Quote: "The practices were told that the study was an audit and to this extent were blinded to the trial of reminders." |
| Blinding of outcome assessment (detection bias) All outcomes | High risk | Quote: "We were unable to blind the second data collector to study groups" but the outcomes were relatively objective (from medical records) |
| Incomplete outcome data (attrition bias) All outcomes | Low risk | Only 2 practices (out of 20) withdrew from the study |
| Selective reporting (reporting bias) | Low risk | Not explicitly mentioned by authors, but it seems low risk |
| Other bias | Low risk | None identified |
Bankhead 2001.
| Methods | Randomised trial | |
| Participants | 13 general practices with low uptake in the second round of breast screening (below 60%) in north‐west London and the West Midlands, United Kingdom. Participants were 1158 women in these practices who were recent non‐attenders for breast screening | |
| Interventions | 4 arms: systematic intervention (general practitioner letter), an opportunistic intervention (flag in women’s notes prompting discussion by health professionals), neither intervention, or both | |
| Outcomes | 1) Attendance for screening 6 months after randomisation 2) Additional cost associated with: the number of additional attendances generated by the interventions (with the ORs derived from the models presented in this paper); any additional consultations (as a result of the letter); lengthier consultations (as a result of the flag); and the production and administrative processes associated with both interventions Assessed at: 6 months |
|
| Notes | 3 relevant comparisons (reminders versus control; reminders versus other QI (patient reminders) intervention; and reminders + other QI versus other QI) Funding: UK Medical Research Council |
|
| Risk of bias | ||
| Bias | Authors' judgement | Support for judgement |
| Random sequence generation (selection bias) | Low risk | Quote: "Randomisation schedules were produced separately for each practice with a random permuted block procedure based on random number tables by two members of the research team involved neither in assessing eligibility nor initiating the interventions." "Individual women were randomised into one of four groups in a factorial design" |
| Allocation concealment (selection bias) | Low risk | Quote: "Sealed envelopes and audited time sheets were used, so these schedules were only available to field workers after they had checked the patients’ notes for eligibility." |
| Blinding of participants and personnel (performance bias) All outcomes | Unclear risk | Participants and personnel were probably not blinded but the impact of this on routinely‐collected outcome data is limited |
| Blinding of outcome assessment (detection bias) All outcomes | Unclear risk | Outcome data seem to be routinely‐collected data |
| Incomplete outcome data (attrition bias) All outcomes | Low risk | Only 1% ‐ 1.5% of patients lost to follow‐up (Figure 1) |
| Selective reporting (reporting bias) | Low risk | Unlikely, although no protocol available |
| Other bias | Low risk | None clearly identified |
Beaulieu 2004.
| Methods | Cluster‐randomised trial | |
| Participants | A sample of 3293 Quebec physicians (Canada) | |
| Interventions | 3 arms: a 1‐page summary of clinical practice guidelines on the treatment of stable angina sent to physicians; the summary plus a reminder; and no intervention (control) | |
| Outcomes | Physicians’ prescribing practices for the treatment of stable angina pectoris Assessed at: 6 months |
|
| Notes | Included in the review but it was not possible to obtain raw data for analysis (only OR from multilevel logistic regression models) Funding: Health Canada. One of the authors received financial support from Aventis Pharma |
|
| Risk of bias | ||
| Bias | Authors' judgement | Support for judgement |
| Random sequence generation (selection bias) | Low risk | Quote: "The physicians identified in our previous study were randomly assigned, using computer‐generated random numbers, to one of three groups." |
| Allocation concealment (selection bias) | Unclear risk | Not mentioned |
| Blinding of participants and personnel (performance bias) All outcomes | Unclear risk | Unclear if physicians could be 'contaminated' by other interventions |
| Blinding of outcome assessment (detection bias) All outcomes | Unclear risk | Not mentioned |
| Incomplete outcome data (attrition bias) All outcomes | High risk | Quote: "Of the 3293 physicians in our initial study, 967 (29.4%) were not in the database in 1999, hence were considered lost to follow‐up." |
| Selective reporting (reporting bias) | Low risk | Unlikely |
| Other bias | Low risk | None identified |
Bishop 2006.
| Methods | Randomised trial | |
| Participants | 428 patients with acute mechanical low back pain and accepted Workers’ Compensation Board claims (British Columbia, Canada) | |
| Interventions | 3 arms: physician reminders, patient reminders, control | |
| Outcomes | Concordance with specific clinical practice guideline recommendation for acute low back pain Assessed at: up to 16 weeks |
|
| Notes | 1 relevant comparison (reminders versus control) | |
| Risk of bias | ||
| Bias | Authors' judgement | Support for judgement |
| Random sequence generation (selection bias) | Low risk | Quote: "A total of 462 family physicians and their patients were enrolled in the study and using a random number generator were randomly assigned to three separate study groups" |
| Allocation concealment (selection bias) | Unclear risk | Not mentioned |
| Blinding of participants and personnel (performance bias) All outcomes | Unclear risk | Participants and the personnel were not blinded to the allocation, but we are not sure about the impact of that on outcomes measurement |
| Blinding of outcome assessment (detection bias) All outcomes | Unclear risk | Not mentioned |
| Incomplete outcome data (attrition bias) All outcomes | Low risk | Only 7.3% of patients lost to follow‐up |
| Selective reporting (reporting bias) | Unclear risk | Unlikely |
| Other bias | Unclear risk | Unclear if there was some adjustment by physicians |
Boekeloo 2003.
| Methods | Randomised trial | |
| Participants | 447 patients aged 12 to 17 years who were seeing primary care providers for health checkups in 5 managed group care practices in Washington DC (USA) | |
| Interventions | 3 arms: usual care, patients reminders (priming), patient reminders + provider reminders (prompting) | |
| Outcomes | Adolescent‐provider communication Assessed at: unclear |
|
| Notes | 1 relevant comparison (reminders + other QI intervention versus other QI intervention) Funding: The study was supported by a grant from the National Institute on Alcohol Abuse and Alcoholism, Bethesda, Md, USA |
|
| Risk of bias | ||
| Bias | Authors' judgement | Support for judgement |
| Random sequence generation (selection bias) | Low risk | Quote: "Study group assignment was based on computer‐generated randomization, stratified by provider as well as adolescent age (12‐ 13, 14‐15, and 16‐17 years) and sex." |
| Allocation concealment (selection bias) | Low risk | Quote: "The principal investigator created sealed envelopes. In the provider’s office just before the health checkup and after administration of an intake questionnaire, the research assistant, who was blinded to the adolescent’s group assignment, opened the sealed envelope containing the adolescent’s random assignment." |
| Blinding of participants and personnel (performance bias) All outcomes | Low risk | From the point of view of participants both of the groups included in the comparison analysed received the same intervention. The prompt to providers was not blinded but we think that the risk of bias is still low |
| Blinding of outcome assessment (detection bias) All outcomes | High risk | Quote: "Interviewers were thus blinded to the adolescent’s group assignment for the previsit interview only." |
| Incomplete outcome data (attrition bias) All outcomes | Low risk | Data available for 294 out of 297 adolescents |
| Selective reporting (reporting bias) | Low risk | Not explicitly mentioned by authors, but it seems low risk |
| Other bias | Low risk | None identified |
Boltri 2007.
| Methods | Cluster‐randomised trial | |
| Participants | 1176 patients with at least 1 risk factor for diabetes in 10 practices from the Georgia‐Mercer Practice Based Research Network (USA) | |
| Interventions | 2 arms: nurse‐based prompt versus usual care | |
| Outcomes | Receiving fasting glucose screening, diet plan, exercise plan, and weight loss plan Assessed at: 3 months |
|
| Notes | 1 relevant comparison (reminders versus control) Funding: This project was supported in part by grants from the Medcen Foundation, Macon Georgia, the US Department of Health and Human Services, Health Resources and Services Administration (HRSA), Grant #5D12HP00159, and National Institutes of Health Grant #1 K07 HL04305‐01 |
|
| Risk of bias | ||
| Bias | Authors' judgement | Support for judgement |
| Random sequence generation (selection bias) | Unclear risk | Quote: "These 10 outpatient private primary care practices that included both internal medicine and family medicine were randomly assigned to either intervention or control groups". But the sequence generation procedure was unclear |
| Allocation concealment (selection bias) | High risk | Quote: "although this study was blinded to the practices, it was not blinded to the research assistants who recruited the subjects." |
| Blinding of participants and personnel (performance bias) All outcomes | Low risk | Quote: "None of the practices knew that the intervention was the nurse prompt" |
| Blinding of outcome assessment (detection bias) All outcomes | Low risk | Quote: "A different research assistant who was blinded to the site (intervention or control) entered all data" |
| Incomplete outcome data (attrition bias) All outcomes | Low risk | It seems that there were no losses to follow‐up |
| Selective reporting (reporting bias) | Unclear risk | Too many outcomes. Not clear which one was the primary |
| Other bias | High risk | There were baseline differences that potentially could impact on effect estimates Quote: "The proportions of women and Blacks were higher in the control group compared to the intervention group (p < 0.001)" "Although there were no significant differences in ADA risk scores, some differences were found between groups" |
Bouza 2004.
| Methods | Randomised trial | |
| Participants | 297 patients with blood stream infections | |
| Interventions | 3 arms: physicians receiving a conventional report, physicians receiving a conventional report and a written alert on the chart with clinical advice, physicians receiving the above plus oral clinical advice | |
| Outcomes | Adequacy of antibiotic therapy, mortality rates Assessed at: unclear |
|
| Notes | 1 relevant comparison (reminders versus control) Funding: Red Espanola de Investigacion en Patologia Infecciosa and the research project of the Fondo de Investigaciones Sanitarias of Spain |
|
| Risk of bias | ||
| Bias | Authors' judgement | Support for judgement |
| Random sequence generation (selection bias) | Low risk | Quote: "We randomly classified the patients with significant episodes of bacteremia into 3 different groups by means of a computer‐assisted random list." |
| Allocation concealment (selection bias) | Unclear risk | Not mentioned |
| Blinding of participants and personnel (performance bias) All outcomes | Unclear risk | Personnel are aware of the intervention and participants (patients) could be unaware, but no information provided. Not sure how this could bias effect sizes |
| Blinding of outcome assessment (detection bias) All outcomes | Unclear risk | Not mentioned |
| Incomplete outcome data (attrition bias) All outcomes | Low risk | No losses to follow‐up reported |
| Selective reporting (reporting bias) | Unclear risk | It seems unlikely, but no protocol available |
| Other bias | High risk | There were differences by services. An adjustment by cluster would had been useful |
Bray 2002.
| Methods | Cluster‐randomised trial | |
| Participants | 12 hospitals across 4 areas in India (during the trial 56,171 units were transfused) | |
| Interventions | Self‐educating transfusion request form | |
| Outcomes | Number of transfusions requests by admission, number of crossmatched per admission, number of units per admission Assessed at: 5 months |
|
| Notes | 1 relevant comparison (reminders versus control) Funding: UK Department for International Development. The UK Medical Research Council |
|
| Risk of bias | ||
| Bias | Authors' judgement | Support for judgement |
| Random sequence generation (selection bias) | Unclear risk | The sequence generation procedure is not clearly established |
| Allocation concealment (selection bias) | Unclear risk | Not mentioned |
| Blinding of participants and personnel (performance bias) All outcomes | Unclear risk | Personnel were aware of the group to which their hospital was allocated but we were not sure about the impact of this on effect estimates |
| Blinding of outcome assessment (detection bias) All outcomes | Unclear risk | Not mentioned |
| Incomplete outcome data (attrition bias) All outcomes | Unclear risk | No information about losses to follow‐up reported |
| Selective reporting (reporting bias) | Unclear risk | Quote: "The main process measure was the number of transfusion requests per in‐patient admission. Information on the number of crossmatches per admission was also collected. The main outcome measures were proportion of in‐patient admissions resulting in transfusion and the number of units transfused" Not sure why authors used these outcomes |
| Other bias | Low risk | None identified |
Buchsbaum 1993.
| Methods | Cluster‐randomised trial | |
| Participants | 83 residents at the ambulatory medicine clinic of the Medical College of Virginia, Richmond (USA) and their 214 patients with harmful drinking habits | |
| Interventions | A letter attached to the front of the patient's medical record with diagnostic information and treatment recommendations | |
| Outcomes | Rates of physician counselling Assessed at: immediately after patient's session |
|
| Notes | 1 relevant comparison (reminders versus control) Funding: This research was supported by a grant for Residency Training in General Internal Medicine, Bureau of Health Professions, Health Resources and Services Administration, Washington, DC, and by grant R01‐AA08278‐02, Improving Physician Management of Alcohol Disorders, National Institute of Alcohol Abuse and Alcoholism, Bethesda, Md |
|
| Risk of bias | ||
| Bias | Authors' judgement | Support for judgement |
| Random sequence generation (selection bias) | Unclear risk | There is no mention of how the sequence was generated |
| Allocation concealment (selection bias) | Unclear risk | Not mentioned |
| Blinding of participants and personnel (performance bias) All outcomes | Unclear risk | Participants were not blinded and it is not clear if patients were blinded |
| Blinding of outcome assessment (detection bias) All outcomes | Unclear risk | No specific information about this |
| Incomplete outcome data (attrition bias) All outcomes | Low risk | No loss to follow‐up reported |
| Selective reporting (reporting bias) | Unclear risk | No specific information reported |
| Other bias | Low risk | None identified |
Burns 2002.
| Methods | Non‐randomised trial | |
| Participants | 997 paediatric patients attending 2 inner‐city primary‐care health centres in urban Pittsburgh (USA) | |
| Interventions | A nurse‐prepared 1‐page form about the child’s immunisation status attached to children's charts (chart prompt) | |
| Outcomes | Up‐to‐date immunisation for age (including an overall outcome and 7 vaccine‐specific outcomes) Assessed at: 12 months |
|
| Notes | 1 relevant comparison (reminders versus control) Funding: the Aetna Foundation |
|
| Risk of bias | ||
| Bias | Authors' judgement | Support for judgement |
| Random sequence generation (selection bias) | High risk | Non randomised sequence Quote: "Each patient chart was systematically assigned to the control or intervention group based on the chart number. A random‐number table was used to generate the following scheme: children with a chart number ending in 0, 1, 2, 5, or 6 were assigned to the intervention group, and children with a chart number ending in 3, 4, 7, 8, or 9 were assigned to the control group." |
| Allocation concealment (selection bias) | High risk | No information reported about this, but because of the non‐random sequence the risk of bias is probably high |
| Blinding of participants and personnel (performance bias) All outcomes | Unclear risk | Participants and personnel were not blinded but unclear about the impact of this in effect estimates |
| Blinding of outcome assessment (detection bias) All outcomes | Low risk | Quote: "The chart reviewer was blinded as to intervention or control status" |
| Incomplete outcome data (attrition bias) All outcomes | Unclear risk | No specific information about this. It seems that there was complete follow‐up |
| Selective reporting (reporting bias) | Low risk | Most of the relevant outcomes included |
| Other bias | High risk | Contamination possible |
Cannon 2000.
| Methods | Randomised trial | |
| Participants | 78 outpatients in a mental health clinic in Salt Lake City, Utah, USA | |
| Interventions | A 3‐page paper checklist inserted into the medical record; a computer reminder system (CaseWalker) | |
| Outcomes | Percentage of cases screened for mood disorders, percentage of cases in which MDD criteria were documented Assessed at: unclear |
|
| Notes | 1 relevant comparison (reminders versus other QI (computerised reminder) intervention) Funding: not reported |
|
| Risk of bias | ||
| Bias | Authors' judgement | Support for judgement |
| Random sequence generation (selection bias) | Low risk | Quote: "randomization was based on a table of random numbers." |
| Allocation concealment (selection bias) | Unclear risk | Not reported |
| Blinding of participants and personnel (performance bias) All outcomes | High risk | Clinicians were not blinded and they could potentially see patients in both arms of the trial |
| Blinding of outcome assessment (detection bias) All outcomes | Unclear risk | Not reported |
| Incomplete outcome data (attrition bias) All outcomes | Low risk | 5 out of 83 patients lost to follow‐up (6%) |
| Selective reporting (reporting bias) | Low risk | The relevant outcomes at which the trial was aimed are reported |
| Other bias | High risk | High risk of contamination |
Chadha 2000.
| Methods | Randomised trial | |
| Participants | 497 women with menorrhagia and 449 women with urinary incontinence seen in gynaecology units in 4 district general hospitals across Scotland | |
| Interventions | Reminder (a disease‐specific reminder comprising a single A4 sheet with the protocol algorithm on one side and brief reference notes on the other) + educational meetings versus educational meetings alone | |
| Outcomes | Process of care within 6 key areas of clinical practice: initial hospital assessment (score of compliance with guidelines, continuous), appropriate use of hospital investigations, inappropriate use of hospital investigations, appropriate first line treatments, appropriate pre‐surgery assessment, and use of surgical treatments
Outcome of care using condition‐specific outcome measures and 4 domains of SF‐36 , and proportion of patients with no periods Assessed at: 6 and 12 months |
|
| Notes | 1 relevant comparison (reminders plus other QI intervention versus other QI intervention) Funding: Grant support for this study was provided by the Chief Scientist Office of the Scottish Office of Home and Health Department, which also funds the Health Services Research Unit, University of Aberdeen, Scotland |
|
| Risk of bias | ||
| Bias | Authors' judgement | Support for judgement |
| Random sequence generation (selection bias) | Low risk | The 2 conditions were allocated at random, stratified by size and location of hospital‐menorrhagia for the 2 hospitals in the west, and urinary incontinence for the 2 in the east of Scotland |
| Allocation concealment (selection bias) | Unclear risk | Not reported |
| Blinding of participants and personnel (performance bias) All outcomes | Unclear risk | Participants were not blinded but unsure about the impact of this in effect estimates |
| Blinding of outcome assessment (detection bias) All outcomes | Unclear risk | It is unclear if the local research assistants were blinded: Quote: "The local research assistant at each hospital abstracted process of care data (details of history, examination, investigation, diagnosis and treatment) from the hospital casenotes onto condition specific data abstraction forms 12 months after date of first consultation" |
| Incomplete outcome data (attrition bias) All outcomes | Low risk | Data available for 94% of eligible women: Quote: "Of these 946 women, 50 did not attend their first clinic appointment, hospital casenotes were not traceable for six, and two referred with urinary incontinence died (from other causes), leaving data available on 888 (94%) women (472 with menorrhagia and 416 with urinary incontinence) for the analysis of process of care" |
| Selective reporting (reporting bias) | Low risk | No specific information reported but unlikely |
| Other bias | High risk | Risk of contamination |
Chan 2002.
| Methods | Cluster‐randomised trial (cross‐over) | |
| Participants | All Washington State physiatrists billing the Medicare program (USA) and 4300 Medicare outpatients seen in Washington State in 1997 | |
| Interventions | Mailings reminding physicians to have their patients immunised against influenza | |
| Outcomes | Influenza immunisation rates Assessed at: 12 months |
|
| Notes | 1 relevant comparison (reminders versus control). Because of the risk of carry‐over effects, we only included data for the first period Funding: not reported |
|
| Risk of bias | ||
| Bias | Authors' judgement | Support for judgement |
| Random sequence generation (selection bias) | Unclear risk | The sequence generation procedure was not reported Quote: "In 1997, the solo practitioners (n 44) and the practitioner groups (n 13) were separately randomized to receive either 4 separate monthly mailings during the influenza season or nothing" |
| Allocation concealment (selection bias) | Unclear risk | Not reported |
| Blinding of participants and personnel (performance bias) All outcomes | Unclear risk | Not reported |
| Blinding of outcome assessment (detection bias) All outcomes | Low risk | Not reported but unlikely because outcome was routinely‐collected data |
| Incomplete outcome data (attrition bias) All outcomes | Unclear risk | No details reported |
| Selective reporting (reporting bias) | Low risk | Unlikely, the outcome seems to be the relevant one in this field |
| Other bias | High risk | No comments about carry‐over effects |
Cheney 1987.
| Methods | Cluster‐randomised trial | |
| Participants | The population studied comprised the 75 members (house officers) of the University of California, San Diego house staff in Internal Medicine (USA) and 200 randomly‐selected records | |
| Interventions | Checklist with preventive care recommendations attached to the medical record | |
| Outcomes | Rates of compliance with preventive care recommendations Assessed at: 12 months |
|
| Notes | Included in the review but not included in the analysis. Number of participants per arm not reported Funding: not reported |
|
| Risk of bias | ||
| Bias | Authors' judgement | Support for judgement |
| Random sequence generation (selection bias) | Unclear risk | No details about the randomisation procedure are provided: Quote: "...randomly assigned to control and experimental groups by postgraduate year and by traditional versus primary care residency track to ensure that house officers..." |
| Allocation concealment (selection bias) | Unclear risk | Not reported |
| Blinding of participants and personnel (performance bias) All outcomes | Unclear risk | Participants were not blinded but unclear about the impact of this in effect estimates |
| Blinding of outcome assessment (detection bias) All outcomes | Unclear risk | Not reported |
| Incomplete outcome data (attrition bias) All outcomes | Unclear risk | Authors audited 200 medical records to measure the outcome but they did not report how many of them were finally included in the analysis |
| Selective reporting (reporting bias) | Low risk | All relevant outcomes seem to be included |
| Other bias | Low risk | No other source of bias identified |
Cibere 2002.
| Methods | Non‐randomised cluster trial (cross‐over) | |
| Participants | 7 rheumatologists from Saskatoon (Canada) and 82 patients at high risk of gastrointestinal bleeding | |
| Interventions | Reminder sheet | |
| Outcomes | Adherence to guidelines in high‐risk NSAID users Assessed at: unclear (12 months?) |
|
| Notes | 1 relevant comparison (reminders versus control). Because of the risk of carry‐over effects, we only included data for the first period Funding: Canadian Institutes of Health Research Clinician Scientist |
|
| Risk of bias | ||
| Bias | Authors' judgement | Support for judgement |
| Random sequence generation (selection bias) | High risk | Non randomised |
| Allocation concealment (selection bias) | High risk | There was no clear concealment of non‐randomised allocation |
| Blinding of participants and personnel (performance bias) All outcomes | Unclear risk | Subjective outcome, probably participants and personnel were unblinded but unclear about the impact of this in effect estimates |
| Blinding of outcome assessment (detection bias) All outcomes | High risk | Because the main outcome is a judgement about guidelines compliance it is likely that the unblinding could have an impact on the effect estimates: Quote: "Charts of all patients that met the inclusion criteria were reviewed by one investigator (JC), who was not blinded to the study intervention" |
| Incomplete outcome data (attrition bias) All outcomes | High risk | We assume that there were 82 high‐risk patients, but data for the first part of the study are available only from 20 patients and for the second part from 17 patients |
| Selective reporting (reporting bias) | Low risk | Unlikely |
| Other bias | High risk | There is a high risk of carry‐over effects |
Clark 2009.
| Methods | Randomised trial | |
| Participants | 256 women attending the High Risk Obstetrical Unit at the Ottawa Hospital (Canada) | |
| Interventions | 4 groups: (1) reminders sent to both physician and patient, (2) reminders sent to the physician but not to the patient, (3) reminders sent to the patient but not to the physician, or (4) no reminder sent (usual care). | |
| Outcomes | Primary: proportion of patients who were screened for diabetes mellitus with an OGTT within 1 year after delivery Secondary: performance of other post‐partum screening tests Assessed at: 12 months |
|
| Notes | 3 relevant comparisons (reminders vs usual care; reminder + other QI versus other QI; and reminders vs other QI (patient reminders) intervention) Funding: provided by the Canadian Institute of Health Research, Knowledge Translation and Exchange (CIHR‐KTE), Grant 200210KTS (H.D.C.). |
|
| Risk of bias | ||
| Bias | Authors' judgement | Support for judgement |
| Random sequence generation (selection bias) | Low risk | Quote: "Randomization was performed with a computer‐generated randomization list" |
| Allocation concealment (selection bias) | Unclear risk | Not reported |
| Blinding of participants and personnel (performance bias) All outcomes | Low risk | Objective outcome (although personnel were not blinded). Participants (patients) were probably blinded |
| Blinding of outcome assessment (detection bias) All outcomes | Low risk | The investigators and statistician were blinded to group allocation because the patients were not seen routinely after delivery in follow‐up |
| Incomplete outcome data (attrition bias) All outcomes | Low risk | Data available from 223 out of 256 patients (87%) |
| Selective reporting (reporting bias) | Low risk | Unlikely. They reported a set of relevant outcomes |
| Other bias | Low risk | No additional source identified |
Cohen 1989a.
| Methods | Cluster‐randomised trial | |
| Participants | 50 dentists in private practice in the Indianopolis area (USA) and their patients (1027) | |
| Interventions | 4 groups: reminder, nicotine gum, nicotine gum + reminder, and control | |
| Outcomes | Smoking cessation rates at 6 and 12 months Assessed at: 6 and 12 months |
|
| Notes | Included in the review but not included in the analysis. Number of participants per group not reported Funding: This research was supported by a grant from the National Cancer Institute, Nº PHS1ROI CA38337 |
|
| Risk of bias | ||
| Bias | Authors' judgement | Support for judgement |
| Random sequence generation (selection bias) | Unclear risk | Quote: "dentists and their entire panel of patients who smoked cigarettes were randomly assigned to.." but the specific randomisation procedure is not described |
| Allocation concealment (selection bias) | Unclear risk | Not reported |
| Blinding of participants and personnel (performance bias) All outcomes | Unclear risk | Personnel seem to be unblinded but it is unclear if participants were or not (in this case the blinding could potentially impact on effect estimates) |
| Blinding of outcome assessment (detection bias) All outcomes | Unclear risk | Not reported |
| Incomplete outcome data (attrition bias) All outcomes | High risk | Although the assumption was that the non‐returners were smoking, 58% and 64% were lost to follow‐up at 6 and 12 months respectively |
| Selective reporting (reporting bias) | Low risk | The outcomes are relevant and it does not seem that important outcomes were missed |
| Other bias | Low risk | No other relevant source identified |
Cohen 1989b.
| Methods | Cluster‐randomised controlled trial | |
| Participants | 97 residents in internal medicine and 15 faculty general internists who staffed the outpatient general medicine clinic of a city county teaching‐hospital in Indianapolis (USA) and their patients (1420) | |
| Interventions | 4 groups: control, gum, reminders, gum + reminders | |
| Outcomes | Smoking cessation rates at 6 and 12 months Assessed at: 6 and 12 months |
|
| Notes | Included in the review but not included in the analysis. Number of participants per group not reported Funding:The National Cancer Institute |
|
| Risk of bias | ||
| Bias | Authors' judgement | Support for judgement |
| Random sequence generation (selection bias) | Unclear risk | Quote: "Before the lecture, physicians and their entire panel of patients who smoked cigarettes were randomly assigned...", but the specific randomisation procedure was not described |
| Allocation concealment (selection bias) | Unclear risk | Not reported |
| Blinding of participants and personnel (performance bias) All outcomes | Unclear risk | Participants and personnel probably not blinded, and it is uncertain if this could have an impact on effect estimates |
| Blinding of outcome assessment (detection bias) All outcomes | Unclear risk | Not reported |
| Incomplete outcome data (attrition bias) All outcomes | High risk | Although the results are presented for the whole group on the assumption that non‐returners are still smokers, the returners were only 895 out of 1420 (63%) |
| Selective reporting (reporting bias) | Low risk | The relevant outcomes have been reported |
| Other bias | Unclear risk | No other source identified |
Cowan 1992.
| Methods | Cluster‐randomised trial | |
| Participants | 29 residents staffing a general medical clinic of the University of Illinois Medical Center (USA). Data were assessed from a sample of 107 patients' charts | |
| Interventions | Periodic health examination fact sheet on the front of every patient's chart (with age‐ and sex‐specific recommendation on 7 periodic health examination actions) | |
| Outcomes | Compliance with periodic health examination actions Assessed at: unclear |
|
| Notes | 1 relevant comparison (reminders versus control) Funding: not reported |
|
| Risk of bias | ||
| Bias | Authors' judgement | Support for judgement |
| Random sequence generation (selection bias) | Unclear risk | Quote: "We randomly assigned one of these groups according to week (odd/even) to receive a periodic health examination fact sheet on the front of every patient's chart" |
| Allocation concealment (selection bias) | High risk | Because the allocation procedure is every other week, it is absolutely predictable |
| Blinding of participants and personnel (performance bias) All outcomes | Unclear risk | The effect of the unblinding of participants and personnel on the effect estimates was not clear |
| Blinding of outcome assessment (detection bias) All outcomes | Low risk | Quote: "One of the two investigators blinded to the resident group assignment reviewed a random sample of 107 charts from a total of..." |
| Incomplete outcome data (attrition bias) All outcomes | Unclear risk | Not reported |
| Selective reporting (reporting bias) | Unclear risk | Although the 7 periodic health examination actions reported seems to be relevant, why they were chosen was not completely clear |
| Other bias | Unclear risk | It is not clear if clustering was considered in the analysis (they mentioned that "we reanalyzed the data using the resident as the unit of analysis, weighting each residents's performance equally regardless of the number of PHE actions for which patients were eligible") |
Daucourt 2003.
| Methods | Cluster‐randomised trial | |
| Participants | 67 wards in 6 hospitals in France (1412 orders for thyroid function tests ordered for 1306 patients) | |
| Interventions | The 2 guideline diffusion interventions tested were a memorandum pocket card (MPC, reminder) and a test request form (TRF, other QI). The MPC summarised the recommendations according to the various clinical or therapeutic situations requiring thyroid exploration. 4 diffusion strategies were compared: TRF MPC (dual intervention group), TRF (order form group), MPC (pocket card group), and a control group | |
| Outcomes | The main outcome measure of effectiveness was the Guideline Conformity Rate (GCR: proportion of thyroid function test ordering in accordance with the guidelines) Assessed at: Unclear |
|
| Notes | 3 relevant comparisons (reminders vs usual care; reminder + other QI versus other QI; and reminders vs other QI) Funding: Supported in part by the Agence Nationale de l’Accreditation et de l’Evaluation en Santé (ANAES) |
|
| Risk of bias | ||
| Bias | Authors' judgement | Support for judgement |
| Random sequence generation (selection bias) | Low risk | Quote: "Randomization was performed by the CCECQA using a random number table" |
| Allocation concealment (selection bias) | Unclear risk | Not reported |
| Blinding of participants and personnel (performance bias) All outcomes | Unclear risk | It seems not to be blinded but the impact of this in effect estimates was unclear |
| Blinding of outcome assessment (detection bias) All outcomes | Unclear risk | Quote: "The research assistant was not blind to the hypothesis but was an independent observer who did not have to judge whether the order for the test was appropriate or not." The effect of this lack of blinding is not clear |
| Incomplete outcome data (attrition bias) All outcomes | Low risk | Out of 1412 tests ordered there were only 37 patients (2,6%) in whom the prescriber status was unknown and 52 (3.68%) in which the indication of test ordering was unknown |
| Selective reporting (reporting bias) | Unclear risk | The rationale behind the selection of a single combined outcome was not clear |
| Other bias | Low risk | No other source identified Quote: "Calculation took into account clustering per ward and stratification per hospital size and activity." |
Dubey 2006.
| Methods | Cluster‐randomised trial | |
| Participants | 4 urban family practice clinics among 38 primary care physicians affiliated with the University of Toronto (Canada) (509 and 608 randomly‐selected chart pre‐ and post‐intervention respectively) | |
| Interventions | Sex‐specific preventive Care Checklist Forms with evidence‐based recommendations on preventive health services and documentation space for routine procedures such as physical examination | |
| Outcomes | Compliance with 13 preventive health manoeuvres Assessed at: 9 ‐ 10 months |
|
| Notes | 1 relevant comparison (reminders versus control) Funding: The PSI Foundation |
|
| Risk of bias | ||
| Bias | Authors' judgement | Support for judgement |
| Random sequence generation (selection bias) | Low risk | Clinics were randomised using a random‐number table |
| Allocation concealment (selection bias) | Unclear risk | Not reported |
| Blinding of participants and personnel (performance bias) All outcomes | Low risk | Personnel were not blinded but they were not aware that the forms were part of a study |
| Blinding of outcome assessment (detection bias) All outcomes | Unclear risk | The impact of unblinding on effect estimates is unclear |
| Incomplete outcome data (attrition bias) All outcomes | Low risk | Different samples for the pre‐intervention (509 out of 697 (73%) analysed) and post‐intervention periods (619 out of 672 (89%) analysed). With a focus on the post‐intervention period the risk seems to be low |
| Selective reporting (reporting bias) | Low risk | Outcome for 13 preselected preventive manoeuvres presented (it is unlikely that some relevant outcome had not been reported) |
| Other bias | High risk | Baseline differences |
Ely 2015.
| Methods | Cluster‐randomised trial | |
| Participants | 14 primary care physicians (100 patients) at the University of Iowa Hospitals and Clinics, in the emergency department and the family medicine same‐day access clinic (USA) | |
| Interventions | A differential diagnosis checklist (printed on 4 × 6 cards with 1 symptom per card. Each symptom included an average of 22 possible diagnoses (SD 10, range 8 – 59). Commonly‐missed diagnoses were marked with an asterisk. 'Don’t miss' diagnoses (those with potentially serious consequences if missed) were marked with an ace of spades) | |
| Outcomes | Frequency of diagnostic errors Assessed at: unclear |
|
| Notes | 1 relevant comparison (reminders versus control) Funding: not reported |
|
| Risk of bias | ||
| Bias | Authors' judgement | Support for judgement |
| Random sequence generation (selection bias) | Low risk | Quote: "Physicians were randomly assigned to usual care vs. checklist using a randomized block design." |
| Allocation concealment (selection bias) | Unclear risk | Not reported |
| Blinding of participants and personnel (performance bias) All outcomes | Unclear risk | Clinicians (and probably patients) were not blinded to the intervention and the impact of this on the effect estimates was unclear |
| Blinding of outcome assessment (detection bias) All outcomes | Low risk | Quote: "The two investigators then independently reviewed this file, blinded to physician identity and study arm to determine the existence of a meaningful discrepancy between the chart diagnosis and the final diagnosis." |
| Incomplete outcome data (attrition bias) All outcomes | Low risk | Only 3 out of 103 lost to follow‐up |
| Selective reporting (reporting bias) | Low risk | Although there is only 1 outcome it seems to be the relevant one |
| Other bias | Low risk | None identified |
Emslie 1993.
| Methods | Cluster‐randomised trial | |
| Participants | 82 general practices (200 couples) in Grampian region (UK) | |
| Interventions | A structured infertility management sheet (guidelines were embedded in the sheet). It consisted of 2 sheets of A4 paper printed on both sides. The front page gave a brief summary of the guidelines and the pointers towards early referral. When the booklet was opened the 2 inner pages were for details of the history and examination of the woman and the man. The final page was a reminder of investigations to perform and a record of the results | |
| Outcomes | Compliance with 14 guideline recommendations related to the couple's sexual history and investigations performed Assessed at: 8 months |
|
| Notes | 1 relevant comparison (reminders versus control) Funding: not reported |
|
| Risk of bias | ||
| Bias | Authors' judgement | Support for judgement |
| Random sequence generation (selection bias) | Unclear risk | No mention about the specific way in which sequence was generated: Quote: "The 82 participating practices were randomised to study and control groups stratified for practice location." |
| Allocation concealment (selection bias) | Unclear risk | Not reported |
| Blinding of participants and personnel (performance bias) All outcomes | Unclear risk | Personnel were clearly unblinded and for participants (patients) was unclear. The impact of this in the effect estimates was unclear |
| Blinding of outcome assessment (detection bias) All outcomes | Unclear risk | Not reported |
| Incomplete outcome data (attrition bias) All outcomes | Low risk | Not mentioned specifically, but the study aimed to recruit 100 couples in each group and they reported results for 100 couples in each group |
| Selective reporting (reporting bias) | Low risk | A comprehensive set of outcomes reported |
| Other bias | Unclear risk | It seems that clustering was not considered in the analysis, but we were not sure |
Etter 2000.
| Methods | Cluster‐randomised trial | |
| Participants | 393 private practitioners in Geneva, Switzerland | |
| Interventions | Physicians in the intervention group received a box containing 500 'Smoker' stickers (diameter, 22 mm) and a letter presenting arguments in favour of systematically labelling the smokers’ charts and of counselling smokers | |
| Outcomes | 7 self‐reported smoking prevention activities Assessed at: 1 month |
|
| Notes | 1 relevant comparison (reminders versus control) Funding: Health Authority of the Canton of Geneva |
|
| Risk of bias | ||
| Bias | Authors' judgement | Support for judgement |
| Random sequence generation (selection bias) | Unclear risk | The randomisation procedure is not reported: Quote: "The 542 physicians (58%) who returned the questionnaire were randomly assigned to receiving the intervention ( n 272) or not ( n 270)" |
| Allocation concealment (selection bias) | Unclear risk | Not reported |
| Blinding of participants and personnel (performance bias) All outcomes | Unclear risk | It is unclear if patients were aware of the intervention |
| Blinding of outcome assessment (detection bias) All outcomes | High risk | Not reported. This could be specially relevant for subjective outcomes (self‐reported by physicians) |
| Incomplete outcome data (attrition bias) All outcomes | High risk | Only 393 out of 542 randomised physicians (72.5%) returned the follow‐up questionnaire |
| Selective reporting (reporting bias) | High risk | Only subjective (self‐reported) outcome reported. No chart audit data |
| Other bias | Unclear risk | No mention about how clustering was considered in the analysis |
Frame 1994.
| Methods | Randomised trial | |
| Participants | Adult members of families in which at least 1 member had been seen in a rural, multiple‐office, non‐profit, fee‐for‐service family practice in Dansville (NY, USA) within the past 2 years (1665 patients) | |
| Interventions | Computer‐based tracking system compared with a manual flowchart‐based system | |
| Outcomes | 1) Provider compliance with a health maintenance protocol (11 manoeuvres) 2) Costs Assessed at: unclear |
|
| Notes | 1 relevant comparison (reminders versus other QI (computer‐generated reminders) intervention) Funding: This project was supported by grant No. HS06283 from the Agency for Health Care Policy and Research, Rockville, Md |
|
| Risk of bias | ||
| Bias | Authors' judgement | Support for judgement |
| Random sequence generation (selection bias) | Low risk | Quote: "Each of the 1008 families included in the study was randomly assigned to either the computerized (trial) or manual (control) group." Quote: "A randomized list of guarantor (roughly equivalent to the head of the household) numbers, distributed among the four participating offices in proportion to the number of active patients at each office, was generated.'" |
| Allocation concealment (selection bias) | Unclear risk | Not reported |
| Blinding of participants and personnel (performance bias) All outcomes | Unclear risk | Quote: "Thus, providers knew they were to use the computer‐based tracking system for red‐dot patients and the manual system for green‐dot patients but they were blinded to which patients were actually included in the study." Unclear if this type of blinding is enough to avoid an impact on effect estimates |
| Blinding of outcome assessment (detection bias) All outcomes | Unclear risk | Not reported |
| Incomplete outcome data (attrition bias) All outcomes | Low risk | Data available for 1469 (725 + 744 from Table 1) out of 1665 patients (88%) |
| Selective reporting (reporting bias) | Low risk | The reported outcomes seem to be a reasonable spectrum of the outcomes in this field |
| Other bias | Unclear risk | Possible risk of contamination between providers |
Gonzalez 1989.
| Methods | Non‐randomised trial | |
| Participants | 14 medical residents (159 patients) at the Medicine Clinic of the New Hanover Memorial Hospital in Wilmington, North Carolina (USA) | |
| Interventions | A screening checklist (procedures not done were previously marked by a nurse) attached to the front of patients' charts | |
| Outcomes | Compliance with 6 health‐promotion and disease‐prevention procedures Assessed at: 5 weeks |
|
| Notes | 1 relevant comparison (reminders versus control) Funding: not reported |
|
| Risk of bias | ||
| Bias | Authors' judgement | Support for judgement |
| Random sequence generation (selection bias) | High risk | Although allocation on alternate days is not randomised, the allocation of residents to a specific day was randomised |
| Allocation concealment (selection bias) | Unclear risk | Not reported |
| Blinding of participants and personnel (performance bias) All outcomes | Unclear risk | Personnel not blinded and this was unclear for participants. The impact of this on effect estimates was unclear |
| Blinding of outcome assessment (detection bias) All outcomes | High risk | Clearly not blinded and this could have an impact on effect estimates |
| Incomplete outcome data (attrition bias) All outcomes | Low risk | Losses to follow‐up seem to be minimal (out of 159 doctor‐patient encounters there seems to be information available from 157) |
| Selective reporting (reporting bias) | Low risk | It does not seems to be an issue |
| Other bias | High risk | No mention of a method to deal with clustering in the analysis |
Grady 1997.
| Methods | Cluster‐randomised trial | |
| Participants | 61 primary care practices (11,716 patients) in greater Dayton, Ohio and Springfield, Massachusetts (USA) | |
| Interventions | 3 arms: (1) education‐only control, (2) education plus cue enhancement, and (3) education plus cue enhancement plus feedback and token rewards | |
| Outcomes | Mean practice mammography referral and completion rates (continuous) Assessed at: 12 months |
|
| Notes | 1 relevant comparison (reminders plus other QI (education) versus other QI intervention) Funding: This research was supported by National Cancer Institute Grant CA58243 |
|
| Risk of bias | ||
| Bias | Authors' judgement | Support for judgement |
| Random sequence generation (selection bias) | Unclear risk | The specific randomisation procedure is not clearly described: Quote: "...was recruited and randomly assigned to one of three conditions: (1) education‐only control, (2) education plus cue enhancement, and (3) education plus cue enhancement plus feedback and token rewards." |
| Allocation concealment (selection bias) | Unclear risk | Not reported |
| Blinding of participants and personnel (performance bias) All outcomes | Unclear risk | Not reported |
| Blinding of outcome assessment (detection bias) All outcomes | Unclear risk | Not reported |
| Incomplete outcome data (attrition bias) All outcomes | Low risk | Quote: "A total of 11,716 patients were identified, and all but 290 (2.5%) who were documented to be lost because of death or transfer out of the practice completed the first year, resulting in a final sample of 11,426." |
| Selective reporting (reporting bias) | Low risk | Unlikely (relevant outcomes reported) |
| Other bias | Low risk | None clearly identified. No cluster adjustment, but the analysis was by cluster |
Halterman 2005.
| Methods | Randomised trial | |
| Participants | 151 3‐ to 7‐year‐old children (and their primary care physicians) entering the Rochester City School District (USA) | |
| Interventions | Primary care providers of children in the provider notification group were sent a letter by facsimile, signed by the principal investigator, indicating the number of days the child experienced daytime and night‐time symptoms during the past 4 weeks and the number of emergency medical visits for asthma during the past 12 months. The letter also included a copy of the 2002 NHLBI guidelines for asthma management and a recommendation to consider medication action based on the child’s current therapy For children assigned to the control group, PCPs were not contacted |
|
| Outcomes | Measures of preventive action taken (e.g. new medication or change in medication) and health care use (e.g. acute visits for asthma) Assessed at: 3 ‐ 6 months |
|
| Notes | 1 relevant comparison (reminders versus control) Funding: The Halcyon Hill Foundation, Rochester; the Robert Wood Johnson Foundation Generalist Physician Faculty Scholars Program |
|
| Risk of bias | ||
| Bias | Authors' judgement | Support for judgement |
| Random sequence generation (selection bias) | Low risk | Quote: "At the conclusion of the baseline interview, children were randomized into either the provider notification group or the control group. Randomization was stratified by use of preventive medications at baseline and was blocked in groups of 6. Randomization cards were made from a table of random numbers" |
| Allocation concealment (selection bias) | Low risk | Quote: "...were kept in sealed, opaque, sequentially numbered envelopes until after the baseline assessment was completed" |
| Blinding of participants and personnel (performance bias) All outcomes | Unclear risk | Unclear if participants were blinded (this could affect outcome measurement) |
| Blinding of outcome assessment (detection bias) All outcomes | Low risk | Quote: "Interviewers blinded to the child’s group assignment called parents 3 to 6 months after randomization to assess the outcome measures for this study." |
| Incomplete outcome data (attrition bias) All outcomes | Low risk | 150 out of 151 children included in the primary analysis (Figure) |
| Selective reporting (reporting bias) | Low risk | Unlikely, although only subjective outcomes reported |
| Other bias | Unclear risk | No adjustment by provider, and a risk of contamination |
Headrick 1992.
| Methods | Cluster‐randomised trial | |
| Participants | Academic group practices (33 residents, 240 patients) of a major urban teaching hospital in Cleveland, Ohio (USA) | |
| Interventions | 3 arms: PCEP (Physicians Cholesterol Education Program) lecture only (group 1, the control group); PCEP lecture supplemented by generic chart reminders of the NCEP (National Cholesterol Education Program) guidelines placed on top of each patient's medical record at the time of his/her visit (group 2, the generic chart reminder group); and the PCEP lecture supplemented by the generic reminders and patient‐specific feedback about current compliance with NCEP guidelines and explicit recommendations about immediate actions to be taken (group 3, the patient‐specific chart reminder group) | |
| Outcomes | Compliance with NCEP guidelines Assessed at: 5 weeks |
|
| Notes | 1 relevant comparison (reminders versus control). Groups 2 and 3 were combined in the analysis (generic & specific reminders) Funding: Merck, Sharp and Dohme, administered by the Clinical Research International Research Triangle |
|
| Risk of bias | ||
| Bias | Authors' judgement | Support for judgement |
| Random sequence generation (selection bias) | Unclear risk | The procedure for generating the sequence is not explicitly described and the different descriptions of what was done were misleading: Quote: "Interventions were assigned to each geographically separated resident group practice." Quote: "The investigation was conducted as a randomized controlled trial of three strategies to improve resident physicians' compliance with published NCEP guidelines for identifying and treating patients with high blood cholesterol levels" |
| Allocation concealment (selection bias) | Unclear risk | Not reported |
| Blinding of participants and personnel (performance bias) All outcomes | Unclear risk | It was unclear if patients were blinded, and the impact of this on effect estimates is uncertain |
| Blinding of outcome assessment (detection bias) All outcomes | Low risk | Quote: "These notes were reviewed blindly by a trained nurse‐reviewer who recorded all lipid testing or management activity (drug therapy initiated or modified, dietary recommendations, or consultations requested), including any documentation of physician intent to undertake these actions." |
| Incomplete outcome data (attrition bias) All outcomes | Unclear risk | Not reported |
| Selective reporting (reporting bias) | Unclear risk | The outcomes reported were a mix of objective (compliance with guidelines) and subjective (knowledge and attitudes) |
| Other bias | Unclear risk | There was no adjustment by clustering |
Hung 2008.
| Methods | Randomised trial | |
| Participants | 194 patients with angiographically‐proven CHD at National Taiwan University Hospital Yun‐Lin Branch (Taiwan) | |
| Interventions | A statement stamped on the paper chart ('Statin can be beneficial to the patients with documented coronary artery disease regardless of their LDL level’), red in colour, written in mixed Chinese and English. Beneath the reminder, the current policy of statin reimbursement issued by National Health Insurance (NHI) of Taiwan was attached | |
| Outcomes | Primary: new prescription of either statin or ezetimibe during the follow‐up period Secondary: the composite result of LLT or lipid profile measurement within 6 months Assessed at: 6 months |
|
| Notes | 1 relevant comparison (reminders versus control) Funding: not reported |
|
| Risk of bias | ||
| Bias | Authors' judgement | Support for judgement |
| Random sequence generation (selection bias) | Low risk | Quote: "The subjects with proven significant CHD without statin use were randomized into two groups on a patient‐by‐patient basis according to the assignment from a computer‐based algorithm." |
| Allocation concealment (selection bias) | Unclear risk | Not reported |
| Blinding of participants and personnel (performance bias) All outcomes | Unclear risk | It was not clear if patients were blinded |
| Blinding of outcome assessment (detection bias) All outcomes | Unclear risk | Not reported |
| Incomplete outcome data (attrition bias) All outcomes | Low risk | No losses to follow‐up reported |
| Selective reporting (reporting bias) | Low risk | The relevant outcomes were considered and reported |
| Other bias | High risk | Authors used logistic regression, but unclear if they controlled by clustering. Some risk of contamination |
Kunz 2007.
| Methods | Cluster‐randomised controlled trial | |
| Participants | 178 practices (417 patients' discharge letter) in the catchment area of the Department of Internal Medicine, Park‐Klinik Weissensee, Berlin (Germany) | |
| Interventions | Short, 1‐sentence evidence summaries appended to the discharge letters (59 different pieces of evidence addressing 15 medical conditions) | |
| Outcomes | Primary: rate of discontinuation of recommended medication (process of care continuous) Assessed at: unclear |
|
| Notes | 1 relevant comparison (reminders versus control) Funding: The study was performed as part of a joint project of the Arztekammer Berlin, Park‐Klinik Weissensee and the Techniker Krankenkasse, Hamburg |
|
| Risk of bias | ||
| Bias | Authors' judgement | Support for judgement |
| Random sequence generation (selection bias) | Low risk | Quote: "Existing practices were randomised using a computer generated random list before establishing the practitioners’ willingness to participate." Quote: "We randomised all primary care practices in the catchment area of the Department of Internal Medicine, Park‐Klinik Weissensee, Berlin, from where patients had been admitted in the preceding year" |
| Allocation concealment (selection bias) | Low risk | Quote: "For practices that opened during the study period, we prepared opaque sealed envelopes that the department secretary opened sequentially." |
| Blinding of participants and personnel (performance bias) All outcomes | High risk | Participants not clearly blinded and the outcome was reported by them |
| Blinding of outcome assessment (detection bias) All outcomes | High risk | They were unblinded but some precautions were taken to minimise bias (unclear if they were effective) |
| Incomplete outcome data (attrition bias) All outcomes | High risk | Outcome data obtained from 122 out of 469 randomised practices (26%). Although they identified some imbalance between control and intervention groups, the risk of bias seems to be high, considering the very high losses to follow‐up |
| Selective reporting (reporting bias) | Low risk | Unlikely |
| Other bias | Low risk | Clustering considered in the analysis. No other source identified |
Laprise 2009.
| Methods | Cluster‐randomised trial | |
| Participants | 122 general practitioners (2344 patients) in 5 regions of Quebec (Canada) | |
| Interventions | Continuous Medical Education (CME) only (control group) compared with CME with practice enablers and reinforces (PER intervention: screening medical records, prompting physicians, filling out and enclosing a 1‐page checklist) | |
| Outcomes | Primary: proportion of patients, undermanaged at baseline for at least 1 recommendation, for which study physicians undertook at least 1 preventive care action in the first visit following patients’ recruitment in the study Secondary: compliance with guideline recommendations (7) Assessed at: variable |
|
| Notes | 1 relevant comparison (reminders plus other QI (educational meetings) compared with other QI intervention) Funding: Sanofi‐Aventis provided an unrestricted research grant to the University of Montreal |
|
| Risk of bias | ||
| Bias | Authors' judgement | Support for judgement |
| Random sequence generation (selection bias) | Low risk | Quote: "Using a computer‐generated list of random numbers ˜MS Excel: RANDOM!, half of the GPs within each stratum were randomly assigned by one investigator to the PER group and the other half to the control group." |
| Allocation concealment (selection bias) | Unclear risk | Not reported |
| Blinding of participants and personnel (performance bias) All outcomes | Unclear risk | Most of the participants were probably blinded to the intervention, but this is not completely clear |
| Blinding of outcome assessment (detection bias) All outcomes | Unclear risk | Quote: "Outcomes were assessed by the investigators, using retrospective audit information abstracted by trained nurses." Probably unblinded but it was not clear |
| Incomplete outcome data (attrition bias) All outcomes | Unclear risk | 122 out of 133 (91.7%) GPs initially randomised were included in the analysis. However only 2344 out of 4488 (52.2%) of the patients were included in the analysis (low consent rates?) |
| Selective reporting (reporting bias) | Low risk | A wide range of relevant outcomes included |
| Other bias | Low risk | An appropriate analysis considering clustering: Quote: "...using logistic regressions, within the framework of a generalized estimating equation." |
Levine 2003.
| Methods | Randomised trial | |
| Participants | 473 patients attending Internal Medicine and Family Medicine clinics in Nashville, Tennessee (USA) | |
| Interventions | A physician‐administered model that used reminders sent by the nurse (reminders) versus a nursing protocol model (other QI intervention). The nurse‐administered protocol‐driven preventive services delivery model was a kind of multifaceted intervention | |
| Outcomes | Proportion of patients with 9 cancer‐related preventive services documented as ordered or completed Assessed at: 9 months |
|
| Notes | Cancer prevention services 1 relevant comparison (reminders versus other QI (multifaceted) intervention) Funding: The Agency for Healthcare Quality and Research and the Health Care Financing Administration |
|
| Risk of bias | ||
| Bias | Authors' judgement | Support for judgement |
| Random sequence generation (selection bias) | Unclear risk | No details about the sequence generation procedure |
| Allocation concealment (selection bias) | Unclear risk | Not reported |
| Blinding of participants and personnel (performance bias) All outcomes | Unclear risk | It was unclear if participants were blinded or not (they knew that were participating in a study). It was unclear if physicians knew they were participating in a study. |
| Blinding of outcome assessment (detection bias) All outcomes | High risk | Nurses (the outcome assessors) were not blinded to the patient allocation and they were part of 1 arm of the trial |
| Incomplete outcome data (attrition bias) All outcomes | Low risk | They did not report any loss to follow‐up |
| Selective reporting (reporting bias) | Low risk | Unlikely; a wide and relevant range of outcomes included |
| Other bias | Unclear risk | Quote: "...using physicians as the unit of analysis did not alter the conclusions." but the way in which this analysis was done is not reported. Contamination? adjustment by nurse? |
Levy 2009.
| Methods | Cluster‐randomised trial | |
| Participants | 5 practices (204 women aged 65 years or older) in the Iowa Research Network (USA) | |
| Interventions | 3 arms: physician chart reminders, physician chart reminders + patient education, usual care. The chart reminder was a sticky note following National Osteoporosis Foundation guidelines that practices could place on the charts wherever they thought it would be most effective. | |
| Outcomes | Primary: rates of completed BMD testing Secondary: percentage of women who asked their physician about a BMD test, percentage of women who discussed a BMD test with their physician Assessed at: 6.7 months (median) |
|
| Notes | Osteoporosis screening 1 relevant comparison (reminders versus control) Funding: Financial support was provided by the American Academy of Family Physicians Foundation through an Advanced Research Training Grant awarded to Barcey Levy and the Department of Family Medicine, University of Iowa Roy J. and Lucille A. Carver College of Medicine, Iowa City |
|
| Risk of bias | ||
| Bias | Authors' judgement | Support for judgement |
| Random sequence generation (selection bias) | Low risk | Quote: "Using a random number generator, the remaining 5 practices were randomized to one of 3 groups: 2 practices to physician chart reminder alone, 2 practices to physician chart reminder plus patient education, and 1 practice to usual care." |
| Allocation concealment (selection bias) | Unclear risk | Not reported |
| Blinding of participants and personnel (performance bias) All outcomes | Unclear risk | Personnel were unblinded and participants could be aware of the sticky note on their charts; the impact of this on effect estimates was unclear |
| Blinding of outcome assessment (detection bias) All outcomes | Unclear risk | Assessors were probably unblinded and the impact of this on effect estimates was unclear (objective primary outcome) |
| Incomplete outcome data (attrition bias) All outcomes | Low risk | Outcome reported for 195 out of 204 (95.6%) patients initially considered |
| Selective reporting (reporting bias) | Unclear risk | Only a single objective outcome |
| Other bias | Low risk | It seems that the analytical methods used (4 different approaches) considered clustering and unit‐of‐analysis errors |
Lilford 1992.
| Methods | Randomised trial | |
| Participants | 2424 women attending the hospital for the first (booking) visit in St James's University Hospital, Leeds (UK) | |
| Interventions | 3 arms (methods of history taking): manual system (usual care), structured checklist (manual reminder), and computerised (on‐screen reminders) | |
| Outcomes | Compliance with 101 clinical actions grouped in 7 medical categories: medical and surgical, obstetric, personal, current symptoms and treatment, related to maternal age, and 2 common actions that would have swamped all others if not grouped separately (carrying out a cervical smear test and giving advice on dental hygiene) Assessed at: unclear |
|
| Notes | Quality of obstetric care (history‐taking) 2 relevant comparisons (reminders versus usual care/control; reminders versus other QI (on‐screen reminders) intervention) Funding: This work was supported by the UK Medical Research Council |
|
| Risk of bias | ||
| Bias | Authors' judgement | Support for judgement |
| Random sequence generation (selection bias) | Low risk | Before each clinic a computer programme randomised women to 1 of the 3 groups in blocks of 3 |
| Allocation concealment (selection bias) | Unclear risk | Not reported |
| Blinding of participants and personnel (performance bias) All outcomes | Unclear risk | Personnel not blinded and participants probably blinded, but unclear |
| Blinding of outcome assessment (detection bias) All outcomes | Low risk | For the first 430 assessments of the notes the research midwives were blinded to the group allocation |
| Incomplete outcome data (attrition bias) All outcomes | Low risk | Outcome data from 2223 out of 2424 randomised patients (91.7%) |
| Selective reporting (reporting bias) | Unclear risk | The clinical actions were grouped, then it is unclear if all the 101 actions were reported |
| Other bias | Unclear risk | It is not clear how relevant each action is for each patient. It is not clear if there is a risk of contamination |
McIsaac 2002.
| Methods | Cluster‐randomised controlled trial | |
| Participants | 97 family physicians (164 randomised) in Ontario (Canada) seeing 621 children and adults with sore throat | |
| Interventions | Chart stickers or forms prompting physicians to use a score‐management approach for the diagnostic of sore throat; control group use a form but without either the sticker or the chart prompt | |
| Outcomes | Primary: prescription of unnecessary antibiotics Secondary: overall antibiotic use Assessed at: unclear |
|
| Notes | 1 relevant comparison (reminders versus control/usual care) Funding: Medical Research Council of Canada |
|
| Risk of bias | ||
| Bias | Authors' judgement | Support for judgement |
| Random sequence generation (selection bias) | Unclear risk | Physicians were randomised but the procedure was unclear |
| Allocation concealment (selection bias) | Unclear risk | Not reported |
| Blinding of participants and personnel (performance bias) All outcomes | Unclear risk | It was unclear if participants (patients) were blinded or not |
| Blinding of outcome assessment (detection bias) All outcomes | Unclear risk | It was not clear how the outcomes were measured |
| Incomplete outcome data (attrition bias) All outcomes | High risk | Only 97 out of 164 (59.1%) randomised physicians completed the study and provided patient data |
| Selective reporting (reporting bias) | Unclear risk | The outcomes seem relevant, but there was a lack of clinically relevant outcomes |
| Other bias | Low risk | Clustering considered in the design (sample size calculation) and in the analysis |
Montgomery 2000.
| Methods | Cluster‐randomised trial | |
| Participants | 27 general practices (614 patients) in Avon (UK) | |
| Interventions | 3 arms: computer‐based clinical decision support system plus cardiovascular risk chart; cardiovascular risk chart alone; and usual care | |
| Outcomes | Percentage of patients in each group with a 5‐year cardiovascular risk of 10% or more, systolic blood pressure, diastolic blood pressure, prescribing of cardiovascular drugs Assessed at: 6 ‐ 12 months |
|
| Notes | 1 relevant comparison (reminders versus control/usual care) Funding: NHS Wales Office of Research and Development |
|
| Risk of bias | ||
| Bias | Authors' judgement | Support for judgement |
| Random sequence generation (selection bias) | Low risk | Quote: "Randomisation was performed with a table of random numbers by a researcher not involved in the study and who was blind to the identity of the practices." |
| Allocation concealment (selection bias) | Unclear risk | Not reported |
| Blinding of participants and personnel (performance bias) All outcomes | Unclear risk | Quote: "...neither the doctors and nurses nor the patients were blind to their study group." The impact of this in effect estimates was unclear |
| Blinding of outcome assessment (detection bias) All outcomes | Unclear risk | Not reported |
| Incomplete outcome data (attrition bias) All outcomes | High risk | At 12 months outcome data were available for 531 out of 810 randomly selected patients (65.5%), but there were few practices lost to follow‐up |
| Selective reporting (reporting bias) | Unclear risk | Unlikely, but protocol not available and most of the outcomes were surrogate |
| Other bias | Low risk | Analysis addressed clustering and other confounding variables: Quote: "Since randomisation was by practice, we also corrected for clustering using procedures in Stata to derive robust estimates of standard error." No other source identified |
Moore 1997.
| Methods | Cluster‐randomised trial | |
| Participants | 26 community‐based office practices of internists and family physicians in Los Angeles, California (USA). 261 patients aged 70 years or more and seeing these physicians for a new visit or a physical examination | |
| Interventions | Screening form and clinical summaries for selected conditions (impairments in hearing, vision, and physical function, weight and memory loss, depression, incontinence, and gait disorders) | |
| Outcomes | Detection of and intervention for conditions screened (impairments in hearing, vision, and physical function, weight and memory loss, depression, incontinence, and gait disorders), and health status 6 months after the intervention Assessed at: 6 months |
|
| Notes | 1 relevant comparison (reminders versus control/usual care) Funding: Robert Wood Johnson Clinical Scholars Program and the National Institute on Aging Geriatric Academic Program |
|
| Risk of bias | ||
| Bias | Authors' judgement | Support for judgement |
| Random sequence generation (selection bias) | Unclear risk | The randomisation procedure was not clear: Quote: "Physician group practices that agreed to participate in the study were matched on specialty and one member of each of the matched pairs was randomized (using a table of random numbers) to the intervention or control group." |
| Allocation concealment (selection bias) | Unclear risk | Not reported |
| Blinding of participants and personnel (performance bias) All outcomes | Unclear risk | It is not clear if patients were blinded or not and the consequences of this on effect estimates |
| Blinding of outcome assessment (detection bias) All outcomes | Unclear risk | Not reported |
| Incomplete outcome data (attrition bias) All outcomes | Unclear risk | 26 out of 35 physician groups (74%) completed the study. Main outcome data at 6 months available from 238 out of 261 (91%) patients |
| Selective reporting (reporting bias) | Low risk | A wide range of relevant outcomes reported |
| Other bias | Unclear risk | Not sure if clustering was considered in the analysis using ANCOVA |
Nendaz 2010.
| Methods | Cluster‐randomised trial | |
| Participants | All eligible patients (1085 acutely ill hospitalised medical patients) from 8 randomly selected Swiss hospitals | |
| Interventions | 3 arms: pocket digital assistant programme (PDA, computerised reminder), pocket cards (manual reminder) and no CDSS (controls) | |
| Outcomes | Adequacy of thromboprophylaxis prescription Assessed at: 4 months |
|
| Notes | 2 relevant comparisons (reminders versus control/usual care; reminders versus other QI (computerised reminder) intervention) Funding: The study was funded by Sanofi‐Aventis (Suisse) |
|
| Risk of bias | ||
| Bias | Authors' judgement | Support for judgement |
| Random sequence generation (selection bias) | Unclear risk | The specific way in which the sequence was generated was not clear: Quote: "We randomly assigned medical services to a pocket digital assistant program (PDA), pocket cards (PC) and no CDSS (controls)." Quote: "The centres without electronic charts were randomly assigned to pocket cards (PC) or pocket digital assistants (PDA) providing the thromboembolic risk score, or received no CDSS (controls, C)." |
| Allocation concealment (selection bias) | Unclear risk | Not reported |
| Blinding of participants and personnel (performance bias) All outcomes | Low risk | Personnel were not blinded, and patients were probably blinded. Because of the setting it is unlikely that this could affect effect estimates |
| Blinding of outcome assessment (detection bias) All outcomes | Unclear risk | No mention of this (data extracted from a standardised form used by a multicentre study, but no explicit mention of who was extracting the data) |
| Incomplete outcome data (attrition bias) All outcomes | Unclear risk | All the randomised units (wards) were included in the analysis (sample of patients were different at baseline than in the follow‐up). The study included 1085 patients with 651 at baseline and 434 post‐CDSS implementation (66.7%) |
| Selective reporting (reporting bias) | Unclear risk | A single outcome, not clear if there were more outcomes reported in the protocol |
| Other bias | Low risk | The analytical plan seems to take clustering into account: Quote: "To take into account a cluster effect at the hospital level, we used a Generalized Estimating Equations (GEE) model with an exchangeable working correlation matrix to assess global adequacy of thromboprophylaxis prescription." No other source identified |
Potter 2009.
| Methods | Cluster‐randomised trial | |
| Participants | 5 primary care clinics (7303 patients) at the University of California, San Francisco (USA) | |
| Interventions | 3 arms: usual care (no intervention), poster‐only arm (reminder: a large colourful poster designed by the American Cancer Society presenting the menu of options for CRCS (Colorectal Cancer Screening) in a multilingual format placed in each exam room), and a poster/phone reminder arm | |
| Outcomes | Proportion of patients with up‐to‐date CRCS Assessed at: 6 to 9 months |
|
| Notes | Colorectal cancer screening 1 relevant comparison (reminders versus control/usual care) Funding: San Francisco Unit of the American Cancer Society, a Cancer Control Career Development Award for Primary Care Physicians from the American Cancer Society (MBP), and a Research ScholarsGrant from the American Cancer Society (JMEW) |
|
| Risk of bias | ||
| Bias | Authors' judgement | Support for judgement |
| Random sequence generation (selection bias) | Unclear risk | The sequence generation procedure is not explicitly described: Quote: "The 5 primary care practices were randomized to 1 of 3 arms..." |
| Allocation concealment (selection bias) | Unclear risk | Not reported |
| Blinding of participants and personnel (performance bias) All outcomes | Low risk | Although participants and personnel were not blinded we think that the impact on effect estimates was minimal, because of the way in which the study is set (non‐intrusive intervention and objective outcome) |
| Blinding of outcome assessment (detection bias) All outcomes | Unclear risk | It is not clear who extracted the data: Quote: "Patient data was extracted from the UCSF EMR." |
| Incomplete outcome data (attrition bias) All outcomes | Unclear risk | There is no mention about losses to follow‐up |
| Selective reporting (reporting bias) | Low risk | The selected outcome seems to be the relevant one |
| Other bias | Unclear risk | It was unclear if the analytical approach used by the authors addressed clustering |
Pritchard 1995.
| Methods | Randomised trial | |
| Participants | 757 women, aged from 36 to 69 years, selected from the registers of a university general practice at Lockridge, near Perth (Australia) | |
| Interventions | 4 arms: a control group (which received opportunistic screening as part of normal practice care); a ‘tagged notes’ group, for which each patient’s notes were tagged with a reminder for the treating doctor to invite the woman to have a Pap smear at a normal consultation or at the special screening clinic (reminders group); a ‘letter only’ group in which each woman received an invitation letter to either attend her normal doctor or make an appointment for the screening clinic, together with instructions for making an appointment at the clinic (patient reminders group 1); and an ‘appointment letter’ group in which each woman received an invitation letter to attend a special screening clinic at a specified date and time (patient reminders group 2). Both patient reminder groups were analysed together | |
| Outcomes | Proportion of patients with a Pap smear taken Assessed at: 12 months |
|
| Notes | Cervical screening in primary care 2 relevant comparisons (reminders versus control/usual care; and reminders versus other QI (patient reminders) intervention) Funding: Australia Health, Housing and Community Services Research and Development Grant |
|
| Risk of bias | ||
| Bias | Authors' judgement | Support for judgement |
| Random sequence generation (selection bias) | Low risk | Quote: "757 were eligible for inclusion in the study and were allocated randomly to one of three intervention groups or a control group." "Eligible women were randomly allocated to one of four groups using a table of random numbers." |
| Allocation concealment (selection bias) | Unclear risk | Not reported |
| Blinding of participants and personnel (performance bias) All outcomes | High risk | Clearly not blind, and the outcome was assess by patient's surveys. For the 'tagged notes' group it was unclear if patients were blinded. For the patient reminder groups it was not clear if personnel were blinded |
| Blinding of outcome assessment (detection bias) All outcomes | Unclear risk | Outcome assessors were not blinded but unclear of the impact of this on the effect estimates, given that the outcome is objective |
| Incomplete outcome data (attrition bias) All outcomes | High risk | 400 out of 757 randomised patients (52.8%) were lost to follow‐up (Table 3) |
| Selective reporting (reporting bias) | Low risk | Although there was a single primary outcome it seems to be the relevant one for this study |
| Other bias | Unclear risk | Because the randomisation was at the patient level the risk of contamination (physicians seeing patients in the different study groups) was unclear. Clustering is not mentioned in the design or analysis contamination; how many women had the same physician? |
Rattanaumpawan 2016.
| Methods | Cluster‐randomised trial | |
| Participants | 6 general medicine wards (874 patients) at Siriraj Hospital, a 2300‐bed tertiary care hospital (Thailand) | |
| Interventions | A self‐inking stamp in the doctor's order sheet versus usual infection control care | |
| Outcomes | Mean duration of all catheter days, mean duration of temporary catheter days, mean duration of Foley’s catheter days, mean duration of central venous catheter days, median length of hospital stay, cumulative incidence (infection episode per hospitalisation), incidence rate (infection episode per catheter day) of CAUTIs and CLABSIs, and hospital mortality Assessed at: unclear |
|
| Notes | Hospital‐acquired infections 1 relevant comparison (reminders versus usual/control care) Funding: This study was primarily supported by the Health Systems Research and Development Project, Faculty of Medicine Siriraj Hospital and Health Systems Research Institute (Thailand) |
|
| Risk of bias | ||
| Bias | Authors' judgement | Support for judgement |
| Random sequence generation (selection bias) | Unclear risk | The sequence generation procedure is not detailed: Quote: "A total of 6 general medicine wards were randomly allocated to receive the usual infection control care either with or without the Catheter Reminder and Evaluation (CARE) program." |
| Allocation concealment (selection bias) | Unclear risk | Not reported |
| Blinding of participants and personnel (performance bias) All outcomes | Unclear risk | Personnel were not blinded but unclear of the impact of this on effect estimates |
| Blinding of outcome assessment (detection bias) All outcomes | Unclear risk | Not reported |
| Incomplete outcome data (attrition bias) All outcomes | Low risk | It seems that losses to follow‐up were minimal |
| Selective reporting (reporting bias) | Low risk | There seem to be a wide range of objective outcomes compatible with the registered protocol (ClinicalTrials.gov identifier: NCT01797146) |
| Other bias | Unclear risk | Possible contamination: Quote: "...therefore, it is possible that residents in the CARE ward may transfer their positive behavior when rotating to the control ward." |
Renzi 2006.
| Methods | Cluster‐randomised trial | |
| Participants | 104 primary care physicians (1612 patients) located in 4 Quebec regions (Canada) | |
| Interventions | Self‐inking paper stamp checklist tool summarising the Canadian Clinical Practice Guidelines for physician assessment of asthmatic patient control and therapy. There were 4 groups: 3 of them including the self‐inking paper stamp + educational materials; the control group was educational materials | |
| Outcomes | Proportion of patients with emergency room visits and with hospitalisations Assessed at: 12 months |
|
| Notes | Asthma management 1 relevant comparison (reminders plus other QI (educational materials) intervention versus other QI) Funding: The Towards Excellence in Asthma Management (TEAM) project of the Quebec Asthma Education Network. |
|
| Risk of bias | ||
| Bias | Authors' judgement | Support for judgement |
| Random sequence generation (selection bias) | Unclear risk | The sequence generation procedure is not clearly described: Quote: "The physicians were then randomly assigned to the four groups listed in Table 1" |
| Allocation concealment (selection bias) | Unclear risk | Not reported |
| Blinding of participants and personnel (performance bias) All outcomes | Unclear risk | It was unclear if patients were blinded; not mentioned but unlikely |
| Blinding of outcome assessment (detection bias) All outcomes | Unclear risk | Not reported |
| Incomplete outcome data (attrition bias) All outcomes | Unclear risk | Not reported |
| Selective reporting (reporting bias) | Unclear risk | Not sure if there were other relevant outcomes |
| Other bias | High risk | No mention of clustering and how it was addressed; no clear report of analysis |
Richards 2001.
| Methods | Cluster‐randomised trial | |
| Participants | 24 general practices with low uptake in the second round of screening (below 60%) in north west London and the West Midlands (UK) Participants were all women (6133) registered with these practices and eligible for screening in the third round |
|
| Interventions | Physicians' reminders (a green card to be inserted into patients’ notes (flag), an encounter form integral to the flag, and an information leaflet), and patient reminders (a letter, translation sheet, and information leaflet). The practices were allocated to one of those interventions, neither of them or both. | |
| Outcomes | Attendance for screening 6 months after the practices had been screened Assessed at: 6 months |
|
| Notes | 3 relevant comparisons (reminders versus usual care; reminders plus other QI intervention versus other QI; and reminders versus other QI (patient reminders) intervention) Funding: UK Medical Research Council |
|
| Risk of bias | ||
| Bias | Authors' judgement | Support for judgement |
| Random sequence generation (selection bias) | Low risk | Quote: "With random number tables, the 24 practices were randomised within strata into one of the four intervention groups" |
| Allocation concealment (selection bias) | Low risk | Quote: "...study staff who were independent of the practice recruitment process and blind to the identities of the individual general practices." |
| Blinding of participants and personnel (performance bias) All outcomes | Low risk | Personnel were not blinded and patients in the flag group unclear, but the impact on effect estimates seems to be low |
| Blinding of outcome assessment (detection bias) All outcomes | Unclear risk | It was not clear if assessors were blinded |
| Incomplete outcome data (attrition bias) All outcomes | Low risk | 5732 out of 6133 (93,5%) women with reported outcomes |
| Selective reporting (reporting bias) | Low risk | Although there was a single outcome it seems to be the relevant one |
| Other bias | Low risk | None relevant. The analysis considered clustering |
Roetzheim 2004.
| Methods | Cluster‐randomised trial | |
| Participants | 8 primary care clinics (1196 patients) in Hillsborough County, Florida (USA) | |
| Interventions | The intervention is a package called Cancer SOS. It has 2 components: a cancer‐screening checklist with chart stickers that indicated whether specific cancer‐screening tests were due, ordered, or completed; and a division of office responsibilities to achieve high screening rates | |
| Outcomes | Proportion of patients up‐to‐date on 1 or more of the following cancer‐screening tests: mammogram, Papanicolaou (Pap) smear, or FOBT Assessed at: 12 months |
|
| Notes | 1 relevant comparison (reminders versus control) The study was included in the review but not considered in the analysis because we were unable to obtain denominators for each group (control and intervention) and then we were unable to compute an absolute effect estimate Funding: This research was supported by National Cancer Institute grant R01 CA77282 |
|
| Risk of bias | ||
| Bias | Authors' judgement | Support for judgement |
| Random sequence generation (selection bias) | Unclear risk | The sequence generation procedure was not described: Quote: "We performed a cluster randomized experimental trial in which 8 clinics meeting eligibility criteria were randomized to either intervention or control conditions." |
| Allocation concealment (selection bias) | Unclear risk | Not reported |
| Blinding of participants and personnel (performance bias) All outcomes | Low risk | Participants were not blinded but the outcome was objective and it was unlikely that this could affect effect estimates |
| Blinding of outcome assessment (detection bias) All outcomes | Unclear risk | They did not mention this issue, although something was done (abstraction was made 3 months after patients had visited the clinic) to prevent the influence of records review from influencing patient or provider screening behaviour |
| Incomplete outcome data (attrition bias) All outcomes | Unclear risk | The number of potentially eligible records (patients) in each arm is not presented |
| Selective reporting (reporting bias) | Low risk | A reasonable set of outcome included |
| Other bias | Low risk | None identified. The analysis took into account clustering and potential confusion variables |
Saillour‐Glénisson 2005.
| Methods | Cluster‐randomised trial | |
| Participants | 6 hospitals (67 wards, 1412 orders for thyroid function tests) in France | |
| Interventions | Pocket card (reminders), test request form (other QI), neither, and both | |
| Outcomes | Guideline Conformity Rate and Cost‐effectiveness ratios Assessed at: unclear (12 months?) |
|
| Notes | 3 relevant comparisons (reminders versus control, reminders plus other QI (test request form) versus other QI intervention, and reminders versus other QI) Funding: Partial funding from l’Agence Nationale d’Accréditation et d’Évaluation en Santé (ANAES) |
|
| Risk of bias | ||
| Bias | Authors' judgement | Support for judgement |
| Random sequence generation (selection bias) | Unclear risk | The sequence generation procedure is not described |
| Allocation concealment (selection bias) | Unclear risk | Not reported |
| Blinding of participants and personnel (performance bias) All outcomes | Unclear risk | Not reported |
| Blinding of outcome assessment (detection bias) All outcomes | Unclear risk | Not reported |
| Incomplete outcome data (attrition bias) All outcomes | Low risk | 1412 orders included without losses to follow‐up reported |
| Selective reporting (reporting bias) | Low risk | Unlikely |
| Other bias | Unclear risk | Risk of contamination? |
Saitz 2003.
| Methods | Cluster‐randomised trial | |
| Participants | Urban academic primary care practice: 41 faculty and resident primary care physicians and 312 patients with hazardous drinking | |
| Interventions | Providing physicians with alcohol screening results (CAGE questionnaire responses, alcohol consumption, and readiness to change) and recommendations for their patients at a visit | |
| Outcomes | Patient self‐report of discussions about alcohol use immediately after the physician visit and alcohol use 6 months later Assessed at: 6 months |
|
| Notes | 1 relevant comparison (reminders versus control/usual care) The study was included in the review but not considered in the analysis because we were not able to calculate the denominators for each stratum: 240 patients for faculty physicians and 72 for residents but not separated by intervention and control groups Funding: the Robert Wood Johnson Foundation (grant031489), Princeton, New Jersey. |
|
| Risk of bias | ||
| Bias | Authors' judgement | Support for judgement |
| Random sequence generation (selection bias) | Low risk | Quote: "Physicians were stratified by level of training (resident or faculty) and were randomly assigned to the intervention or control group at the start of the study.' 'The computer‐generated randomization was done by off‐site data management personnel who had no patient or physician contact." |
| Allocation concealment (selection bias) | Low risk | Quote: "...was done by off‐site data management personnel who had no patient or physician contact." |
| Blinding of participants and personnel (performance bias) All outcomes | Unclear risk | It was unclear if participants were blinded and this could have an important impact on effect estimates |
| Blinding of outcome assessment (detection bias) All outcomes | Low risk | Quote: "...at follow‐up, interviews were done without knowledge of group assignment." |
| Incomplete outcome data (attrition bias) All outcomes | Low risk | 41 out of 50 (82%) randomised physicians completed the study |
| Selective reporting (reporting bias) | Unclear risk | Only patient‐reported outcomes included |
| Other bias | Unclear risk | No additional source identified |
Scholes 2006.
| Methods | Randomised trial | |
| Participants | 3509 sexually active women, aged 14 ‐ 20 years, enrolled in a mixed‐model managed care system in Washington state (USA) | |
| Interventions | Chart prompt reminders: a brief, highly visible prompt placed in the front of randomly selected patient charts, stated ‘High Risk Age Group for Chlamydia. Consider Screening,’ and included the guideline web‐link | |
| Outcomes | Chlamydia testing rates during 12 months post‐intervention Assessed at: 12 months |
|
| Notes | This study included 2 parts: a clinic‐level intervention and a chart prompt intervention. We only used data from the latter (reminders versus control) Funding: Financial support was provided by R01 HS10514 from the Agency for Healthcare Research, as part of the Translating Research into Practice (TRIP) initiative; and K07 CA84603 (JBM) |
|
| Risk of bias | ||
| Bias | Authors' judgement | Support for judgement |
| Random sequence generation (selection bias) | Unclear risk | It was clearly randomised but the sequence generation procedure was unclear: Quote: "We employed a two‐by‐two factorial randomized trial design to evaluate two conceptually‐based interventions to increase adherence to the GHC chlamydia screening guideline in usual clinical practice" Quote: "Randomization at the enrollee level was employed to evaluate the effect of a chart prompt encouraging providers to screen for chlamydia per guideline recommendations" |
| Allocation concealment (selection bias) | Unclear risk | Not reported |
| Blinding of participants and personnel (performance bias) All outcomes | Low risk | Participants unblinded but unlikely to impact on effect estimates |
| Blinding of outcome assessment (detection bias) All outcomes | Unclear risk | No mention of this issue. It could potentially affect outcome adjudication in some borderline cases |
| Incomplete outcome data (attrition bias) All outcomes | Low risk | No report of losses to follow‐up |
| Selective reporting (reporting bias) | Low risk | Although there was a single outcome it seems to be the relevant one for this intervention |
| Other bias | High risk | There is mention of clustering in the analysis of the clinical‐level intervention but not for the individual‐level intervention: Quote: "we tested for interaction between the clinic‐ level and chart prompt interventions. Since this interaction was not significant, the results for the two interventions are reported separately." Even when this was done there is a risk of contamination in the individual‐level intervention |
Seto 1989.
| Methods | Cluster‐randomised trial | |
| Participants | Nine randomly‐selected wards (235 participant nurses) in Queen Mary Hospital, Hong Kong | |
| Interventions | 3 groups: a simple announcement through the nursing hierarchy in group A (control), a passive method (posters and pamphlets) was added in group B, and both passive and active methods (in‐service lectures) were used in group C | |
| Outcomes | Proportion of nurses no longer recapping needles, proportion of needles uncapped Assessed at: 5 weeks and 6 months |
|
| Notes | 1 relevant comparison (reminders (group B) versus control/usual care (group A)) Funding: not reported |
|
| Risk of bias | ||
| Bias | Authors' judgement | Support for judgement |
| Random sequence generation (selection bias) | Low risk | Quote: "From the 36 clinical wards of the hospital, nine were randomly selected and divided into three groups (A: B and C) of three wards each with the help of a computerised randomization programme." |
| Allocation concealment (selection bias) | Unclear risk | Not reported |
| Blinding of participants and personnel (performance bias) All outcomes | Unclear risk | Personnel unblinded but unclear if they were aware of the other interventions |
| Blinding of outcome assessment (detection bias) All outcomes | Unclear risk | Not mentioned, but this could have an impact on effect estimates |
| Incomplete outcome data (attrition bias) All outcomes | Low risk | Outcome data available from 208 out of 235 (88.5%) of nurses |
| Selective reporting (reporting bias) | Low risk | No protocol available, but the scope of the study is very specific and the outcome seems to be the relevant ones |
| Other bias | High risk | Clustering not considered in the analysis and risk of contamination in a single hospital |
Shannon 2001.
| Methods | Cluster‐randomised trial | |
| Participants | All (31) resident physicians (608 patients) in a large family practice residency programme in New Hampshire (USA) | |
| Interventions | The Comprehensive Annotated Reminder Tool (CART): a form for documenting history and physical examinations containing 8 sections: history, screening history, physical examination, screening mnemonics, laboratory screens, prophylaxis, counselling, and the assessment/plan section. The CART forms were placed on all charts of new patients seen by physicians in the experimental group | |
| Outcomes | Physician adherence to up to 49 preventive services recommendations Assessed at: unclear |
|
| Notes | 1 relevant comparison (reminders versus control/usual care) Funding: a grant from the Advocate Christ Hospital Medical Fund |
|
| Risk of bias | ||
| Bias | Authors' judgement | Support for judgement |
| Random sequence generation (selection bias) | Unclear risk | The sequence generation procedure is not described: Quote: "Resident physicians were randomly assigned to a treatment group that was exposed to the CART (n=15) and a control group that used existing blank history and physical examination forms (n=16)." |
| Allocation concealment (selection bias) | Unclear risk | Not reported |
| Blinding of participants and personnel (performance bias) All outcomes | Unclear risk | It was unclear if patients were blinded to the intervention |
| Blinding of outcome assessment (detection bias) All outcomes | Unclear risk | It was not clear if only a subgroup of all the charts was assessed without knowing the allocated group: Quote: "Blinded chart reviews were performed by the principal investigator and 2 other independent reviewers on randomly selected new patients." |
| Incomplete outcome data (attrition bias) All outcomes | Unclear risk | Not reported |
| Selective reporting (reporting bias) | Low risk | A comprehensive set of preventive measures included in the analysis |
| Other bias | High risk | Clustering was not considered in the analysis, and some risk of contamination in a single practice |
Shaw 2000.
| Methods | Cluster‐randomised trial | |
| Participants | 52 first‐ and second‐year paediatric residents (626 well‐child care visits) in a hospital‐based continuity clinic in Boston (USA) | |
| Interventions | Manual prompts of immunisations due: a trained research assistant manually determined the immunisations due, using an algorithm. A list of all possible immunisations along with the words 'due today' was stamped on to the computerised printout, and the appropriate immunisations due that day were checked off. Although there were computers in the prompt elaboration process we think that the core mechanism of action of the intervention was similar to other paper reminders interventions | |
| Outcomes | Proportion of visits with unacceptable missed opportunities/vaccine administration errors (reversed for analysis) Assessed at: unclear (immediate?) |
|
| Notes | 1 relevant comparison (reminders versus control/usual care) Funding: This study was supported in part by the Massachusetts Immunization Program, Massachusetts Department of Public Health |
|
| Risk of bias | ||
| Bias | Authors' judgement | Support for judgement |
| Random sequence generation (selection bias) | Unclear risk | The sequence generation procedure is described but we were not sure if this was fully randomised (alternation): Quote: "First‐ and second‐year residents were randomized by day of the week and assigned to either the control (no prompting) or the intervention (prompting) group." "...random‐ number generator program was used to choose 2 days for the intervention group and 3 days for the control group." |
| Allocation concealment (selection bias) | High risk | Assigning days of the week makes the randomisation sequence known to everyone |
| Blinding of participants and personnel (performance bias) All outcomes | Unclear risk | It is unclear if patients were blinded. It is unclear if unblinding of residents could have had some impact on effect estimates |
| Blinding of outcome assessment (detection bias) All outcomes | Low risk | The final judgment of vaccination error was made by a clinician blinded to the allocation: Quote: "At the end of each day, post encounter chart review of all charts in both groups compared immunizations administered with those due. No variance was defined as complete administration of the vaccines due. Those charts with any variance from complete administration of immunizations due were further reviewed by a clinician, blinded to the resident groups." |
| Incomplete outcome data (attrition bias) All outcomes | Low risk | 626 out of 686 visits (91%) were used in the analysis |
| Selective reporting (reporting bias) | Unclear risk | It seem to be a reasonable outcome to be measured in the context of this study but no protocol available |
| Other bias | High risk | Clustering was incorporated in the analysis, but contamination among residents was possible |
Somkin 1997.
| Methods | Randomised trial | |
| Participants | 7077 female health maintenance organization members (aged 50 ‐ 74 years with no prior mammogram in the previous 30 months or aged 20 ‐ 64 years with no prior Pap smear in the previous 36 months) in Northern California (USA) | |
| Interventions | A letter sent to participants described some common barriers to screening and provided women with a telephone number to call and schedule a 'VIP mammogram' or a 'VIP Pap smear' (patient reminder group), the patient reminder plus a chart reminder (yellow form) placed manually by a chart room clerk (patient plus provider reminder group), and control/usual care group | |
| Outcomes | Proportion of patients with mammograms and Pap smear done Assessed at: 6 months |
|
| Notes | 1 relevant comparison (reminders plus other QI intervention (patient reminders) compared with other QI intervention) Funding: This study was funded by the Innovation Program, Clinical Services Branch, The Permanente Medical Group, Oackland, California |
|
| Risk of bias | ||
| Bias | Authors' judgement | Support for judgement |
| Random sequence generation (selection bias) | Unclear risk | The sequence generation procedure is not described: Quote: "Separately for the mammography and Pap smear samples, we randomly selected 594 eligible women during each of 6 months (March to August 1994) and randomized them to receive one of the following..." |
| Allocation concealment (selection bias) | Unclear risk | Not reported |
| Blinding of participants and personnel (performance bias) All outcomes | Low risk | None of them was blinded to the intervention group, but the impact on outcome measurement seems to be limited because the outcome was routinely‐collected data |
| Blinding of outcome assessment (detection bias) All outcomes | Low risk | Although this was unclear the outcome was assessed from a routinely‐used computerised system |
| Incomplete outcome data (attrition bias) All outcomes | Unclear risk | This is not mentioned, although the follow‐up seems to be almost complete (?) |
| Selective reporting (reporting bias) | Low risk | Outcomes assessed were the relevant ones |
| Other bias | High risk | Authors incorporated a number of covariates in the analysis but unclear if this considered clustering appropriately. Risk of contamination? |
Stamos 2001.
| Methods | Cluster‐randomised trial | |
| Participants | 16 physicians (290 patients) in a Cardiology Clinic at Cook County Hospital, Chicago, Illinois (USA) | |
| Interventions | Reminders attached to the front of the chart when patients were not being managed in accordance with NCEP guidelines. They were individualised to the given patient, citing the NCEP guidelines and suggesting the specific change in management required to restore compliance with the guidelines | |
| Outcomes | Compliance with the NCEP guidelines Assessed at: unclear |
|
| Notes | 1 relevant comparison (reminders versus control/usual care) Funding: This study was supported in part by an unrestricted educational grant from the Bristol‐Myers Squibb Corporation, New York, New York |
|
| Risk of bias | ||
| Bias | Authors' judgement | Support for judgement |
| Random sequence generation (selection bias) | Unclear risk | No details about the sequence generation procedure: Quote: "This was a randomized, controlled study,..." "by 8 physicians (4 cardiology fellows, 4 attending cardiologists) randomly chosen to receive the intervention." |
| Allocation concealment (selection bias) | Unclear risk | Not reported |
| Blinding of participants and personnel (performance bias) All outcomes | Unclear risk | It was not clear if patients were blinded (and this could affect effect estimates through them asking specific clinical questions) |
| Blinding of outcome assessment (detection bias) All outcomes | Unclear risk | The impact of unblinded outcome assessor in effect estimates is uncertain, especially for routinely‐collected outcomes Quote: "Laboratory data were collected using the hospital laboratory information system." |
| Incomplete outcome data (attrition bias) All outcomes | Low risk | There is no mention of losses to follow‐up at either the physician or at the patient level |
| Selective reporting (reporting bias) | Low risk | The outcomes chosen seem to be the relevant ones, but no information from the protocol available |
| Other bias | High risk | We were not sure if the analytical plan addressed clustering. Some risk of contamination among physicians in a single clinic |
Strecher 1991.
| Methods | Cluster‐randomised trial | |
| Participants | 11 residency programmes (261 residents, 937 patients), in internal medicine (6), family medicine (3), and pediatrics (2). Programmes were located in 3 university medical centres and 4 university‐affiliated community hospitals in North Carolina and Pennsylvania (USA) | |
| Interventions | 2 interventions (tutorial and prompt) and 4 groups. The tutorial was a 2‐hour educational programme in minimal‐contact smoking cessation counselling for residents. The prompt was a chart‐based reminder to assist physician counselling. 1 group of residents received the tutorial; 1, the prompt; and one, both. A 4th group received no intervention | |
| Outcomes | The primary physician outcomes were 2 counselling practices: frequency (the percentage of smokers residents counselled who returned); and content (the number of 5 specific techniques residents reported having used in counselling). Several secondary physician outcomes were also examined, including the use of techniques to motivate patients to quit smoking and the presence of 3 attitudes toward smoking cessation counselling: confidence, perceived preparedness, and perceived success. The primary patient‐related outcome was the patient quitting rate: the percentage of each resident's patients with a self‐report of smoking cessation within 6 months of the exit interview Assessed at: 6 months |
|
| Notes | 3 relevant comparisons (reminders versus control/usual care; reminders plus other QI (tutorial) versus other QI intervention; and reminders versus other QI (educational meetings) intervention) Funding: supported by the University of North Carolina Faculty Development Program in General Medicine and General Pediatrics and by grants from the Cancer Prevention and Control Program of the National Cancer Institute, the North Carolina Chapter of the American Heart Association and the University of North Carolina Center for Health Promotion and Disease Prevention |
|
| Risk of bias | ||
| Bias | Authors' judgement | Support for judgement |
| Random sequence generation (selection bias) | Unclear risk | The sequence generation procedure was not clearly described: Quote: "We used a randomized factorial design to test the two interventions, alone and in combination" Quote: "After the pretest, residents were randomly assigned by clinic half‐day session to one of four groups: tutorial only, prompt only, tutorial + prompt, and control." |
| Allocation concealment (selection bias) | Unclear risk | Not reported |
| Blinding of participants and personnel (performance bias) All outcomes | Unclear risk | It was unclear if patients were blinded to interventions and this could have had an impact on patient‐reported outcomes |
| Blinding of outcome assessment (detection bias) All outcomes | Unclear risk | For the patients' self‐report the outcome assessor was blind. But it is not clear for the physicians' self‐report (probably not). It was not clear how self‐reported outcomes data were collected: Quote: "Six months after the initial exit interview, telephone interviewers, who were blind to residents' and patients' group assignments, obtained patient reports on current smoking status." |
| Incomplete outcome data (attrition bias) All outcomes | Low risk | 6% lost from physicians, 29% lost from patients |
| Selective reporting (reporting bias) | Unclear risk | A comprehensive set of outcomes was reported but no information about the protocol |
| Other bias | Unclear risk | Not clear if they adjusted by physicians (clustering) |
Szilagyi 1996.
| Methods | Randomised trial | |
| Participants | 878 patients attending a Pediatric Continuity Clinic in a teaching hospital in Rochester, NY (USA) | |
| Interventions | "The No Missed Opportunities intervention has 3 components: (1) medical charts of study patients were marked with a conspicuous black dot to indicate that they were assigned to the NMO study group; (2) staff nurses were instructed to screen these medical charts for immunization status at all visit types, including acute‐illness, follow‐up, and nurse only visits; and (3) if a vaccination was due, a brightly colored immunization reminder card was to be attached by the triage nurse to the front of the medical chart indicating that a vaccination was due and listing the valid contraindications to immunizations according to current Advisory Committee on Immunization Practices and American Academy of Pediatrics criteria" | |
| Outcomes | Missed opportunity rates, proportion of patients with up‐to‐date immunisation schedule Assessed at: 9 ‐ 18 months |
|
| Notes | The study assessed two different strategies in separate studies: the No Missed Opportunities (NMO) intervention and the Vaccination Without Legal Guardian's Signature intervention. We only included the first one in our analysis 1 relevant comparison (reminders versus control/usual care) Funding: Supported by grants from the New York State Department of Health in Albany (Contract C‐008975), and the Strong Children's Research Center, Rochester |
|
| Risk of bias | ||
| Bias | Authors' judgement | Support for judgement |
| Random sequence generation (selection bias) | Unclear risk | The sequence generation procedure was not detailed: Quote: "The study was a randomized controlled clinical trial as shown in Figure 1" |
| Allocation concealment (selection bias) | Unclear risk | Not reported |
| Blinding of participants and personnel (performance bias) All outcomes | Unclear risk | It is not clear if patients were blinded and the impact of this on effect estimates is uncertain; Unlikely but outcome objective |
| Blinding of outcome assessment (detection bias) All outcomes | Low risk | Quote: "At the end of the study, a medical chart review was performed by chart abstractors who were blind to the study hypotheses and to group allocations." |
| Incomplete outcome data (attrition bias) All outcomes | Unclear risk | This information is not clearly reported |
| Selective reporting (reporting bias) | Unclear risk | The outcomes are not presented in a clear format and it is not always easy to identify effect estimates (not sure if this is related to selective reporting) |
| Other bias | High risk | Not adjusted by centre, and contamination possible in a single centre where physicians could see patients in both groups |
Thapar 2002.
| Methods | Cluster‐randomised trial | |
| Participants | 82 general practices (1275 patients with active epilepsia) in 4 areas of Greater Manchester (UK) | |
| Interventions | The intervention consisted of an evidence‐based epilepsy prompt and reminder card for GPs to complete. The card had 2 main parts: first, ‘prompts’ to collect key clinical information about an individual’s epilepsy; and secondly, evidence‐based information (‘reminders’) on which to then base any subsequent patient management decision. The card was passport‐sized, bright yellow in colour, and consisted of 9 sections (including seizure frequency and pattern, seizure classification, medication, side‐effects and indications for medication withdrawal, checking serum levels, information provision, and monitoring). In 1 group (provider reminder) the card was inserted into the patients’ records, and in other group (patient reminder) the patient held the card | |
| Outcomes | The primary outcome measures were recording of seizure frequency and self‐reported seizure frequency in the previous year. Secondary outcome measures were the retrieval rate and completion rate of the epilepsy card, the proportion of patients on monotherapy with anticonvulsants, the proportion of patients reporting medication side‐effects, whether serum levels of anticonvulsants were checked appropriately, the levels of patient satisfaction with GP care, and level of satisfaction with information provision by the GP Assessed at: 12 months |
|
| Notes | Clinical area: management of epilepsy 2 relevant comparisons (reminders versus control/usual care, and reminders versus other QI (patient reminders) intervention) Funding: the UK Department of Health Implementation of research methods programme (IMP 15/12) and Sanofi Pharmaceuticals |
|
| Risk of bias | ||
| Bias | Authors' judgement | Support for judgement |
| Random sequence generation (selection bias) | Low risk | Quote: "The study was a pragmatic randomised trial. Practices were stratified into small (fewer than three partners in practice) or large (three or more partners in the practice). Using a random number table, practices were either allocated to the ‘control’ group, to the ‘doctor‐held card’ group (where the card was inserted into the patients’ records) or to the ‘patient‐held card’ group (where the patient held the card)." |
| Allocation concealment (selection bias) | Unclear risk | Not reported |
| Blinding of participants and personnel (performance bias) All outcomes | Unclear risk | Neither participants nor personnel were blinded and the impact of this on effect estimates is unclear |
| Blinding of outcome assessment (detection bias) All outcomes | Unclear risk | Not reported |
| Incomplete outcome data (attrition bias) All outcomes | Low risk | It was low for outcomes from medical records (94.9%) and borderline for outcomes from questionnaires (76.5%) |
| Selective reporting (reporting bias) | Low risk | It seems to be a complete set of outcomes (although no protocol available) |
| Other bias | Low risk | None identified |
Vinker 2002.
| Methods | Cluster‐randomised trial | |
| Participants | 6 family physicians (2315 patients) in 2 primary care clinics in Israel | |
| Interventions | In the physician intervention arm (group I), a reminder note to the physician was placed in the patient file. It advised physicians to direct patients to perform a FOBT according to accepted practice. In the patient intervention arm (group II), 2 reminder strategies were used (a reminder letter or a phone call). In the control group (group III), physicians continued administering their usual level of care | |
| Outcomes | The main outcome measure was the percentage of patients performing screening FOBT at the conclusion of the 1‐year study period Assessed at: 12 months |
|
| Notes | Clinical area: Colorectal cancer screening 2 relevant comparisons (reminders versus control/usual care; and reminders versus other QI (patient reminders) intervention) Funding: The project was supported by a grant from the Israel Cancer Society. |
|
| Risk of bias | ||
| Bias | Authors' judgement | Support for judgement |
| Random sequence generation (selection bias) | Unclear risk | The sequence generation procedure was not clearly described |
| Allocation concealment (selection bias) | Unclear risk | Not reported |
| Blinding of participants and personnel (performance bias) All outcomes | Unclear risk | Not reported |
| Blinding of outcome assessment (detection bias) All outcomes | Unclear risk | Not reported |
| Incomplete outcome data (attrition bias) All outcomes | Unclear risk | Although in Figure 1 there was no losses to follow‐up the issue is not explicitly described |
| Selective reporting (reporting bias) | Low risk | A single outcome but it seems to be the relevant one in this field |
| Other bias | High risk | Clustering not considered in the design or the analysis. The reminders arm included both manual and computerised reminders; we used only manual reminders in our analysis (breaking in some way randomisation) |
Walker 1999.
| Methods | Cluster‐randomised trial | |
| Participants | 10 physicians (74 patients) at the University of Cincinnati's Infectious Disease Center (USA) | |
| Interventions | A coloured chart reminder placed on charts not earlier than 24 hours before patient's primary care clinic visit | |
| Outcomes | Rate of documentation by the physician of discussion about advanced directives (AD) and the rate of AD completion by the patient by the end of the 6‐month study period Assessed at: 6 months |
|
| Notes | Clinical area: care of HIV/AIDS patients 1 relevant comparison (reminders versus control/usual care) Funding: not reported |
|
| Risk of bias | ||
| Bias | Authors' judgement | Support for judgement |
| Random sequence generation (selection bias) | Unclear risk | They mention that groups were randomly assigned but they were also grouped by day of their clinical practice. The sequence generation procedure is not described |
| Allocation concealment (selection bias) | Unclear risk | Not reported |
| Blinding of participants and personnel (performance bias) All outcomes | Unclear risk | Quote: "Physicians were unaware of the purpose of the study" but it is not clear if they were unblinded |
| Blinding of outcome assessment (detection bias) All outcomes | Unclear risk | Outcome assessors were independent from the researchers group but not blinded. Although the criteria to assess outcomes was objective it was not clear if this protected completely from bias |
| Incomplete outcome data (attrition bias) All outcomes | Low risk | No mention of losses to follow‐up |
| Selective reporting (reporting bias) | Low risk | The 2 outcomes seems to be the relevant ones in this field |
| Other bias | Unclear risk | Contamination was unlikely because they grouped physicians by day of their clinical practice. They did a physician‐level sensitivity analysis to determine if the intervention was physician‐dependent. However, we were not sure if the analysis described (physician‐level analysis using Fisher's exact test or multiple logistic regression) considered clustering |
Wang 1994.
| Methods | Cluster‐randomised trial | |
| Participants | 27 second‐ and third‐year residents, part‐time and full‐time staff physicians (93 patients) at the Department of Family Medicine, National Taiwan University Hospital | |
| Interventions | 1 group of physicians received 2 lectures on the stages‐of‐change model on which to base their patient counselling (educational meetings). A second group was not exposed to the model but received a reminder reading 'Ask your patients to stop smoking' (reminders). A third group was not exposed to any of the previous interventions (usual care) | |
| Outcomes | Quit‐smoking rate Assessed at: 6 months |
|
| Notes | Clinical area: smoking cessation 2 relevant comparisons: reminders versus control/usual care; and reminders versus other QI intervention (educational meetings) Funding: This study was supported by a grant from the Department of Health |
|
| Risk of bias | ||
| Bias | Authors' judgement | Support for judgement |
| Random sequence generation (selection bias) | Unclear risk | Quote: "All physicians were numbered and randomly assigned to one of 3 groups by number of years in practice" (not truly randomised?) |
| Allocation concealment (selection bias) | Unclear risk | Not reported |
| Blinding of participants and personnel (performance bias) All outcomes | Unclear risk | Unclear if participants were blinded and this could be relevant in a self‐reported outcome |
| Blinding of outcome assessment (detection bias) All outcomes | Unclear risk | Not reported |
| Incomplete outcome data (attrition bias) All outcomes | Low risk | It seems that all the 27 physicians were at follow‐up and 82 out of 93 patients (88.2%) completed the follow‐up interview (although losses were higher in the control group) |
| Selective reporting (reporting bias) | Unclear risk | Smoking cessation was reported by patients without a complementary 'objective' outcome. Not sure if this could affect effect estimates |
| Other bias | High risk | Clustering not considered in the design or analysis Smoking cessation was reported by patients with no 'objective' confirmation Because residents were in a single hospital there is some risk of contamination |
Wigton 1981.
| Methods | Non‐randomised trial | |
| Participants | 291 patients with abnormal haemoglobin values evaluated at the University of Nebraska Hospital (USA) | |
| Interventions | Chart reminder: it took the form of a questionnaire the same size as the progress note sheets. The questionnaire stated it was part of a study of anaemia, listed the patient's abnormal haemoglobin value, and then asked 2 questions | |
| Outcomes | Diagnosis of anaemia, number of non‐substantive errors (in the management of anaemia) A non‐comparative description of costs included Assessed at: 6 months |
|
| Notes | 1 relevant comparison (reminders versus control/usual care) Funding: not reported |
|
| Risk of bias | ||
| Bias | Authors' judgement | Support for judgement |
| Random sequence generation (selection bias) | High risk | Non‐randomised allocation: Quote: "Those with low abnormal values were assigned alternately to a study group or to a control group." |
| Allocation concealment (selection bias) | High risk | No mention of this issue but probably not considered |
| Blinding of participants and personnel (performance bias) All outcomes | Unclear risk | It was not clear if patients were blinded and this could potentially affect the effect of intervention |
| Blinding of outcome assessment (detection bias) All outcomes | Low risk | Quote: "Neither the committee members nor the records administrator knew which cases were assigned to the study group" |
| Incomplete outcome data (attrition bias) All outcomes | High risk | 173 out of 291 (59.4%) of patients with outcome data. The characteristics of the other 118 patients are not detailed |
| Selective reporting (reporting bias) | Unclear risk | It was not possible to make a judgement with the information available |
| Other bias | High risk | Possible contamination |
Yazdany 2011.
| Methods | Randomised trial | |
| Participants | 204 patients seen at the gynaecology outpatient clinics at Harbor UCLA Medical Center, Los Angeles (USA) | |
| Interventions | A chart‐alert sticker reading ‘do you leak urine’. The chart alert appeared as a white sticker placed on the lower left‐hand corner of the preprinted history and physical intake assessment sheet, using the same text format as the form | |
| Outcomes | Primary: documentation of urinary incontinence Secondary: diagnosis made, initiation of workout treatment plan, referral to an urogynaecologist Assessed at: unclear |
|
| Notes | 1 relevant comparison (reminders versus control/usual care) Funding: not reported |
|
| Risk of bias | ||
| Bias | Authors' judgement | Support for judgement |
| Random sequence generation (selection bias) | Low risk | Quote: "We randomized 350 new patient consults to receive a chart‐alert sticker reading ‘‘do you leak urine’’ versus no sticker using a simple randomization computer‐generated sequence. A team research assistant created the randomization sequence |
| Allocation concealment (selection bias) | Low risk | Quote: "The randomization sequence list was secured until completion of the study." |
| Blinding of participants and personnel (performance bias) All outcomes | Low risk | Quote: "Residents were not informed about the study and, if they asked, were told the reasons for the sticker were confidential." |
| Blinding of outcome assessment (detection bias) All outcomes | Unclear risk | It was not clear if those involved in assessing the outcomes (reviewing the charts) were blinded |
| Incomplete outcome data (attrition bias) All outcomes | High risk | Outcome data for 190 out of 350 (54.3% ) patients' charts randomised |
| Selective reporting (reporting bias) | Low risk | Unlikely. A relevant set of outcomes reported |
| Other bias | High risk | Probable contamination ‐ not adjusted by resident |
Zenni 1996.
| Methods | Cluster‐randomised trial | |
| Participants | 53 paediatric house officers and 153 parents in the paediatric house staff continuity clinic at Lucile Salter Packard Children's Hospital, Stanford, California (USA) | |
| Interventions | Structured encounter forms including checklists for developmental milestones and checklists for anticipatory guidance/preventive care | |
| Outcomes | Patient satisfaction and compliance with recommended standards Assessed at: unclear |
|
| Notes | 1 relevant comparison (reminders versus usual care) Funding: this work was performed during the tenure of a Clinician‐Scientist Award from the American Heart Association, Dallas, Texas |
|
| Risk of bias | ||
| Bias | Authors' judgement | Support for judgement |
| Random sequence generation (selection bias) | Unclear risk | The sequence generation procedure was not clearly detailed |
| Allocation concealment (selection bias) | Unclear risk | Not reported |
| Blinding of participants and personnel (performance bias) All outcomes | Unclear risk | It is not clear that patients were blinded (because of the way in which outcomes are measured this could have an impact on effect estimates) Quote: "House staff were informed that the general purpose of the study was to assess the impact of SEFs and were asked for their voluntary, confidential participation." |
| Blinding of outcome assessment (detection bias) All outcomes | Low risk | Quote: "Patient and parent names were not labeled on the audiotapes. Audio‐ tapes were scored by the 2 authors independently using structured checklists, and group assignments of subjects were blind." |
| Incomplete outcome data (attrition bias) All outcomes | Unclear risk | Of the 53 eligible residents, 50 (94%) completed both a pretest and post‐test. However very few audiotapes were analysed |
| Selective reporting (reporting bias) | Unclear risk | It was not possible to make a judgement with the information available |
| Other bias | Unclear risk | Authors tested the assumption of no more correlation between the intervention responses of patients in the same clinic group than there is between the intervention responses of those in different clinic groups, but unclear if the method used for that is appropriate |
BMD: bone mineral density; CAGE:cut‐annoy‐guilty‐eye; CAUTI: catheter‐associated urinary tract infection; CLABSI: central line‐associated blood stream infection; CDSS: clinical decision support systems; CHD: coronary heart disease; FOBT: faecal occult blood test; GHC: group health cooperative; GP: general practitioners; HIV/AIDS: human immunodeficiency virus/acquired immune deficiency syndrome; LDL: low density lipoproteins; LLT: lipid lowering therapy; MDD: major depressive disorder; NCEP: National Cholesterol Education Program ; NHLBI: National Health, Lung and Blood Institute; NSAID: non‐steroidal anti‐inflammatory drugs; OGTT: oral glucose tolerance test; OR: odds ratio; PCEP: Physician Cholesterol Education Program; PCP: primary care providers; QI: quality improvement; VIP: very important person.
Characteristics of excluded studies [ordered by study ID]
| Study | Reason for exclusion |
|---|---|
| Albada 2012 | Patient‐reminders study |
| Althabe 2008 | Multifaceted intervention |
| Ashe 2004 | Multifaceted intervention |
| Aspesi 2012 | Multifaceted intervention |
| Aspesi 2013 | Multifaceted intervention |
| Aspy 2008 | Multifaceted intervention |
| Avorn 1988 | Multifaceted intervention (mainly educational materials) |
| Baker 2011 | Computer‐generated reminders study |
| Barkun 2013 | Multifaceted intervention |
| Baskerville 2001 | Non‐reminders study |
| Bates 1995 | Computer‐generated reminders study |
| Bazian 2005 | Computer‐generated reminders study |
| Bejes 1992 | Multifaceted intervention |
| Bekkering 2005 | Multifaceted intervention |
| Berwanger 2012 | Multifaceted intervention |
| Bhatia 2013 | Multifaceted intervention |
| Bloomfield 2005 | Multifaceted intervention |
| Boekeloo 1990 | Non‐eligible population (Internal Medicine interns) |
| Brink 1989 | Computer‐generated reminder study |
| Buchner 1987 | Patient‐reminders study |
| Burack 1996 | Computer‐generated reminders study |
| Burack 1998 | Computer‐generated reminders study |
| Burack 2003 | Computer‐generated reminders study |
| Canada 2011 | Multifaceted intervention |
| Carruthers 2007 | Computer‐generated reminders study |
| Carter 2005 | Patient‐reminders study |
| Casebeer 1999 | Multifaceted intervention |
| Cayten 1983 | Non‐reminders study |
| Champion 2003 | Multifaceted intervention |
| Chanques 2006 | Ineligible study design |
| Chase 1983 | Computer‐generated reminders study |
| Chokshi 2017 | Computer‐generated reminders study |
| Christensen 2004 | Multifaceted intervention |
| Coenen 2004 | Multifaceted intervention |
| Coleman 2003 | Multifaceted intervention |
| Cox 2003 | Non‐reminder study |
| Coyle 2004 | On‐screen reminders study |
| Cranney 2008 | Multifaceted intervention |
| Crawford 2011 | Patient‐reminders study |
| Cummings 1989 | Non‐reminders study |
| Dainty 2011 | Multifaceted intervention |
| Daley 2004 | Multifaceted intervention |
| Damery 2012 | Patient‐reminders study |
| Danchaivijitr 1992 | Insufficient information to make a judgement |
| Davis 2007 | Multifaceted intervention |
| Derose 2005 | Computer‐generated reminders study |
| Desai 2013 | Computer‐generated reminders study |
| Dexter 2004 | Computer‐generated reminders study |
| Dijkstra 2006 | Multifaceted intervention |
| Duke 2013 | On‐screen reminders study |
| Eccles 2001 | Computer‐generated reminders study |
| Eschmann 2015 | On‐screen reminders study |
| Feder 1999 | Multifaceted intervention |
| Fiscella 2010 | Multifaceted intervention |
| Flottorp 2002 | Non‐reminders study |
| Foy 2011 | Computer‐generated reminders study |
| Frances 2001 | Mixed (computerised and manual paper) reminders study |
| Frasure 2014 | Multifaceted intervention |
| Frazier 1991 | Non‐reminders study |
| Friedmann 2006 | Multifaceted intervention |
| Gans 1994 | Multifaceted intervention |
| Garcia 2009 | Computer‐generated reminders study |
| Gilutz 2009 | Computer‐generated reminders study |
| Girotti 1990 | Ineligible study design (seems to be a kind of retrospective audit) |
| Goff 2003 | Multifaceted intervention |
| Goldstein 2005 | Computer‐generated reminders |
| Goodey 2000 | Non‐reminders study |
| Hagerman 1978 | Patient‐reminders study |
| Hajek 2002 | Non‐reminders study |
| Halbert 1999 | Patient‐reminders study |
| Hambidge 2004 | Multifaceted intervention |
| Heard 2002 | Non‐reminders study |
| Heidenreich 2005 | Computer‐generated reminders study |
| Heidenreich 2007 | Computer‐generated reminders study |
| Heiman 2004 | Computer‐generated reminders study |
| Heranney 2011 | Patient‐reminders study |
| Hiatt 2001 | Multifaceted intervention |
| Hilleman 2001 | Multifaceted intervention |
| Holton 2010 | Multifaceted intervention |
| Huang 2004 | Non‐reminder study |
| Ivers 2012 | Patient‐reminders study |
| Jones 1996 | Multifaceted intervention |
| Karikoski 2003 | Patient‐reminders study |
| Kaye 2005 | Multifaceted intervention |
| Kinsman 2009 | Multifaceted intervention |
| Kirwin 2010 | Computer‐generated reminders study |
| Lafata 2007 | Computer‐generated reminders study |
| Lainscak 2016 | Computer‐generated reminders study |
| Lantz 1995 | Patient‐reminders study |
| Lesprit 2010 | Non‐reminders study |
| Levy 2013 | Mixed (computerised and manual paper) reminders study |
| Linares 2011 | Computer‐generated reminders study |
| Lipkus 1999 | Patient‐reminders study |
| Luszczynska 2012 | Non‐reminders study |
| MacIntyre 2003 | Computerised‐reminder study |
| MacLean 2009 | Multifaceted intervention |
| Majumdar 2004 | Multifaceted intervention |
| Majumdar 2008 | Multifaceted intervention |
| Majumdar 2012 | Computer‐generated reminders study |
| Manfredi 1998 | Multifaceted intervention |
| Manns 2012 | Ineligible comparator |
| Margolis 1992 | Multifaceted intervention |
| Martin 2004 | Multifaceted intervention |
| Martin 2007 | Multifaceted intervention |
| McAlister 2009 | Computer‐generated reminders study |
| McDermott 2001 | Non‐reminders study |
| McDermott 2003 | Non‐reminders study |
| Meulepas 2007 | Non‐reminders study |
| Myers 2004 | Multifaceted intervention |
| Naunton 2004 | Multifaceted intervention |
| NCT00355004 2013 | On‐screen reminders study |
| NCT00699439 2009 | Computer‐generated reminders study |
| NCT01057888 2015 | Patient‐reminders study |
| NCT01126034 2015 | Computer‐generated reminders study |
| NCT01207232 2010 | Patient‐reminders study |
| NCT01325116 2014 | Patient‐reminders study |
| NCT01352390 2017 | Patient‐reminders study |
| NCT01401621 2015 | Patient‐reminders study |
| NCT01729429 2015 | Patient‐reminders study |
| NCT02411006 2016 | Patient‐reminders study |
| NCT02411032 2017 | Patient‐reminders study |
| NCT02438943 2015 | Multifaceted intervention |
| NCT02564653 2015 | Multifaceted intervention |
| NCT02640521 2016 | Multifaceted intervention |
| NCT02927743 2016 | Non‐reminders study |
| Nguyen 2000 | Multifaceted intervention |
| Ockene 1994 | Multifaceted intervention |
| Okano 1995 | Non‐reminders study |
| Olivarius 2001 | Multifaceted intervention |
| Olson 2009 | On‐screen reminders study |
| Ong 2011 | Ineligible participants |
| Otero‐Sabogal 2006 | Multifaceted intervention |
| Paskett 1999 | Multifaceted intervention |
| Peccoralo 2012 | Non‐reminders study |
| Piazza 2013 | No paper reminders study |
| Polinski 2011 | Non‐reminders study |
| Pollack 1991 | Multifaceted intervention |
| Puech 1998 | Patient‐reminders study |
| Raebel 2005 | Computer‐generated reminders study |
| Ray 2001 | Multifaceted intervention |
| Rimer 1999 | Computer‐generated reminders study |
| Robie 1988 | Multifaceted intervention |
| Rodewald 1999 | Multifaceted intervention |
| Rossignol 2005 | Non‐reminders study |
| Roy 2014 | Multifaceted intervention |
| Saint 2005 | Multifaceted intervention |
| Scheel 2002 | Multifaceted intervention |
| Sellors 2004 | Computer‐generated reminders study |
| Shelley 2010 | Multifaceted intervention |
| Shevlin 2002 | Multifaceted intervention |
| Shirai 2012 | Patient‐reminders study |
| Simon 2001 | Patient‐reminders study |
| Solomon 2004 | Multifaceted intervention |
| Solomon 2007 | Non‐reminders study |
| Soureti 2011 | Patient‐reminders study |
| Srikrajang 2005 | Multifaceted intervention |
| Stiell 2009 | Multifaceted intervention |
| Stiell 2010 | Multifaceted intervention |
| Stone 2005 | Non‐paper reminder study (reminder delivered by telephone) |
| Szilagyi 2015 | On‐screen reminders study |
| Taylor 1999 | Multifaceted intervention |
| Thomas 2006 | Computer‐generated reminders study |
| Thomas 2007 | Patient‐reminders study |
| Thompson 2008 | Multifaceted intervention |
| Tierney 1986 | Computer‐generated reminders study |
| Tseng 2002 | Multifaceted intervention |
| Valanis 2002 | Non‐reminders study |
| Van der Sanden 2005 | Multifaceted intervention |
| Van der Weijden 2001 | Non‐reminders study |
| Vissers 1996 | On‐screen reminders study |
| Wadland 2007 | Non‐reminders study |
| Waldorff 2009 | Multifaceted intervention |
| Wee 2016 | Multifaceted intervention |
| Weiss 2011 | Non‐paper reminders study |
| Zeler 1992 | Multifaceted intervention |
Characteristics of ongoing studies [ordered by study ID]
Feitosa‐Souza 2018.
| Trial name or title | Creation of reminders to improve nurses' practice regarding registration and use of the child's book |
| Methods | Cluster‐randomised trial |
| Participants | 10 primary care nurses in Brazil |
| Interventions | Chart reminders |
| Outcomes | Adherence to guidelines on child development |
| Starting date | 2018 |
| Contact information | |
| Notes |
Differences between protocol and review
Adjustment by baseline differences
Since important baseline differences between intervention and control groups are frequently found in cluster‐randomised trials, in the protocol our primary analysis was based on estimates of effect adjusted for baseline differences. For dichotomous outcome data, we had planned to calculate the adjusted risk difference (RD) as the difference in compliance after the intervention minus the difference before the intervention. For continuous outcome data, we had planned to calculate adjusted change relative to the control group as the post‐intervention difference in means minus the baseline difference in means divided by the baseline control group mean. We had therefore initially planned to include in the primary statistical analysis only those studies providing data on baseline compliance (performance). However, because only 23 out of 57 studies included some sort of outcome baseline data, we based the primary analysis on post‐intervention differences between intervention and control groups, and we tested this in a sensitivity analysis (See Effects of interventions).
Assessment of publication bias
We had planned to use a funnel plot to visually explore the risk of publication bias, using the number of health professionals as a proxy for the precision of the estimate and the adjusted RD as the treatment effect. We did not do this because the number of health professionals/practices in most of the studies was not reported in a reliable way. We also used the unadjusted post‐intervention RD as the primary treatment effect measure.
Data synthesis
We considered weighting the median effect size of each study by the number of health professionals involved in the trial, in order to ensure that very small trials did not contribute the same weight to the overall estimates as larger trials. Thus, we would have obtained a weighted median‐adjusted RD or weighted median‐adjusted change relative to baseline control as a summary effect size. However, because of the unreliable reporting of the number of health professionals/practices in the included studies we did not perform this analysis. For the primary analysis we had planned to exclude studies at high risk of bias, but we finally included all the studies in the primary analysis, and tested this assumption in a sensitivity analysis (See Effects of interventions).
Effect modifiers
We had planned to use reminder delivery mode (e.g. checklist versus coloured stickers), and baseline adherence (studies with high adherence rates in control groups versus those with low adherence) as effect modifiers to be used in subgroup analyses. However, we were not able to identify the delivery mode in a reliable way, or to clearly define a threshold for adherence rates, in view of the diversity of populations, clinical conditions and outcomes included in the studies.
Changes in outcomes terms
Because of a lack of consistency in the terms used to label the outcomes, we have combined dichotomous process adherence outcomes and continuous process outcomes under the label "professional practice outcomes", and dichotomous clinical outcomes and continuous clinical outcomes under the label "patient outcomes".
Electronic searches
The EPOC register was searched as part of CENTRAL.
Authorship
The following authors left the review author team: Alberto Romero, Flora Haaijer Ruskamp, Javiera Leniz, Jeremy Wyatt, Michael E Green, Nicholas Hicks, Paramjit Gill, Petra Denig, Pierre Durieux, Rachel Rowe, Richard Gordon. Carla Castañon and Javiera Leniz Martelli joined the review author team.
Contributions of authors
TP led the writing of the protocol. JMG, NC, JL and CC provided comments and feedback. For the full review, TP, NC, JL and CC screened records for eligibility. JMG acted as arbiter when disagreements arose. TP, JL and CC abstracted data, undertook analysis and wrote up the review. JMG contributed to the interpretation of results.
Sources of support
Internal sources
No sources of support supplied
External sources
EBM_Network, FIS‐G03/90 Program, Spain.
Departamento de Medicina Familiar, P Universidad Católica de Chile, Chile.
Declarations of interest
TP is an editor with the Cochrane Effective Practice and Organisation of Care (EPOC) group.
JMG: none known.
NC: none known.
CC: none known.
JLM: none known.
New
References
References to studies included in this review
Ayanian 2008 {published data only}
- Ayanian JZ, Sequist TD, Zaslavsky AM, Johannes RS. Physician reminders to promote surveillance colonoscopy for colorectal adenomas: a randomized controlled trial. Journal of General Internal Medicine 2008;23(6):762‐7. [DOI] [PMC free article] [PubMed] [Google Scholar]
Baker 1997 {published data only}
- Baker R, Farooqi A, Tait C, Walsh S. Randomised controlled trial of reminders to enhance the impact of audit in general practice on management of patients who use benzodiazepines. Quality in Health Care 1997;6(1):14‐18. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Baker R, Farooqi A, Tait C, Walsh S. The effectiveness of reminders. A randomised controlled trial of reminders with feedback compared to feedback alone for implementing guidelines for the management of patients receiving long term benzodiazepines in general practice. Eli Lilly National Clinical Audit. Department of General Practice & Primary Health Care, University of Leicester 1996.
Bankhead 2001 {published data only}
- Bankhead C, Richards SH, Peters TJ, Sharp DJ, Hobbs FD, Brown J, et al. Improving attendance for breast screening among recent non‐attenders: a randomised controlled trial of two interventions in primary care. Journal of Medical Screening 2001;8(2):99‐105. [DOI] [PubMed] [Google Scholar]
Beaulieu 2004 {published data only}
- Beaulieu MD, Brophy J, Jacques A, Blais R, Battista R, Lebeau R. Drug treatment of stable angina pectoris and mass dissemination of therapeutic guidelines: a randomized controlled trial. Quarterly Journal of Medicine 2004;97(1):21‐31. [DOI] [PubMed] [Google Scholar]
Bishop 2006 {published data only}
- Bishop PB, Wing PC. Knowledge transfer in family physicians managing patients with acute low back pain: a prospective randomized control trial. Spine Journal 2006;6(3):282‐8. [DOI] [PubMed] [Google Scholar]
Boekeloo 2003 {published data only}
- Boekello BO, Bobbin MP, Lee WI, Worrell KD, Hamburger EK, Russek‐Cohen E. Effect of patient priming and primary care provider prompting on adolescent‐provider communication about alcohol. Archives of Pediatrics & Adolescent Medicine 2003;157(5):433‐9. [DOI] [PubMed] [Google Scholar]
Boltri 2007 {published data only}
- Boltri JM, Okosun I, Davis‐Smith YM, Seale JP, Roman P, Tobin BW. A simple nurse‐based prompt increases screening and prevention counseling for diabetes. Diabetes Research and Clinical Practice 2007;75(1):81‐7. [DOI] [PubMed] [Google Scholar]
Bouza 2004 {published data only}
- Bouza E, Sousa D, Muñoz P, Rodríguez‐Créixems M, Fron C, Garcia Lechuz J. Bloodstream infections: a trial of the impact of different methods of reporting positive blood culture results. Clinical Infectious Diseases 2004;39(8):1161‐9. [DOI] [PubMed] [Google Scholar]
Bray 2002 {published data only}
- Bray TJ, Salil P, Weiss HA, Porter JD. Intervention to promote appropriate blood use in India. Transfusion Medicine 2002;12(6):357‐66. [DOI] [PubMed] [Google Scholar]
Buchsbaum 1993 {published data only}
- Buchsbaum DG, Buchanan RG, Lawton MJ, Elswick RK, Schnoll SH. A program of screening and prompting improves short‐term physician counseling of dependent and non‐dependent harmful drinkers. Archives of Internal Medicine 1993;153:1573‐7. [PubMed] [Google Scholar]
Burns 2002 {published data only}
- Burns IT, Zimmerman RK, Santibanez TA. Effectiveness of a chart prompt about immunizations in an urban health center. Journal of Family Practice 2002;51(12):1018. [PubMed] [Google Scholar]
Cannon 2000 {published data only}
- Cannon DS, Allen SN. A comparison of the effects of computer and manual reminders on compliance with a mental health clinical practice guideline. JAMIA 2000;7(2):196‐203. [DOI] [PMC free article] [PubMed] [Google Scholar]
Chadha 2000 {published data only}
- Chadha Y, Mollison J, Howie F, Grimshaw J, Hall M, Russell I. Guidelines in gynaecology: evaluation in menorrhagia and in urinary incontinence. British Journal of Obstetrics and Gynaecology 2000;107(4):535‐43. [DOI] [PubMed] [Google Scholar]
Chan 2002 {published data only}
- Chan L, MacLehose RF, Houck PM. Impact of physician reminders on the use of influenza vaccinations: a randomized trial. Archives of Physical Medicine and Rehabilitation 2002;83(3):371‐5. [DOI] [PubMed] [Google Scholar]
Cheney 1987 {published data only}
- Cheney C, Ramsdel JW. Effect of medical records' checklists on implementation of periodic health measures. American Journal of Medicine 1987;83(1):129‐36. [DOI] [PubMed] [Google Scholar]
Cibere 2002 {published data only}
- Cibere J, Sibley JT, Haga M. Rheumatologists' adherence to guidelines for misoprostol use in patients at high risk for nonsteroidal antiinflammatory drug gastropathy. Journal of Rheumatology 2002;29(2):339‐46. [PubMed] [Google Scholar]
Clark 2009 {published data only}
- Clark HD, Graham ID, Karovitch A, Keely EJ. Do postal reminders increase postpartum screening of diabetes mellitus in women with gestational diabetes mellitus? A randomized controlled trial. American Journal of Obstetrics and Gynecology 2009;200:634.e1‐634.e7. [DOI] [PubMed] [Google Scholar]
Cohen 1989a {published data only}
- Cohen SJ, Christen AG, Katz BP, Drook CA, Davis BJ, Smith DM, et al. Counseling medical and dental patients about cigarette smoking: the impact of nicotine gum and chart reminders. American Journal of Public Health 1987;77(3):313‐6. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Cohen SJ, Stookey GK, Katz BP, Drook CA, Christen AG. Helping smokers quit: a randomized controlled trial with private practice dentists. Journal of the American Dental Association (1939) 1989;118:41‐5. [DOI] [PubMed] [Google Scholar]
Cohen 1989b {published data only}
- Cohen SJ, Christen AG, Katz BP, Drook CA, Davis BJ, Smith DM, et al. Counseling medical and dental patients about cigarette smoking: the impact of nicotine gum and chart reminders. American Journal of Public Health 1987;77(3):313‐6. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Cohen SJ, Stookey GK, Katz BP, Drook CA, Smith DM. Encouraging primary care physicians to help smokers quit. A randomized, controlled trial. Annals of Internal Medicine 1989;110(8):648‐52. [DOI] [PubMed] [Google Scholar]
Cowan 1992 {published data only}
- Cowan JA, Heckerling PS, Parker JB. Effect of a fact sheet reminder on performance of the periodic health examination: a randomized controlled trial. American Journal of Preventive Medicine 1992;8(2):104‐9. [PubMed] [Google Scholar]
Daucourt 2003 {published data only}
- Daucourt V, Saillour‐Glenisson F, Michel P, Jutand MA, Abouelfath A. A multicenter cluster randomized controlled trial of strategies to improve thyroid function testing. Medical Care 2003;41(3):432‐41. [DOI] [PubMed] [Google Scholar]
Dubey 2006 {published data only}
- Dubey V, Mathew R, Iglar K, Moineddin R, Glazier R. Improving preventive service delivery at adult complete health check‐ups: the Preventive health Evidence‐based Recommendation Form (PERFORM) cluster randomized controlled trial. BMC Family Medicine 2006;7:44. [DOI] [PMC free article] [PubMed] [Google Scholar]
Ely 2015 {published data only}
- Ely JW, Graber MA. Checklists to prevent diagnostic errors: A pilot randomized controlled trial. Diagnosis 2015;2(3):163‐9. [DOI] [PubMed] [Google Scholar]
Emslie 1993 {published data only}
- Emslie C, Grimshaw J, Templeton A. Do clinical guidelines improve general practice management and referral of infertile couples?. BMJ 1993;306(6894):1728‐31. [DOI] [PMC free article] [PubMed] [Google Scholar]
Etter 2000 {published data only}
- Etter JF, Rielle JP, Perneger TV. Labeling smokers’ charts with a “Smoker” sticker. Results of a randomized controlled trial among private practitioners. Journal of General Internal Medicine 2000;15(6):421‐4. [DOI] [PMC free article] [PubMed] [Google Scholar]
Frame 1994 {published data only}
- Frame PS, Zimmer JG, Werth PL, Hall WJ, Eberly SW. Computer‐based vs manual health maintainance tracking. A controlled trial. Archives of Family Medicine 1994;3(7):581‐8. [DOI] [PubMed] [Google Scholar]
Gonzalez 1989 {published data only}
- Gonzalez JJ, Ranney J, West J. Nurse‐initiated health promotion prompting system in an Internal Medicine Residents' clinic. Southern Medical Journal 1989;82(3):342‐4. [DOI] [PubMed] [Google Scholar]
Grady 1997 {published data only}
- Grady KE, Lemkau JP, Lee NR, Caddell C. Enhancing mammography referral in primary care. Preventive Medicine 1997;26(6):791‐800. [DOI] [PubMed] [Google Scholar]
Halterman 2005 {published data only}
- Halterman JS, McConnochie KM, Conn KM, Yoos HL, Callahan PM, Neely TL, et al. A randomized trial of primary care provider prompting to enhance preventive asthma therapy. Archives of Pediatrics & Adolescent Medicine 2005;159(5):422‐7. [DOI] [PubMed] [Google Scholar]
Headrick 1992 {published data only}
- Headrick LA, Speroff T, Pelecanos HI, Cebul RD. Efforts to improve compliance with the National Cholesterol Education Program guidelines. Results of a randomized controlled trial. Archives of Internal Medicine 1992;152(12):2490‐6. [PubMed] [Google Scholar]
Hung 2008 {published data only}
- Hung CS, Lin JW, Hwang JJ, Tsai RY, Li AT. Using paper chart based clinical reminders to improve guideline adherence to lipid management. Journal of Evaluation in Clinical Practice 2008;14(5):861‐8. [DOI] [PubMed] [Google Scholar]
Kunz 2007 {published data only}
- Kunz R, Wegscheider K, Guyatt G, Zielinski W, Rakowsky N, Donner‐Banzhoff N, et al. Impact of short evidence summaries in discharge letters on adherence of practitioners to discharge medication. A cluster randomised controlled trial. Quality & Safety in Health Care 2007;16(6):456‐61. [DOI] [PMC free article] [PubMed] [Google Scholar]
Laprise 2009 {published data only}
- Laprise R, Thivierge R, Gosselin G, Bujas‐Bobanovic M, Vandal S, Paquette D, et al. Improved cardiovascular prevention using best CME practices: a randomized trial. Journal of Continuing Education in the Health Professions 2009;29(1):16‐31. [DOI] [PubMed] [Google Scholar]
Levine 2003 {published data only}
- Levine RS, Husaini BA, Emerson JS, Hull PC, Briggs NC, Moriarty CJ, et al. Using a nurse protocol to assure equitable delivery of cancer‐related prevention services. Cellular and Molecular Biology 2003;49(8):1229‐32. [PubMed] [Google Scholar]
Levy 2009 {published data only}
- Levy BT, Hartz A, Woodworth G, Xu Y, Sinift S. Interventions to improving osteoporosis screening: an Iowa Research Network (IRENE) study. Journal of the American Board of Family Medicine 2009;22(4):360‐7. [DOI] [PubMed] [Google Scholar]
Lilford 1992 {published data only}
- Lilford RJ, Kelly M, Baines A, Cameron S, Cave M, Guthrie K, et al. Effect of using protocols on medical care: randomised trial of three methods of taking an antenatal history. BMJ 1992;305(6863):1181‐4. [DOI] [PMC free article] [PubMed] [Google Scholar]
McIsaac 2002 {published data only}
- McIsaac WJ, Goel V, To T, Permaul JA, Low DE. Effect on antibiotic prescribing of repeated clinical prompts to use a sore throat score. Lessons from a failed community intervention study. Journal of Family Practice 2002;51(4):339‐44. [PubMed] [Google Scholar]
Montgomery 2000 {published data only}
- Montgomery AA, Fahey T, Peters TK, MacIntosh C, Sharp DJ. Evaluation of computer based clinical decision support system and risk chart for management of hypertension in primary care: randomised controlled trial. BMJ 2000;320(7236):686‐90. [DOI] [PMC free article] [PubMed] [Google Scholar]
Moore 1997 {published data only}
- Moore AA, Siu AL, Partridge JM, Hays RD, Adams J. A randomized trial of office‐based screening for common problems in older persons. American Journal of Medicine 1997;102(4):371‐8. [DOI] [PubMed] [Google Scholar]
Nendaz 2010 {published data only}
- Nendaz MR, Chopard P, Lovis C, Kucher N, Asmis LM, Dörffler J, et al. Adequacy of venous thromboprophylaxis in acutely ill medical patients (IMPART): multisite comparison of different clinical decision support systems. Journal of Thrombosis and Haemostasis 2010;8(6):1230‐4. [DOI] [PubMed] [Google Scholar]
Potter 2009 {published data only}
- Potter MB, Namvargolian Y, Hwang J, Walsh JM. Improving colorectal cancer screening: a partnership between primary care practices and the American Cancer Society. Journal of Cancer Education 2009;24(1):22‐7. [DOI] [PubMed] [Google Scholar]
Pritchard 1995 {published data only}
- Pritchard DA, Straton JA, Hyndman J. Cervical screening in general practice. Australian and New Zealand Journal of Public Health 1995;19(2):167‐72. [DOI] [PubMed] [Google Scholar]
Rattanaumpawan 2016 {published data only}
- Rattanaumpawan P, Teeratorn N, Thamlikitkul V. A cluster randomized controlled trial of the Catheter Reminder and Evaluation Program. Infection Control & Hospital Epidemiology 2016;37(2):231‐3. [NCT01797146] [DOI] [PubMed] [Google Scholar]
Renzi 2006 {published data only}
- Renzi PM, Ghezzo H, Goulet S, Dorval E, Thivierge RL. Paper stamp checklist tool enhances asthma guidelines knowledge and implementation by primary care physicians. Canadian Respiratory Journal 2006;13(4):193‐7. [DOI] [PMC free article] [PubMed] [Google Scholar]
Richards 2001 {published data only}
- Richards SH, Bankhead C, Peters TJ, Austoker J, Hobbs FD, Brown J, et al. Cluster randomised controlled trial comparing the effectiveness and cost‐effectiveness of two primary care interventions aimed at improving attendance for breast screening. Journal of Medical Screening 2001;8(2):91‐8. [DOI] [PubMed] [Google Scholar]
Roetzheim 2004 {published data only}
- Roetzheim RG, Christman LK, Jacobsen PB, Cantor AB, Schroeder J, Abdulla R, et al. A randomized controlled trial to increase cancer screening among attendees of community health centers. Annals of Family Medicine 2004;2(4):294‐300. [DOI] [PMC free article] [PubMed] [Google Scholar]
Saillour‐Glénisson 2005 {published data only}
- Saillour‐Glénisson F, Michel P, Daucourt V. Medico‐economic assessment of two methods for implementing thyroid testing guidelines [Évaluation médico‐économique de la mise en oeuvre de recommandationsde prescription des examens biologiques explorant la fonction thyroïdienne]. Revue d'Épidémiologie et de Santé Publique 2005;53:1S79‐1S88. [PubMed] [Google Scholar]
Saitz 2003 {published data only}
- Saitz R, Horton NJ, Sullivan LM, Moskowitz MA, Samet JH. Addressing alcohol problems in primary care: a cluster randomized, controlled trial of a systems intervention. The screening and intervention in primary care (SIP) study. Annuals of Internal Medicine 2003;138(5):372‐82. [DOI] [PubMed] [Google Scholar]
Scholes 2006 {published data only}
- Scholes D, Grothaus L, McClure J, Reid R, Fishman P, Sisk C, et al. A randomized trial of strategies to increase chlamydia screening in young women. Preventive Medicine 2006;43(4):343‐50. [DOI] [PubMed] [Google Scholar]
Seto 1989 {published data only}
- Seto WH, Ching PT, Fung JP, Fielding R. The role of communication in the alteration of patient‐care practices in hospital ‐ a prospective study. Journal of Hospital Infection 1989;14(1):29‐37. [DOI] [PubMed] [Google Scholar]
Shannon 2001 {published data only}
- Shannon KC, Sinacore JM, Bennett SG, Joshi AM, Sherin KM, Deitrich A. Improving delivery of preventive health care with the comprehensive annotated reminder tool (CART). Journal of Family Practice 2001;50(9):767‐71. [PubMed] [Google Scholar]
Shaw 2000 {published data only}
- Shaw JS, Samuels RC, Larusso EM, Bernstein HH. Impact of an encounter‐based prompting system on resident vaccine administration performance and immunization knowledge. Pediatrics 2000;105(4 Pt 2):978‐83. [PubMed] [Google Scholar]
Somkin 1997 {published data only}
- Somkin CP, Hiatt RA, Hurley LB, Gruskin E, Ackerson L, Larson P. The effect of patient and provider reminders on mammography and papanicolaou smear screening in a large Health Maintenance Organization. Archives of Internal Medicine 1997;157(15):1658‐64. [PubMed] [Google Scholar]
Stamos 2001 {published data only}
- Stamos TD, Shaltoni H, Girard SA, Parrillo JE, Calvin JE. Effectiveness of chart prompts to improve physician compliance with the National Cholesterol Education Program Guidelines. American Journal of Cardiology 2001;88(12):1420‐3. [DOI] [PubMed] [Google Scholar]
Strecher 1991 {published data only}
- Strecher V, O'Malley MS, Villagra VG, Campbell EE, Gonzalez JI, Irons TG, et al. Can residents be trained to counsel patients about quitting smoking? Results from a randomized trial. Journal of General Internal Medicine 1991;6(1):9‐17. [DOI] [PubMed] [Google Scholar]
Szilagyi 1996 {published data only}
- Szilagyi PG, Rodewald LE, Humiston SG, Pollard L, Klossner K, Jones AM, et al. Reducing missed opportunities for immunizations. Easier said than done. Archives of Pediatrics & Adolescent Medicine 1996;150(11):1193‐200. [DOI] [PubMed] [Google Scholar]
Thapar 2002 {published data only}
- Thapar A, Jacoby A, Richens A, Russell I, Roberts C, Porter E, et al. A pragmatic randomised controlled trial of a prompt and reminder card in the care of people with epilepsy. British Journal of General Practice 2002;52(475):93‐8. [PMC free article] [PubMed] [Google Scholar]
Vinker 2002 {published data only}
- Vinker S, Nakar S, Rosenberg E, Kitai E. The role of family physcians in increasing annual fecal occult blood test screening coverage: a prospective intervention study. Israel Medical Association Journal 2002;4(6):424‐5. [PubMed] [Google Scholar]
Walker 1999 {published data only}
- Walker NM, Mandell KL, Tsevat J. Use of chart reminders for physicians to promote discussion of advance directives in patients with AIDS. AIDS Care 1999;11(3):345‐53. [DOI] [PubMed] [Google Scholar]
Wang 1994 {published data only}
- Wang WD. Feasibility and effectiveness of a stage‐of‐change model in cigarette smoking cessation counselling. Journal of the Formosan Medical Association 1994;93:752‐7. [PubMed] [Google Scholar]
Wigton 1981 {published data only}
- Wigton RS, Zimmer JL, Wigton JH, Patil KD. Chart reminders in the diagnosis of anemia. JAMA 1981;245(17):1745‐7. [PubMed] [Google Scholar]
Yazdany 2011 {published data only}
- Yazdany T, Wong M, Bhatia NN. Improving resident screening and workup of urinary incontinence in an OB/GYN residency program: a randomized controlled study. Female Pelvic Medicine & Reconstructive Surgery 2011;17(5):242‐5. [DOI] [PubMed] [Google Scholar]
Zenni 1996 {published data only}
- Zenni EA, Robinson TN. Effects of structured encounter forms on pediatric house staff knowledge, parent satisfaction and quality of care. Archives of Pediatrics & Adolescent Medicine 1996;150(9):975‐80. [DOI] [PubMed] [Google Scholar]
References to studies excluded from this review
Albada 2012 {published data only}
- Albada A, Dulmen S, Ausems M G, Bensing J M. A pre‐visit website with question prompt sheet for counselees facilitates communication in the first consultation for breast cancer genetic counseling: findings from a randomized controlled trial. Genetics in Medicine 2012;14(5):535‐42. [DOI] [PubMed] [Google Scholar]
Althabe 2008 {published data only}
- Althabe F, Buekens P, Bergel E, Belizán JM, Campbell MK, Moss N, et al. A behavioral intervention to improve obstetrical care. New England Journal of Medicine 2008;358(18):1929‐40. [DOI] [PubMed] [Google Scholar]
Ashe 2004 {published data only}
- Ashe M, Khan K, Guy P, Kruse K, Hughes K, O'Brien P, et al. Wristwatch‐distal radial fracture as a marker for osteoporosis investigation: a controlled trial of patient education and a physician alerting system. Journal of Hand Therapy 2004;17(3):324‐8. [DOI] [PubMed] [Google Scholar]
Aspesi 2012 {published data only}
- Aspesi AV, Kauffmann G, Davis AM, Schulwolf E, Press VG, Arora V. IBCD: Effectiveness and sustainability of a checklist to improve quality of care for hospitalized general medical patients. Journal of General Internal Medicine 2012;27:S213‐4. [DOI] [PMC free article] [PubMed] [Google Scholar]
Aspesi 2013 {published data only}
- Aspesi A V, Kauffmann GE, Davis AM, Schulwolf EM, Press VG, Stupay KL, et al. IBCD: development and testing of a checklist to improve quality of care for hospitalized general medical patients. Joint Commission Journal on Quality and Patient Safety / Joint Commission Resources 2013;39(4):147‐56. [DOI] [PMC free article] [PubMed] [Google Scholar]
Aspy 2008 {published data only}
- Aspy CB, Enright M, Halstead L, Mold JW. Improving mammography screening using best practices and practice enhancement assistants: an Oklahoma Physicians Resource/Research Network(OKPRN) Study. Journal of the American Board of Family Medicine 2008;21(4):326‐33. [DOI] [PubMed] [Google Scholar]
Avorn 1988 {published data only}
- Avorn J, Soumerai SB, Taylor W, Wessels MR, Janousek J, Weiner M. Reduction of incorrect antibiotic dosing through a structured educational order form. Archives of Internal Medicine 1988;148(8):1720‐4. [PubMed] [Google Scholar]
Baker 2011 {published data only}
- Baker DW, Persell SD, Kho AN, Thompson JA, Kaiser D. The marginal value of pre‐visit paper reminders when added to a multifaceted electronic health record based quality improvement system. Journal of the American Medical Informatics Association: JAMIA 2011;18(6):805‐11. [DOI] [PMC free article] [PubMed] [Google Scholar]
Barkun 2013 {published data only}
- Barkun AN, Bhat M, Armstrong D, Dawes M, Donner A, Enns R, et al. Effectiveness of disseminating consensus management recommendations for ulcer bleeding: a cluster randomized trial. Canadian Medical Association Journal 2013;185(3):E156‐66. [DOI] [PMC free article] [PubMed] [Google Scholar]
Baskerville 2001 {published data only}
- Baskerville NB, Hogg W, Lemeux J. Process evaluation of a tailored multifaceted approach to improving preventive care. Journal of Family Practice 2001;50(3):241. [PubMed] [Google Scholar]
Bates 1995 {published data only}
- Bates DW, Kuperman GJ, Rittenberg E, Teich JM, Onderdonk A, Winkelman J, et al. Reminders for redundant tests: results of a randomized controlled trial. Proceedings of the Annual Symposium on Computer Application. 1995:935.
Bazian 2005 {published data only}
- Bazian Ltd. Automated standing orders to nurses increase influenza and pneumococcal vaccination rates among inpatients compared with reminders to physicians. Evidence‐based Healthcare and Public Health 2005;9(3):211‐2. [Google Scholar]
Bejes 1992 {published data only}
- Bejes C, Marvel MK. Attempting the improbable: offering colorectal cancer screening to all appropriate patients. Family Practice Research Journal 1992;12(1):83‐90. [PubMed] [Google Scholar]
Bekkering 2005 {published data only}
- Bekkering GE, Hendriks HJM, Tulder MW, Knol DL, Hoeijenbos M, Oostendorp RA, et al. Effect on the process of care of an active strategy to implement clinical guidelines on physiotherapy for low back pain: a cluster randomised controlled trial. Quality & Safety in Health Care 2005;14(2):107‐12. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Bekkering GE, Tulder MW, Hendriks EJ, Koopmanschap MA, Knol DL, Bouter LM, et al. Implementation of clinical guidelines on physical therapy for patients with low back pain: randomized trial comparing patient outcomes after a standard and active implementation strategy. Physical Therapy 2005;85(6):544‐55. [PubMed] [Google Scholar]
Berwanger 2012 {published data only}
- Berwanger O, Guimaraes HP, Laranjeira LN, Cavalcanti AB, Kodama AA, Zazula AD, et al. Effect of a multifaceted intervention on use of evidence‐based therapies in patients with acute coronary syndromes in Brazil: the BRIDGE‐ACS randomized trial. JAMA 2012;307(19):2041‐9. [DOI] [PubMed] [Google Scholar]
Bhatia 2013 {published data only}
- Bhatia RS, Milford CE, Picard MH, Weiner RB. An educational intervention reduces the rate of inappropriate echocardiograms on an inpatient medical service. JACC. Cardiovascular Imaging 2013;6(5):545‐55. [DOI] [PubMed] [Google Scholar]
Bloomfield 2005 {published data only}
- Bloomfield HE, Nelson DB, Ryn M, Neil BJ, Koets NJ, Basile JN, et al. A trial of education, prompts, and opinion leaders to improve prescription of lipid modifying therapy by primary care physicians for patients with ischemic heart disease. Quality & Safety in Health Care 2005;14(4):258‐63. [DOI] [PMC free article] [PubMed] [Google Scholar]
Boekeloo 1990 {published data only}
- Boekeloo BO, Becker DM, Levine DM, Belitsos PC, Pearson TA. Strategies for increasing house staff management of cholesterol with inpatients. American Journal of Preventive Medicine 1990;6(suppl 1):51‐9. [PubMed] [Google Scholar]
Brink 1989 {published data only}
- Brink SG. Provider reminders: changing information format to increase infant immunizations. Medical Care 1989;27(6):648‐53. [DOI] [PubMed] [Google Scholar]
Buchner 1987 {published data only}
- Buchner DM, Larson EB, White RF. Influenza vaccination in community elderly. A controlled trial of postcard reminders. Journal of the American Geriatrics Society 1987;35(8):755‐60. [DOI] [PubMed] [Google Scholar]
Burack 1996 {published data only}
- Burack RC, Gimotty PA, George J, Simon MS, Dews P, Moncrease A. The effect of patient and physician reminders on use of screening mammography in a health maintenance organization: results of a randomized controlled trial. Cancer 1996;78(8):1708‐21. [PubMed] [Google Scholar]
Burack 1998 {published data only}
- Burack RC, Gimotty PA, George J, McBride S, Moncrease A, Simon MS, et al. How reminders given to patients and physicians affected pap smear use in a health maintenance organization: results of a randomized controlled trial. Cancer 1998;83(12):2391‐400. [DOI] [PubMed] [Google Scholar]
Burack 2003 {published data only}
- Burack RC, Gimotty PA, Simon M, Moncrease A, Dews P. The effect of adding Pap smear information to a mammography reminder system in an HMO: results of randomized controlled trial. Preventive Medicine 2003;36:547‐54. [DOI] [PubMed] [Google Scholar]
- Sellors JW. Adding Pap smear reminders to mammogram reminders increases Pap smear uptake with no effect on mammography uptake. Evidence Based Health Care 2004;8:18‐20. [Google Scholar]
Canada 2011 {published data only}
- Canada A, Martin C, Soto S, Abanades J C, Salinero M. Random clinical trial to evaluate the effect of a multimodal intervention in hand hygiene in primary care in Madrid. BMC Proceedings. Conference: International Conference on Prevention and Infection Control, ICPIC 2011;5(Suppl 6):P257. [Google Scholar]
Carruthers 2007 {published data only}
- Carruthers SG. Reminders in echocardiography reports increased use of ß‐blockers in reduced left ventricular ejection fraction. Evidence‐Based Medicine 2007;12:185. [DOI] [PubMed] [Google Scholar]
Carter 2005 {published data only}
- Carter GL, Clover K, Whyte IM, Dawson AH, D'Este C. Postcards from the EDge project: randomised controlled trial of an intervention using postcards to reduce repetition of hospital treated deliberate self poisoning. BMJ 2005;331(7520):805. [DOI] [PMC free article] [PubMed] [Google Scholar]
Casebeer 1999 {published data only}
- Casebeer LL, Klapow JC, Centor RM, Stafford MA, Renkl LA, Mallinger AP, et al. An intervention to increase physicians' use of adherence‐enhancing strategies in managing hypercholesterolemic patients. Academic Medicine 1999;74(12):1334‐9. [DOI] [PubMed] [Google Scholar]
Cayten 1983 {published data only}
- Cayten CG, Oler J, Staroscik R, Morganroth J, Walker K, Cole L, et al. Clinical algorithms for prehospital cardiac care. Medical Care 1983;21(2):147‐56. [DOI] [PubMed] [Google Scholar]
Champion 2003 {published data only}
- Champion V, Maraj M, Hui S, Perkins AJ, Tierney W, Menon U, et al. Comparison of tailored interventions to increase mammography screening in nonadherent older women. Preventive Medicine 2003;36(2):150‐8. [DOI] [PubMed] [Google Scholar]
Chanques 2006 {published data only}
- Chanques G, Jaber S, Barbotte E, Violet S, Sebbane M, Perrigault P‐F, et al. Impact of systematic evaluation of pain and agitation in an intensive care unit. Critical Care Medicine 2006;34(6):1691‐9. [DOI] [PubMed] [Google Scholar]
Chase 1983 {published data only}
- Chase CR, Vacek PM, Shinozaki T, Giard AM, Ashikaga T. Medical information management: improving the transfer of research results to presurgical evaluation. Medical Care 1983;21(4):410‐24. [DOI] [PubMed] [Google Scholar]
Chokshi 2017 {published data only}
- Chokshi M, McNamara RL, Rajeswaran Y, Lampert R. Effect of a reminder statement on echocardiography reports on referrals for implantable cardioverter‐defibrillators for primary prevention. American Journal of Cardiology 2017;119(3):478‐82. [DOI] [PubMed] [Google Scholar]
Christensen 2004 {published data only}
- Christensen MB, Christensen B, Mortensen JT, Olesen F. Intervention among frequent attenders of the out‐of‐hours service: a stratified cluster randomized controlled trial. Scandinavian Journal of Primary Health Care 2004;22(3):180‐6. [DOI] [PubMed] [Google Scholar]
Coenen 2004 {published data only}
- Coenen S, Royen P, Michiels B, Denekens J. Optimizing antibiotic prescribing for acute cough in general practice: a cluster‐randomized controlled trial. Journal of Antimicrobial Chemotherapy 2004;54(3):661‐72. [DOI] [PubMed] [Google Scholar]
Coleman 2003 {published data only}
- Coleman EA, Lord J, Heard J, Coon S, Cantrell M, Mohrmann C, et al. The Delta Project: increasing breast cancer screening among rural minority and older women by targeting rural healthcare providers. Oncology Nursing Forum 2003;30(4):669‐77. [DOI] [PubMed] [Google Scholar]
Cox 2003 {published data only}
- Cox JL, Wheatley M, Nemis‐White J. Optimizing anti‐coagulant therapy: a five‐year study in Nova Scotia has identified ways to improve physicians' prescribing practices for atrial fibrillation therapy. Canadian Nurse 2003;99(6):16‐9. [PubMed] [Google Scholar]
Coyle 2004 {published data only}
- Coyle CM, Currie BP. Improving the rates of inpatient pneumococcal vaccination: impact of standing orders versus computerized reminders to physicians. Infection Control and Hospital Epidemiology 2004;25(11):904‐7. [DOI] [PubMed] [Google Scholar]
Cranney 2008 {published data only}
- Cranney A, Lam M, Ruhland L, Brison R, Godwin M, Harrison MM, et al. A multifaceted intervention to improve treatment of osteoporosis in postmenopausal women with wrist fractures: a cluster randomized trial. Osteoporosis International 2008;19(12):1733‐40. [DOI] [PubMed] [Google Scholar]
Crawford 2011 {published data only}
- Crawford N, Royle J, Sonja Elia RN, South M, Buttery J. Effect of a postcard immunisation reminder in hospital outpatients: a randomised controlled trial. Journal of Paediatrics and Child Health. Conference: 2011 Paediatrics and Child Health Annual Meeting 2011;47:20. [Google Scholar]
Cummings 1989 {published data only}
- Cummings SR, Richard RJ, Duncan CL, Hansen B, Martin RV, Gerbert B, et al. Training physicians about smoking cessation: a controlled trial in private practices. Journal of General Internal Medicine 1989;4(6):482‐9. [DOI] [PubMed] [Google Scholar]
Dainty 2011 {published data only}
- Dainty KN, Scales DC, Brooks SC, Needham DM, Dorian P, Ferguson N, et al. A knowledge translation collaborative to improve the use of therapeutic hypothermia in post‐cardiac arrest patients: protocol for a stepped wedge randomized trial. Implementation Science 2011;6:4. [DOI] [PMC free article] [PubMed] [Google Scholar]
Daley 2004 {published data only}
- Daley MF, Steiner JF, Kempe A, Beaty BL, Pearson KA, Jones JS, et al. Quality improvement in immunization delivery following an unsuccessful immunization recall. Ambulatory Pediatrics 2004;4(3):217‐23. [DOI] [PubMed] [Google Scholar]
Damery 2012 {published data only}
- Damery S, Smith S, Clements A, Holder R, Nichols L, Draper H, et al. Evaluating the effectiveness of GP endorsement on increasing participation in the NHS Bowel Cancer Screening Programme in England: study protocol for a randomized controlled trial. Trials 2012;13:18. [DOI] [PMC free article] [PubMed] [Google Scholar]
Danchaivijitr 1992 {published data only}
- Danchaivijitr S, Chokloikaew S, Tangtrakool T, Waitayapiches S. Does indication sheet reduce unnecessary urethral catheterization?. Journal of the Medical Association of Thailand 1992;75 Suppl 2:1‐5. [PubMed] [Google Scholar]
Davis 2007 {published data only}
- Davis JC, Guy P, Ashe MC, Liu‐Ambrose T, Khan K. HipWatch: Osteoporosis investigation and treatment after a hip fracture: a 6‐month randomized controlled trial. Journals of Gerontology. Series A, Biological Sciences and Medical Sciences 2007;62A(8):888‐91. [DOI] [PubMed] [Google Scholar]
Derose 2005 {published data only}
- Derose SF, Dudl JR, Benson VM, Contreras R, Nakahiro RK, Ziel FH. Point‐of‐service reminders for prescribing cardiovascular medications. American Journal of Managed Care 2005;11(5):298‐304. [PubMed] [Google Scholar]
Desai 2013 {published data only}
- Desai S, Szent‐Gyorgyi L, Turchin A, Lu B, Bogdanova AA, Weinblatt M, et al. Increasing pneumococcal vaccination for immunosuppressed patients: a cluster quality improvement trial. Arthritis and Rheumatism 2012;64:S874. [DOI] [PubMed] [Google Scholar]
- Desai SP, Lu B, Szent‐Gyorgyi LE, Bogdanova AA, Turchin A, Weinblatt M, et al. Increasing pneumococcal vaccination for immunosuppressed patients: a cluster quality improvement trial. Arthritis and Rheumatism 2013;65(1):39‐47. [DOI] [PubMed] [Google Scholar]
Dexter 2004 {published data only}
- Dexter PR, Perkins SM, Maharry KS. Inpatient computer‐based standing orders vs physician reminders to increase influenza and pneumococcal vaccination rates: a randomized trial. JAMA 2004;292(19):2366‐71. [DOI] [PubMed] [Google Scholar]
Dijkstra 2006 {published data only}
- Dijkstra RF, Niessen LW, Braspenning JC, Adang E, Grol RT. Patient‐centred and professional‐directed implementation strategies for diabetes guidelines: a cluster‐randomized trial‐based cost‐effectiveness analysis. Diabetic Medicine 2006;23(2):164‐70. [DOI] [PubMed] [Google Scholar]
Duke 2013 {published data only}
- Duke J. Enhancing drug interaction alerts with contextual laboratory data. Clinical and Translational Science. Conference: Translational Science 2012. Washington DC, 2012; Vol. 5:159.
- Duke JD, Li X, Dexter P. Adherence to drug‐drug interaction alerts in high‐risk patients: a trial of context‐enhanced alerting. Journal of the American Medical Informatics Association: JAMIA 2013;20(3):494‐8. [DOI] [PMC free article] [PubMed] [Google Scholar]
Eccles 2001 {published data only}
- Eccles M, Steen N, Grimshaw J, Thomas L, McNamee P, Soutter J, et al. Effect of audit and feedback, and reminder messages on primary care radiology referrals: a randomised trial. Lancet 2001;357(9266):1406‐9. [DOI] [PubMed] [Google Scholar]
Eschmann 2015 {published data only}
- Eschmann E, Beeler PE, Blaser J. Impact of specific alerts in potassium‐increasing drug‐drug interactions. Studies in Health Technology and Informatics 2015;216:949. [PubMed] [Google Scholar]
Feder 1999 {published data only}
- Feder G, Griffiths C, Eldridge S, Spence M. Effect of postal prompts to patients and general practitioners on the quality of primary care after a coronary event (POST): randomised controlled trial. BMJ 1999;318(7197):1522‐6. [DOI] [PMC free article] [PubMed] [Google Scholar]
Fiscella 2010 {published data only}
- Fiscella K, Yosha A, Hendren SK, Humiston S, Winters P, Ford P, et al. Get screened: a pragmatic randomized controlled trial to increase mammography and colorectal cancer screening in a large, safety net practice. BMC Health Services Research 2010;10:280. [DOI] [PMC free article] [PubMed] [Google Scholar]
Flottorp 2002 {published data only}
- Flottorp S, Oxman AD, Håvelsrud K, Treweek S, Herrin J. Cluster randomised controlled trial of tailored interventions to improve the management of urinary tract infections in women and sore throat. BMJ 2002;325(7360):367. [DOI] [PMC free article] [PubMed] [Google Scholar]
Foy 2011 {published data only}
- Foy R, Eccles MP, Hrisos S, Hawthorne G, Steen N, Gibb I, et al. A cluster randomised trial of educational messages to improve the primary care of diabetes. Implementation Science 2011;6:129. [DOI] [PMC free article] [PubMed] [Google Scholar]
Frances 2001 {published data only}
- Frances CD, Alperin P, Adler JS, Grady D. Does a fixed physician reminder system improve the care of patients with coronary artery disease? A randomized controlled trial. Western Journal of Medicine 2001;175(3):165‐6. [DOI] [PMC free article] [PubMed] [Google Scholar]
Frasure 2014 {published data only}
- Frasure JS. The effectiveness of four translation strategies on nurses' adoption of an evidence‐based bladder protocol. Journal of Neuroscience Nursing 2014;46(4):218‐26. [DOI] [PubMed] [Google Scholar]
Frazier 1991 {published data only}
- Frazier LM, Brown JT, Divine GW, Fleming GR, Philips NM, Siegal WC, et al. Can physician education lower the cost of prescription drugs? A prospective, controlled trial. Annals of Internal Medicine 1991;115(2):116‐21. [DOI] [PubMed] [Google Scholar]
Friedmann 2006 {published data only}
- Friedmann PD, Rose J, Hayaki J, Ramsey S, Charuvastra A, Dube C, et al. Training primary care clinicians in maintenance care for moderated alcohol use. Journal of General Internal Medicine 2006;21(12):1269‐75. [DOI] [PMC free article] [PubMed] [Google Scholar]
Gans 1994 {published data only}
- Gans KM, Lapane KL, Lasater TM, Carleton RA. Effects of intervention on compliance to referral and lifestyle recommendations given at cholesterol screening programs. American Journal of Preventive Medicine 1994;10(5):275‐82. [PubMed] [Google Scholar]
Garcia 2009 {published data only}
- Garcia DA, Highfill J, Finnerty K, Varoz E, McConkey S, Hutchinson K, et al. A prospective, controlled trial of a pharmacy‐driven alert system to increase thromboprophylaxis rates in medical inpatients. Blood Coagulation & Fibrinolysis 2009;20(7):541‐5. [DOI] [PubMed] [Google Scholar]
Gilutz 2009 {published data only}
- Gilutz H, Novack L, Shvartzman P, Zelingher J, Bonneh D Y, Henkin Y, et al. Computerized community cholesterol control(4C): Meeting the challenge of secondary prevention. Israel Medical Association Journal 2009;11(1):23‐9. [PubMed] [Google Scholar]
Girotti 1990 {published data only}
- Girotti MJ, Fodoruk S, Irvine‐Meek J, Rotstein OD. Antibiotic handbook and pre‐printed perioperative order forms for surgical antibiotic prophylaxis: do they work?. Canadian Journal of Surgery. Journal Canadien de Chirurgie 1990;33(5):385‐8. [PubMed] [Google Scholar]
Goff 2003 {published data only}
- Goff DC, Gu L, Cantley LK, Sheedy DJ, Cohen SJ. Quality of care for secondary prevention for patients with coronary heart disease: results of the Hastening the Effective Application of Research through Technology (HEART) trial. American Heart Journal 2003;146(6):1045‐51. [DOI] [PubMed] [Google Scholar]
Goldstein 2005 {published data only}
- Goldstein MK, Lavori P, Coleman R, Advani A, Hoffman BB. Improving adherence to guidelines for hypertension drug prescribing: cluster‐randomized controlled trial of general versus patient‐specific recommendations. American Journal of Managed Care 2005;11(11):677‐85. [PubMed] [Google Scholar]
Goodey 2000 {published data only}
- Goodey RD, Brickley MR, Hill CM, Shepherd JP. A controlled trial of three referral methods for patients with third molars. British Dental Journal 2000;189(10):556‐60. [DOI] [PubMed] [Google Scholar]
Hagerman 1978 {published data only}
- Hagerman GA. Testing the mailed appointment reminder in Family Practice. Journal of Family Practice 1978;7(1):199‐201. [PubMed] [Google Scholar]
Hajek 2002 {published data only}
- Hajek P, Taylor TZ, Mills P. Brief intervention during hospital admission to help patients to give up smoking after myocardial infarction and bypass surgery: randomised controlled trial. BMJ 2002;324(7329):87‐9. [DOI] [PMC free article] [PubMed] [Google Scholar]
Halbert 1999 {published data only}
- Halbert RJ, Leung K, Nichol JM, Legorreta AP. Effect of multiple patient reminders in improving diabetic retinopathy. A randomized trial. Diabetes Care 1999;22(5):752‐5. [DOI] [PubMed] [Google Scholar]
Hambidge 2004 {published data only}
- Hambidge SJ, Davidson AJ, Phibbs SL, Chandramouli V, Zerbe G, LeBaron CW, et al. Strategies to improve immunization rates and well‐child care in a disadvantaged population. A cluster randomized controlled trial. Archives of Pediatrics & Adolescent Medicine 2004;158(2):162‐9. [DOI] [PubMed] [Google Scholar]
Heard 2002 {published data only}
- Heard A, Kalucy E, Richardson D, Battersby MW, Frith P, McGowan C. An evaluation of the care planning process in coordinated care trials: what differences do care plans make to service provision?. Australian Journal of Primary Health 2002;8(1):52‐6. [Google Scholar]
Heidenreich 2005 {published data only}
- Heidenreich P. Improving heart failure care with a reminder attached to the echocardiography report. American Journal of Medicine 2008;121(10):853‐4. [DOI] [PubMed] [Google Scholar]
- Heidenreich PA, Chacko M, Goldstein MK, Atwood E. ACE inhibitor reminders attached to echocardiography reports of patients with reduced left ventricular ejection fraction. American Journal of Medicine 2005;118(9):1034‐7. [DOI] [PubMed] [Google Scholar]
Heidenreich 2007 {published data only}
- Carruthers SG. Reminders in echocardiography reports increased use of ß‐blockers in reduced left ventricular ejection fraction. Evidence‐Based Medicine 2007;12:185. [DOI] [PubMed] [Google Scholar]
- Heidenreich P. Improving heart failure care with a reminder attached to the echocardiography report. American Journal of Medicine 2008;121(10):853‐4. [DOI] [PubMed] [Google Scholar]
- Heidenreich PA, Gholami P, Sahay A, Massie B, Goldstein MK. Clinical reminders attached to echocardiography reports of patients with reduced left ventricular ejection fraction increase use of ß‐Blockers: a randomized trial. Circulation 2007;115:2829‐34. [DOI] [PubMed] [Google Scholar]
Heiman 2004 {published data only}
- Heiman H, Bates DW, Fairchild D, Shaykevich S, Lehmann LS. Improving completion of advance directives in the primary care setting: a randomized controlled trial. American Journal of Medicine 2004;117(5):318‐24. [DOI] [PubMed] [Google Scholar]
Heranney 2011 {published data only}
- Heranney D, Fender M, Velten M, Baldauf JJ. A prospective randomized study of two reminding strategies: telephone versus mail in the screening of cervical cancer in women who did not initially respond. Acta Cytologica 2011;55(4):334‐40. [DOI] [PubMed] [Google Scholar]
Hiatt 2001 {published data only}
- Hiatt RA, Pasick RJ, Stewart S, Bloom J, Davis P, Gardiner P, et al. Community‐based cancer screening for underserved women: design and baseline findings from the Breast and Cervical Cancer Intervention Study. Preventive Medicine 2001;33(3):190‐203. [DOI] [PubMed] [Google Scholar]
Hilleman 2001 {published data only}
- Hilleman DE, Monaghan MS, Ashby CL, Mashni JE, Woolley K, Amato CM. Physician‐prompting statin therapy intervention improves outcomes in patients with coronary heart disease. Pharmacotherapy 2001;21(11):1415‐21. [DOI] [PubMed] [Google Scholar]
Holton 2010 {published data only}
- Holton CH, Beilby JJ, Harris MF, Harper CE, Proudfoot JG, Ramsay EN, et al. Systematic care for asthma in Australian general practice: a randomised controlled trial. Medical Journal of Australia 2010;193(6):332‐7. [DOI] [PubMed] [Google Scholar]
Huang 2004 {published data only}
- Huang W‐C, Wann S‐R, Lin S‐L, Kunin CM, Kung M‐H, Lin C‐H, et al. Catheter‐associated urinary tract infections in Intensive Care Units can be reduced by prompting physicians to remove unnecessary catheters. Infection Control and Hospital Epidemiology 2004;25(11):974‐8. [DOI] [PubMed] [Google Scholar]
Ivers 2012 {published data only}
- Ivers NM, Schwalm JD, Grimshaw JM, Witteman H, Taljaard M, Zwarenstein M, et al. Delayed educational reminders for long‐term medication adherence in ST‐elevation myocardial infarction (DERLA‐STEMI): protocol for a pragmatic, cluster‐randomized controlled trial. Implementation Science 2012;7:54. [DOI] [PMC free article] [PubMed] [Google Scholar]
Jones 1996 {published data only}
- Jones DL, Kroenke K, Landry FJ, Tomich DJ, Ferrel RJ. Cost savings using a stepped‐care prescribing protocol for nonsteroidal anti‐inflammatory drugs. JAMA 1996;275(12):926‐30. [PubMed] [Google Scholar]
Karikoski 2003 {published data only}
- Karikoski A, Ilanne‐Parikka P, Murtomaa H. Oral health promotion among adults with diabetes in Finland. Community Dentistry and Oral Epidemiology 2003;31(6):447‐53. [DOI] [PubMed] [Google Scholar]
Kaye 2005 {published data only}
- Kaye KI, Welch SA, Graudins LV, Graudins A, Rotem T, Davis SR, et al. Pethidine in emergency departments: promoting evidence‐based prescribing. Medical Journal of Australia 2005;183(3):129‐33. [PubMed] [Google Scholar]
Kinsman 2009 {published data only}
- Kinsman LD, Buykx P, Humphreys JS, Snow PC, Willis J. A cluster randomised trial to assess the impact of clinical pathways on AMI management in rural Australian emergency departments. BMC Health Services Research 2009;9:83. [DOI] [PMC free article] [PubMed] [Google Scholar]
Kirwin 2010 {published data only}
- Kirwin JL, Cunningham RJ, Sequist TD. Pharmacist recommendations to improve the quality of diabetes care: a randomized controlled trial. Journal of Managed Care Pharmacy 2010;16(2):104‐13. [DOI] [PMC free article] [PubMed] [Google Scholar]
Lafata 2007 {published data only}
- Lafata JE, Kolk D, Peterson EL, McCarthy BD, Weiss TW, Chen YT, et al. Improving osteoporosis screening: results from a randomized cluster trial. Journal of General Internal Medicine 2007;22(3):346‐51. [DOI] [PMC free article] [PubMed] [Google Scholar]
Lainscak 2016 {published data only}
- Lainscak M, Roblek T, Deticek A, Leskovar B, Horvat M, Belic A, et al. Clinical‐pharmacist intervention reduces clinically relevant drug‐drug interactions in patients with heart failure: a randomized, double‐blind, controlled trial. European Journal of Heart Failure 2016;18:187. [DOI] [PubMed] [Google Scholar]
Lantz 1995 {published data only}
- Lantz PM, Stencil D, Lippert MT, Beversdorf S, Jaros L, Remington PL. Breast and cervical cancer screening in a low‐income managed care sample: the efficacy of physician letters and phone calls. American Journal of Public Health 1995;85(6):834‐6. [DOI] [PMC free article] [PubMed] [Google Scholar]
Lesprit 2010 {published data only}
- Lesprit P, Landelle C, Girou E, Brun‐Buisson C. Reassessment of intravenous antibiotic therapy using a reminder or direct counselling. Journal of Antimicrobial Chemotherapy 2010;65(4):789‐95. [DOI] [PubMed] [Google Scholar]
Levy 2013 {published data only}
- Levy BT, Xu Y, Daly JM, Ely JW. A randomized controlled trial to improve colon cancer screening in rural family medicine: an Iowa Research Network (IRENE) study. Journal of the American Board of Family Medicine 2013;26(5):486‐97. [DOI] [PubMed] [Google Scholar]
- NCT01477814. Interventions to improve colon cancer screening in poor rural Iowa counties. clinicaltrials.gov/ct2/show/NCT01477814 (first received 23 November 2011).
Linares 2011 {published data only}
- Linares LA, Thornton DJ, Strymish J, Baker E, Gupta K. Electronic memorandum decreases unnecessary antimicrobial use for asymptomatic bacteriuria and culture‐negative pyuria. Infection Control Hospital Epidemiology 2011;32(7):644‐8. [DOI] [PubMed] [Google Scholar]
Lipkus 1999 {published data only}
- Lipkus IM, Lyna PR, Rimer BK. Using tailored interventions to enhance smoking cessation among African‐Americans at a community health center. Nicotine & Tobacco Research 1999;1(1):77‐85. [DOI] [PubMed] [Google Scholar]
Luszczynska 2012 {published data only}
- Luszczynska A, Durawa AB, Dudzinska M, Kwiatkowska M, Knysz B, Knoll N. The effects of mortality reminders on posttraumatic growth and finding benefits among patients with life‐threatening illness and their caregivers. Psychology & Health 2012;27(10):1227‐43. [DOI] [PubMed] [Google Scholar]
MacIntyre 2003 {published data only}
- MacIntyre CR, Kainer MA, Brown GV. A randomised, clinical trial comparing the effectiveness of hospital and community‐based reminder systems for increasing uptake of influenza and pneumococcal vaccine in hospitalised patients aged 65 years and over. Gerontology 2003;49(1):33‐40. [DOI] [PubMed] [Google Scholar]
MacLean 2009 {published data only}
- MacLean CD, Gagnon M, Callas P, Littenberg B. The Vermont diabetes information system: a cluster randomized trial of a population based decision support system. Journal of General Internal Medicine 2009;24(12):1303‐10. [DOI] [PMC free article] [PubMed] [Google Scholar]
Majumdar 2004 {published data only}
- Majumdar SR, Johnson JA, Lier DA, Russell AS, Hanley DA, Blitz S, et al. Persistence, reproducibility, and cost‐effectiveness of an intervention to improve the quality of osteoporosis care after a fracture of the wrist: Results of a controlled trial. Osteoporosis International 2007;18:261‐270. [DOI] [PubMed] [Google Scholar]
- Majumdar SR, Rowe BH, Folk D, Johnson JA, Holroyd BH, Morrish DW, et al. A controlled trial to increase detection and treatment of osteoporosis in older patients with a wrist fracture. Annals of Internal Medicine 2004;141:366‐73. [DOI] [PubMed] [Google Scholar]
Majumdar 2008 {published data only}
- Majumdar SR, Johnson JA, McAlister FA, Bellerose D, Russell AS, Hanley DA, et al. Multifaceted intervention to improve diagnosis and treatment of osteoporosis in patients with recent wrist fracture: a randomized controlled trial. Canadian Medical Association Journal 2008;178(5):569‐75. [DOI] [PMC free article] [PubMed] [Google Scholar]
Majumdar 2012 {published data only}
- Majumdar SR, McAlister FA, Johnson JA, Bellerose D, Siminoski K, Hanley DA, et al. Interventions to increase osteoporosis treatment in patients with incidentally detected vertebral fractures. American Journal of Medicine 2012;125(9):929‐36. [DOI] [PubMed] [Google Scholar]
Manfredi 1998 {published data only}
- Manfredi C, Czaja R, Freels S, Trubitt M, Warnecke R, Lacey L. Prescribe for health. Improving cancer screening in physician practices serving low‐income and minority populations. Archives of Family Medicine 1998;7(4):329‐37. [DOI] [PubMed] [Google Scholar]
Manns 2012 {published data only}
- Manns B, Tonelli M, Culleton B, Faris P, McLaughlin K, Chin R, et al. A cluster randomized trial of an enhanced eGFR prompt in chronic kidney disease. Clinical Journal of The American Society of Nephrology: CJASN 2012;7(4):565‐72. [DOI] [PMC free article] [PubMed] [Google Scholar]
Margolis 1992 {published data only}
- Margolis KL, Nichol KL, Wuorenma J, Sternberg TL. Exporting a successful influenza vaccination program from a teaching hospital to a community outpatient setting. Journal of the American Geriatrics Society 1992;40(10):1021‐3. [DOI] [PubMed] [Google Scholar]
Martin 2004 {published data only}
- Martin CM, Doig GS, Heyland DK, Morrison T, Sibbald WJ, for the Southwestern Ontario Critical Care Research Network. Multicentre, cluster‐randomized clinical trial of algorithms for critical‐care enteral and parenteral therapy (ACCEPT). Canadian Medical Association Journal 2004;170(2):197‐204. [PMC free article] [PubMed] [Google Scholar]
Martin 2007 {published data only}
- Martin EW, Miller SC, Welch LC, Burrill J. Improving access to hospice: the physician feedback and reminders to improve access to hospice (PFRIAH) study. Medicine and Health, Rhode Island 2007;90(12):388‐90. [PubMed] [Google Scholar]
McAlister 2009 {published data only}
- McAlister FA, Fradette M, Majumdar SR, Williams R, Graham M, McMeekin J, et al. The Enhancing Secondary Prevention in Coronary Artery Disease trial. Canadian Medical Association Journal 2009;181(12):897‐904. [DOI] [PMC free article] [PubMed] [Google Scholar]
McDermott 2001 {published data only}
- McDermott R, Schmidt B, Sinha A, Mills P. Improving diabetes care in the primary healthcare settings: a randomised cluster trial in remote Indigenous communities. Medical Journal of Australia 2001;174:497‐502. [DOI] [PubMed] [Google Scholar]
McDermott 2003 {published data only}
- McDermott R, Tulip F, Schmidt B, Sinha A. Sustaining better diabetes care in remote indigenous Australian communities. BMJ 2003;327(7412):428‐30. [DOI] [PMC free article] [PubMed] [Google Scholar]
Meulepas 2007 {published data only}
- Meulepas MA, Braspenning JC, Grauw WJ, Lucas AE, Harms L, Akkermans RP, et al. Logistic support service improves processes and outcomes of diabetes care in general practice. Family Practice 2007;24(1):20‐5. [DOI] [PubMed] [Google Scholar]
Myers 2004 {published data only}
- Myers RE, Turner B, Weinberg D, Hyslop T, Hauck WW, Brigham T, et al. Impact of a physician‐oriented intervention on follow‐up in colorectal cancer screening. Preventive Medicine 2004;38(4):375‐81. [DOI] [PubMed] [Google Scholar]
Naunton 2004 {published data only}
- Naunton M, Peterson GM, Jones G, Griffin GM, Bleasel MD. Multifaceted educational program increases prescribing of preventive medication for corticosteroid induced osteoporosis. Journal of Rheumatology 2004;31(3):550‐6. [PubMed] [Google Scholar]
NCT00355004 2013 {published data only}
- NCT00355004. Computerized and mailed reminders in increasing the rate of colorectal cancer screening in adults with an average risk for colorectal cancer. clinicaltrials.gov/ct2/show/NCT00355004 (first received 20 July 2006).
NCT00699439 2009 {published data only}
- NCT00699439. A reminder system for paper‐based asthma guidelines in the pediatric emergency department. clinicaltrials.gov/ct2/show/NCT00699439 (first received 18 June 2008).
NCT01057888 2015 {published data only}
- NCT01057888. Evaluation of vaccination reminder/recall systems for adolescent patients. clinicaltrials.gov/ct2/show/NCT00715234 (first received 15 July 2008).
NCT01126034 2015 {published data only}
- NCT01126034. Improving metoclopramide prescribing practices at Penn through a physician‐targeted intervention. clinicaltrials.gov/ct2/show/NCT01126034 (first received 19 May 2010).
NCT01207232 2010 {published data only}
- NCT01207232. The effect of a planning prompt on seasonal influenza vaccination rates. clinicaltrials.gov/ct2/show/NCT01207232 ((first received 22 September 2010).
NCT01325116 2014 {published data only}
- NCT01325116. Delayed educational reminders in acute myocardial infarction (MI). clinicaltrials.gov/ct2/show/NCT01325116 (first received 29 March 2011).
NCT01352390 2017 {published data only}
- NCT01352390. The effect of a coloring prompt on health engagement. clinicaltrials.gov/ct2/show/results/NCT01352390 (first received 11 May 2011).
NCT01401621 2015 {published data only}
- NCT01401621. The effect of a scratch off prompt on health engagement. clinicaltrials.gov/ct2/show/NCT01401621 (first received 25 July 2011).
NCT01729429 2015 {published data only}
- NCT01729429. An intervention promoting HPV vaccination in safety‐net clinics. clinicaltrials.gov/ct2/show/NCT01729429 (first received 20 November 2011).
NCT02411006 2016 {published data only}
- NCT02411006. Effect of reminders on adherence. clinicaltrials.gov/ct2/show/NCT02411006 (first received 8 April 2015).
NCT02411032 2017 {published data only}
- NCT02411032. Fresh Start Experiment. clinicaltrials.gov/ct2/show/NCT02411032 (first received 8 April 2015).
NCT02438943 2015 {published data only}
- NCT02438943. An intervention to improve management of dyslipidemia in Primary Care. clinicaltrials.gov/ct2/show/NCT02438943 (first received 8 Paril 2015).
NCT02564653 2015 {published data only}
- NCT02564653. Implementing tobacco use guidelines in community health centers in Vietnam public health system. clinicaltrials.gov/ct2/show/NCT02564653 (first received 1 October 2015).
NCT02640521 2016 {published data only}
- NCT02640521. Smoking Cessation Peds Asthma. clinicaltrials.gov/ct2/show/NCT02640521 (first received 29 December 2015).
NCT02927743 2016 {published data only}
- NCT02927743. Effect of evidence‐based reminders on use of antibiotics. clinicaltrials.gov/ct2/show/NCT02927743 (first received 7 October 2016).
Nguyen 2000 {published data only}
- Nguyen BH, Nguyen KP, McPhee SJ, Nguyen AT, Tran DQ, Jenkins CN. Promoting cancer prevention activities among Vietnamese physicians in California. Journal of Cancer Education 2000;15(2):82‐5. [DOI] [PubMed] [Google Scholar]
Ockene 1994 {published data only}
- Ockene JK, Adams A, Pbert L, Luippold R, Hebert JR, Quirk M, et al. The Physician‐Delivered Smoking Intervention Project: factors that determine how much the physician intervenes with smokers. Journal of General Internal Medicine 1994;9(7):379‐84. [DOI] [PubMed] [Google Scholar]
Okano 1995 {published data only}
- Okano GJ, Rascati KL. Effects of Medicaid drug utilization review intervention letters. Clinical Therapeutics 1995;17(3):525‐33. [DOI] [PubMed] [Google Scholar]
Olivarius 2001 {published data only}
- Olivarius N, BeckNielsen H, Andreasen AH, Hørder M, Pedersen PA. Randomised controlled trial of structured personal care of type 2 diabetes mellitus. BMJ 2001;323(7319):1‐9. [DOI] [PMC free article] [PubMed] [Google Scholar]
Olson 2009 {published data only}
- Olson KL, Delate T, Rasmussen J, Humphries TL, Merenich A, Clinical Pharmacy Cardiac Risk Service Study Group. Outcomes of patients discharged from pharmacy‐managed cardiovascular disease management. American Journal of Managed Care 2009;15(8):497‐503. [PubMed] [Google Scholar]
Ong 2011 {published data only}
- Ong MS, Coiera E. Evaluating the effectiveness of clinical alerts: a signal detection approach. AMIA Annual Symposium Proceedings 2011;2011:1036‐44. [PMC free article] [PubMed] [Google Scholar]
Otero‐Sabogal 2006 {published data only}
- Otero‐Sabogal R, Owens D, Canchola J, Tabnak F. Improving rescreening in community clinics: does a system approach work?. Journal of Community Health 2006;31(6):497‐519. [DOI] [PubMed] [Google Scholar]
Paskett 1999 {published data only}
- Paskett ED, Tatum CM, D'Agostino R. Community‐based interventions to improve breast and cervical cancer screening: results of the Forsyth County Cancer Screening (FoCaS) Project. Cancer Epidemiology, Biomarkers & Prevention 1999;8(5):453‐9. [PubMed] [Google Scholar]
Peccoralo 2012 {published data only}
- Peccoralo L, Karani R, Coplit L, Korenstein D. Pocket card and dedicated feedback session to improve feedback to ward residents: a randomized trial. Journal of Hospital Medicine 2012;7(1):35‐40. [DOI] [PubMed] [Google Scholar]
Piazza 2013 {published data only}
- Piazza G, Anderson FA, Ortel TL, Cox MJ, Rosenberg DJ, Rahimian S, et al. Randomized trial of physician alerts for thromboprophylaxis after discharge. American Journal of Medicine 2013;126(5):435‐42. [DOI] [PubMed] [Google Scholar]
Polinski 2011 {published data only}
- Polinski JM, Schneeweiss S, Maclure M, Marshall B, Ramsden S, Dormuth C. Time series evaluation of an intervention to increase statin tablet splitting by general practitioners. Clinical Therapeutics 2011;33(2):235‐43. [DOI] [PMC free article] [PubMed] [Google Scholar]
Pollack 1991 {published data only}
- Pollack MM, Getson PR. Pediatric critical care cost containment: combined actuarial and clinical program. Critical Care Medicine 1991;19(1):12‐20. [DOI] [PubMed] [Google Scholar]
Puech 1998 {published data only}
- Puech M, Ward J, Lajoie V. Postcard reminders from GPs for influenza vaccine: are they more effective than an ad hoc approach?. Australian and New Zealand Journal of Public Health 1998;22(2):254‐6. [DOI] [PubMed] [Google Scholar]
Raebel 2005 {published data only}
- Raebel MA, Lyons EE, Chester EA, Bodily MA, Kelleher JA, Long CL, et al. Improving laboratory monitoring at initiation of drug therapy in ambulatory care. A randomized trial. Archives of Internal Medicine 2005;165(20):2395‐401. [DOI] [PubMed] [Google Scholar]
Ray 2001 {published data only}
- Ray WA, Stein CM, Byrd V, Shorr R, Pichert JW, Gideon P, et al. Educational program for physicians to reduce use of non‐steroidal anti‐inflammatory drugs among community‐dwelling elderly persons: a randomized controlled trial. Medical Care 2001;39(5):425‐35. [DOI] [PubMed] [Google Scholar]
Rimer 1999 {published data only}
- Rimer BK, Conaway M, Lyna P, Glassman B, Yarnall KS, Lipkus I, et al. The impact of tailored interventions on a community health center population. Patient Education and Counseling 1999;37(2):125‐40. [DOI] [PubMed] [Google Scholar]
Robie 1988 {published data only}
- Robie PW. Improving and sustaining outpatient cancer screening by medicine residents. Southern Medical Journal 1988;81(7):902‐5. [DOI] [PubMed] [Google Scholar]
Rodewald 1999 {published data only}
- Rodewald LE, Szilagyi PG, Humiston SG, Barth R, Kraus R, Raubertas RF. A randomized study of tracking with outreach and provider prompting to improve immunization coverage and primary care. Pediatrics 1999;103(1):31‐8. [DOI] [PubMed] [Google Scholar]
Rossignol 2005 {published data only}
- Rossignol M, Allaert FA, Rozenberg S, Valat JP, Avouac B, Peres G, et al. Measuring the contribution of pharmacological treatment to advice to stay active in patients with subacute low‐back pain: a randomised controlled trial. Pharmacoepidemiology and Drug Safety 2005;14(12):861‐7. [DOI] [PubMed] [Google Scholar]
Roy 2014 {published data only}
- Roy PM, Rachas A, Meyer G, Gal G, Durieux P, Schmidt J, et al. Impact of a multifaceted intervention to prevent venous thromboembolic disease in patients admitted to emergency ward and hospitalized for acute medical illness: a multicentre cluster randomized trial. European Heart Journal 2014;35:1064‐5. [Google Scholar]
Saint 2005 {published data only}
- Saint S, Kaufman SR, Thompson M, Rogers MA, Chenoweth CE. A reminder reduces urinary catheterization in hospitalized patients. Joint Commission Journal on Quality and Patient Safety 2005;31(8):455‐62. [DOI] [PubMed] [Google Scholar]
Scheel 2002 {published data only}
- Scheel IB, Hagen KB, Herrin J, Oxman AD. A randomized controlled trial of two strategies to implement active sick leave for patients with low back pain. Spine 2002;27(6):561‐6. [DOI] [PubMed] [Google Scholar]
Sellors 2004 {published data only}
- Sellors JW, Coleman A. Adding pap smear reminders to mammogram reminders increases pap smear uptake with no effect on mammography uptake. Evidence‐based Healthcare 2004;8(1):18‐20. [Google Scholar]
Shelley 2010 {published data only}
- Shelley D, Cantrell J. The effect of linking community health centers to a state‐level smokers quitline on rates of cessation assistance. BMC Health Services Research 2010;10:25. [DOI] [PMC free article] [PubMed] [Google Scholar]
Shevlin 2002 {published data only}
- Shevlin JD, Summers‐Bean C, Thomas D, Whitney CG, Todd D, Ray SM. A systematic approach for increasing pneumococcal vaccination rates at an inner‐city public hospital. American Journal of Preventive Medicine 2002;22(2):92‐7. [DOI] [PubMed] [Google Scholar]
Shirai 2012 {published data only}
- Shirai Y, Fujimori M, Ogawa A, Yamada Y, Nishiwaki Y, Ohtsu A, et al. Patients perception of the usefulness of a question prompt sheet for advanced cancer patients when deciding the initial treatment: a randomized, controlled trial. Psycho‐Oncology 2012;21(7):706‐13. [DOI] [PubMed] [Google Scholar]
Simon 2001 {published data only}
- Simon MS, Gimotty PA, Moncrease A, Dews P, Burack RC. The effect of patient reminders on the use of screening mammography in an urban health department primary care setting. Breast Cancer Research and Treatment 2001;65(1):63‐70. [DOI] [PubMed] [Google Scholar]
Solomon 2004 {published data only}
- Solomon DH, Katz JN, Tourette AM, Coblyn JS. Multifaceted intervention to improve rheumatologists' management of glucocorticoid‐induced osteoporosis: a randomized controlled trial. Arthritis and Rheumatism 2004;51(3):383‐7. [DOI] [PubMed] [Google Scholar]
Solomon 2007 {published data only}
- Solomon DH, Polinski JM, Stedman M, Truppo C, Breiner L, Egan C, et al. Improving care of patients at‐risk for osteoporosis: a randomized controlled trial. Journal of General Internal Medicine 2007;22(3):362‐7. [DOI] [PMC free article] [PubMed] [Google Scholar]
Soureti 2011 {published data only}
- Soureti A, Murray P, Cobain M, Chinapaw M, Mechelen W, Hurling R. Exploratory study of web‐based planning and mobile text reminders in an overweight population. Journal of Medical Internet Research 2011;13(4):e118. [DOI] [PMC free article] [PubMed] [Google Scholar]
Srikrajang 2005 {published data only}
- Srikrajang J, Pochamarn C, Chittreecheur J, Apisarnthanarak A, Danchaivijitr S. Effectiveness of education and problem solving work group on nursing practices to prevent needlestick and sharp injury. Journal of the Medical Association of Thailand 2005;88(Suppl 10):S115‐9. [PubMed] [Google Scholar]
Stiell 2009 {published data only}
- Stiell IG, Clement CM, Grimshaw J, Brison RJ, Rowe BH, Schull MJ, et al. Implementation of the Canadian C‐Spine Rule: prospective 12 centre cluster randomised trial. BMJ 2009;339:b4146. [DOI] [PMC free article] [PubMed] [Google Scholar]
Stiell 2010 {published data only}
- Stiell IG, Clement CM, Grimshaw JM, Brison RJ, Rowe BH, Lee JS, et al. A prospective cluster‐randomized trial to implement the Canadian CT Head Rule in emergency departments. Canadian Medical Association Journal 2010;182(14):1527‐32. [DOI] [PMC free article] [PubMed] [Google Scholar]
Stone 2005 {published data only}
- Stone RA, Mor MK, Lave JR, Hough LJ, Fine MJ. Implementation of an inpatient management and discharge strategy for patients with community acquired pneumonia. American Journal of Managed Care 2005;11(8):491‐9. [PubMed] [Google Scholar]
Szilagyi 2015 {published data only}
- Szilagyi PG, Serwint JR, Humiston SG, Rand CM, Schaffer S, Vincelli P, et al. Effect of provider prompts on adolescent immunization rates: a randomized trial. Academic Pediatrics 2015;15(2):149‐57. [DOI] [PMC free article] [PubMed] [Google Scholar]
Taylor 1999 {published data only}
- Taylor V, Thompson B, Lessler D, Yasui Y, Montano D, Johnson KM, et al. A clinic‐based mammography intervention targeting inner‐city women. Journal of General Internal Medicine 1999;14(2):104‐11. [DOI] [PubMed] [Google Scholar]
Thomas 2006 {published data only}
- Thomas RE, Croal BL, Ramsay C, Eccles M, Grimshaw J. Effect of enhanced feedback and brief educational reminder messages on laboratory test requesting in primary care: a cluster randomised trial. Lancet 2006;367(9527):1990‐6. [DOI] [PubMed] [Google Scholar]
Thomas 2007 {published data only}
- Thomas KG, Thomas MR, Stroebel RJ, McDonald FS, Hanson GJ, Naessens JM, et al. Use of a registry‐generated audit, feedback, and patient reminder intervention in an internal medicine resident clinic: a randomized trial. Journal of General Internal Medicine 2007;22(12):1740‐4. [DOI] [PMC free article] [PubMed] [Google Scholar]
Thompson 2008 {published data only}
- Thompson A, Sullivan SA, Barley M, Strange SO, Moore L, Rogers P, et al. The DEBIT trial: an intervention to reduce antipsychotic polypharmacy prescribing in adult psychiatry wards ‐ a cluster randomized controlled trial. Psychological Medicine 2008;38(5):705‐15. [DOI] [PubMed] [Google Scholar]
Tierney 1986 {published data only}
- Tierney WM, Hui SL, McDonald CJ. Delayed feedback of physician performance versus immediate reminders to perform preventive care: effects on physician compliance. Medical Care 1986;24(8):659‐66. [DOI] [PubMed] [Google Scholar]
Tseng 2002 {published data only}
- Tseng DS. Organisational changes are more effective than education and reminders for raising adult immunisation and cancer screening rates. Evidence‐based Healthcare 2002;6(4):186‐7. [Google Scholar]
Valanis 2002 {published data only}
- Valanis BG, Glasgow RE, Mullooly J, Vogt TM, Whitlock EP, Boles SM, et al. Screening HMO women overdue for both mammograms and Pap tests. Preventive Medicine 2002;34(1):40‐50. [DOI] [PubMed] [Google Scholar]
Van der Sanden 2005 {published data only}
- Sanden WJ, Mettes DG, Plasschaert AJ, Grol RP, Mulder J, Verdonschot EH. Effectiveness of clinical practice guideline implementation on lower third molar management in improving clinical decision‐making: a randomized controlled trial. European Journal of Oral Sciences 2005;113(5):349‐54. [DOI] [PubMed] [Google Scholar]
Van der Weijden 2001 {published data only}
- Weijden T. Effect of implementation of cholesterol guidelines on performance. A randomised controlled trial in 20 general practices. Evaluation of Cholesterol Guidelines in General Practice [PhD thesis]. Maastricht: Universiteit Maastricht, 2001. [Google Scholar]
Vissers 1996 {published data only}
- Vissers MC, Biert J, Linden CJ, Hasman A. Effects of a supportive protocol processing system (ProtoVIEW) on clinical behaviour of residents in the accident and emergency department. Computer Methods and Programs in Biomedicine 1996;49(2):177‐84. [DOI] [PubMed] [Google Scholar]
Wadland 2007 {published data only}
- Wadland WC, Holtrop JS, Weismantel D, Pathak PK, Fadel H, Powell J. Practice‐based referrals to a tobacco cessation quit line: assessing the impact of comparative feedback vs general reminders. Annals of Family Medicine 2007;5(2):135‐42. [DOI] [PMC free article] [PubMed] [Google Scholar]
Waldorff 2009 {published data only}
- Waldorff FB, Almind G, Mäkelä M, Møller S, Waldemar G. Implementation of a clinical dementia guideline. Scandinavian Journal of Primary Health Care 2009;21(3):142‐7. [DOI] [PubMed] [Google Scholar]
Wee 2016 {published data only}
- Wee AG, Zimmerman LM, Anderson JR, Nunn ME, Loberiza FR, Sitorius MA, et al. Promoting oral cancer examinations to medical primary care providers: a cluster randomized trial. Journal of Public Health Dentistry 2016;76(4):340‐9. [DOI] [PubMed] [Google Scholar]
Weiss 2011 {published data only}
- Weiss CH, Feinglass J, Moazed F, McEvoy C, Singer BD, Wunderink RG. Prompting to use a daily rounding checklist reduces costs associated with hospitalization. American Journal of Respiratory and Critical Care Medicine 2011;183:A104. [Google Scholar]
- Weiss CH, Moazed F, McEvoy C, Singer B, Szleifer I, Amaral LAN, et al. Checklist‐based prompting improves ICU outcomes. American Journal of Respiratory and Critical Care Medicine 2010;181:A2397. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Weiss CH, Moazed F, McEvoy CA, Singer BD, Szleifer I, Amaral LAN, et al. Prompting Physicians to Address a Daily Checklist and Process of Care and Clinical Outcomes: A Single‐Site Study. American Journal of Respiratory and Critical Care Medicine 2011;184:680‐6. [DOI] [PMC free article] [PubMed] [Google Scholar]
Zeler 1992 {published data only}
- Zeler KM, McPharlane TJ, Salamonsen RF. Effectiveness of nursing involvement in bedside monitoring and control of coagulation status after cardiac surgery. American Journal of Critical Care 1992;1(2):70‐5. [PubMed] [Google Scholar]
References to ongoing studies
Feitosa‐Souza 2018 {published data only}
- Feitosa‐Souza MA. Behavioural technology for nurses to enrol and use the child's book [Tecnologia comportamental para adesão dos enfermeiros ao registro e utilização da caderneta da criança]. Brazilian Registry of Clinical Trials 2018. [U1111‐1212‐3312]
Additional references
Arditi 2017
- Arditi C, Rège‐Walther M, Durieux P, Burnand B. Computer‐generated reminders delivered on paper to healthcare professionals: effects on professional practice and healthcare outcomes. Cochrane Database of Systematic Reviews 2017, Issue 7. [DOI: 10.1002/14651858.CD001175.pub4] [DOI] [PMC free article] [PubMed] [Google Scholar]
Austin 1994
- Austin SM, Balas EA, Mitchell JA, Ewigman BG. Effect of physician reminders on preventive care: meta‐analysis of randomized clinical trials. Annual Symposium on Computer Application in Medical Care. 1994:121‐4. [PMC free article] [PubMed]
Balas 2000
- Balas EA, Weingarten S, Garb CT, Blumenthal D, Boren SA, Brown GD. Improving preventive care by prompting physicians. Archives of Internal Medicine 2000;160(3):301‐8. [DOI] [PubMed] [Google Scholar]
Baron 2010
- Baron RC, Melillo S, Rimer BK, Coates RJ, Kerner J, Habarta N, et al. Task Force on Community Preventive Services. Intervention to increase recommendation and delivery of screening for breast, cervical, and colorectal cancers by healthcare providers: a systematic review of provider reminders. American Journal of Preventive Medicine 2010;38(1):110‐7. [DOI] [PubMed] [Google Scholar]
Bero 1998
- Bero LA, Grilli R, Grimshaw JM, Harvey E, Oxman AD, Thomson MA. Getting research findings into practice: closing the gap between research and practice: an overview of systematic reviews of interventions to promote the implementation of research findings. BMJ 1998;317(7156):465‐8. [DOI] [PMC free article] [PubMed] [Google Scholar]
Bosch‐Capblanch 2017
- Bosch‐Capblanch X, Auer C, Njepuome N, Saric J, Jarrett C, Guterman A, et al. Characterisation of the Health Information System in Nigeria: Report of findings (PHISICC project). paperbased.info/wp‐content/uploads/2017/10/PHISICC3_NGA_Report_v08.pdf (accessed prior to 13 November 2019).
Buntinx 1993
- Buntinx F, Winkens R, Grol R, Knottnerus JA. Influencing diagnostic and preventive performance in ambulatory care by feedback and reminders. A review. Family Practice 1993;10(2):219‐28. [DOI] [PubMed] [Google Scholar]
Carlsen 2007
- Carlsen B, Glenton C, Pope C. Thou shalt versus thou shalt not: a meta‐synthesis of GPs' attitudes to clinical practice guidelines. British Journal of General Practice 2007;57(545):971‐8. [DOI] [PMC free article] [PubMed] [Google Scholar]
Cheung 2012
- Cheung A, Weir M, Mayhew A, Kozloff N, Brown K, Grimshaw J. Overview of systematic reviews of the effectiveness of reminders in improving healthcare professional behavior. Systematic Reviews 2012;1:36. [DOI] [PMC free article] [PubMed] [Google Scholar]
Cohen 1994
- Cohen SJ, Halvorson HW, Gosselink CA. Changing physician behavior to improve disease prevention. Preventive Medicine 1994;23(3):284‐91. [DOI] [PubMed] [Google Scholar]
CRD 1999
- Centre for Reviews and Dissemination (CRD), The University of York. Getting evidence into practice. Effective Health Care bulletins. February 1999 Volume 5 Number 1. Available from www.york.ac.uk/media/crd/ehc51.pdf.
Dexheimer 2008
- Dexheimer JW, Talbot TR, Sanders DL, Rosenbloom ST, Aronsky D. Prompting clinicians about preventive care measures: a systematic review of randomized controlled trials. Journal of the American Medical Informatics Association : JAMIA 2008;15(3):311‐20. [DOI] [PMC free article] [PubMed] [Google Scholar]
EPOC 2015
- Cochrane Effective Practice, Organisation of Care (EPOC). EPOC taxonomy. epoc.cochrane.org/epoc‐taxonomy (accessed 12 November 2019).
EPOC 2017a
- Cochrane Effective Practice, Organisation of Care (EPOC). What study designs can be considered for inclusion in an EPOC review and what should they be called? EPOC Resources for review authors. epoc.cochrane.org/resources/epoc‐resources‐review‐authors (accessed on 12 November 2019).
EPOC 2017b
- Cochrane Effective Practice, Organisation of Care (EPOC). Data collection form. EPOC resources for review authors. epoc.cochrane.org/epoc‐specific‐resources‐review‐authors (accessed 12 November 2019).
EPOC 2017c
- Cochrane Effective Practice, Organisation of Care (EPOC). Suggested risk of bias criteria for EPOC reviews. EPOC Resources for review author. epoc.cochrane.org/resources/epoc‐resources‐review‐authors (accessed 12 November 2019).
EPOC 2017d
- Cochrane Effective Practice, Organisation of Care (EPOC). EPOC worksheets for preparing a Summary of Findings (SoF) table using GRADE. EPOC Resources for review authors. epoc.cochrane.org/resources/epoc‐resources‐review‐authors (accessed 15 May 2019).
Flodgren 2019
- Flodgren G, Parmelli E, Doumit G, Gattellari M, O'Brien MA, Grimshaw J, et al. Local opinion leaders: effects on professional practice and healthcare outcomes. Cochrane Database of Systematic Reviews 2019, Issue 6. [DOI: 10.1002/14651858.CD000125.pub4] [DOI] [PMC free article] [PubMed] [Google Scholar]
Grimshaw 2001
- Grimshaw JM, Shirran L, Thomas R, Mowatt G, Fraser C, Bero L, et al. Changing provider behavior: an overview of systematic reviews of interventions. Medical Care 2001;39(8 Suppl 2):II2‐45. [PubMed] [Google Scholar]
Grimshaw 2012
- Grimshaw JM, Eccles MP, Lavis JN, Hill SJ, Squires JE. Knowledge translation of research findings. Implementation Science 2012;7:50. [DOI] [PMC free article] [PubMed] [Google Scholar]
Grol 2003
- Grol R, Grimshaw J. From best evidence to best practice: effective implementation of change in patients' care. Lancet 2003;362(9391):1225‐30. [DOI] [PubMed] [Google Scholar]
Grol 2005
- Grol R, Wensing M, Eccles M. Improving Patient Care. The Implementation of Change in Clinical Practice. Edinburgh: Elsevier, 2005. [Google Scholar]
Guyatt 2008
- Guyatt GH, Oxman AD, Vist GE, Kunz R, Falck‐Ytter Y, Alonso‐Coello P, et al. GRADE: an emerging consensus on rating quality of evidence and strength of recommendations. BMJ 2008;336:924‐6. [DOI] [PMC free article] [PubMed] [Google Scholar]
Higgins 2011
- Higgins JP, Deeks JJ, Altman DG, editor(s). Chapter 16: Special topics in statistics. In: Higgins JP, Green S, editor(s). Cochrane Handbook for Systematic Reviews of Interventions. Version 5.1.0 (updated March 2011). The Cochrane Collaboration, 2011. Available from www.cochrane‐handbook.org.
Higgins 2017
- Higgins JP, Altman DG, Sterne JA, editor(s). Chapter 8: Assessing risk of bias in included studies. In: Higgins JP, Churchill R, Chandler J, Cumpston MS, editor(s), Cochrane Handbook for Systematic Reviews of Interventions version 5.2.0 (updated June 2017), The Cochrane Collaboration, 2017. Available from www.training.cochrane.org/handbook.
Ivers 2012a
- Ivers N, Jamtvedt G, Flottorp S, Young JM, Odgaard‐Jensen J, French SD, et al. Audit and feedback: effects on professional practice and healthcare outcomes. Cochrane Database of Systematic Reviews 2012, Issue 6. [DOI: 10.1002/14651858.CD000259.pub3] [DOI] [PMC free article] [PubMed] [Google Scholar]
Johnson 2015
- Johnson MJ, May CR. Promoting professional behaviour change in healthcare: what interventions work, and why? A theory‐led overview of systematic reviews. BMJ Open 2015;5:e008592. [DOI] [PMC free article] [PubMed] [Google Scholar]
Johnston 1994
- Johnston ME, Langton KB, Haynes RB, Mathieu A. Effects of computer‐based clinical decision support systems on clinician performance and patient outcome: a critical appraisal of research. Annals of Internal Medicine 1994;120(2):135‐42. [DOI] [PubMed] [Google Scholar]
Litzelman 1993
- Litzelman DK, Dittus RS, Miller ME, Tierney WM. Requiring physicians to respond to computerized reminders improves their compliance with preventive care protocols. Journal of General Internal Medicine 1993;8(6):311‐7. [DOI] [PubMed] [Google Scholar]
Mandelblatt 1995
- Mandelblatt J, Kanetsky PA. Effectiveness of interventions to enhance physician screening for breast cancer. Journal of Family Practice 1995;40(2):162‐71. [PubMed] [Google Scholar]
Mangione‐Smith 2007
- Mangione‐Smith R, DeCristofaro AH, Setodji CM, Keesey J, Klein DJ, Adams JL, et al. The quality of ambulatory care delivered to children in the United States. New England Journal of Medicine 2007;357(15):1515‐23. [DOI] [PubMed] [Google Scholar]
Mason 1999
- Mason J, Wood J, Freemantle N. Designing evaluations of interventions to change professional practice. Journal of Health Services Research & Policy 1999;4(2):106‐11. [DOI] [PubMed] [Google Scholar]
McDonald 1976
- McDonald CJ. Protocol‐based computer reminders, the quality of care and the non‐perfectability of man. New England Journal of Medicine 1976;295(24):1351‐5. [DOI] [PubMed] [Google Scholar]
McGlynn 2003
- McGlynn EA, Asch SM, Adams J, Keesey J, Hicks J, DeCristofaro A, et al. The quality of health care delivered to adults in the United States. New England Journal of Medicine 2003;348(26):2635‐45. [DOI] [PubMed] [Google Scholar]
Moher 2009
- Moher D, Liberati A, Tetzlaff J, Altman DG, The PRISMA Group. Preferred Reporting Items for Systematic Reviews and Meta‐Analyses: The PRISMA Statement. PLoS Medicine 2009;6(7):e1000097. [DOI] [PMC free article] [PubMed] [Google Scholar]
Mugford 1991
- Mugford M, Bnafield P, O'Hanlon M. Effect of feedback of information on clinical practice: a review. BMJ 1991;303(6799):398‐402. [DOI] [PMC free article] [PubMed] [Google Scholar]
Ogrinc 2016
- Ogrinc G, Davies L, Goodman D, Batalden P, Davidoff F, Stevens D. SQUIRE 2.0 (Standards for QUality Improvement Reporting Excellence): revised publication guidelines from a detailed consensus process. BMJ Quality & Safety 2016;25(12):986‐992. [DOI] [PMC free article] [PubMed] [Google Scholar]
Oxman 1995
- Oxman AD, Thomson MA, Davis DA, Haynes RB. No magic bullets: a systematic review of 102 trials of interventions to improve professional practice. Canadian Medical Association Journal 1995;153(10):1423‐31. [PMC free article] [PubMed] [Google Scholar]
Reckmann 2009
- Reckmann MH, Westbrook JI, Koh Y, Lo C, Day RO. Does computerized provider order entry reduce prescribing errors for hospital inpatients?. Journal of the American Medical Informatics Association 2009;16(5):613‐23. [DOI] [PMC free article] [PubMed] [Google Scholar]
Schedlbauer 2009
- Schedlbauer A, Prasad V, Mulvaney C, Phansalkar S, Stanton W, Bates DW, et al. What evidence supports the use of computerized alerts and prompts to improve clinicians' prescribing behavior?. Journal of the American Medical Informatics Association 2009;16(4):531‐8. [DOI] [PMC free article] [PubMed] [Google Scholar]
Schulz 2010
- Schulz KF, Altman DG, Moher D, for the CONSORT Group. CONSORT 2010 Statement: updated guidelines for reporting parallel group randomised trials. BMJ 2010;340:c332. [DOI] [PMC free article] [PubMed] [Google Scholar]
Shojania 2004
- Shojania KG, Grimshaw JM. Still no magic bullets: pursuing more rigorous research in quality improvement. American Journal of Medicine 2004;116(11):778‐80. [DOI] [PubMed] [Google Scholar]
Shojania 2005
- Shojania KG, Grimshaw JM. Evidence‐based quality improvement: the state of the science. Health Affairs 2005;24(1):138‐50. [DOI] [PubMed] [Google Scholar]
Shojania 2009
- Shojania KG, Jennings A, Mayhew A, Ramsay CR, Eccles MP, Grimshaw J. The effects of on‐screen, point of care computer reminders on processes and outcomes of care. Cochrane Database of Systematic Reviews 2009, Issue 3. [DOI: 10.1002/14651858.CD001096.pub2] [DOI] [PMC free article] [PubMed] [Google Scholar]
StataCorp 2015 [Computer program]
- StataCorp LP. Stata Statistical Software: Release 14. College Station, TX: StataCorp LP, 2015.
Szilagyi 2000
- Szilagyi PG, Bordley C, Vann JC, Chelminski A, Kraus RM, Margolis PA, et al. Effect of patient reminder/recall interventions on immunization rates: A review. JAMA 2000;284(14):1820‐7. [DOI] [PubMed] [Google Scholar]
Ukoumunne 1999
- Ukoumunne OC, Gulliford MC, Chinn S, Sterne JA, Burney PG. Methods for evaluating area‐wide and organisation‐based interventions in health and health care: a systematic review. Health Technology Assessment 1999;3(5):iii‐92. [PubMed] [Google Scholar]
Wensing 1994
- Wensing M, Grol R. Single and combined strategies for implementing changes in primary care: a literature review. International Journal for Quality in Health Care 1994;6(2):115‐32. [DOI] [PubMed] [Google Scholar]
References to other published versions of this review
Pantoja 2014
- Pantoja T, Green ME, Grimshaw J, Denig P, Durieux P, Gill P, Colomer N, Castañon C, Leniz J. Manual paper reminders: effects on professional practice and health care outcomes. Cochrane Database of Systematic Reviews 2014, Issue 9. [DOI: 10.1002/14651858.CD001174.pub3] [DOI] [Google Scholar]