Computer‐generated reminders delivered on paper to healthcare professionals: effects on professional practice and healthcare outcomes

Chantal Arditi; Myriam Rège‐Walther; Pierre Durieux; Bernard Burnand

doi:10.1002/14651858.CD001175.pub4

. 2017 Jul 6;2017(7):CD001175. doi: 10.1002/14651858.CD001175.pub4

Computer‐generated reminders delivered on paper to healthcare professionals: effects on professional practice and healthcare outcomes

Chantal Arditi ^1,^✉, Myriam Rège‐Walther ², Pierre Durieux ³, Bernard Burnand ¹

Editor: Cochrane Effective Practice and Organisation of Care Group

PMCID: PMC6483307 PMID: 28681432

Abstract

Background

Clinical practice does not always reflect best practice and evidence, partly because of unconscious acts of omission, information overload, or inaccessible information. Reminders may help clinicians overcome these problems by prompting them to recall information that they already know or would be expected to know and by providing information or guidance in a more accessible and relevant format, at a particularly appropriate time. This is an update of a previously published review.

Objectives

To evaluate the effects of reminders automatically generated through a computerized system (computer‐generated) and delivered on paper to healthcare professionals on quality of care (outcomes related to healthcare professionals' practice) and patient outcomes (outcomes related to patients' health condition).

Search methods

We searched CENTRAL, MEDLINE, Embase, six other databases and two trials registers up to 21 September 2016 together with reference checking, citation searching and contact with study authors to identify additional studies.

Selection criteria

We included individual‐ or cluster‐randomized and non‐randomized trials that evaluated the impact of computer‐generated reminders delivered on paper to healthcare professionals, alone (single‐component intervention) or in addition to one or more co‐interventions (multi‐component intervention), compared with usual care or the co‐intervention(s) without the reminder component.

Data collection and analysis

Review authors working in pairs independently screened studies for eligibility and abstracted data. For each study, we extracted the primary outcome when it was defined or calculated the median effect size across all reported outcomes. We then calculated the median improvement and interquartile range (IQR) across included studies using the primary outcome or median outcome as representative outcome. We assessed the certainty of the evidence according to the GRADE approach.

Main results

We identified 35 studies (30 randomized trials and five non‐randomized trials) and analyzed 34 studies (40 comparisons). Twenty‐nine studies took place in the USA and six studies took place in Canada, France, Israel, and Kenya. All studies except two took place in outpatient care. Reminders were aimed at enhancing compliance with preventive guidelines (e.g. cancer screening tests, vaccination) in half the studies and at enhancing compliance with disease management guidelines for acute or chronic conditions (e.g. annual follow‐ups, laboratory tests, medication adjustment, counseling) in the other half.

Computer‐generated reminders delivered on paper to healthcare professionals, alone or in addition to co‐intervention(s), probably improves quality of care slightly compared with usual care or the co‐intervention(s) without the reminder component (median improvement 6.8% (IQR: 3.8% to 17.5%); 34 studies (40 comparisons); moderate‐certainty evidence).

Computer‐generated reminders delivered on paper to healthcare professionals alone (single‐component intervention) probably improves quality of care compared with usual care (median improvement 11.0% (IQR 5.4% to 20.0%); 27 studies (27 comparisons); moderate‐certainty evidence). Adding computer‐generated reminders delivered on paper to healthcare professionals to one or more co‐interventions (multi‐component intervention) probably improves quality of care slightly compared with the co‐intervention(s) without the reminder component (median improvement 4.0% (IQR 3.0% to 6.0%); 11 studies (13 comparisons); moderate‐certainty evidence).

We are uncertain whether reminders, alone or in addition to co‐intervention(s), improve patient outcomes as the certainty of the evidence is very low (n = 6 studies (seven comparisons)). None of the included studies reported outcomes related to harms or adverse effects of the intervention.

Authors' conclusions

There is moderate‐certainty evidence that computer‐generated reminders delivered on paper to healthcare professionals probably slightly improves quality of care, in terms of compliance with preventive guidelines and compliance with disease management guidelines. It is uncertain whether reminders improve patient outcomes because the certainty of the evidence is very low. The heterogeneity of the reminder interventions included in this review also suggests that reminders can probably improve quality of care in various settings under various conditions.

Plain language summary

The effect of automatically generated reminders delivered to providers on paper on quality of care and patient outcomes

What is the aim of this review?

The aim of this Cochrane review was to find out if reminders, automatically generated through a computer, but delivered on paper to doctors help them provide the best recommended care. Cochrane researchers identified 35 studies and analyzed 34 of these studies to answer this question.

Key messages

Providing reminders to doctors probably improves slightly the quality of care patients receive. However, because the certainty of the evidence is moderate, more high‐quality studies on the effectiveness of reminders are needed to confirm to findings of this review.

What was studied in the review?

Doctors do not always provide care that is recommended or that reflects the latest research, partly because of too much information or inaccessible information. Reminders may help doctors overcome these problems by reminding them about guidelines and research findings, or by providing advice, in a more accessible and relevant format, at a particularly appropriate time. For example, when a doctor sees a patient for an annual check‐up, the doctor would receive the patient's chart with a reminder section listing the screening tests due that year, such as colorectal cancer screening. In this review, we evaluated the effects of reminders on the quality of care delivered by physicians, on patient outcomes, and on adverse effects. These reminders were automatically generated through a computer system but delivered on paper.

What are the main results of the review?

Twenty‐nine studies were from the USA and six studies were from Canada, France, Israel and Kenya. The studies examined reminders to doctors to order screening tests, to provide vaccinations, to prescribe specific medications, or to discuss care with patients.

The review shows that:

‐ overall, reminders probably improve slightly quality of care by 6.8% (in 34 studies (40 comparisons), moderate‐certainty evidence);

‐ reminders alone (single‐component intervention) probably improve quality of care by 11.0% compared with usual care (in 27 studies (27 comparisons), moderate‐certainty evidence);

‐ adding reminders to one or more co‐interventions (multi‐component intervention) probably improve slightly quality of care by 4.0% compared with the co‐intervention(s) without the reminder component (in 11 studies (13 comparisons), moderate‐certainty evidence);

‐ it is uncertain whether reminders improve patient outcomes because the certainty of the evidence is very low;

‐ none of the included studies reported outcomes related to harms or adverse effects.

How up to date is this review?

The review authors searched for studies that had been published up to 21 September 2016.

Summary of findings

for the main comparison.

Computer‐generated reminders delivered on paper to healthcare professionals, alone or in addition to co‐intervention(s), compared with usual care or the co‐intervention(s) without the reminder component
Patient or population: Healthcare professionals Settings: Outpatient care in Canada, France, Israel, Kenya and USA Intervention: Reminders automatically generated through a computerized system (computer‐generated) and delivered on paper to healthcare professionals, alone or in addition to one or more co‐interventions, aimed at enhancing compliance with preventive guidelines (e.g. cancer screening tests, vaccination) or disease management guidelines for acute or chronic conditions (e.g. annual follow‐ups, laboratory tests, medication adjustment, counseling) Comparison: Usual care or co‐intervention(s) without reminder component
Outcomes	Median improvement	Number of studies (comparisons)	Certainty of the evidence  (GRADE)	Comments
Quality of care	Pooling data across the 40 comparisons, the median improvement in quality of care associated with the reminder intervention was 6.8% (IQR 3.8% to 17.5%).	34 studies (40 comparisons)	⊕⊕⊕⊝  MODERATE¹	Quality of care was measured by various rates: e.g. test ordering rates, vaccination rates, follow‐up rates, prescription rates, overall compliance rate.
Patient outcomes	Not estimable	6 studies (7 comparisons)	⊕⊝⊝⊝  VERY LOW²	No measurable effect on i) blood pressure, glycated hemoglobin and cholesterol levels, ii) reaching blood pressure, glycated hemoglobin and cholesterol targets, and iii) mortality.
Adverse effects	Not reported	‐	‐	None of the included studies reported outcomes related to harms or adverse effects of reminders.
IQR: interquartile range
GRADE Working Group grades of evidence   High certainty: Further research is very unlikely to change our confidence in the estimate of effect.  Moderate certainty: Further research is likely to have an important impact on our confidence in the estimate of effect and may change the estimate.  Low certainty: Further research is very likely to have an important impact on our confidence in the estimate of effect and is likely to change the estimate.  Very low certainty: We are very uncertain about the estimate.

Open in a new tab

¹ We downgraded the level of certainty of the evidence from high to moderate because of methodological limitations in the included studies and possible publication bias. We did not find other serious limitations in the other factors (indirectness of evidence, inconsistency of results, and imprecision of results).

² We downgraded the level of certainty of the evidence from high to very low because of methodological limitations in the included studies, imprecision of results (wide confidence intervals) and inconsistency of the results.

2.

Computer‐generated reminders delivered on paper to healthcare professionals alone (single‐component intervention) compared with usual care
Patient or population: Healthcare professionals Settings: Outpatient care in Canada, France, Israel, Kenya and USA Intervention: Computer‐generated reminders delivered on paper alone (single‐component intervention) Comparison: Usual care
Outcomes	Median improvement	Number of studies (comparisons)	Certainty of the evidence  (GRADE)	Comments
Quality of care	Pooling data across the 27 comparisons, the median improvement in quality of care associated with the reminder intervention was 11.0% (IQR 5.4% to 20.0%)	27 studies (27 comparisons)	⊕⊕⊕⊝  MODERATE¹	Quality of care was measured by various rates: e.g. test ordering rates, vaccination rates, follow‐up rates, prescription rates, overall compliance rate.
Patient outcomes	Not estimable	4 studies (4 comparisons )	⊕⊝⊝⊝  VERY LOW²	No measurable effect on i) blood pressure, glycated hemoglobin and cholesterol levels, ii) reaching blood pressure, glycated hemoglobin and cholesterol targets, and iii) mortality.
Adverse effects	Not reported	‐	‐	None of the included studies reported outcomes related to harms or adverse effects of reminders.
IQR: interquartile range
GRADE Working Group grades of evidence   High certainty: Further research is very unlikely to change our confidence in the estimate of effect.  Moderate certainty: Further research is likely to have an important impact on our confidence in the estimate of effect and may change the estimate.  Low certainty: Further research is very likely to have an important impact on our confidence in the estimate of effect and is likely to change the estimate.  Very low certainty: We are very uncertain about the estimate.

Open in a new tab

3.

Computer‐generated reminders delivered on paper to healthcare professionals in addition to one or more co‐interventions (multi‐component intervention) compared with the co‐intervention(s) without the reminder component
Patient or population: Healthcare professionals Settings: Outpatient care in Canada and USA Intervention: Computer‐generated reminders delivered on paper in addition to one or more co‐interventions (multi‐component intervention) Comparison: Co‐intervention(s) without the reminder component
Outcomes	Median improvement (interquartile range)	Number of studies (comparisons)	Certainty of the evidence  (GRADE)	Comments
Quality of care	Pooling data across the 13 comparisons, the median improvement in quality of care associated with the reminder intervention was 4.0% (3.0% to 6.0%)	11 studies (13 comparisons)	⊕⊕⊕⊝  MODERATE¹	Quality of care was measured by various rates: e.g. test ordering rates, vaccination rates, follow‐up rates, prescription rates, overall compliance rate.
Patient outcomes	Not estimable	2 studies (3 comparisons)	⊕⊝⊝⊝  VERY LOW²	No measurable effect on i) blood pressure, glycated hemoglobin and cholesterol levels, ii) reaching blood pressure, glycated hemoglobin and cholesterol targets, and iii) mortality.
Adverse effects	Not reported	‐	‐	None of the included studies reported outcomes related to harms or adverse effects of reminders.
IQR: interquartile range
GRADE Working Group grades of evidence   High certainty: Further research is very unlikely to change our confidence in the estimate of effect.  Moderate certainty: Further research is likely to have an important impact on our confidence in the estimate of effect and may change the estimate.  Low certainty: Further research is very likely to have an important impact on our confidence in the estimate of effect and is likely to change the estimate.  Very low certainty: We are very uncertain about the estimate.

Open in a new tab

² We downgraded the level of certainty of the evidence from high to very low because of methodological limitations in the included studies and imprecision of results (wide confidence intervals).

Background

Description of the condition

Clinical practice does not always reflect best evidence, partly because of unconscious acts of omission, information overload or inaccessible information (McDonald 1976). A number of recent studies suggest that fragmented and inaccessible clinical information adversely affects both the cost and quality of health care as well as compromising patient safety (e.g. Anderson 2007). Healthcare professionals are constantly confronted with multiple clinical decisions to be made about diagnosing, treating, and counseling, in various settings. In addition, physicians are increasingly expected to perform tasks related to health maintenance and preventive care that are not directly related to the patient's acute problem, such as cancer screening and chronic disease management. Because the vast amount of information that is needed to achieve appropriate decisions, various support systems have been developed to convey the proper information at the right place and time. A number of interventions have been designed to reduce omissions and the gap between best practice and routine care: educational interventions (directed at clinicians or patients), clinical practice guidelines, reminders (directed at clinicians or patients), audit and feedback of clinical performance, financial incentives, local opinion leaders, information and communication technologies (e‐health) and organizational changes. Previous reviews have shown that such interventions may have the potential to foster better knowledge translation; however the effects are most often modest on average, have shown large variations in practice and are most frequently based on weak quality of evidence (e.g. Baker, 2015; Fiander 2015; Flodgren 2011; Forsetlund 2009; Gagnon 2009; Giguère 2012; Grimshaw 2004; Ivers 2012; Morris 2002; Shojania 2009; Thomas 1999).

Description of the intervention

According to the US National Library of Medicine, "reminder systems" are approaches, techniques or procedures "used to prompt or aid the memory" of healthcare professionals. "The systems can be computerized reminders, colour coding, telephone calls, or devices such as letters and postcards." (www.ncbi.nlm.nih.gov/pubmed) Reminders have been used for many years and in many different forms. Reminders can be generated electronically or manually, and can be delivered on the computer screen, via email or fax, or in patient paper charts. They also vary in format (e.g. flow chart, electronic message, checklist, sticker) and content (e.g. suggested test date, reference to literature, preventive care suggestions). They can be completely automated and computerized, such as an alert system embedded into computerized provider order entry systems, or completely paper‐based without any involvement of a computer, such as simple notes attached by nursing personnel to the front of charts. A third type of reminder, computer‐generated reminders delivered on paper, combines the two previous approaches. These are automatically generated through a computerized system, but are delivered on paper to the healthcare professional, usually along with the paper‐based medical record, but potentially as a letter they receive outside the consultation.

How the intervention might work

Reminder systems help clinicians overcome barriers to knowledge transfer, remind them to perform tests or interventions that should be performed regularly, e.g. regular foot examination in diabetic patients or yearly influenza vaccine in elderly patients. Indeed, reminders systems prompt clinicians to recall information that they already know or would be expected to know and by providing information or guidance in a more accessible and relevant format, at a particularly appropriate time. Studies and systematic reviews have indicated that reminders to healthcare professionals can be effective in promoting change in healthcare professional practice across a variety of clinical areas and settings (Balas 2000; Buntinx 1993; Kawamoto 2005; Mandelblatt 1995; Shea 1996; Wensing 1994). Reminder systems have been used to target provider behavior across a range of clinical circumstances including preventive, acute and chronic care and to target various behaviors, such as test ordering, vaccination, drug selection, dosing and prescribing, and improving general disease management.

Why it is important to do this review

Previous comprehensive and systematic reviews have covered reminders as one of a wide range of interventions aimed at improving professional practice (Davis 1992; Davis 1995; Garg 2005; Grimshaw 2004; Hunt 1998; Johnston 1994;Oxman 1995), or have focused on computerized reminders (Schedlbauer 2009) or the effectiveness of reminders for a specific behavior, such as preventive care (Balas 2000; Dexheimer 2008; Shea 1996), cancer screening (Baron 2010), vaccination (Ndiaye 2005), diabetes care (Balas 2004), or prescribing practices (Bennett 2003; Pearson 2009). In addition, factors that may modify the effectiveness of reminders have not been systematically considered. For example, specific suggestions or advice have been used by several reviews (Axt‐Adam 1993; Buntinx 1993; Haynes 1987) to distinguish between types of reminder, but few conclusions have been drawn about their impact on the effectiveness of reminders. This may reflect the difficulty of distinguishing explicit advice from implicit advice in many reports of reminder studies. In our view and based on the literature, the effectiveness of reminders may be influenced by their content: whether they provide generic or patient‐specific information; whether they require the healthcare professional to record a response; whether they provide a recommendation for care and not just an assessment; whether they include an explanation or justification of the decision support; whether they are explicitly from, or justified by reference to an influential source; and whether reminders are available at point‐of‐care (Kawamoto 2005; Litzelman 1993). Another potential effect modifier may be the type of targeted behavior. Finally, reminders may also prove useful in low‐ and middle‐income countries; due to a shortage of healthcare workers, support and reminder systems may help volunteer or community health workers to contribute to appropriate care delivery (Mahmud 2010; Tierney 2007). Moreover, a systematic review aimed at evaluating the effectiveness of interventions targeting the transfer of evidence‐based information into practice in developing countries did not find conclusive evidence (Siddiqi 2005).

This review is one of a series covering three major categories of reminder and a fourth that will compare all of these. As well as carrying major resource implications, these categories may influence reminder effectiveness.

Manual paper reminders: no computer is involved in the production or delivery of the reminder, nor in selecting target patients (Pantoja 2014).
Computer‐generated reminders delivered on paper: a computer is used either to generate paper reminders or to identify patients for whom clinicians should receive a paper reminder.
On‐screen reminders: reminders are delivered to clinicians on computer screen (Shojania 2009).

The primary objective of the series is to guide the development and use of clinical reminder systems. When implementing a reminder system, the decision to use manual methods or a computer to produce or deliver reminders has major resource implications as well as usability implications. Although more and more providers adopt electronic medical records (EMR), their comprehensiveness varies and their widespread use is still limited. In 2001 only 29% of primary care physicians in the European Union had implemented electronic medical records, while in the USA less than 17% of primary care physicians routinely use EMRs in their practices (Anderson 2007). Another recent study found that, depending on the definition used, between 8% and 12% of U.S. hospitals have a basic electronic‐records system (Jha 2009). Using a computer to carry out case finding and to generate paper reminders combines the benefits of the speed and accuracy of computers, compared with manual selection of cases by a person, and the low technology paper delivery method that continues to dominate much clinical practice worldwide.

Objectives

In this review, we examined the effects of reminders automatically generated through a computerized system (computer‐generated) and delivered on paper to healthcare professionals on quality of care (outcomes related to healthcare professionals' practice) and patient outcomes (outcomes related to patients' health condition). We addressed the following primary question and subsidiary questions.

Are computer‐generated reminders delivered on paper to healthcare professionals effective in improving quality of care and patient outcomes?

- Are computer‐generated reminders delivered on paper to healthcare professionals alone (single‐component intervention) more effective than usual care?
- Are computer‐generated reminders delivered on paper to healthcare professionals in addition to one or more co‐interventions (multi‐component intervention) more effective than the co‐intervention(s) without the reminder component?

We also addressed the following secondary questions, to identify factors that may systematically modify the effectiveness of reminders, based on features that have been suggested to be effect modifiers in the literature (Baron 2010; Dexheimer 2008; Kawamoto 2005; Litzelman 1993; Mollon 2009; Shiffman 1999).

Content of reminder

Are reminders that include some individual patient‐specific information more effective than generic reminders (i.e. same message for all patients)?
Are reminders that include space for a response from the clinician more effective than reminders that do not include this?
Are reminders that offer specific advice on patient management (i.e. recommendation for care) more effective than reminders that offer general information only (e.g. prevalence of a disease)?
Are reminders that include an explanation of their content or advice (e.g. background information, risk definition) more effective than reminders that do not include this?
Are reminders that are explicitly from, or justified by reference to an influential source more effective than anonymous reminders or those from another source? An influential source can be a systematic review, a practice guideline, a bibliographic citation, or a person or body likely to be perceived as credible by the target clinician.

Delivery of reminder

Are reminders available at point‐of‐care (i.e. at patient's visit) more effective than reminders available at another time (e.g. mailed reminders received after patient's visit)?

Behavior targeted by reminder

Do reminders vary in effectiveness according to the targeted behavior (e.g. test ordering, prescription)?

Methods

Criteria for considering studies for this review

Types of studies

We included trials where individuals (patients or providers) or other units (e.g. practice, hospital) were definitely or possibly assigned prospectively by the investigators to one of two (or more) alternative forms of health care using random allocation (randomized trial) or non‐random method of allocation (non‐randomized trial) such as alternation, date of birth or medical record number, according to EPOC guidance on study designs (EPOC 2015b). We included non‐randomized trials because in complex interventions that are evaluated in routine practice, conducting a randomized trial may be neither feasible nor acceptable. Non‐randomized trial designs can be better suited for real‐life situations and may better reflect the effectiveness of the intervention.

Types of participants

Any qualified healthcare professional, or a population where qualified healthcare professionals form the majority of the study population.

Types of interventions

Reminders are patient‐ or encounter‐specific information, which are designed or intended to prompt a healthcare professional to recall information usually encountered through their general medical education, in the medical records or through interaction with peers, and to remind them to perform or avoid some action to aid individual patient care. Reminders differ from feedback interventions in terms of content: feedback consists of a summary of clinical performance over a specified period of time, and typically aggregates information on multiple patients. Reminders also must not contain any new information about the patient such as a laboratory result that is not in the case notes or a score derived from a clinical prediction rule that was previously unknown to the clinician.

This review considered computer‐generated reminders delivered on paper. A computer had to be involved in producing the reminder for eligible patients or in selecting the patients about whom the clinician received a reminder, or both. If a computer was merely used as a medium to print the reminder without any other function, the reminder was not considered as computer‐generated. We also included applications of computerized algorithms to identify eligible patients, for whom the prompt is printed out and placed in the chart. Information was usually obtained from computerized medical records or a computerized database. Once generated, the reminder had to be delivered on paper (fax included), and not on a computer screen or via email or text message.

To be included in the review, the reminder had to target a healthcare professional who delivered the care directly to patients, not an intermediary (e.g. clinic receptionist, clinician manager). Expert systems for facilitating diagnosis or estimating prognosis were not considered as reminders, even if their output was printed out. A document listing all the drugs a patient was currently taking (e.g. drug profile) or a document summarizing the medical records, with no rules applied in the computer, were not considered as reminders, but as an organizational intervention (i.e. changes in the medical records systems). New clinical information collected directly from patients on a computer and given to the provider as a prompt was not considered as a reminder intervention, but as a patient‐mediated intervention.

Types of outcome measures

Quality of care is the primary outcome of this review as the main purpose of reminders is to change healthcare professional practice and affect a quality of care endpoint, such as ordering a test or initiating a treatment. This targeted practice change should, in turn, improve patient outcomes, based on evidence. Thus, if the reminder is aimed at modifying a drug prescription for a simpler or cheaper treatment, the latter prescription should have been shown as having at least similar effectiveness as the current treatment (indirect evidence). Studies of reminders rarely target changes in patient outcomes directly. Moreover, the targeted modification may not be linked to an actual change in patient outcome, for instance when replacing a proprietary drug by a generic equivalent.

Primary outcomes

Quality of care

Dichotomous outcomes related to healthcare professionals' practice: the percentage of patients receiving a target process of care (e.g. ordering of a test, prescription for a medication) or whose care was in compliance with an overall guideline (e.g. percentage of women up‐to‐date with a breast cancer screening recommendation). Instead of patients in the denominator, this could be patient encounters or reminders (e.g. number of recommendations followed over the number of recommendations due during an encounter).
Continuous outcomes related to healthcare professionals' practice: any continuous measure of how providers delivered care (e.g. duration of therapy, time to event).

Secondary outcomes

Patient outcomes

Dichotomous outcomes related to patients' health condition: the percentage of clinical endpoints (e.g. death, development of a disease such as pneumonia, stroke, heart attack, etc.) or the percentage of surrogate or intermediate endpoints, such as a continuous measures of disease control that have been dichotomized and reported as percentage of patients with sufficient or insufficient control (e.g. percentage of diabetics reaching the glycated hemoglobin target (< 7%), percentage of patients reaching systolic blood pressure target (< 140 mmHg)).
Continuous outcomes related to patients' health condition: various markers of disease or health status (e.g. blood pressure, body mass index, glycated hemoglobin levels) that were captured and analyzed as continuous variables.

Adverse effects outcomes: any adverse effects described in the study, such as redundant testing or overdiagnosis.

Search methods for identification of studies

Electronic searches

The Information Specialist for the EPOC Group conducted the searches on 21 September 2016; exact search dates, search terms, syntax and number of results are provided for each database and may be found in Appendix 1. Previous searches can be found in the previous version of the review (Arditi 2012).

We searched the following databases.

Cochrane Central Register of Controlled Trials (CENTRAL; 2016, Issue 8) in the Cochrane Library
Cochrane Database of Systematic Reviews (CDSR; 2016, Issue 9) in the Cochrane Library
Health Technology Assessment Database (HTA; 2016, Issue 3) in the Cochrane Library
Database of Abstract of Reviews of Effectiveness (DARE; 2015, Issue 2) in the Cochrane Library
NHS Economic Evaluations Database (NHSEED; 2015, Issue 2) in the Cochrane Library
MEDLINE via OVID (from 1946)
Embase via OVID (from 1974)
CINAHL (Cumulative Index to Nursing and Allied Health Literature) via Ebscohost (from 1980)
INSPEC via Web of Science(from 1969)

Searching other resources

In addition to database searching, we examined reference lists of key articles and relevant reviews, handsearched the WHO International Clinical Trials Registry (http://apps.who.int/trialsearch/), the US National Institutes of Health (NIH) Clinical Trials Registry (http://clinicaltrials.gov/) and contacted authors of relevant reviews and studies regarding any further published or unpublished work.

Data collection and analysis

For this update, we used the same data collection tool defined in the protocol and used in the previous version of this systematic review (Arditi 2012).

Selection of studies

Two assessors (JW, SY), working independently, screened titles and abstracts of references located by the literature search for potential relevance. We retrieved full‐text copies of all potentially relevant studies for full‐text assessment. Many studies were rated as potentially relevant in the first selection process, as it was often unclear whether computerized reminders were provided to the healthcare professional on paper or on a computer screen, and whether the reminders were computer‐generated. Two assessors, again working in pairs (CA, SY), independently assessed studies for inclusion. Studies that appear to meet the inclusion criteria, but on closer examination failed to, are detailed in the table of excluded studies.

Data extraction and management

Two assessors independently carried out data extraction (SY, CA), using the EPOC Data Collection Checklist modified to capture more detailed information in some areas (e.g. content of the reminder). Any discrepancies between assessors arising from the inclusion assessment or from the data extraction process were resolved by discussion and the involvement of a third review author. Decisions that could not be resolved easily were referred to the EPOC contact editor.

Assessment of risk of bias in included studies

The risk of bias for all included studies was independently assessed in pairs (SY, CA) using the nine suggested risk of bias criteria for EPOC reviews (EPOC 2015a).

Measures of treatment effect

For each study we reported the main results in natural units in a results table. Where baseline results were available, pre‐intervention proportions and means were also reported for both study and control groups. The unadjusted and adjusted (for baseline imbalance) differences (in proportion or mean) between study and control groups at endpoint were calculated for the outcomes. The direction of the effect size was standardized so that a positive difference between post‐intervention percentages or means indicated a positive outcome.

Unit of analysis issues

We anticipated that cluster‐randomized trials would be common, which is often the case in interventions aimed at healthcare professionals. There is a high risk of contamination when patients are randomized rather than professionals since clinicians’ experience of applying the intervention to patients receiving the experimental management may contaminate the way they treat control patients (Biau 2008; Kahan 2013). We also expected that such trials would rarely take into account the cluster effect in the analysis (i.e. unit of analysis error resulting in artificially extreme P values and over narrow confidence intervals (Ukoumunne 1999). Performing a meta‐analysis involving both trials randomizing patients and clusters would require us to make assumptions about unknown parameters, such as intra‐class correlation coefficients and the distribution of patients across clusters, to avoid spurious precision in 95% confidence intervals. In addition, we expected a large variety of interventions, outcomes and response scales, as well as a very wide contextual and clinical heterogeneity in existing studies’ reports. We thus decided to report the median improvement and interquartile range (IQR) across the included studies in order to avoid unit of analysis issues when combining results from cluster‐ and patient‐randomized trials.

Dealing with missing data

No data were missing.

Assessment of heterogeneity

We explored heterogeneity visually by preparing box plots displaying median effects and IQRs (see Subgroup analysis and investigation of heterogeneity for further details).

Assessment of reporting biases

We explored the possibility of publication bias by plotting the number of patients and professionals included in the studies against the median effect size.

Data synthesis

We combined cluster‐ and patient‐randomized trials using the median improvement and IQR. This approach was first developed in a large review of guideline dissemination and implementation strategies (Grimshaw 2004) and used in the systematic review on the effects of on‐screen, point‐of‐care reminders (Shojania 2009). Briefly, each study is represented by a single representative outcome and the median effect size and IQR are calculated across the included studies. By using the median rather than the mean, the summary estimate is less likely to be influenced by outlying results (e.g. large effects from methodologically poor studies). In contrast to conventional meta‐analysis, where each study is given a weight based on the precision of the results, here each study is given equal weight. The impact of study size and various methodological features were investigated in pre‐specified subgroup analyses.

The representative outcome of studies reporting more than one outcome was the primary outcome measure when it was defined as such by the authors of the study. If authors did not specify the primary outcome but provided an aggregated outcome (e.g. overall physician compliance), we selected that aggregated outcome as a representative outcome. If a primary outcome was not available, we calculated the median effect size across all reported outcomes. For example, if the study reported five dichotomous quality of care outcomes and none of them were denoted the primary outcome, we ranked the effect sizes for the five quality of care outcomes and took the median value. If there was an even number of outcomes, we calculated the average of the two middle outcomes.

Summary of findings

We summarized the findings in three 'Summary of findings' tables to draw conclusions about the certainty of the evidence. Two review authors (BB, CA) independently assessed the certainty of the evidence (high, moderate, low, and very low) using the five GRADE considerations (study limitations, consistency of effect, imprecision, indirectness, and publication bias) (Guyatt 2011). We used methods and recommendations described in Section 8.5 and Chapter 12 of the Cochrane Handbook for Systematic Reviews of Interventions (Higgins 2011), the EPOC worksheets (EPOC 2015c), and by using GRADEpro software (GRADEpro GDT 2015). We resolved disagreements on certainty ratings by discussion and provided justification for decisions to down‐ or up‐grade the ratings using footnotes in the tables and made comments to aid readers' understanding of the review where necessary. We used plain language statements to report these findings throughout the review.

Subgroup analysis and investigation of heterogeneity

We compared the median improvement across studies according to the following potential effect modifiers, pre‐defined in the protocol and based on features that have been suggested to be effect modifiers in the literature (Baron 2010; Dexheimer 2008; Kawamoto 2005; Litzelman 1993; Mollon 2009; Shiffman 1999):

patient‐specific: whether the reminder provided generic knowledge or advice with no patient data or patient‐specific advice (i.e. same message or advice for all patients) or patient‐specific knowledge or advice;
space for response: whether the reminder provided space for the healthcare professional to record a response/comment (e.g. a box to tick or line to write on) or not;
specific advice: whether the reminder provided advice on patient management or recommendation for care (e.g. consider reducing dosage of drug) or not (e.g. prevalence of disease);
explanation: whether the reminder was supported by an explanation (e.g. background information, definitions, risks, rationale) or not (e.g. last pap smear test date);
reference: reminders were explicitly from or justified by reference to an influential source (e.g. clear reference to a systematic review or national guidelines) or not;
at point‐of‐care: whether the reminder was delivered to healthcare professional when providing care to the patient (at point‐of‐care) or not (e.g. reminder sent by mail after patient's visit).

We also compared the median improvement across studies according to the type of behavior targeted by the reminder (e.g. prescription, test ordering) and the following features of the study: study design, allocation method, sample size (patients and professionals), setting, country, duration of intervention, and publication year. We also investigated the median improvement in disadvantaged populations, in terms of economic status, place of residence and ethnicity.

We used the non‐parametric Wilcoxon rank‐sum test (also known as the Mann‐Whitney two‐sample statistic) for two‐levels variables and the Kruskal‐Wallis test for variables with more than two levels. We performed all statistical analyses using Stata version 10 (Stata 2007).

Sensitivity analysis

We performed sensitivity analyses based on study design criteria deemed important in the context of this review (only including studies with allocation concealment and complete outcome data) and data availability (excluding trials where data were estimated from graphs). We also re‐analyzed the data using three alternative methods for representing the outcome from each study: using the median outcome as representative outcome, even for studies reporting a primary outcome; using the reported outcome showing the largest improvement (largest outcome); and using the reported outcome showing the smallest improvement (smallest outcome).

Results

Description of studies

See: Characteristics of included studies; Characteristics of excluded studies.

Results of the search

See: Figure 1.

Study flow diagram

*Ineligible comparison or inappropriate control: e.g. physician reminder combined with another intervention vs usual care, physician reminder with a specific feature vs physician reminder without it, physician reminder vs another intervention

^$Not a provider reminder: e.g. audit and feedback, changes in medical records system, expert system for estimating diagnosis/risk/dosage, patient‐mediated intervention

We identified 7225 records (including three studies awaiting classification in the original review and three studies identified via handsearching), of which 7056 were excluded after screening the title and abstract. After assessing full‐texts for the remaining 169 records, we retained three new studies (Gilutz 2009; Le Breton 2016; Were 2013). In total, we included 35 studies in the qualitative synthesis and 34 studies in the quantitative synthesis (one study did not report usable outcome data). Six studies (Burack 1996; Burack 1998; McPhee 1989; Ornstein 1991; Tierney 1986; Ziemer 2006) contained four study groups (i.e. reminders alone, reminders with co‐intervention(s), co‐intervention(s) without reminder component, usual care), resulting in 40 eligible comparisons in the quantitative analyses.

Included studies

Design

Thirty studies were randomized trials, including one cross‐over trial (McDonald 1980), and five studies were non‐randomized trials (Mazzuca 1990; McDonald 1976a; Morgan 1978; Oniki 2003; Turner 1989), including one cross‐over trial (McDonald 1976a). Among the 35 included studies, 15 allocated patients to study groups (Barnett 1983; Becker 1989; Binstock 1997; Burack 1996; Burack 1998; Chambers 1989; Heidenreich 2005; Heidenreich 2007; Javitt 2005; McDonald 1976b; Morgan 1978; Oniki 2003; Thomas 1983; Were 2013; White 1984), while the other studies used cluster‐allocation methods. The unit of allocation was the health professional in 10 studies (Chambers 1991; Le Breton 2016; Lobach 1997; Majumdar 2007; McAlister 2009; McDonald 1976a; McDonald 1980; McPhee 1989; Nilasena 1995; Rossi 1997), the clinic, clinic session or health professional team in nine studies (Dexter 1998; Gilutz 2009; Heiman 2004; Mazzuca 1990; McDonald 1984; Ornstein 1991; Tierney 1986; Turner 1989; Ziemer 2006), and the family in one study (Rosser 1991).

Participants, setting and publication date

All studies included at least 100 patients in the analyses (median 751, mean 2275); the number of patients was not reported in two studies (Javitt 2005; McDonald 1980). The healthcare professionals were primarily physicians, although some studies also included other professionals such as nurse practitioners. One study included only nurses (Oniki 2003). Healthcare professionals' level of training varied across studies. In the cluster‐randomized studies, the number of professionals, for whom outcome data were obtained, varied between nine and 600 (median 57, mean 104).

Most included studies were based in North America (29 in the USA, three in Canada). The three remaining studies were based in France (Le Breton 2016), Israel (Gilutz 2009), and Kenya (Were 2013). Most studies took place in outpatient settings, while two took place in inpatient settings (Oniki 2003; White 1984) and three in mixed settings (Heidenreich 2005; Heidenreich 2007; Javitt 2005).

About 70% of the studies were published between 1980 and 2000.

Interventions

Physician reminders alone (single‐component intervention) were compared with usual care in 28 studies (Barnett 1983; Becker 1989; Binstock 1997; Burack 1996; Burack 1998; Chambers 1989; Chambers 1991; Dexter 1998; Gilutz 2009; Heidenreich 2005; Heidenreich 2007; Heiman 2004; Javitt 2005; Le Breton 2016; Lobach 1997; McDonald 1976a; McDonald 1976b; McDonald 1980; McDonald 1984; McPhee 1989; Morgan 1978; Oniki 2003; Rosser 1991; Rossi 1997; Thomas 1983; Tierney 1986; Were 2013; White 1984). Physician reminders in addition to one or more co‐interventions (multi‐component intervention) were compared with the co‐intervention(s) without the reminder component in 11 studies. There was one co‐intervention in seven studies (Burack 1996; Burack 1998; Majumdar 2007; Mazzuca 1990; McAlister 2009; Nilasena 1995; Tierney 1986), two co‐interventions in four studies (McPhee 1989; Ornstein 1991; Turner 1989; Ziemer 2006), and three co‐interventions in study groups of two studies (Ornstein 1991; Ziemer 2006). The most common co‐interventions were patient reminder, educational meeting for healthcare professionals, and audit and feedback.

The same reminder was provided for all eligible patients (e.g. order a pap smear test) in 15 comparisons. Between two and 10 different reminders could be provided for patients in 19 studies, while over 10 different reminders could be provided for eligible patients in the remaining seven comparisons (McDonald 1980; McDonald 1984; Nilasena 1995; Thomas 1983; Tierney 1986; Tierney 1986; Were 2013).

The categorization of reminders for each included study is provided in the Characteristics of included studies tables. Reminders in all comparisons except one (Chambers 1991) were patient‐specific. The use of the computer to select patients allowed the reminders to be sent to eligible patient records only and thus be patient‐specific. Reminders in 19 comparisons included space for the provider to respond to the reminder (e.g. a check box to order a mammogram). Reminders offered specific advice on patient management (i.e. recommendation for care) in 35 comparisons and included an explanation of their content or advice (e.g. background information, risk definition) in 13 comparisons. Reminders were explicitly from or justified by reference to an influential source (e.g. systematic review, bibliographic citation) in 11 comparisons. Reminders were provided to physicians at the point‐of‐care (i.e. during the patient's visit) in all comparisons except five, where reminders were sent after patients' visits directly to physicians.

The median duration of the reminder intervention was 11 months (range two to 56 months); the duration was not reported in two studies (Binstock 1997; Dexter 1998).

Clinical domain and targeted behavior

Reminders were aimed at prompting the physicians to provide preventive care services in half of the comparisons. In these studies, the most common objective was to enhance compliance with cancer screening tests (e.g. mammography, Papanicolaou smear, rectal examination) or vaccination. In the remaining comparisons, reminders were provided to physicians seeing patients with an acute or chronic condition, such as diabetes, HIV and cardiovascular disease, to enhance compliance with disease management guidelines (e.g. foot examination in diabetes patients, blood pressure check in hypertensive patients, prescribing angiotensin converting enzyme (ACE) inhibitors in patients with cardiovascular problems).

In 23 comparisons, reminders targeted one type of behavior. The behavior was test ordering (e.g. mammography, glycated hemoglobin) in 10 comparisons, vaccination in one comparison, prescribing in seven comparisons, professional‐patient communication in two comparisons, and general management in three comparisons. In the remaining 17 comparisons, reminders targeted multiple behaviors: two types of behaviors in nine comparisons and three or four types of behaviors in the other eight comparisons. In one comparison, the number of behaviors was unclear.

Outcome measures

There were large variations in the kind of outcome measure, and many studies reported multiple outcomes, especially studies on compliance with more than one guideline. Most trials measured quality of care outcomes, such as prescribing or test ordering rates. Six studies also reported patient outcomes such as blood pressure or cholesterol levels (Barnett 1983; Gilutz 2009; Heidenreich 2005; McAlister 2009; Rossi 1997; Ziemer 2006). All studies except one (Oniki 2003) reported at least one dichotomous quality of care outcome.

Excluded studies

We excluded 166 studies in this update, in addition to the 297 studies excluded in the original review. Twenty‐seven studies were excluded because of ineligible comparison or inappropriate control (e.g. physician reminder combined with another intervention versus usual care, physician reminder with a specific feature versus physician reminder without it, physician reminder versus another intervention). Four studies were excluded because reminders were presented to physicians on paper and onscreen at the same time, thus not allowing us to determine the effect of the paper reminder alone. When we retrieved full‐texts, we found that reminders in 46 studies were presented to physicians on a computer screen or sent by email. Computers were not involved in generating the reminder in 10 studies. In 21 studies, interventions were not provider reminders (e.g. audit and feedback, changes in medical records system, expert system for estimating diagnosis/risk/dosage, patient‐mediated intervention). Fourty‐four studies were excluded because of study design and 11 because the publication was not an original study. We excluded two studies because their objective was not to improve professional practice and one study did not provide sufficient information to determine its eligibility. We listed 52 of the 166 excluded studies in the Characteristics of excluded studies that may appear to meet the eligibility criteria to readers.

Risk of bias in included studies

See Figure 2; Figure 3 for summaries of risk of bias, and the Characteristics of included studies for details of risk of bias in each study.

Methodological quality summary: review authors' judgements about each methodological quality item for each included study.

Methodological quality graph: review authors' judgements about each methodological quality item presented as percentages across all included studies.

Allocation

Among the 30 randomized trials, the 15 studies that described the sequence generation by referring to a computerized randomization program or a random number table (Chambers 1989; Chambers 1991; Heidenreich 2005; Heidenreich 2007; Heiman 2004; Javitt 2005; Le Breton 2016; Lobach 1997; Majumdar 2007; McAlister 2009; McDonald 1976b; Rosser 1991; Rossi 1997; Were 2013; White 1984) were at low risk of bias. The process of sequence generation was unclear for the other 15 randomized trials, which merely stated that the study groups were randomly allocated (Barnett 1983; Becker 1989; Binstock 1997; Burack 1996; Burack 1998; Dexter 1998; Gilutz 2009; McDonald 1980; McDonald 1984; McPhee 1989; Nilasena 1995; Ornstein 1991; Thomas 1983; Tierney 1986; Ziemer 2006). Allocation concealment occurred in nine randomized trials, while it was unclear in the remaining randomized trials.

The five non‐randomized trials were at high risk of bias for sequence generation and allocation concealment.

Unit of allocation issues

Of the 20 studies with a cluster design, only seven analyzed results at the level of the cluster (Lobach 1997; Mazzuca 1990; McPhee 1989; Nilasena 1995; Tierney 1986; Turner 1989; Ziemer 2006), while the other studies analyzed results at the patient level (Chambers 1991; Dexter 1998; Gilutz 2009; Heiman 2004; Le Breton 2016; Majumdar 2007; McAlister 2009; McDonald 1984; Ornstein 1991; Rosser 1991, Rossi 1997) or the reminder level (McDonald 1976a; McDonald 1980). Such unit of analysis errors artificially increase the precision of statistical tests and may lead to inappropriate conclusions. Five of these studies re‐analyzed the data taking into account the clustering effect (Dexter 1998; Heiman 2004; Le Breton 2016; McAlister 2009; Rossi 1997). One study (Majumdar 2007) minimized the unit of analysis error by not allowing physicians to contribute more than five patients.

Blinding

Due to the nature of the intervention, blinding was only assessed with regards to the outcome assessment method. Five studies reported that outcome assessors were blinded (Dexter 1998; Le Breton 2016; Majumdar 2007; McAlister 2009; White 1984) and two studies performed an audit of outcome assessments (Lobach 1997; McPhee 1989). Ten further studies reported that outcomes were derived from a computerized medical records system, minimizing risk of bias (Binstock 1997; Burack 1996; Gilutz 2009; Heidenreich 2007; Mazzuca 1990; McDonald 1984; Oniki 2003; Ornstein 1991; Rossi 1997; Tierney 1986). While two studies reported that outcomes were not assessed blindly (Heiman 2004; Turner 1989); the other studies did not report on blinding procedures.

Incomplete outcome data

Outcome data were considered complete when 80% or more of the patients randomized were included in the analyses or when reasons for attrition were similar across groups. These were reported in 16 studies (Barnett 1983; Becker 1989; Burack 1996; Chambers 1991; Dexter 1998; Gilutz 2009; Heidenreich 2005; Heidenreich 2007; Heiman 2004; Javitt 2005; Le Breton 2016; Lobach 1997; Majumdar 2007; McAlister 2009; McDonald 1984; Rosser 1991). Outcome data were considered incomplete in four studies, where the percentage of patients analyzed was less than 80% of patients randomized and no reason was given for the missing data (Burack 1998; McPhee 1989; Nilasena 1995; Thomas 1983). In the remaining studies, the number of patients lost to follow‐up was unclear.

Other potential sources of bias

Baseline measurement of the outcome of interest was reported in 13 studies. Among these studies, 10 reported that study groups were comparable at baseline (Chambers 1989; Heidenreich 2005; Heidenreich 2007; Heiman 2004; Le Breton 2016; Lobach 1997; Mazzuca 1990; McAlister 2009; McPhee 1989; Ziemer 2006), while three reported significant differences (Nilasena 1995; Ornstein 1991; Turner 1989). Across studies reporting a baseline measurement of outcome, the median difference between intervention and control groups at baseline was 1%.

Two thirds of the studies reported patient characteristics at baseline that permitted assessment of baseline heterogeneity in characteristics between study groups. Six studies reported significant differences (Dexter 1998; Gilutz 2009; Heiman 2004; Le Breton 2016; Ornstein 1991; Rossi 1997).

Lack of protection against contamination is a potential source of bias in interventions targeting healthcare professionals. Indeed, there is a risk that physicians who receive reminders for some patients but no reminders for other patients may improve their behavior in both groups, thus reducing the chance of measuring a difference between the study groups. Sixteen studies prevented contamination by allocating physicians or practices to study groups, eliminating the risk of physicians receiving reminders for some patients and no reminders for others.

Effects of interventions

See: Table 1; Table 2; Table 3

The outcomes considered for each study included in the analyses are described in detail in Table 4. The absolute improvement in quality of care for studies reporting a primary outcome and the median improvement and interquartile range (IQR) for studies reporting more than one eligible outcome are displayed in Figure 4.

1. Improvement rates of quality of care, by study.

Study ID	Primary outcome	Other outcomes (n)	Absolute improvement ‐ using primary outcome	Median absolute improvement ‐ using other outcomes (interquartile range)
Barnett 1983		percentage of eligible patients with: blood pressure values on record, follow‐up (2)		38.8% (18.4% to 59.1%)
Becker 1989	overall compliance rate with preventive care recommendations	percentage of eligible patients with: dental check, ocular pressure check, FOBT, flu vacc, pneumo vacc, tetanus vacc, mammography, pap smear (8)	4.7%	5.8% (3.0% to 10.2%)
Binstock 1997	percentage of eligible patients with pap smear		7.6%
Burack 1996_1	percentage of eligible patients with mammography		6.0%
Burack 1996_2	percentage of eligible patients with mammography		4.0%
Burack 1998_1	percentage of eligible patients with pap smear		1.0%
Burack 1998_2	percentage of eligible patients with pap smear		3.0%
Chambers 1989	percentage of eligible patients with mammography		7.1%
Chambers 1991	percentage of eligible patients with flu vacc		20.7%
Dexter 1998		percentage of eligible patients with: discussion of directives, completion of directives (2)		9.2% (6.1% to 12.3%)
Gilutz 2009		percentage of patients with adequate monitoring, percentage of eligible patients with initiation or up‐titration of statin therapy, percentage of eligible patients with up‐titration (3)		5.4% (1.2% to 6.1%)
Heidenreich 2005	percentage of eligible patients with ACE inhibitor prescription		11.5%
Heidenreich 2007	percentage of eligible patients with any β‐blocker prescription	percentage of eligible patients with recommended β‐blocker prescription	7.9%	6.7% (5.4% to 7.9%)
Heiman 2004	percentage of eligible patients with advance directives	percentage of eligible patients with: discussion or completion of directives, completion of healthcare proxy, completion of living will (3)	‐0.2%	‐0.3% (‐0.9% to ‐0.2%)
Javitt 2005	compliance rate with prescription reminders (start a new drug) (denominator: reminders)		7.0%
Le Breton 2016	adherence to colorectal cancer screening		1.7%
Lobach 1997	overall physician compliance rate	physician compliance rate with: foot exam, physical exam, glycated hemoglobin, urine protein determination, cholesterol level, eye exam, flu vacc, pneumo vacc (8)	16.4%	22.7% (11.0% to 28.4%)
Majumdar 2007	overall compliance rate with prescription reminders	percentage of eligible patients with: ACE inhibitor or ARB therapy prescription, statins prescription	6.0%	9.0% (0.0% to 18.0%)
Mazzuca 1990		physician compliance rate with: glycated hemoglobin, fasting blood glucose, home‐monitored blood glucose, diet clinic referral, oral hypoglycemic agents (5)		4.0% (4.0% to 5.0%)
McAlister 2009	overall compliance rate with prescription reminders	percentage of eligible patients with: statins, standardized statin dose, another lipid‐lowering drug, acetylsalicylic acid, acetylsalicylic acid or thienopyridine, ACE inhibitor, ACE inhibitor or ARB, β‐blocker, triple therapy (8)	6.6%	0.5% (‐0.4% to 2.2%)
McDonald 1976a	overall compliance rate with prescription reminders (denominator: reminders)	compliance with: observing a physical finding or inquiring about a symptom, ordering a diagnostic study, changing or initiating a therapeutic regimen (3)	28.9%	24.7% (21.1% to 38.8%)
McDonald 1976b	overall compliance rate with reminders (denominator: reminders)	percentage of patients with: test order, therapeutic change (2)	23.5%	20.3% (14.9% to 25.7%)
McDonald 1980	overall compliance rate with reminders (denominator: reminders)		18.6%
McDonald 1984	overall compliance rate with reminders	percentage of patients with: FOBT, pap smear, chest roentgenogram, pneumo vacc, tuberculosis skin test, serum potassium, mammogram, flu vacc, diet, digitalis, antacids, β‐blockers (12)	20.0%	13.0% (10.5% to 24.5%)
McPhee 1989_1		physician compliance rate with: FOBT, rectal exam, sigmoidoscopy, pap smear, pelvic exam, breast exam, mammography (7)		23.0% (20.0% to 33.0%)
McPhee 1989_2		physician compliance rate with: breast exam, mammography (2)		23.2% (20.0% to 26.5%)
Morgan 1978		percentage of patients with: blood group and type, syphilis serology, prenatal counseling, pregnancy diet counseling, sickle cell preparation (5)		0.1% (‐1.9% to 2.0%)
Nilasena 1995	overall physician compliance rate with reminders		3.9%
Ornstein 1991_1		percentage of eligible patients with: FOBT, mammography, tetanus vacc, cholesterol, pap smear (5)		4.4% (3.9% to 6.9%)
Ornstein 1991_2		percentage of eligible patients with: FOBT, mammography, tetanus vacc, cholesterol, pap smear (5)		6.1% (3.9% to 7.0%)
Rosser 1991	overall compliance rate	percentage of eligible patients with: flu vacc, tetanus vacc, BP reading, pap smear (4)	19.2%	11.4% (6.0% to 16.4%)
Rossi 1997	percentage of eligible patients with prescription change		11.0%
Thomas 1983	compliance rate with reminders		12.9%
Tierney 1986_1		physician compliance rate with: FOBT, pneumo vacc, antacids, TB skin testing, β‐blockers, nitrates, anti‐depressants, calcium supplements, pap smear, mammography, metronidazole, digitalis, salicylates (13)		1.5% (0.5% to 11.0%)
Tierney 1986_2		physician compliance rate with: FOBT, pneumo vacc, antacids, TB skin testing, β‐blockers, nitrates, anti‐depressants, calcium supplements, pap smear, mammography, metronidazole, digitalis, salicylates (13)		1.0% (‐0.5% to 2.0%)
Turner 1989		physician compliance rate with: FOBT, rectal exam, pap smear, breast exam, mammography (5)		3.6% (‐5.8% to 10.1%)
Were 2013	completion of overdue clinical tasks (denominator: reminders)	completion of overdue clinical task for: ordering chest x‐ray, ordering 18‐mo human immunodeficiency virus enzyme‐linked immunosorbent assay, ordering other laboratory tests, beginning antiretroviral therapy, referring to nutritional support (5)	50.0%	39.0% (26.0% to 54.0%)
White 1984	compliance rate with reminders (denominator: reminders)		12.0%
Ziemer 2006_1	physician compliance rate		0.2%
Ziemer 2006_2	physician compliance rate		0.7%

Open in a new tab

ACE: angiotensin‐converting enzyme, ARB: angiotensin II receptor blockers, BP: blood pressure, flu: influenza, FOBT: fecal occult blood test, pneumo: pneumococcal, TB: tuberculosis, Vacc: vaccination

Quality of care

Computer‐generated reminders delivered on paper to healthcare professionals, alone (single‐component intervention) or in addition to co‐intervention(s) (multi‐component intervention), probably improve slightly quality of care compared with usual care or the co‐intervention(s) without the reminder component (median improvement 6.8% (IQR: 3.8% to 17.5%); 34 studies (40 comparisons); moderate‐certainty evidence) (see Table 1).

Computer‐generated reminders delivered on paper to healthcare professionals alone (single‐component intervention) probably improves quality of care compared with usual care (median improvement 11.0% (IQR 5.4% to 20.0%); 27 studies (27 comparisons); moderate‐certainty evidence) (see Table 2). Adding computer‐generated reminders delivered on paper to healthcare professionals to one or more co‐interventions (multi‐component intervention) probably improves quality of care slightly compared with the co‐intervention(s) without the reminder component (median improvement 4.0% (IQR 3.0% to 6.0%); 11 studies (13 comparisons); moderate‐certainty evidence) (see Table 3).

A possible explanation for the different magnitude of effect according to the presence of co‐intervention(s) would be that co‐interventions delivered to both groups leave little room for reminders to demonstrate additional improvement. Indeed, the median post‐intervention quality of care rate in the additional intervention(s) alone control groups was higher than the rate in the usual care groups (median: 27.4% versus 21.8%).

Of the 40 comparisons, 14 reported baseline quality of care rates for study groups. For these comparisons, the median marginal improvement in the intervention group (i.e. the improvement in the intervention group minus the improvement in the control group) was 3.9% (IQR 0.5% to 7%).

Subgroup analyses: impact of reminder features on quality of care effect size

We examined the impact of a number of characteristics of the reminders on the magnitude of effect (Figure 5). Effect size was associated with three features: the availability of space for healthcare professionals to enter a response (median 13.7% versus 4.3% for no space, P = 0.01), reminders including an explanation of their content or advice (median 12.0% versus 4.2% for no explanation, P = 0.02), and reminders explicitly from or justified by reference to an influential source (median 20.0% versus 5.4% for no reference, P = 0.04). The following reminder features were not associated: specific advice included in the reminder (median 6.1% versus 13.9%, P = 0.49), and reminders available at point‐of‐care (median 7.1% versus 6.0%, P = 0.93). The impact of whether the reminder was patient‐specific or generic was not assessed, as only one study examined generic reminders.

Median effect and interquartile range (IQR) across comparisons by reminder feature (P values reflect Mann–Whitney test)

The median improvement in quality of care associated with reminders differed according to the targeted behavior but not the number of targeted behaviors. The largest improvement seen was in vaccination, with a median improvement of 13.1% (IQR 12.2% to 20.7%), while the smallest improvement seen was for professional‐patient communication, with a median reduction of ‐0.2% (IQR ‐2% to 9.2%).

Subgroup analyses: impact of study features on quality of care effect size

There were sufficient comparisons to permit analyses of potential associations between various study features and the magnitude of effect (Figure 6). No association was found between effect size and study features, except for patient sample size. Studies with a small patient sample size achieved larger improvements than studies with a large patient sample size (median 11.8% versus 4.9%, P = 0.05).

Median effect and interquartile range (IQR) across comparisons by study feature (*Kruskall–Wallis test; other P values reflect Mann–Whitney test)

Studies published up to 1990 showed larger improvements than those published after 1990 (median 12.4% for up to 1990, 6.1% for 1991 to 2000 and 6.0% for 2001 to 2015, P = 0.28). To determine whether this reflected temporal changes in baseline rates, we examined the baseline quality of care rates in the control and intervention groups in the 14 comparisons reporting baseline data; there was no temporal trend in either group. Baseline adherence rates were actually higher in the studies published before 1990 reporting baseline rates than in those published after 1990. We also looked at the post‐intervention quality of care rates in the control and intervention groups in all 40 comparisons, which were similar across the years.

Only the two cross‐over studies (McDonald 1976a; McDonald 1980) examined quality of care after the end of the reminder delivery. Neither study showed a statistical carry‐over effect of experimental effect into control periods.

Subgroup analyses: disadvantaged populations

The effect of provider reminders in settings serving disadvantaged and minority populations has been specifically evaluated in 10 studies. Eight studies took place in inner‐cities in the USA, with high rates of African‐American, economically disadvantaged, medicaid eligible and uninsured populations, aiming to improve preventive care rates (Becker 1989; Burack 1996; Burack 1998; Chambers 1989; Chambers 1991; Ornstein 1991; Turner 1989) or to improve diabetes care (Ziemer 2006). In addition, the French study (Le Breton 2016) aimed to improve screening rates in a population where a quarter lived in socio‐economically deprived areas and the Kenyan study (Were 2013) aimed to improve pediatric HIV care in a resource‐limited setting. The improvement of quality of care achieved with reminders in these studies with disadvantaged populations (median 4.2%, IQR 1.7% to 6.1%, 14 comparisons) was lower than the median improvement in studies not focusing on disadvantaged populations (median 10.3%, IQR 5.4% to 19.2%, 26 comparisons), and the overall median improvement (median 6.8%, IQR 3.8% to 17.5%, 40 comparisons). Also, the baseline quality of care rates in the studies in disadvantaged population (19.5% in the control group and 21.8% in the intervention group, seven comparisons with baseline data) was lower than the baseline quality of care rates in the studies in general population (34.6% in the control group and 38.0% in the intervention group, seven comparisons with baseline data).

Sensitivity analyses

Similar median improvement of quality of care was observed when only studies with allocation concealment and complete outcome data were considered (median improvement: 6.8%, IQR 3.9% to 9.7%) and when excluding the six studies with estimated data (median improvement: 5.0%, IQR: 1.5% to 23.0%).

Table 5 shows the results obtained when we re‐analyzed the median improvement of quality of care using the outcome with the largest improvement and the outcome with the smallest improvement for the representative outcome for each study, respectively. As expected, median improvement was larger when using the largest outcome and smaller when using the smallest outcome for all three comparisons. The IQR range included 0 in one comparison: when using the smallest outcome in the reminder with co‐intervention comparison.

2. Median improvement of quality of care across all comparisons and according to the presence of co‐interventions.

	Median improvement (interquartile range)
	Using primary (or median) outcome	Using largest outcome	Using smallest outcome
All (n = 40)	6.8% (3.8% to 17.5%)	12.0% (6.1% to 20.2%)	4.0% (0.5% to 11.3%)
Reminders alone (n = 27)	11.0% (5.4% to 20.0%)	12.3% (7.0% to 33.5%)	6.1% (1.2% to 12.9%)
Reminders with co‐intervention(s) (n = 13)	4.0% (3.0% to 6.0%)	9.8% (3.9% to 12.5%)	0.7% (‐1.9% to 3.6%)

Open in a new tab

We also re‐analyzed the impact of reminder and study features on effect size using the largest and smallest outcome for the representative outcome for each study. None of these analyses yielded substantially different findings compared with the findings using the primary (or median) outcome. The direction of the impact of the reminder and study features remained the same.

Patient outcomes

Six studies reported patient outcomes (see Table 6), but we were unable to pool them because of heterogeneity: they measured different clinical outcomes in different populations. In these studies, reminders had no measurable effect on i) blood pressure, glycated hemoglobin and cholesterol levels, ii) reaching blood pressure, glycated hemoglobin and cholesterol targets, and iii) mortality.

3. Improvement of patient outcomes, by study.

Study ID	Patient outcome: percentage difference between groups at follow‐up	Patient outcome: mean difference between groups at follow‐up
Barnett 1983	Percentage of patients with BP<100 or receiving treatment at 12 mo: 18.1%
Gilutz 2009	Event‐free survival: ‐2.1%	LDL level: ‐2.4 mg/dL
Heidenreich 2005	Mortality: hazard ratio: 0.98 (95% CI: 0.78 to 1.23)	Diastolic BP: 0 Systolic BP: 0
McAlister 2009	Mortality: 1%
Rossi 1997		Diastolic BP: ‐4 Systolic BP: 0
Ziemer 2006_1	Percentage of patients with Hba1c<7.0%: OR: 0.98 (95% CI: 0.86 to 1.12) Percentage of patients with systolic BP<130: OR: 1.04 (95% CI: 0.94 to 1.16) Percentage of patients with LDL<100: OR 0.92 (95% CI: 0.79 to 1.08)	Hba1c: 0.1 Systolic BP: ‐1.2 LDL level: 2.5 mg/dL
Ziemer 2006_2	Percentage of patients with Hba1c<7.0%: OR: 0.99 (95% CI: 0.82 to 1.19) Percentage of patients with systolic BP<130: OR: 0.92 (95% CI: 0.79 to 1.06) Percentage of patients with LDL<100: OR 1.05 (95% CI: 0.84 to 1.31)	Hba1c: 0.4 Systolic BP: 0.8 LDL level: 3.0 mg/dL

Open in a new tab

BP: blood pressure, Hba1c: glycated hemoglobin, LDL: low‐density lipoprotein, mo: months

We are thus uncertain whether reminders, alone (single‐component intervention) or in addition to co‐intervention(s) (multi‐component intervention), improve patient outcomes compared with usual care or the co‐intervention(s) without the reminder component as the certainty of the evidence is very low (n = 6 studies (seven comparisons)) (see Table 1).

We are uncertain whether reminders alone improve patient outcomes compared with usual care (n = 4 studies (four comparisons), very low‐certainty evidence) (see Table 2). We are also uncertain whether adding reminders to one or more co‐interventions improve patient outcomes compared with the co‐intervention(s) without the reminder component (n = 2 studies (three comparisons), very low‐certainty evidence) (see Table 3).

Adverse effects

None of the included studies reported outcomes related to harms or adverse effects of the intervention.

Discussion

Summary of main results

Computer‐generated reminders delivered on paper to healthcare professionals, alone or in addition to co‐intervention(s), probably improve slightly quality of care compared with usual care or the co‐intervention(s) without the reminder component (median improvement 6.8% (interquartile range (IQR): 3.8% to 17.5%); 34 studies (40 comparisons); moderate‐certainty evidence) (see Table 1).

Computer‐generated reminders delivered on paper to healthcare professionals alone (single‐component intervention) probably improve quality of care compared with usual care (median improvement 11.0% (IQR 5.4% to 20.0%); 27 studies (27 comparisons); moderate‐certainty evidence) (see Table 2). Adding computer‐generated reminders delivered on paper to healthcare professionals to one or more co‐interventions (multi‐component intervention) probably improve slightly quality of care compared with the co‐intervention(s) without the reminder component (median improvement 4.0% (IQR 3.0% to 6.0%); 11 studies (13 comparisons); moderate‐certainty evidence) (see Table 3).

We are uncertain whether reminders, alone or in addition to co‐intervention(s), improve patient outcomes compared with usual care or the co‐intervention(s) without the reminder component because the certainty of the evidence is very low. None of the included studies reported outcomes related to harms or adverse effects of the intervention, such as redundant testing or overdiagnosis.

As the authors of the on‐screen reminders have suggested (Shojania 2009), the lower improvement rate in multi‐component interventions could be due to the improved quality of care achieved by the other components of the multi‐component intervention, leaving less room for improvement by the reminder. Our analyses support this explanation as post‐intervention compliance rates were higher in the multi‐component intervention control group than the rate in the usual care group. An additional explanation offered by Shojania and colleagues might be that investigators chose to incorporate reminders in multi‐component interventions when attempting to change more complex (and therefore difficult to change) behaviors than those addressed by reminders alone.

Three reminder features were associated with larger effect sizes: providing space for the provider to enter a response, providing an explanation for the reminder, and providing a reference to an influential source. Providing space for a response is likely to increase physician attention to the reminder. Indeed, in a study that specifically investigated this aspect (Litzelman 1993), requiring physicians to respond to reminders improved their overall compliance with cancer screening test by 8% compared with reminders that did not require a response (P = 0.002). Concerning the second feature, providing an explanation for the reminder may allow physicians to understand why they received a reminder and encourage them to respond to the reminder and not discard it. The third feature, providing a reference to an influential source, may further legitimate the reminder and convince the physician to follow the reminder.

Most studies focused on quality of care outcomes. Patient outcomes were most often secondary or non targeted outcomes in the individual studies. Only two of the 35 studies used an endpoint patient outcome (mortality, event‐free survival), whereas four others used an intermediary patient outcome (blood pressure, laboratory tests). Although improving patient outcomes is the ultimate objective of any quality improvement intervention, showing or not showing an impact on patient outcome does not mean the intervention is not effective. The degree to which provider behavior changes ultimately improve patient outcomes will vary according to the strength of the relationship between the targeted process of interest and patient‐level outcomes.

Overall completeness and applicability of evidence

The studies included in this review were conducted over the last 35 years. While three studies were executed as early as in the 1970s, when computerized medical record systems started to be implemented in hospitals (McDonald 1976a; McDonald 1976b; Morgan 1978), most studies were performed in the 1980s and the 1990s, when computerized databases became more widespread. Within the last 10 years, studies on computer‐generated reminders delivered on paper continued to be conducted, parallel to studies on onscreen reminders. A review showed that reminders including a paper‐based component remained the most frequent implementation strategy and had a similar average effect as computerized reminders (Dexheimer 2008). The paper record appears to remain an important source of information and documentation instrument in both hospitals and clinics. Paper‐based solutions may indeed be easier to integrate with the clinical workflow, as compared with designing a completely computerized reminder that relies on the providers' workstation use. This same review showed however that studies on the impact of computer‐generated reminders delivered on paper have tended to decrease, while computerized reminders have increased (Dexheimer 2008). This increase in computerized reminder strategies may suggest that clinical information systems are increasingly being adopted, providing the infrastructure to implement computerized reminders.

The effectiveness of reminders in improving patient outcomes could not be estimated because the few studies that reported outcomes of care, reported data too heterogenous to combine. Most studies measured whether reminders improved quality of care, i.e. intermediate outcomes, rather than improved patient outcomes, the ultimate goal of any quality improvement intervention.

Using the median effect across studies as effect size limits the interpretation of the results (precision of study effect size not taken into account). However, conventional methods of meta‐analysis were not appropriate in our context, due to study heterogeneity and clustering effects that could not be taken into account in many studies failing to report the intra‐class correlation. The median effect approach is increasingly used in Cochrane reviews (e.g. Farmer 2008; O'Brien 2007; Shojania 2009). This method allows for reporting on the range of effects associated with the intervention under study and allows for analysis of potential effect modifiers. Although we performed bivariate analyses, we were not able to perform multivariate analyses due to missing data on intervention details related to effect modifiers in many studies.

Studies included in this review were mostly carried out in outpatient settings, for preventive and chronic or acute care. While studies were exclusively based in the USA and Canada in the first publication of the review, the three studies added in this update were all based outside of North America (France, Israel, Kenya). Most studies were based in university hospitals, university‐affiliated clinics or large healthcare organizations (e.g. Veterans Health Administration, private Health Maintenance Orgnizations (HMO)), which often developed their own electronic medical record system and embedded reminder system (e.g. Computer‐Stored Ambulatory Record (COSTAR) developed at Massachusetts General Hospital for Harvard Community Health Plan, Regenstrief Medical Records System developed at the Indiana University School of Medicine). This limits the applicability of the evidence to other types of healthcare delivery systems. It appears that the improvement of quality of care achieved with reminders is more modest in studies targeting disadvantaged populations compared with studies targeting general populations. Also, the quality of care provided to disadvantaged populations before the reminder intervention was lower than the quality of care provided to the general population, among our included studies. Reminder interventions do not appear as beneficial in context of disadvantaged groups where access to care and out‐of‐pocket expenses, for instance, might be a stronger barrier to better quality of care than healthcare providers' behavior.

Certainty of the evidence

Overall, the certainty of the evidence regarding the effectiveness of computer‐generated reminders delivered on paper to improve quality of care was rated as moderate according to the GRADE approach (see Table 1; Table 2; Table 3). We downgraded the level of certainty of the evidence from high to moderate because of methodological limitations of the included randomized trials. The quality of the studies was fairly low. One reason for the low quality of studies was that reporting of earlier studies was very poor, thus making it difficult to assess whether appropriate measures were taken to reduce bias. Actually, this may be the reason for the observed decreasing time trend in the size of the effect. In addition, as few studies reported baseline adherence rates, marginal improvement of reminders could not be correctly evaluated. The majority of studies did not implement nor report allocation concealment and appropriate analyses according to unit of allocation. Authors of approximately one half of studies (14) did report follow‐up features allowing us to conclude that the risk for a potential attrition bias was low. We did not find other serious limitations in the other factors (indirectness of evidence, inconsistency of results, and imprecision of results).

The certainty of the evidence regarding the effectiveness of computer‐generated reminders delivered on paper to improve patient outcomes was rated as very low according to the GRADE approach (see Table 1; Table 2; Table 3). We downgraded the level of certainty of the evidence from high to very low because of methodological limitations in the included randomized trials, wide variation of results (inconsistency) and wide confidence intervals (imprecision).

We can not rule out the presence of publication bias: smaller studies (in terms of included patients and healthcare professionals) showed greater effect size than larger studies (figures available on request). It is thus possible that smaller studies with negative or non‐significant results were not published, further reducing the true overall effect size.

Potential biases in the review process

We conducted extensive literature searches including multiple databases with overlapping periods to avoid publication bias. Two people, working independently, screened all search results to minimize the risk of missing a relevant study. Study selection was discussed thoroughly to check whether the reminder system met the inclusion criteria. Using the median effect size as analytic approach allowed us to avoid the unit of analysis issues in unadjusted cluster trials, but it limits the interpretability of the results as there are no confidence intervals of individual effect sizes indicating the degree of imprecision of the results.

Agreements and disagreements with other studies or reviews

There are a number of previous reviews on the effectiveness of reminders alone (e.g. Balas 2000; Baron 2010) or reminders as part of a multi‐component intervention (e.g. Garg 2005; Grimshaw 2004), focusing on various types of reminders (e.g. computerized, manual). Compared with the median improvement in quality of care of 4.2% reported in the Cochrane systematic review of on‐screen point‐of‐care computer reminders including 28 studies (Shojania 2009), the median improvement of 6.8% we observed in our review was only slightly higher. In a review comparing the effect of various intervention techniques for prompting physicians, the average rate difference for adherence to recommended preventive care strategies using computer‐ generated reminders did not differ from non‐computerized prompting approaches (Balas 2000). We also observed that the largest improvement in quality of care was seen in vaccination rates, with a median improvement of 13.1%. Similar rates were found in other systematic reviews focusing on vaccination and preventive care performance (Balas 2000; Baron 2010; Dexheimer 2008).

Regarding effect modifiers, unlike our review where we observed differences in effect according to whether the reminder provided space for the provider to enter a response, provided an explanation for the reminder, and provided a reference to an influential source, other systematic reviews found no specific reminder or contextual features significantly associated with effect magnitude (Baron 2010; Shojania 2009).

Authors' conclusions

Implications for practice.

The findings of this review provide moderate evidence to support the use of computer‐generated reminders delivered on paper to healthcare professionals to improve quality of care. Providing space on the reminder for a response from the clinician, providing an explanation of the reminder’s content or advice, and providing a reference to an influential source were associated with larger effect sizes. The heterogeneity of the reminder interventions included in this review also suggests that reminders can be implemented in various settings for various health conditions.

Implications for research.

To improve the quality of the rather mixed evidence base in this important area and address further policy‐relevant questions, we suggest that researchers and research funders consider the following suggestions for future research on reminders.

Better reporting of methods (for randomization, allocation concealment, etc.) in compliance with existing reporting standards, such as the CONSORT checklist (Moher 2010).
Better reporting of quality improvement interventions (Hoffmann 2014; Ogrinc 2016) to allow better classification and comparisons of reminder features.
Develop and apply better approaches, definitions, analyses and reporting of complex interventions (Craig 2008; Mohler 2015).
Report quality of care and patient outcomes at baseline and at follow‐up.
Consider the probability, nature and process of contamination before designing the study. Cluster‐randomization may or may not be appropriate and should not be uncritically assumed always to be a solution as it holds statistical disadvantage (larger sample size required) (Keogh‐Brown 2007).
If using a cluster design, use rigorous statistical methods and report all relevant data (Campbell 2007).
Use blind assessment of outcomes to reduce assessment bias.
Investigate the effectiveness of reminders in various healthcare delivery systems, outside North America and university‐affiliated hospitals.
Investigate the learning effect of reminders (after the end of the intervention).

What's new

Date	Event	Description
16 August 2016	New citation required but conclusions have not changed	Overall conclusions have not changed.
16 August 2016	New search has been performed	We searched for evidence to 21 September 2016 and included three new studies. The review now has 35 studies. We have updated the methods and other sections of the review in line with current Cochrane and EPOC guidance

Open in a new tab

History

Protocol first published: Issue 3, 1998  Review first published: Issue 12, 2012

Date	Event	Description
18 July 2012	Amended	Review amended following first peer‐review process.
26 May 2008	Amended	Review stage
25 May 2008	Amended	Converted to new review format.

Open in a new tab

Acknowledgements

For this update, we would like to acknowledge the contribution of the following individuals who helped with data selection and extraction: Julia Worswick (JW), Sebastien Bacher (SB), and Sharlini Yogasingam (SY).

We would also like to acknowledge the support of the National Institute for Health Research (NIHR), via Cochrane Infrastructure funding to the Effective Practice and Organisation of Care (EPOC) Group, and the Ottawa Hospital Research Institute (OHRI) supporting JW and SY. The views and opinions expressed therein are those of the authors and do not necessarily reflect those of the Systematic Reviews Programme, NIHR, National Health Service (NHS), or the Department of Health.

We were unable to contact Jeremy Wyatt (co‐author of the original version of the review) for this update. We acknowledge here his contribution to the original version of the review.

Appendices

Appendix 1. Update search strategies

MEDLINE (OVID)

Epub Ahead of Print, In‐Process & Other Non‐Indexed Citations, Ovid MEDLINE(R) Daily and Ovid MEDLINE(R) 1946 to Present

No.	Search terms	Results
1	reminder systems/ or computer*.mp. or decision support systems, clinical/ or physician's practice patterns/ or "forms and records control"/ or exp hospital information systems/ or exp artificial intelligence/ or exp decision making, computer‐assisted/ or "appointments and schedules"/ or exp medical records/ or exp clinical protocols/	1036105
2	(remind* or sticker* or (flowsheet* or flow sheet* or flowchart* or flow chart) or (order adj1 (form or sheet)) or ((request or encounter) adj1 form) or checklist* or ((tag or tagged or sticker or annot) adj5 (note or record* or sheet* or chart* or form)) or (computer adj3 feedback) or (information adj3 feedback) or prompt*).tw.	155205
3	(randomized controlled trial or controlled clinical trial).pt. or randomi#ed.ab. or placebo.ab. or clinical trials as topic.sh. or randomly.ab. or trial.ti.	1097542
4	exp animals/ not humans.sh.	4319541
5	3 not 4	1013244
6	intervention?.ti. or (intervention? adj6 (clinician? or collaborat* or community or complex or design* or doctor? or educational or family doctor? or family physician? or family practitioner? or financial or gp or general practice? or hospital? or impact? or improv* or individuali?e? or individuali?ing or interdisciplin* or multicomponent or multi‐component or multidisciplin* or multi‐disciplin* or multifacet* or multi‐facet* or multimodal* or multi‐modal* or personali?e? or personali?ing or pharmacies or pharmacist? or pharmacy or physician? or practitioner? or prescrib* or prescription? or primary care or professional* or provider? or regulatory or regulatory or tailor* or target* or team* or usual care)).ab.	220471
7	(pre‐intervention? or preintervention? or "pre intervention?" or post‐intervention? or postintervention? or "post intervention?").ti,ab.	15479
8	(hospital* or patient?).hw. and (study or studies or care or health* or practitioner? or provider? or physician? or nurse? or nursing or doctor?).ti,hw.	814580
9	demonstration project?.ti,ab.	2226
10	(pre‐post or "pre test" or pretest or posttest* or "post test*" or (pre adj5 post)).ti,ab.	87464
11	(pre‐workshop or post‐workshop or (before adj3 workshop) or (after adj3 workshop)).ti,ab.	825
12	trial.ti. or ((study adj3 aim?) or "our study").ab.	848755
13	(before adj10 (after or during)).ti,ab.	417939
14	("quasi‐experiment" or quasiexperiment or "quasi random" or quasirandom or "quasi control" or quasicontrol or ((quasi* or experimental) adj3 (method* or study or trial or design*))).ti,ab,hw.	124707
15	("time series" adj2 interrupt*).ti,ab,hw.	1751
16	(time points adj3 (over or multiple or three or four or five or six or seven or eight or nine or ten or eleven or twelve or month* or hour? or day? or "more than")).ab.	12834
17	pilot.ti.	52038
18	pilot projects/	97260
19	(clinical trial or controlled clinical trial or multicenter study).pt.	684859
20	(multicentre or multicenter or multi‐centre or multi‐center).ti.	37868
21	random*.ti,ab. or controlled.ti.	940170
22	(control adj3 (area or cohort? or compare? or condition or design or group? or intervention? or participant? or study)).ab. not (controlled clinical trial or randomized controlled trial).pt.	510099
23	(control year? or experimental year? or (control period? or experimental period?)).ti,ab.	15070
24	evaluation studies as topic/ or prospective studies/ or retrospective studies/	1135413
25	(utili?ation or programme or programmes).ti.	64750
26	(during adj5 period).ti,ab.	347377
27	((strategy or strategies) adj2 (improv* or education*)).ti,ab.	25348
28	(purpose adj3 study).ab.	281494
29	"comment on".cm. or review.pt. or (review not "peer review*").ti. or randomized controlled trial.pt.	3417623
30	(rat or rats or cow or cows or chicken? or horse or horses or mice or mouse or bovine or animal?).ti,hw. or veterinar*.ti,ab,hw.	6113916
31	exp animals/ not humans.sh.	4319541
32	(or/6‐28) not (or/29‐31)	3429439
33	1 and 2 and (5 or 32)	6104

Open in a new tab

Embase (OVID)

Embase 1974 to 2016 September 21

No.	Search terms	Results
1	reminder system/ or remind.mp. or computer.mp. or decision support system/ or decision support.tw. or clinical practice/ or medical order/ or (order adj2 entry).tw. or hospital information system/ or information system?.tw. or artificial intelligence/ or artificial intelligence.tw. or patient scheduling/ or (patient? adj2 (schedul or appointment?)).tw. or medical record/ or medical record?.tw. or clinical protocol/	1628912
2	remind.mp. or sticker.tw. or (flowsheet* or flow sheet* or flowchart* or flow chart).tw. or (order adj1 (form or sheet)).tw. or ((request or encounter) adj1 form).tw. or checklist.tw. or ((tag or tagged or sticker or annot) adj5 (note* or record* or sheet* or chart* or form)).tw. or (computer adj3 feedback).tw. or (information adj3 feedback).tw. or prompt*.tw.	205015
3	1 and 2	41915
4	((education* adj2 (program* or intervention? or meeting? or session? or strateg* or workshop? or visit?)) or (behavio?r* adj2 intervention?) or (leaflet? or booklet? or poster or posters) or ((written or printed or oral) adj information) or (information* adj2 campaign) or (education* adj1 (method? or material?)) or outreach or ((opinion or education* or influential) adj1 leader?) or facilitator? or academic detailing or consensus conference? or practice guideline? or (guideline? adj2 (introduc* or issu* or impact or effect? or disseminat* or distribut)) or ((introduc or impact or effect? or implement* or computer* or compli) adj2 protocol?) or ((introduc or impact or effect? or implement* or computer* or compli) adj2 algorithm?) or clinical pathway? or critical pathway? or ((effect? or impact or evaluat or introduc* or compar) adj2 training program) or reminder? or (recall adj2 system) or (prompter? or prompting) or advance directive?).tw. or feedback/ or feedback.tw. or chart review.tw. or ((effect? or impact or records or chart?) adj2 audit).tw. or compliance.tw. or marketing.tw. or ((cost or clinical or medical) adj information).tw. or medical education/ or medical audit/ or continuing education/ or *postgraduate education/	711613
5	(fee for service or cost shar* or (copayment? or co payment?) or (prepay* or prepaid or prospective payment?) or formular? or fundhold? or blue cross or voucher? or (free adj2 care)).tw. or exp health insurance/ or health care costs/ or health care financing/ or medical fee/ or *prospective payment/	137932
6	((nurse adj (rehabilitator? or clinician? or practitioner? or midwi)) or ((nurse or midwi or practitioner) adj managed) or clinical pharmacist? or paramedic?).tw. or exp paramedical personnel/ or general practitioner/ or physician/ or (team? adj2 (care or treatment or assessment or consultation)).tw. or (integrat adj2 (care or service?)).tw. or (care adj2 (coordinat* or program* or continuity)).tw. or (case adj1 management).tw. or patient care/ or (chang adj1 location?).tw. or domiciliary.tw. or (home adj1 (treat* or visit?)).tw. or day surgery.tw. or exp primary health care/ or ambulatory surgery/ or nursing home/ or day hospital/ or outpatient care/ or terminal care/ or group practice/ or general practice/ or rural health care/ or community mental health center/ or information system/ or medical record/ or (information adj2 (management or system?)).tw. or peer review/ or professional standards review organization/ or exp clinical practice/ or quality assurance.tw. or exp health care delivery/ or health care quality/ or professional practice/ or (early adj1 discharg).tw. or discharge planning.tw. or offset.tw. or triage.tw. or near patient testing.tw. or patient referral/ or (physician patient adj (interaction? or relationship?)).tw. or managed care.tw. or health care organization/ or health maintenance organization/ or health care system/ or health care access/ or (hospital? adj1 merg).tw. or (computer adj2 (dosage or dosing or diagnosis therapy or decision?)).tw. or (computer* adj2 (diagnosis or therapy)).tw. or gatekeep*.tw.	1445240
7	(((standard or usual or routine or regular or traditional or conventional or pattern) adj2 care) or (program* adj2 (reduc* or increas* or decreas* or chang* or improv* or modify* or monitor* or care)) or (program* adj1 (health or care or intervention?)) or ((effect or impact or introduc) adj2 (legislation or regulations or policy)) or ((effect? or impact or evaluat or introduc* or compar) adj2 treatment program) or ((effect? or impact or evaluat* or introduc* or compar) adj2 care program) or ((effect? or impact or evaluat* or introduc* or compar) adj2 screening program) or ((effect? or impact or evaluat* or introduc* or compar) adj2 prevent program*)).tw.	172495
8	4 or 5 or 6 or 7	2227244
9	(computer* or electronic* or print or printed or print‐out* or paper or paper‐based).ti,ab.	1157926
10	randomized controlled trial/ or (randomised or randomized).tw. or experiment.tw. or (time adj series).tw. or (pre test or pretest or posttest or post test).tw. or impact.tw. or intervention?.tw. or chang.tw. or evaluat.tw. or effect?.tw. or compar.tw. or (controlled adj study).tw.	13542751
11	(random sampl* or random digit* or random effect* or random survey or random regression).ti,ab. not randomized controlled trial/ [per bmj clinical evidence filter]	74330
12	(exp animals/ or exp invertebrate/ or animal experiment/ or animal model/ or animal tissue/ or animal cell/ or nonhuman/) and (human/ or normal human/ or human cell/)	17977645
13	(exp animals/ or exp invertebrate/ or animal experiment/ or animal model/ or animal tissue/ or animal cell/ or nonhuman/) not 12	5840407
14	10 not (or/11,13)	10163810
15	3 and 8 and 9 and 14	4430

Open in a new tab

Cochrane Library (Wiley)

No.	Search terms	Results
#1	[mh "reminder systems"]	749
#2	(prompt? near/5 booklet? or prompt? near/5 chart? or prompt? near/5 checklist? or prompt? near/5 check‐list? or prompt? near/5 display? or prompt? near/5 flowchart? or prompt? near/5 flow sheet? or prompt? near/5 flowsheet? or prompt? near/5 form? or prompt? near/5 "hard copy" or prompt? near/5 "hard copies" or prompt? near/5 insert? or prompt? near/5 leaflet$ or prompt? near/5 manual or prompt? near/5 mail$ or prompt? near/5 pamphlet$ or prompt? near/5 paper or prompt? near/5 paper‐based or prompt? near/5 postal? or prompt? near/5 postcard? or prompt? near/5 post‐card? prompt? near/5 poster? or prompt? near/5 print$ or prompt? near/5 sheet? or prompt? near/5 written or prompt? near/5 handwritten):ti,ab,kw	15
#3	(alert? near/3 chart? or alert? near/3 checklist? or alert? near/3 check‐list? or alert? near/3 handwritten or alert? near/3 "hard copy" or alert? near/3 "hard copies" or alert? near/3 insert$ or alert? near/3 leaflet$ or alert? near/3 manual or alert? near/3 mail$ or alert? near/3 pamphlet$ or alert? near/3 paper or alert? near/3 paper‐based or alert? near/3 postal? or alert? near/3 postcard? or alert? near/3 post‐card? or alert? near/3 poster? or alert? near/3 print$ or alert? near/3 sheet? or alert? near/3 written):ti,ab,kw	12
#4	(chart* near/2 insert* or medical record* near/2 insert*):ti,ab,kw	6
#5	(chart* near/4 stamp* or record* near/4 stamp* or chart* near/4 sticker* or record* near/4 sticker*):ti,ab,kw	19
#6	(alert* near/4 stamp* or alert* near/4 sticker* or prompt* near/4 stamp* or prompt* near/4 sticker*):ti,ab,kw	3
#7	(alert* near/4 stamp* or alert* near/4 sticker* or prompt* near/4 stamp* or prompt* near/4 sticker*):ti,ab,kw	3
#8	(prompt* near/3 record* or prompt* near/3 chart* or prompt* near/3 progress note* or alert* near/3 record* or alert* near/3 chart* or alert* near/3 progress note*):ti,ab,kw	65
#9	(prompt* next physician* or prompt* next provider* or prompt* next practitioner* or alert* next physician* or alert* next provider* or alert* next practitioner*):ti,ab,kw	54
#10	(alert* near/5 patient* profile* or alert* near/5 cue sheet* or alert* near/5 check list* or alert* near/5 checklist* or alert* near/5 patient‐specific or alert* near/5 gener* information or prompt* near/5 patient* profile* or prompt* near/5 cue sheet* or prompt* near/5 check list* or prompt* near/5 checklist* or prompt* near/5 patient‐specific or prompt* near/5 gener* information or sticker* near/5 patient* profile* or sticker* near/5 cue sheet* or sticker* near/5 check list* or sticker* near/5 checklist* or sticker* near/5 patient‐specific or sticker* near/5 gener* information or stamp* near/5 patient* profile* or stamp* near/5 cue sheet* or stamp* near/5 check list* or stamp* near/5 checklist* or stamp* near/5 patient‐specific or stamp* near/5 gener* information):ti,ab,kw	84
#11	(remind*):ti,ab,kw	2466
#12	{or #2‐#10}	239
#13	(computer* or electronic* or print or printed or print‐out* or paper or paper‐based):ti,ab,kw	45815
#14	[mh "medical records systems, computerized"]	521
#15	[mh computers]	1194
#16	{or #13‐#15}	45974
#17	(#1 or #12) and #16	360
#18	(#11 and #13)	716
#19	#17 or #18	775

Open in a new tab

CINAHL (Ebsco)

No.	Search terms	Results
S1	(MH "Reminder Systems")	1,483
S2	MH Decision Support Systems, Clinical	1,759
S3	(MH "Hospital Information Systems")	1,908
S4	(MH "Artificial Intelligence+")	5,003
S5	(MH "Decision Making, Computer Assisted+")	11,706
S6	(MH "appointments and schedules")	4,000
S7	(MH "Medical Records+")	61,477
S8	AB ( computer* OR clinical protocol* ) or TI ( computer* OR clinical protocol* ) or MW ( computer* OR clinical protocol* )	82,586
S9	TI ( form* N3 control OR record* N3 control ) or AB ( form* N3 control OR record* N3 control )	1,580
S10	TI ( clinical n3 practice OR community n3 practice OR primary n2 care or specialty n3 practice* or physician* n3 practice* ) or AB ( clinical n3 practice OR community n3 practice OR primary n2 care or specialty n3 practice* or physician* n3 practice* )	77,426
S11	MW clinical n3 practice OR community n3 practice OR primary n2 care or specialty n3 practice* or physician* n3 practice*	60,690
S12	TI ( remind* OR sticker* OR flowsheet* or flow sheet* or flowchart* or flow chart* OR checklist* OR prompt* or order form* or order sheet* OR request form* or encounter form* OR tag* n5 note* OR tag* n5 record* or tag* n5 sheet* or tag* n5 chart* or tag* n5 form* OR sticker* n5 note* OR sticker* n5 record* or sticker* n5 sheet* or sticker* n5 chart* or sticker* n5 form* OR annot* n5 note* OR annot* n5 record* or annot* n5 sheet* or annot* n5 chart* or annot* n5 form* ) or AB ( remind* OR sticker ...	7,134
S13	TI ( computer* n3 feedback OR information n3 feedback ) or AB ( computer* n3 feedback OR information n3 feedback )	339
S14	(MH "Clinical Trials")	85,055
S15	(MH "random assignment")	33,773
S16	TI ( experiment* or time series or pre‐test or pretest or post‐test or posttest or Comparative Stud* ) or AB ( experiment* or time series or pre‐test or pretest or post‐test or posttest or Comparative Stud* )	61,170
S17	(MH "Evaluation Research")	22,078
S18	(MH "Experimental Studies") or (MH "Intervention Trials") or (MH "Triple‐Blind Studies") or (MH "Double‐Blind Studies") or (MH "Single‐Blind Studies") or (MH "Solomon Four‐Group Design") or (MH "Pretest‐Posttest Design+")	71,973
S19	S14 or S15 or S16 or S17 or S18	224,649
S20	S1 or S2 or S3 or S4 or S5 or S6 or S7 or S8 or S9 or S10 or S11	232,613
S21	S12 or S13	7,469
S22	S19 AND S20 AND S21	438
S23	(MM "Reminder Systems")	770
S24	S19 and S23	213
S25	S22 or S24	512
S26	S25	102

Open in a new tab

INSPEC (Web of Science)

1969‐present

No.	Search terms	Results
# 1	TS=((reminder) AND (print OR paper*))	896
# 2	#1 Indexes=Inspec Timespan=2015‐2016	132
# 3	TI=(computer* OR patient* OR physician* OR care OR clinic* OR practice* OR practitioner* OR trial* OR random* OR study OR studies)	1,057,519
# 4	#3 AND #2	16

Open in a new tab

Characteristics of studies

Characteristics of included studies [ordered by study ID]

Barnett 1983.

Methods	Randomized trial, unit of allocation: patient
Participants	Patients: blood pressure ≥ 100, "poor" follow‐up, women: 49%, mean age: 43, n = 115 Professionals: physicians, nurses, n = not clear Setting: outpatient (Harvard Community Health Plan (HMO), Boston USA)
Interventions	Control: usual care Intervention: physician reminder Duration of intervention: 20 months
Outcomes	Quality of care: rate of repeat blood pressure recorded at 12 months; rate of follow‐up attempted or achieved at 12 months (median) Patient outcomes: percentage patients with blood pressure < 100 or on treatment
Clinical area and targeted activity	Hypertension (general management)
Reminder	Description: If there were not 2 repeat visits that included blood pressure measurement within 6 months after the initial recording of the elevated blood pressure, the COSTAR system was programmed to automatically generate a reminder notice to the patient's primary physician. The physician was notified of the deviation from the standard, and was given an encounter form on which he or she recorded when the next follow‐up visit should occur. Typology: patient‐specific: YES, space for response: YES, explicit advice: NO; explanation: NO; reference: NO; at point‐of‐care: NO
Notes
*Risk of bias*
Bias	Authors' judgement	Support for judgement
Random sequence generation (selection bias)	Unclear risk	"randomly assigned to a control or to an experimental group"
Allocation concealment (selection bias)	Unclear risk	not described
Blinding of outcome assessment (detection bias)   All outcomes	Unclear risk	not described
Incomplete outcome data (attrition bias)   All outcomes	Low risk	8 dropouts in intervention group and 10 in control group
Selective reporting (reporting bias)	Unclear risk	not clear
Other bias	Unclear risk	not clear
Outcomes at baseline similar?	Unclear risk	not reported
Characteristics at baseline similar?	Low risk	groups stratified by age and diastolic blood pressure level before randomization
Adequate protection against contamination?	High risk	physicians treated both intervention and control patients

Open in a new tab

Becker 1989.

Methods	Randomized trials, unit of allocation: patient
Participants	Patients: 40‐60 years old; women: 68%, mean age: 51, n = ˜700 randomized (395 analyzed in included study groups) Professionals: physicians, n = not clear Setting: outpatient (University of Virginia internal medicine clinic, Charlottesville, USA)
Interventions	Control: usual care Intervention: physician reminder Duration of intervention: nine months
Outcomes	Quality of care: overall compliance rate with preventive care guidelines (primary); compliance with: dental exam, ocular pressure check, fecal occult blood test, influenza vaccination, pneumococcal vaccination, tetanus toxoid, mammography, pap smear
Clinical area and targeted activity	Preventive care (multiple: test ordering, vaccination)
Reminder	Description: A standardized telephone questionnaire and computer program were used to create an individualized schedule for preventive care needs. The reminder specified for the patient which services were necessary and when they should be obtained. Physician received the reminder as a memorandum appended to each patient's chart at the first visit after the telephone interview. If there was no scheduled visit, the reminder was mailed to the patient's primary physician. Typology: patient‐specific: YES; space for response: NO; explicit advice: YES; explanation: NOT CLEAR ; reference: NO; at point‐of‐care: YES
Notes	Additional study intervention excluded from analyses: patient reminder
*Risk of bias*
Bias	Authors' judgement	Support for judgement
Random sequence generation (selection bias)	Unclear risk	"eligible patients were randomly assigned to three study groups"
Allocation concealment (selection bias)	Unclear risk	not described
Blinding of outcome assessment (detection bias)   All outcomes	Unclear risk	data gathered through outpatient medical record review, but no mention of blinding
Incomplete outcome data (attrition bias)   All outcomes	Low risk	authors gave reasons for data excluded from analysis and compared a random sample of excluded patients with included patients
Selective reporting (reporting bias)	Unclear risk	not clear
Other bias	Unclear risk	not clear
Outcomes at baseline similar?	Unclear risk	not reported
Characteristics at baseline similar?	Low risk	"The experimental groups were similar in all characteristics."
Adequate protection against contamination?	High risk	patients randomized

Open in a new tab

Binstock 1997.

Methods	Randomized trial, unit of allocation: patient
Participants	Patients: women 25‐49 years old without pap smear in previous three years, n = 3052 Professionals: physicians, n = not clear Setting: outpatient (3 medical centers of a Kaiser Permanente HMO, USA)
Interventions	Control: usual care Intervention: physician reminder Duration of intervention: not clear
Outcomes	Quality of care: pap smear rate within 12 months if due (primary) Resource use/financial: total estimated costs; estimated costs per pap smear obtained
Clinical area and targeted activity	Preventive care/cervical cancer (test ordering)
Reminder	Description: A chart reminder was affixed to the outside of the patient's medical record. Computerized laboratory files were used to identify women without a pap smear in the previous 3 years. Typology: patient‐specific: YES; space for response: NOT CLEAR; explicit advice: NOT CLEAR; explanation: NOT CLEAR; reference: NOT CLEAR; at point‐of‐care: YES
Notes
*Risk of bias*
Bias	Authors' judgement	Support for judgement
Random sequence generation (selection bias)	Unclear risk	"randomly assigned"
Allocation concealment (selection bias)	Unclear risk	not described
Blinding of outcome assessment (detection bias)   All outcomes	Low risk	outcome data from computerized laboratory records
Incomplete outcome data (attrition bias)   All outcomes	Unclear risk	not clear
Selective reporting (reporting bias)	Unclear risk	not clear
Other bias	Unclear risk	not clear
Outcomes at baseline similar?	Unclear risk	not reported
Characteristics at baseline similar?	Unclear risk	not reported
Adequate protection against contamination?	High risk	patients randomized

Open in a new tab

Burack 1996.

Methods	Randomized trial, unit of allocation: patient
Participants	Patients: women, ≥ 40 years old, mean age: not clear, n = 2368 women randomized (1527 analyzed in included study groups) Professionals: 20 physicians (2 family medicine physicians, 9 internal medicine physicians, 9 gynecologists) Setting: outpatient (2 sites of a large HMO, Detroit, USA)
Interventions	Control (comparison 1): usual care Intervention (comparison 1): physician reminder Active control (comparison 2): patient reminder Intervention (comparison 2): physician reminder; patient reminder Duration of intervention: 12 months
Outcomes	Quality of care: mammography rate if due (primary)
Clinical area and targeted activity	Preventive care/breast cancer (test ordering)
Reminder	Description: Electronic HMO administrative records and previous mammograms were employed to assess eligibility. The reminder forms were generated off‐site and placed in medical records by the research team. The physician reminder was a brightly colored single page notice placed in the medical chart 1 month before the due date. In addition, the reminder displayed information concerning previous mammograms and allowed the physician to recommend an alternative due date. The reminder was removed from the chart once documentation of completed mammography was obtained. Typology: patient‐specific: YES; space for response: YES; explicit advice: YES; explanation: NO; reference: NO; at point‐of‐care: YES
Notes	Data extracted from graphics. Author not contacted because publication date > 10 years.
*Risk of bias*
Bias	Authors' judgement	Support for judgement
Random sequence generation (selection bias)	Unclear risk	"randomly assigned"
Allocation concealment (selection bias)	Unclear risk	not described
Blinding of outcome assessment (detection bias)   All outcomes	Low risk	electronic administrative records were used
Incomplete outcome data (attrition bias)   All outcomes	Low risk	2368 women randomized ‐ 741 with no visits = 1627 analyzed (68.7%)
Selective reporting (reporting bias)	Unclear risk	not clear
Other bias	Unclear risk	not clear
Outcomes at baseline similar?	Unclear risk	not reported
Characteristics at baseline similar?	Low risk	"There were no significant differences among characteristics of the intervention groups"
Adequate protection against contamination?	High risk	patients randomized

Open in a new tab

Burack 1998.

Methods	Randomized trial, unit of allocation: patient
Participants	Patients: women, 18‐40 years old, mean age: not clear, n = 5801 women randomized (3848 analyzed in included study groups) Professionals: 20 physicians (2 family medicine physicians, 9 internal medicine physicians, 9 gynecologists) Setting: outpatient (three sites of a large HMO, Detroit, USA)
Interventions	Control (comparison 1): usual care Intervention (comparison 1): physician reminder Active control (comparison 2): patient reminder Intervention (comparison 2): physician reminder; patient reminder Duration of intervention: 14 months
Outcomes	Quality of care: pap smear rate if due (primary)
Clinical area and targeted activity	Preventive care/cervical cancer (test ordering)
Reminder	Description: The reminder forms were computer‐generated off‐site and placed in medical records by the research team. The physician reminder was a brightly colored single page notice that included patient specific pap smear information. The notice was prominently placed at the front of the patient's medical chart 2 months before the due date. The reminder was removed from chart once documentation of completed pap smear was obtained. Typology: patient‐specific: YES; space for response: NO; explicit advice: YES; explanation: NO; reference: NO; at point‐of‐care: YES
Notes	2 of the 3 sites had previously participated in a related trial on reminders.
*Risk of bias*
Bias	Authors' judgement	Support for judgement
Random sequence generation (selection bias)	Unclear risk	"women were first randomly assigned, using a site specific, stratified randomization procedure to physician reminder intervention; women were then randomized to the patient reminder intervention on a weekly basis in groups of 156"
Allocation concealment (selection bias)	Unclear risk	not described
Blinding of outcome assessment (detection bias)   All outcomes	Unclear risk	paper copies of pap smear results received: no indication of blinding.
Incomplete outcome data (attrition bias)   All outcomes	High risk	5801 women randomized but 3848 analyzed (66.3%)
Selective reporting (reporting bias)	Unclear risk	not clear
Other bias	Unclear risk	not clear
Outcomes at baseline similar?	Unclear risk	not reported
Characteristics at baseline similar?	Low risk	characteristics of women were similar and there were no significant differences
Adequate protection against contamination?	High risk	patients randomized

Open in a new tab

Chambers 1989.

Methods	Randomized trial, unit of allocation: patient
Participants	Patients: women, ≥ 40 years old, mean age: 62, n = 1262 Professionals: 30 physicians (12 faculty and 18 residents) (2 providers involved in study excluded from analyses) Setting: outpatient (Family practice center of the department of Family Medicine, University hospital, Philadelphia, USA)
Interventions	Control: usual care Intervention: physician reminder Duration of intervention: 6 months
Outcomes	Quality of care: mammography rate if due (primary); mammography rate
Clinical area and targeted activity	Preventive care/breast cancer (test ordering)
Reminder	Description: The date of the last mammogram ordered and entered into the database was displayed in the comments section of the encounter form for each visit. The information was printed as "last mammogram: date" or, if no mammogram was on record in the database (none since 1984), the notation listed "last mammogram?" The physician ordered the mammogram by writing for one in the test ordered section of the encounter form. Typology: patient‐specific: YES; space for response: YES; explicit advice: NO; explanation: NO; reference: NO; at point‐of‐care: YES
Notes
*Risk of bias*
Bias	Authors' judgement	Support for judgement
Random sequence generation (selection bias)	Low risk	"assigned according to a computer‐generated random number program"
Allocation concealment (selection bias)	Unclear risk	not described
Blinding of outcome assessment (detection bias)   All outcomes	Unclear risk	"data entered by office receptionists"
Incomplete outcome data (attrition bias)   All outcomes	Unclear risk	patients (n = not clear) from the 2 physician investigators excluded from analysis
Selective reporting (reporting bias)	Unclear risk	not clear
Other bias	Unclear risk	not clear
Outcomes at baseline similar?	Low risk	some proportion of women up‐to‐date at baseline
Characteristics at baseline similar?	Low risk	"the subjects in the experimental and control groups were similar in age, race, insurance coverage and complexity of disease".
Adequate protection against contamination?	High risk	physicians likely saw patients with and without reminders

Open in a new tab

Chambers 1991.

Methods	Cluster randomized trial, unit of allocation: health professional (stratified by level of training)
Participants	Patients: ≥ 65 years old or with any of the following clinical diagnoses: diabetes, renal failure, anemia, congestive heart failure, asthma, or chronic obstructive pulmonary disease; women: 74%; mean age: not clear, n = 489 analyzed Professionals: 30 physicians (12 faculty and 18 residents) Setting: outpatient (Family practice center, Thomas Jefferson University, Philadelphia, USA)
Interventions	Control: usual care Intervention: physician reminders (always) Duration of intervention: 2 months
Outcomes	Quality of care: vaccination rate if due (primary)
Clinical area and targeted activity	Preventive care/influenza (vaccination)
Reminder	Description: "Flu vac in 1987?__" appears on the encounter form, computer‐generated for each visit when a patient arrives at the receptionist's desk, which is then attached to the front of the patient chart; it includes space for information regarding tests and procedures ordered. Typology: patient‐specific: NO; space for response: YES; explicit advice: NO; explanation: NO; reference: NO; at point‐of‐care: YES
Notes	Additional study intervention excluded from analyses: physicians reminders (sometimes: printed for half the eligible patients)
*Risk of bias*
Bias	Authors' judgement	Support for judgement
Random sequence generation (selection bias)	Low risk	"Physician randomly assigned via a computerized randomization program"
Allocation concealment (selection bias)	Unclear risk	not described
Blinding of outcome assessment (detection bias)   All outcomes	Unclear risk	not clear
Incomplete outcome data (attrition bias)   All outcomes	Low risk	864 patients randomized ‐ 51 received vaccination before start ‐ 93 saw multiple physicians ‐ 24 drop‐in visits = 686 analyzed (79.4%)
Selective reporting (reporting bias)	Unclear risk	not clear
Other bias	Unclear risk	not clear
Outcomes at baseline similar?	Unclear risk	not reported
Characteristics at baseline similar?	Unclear risk	not reported
Adequate protection against contamination?	Low risk	physician assignment overruled patient assignment: physicians in control group never received a reminder

Open in a new tab

Dexter 1998.

Methods	Cluster randomized trial (2 x 2 factorial design), unit of allocation: half‐day session, n = 30
Participants	Patients: ≥ 75 years old or ≥ 50 with serious underlying disease, women: 66%, mean age: 65, n = 1394 eligible patients (1009 analyzed), n = 1160 patient/physician pairs Professionals: 147 providers (39 faculty, 108 residents) Setting: outpatient (4 practices with 8 half‐day sessions each, Indiana University, USA)
Interventions	Control: usual care Intervention (3 intervention arms combined): physician reminder Duration of intervention: not clear
Outcomes	Quality of care: rate of patients who completed either directives; patient/physician pair who discussed directives
Clinical area and targeted activity	Advance directives for patients at risk for acute deterioration (professional‐patient communication)
Reminder	Description: All physicians routinely received computer‐generated reminders for patients with scheduled visits. They were reminded to give preventive care, note abnormal results, and avoid drug interactions. These reminders appeared at the bottom of computer‐generated printed encounter forms. The advance directive reminders were followed by a choice list (discussed today, next visit, not applicable, patient too ill, patient refuses to discuss, I disagree with advance directives). Typology: patient‐specific: YES; space for response: YES; explicit advice: YES; explanation: NO; reference: NO; at point‐of‐care: YES
Notes	Intervention a: physician reminder for instruction directive Intervention b: physician reminder for proxy directive Intervention c: physician reminder for both
*Risk of bias*
Bias	Authors' judgement	Support for judgement
Random sequence generation (selection bias)	Unclear risk	"we randomly assigned all of the physicians who worked in a particular half‐day session to the same reminder category."
Allocation concealment (selection bias)	Unclear risk	not described
Blinding of outcome assessment (detection bias)   All outcomes	Low risk	research assistants who collected data were blinded at all times to the patient's study groups
Incomplete outcome data (attrition bias)   All outcomes	Low risk	1394 eligible patients ‐ 42 missed by research assistant ‐ 83 low scores ‐ 36 in nursing homes or prison ‐ 29 deaf or spoke no English ‐ 9 completed advanced directive ‐ 5 for other reasons = 1190 patients interviewed and 1042 enrolled ‐ 33 patients cared for by 10 physicians who changed sessions = 1009 analyzed
Selective reporting (reporting bias)	Unclear risk	not clear
Other bias	Unclear risk	not clear
Outcomes at baseline similar?	Unclear risk	not reported
Characteristics at baseline similar?	High risk	similar, except for ethnicity and coronary heart disease and significant difference among groups for "had ever thought about advance directive": higher for control group
Adequate protection against contamination?	Low risk	"16 physicians practiced in more than one session per week: therefore we randomly assigned the sessions in a stepped manner by first allocating the 16 physicians and all of their associated sessions; we then randomly assigned the remaining eight sessions"

Open in a new tab

Gilutz 2009.

Methods	Cluster‐randomized trial, unit of allocation: primary clinics (n = 112)
Participants	Patients: women: 37.5% , mean age: 65.6, n = 7448 Professionals: physicians, nurses, n = 600 Setting: outpatient (112 primary care clinics of a HMO, Israel)
Interventions	Control: usual care Intervention: physician reminder Duration of intervention: 6‐36 months (mean: 21)
Outcomes	Quality of care: rate of appropriate lipoprotein monitoring; rate of initiation or up‐titration of statin therapy; rate of up‐titration in eligible patients Patient outcomes: LDL levels, event‐free survival (only intervention group data)
Clinical area and targeted activity	Cardiovascular disease (multiple: general management, prescription)
Reminder	A written reminder with patient‐tailored recommendations was mailed to the primary care physicians and nurses. The recommendations were based on the previous 6 months data for new patients, and 4 months for patients in periodic follow‐up. The reminder indicated the patient’s risk factors, lipoprotein values, and known dispensed medications. Lipid‐lowering drug treatment was recommended only in patients with LDL > 110 mg/dL and consisted of either statin initiation (simvastatin 20 mg/day), statin up‐titration (doubling the last registered dose), changing to a more potent statin or compliance evaluation. For unresponsive and compliant patients it was recommended that they be referred to a metabolic clinic. Typology: patient‐specific: YES; space for response: NOT CLEAR; explicit advice: YES; explanation: YES; reference: NOT CLEAR; at point‐of‐care: NO
Notes
*Risk of bias*
Bias	Authors' judgement	Support for judgement
Random sequence generation (selection bias)	Unclear risk	Method of randomization not described in paper.
Allocation concealment (selection bias)	Unclear risk	insufficient information
Blinding of outcome assessment (detection bias)   All outcomes	Low risk	administrative data. "data were collected from three routinely used databases"
Incomplete outcome data (attrition bias)   All outcomes	Low risk	The number of attrition and exclusions pre‐randomization were reported, as well as reasons for attrition/exclusions. No exclusions after randomization.
Selective reporting (reporting bias)	High risk	Outcomes defined in the methods are not reported in the results (initiation of statin, LDL levels in middle and low groups, and secondary outcomes (mortality and hospitalizations))
Other bias	Unclear risk	not clear.
Outcomes at baseline similar?	Unclear risk	No baseline measures of outcomes
Characteristics at baseline similar?	High risk	"Patients in the intervention and control groups were similar in most parameters. However, there were significantly more patients with a history of myocardial infarction (33.0% vs. 29.9%, P = 0.004) and percutaneous coronary intervention (26.2% vs. 23.8%, P = 0.019) in the intervention arm."
Adequate protection against contamination?	Low risk	Clinic randomized to avoid contamination.

Open in a new tab

Heidenreich 2005.

Methods	Randomized trial, unit of allocation: patient
Participants	Patients: ventricular ejection fraction < 40%; women: 0.7%; mean age: 67.5, n = 600 [277 analyzed] Professionals: not clear Setting: inpatient and outpatient (VA, Palo Alto, USA)
Interventions	Control: usual care Intervention: physician reminder Duration of intervention: 18 months
Outcomes	Quality of care: ACE inhibitor use rate (primary) Patient outcomes: mean systolic/diastolic blood pressure; mean creatinine level; mortality
Clinical area and targeted activity	Ventricular dysfunction (prescribing)
Reminder	Description: Echocardiography reports included this statement: "Note: patients with ejection fraction < 40% have a survival benefit with ACE inhibitors (goal dose lisinopril or fosinopril 30‐40 mg/day)". Typology: patient‐specific: YES; space for response: NO; explicit advice: YES; explanation: YES; reference: NO; at point‐of‐care: YES
Notes
*Risk of bias*
Bias	Authors' judgement	Support for judgement
Random sequence generation (selection bias)	Low risk	"Patient selection and randomization were computerized and performed in conjunction with the generation of the reminder: patients who met study entry criteria were randomized using a computerized random number generator."
Allocation concealment (selection bias)	Low risk	"Allocation was concealed from all echocardiographers until the reminder appeared on the report."
Blinding of outcome assessment (detection bias)   All outcomes	Unclear risk	"medication use was determined from review of inpatient or outpatient encounters"
Incomplete outcome data (attrition bias)   All outcomes	Low risk	Among the 600 eligible and randomized patients, 201 were excluded from analyses as they were already on ACE‐inhibitor or appropriate alternative at the time of randomization (96 [32.9%] in intervention group and 105 [34.1%] in control group), 46 died within 2 months of the echocardiogram (20 [6.8%] in intervention group and 26 [8.4%] in control group), 71 left the VA healthcare system (36 [12.3%] in intervention group and 35 [11.4%] in control group) , and 5 had an allergy or adverse reaction to ACE‐inhibitors (3 [1%] in intervention group and 2 [0.6%] in control group). In total, 323 [53.8%] were excluded from analyses (155 [53.1%] in intervention group and 168 [54.5%] in control group).
Selective reporting (reporting bias)	Unclear risk	not clear
Other bias	Unclear risk	not clear
Outcomes at baseline similar?	Low risk	no significant difference between groups for ACE inhibitor use
Characteristics at baseline similar?	Low risk	no significant difference except for history of heart failure, but not extreme. Adjusted for in multivariate analyses.
Adequate protection against contamination?	High risk	patients randomized rather than physicians

Open in a new tab

Heidenreich 2007.

Methods	Randomized trial, unit of allocation: patient
Participants	Patients: men: 98%, mean age: 69, n = 1546 [1271 analyzed] Professionals: physicians (n = 45) and nurse practitioners (n = 5), n = 50 Setting: outpatient and inpatient (VA, Palo Alto, USA)
Interventions	Control: usual care Intervention: physician reminder Duration of intervention: 56 months
Outcomes	Quality of care: prescription rate of any beta‐blocker within 9 months (primary); prescription rate of recommended beta‐blockers within 9 months
Clinical area and targeted activity	Ventricular dysfunction (prescribing)
Reminder	Description: In the process of printing the completed echocardiography report, the computer algorithm checked the electronic report and had a reminder attached to the report of eligible patients meeting the pre‐defined criteria. The reminder included the following statement: "Note: Patients with reduced left ventricular ejection fraction have a survival benefit with beta‐blockers (initial dose: carvedilol 3.125 mg BID or metoprolol succinate 12.5 mg BID)". The reminder also recommended cardiology follow‐up if the patient had New York Heart Association class II or IV symptoms. Typology: patient‐specific: YES; space for response: NO; explicit advice: YES; explanation: YES; reference: NO; at point‐of‐care: YES
Notes
*Risk of bias*
Bias	Authors' judgement	Support for judgement
Random sequence generation (selection bias)	Low risk	"Randomization was carried out in conjunction with the formation of the reminder with an electronic database: patients meeting study criteria were randomized with a computerized random number generator. Randomization was performed separately at each site"
Allocation concealment (selection bias)	Low risk	"Allocation was concealed from all echocardiographers until the reminder appeared on the report."
Blinding of outcome assessment (detection bias)   All outcomes	Low risk	"prescription was determined with the VA pharmacy database"
Incomplete outcome data (attrition bias)   All outcomes	Low risk	See Figure 1: Among the 1546 eligible and randomized patients, 89 died within 30 days of the echocardiogram (44 [5.8%] in intervention group and 45 [5.7%] in control group), 180 left the healthcare system (88 [11.7%] in intervention group and 92 [11.6%] in control group) , and 6 had an echocardiography at >1 site (2 [0.3%] in intervention group and 4 [0.5%] in control group). In total, 275 [17.8%] were excluded from analyses (134 [17.7%] in intervention group and 141 [17.8%] in control group). Exclusions did not significantly differ between groups.
Selective reporting (reporting bias)	Unclear risk	not clear
Other bias	Unclear risk	not clear
Outcomes at baseline similar?	Low risk	no significant difference between groups for β‐Blocker use
Characteristics at baseline similar?	Low risk	no significant difference except for percentage of male, but not extreme.
Adequate protection against contamination?	High risk	patients randomized rather than physicians.

Open in a new tab

Heiman 2004.

Methods	Cluster‐randomized trial, unit of allocation: clinic, n = 5
Participants	Patients: ≥ 70 yr or ≥ 50 yr with severe chronic illness, without advanced directives; women: 68%; mean age: 72; n = 719 in included study arms Professionals: 31 providers in included study arms Setting: outpatient (5 clinics in 5 general practices in Boston, USA)
Interventions	Control: usual care Intervention: physician reminder Duration of intervention: 7 months
Outcomes	Quality of care: completion rate of any advanced directives (primary); discussion or completion rate of advanced directives; completion rate of healthcare proxy; completion rate of living will
Clinical area and targeted activity	Advance directives (professional‐patient communication)
Reminder	Description: Reminders were printed at the bottom of patient summary sheets at every patient visit during the study period. It read: "Your patient is x years old with a history of y medical condition and is missing key information about advanced directives". It instructed physicians to enter data about discussion or completion of the living will and healthcare proxy in either the inpatient or the outpatient electronic medical record. Reminders were already in use for screening tests. Typology: patient‐specific: YES; space for response: YES; explicit advice: YES; explanation: NO; reference: NO; at point‐of‐care: YES
Notes	Additional study intervention excluded from analyses: physician reminder; patient reminder; patient educational material
*Risk of bias*
Bias	Authors' judgement	Support for judgement
Random sequence generation (selection bias)	Low risk	"A blinded programmer using a random number generator randomly assigned each group"
Allocation concealment (selection bias)	Low risk	"A blinded programmer using a random number generator randomly assigned each group"
Blinding of outcome assessment (detection bias)   All outcomes	High risk	"The study investigator who assessed outcomes was not blinded".
Incomplete outcome data (attrition bias)   All outcomes	Low risk	Among the 950 randomized patients, 41 were ineligible (17 [3.4%] in intervention group and 24 [5.4%] in control group) and 190 refused to participate (102 [20.2%] in intervention group and 88 [19.7%] in control group). In total, 719 [75.7%] were included in the intention‐to‐treat analyses (385 [76.4%] in intervention group and 334 [74.9%] in control group).
Selective reporting (reporting bias)	Unclear risk	not clear
Other bias	Unclear risk	not clear
Outcomes at baseline similar?	Unclear risk	Not reported
Characteristics at baseline similar?	High risk	significant differences at baseline for age, gender, private insurance, and number of qualifying illnesses
Adequate protection against contamination?	Low risk	physicians randomized

Open in a new tab

Javitt 2005.

Methods	Randomized trial, unit of allocation: patient
Participants	Patients: all members of a managed care plan, 12‐64 years old; women: 58%; mean age: 38, n = 41,870 [35,447 completed study] (n = not clear for included analyses) Professionals: not clear Setting: mixed (managed care plan, USA)
Interventions	Control: usual care Intervention: physician reminder Duration of intervention: 12 months
Outcomes	Quality of care: compliance rate with starting a new drug (primary) (other outcomes not available) Resource use/cost: mean hospital admissions; mean inpatient days; mean length of stay; inpatient charges
Clinical area and targeted activity	General management: increase compliance with evidence‐based practices (prescribing)
Reminder	Description: The sentinel system is designed as a rule‐based artificial intelligence engine combined with an automated message generator that conveys clinical recommendations and supporting literature to treating physicians. Daily data inputs include physician‐generated insurance claims, hospital discharge and outpatient claims, laboratory claims and laboratory test results, and pharmacy claims. Typical issues targeted by the rules engine include the following: a) absence of ACE inhibitor therapy in patients with congestive heart failure and in those who meet the HOPE trial criteria; b) absence of β‐blocker use in patients with myocardial infarction; c) absence of anticoagulation in patients with atrial fibrillation and structural heart disease; d) absence of documented laboratory monitoring in patients taking warfarin sodium, glitazones, and other medications that require specific laboratory tests. The system contains more than 1000 decision matrices that, when triggered, result in the transmission of a communication to the treating physician (on paper). All recommendations make clear that the communication is merely for the physician's consideration and that there may be mitigating circumstances that might render the recommendation inappropriate. Typology: patient‐specific: YES; space for response: NO; explicit advice: YES; explanation: NOT CLEAR; reference: NO; at point‐of‐care: YES
Notes
*Risk of bias*
Bias	Authors' judgement	Support for judgement
Random sequence generation (selection bias)	Low risk	"Patients were assigned to an intervention or a control group, using an individually assigned random number. Assignment occurred on a single date at study entry."
Allocation concealment (selection bias)	Low risk	"Assignment occurred on a single date at study entry. Neither patients nor treating physicians were informed of the allocation, although it is likely that physicians who received communications about specific patients surmised that those patients were part of the intervention group."
Blinding of outcome assessment (detection bias)   All outcomes	Unclear risk	Not specified in article. Likely from a computer database.
Incomplete outcome data (attrition bias)   All outcomes	Low risk	41870 patients randomized ‐ 2408 withdrew before study inception (1206 [5.9%] in control group and 1202 [5.8%] in intervention group) ‐ 4015 withdrew between 6 and 12 months (2088 [10%] in control group and 1927 [9.2%] in intervention group) = 35447 [84.7%] completed study (17635 [84.3%] in control group and 17812 [85.1%] in intervention group)
Selective reporting (reporting bias)	Unclear risk	not clear
Other bias	Unclear risk	not clear
Outcomes at baseline similar?	Unclear risk	not reported
Characteristics at baseline similar?	Low risk	"at baseline, no significant differences were observed in age or sex between the intervention and control group subjects"
Adequate protection against contamination?	High risk	physicians likely care for patients in intervention and control group

Open in a new tab

Le Breton 2016.

Methods	Cluster‐randomized trial, unit of allocation: health professional (n = 144)
Participants	Patients: age 50‐74, registered with a study provider and visited a study provider during study period; women: 55.8%; median age: 60; n = 20788 Professionals: 144 general practitioners Setting: outpatient (individual or group primary care practices, Val‐de‐Marne, France)
Interventions	Control: usual care Intervention: physician reminder Duration of intervention: 17 months
Outcomes	Quality of care: unadjusted patient adherence to colorectal cancer screening (completion of FOBT or exclusion from FOBT for medical reason), adjusted patient adherence (adjusted for clustering)
Clinical area and targeted activity	Preventive care/colorectal cancer (test ordering)
Reminder	The screening centre mailed three reminders to the intervention‐group GPs at 4‐month intervals. The reminders were lists of patients who had not performed a scheduled FOBT. Typology: patient‐specific: YES; space for response: NO; explicit advice: YES; explanation: NO; reference: NO; at point‐of‐care: NO
Notes
*Risk of bias*
Bias	Authors' judgement	Support for judgement
Random sequence generation (selection bias)	Low risk	"GPs were assigned randomly in a 1:1 proportion to the intervention or the control group, in permuted blocks of 2 or 4"
Allocation concealment (selection bias)	Low risk	unit of allocation by professional (GP)
Blinding of outcome assessment (detection bias)   All outcomes	Low risk	"Outcome data were collected for all patients in the ADOC94 database throughout the study period and over the 9‐month period following the last reminder by data abstracters who were blinded to group assignment."
Incomplete outcome data (attrition bias)   All outcomes	Low risk	"We included 20 778 patients who were eligible for CRC screening and had visited a study GP for any reason during the study period". No missing data. See Figure 1 in paper.
Selective reporting (reporting bias)	Low risk	Primary outcome in trial register matches the primary outcome in the published paper.
Other bias	Unclear risk	Low GP participation rate. Only 15% of contacted GPs agreed to participate in the study.
Outcomes at baseline similar?	Low risk	No SS difference at baseline. "CRC screening adherence rates were 21.8% (95% CI 18.7–26.1) in the intervention group and 21.9% (95% CI 18.9–26.5) in the control group."
Characteristics at baseline similar?	High risk	Some SS differences. "Baseline GP characteristics were similar in the two groups, except for numbers of GPs in group practice and charging above‐standard fees, which were higher in the control group. The only significant difference in baseline patient characteristics was a slightly higher proportion of patients from deprived areas in the intervention group."
Adequate protection against contamination?	Low risk	Randomization at the GP level. Low risk of contamination.

Open in a new tab

Lobach 1997.

Methods	Cluster‐randomized trial, unit of allocation: health professional
Participants	Patients: diabetic; n = 497 eligible patients (359 analyzed); encounters: 1265 (884 analyzed) Professionals: 58 primary care providers randomized (20 family physicians, 1 general internist, 2 physician's assistants, 2 nurse practitioners, 33 residents) [30 analyzed] Setting: outpatient (Duke Family Medicine Center, USA)
Interventions	Control: usual care Intervention: physician reminder Duration of intervention: six months
Outcomes	Quality of care: clinician compliance rate overall (number of recommendations completed/total number of recommendations) (primary); clinician compliance rate for 8 recommendations: foot exam every month in patients with diabetic neuropathy or history of lower limb ulcer, annual complete physical exam, glycated hemoglobin every 6 months, annual urine protein determination, annual cholesterol level, annual eye exam, seasonal influenza vaccination, pneumococcal vaccination
Clinical area and targeted activity	Diabetes (multiple: test ordering, vaccination)
Reminder	Description: The Computer‐Assisted Management Protocol generates a set of disease‐specific care recommendations customized to an individual patient, based on data stored in the patient's electronic medical record, that advises the clinician regarding which studies/procedures should be done during the current visit and which studies/procedures are next due in order to assist the clinician with managing the diabetic patient in accordance with a clinical practice guideline. The output is printed on the 1st page of the paper encounter form. Additional flexibility is included for the clinicians to designate that the recommendation was declined by the patient ("D") or never to be done for the patient ("N"). Clinicians can order the appropriate studies/procedures on the encounter form and indicate results of procedures done during the encounter. Typology: patient‐specific: YES; space for response: YES; explicit advice: YES; explanation: YES; reference: NO; at point‐of‐care: YES
Notes
*Risk of bias*
Bias	Authors' judgement	Support for judgement
Random sequence generation (selection bias)	Low risk	"all primary care providers were randomly assigned by standard randomization techniques"
Allocation concealment (selection bias)	Low risk	"all primary care providers were randomly assigned by standard randomization techniques"
Blinding of outcome assessment (detection bias)   All outcomes	Low risk	"chart audit with audit protocol: intra‐auditor consistency > 90%"
Incomplete outcome data (attrition bias)   All outcomes	Low risk
Selective reporting (reporting bias)	Unclear risk	not clear
Other bias	Unclear risk	not clear
Outcomes at baseline similar?	Low risk	"compliance scores prior to study were not statistically significantly different"
Characteristics at baseline similar?	Low risk	"patients did not differ significantly by age, race or gender"
Adequate protection against contamination?	Low risk	providers randomized

Open in a new tab

Majumdar 2007.

Methods	Cluster‐randomized trial; unit of allocation: health professional
Participants	Patients: patients with heart failure or with ischemic heart disease; women: 60%; median age: 75; n = 171 Professionals: primary care physicians, n = 769 potentially eligible randomized (128 analyzed) Setting: outpatient (Alberta, Canada)
Interventions	Active control: medication profile Intervention: physician reminder; medication profile Duration of intervention: 3.5 year
Outcomes	Quality of care: prescription rate of efficacious therapies within 6 months (primary); prescription rate of ACE inhibitors or ARBs in heart failure patients; prescription rate of statins in ischemic heart disease patients
Clinical area and targeted activity	Cardiovascular disease (prescribing)
Reminder	Description: For each patient, a condition‐specific one‐page evidence summary was generated in the form of a letter addressed to the primary care physician and was faxed to him. The letters identified patients and their diagnoses, briefly described the key evidence in support of the study medications, and were signed by opinion leaders. The intent of the intervention was that the evidence summary (and medical profile) would become part of the patient's medical record and act as a point‐of‐care reminder for the next patient visit. Typology: patient‐specific: YES; space for response: NO; explicit advice: YES; explanation: YES; reference: YES; at point‐of‐care: NO
Notes
*Risk of bias*
Bias	Authors' judgement	Support for judgement
Random sequence generation (selection bias)	Low risk	"simple randomization with concealment of allocation was performed with the use of a computer‐generated sequence"
Allocation concealment (selection bias)	Low risk	"simple randomization with concealment of allocation was performed with the use of a computer‐generated sequence"
Blinding of outcome assessment (detection bias)   All outcomes	Low risk	"all outcomes were ascertained in an independent and blinded fashion, and allocation was concealed from patients, investigators, data collectors, and analysts"
Incomplete outcome data (attrition bias)   All outcomes	Low risk	"no study patients were lost to follow‐up". 769 professionals were randomized, but 128 were analyzed as most did not contribute an eligible patient.
Selective reporting (reporting bias)	High risk	results on secondary outcomes not reported.
Other bias	Low risk	not clear
Outcomes at baseline similar?	Unclear risk	not reported
Characteristics at baseline similar?	Low risk	"the intervention and control subjects were comparable, with no important differences"
Adequate protection against contamination?	Low risk	physicians allocated to intervention

Open in a new tab

Mazzuca 1990.

Methods	Cluster non‐randomized trial; unit of allocation: clinical area, n = 4
Participants	Patients: patients with non‐insulin dependent diabetes mellitus; women: not clear; mean age: not clear; total n = 2791 (estimated 1395 in the 2 included study arms) Professionals: 99 internal medicine residents and 15 faculty internists (total n = 114) in four clinical areas (estimated 57 in the 2 included study arms) Setting: outpatient (4 clinic areas of a general medicine clinic, Indiana University, USA)
Interventions	Active control: educational meeting (postgraduate seminar) Intervention: physician reminder; educational meeting (postgraduate seminar) Duration of intervention: 11 months
Outcomes	Quality of care: adherence rate to recommendations for: lab orders (glycated hemoglobin, fasting blood glucose) and therapies (home‐monitored blood glucose, diet clinic referral, oral hypoglycemic agents) (median)
Clinical area and targeted activity	Diabetes (multiple: test ordering, prescribing, referral)
Reminder	Description: Printed reminders were placed in patients' clinic records whenever the computer detected history, physical, laboratory, or pharmacy data indicating the need to consider a recommendation (e.g. if the patient was obese and without a diet on record, the computer would print out the following reminder on a separate sheet). (same system as McDonald 1984) Typology: patient‐specific: YES; space for response: NO; explicit advice: YES; explanation: NOT CLEAR; reference: NO; at point‐of‐care: YES
Notes	Additional study interventions excluded from analyses: a) physician reminder; educational meeting (postgraduate seminar); physician educational material; b) physician reminder; educational meeting (postgraduate seminar); physician educational material; patient education service
*Risk of bias*
Bias	Authors' judgement	Support for judgement
Random sequence generation (selection bias)	High risk	"the four areas were assigned arbitrarily to study conditions according to a nonequivalent control group design"
Allocation concealment (selection bias)	High risk	"the four areas were assigned arbitrarily to study conditions according to a nonequivalent control group design"
Blinding of outcome assessment (detection bias)   All outcomes	Low risk	"computerized audit of medical record"
Incomplete outcome data (attrition bias)   All outcomes	Unclear risk	not clear
Selective reporting (reporting bias)	Unclear risk	not clear
Other bias	Unclear risk	not clear
Outcomes at baseline similar?	Low risk	"baseline interviews with a random sample of 175 patients showed the 4 groups to be equivalent with respect to general therapeutic practices"
Characteristics at baseline similar?	Unclear risk	not clear
Adequate protection against contamination?	Low risk	randomization by clinical area

Open in a new tab

McAlister 2009.

Methods	Cluster‐randomized trial, unit of allocation: health professional
Participants	Patients: > 18 years old, diagnosed with coronary artery disease (stenosis in at least one coronary vessel of ≥ 50%) eligible for but not already taking a statin or who were on a suboptimal regimen, women: 21%, mean age: 64, n = 480 Professionals: primary care physicians, n = 252 Setting: outpatient (252 general practices in Alberta, Canada)
Interventions	Active control: coronary artery diagram Intervention (two intervention arms combined): coronary artery diagram; physician reminder Duration of intervention: 6 months
Outcomes	Quality of care: compliance with recommendation at six months (statin initiation or increase in dose) (primary); taking a statin; standardized statin dose; taking another lipid‐lowering drug; acetylsalicylic acid; acetylsalicylic acid or thienopyridine; ACE inhibitor; ACE inhibitor or ARB; β‐Blocker; triple therapy Patient outcomes: mortality rate Resource use/cost: ED visits, hospitalizations
Clinical area and targeted activity	Cardiovascular disease (prescribing)
Reminder	Description: The statement was a 1‐page summary of evidence‐based secondary prevention strategies and treatment recommendations. The Local Opinion Leader Statement contained the signatures of 5 local opinion leaders, while the Unsigned Evidence Statement was unsigned. These statements were imprinted with the name of the patient, addressed directly to the patient's physician, and faxed automatically by a software program, along with a coronary artery diagram documenting the extent of the patient's coronary atherosclerosis. These statements were sent to physicians within a few days of the angiogram. Typology: patient‐specific: YES; space for response: NO; explicit advice: YES; explanation: YES; reference: YES for intervention a/NO for intervention b; at point‐of‐care: NO
Notes	Intervention a: coronary artery diagram; physician reminder (local opinion leader statement) Intervention b: coronary artery diagram; physician reminder (unsigned evidence statement) Intervention groups considered separately in 1 analysis (content of reminder: reference versus no reference)
*Risk of bias*
Bias	Authors' judgement	Support for judgement
Random sequence generation (selection bias)	Low risk	"computer‐generated central randomization system with concealment of the randomization list"
Allocation concealment (selection bias)	Low risk	"computer‐generated central randomization system with concealment of the randomization list"
Blinding of outcome assessment (detection bias)   All outcomes	Low risk	"Blinding of outcome assessors and analysts"
Incomplete outcome data (attrition bias)   All outcomes	Low risk	reasons provided for excluded patients; intention‐to‐treat analyses
Selective reporting (reporting bias)	Low risk	outcomes reported match outcomes described in protocol
Other bias	Low risk	no recruitment bias; GEE analyses
Outcomes at baseline similar?	Low risk	"No appreciable differences between the treatment arms" for statin use (P = 0‐87) and statin dosing (P = 0.84)
Characteristics at baseline similar?	Low risk	"At baseline, there were no statistically significant differences between groups".
Adequate protection against contamination?	Low risk	randomization by health professional

Open in a new tab

McDonald 1976a.

Methods	Cluster non‐randomized trial (cross‐over), unit of allocation: health professional
Participants	Patients: percentage of women: not clear, mean age: not clear, n = 189 Professionals: physicians, n = 9 Setting: outpatient care (general medicine clinic at Wishard Memorial Hospital, Indianapolis, USA)
Interventions	Control: usual care Intervention: physician reminder Duration of intervention: 17 weeks
Outcomes	Quality of care: overall compliance rate (primary); compliance rate with observing a physical finding or inquiring about a symptom; compliance with ordering a diagnostic study; compliance with changing or initiating a therapeutic regimen
Clinical area and targeted activity	General care (multiple: prescribing, test ordering, other)
Reminder	Description: The Regenstrief Medical records system searches its records for events and makes recommendations about the management. Each recommendation consists of a reminder to the physician that a particular event has occurred and a suggested course of "action" for correcting that event. The study involved 390 protocols (recommendations). The computer prints 3 reports. The first is the surveillance report, which contains all the computer recommendations for a given patient. The second is a computer‐tailored encounter form, which provides space for recording findings. The third report, the summary report, is not influenced by the computer protocols and is a flow‐sheet summary. Typology: patient‐specific: YES; space for response: YES; explicit advice: YES; explanation: YES; reference: NO; at point‐of‐care: YES
Notes
*Risk of bias*
Bias	Authors' judgement	Support for judgement
Random sequence generation (selection bias)	High risk	no mention of randomization
Allocation concealment (selection bias)	High risk	not described
Blinding of outcome assessment (detection bias)   All outcomes	Unclear risk	outcome data gathered through medical record review, no indication of blinding
Incomplete outcome data (attrition bias)   All outcomes	Unclear risk	not clear
Selective reporting (reporting bias)	Unclear risk	not clear
Other bias	Unclear risk	not clear
Outcomes at baseline similar?	Unclear risk	not reported
Characteristics at baseline similar?	Unclear risk	not reported
Adequate protection against contamination?	High risk	cross‐over design

Open in a new tab

McDonald 1976b.

Methods	Randomized trial, unit of allocation: patient
Participants	Patients: adult diabetic patients; percentage of women: not clear, mean age: 60, n = 257 (226 analyzed) Professionals: diabetologists, residents, interns, senior medical residents, nurse clinicians, n = 63 Setting: outpatient (diabetic clinic of Wishard Memorial Hospital, Indianapolis, USA)
Interventions	Control: usual care Intervention: physician reminder Duration of intervention: 8 months
Outcomes	Quality of care: compliance rate with recommendations for test ordering/therapeutical change (median)
Clinical area and targeted activity	Diabetes (multiple: prescribing, test ordering)
Reminder	Description: The computerized medical record system (Regenstrief) generated prospective, protocol‐driven recommendations, which alerted the clinician to the existence of, and the proper response to, simple events. Two types of protocols: 1) taking measurement at proper intervals, based on "if on drug A and no test B for X months then order test B"; 2) reacting to measures that implied that treatment was either insufficient, excessive, or dangerous, based on "if on drug A and last test B abnormal, then warn about possible changed drug action". The computer printed 3 reports: the summary report, with a flow‐sheet; the patient encounter form with space for writing new medication orders; the surveillance report, with the protocol‐generated suggestions to physician, with recommended tests, date of last test, treatments triggering recommendation, recommendations for specific changes in therapeutics, with rationale for the change suggested. Typology: patient‐specific: YES; space for response: YES; explicit advice: YES; explanation: YES; reference: YES; at point‐of‐care: YES
Notes
*Risk of bias*
Bias	Authors' judgement	Support for judgement
Random sequence generation (selection bias)	Low risk	randomized by computer‐generated random number
Allocation concealment (selection bias)	Unclear risk	not described
Blinding of outcome assessment (detection bias)   All outcomes	Unclear risk	not clear
Incomplete outcome data (attrition bias)   All outcomes	Unclear risk	not clear
Selective reporting (reporting bias)	Unclear risk	not clear
Other bias	Unclear risk	not clear
Outcomes at baseline similar?	Unclear risk	not reported
Characteristics at baseline similar?	Unclear risk	not reported
Adequate protection against contamination?	High risk	most practitioners saw patients from both the study and control groups

Open in a new tab

McDonald 1980.

Methods	Cluster‐randomized trial (cross‐over trial), unit of allocation: health professional
Participants	Patients: n = not clear; conditions: 3691 Professionals: 31 providers (9 interns, 17 residents, 5 nurse‐practitioners) Setting: outpatient (General medicine service, University Hospital, Indianapolis, USA)
Interventions	Control: usual care Intervention (2 intervention arms combined): physician reminder Duration of intervention: 15 weeks
Outcomes	Quality of care: compliance rate
Clinical area and targeted activity	Preventive care (multiple: test ordering, prescribing, general management)
Reminder	Description: At each visit, the computer provides a tailored encounter form and a flow sheet summary of each patient's medical history and follows physician‐authored management rules (n = 410) to remind the physician about patient conditions requiring his attention. The report containing these reminders is called the "surveillance" report. Practitioners order all diagnostic treatments and referrals by recording them on the encounter form. Typology: patient‐specific: YES; space for response: YES; explicit advice: YES; explanation: YES; reference: YES for intervention a / NO for intervention b; at point‐of‐care: YES
Notes	Intervention a: physician reminder (with bibliographic citations) Intervention b: physician reminder (without citations) Intervention groups considered separately in 1 analysis (content of reminder: reference versus no reference)
*Risk of bias*
Bias	Authors' judgement	Support for judgement
Random sequence generation (selection bias)	Unclear risk	physicians were randomly assigned to the order to which they received their intervention
Allocation concealment (selection bias)	Unclear risk	not described
Blinding of outcome assessment (detection bias)   All outcomes	Unclear risk	not clear
Incomplete outcome data (attrition bias)   All outcomes	Unclear risk	not clear
Selective reporting (reporting bias)	Unclear risk	not clear
Other bias	Unclear risk	not clear
Outcomes at baseline similar?	Unclear risk	not reported
Characteristics at baseline similar?	Unclear risk	not reported
Adequate protection against contamination?	High risk	cross‐over trial

Open in a new tab

McDonald 1984.

Methods	Cluster‐randomized trial, unit of allocation: practice team, n = 27
Participants	Patients: women: 65%; mean age: not clear; n = 12,467 Professionals: 130 providers (115 residents, 11 faculty, 4 nurses) within practice teams (n = 115 in included study groups) Setting: outpatient (27 teams in a general medicine service, University Hospital, Indianapolis USA)
Interventions	Control: usual care Intervention: physician reminder Duration of intervention: 2 years
Outcomes	Quality of care: overall compliance rate (primary); compliance with occult blood, cervical smear, chest roentgenogram, pneumococcal vaccine, tuberculosis skin test, serum potassium, mammogram, influenza vaccine, diet, digitalis, antacids, beta blockers (12 reminders)
Clinical area and targeted activity	Preventive care (multiple: vaccination, test ordering, prescribing)
Reminder	Description: The computerized medical record was programmed with 1491 rules that could generate 751 different reminder messages. The set of rules included reminders about preventive care, obtaining tests needed to complete the initial database or to identify the cause of existing abnormalities, about prophylactic treatment and treatment of active problems. The computer used these rules to review each patient's electronic medical record the day before each visit. For each patient, when it found conditions satisfying the reminder rule, a reminder was stored on a disc file. The computer gathered these messages as a printed report that was attached to the charts of scheduled patients. The reminder messages included citations to the relevant medical literature. Typology: patient‐specific: YES; space for response: YES; explicit advice: YES; explanation: YES; reference: YES; at point‐of‐care: YES
Notes
*Risk of bias*
Bias	Authors' judgement	Support for judgement
Random sequence generation (selection bias)	Unclear risk	"Each team was randomized as to study or control"
Allocation concealment (selection bias)	Unclear risk	not described
Blinding of outcome assessment (detection bias)   All outcomes	Low risk	outcomes measured using computer
Incomplete outcome data (attrition bias)   All outcomes	Low risk	physicians with fewer than 100 reminder messages during study period excluded (2%)
Selective reporting (reporting bias)	Unclear risk	not clear
Other bias	Unclear risk	not clear
Outcomes at baseline similar?	Unclear risk	Not reported
Characteristics at baseline similar?	Low risk	No significant difference in gender and age for patients
Adequate protection against contamination?	Unclear risk	practice team randomized, but faculty members and nurse‐clinicians saw both study groups

Open in a new tab

McPhee 1989.

Methods	Cluster‐randomized trial, unit of allocation: health professional
Participants	Patients: ≥ 40 years old, women: 67%, mean age: not clear; n = ˜1291 patients in four included study arms Professionals: 42 residents in four included study arms Setting: outpatient (General Internal Medicine Practice, University hospital, San Francisco USA)
Interventions	Control (comparison 1): usual care Intervention (comparison 1): physician reminder Active control (comparison 2): distribution of educational material to patients; patient reminder Intervention (comparison 2): physician reminder; distribution of educational material to patients; patient reminder Duration of intervention: nine months
Outcomes	Quality of care: physician compliance rate with: FOBT, rectal exam, sigmoidoscopy, pap smear, pelvic exam, breast exam, and mammography (median) Cost: cost per patient; cost per additional screening test
Clinical area and targeted activity	Preventive care/colorectal, breast, cervical cancer (test ordering)
Reminder	Description: The cancer screening reminders provided residents with up‐to‐date information about their patient's screening status. The research staff printed cancer screening reminders at the time of each patient appointment and updated existing microcomputer files by re‐auditing patient medical records in preparation for return appointments. At the time of a patient's visit to the resident, a cancer screening reminder was attached to the regular encounter form or medical record. A new reminder was generated for each patient encounter. Reminders displayed the list of appropriate cancer screening procedures (based on the patient's sex and age), the recommended testing intervals, the last performance date, the due date for each next test, and the patient's current "due" status. The physician indicated on the form whether or not each test was performed or ordered during the current visit. Typology: patient‐specific: YES; space for response: YES; explicit advice: YES; explanation: NO; reference: YES; at point‐of‐care: YES
Notes	Data extracted from graphics. Author not contacted as publication date > 10 years. Additional study interventions excluded from analyses: a) audit and feedback, b) audit and feedback; distribution of educational material to patients; patient reminder
*Risk of bias*
Bias	Authors' judgement	Support for judgement
Random sequence generation (selection bias)	Unclear risk	"residents were randomly assigned to one of six intervention groups"
Allocation concealment (selection bias)	Unclear risk	not described
Blinding of outcome assessment (detection bias)   All outcomes	Low risk	Medical records audit: no blinding but reliability test performed on random sample (90‐98%)
Incomplete outcome data (attrition bias)   All outcomes	High risk	1969 records audited at baseline = 72% of eligible patients; 1936 records audited during study = 71% of eligible patients
Selective reporting (reporting bias)	Unclear risk	not clear
Other bias	Unclear risk	not clear
Outcomes at baseline similar?	Low risk	compliance scores at baseline not significantly different
Characteristics at baseline similar?	Unclear risk	not reported
Adequate protection against contamination?	Low risk	providers randomized

Open in a new tab

Morgan 1978.

Methods	Non‐randomized trial, unit of allocation: patient
Participants	Patients: pregnant women; mean age: not clear, n = 279 Professionals: physicians, n = 5 Setting: outpatient (Harvard Community Health Plan ambulatory care center, Boston, USA)
Interventions	Control: usual care Intervention: physician reminder Duration of intervention: 18 months
Outcomes	Quality of care: compliance rate with 5 minimum standard of care (at 18months): blood group and type, syphilis serology, prenatal counseling, pregnancy diet counseling, sickle cell preparation (median)
Clinical area and targeted activity	Prenatal care (multiple: test ordering, professional‐patient communication)
Reminder	Description: COSTAR was programmed to automatically check the records of patients at the time of each prenatal visit to determine if physicians had complied with specific indices of care. An updated summary record was printed prior to the patient's visit with the list of missing items, under the heading "Data incomplete". Typology: patient‐specific: YES; space for response: NO; explicit advice: NO; explanation: NO; reference: NO; at point‐of‐care: YES
Notes
*Risk of bias*
Bias	Authors' judgement	Support for judgement
Random sequence generation (selection bias)	High risk	"patients with odd‐numbered medical records were assigned to the experimental group (...) patients with even‐numbered medical records were assigned to a control group".
Allocation concealment (selection bias)	High risk	allocation based on odd/even number of medical record
Blinding of outcome assessment (detection bias)   All outcomes	Unclear risk	outcomes likely assessed by computer, but no specific indication in text
Incomplete outcome data (attrition bias)   All outcomes	Unclear risk	not clear
Selective reporting (reporting bias)	Unclear risk	not clear
Other bias	Unclear risk	not clear
Outcomes at baseline similar?	Unclear risk	not reported
Characteristics at baseline similar?	Unclear risk	not reported
Adequate protection against contamination?	High risk	physicians saw patients from intervention and control groups

Open in a new tab

Nilasena 1995.

Methods	Cluster‐randomized trial (blocked by site and level of training), unit of allocation: health professional
Participants	Patients: women: not clear; mean age: not clear; n = 480 identified (164 analyzed) Professionals: internal medicine residents, n = 35 randomized out of 36 Setting: outpatient (2 clinics, USA)
Interventions	Active control: blank encounter form; educational meeting Intervention: physician reminder; blank encounter form; ; educational meeting Duration of intervention: 6 months
Outcomes	Quality of care: overall compliance rate (primary)
Clinical area and targeted activity	Diabetes (multiple: test ordering, prescribing, general management, professional‐patient communication)
Reminder	Description: The computer program outputs a printed paper health maintenance report for the patient's primary physician based on the currently available data for the patient. The report summarizes the patient's diabetes preventive‐health status, and lists a schedule of upcoming or past due preventive‐health activities for the patient. Clinical alerts about high‐risk aspects of the patient's current profile are also presented. The report is placed at the front of the patient's chart. Typology: patient‐specific: YES; space for response: NO; explicit advice: YES; explanation: NOT CLEAR; reference: NO; at point‐of‐care: YES
Notes
*Risk of bias*
Bias	Authors' judgement	Support for judgement
Random sequence generation (selection bias)	Unclear risk	"resident physicians were randomly assigned"
Allocation concealment (selection bias)	Unclear risk	not described
Blinding of outcome assessment (detection bias)   All outcomes	Unclear risk	manual chart review ‐ no blinding procedure described
Incomplete outcome data (attrition bias)   All outcomes	High risk	480 patients identified, but 164 analyzed (34.2%). No exclusion reasons.
Selective reporting (reporting bias)	Unclear risk	not clear
Other bias	Unclear risk	not clear
Outcomes at baseline similar?	High risk	baseline compliance lower in control group (34.6%) than intervention group (38.0%)
Characteristics at baseline similar?	Unclear risk	not reported
Adequate protection against contamination?	Low risk	health professional randomized

Open in a new tab

Oniki 2003.

Methods	Non‐randomized trial, unit of allocation: patient
Participants	Patients: women: not clear; mean age: not clear, n = 120 Professionals: nurses (n = 109) Setting: inpatient (2 intensive care units in LDS Hospital, Department of Critical Care, Utah, USA)
Interventions	Control: usual care Intervention: physician reminder Duration of intervention: 59 days (45 days in control group)
Outcomes	Quality of care: mean deficiencies per day per patient (patient's total number of deficiencies/number of valid days in study) (primary)
Clinical area and targeted activity	Intensive care (multiple: record keeping, general management)
Reminder	Description: The computer system stored any mid‐day (13:00) reminders generated in a database. For each patient in the study group, the program generated a reminder report which listed patient's room, patient number/name, the date and any reminder (e.g. no Glasgow Coma Score between 7:00 and 13:00). Any reports containing reminders were delivered to charge nurse, who delivered them to the bedside nurse. Typology: patient‐specific: YES; space for response: NO; explicit advice: NO; explanation: NO; reference: NO; at point‐of‐care: YES
Notes
*Risk of bias*
Bias	Authors' judgement	Support for judgement
Random sequence generation (selection bias)	High risk	patients assigned according to ICU room to which they were admitted
Allocation concealment (selection bias)	High risk	patients assigned according to ICU room to which they were admitted
Blinding of outcome assessment (detection bias)   All outcomes	Low risk	outcome measure collected from computer system
Incomplete outcome data (attrition bias)   All outcomes	Unclear risk	not clear
Selective reporting (reporting bias)	Unclear risk	not clear
Other bias	Unclear risk	not clear
Outcomes at baseline similar?	Unclear risk	not reported
Characteristics at baseline similar?	Unclear risk	not reported
Adequate protection against contamination?	High risk	40 nurses cared for study and control patients

Open in a new tab

Ornstein 1991.

Methods	Cluster‐randomized trial, unit of allocation: practice, n = 4
Participants	Patients: > 18 years old, due for prevention services; women: 61%; mean age: 40; n = 7,397 (3564 in comparison 1; 3833 in comparison 2) Professionals: 49 family physicians (6 faculty, 1 fellow, 42 residents) Setting: outpatient (four practice groups in the Family Medicine Center at the Medical University of South Carolina, USA)
Interventions	Active control (comparison 1): educational meeting; audit and feedback Intervention (comparison 1): physician reminder; educational meeting; audit and feedback Active control (comparison 2): patient reminder; educational meeting; audit and feedback Intervention (comparison 2): physician reminder; patient reminder; educational meeting; audit and feedback Duration of intervention: 12 months
Outcomes	Quality of care: physician adherence rate to 5 preventive services: FOBT, mammography, tetanus vaccine, cholesterol, pap smear (median)
Clinical area and targeted activity	Preventive care/colorectal, breast, cervical cancer, tetanus vaccination and cholesterol prevention (multiple: test ordering, vaccination)
Reminder	Description: Reminder forms were generated by the computer system for each patient the night before a scheduled appointment. The reminders were generated by scanning each patient record for deficient preventive services based on the patient's age, sex, and last recorded time of the service. Forms were printed on single sheets of paper and attached to the medical record by nursing personnel the morning of the scheduled visit. The top half of the form listed identifying information and 0 to 5 deficient preventive services. It contained boxed for the physician to mark, indicating his or her action on each particular reminder. Actions included ordering the preventive service that day, scheduling the patient to return for it another day, noting that it was not indicated for the patient, offering it to the patient but having the patient refuse, or not discussing it. The bottom half of the reminder form listed any of the 5 preventive services appropriate for the patient's age and sex, and the date the item was last received. Typology: patient‐specific: YES; space for response: YES; explicit advice: YES; explanation: NO; reference: NO; at point‐of‐care: YES
Notes
*Risk of bias*
Bias	Authors' judgement	Support for judgement
Random sequence generation (selection bias)	Unclear risk	"patients and their physicians were randomly assigned by practice group into study groups"
Allocation concealment (selection bias)	Unclear risk	not described
Blinding of outcome assessment (detection bias)   All outcomes	Low risk	"[outcome] assessed through computerized medical records"
Incomplete outcome data (attrition bias)   All outcomes	Unclear risk	not clear
Selective reporting (reporting bias)	Unclear risk	not clear
Other bias	Unclear risk	not clear
Outcomes at baseline similar?	High risk	adherence rate at baseline differ according to study groups
Characteristics at baseline similar?	High risk	"statistically significant differences between study groups were present for race, insurance coverage, and visit frequency"
Adequate protection against contamination?	Low risk	physicians randomized

Open in a new tab

Rosser 1991.

Methods	Cluster‐randomized trial, unit of allocation: family, n = 4450 families randomized
Participants	Patients: 15 years old or more; percentage of women: not clear; mean age: not clear; 1403 patients (1056 families) randomized to usual care and 1471 patients (1122 families) randomized to included study arm Flu arm: 822 families randomized, 939 patients; blood pressure arm: 4247 families randomized, 5744 patients; Pap arm: 1406 women randomized; Tetanus arm: 4247 families randomized, 5589 patients Professionals: staff physicians, residents, and nurses, n = not clear Setting: outpatient (four practices, University of Ottawa Family Medicine Center, Canada)
Interventions	Control: usual care Intervention: physician reminder Duration of intervention: 12 months (69 days for flu arm)
Outcomes	Quality of care: rate of eligible patients for whom the recommended procedure was performed (primary); rate of eligible patients with: influenza vaccination, blood pressure reading, pap test, tetanus vaccination
Clinical area and targeted activity	Preventive care/influenza vaccination, blood pressure, smoking status, pap test, tetanus (multiple: test ordering, vaccination)
Reminder	Description: Computer‐generated reminders were included on the routinely printed encounter form before any visit to the office to remind the physician of outstanding preventive procedures. Until the procedure or reading was recorded, the computer continued to generate reminders on subsequent visits. Influenza vaccination reminder: for patients who had not already been vaccinated, the message stated "Patient 65 or older: check flu immunization". Blood pressure reminder: When the patient booked an appointment, the computer printed a reminder advising the doctor to "Check blood pressure". Pap reminder: for women who had not been screened during previous year, the computer printed a message to the physician to recommend cervical screening. Tetanus vaccination reminder: "ask patient about tetanus vaccination" was included on the routinely printed encounter form. Typology: patient‐specific: YES; space for response: NO; explicit advice: YES; explanation: NO; reference: NO; at point‐of‐care: YES
Notes	Additional study interventions excluded from analyses: a) patient phone reminder, b) patient letter reminder
*Risk of bias*
Bias	Authors' judgement	Support for judgement
Random sequence generation (selection bias)	Low risk	"with the use of a standard randomization computer program"
Allocation concealment (selection bias)	Unclear risk	not described
Blinding of outcome assessment (detection bias)   All outcomes	Unclear risk	Influenza vaccination recorded in the computer, but data completed by phone calls which are not described as blinded
Incomplete outcome data (attrition bias)   All outcomes	Low risk	possible that patients had undergone preventive measures somewhere else, but similar situation across groups
Selective reporting (reporting bias)	Unclear risk	not clear
Other bias	Unclear risk	not clear
Outcomes at baseline similar?	Unclear risk	not reported
Characteristics at baseline similar?	Low risk	no significant difference for age and gender (family size was different before exclusion of practices)
Adequate protection against contamination?	Unclear risk	families (patients) randomized; physicians likely saw both study and control patients

Open in a new tab

Rossi 1997.

Methods	Cluster‐randomized trial, unit of allocation: health professional
Participants	Patients: men: 96%, mean age: 68, n = 719 patients Professionals: physicians (n = 15), residents/fellows (n = 44) and nurse practitioners (n = 12) Setting: outpatient (General internal medicine clinic of the VA Pounget Sound Health Care System, USA)
Interventions	Control: usual care Intervention: physician reminder Duration of intervention: 6 months
Outcomes	Quality of care: prescription change rate (primary) Patient outcomes: mean blood pressure Resource use: mean clinic visits per patient, mean ED visits per patient, mean creatinine tests per patient, mean total cholesterol tests per patient
Clinical area and targeted activity	Hypertension (prescribing)
Reminder	Description: An automated computer query system identified eligible patients and their providers. For each clinic visit, for each eligible patient, providers had a 1‐page guideline reminder placed in the patient chart by the clinic pharmacist, attached to the medication refill forms that are given to providers at every patient visit. The reminder highlighted the prescription and offered alternative drugs and doses. For continued medication use, the reminder also asked provider to designate 1 of 4 indications. Typology: patient‐specific: YES; space for response: YES; explicit advice: YES; explanation: YES; reference: YES; at point‐of‐care: YES
Notes
*Risk of bias*
Bias	Authors' judgement	Support for judgement
Random sequence generation (selection bias)	Low risk	"A random number generator was used to randomize providers, stratified to whether they were staff physicians, nurse practitioners, or residents or fellows".
Allocation concealment (selection bias)	Low risk	randomization at the physician level; providers were assigned numeric codes and study investigators were blinded to the coding identifiers
Blinding of outcome assessment (detection bias)   All outcomes	Low risk	outcome data from computer database
Incomplete outcome data (attrition bias)   All outcomes	Unclear risk	not clear
Selective reporting (reporting bias)	Unclear risk	not clear
Other bias	Unclear risk	not clear
Outcomes at baseline similar?	Unclear risk	not reported
Characteristics at baseline similar?	High risk	provider characteristics (type, gender, number patients, number patient visits) were not statistically different; patient characteristics (age, race, gender, weight, blood pressure) were not statistically different, but mean prescriptions per patient and percentage on ß‐blockers and ACE inhibitors were different
Adequate protection against contamination?	Low risk	physicians randomized

Open in a new tab

Thomas 1983.

Methods	Randomized trial, unit of allocation: patient
Participants	Patients: diabetic patients; percentage of women: not clear, mean age: not clear; n = 185 (133 analyzed) Professionals: physicians, n = not clear Setting: outpatient (University of Texas Medical School Ambulatory Clinic, USA)
Interventions	Control: usual care Intervention: physician reminder Duration of intervention: 12 months (not finished)
Outcomes	Quality of care: compliance rate (primary) Resource use/cost: percentage of patients hospitalized, mean days of hospitalization, mean number of patient visits, mean costs per patient
Clinical area and targeted activity	Diabetes (multiple: not specified)
Reminder	Description: An automated chart audit is obtained for each patient scheduled to attend the clinic, based on protocol‐driven algorithms, and prints a report for the physician. 80% of suggestions are recommendations concerning general medicine and preventive care, and 20% are for specific speciality problems. Typology: patient‐specific: YES; space for response: NO; explicit advice: YES; explanation: NOT CLEAR; reference: NOT CLEAR; at point‐of‐care: YES
Notes
*Risk of bias*
Bias	Authors' judgement	Support for judgement
Random sequence generation (selection bias)	Unclear risk	"Patients were assigned through the use of stratified random sample"
Allocation concealment (selection bias)	Unclear risk	not described
Blinding of outcome assessment (detection bias)   All outcomes	Unclear risk	"a medical research assistant interviewed each patient for every visit during the study; all study data were then coded and entered into the automated medical record system"
Incomplete outcome data (attrition bias)   All outcomes	High risk	305 patients entered study; 52 dropped out; 133 with first 12 mo data (43.6%)
Selective reporting (reporting bias)	Unclear risk	not clear
Other bias	Unclear risk	not clear
Outcomes at baseline similar?	Unclear risk	not reported
Characteristics at baseline similar?	Unclear risk	not reported
Adequate protection against contamination?	High risk	patients randomized

Open in a new tab

Tierney 1986.

Methods	Cluster‐randomized trial, unit of allocation: clinic session (n = 32)
Participants	Patients: percentage women: not clear, mean age: not clear; n = 6045 Professionals: faculty, interns and residents, n = 135 Setting: outpatient (general medicine clinic of Wishard Memorial Hospital, Indianapolis, USA)
Interventions	Control (comparison 1): usual care Intervention (comparison 1): physician reminder Active control (comparison 2): feedback (delayed reminder) Intervention (comparison 2): physician reminder; feedback (delayed reminder) Duration of intervention: 7 months
Outcomes	Quality of care: physician compliance with 13 protocols: FOBT, pneumococcal vaccination, antacids, TB skin testing, beta‐blockers, nitrates, anti‐depressants, calcium supplements, pap smear, mammography, metronidazole, digitalis, salicylates (median)
Clinical area and targeted activity	Preventive care (multiple: test ordering, vaccination, prescribing)
Reminder	Description: The reminders were generated the night before scheduled appointments and were placed in patients' clinical charts. Each reminder identified the patient and listed the suggested preventive care along with data from the computer record that made the patient eligible for the action, along with supporting references (same system as McDonald 1984). Typology: Patient‐specific: YES; space for response: NO; explicit advice: YES; explanation: NO; reference: YES; at point‐of‐care: YES
Notes	Data extracted from graphics. Author not contacted as publication date > 10 years.
*Risk of bias*
Bias	Authors' judgement	Support for judgement
Random sequence generation (selection bias)	Unclear risk	"we randomized house staff by their clinic session"
Allocation concealment (selection bias)	Unclear risk	not described
Blinding of outcome assessment (detection bias)   All outcomes	Low risk	computerized data
Incomplete outcome data (attrition bias)   All outcomes	Unclear risk	not clear
Selective reporting (reporting bias)	Unclear risk	not clear
Other bias	Low risk	not clear
Outcomes at baseline similar?	Unclear risk	not reported
Characteristics at baseline similar?	Unclear risk	not reported
Adequate protection against contamination?	Low risk	allocation by clinic session

Open in a new tab

Turner 1989.

Methods	Cluster non‐randomized trial, unit of allocation: clinic team (n = 5)
Participants	Patients: with multiple chronic diseases, ≥ 20 yr, women: 74%, mean age: 60; n = 253 (n = 150 in included study arms) Professionals: junior and senior residents (n = 39) in 5 clinic teams (n = 25 in included study arms) Setting: outpatient (general medicine clinic, USA)
Interventions	Active control: patient‐mediated intervention (questionnaire given to physician); distribution of educational material to patient Intervention: physician reminder; patient‐mediated intervention (questionnaire given to physician); distribution of educational material to patient Duration of intervention: 6 months
Outcomes	Quality of care: adherence rates to recommendations: FOBT, rectal exam, mammography, pap smear, breast exam (median)
Clinical area and targeted activity	Preventive care (test ordering)
Reminder	Description: The computer record/reminder system used age‐ and sex‐standardized criteria for preventive care to generate patient‐specific reminder that was printed at the bottom of each patient's visit record sheet. Physicians were instructed to write in the date on which each listed service was last completed (C) or requested (R). On subsequent visits, the computer printed the updated information on the visit record; an asterisk in the "due " column indicated that it was time to repeat the service. Typology: Patient‐specific: YES; space for response: YES; explicit advice: YES; explanation: NO; reference: NO; at point‐of‐care: YES
Notes	Additional study intervention excluded from analyses: distribution of educational material to patient
*Risk of bias*
Bias	Authors' judgement	Support for judgement
Random sequence generation (selection bias)	High risk	allocation by day
Allocation concealment (selection bias)	High risk	allocation by day
Blinding of outcome assessment (detection bias)   All outcomes	High risk	"chart abstractors were not blind to the clinic groups of the patients in the audit"
Incomplete outcome data (attrition bias)   All outcomes	Unclear risk	not clear
Selective reporting (reporting bias)	High risk	tetanus vaccination rate not shown, as performance rate remained at less than 10% despite the interventions
Other bias	Unclear risk	not clear
Outcomes at baseline similar?	High risk	"extremely poor performance of mammography by the residents in the computer/questionnaire group"
Characteristics at baseline similar?	Low risk	"no significant differences were found among groups"
Adequate protection against contamination?	Low risk	clinic teams allocated, on different days

Open in a new tab

Were 2013.

Methods	Randomized trial, unit of allocation: patient
Participants	Patients: women: 49.3% , mean age: 7, n = 1619 randomized (1611 analyzed) Professionals: clinical officer, physician, nurse (n = 30) Setting: outpatient (pediatric HIV clinic, Kenya)
Interventions	Control: usual care Intervention: physician reminder Duration of intervention: 5 months
Outcomes	Quality of care: completion of overdue clinical tasks (primary); mean time (days) from when a patient had an overdue clinical task to the completion of the task
Clinical area and targeted activity	HIV care (multiple: prescription, test ordering, referral)
Reminder	The patient‐specific clinical summary, tailored for pediatric care, displayed selected information from the patient’s EHR to provide a quick reference to the most relevant data needed by clinicians. The module also contained CDSS functionality that appended patient‐ specific care reminders (overdue tests and treatments) to the bottom of the clinical summary. All summaries were attached to the relevant patients’ paper charts for clinicians to review during a patient’s clinic visit. Clinicians were asked to document their response to each reminder. No more than 5 reminders were displayed for each patient per visit. Typology: patient‐specific: YES; space for response: YES; explicit advice: YES; explanation: YES; reference: YES; at point‐of‐care: YES
Notes
*Risk of bias*
Bias	Authors' judgement	Support for judgement
Random sequence generation (selection bias)	Low risk	"All patients, both HIV‐exposed and HIV‐infected, previously enrolled at the study site were randomly assigned to either the intervention or control group in a 1:1 ratio by using a 4‐block randomization scheme."
Allocation concealment (selection bias)	Unclear risk	method of concealment not described
Blinding of outcome assessment (detection bias)   All outcomes	High risk	Authors don't mention blinding of outcome assessors.
Incomplete outcome data (attrition bias)   All outcomes	Unclear risk	8 patients were excluded from the analyses because they were seen by a clinician involved in the study. It is unclear if all the charts have been reviewed and if no data is missing.
Selective reporting (reporting bias)	Low risk	Primary outcome stated in the trial registry matches the primary outcome in the published results.
Other bias	Unclear risk	not clear.
Outcomes at baseline similar?	Unclear risk	no baseline outcome measures
Characteristics at baseline similar?	Low risk	"There were no statistically significant differences in the demographic characteristics between the control and intervention groups."
Adequate protection against contamination?	High risk	Clinicians received reminders for some patients but not others. "We randomized by patient in‐ stead of by clinician, because patients typically saw whichever clinician was first available at the time of their visit, and it was not possible to tell in advance which patient a clinician would see. We understood that this could sensitize clinicians to order the indicated care for control patients, which might bias our study against finding a significant effect for the reminders."

Open in a new tab

White 1984.

Methods	Randomized trial, unit of allocation: patient
Participants	Patients: women: 50% , mean age: 69, n = 396 Professionals: physicians, n = not clear Setting: inpatient (Hospital, Utah, USA)
Interventions	Control: usual care Intervention: physician reminder Duration of intervention: 3 months
Outcomes	Quality of care: compliance rate (primary)
Clinical area and targeted activity	Digoxin intoxication (prescribing)
Reminder	Description: Each night, the computer program activates the alert modules for all patients in the hospital. Alert message are formatted into a "digoxin alert report" that is sent out to a line printer in the nursing division nearest the patient. This report is placed in the patient chart by nursing personnel. Typology: Patient‐specific: YES; space for response: NO, explicit advice: YES; explanation: YES; reference: NO; at point‐of‐care: YES
Notes
*Risk of bias*
Bias	Authors' judgement	Support for judgement
Random sequence generation (selection bias)	Low risk	"a random number generator was used to assign patients to an alert or nonalert group"
Allocation concealment (selection bias)	Unclear risk	not described
Blinding of outcome assessment (detection bias)   All outcomes	Low risk	"a blinded medical record review was carried out in accordance with a protocol aimed at identifying physician actions with possible relation to the digoxin alerts"
Incomplete outcome data (attrition bias)   All outcomes	Unclear risk	not clear
Selective reporting (reporting bias)	Unclear risk	not clear
Other bias	Unclear risk	not clear
Outcomes at baseline similar?	Unclear risk	not reported
Characteristics at baseline similar?	Low risk	"there was no statistically significant difference between groups for sex, age, and medical/surgical service"
Adequate protection against contamination?	High risk	patients randomized

Open in a new tab

Ziemer 2006.

Methods	Cluster‐randomized trial (2 x 2 factorial design), unit of allocation: half‐day session (n = 40)
Participants	Patients: women: 67%; mean age: 59; 4,138 patients (n = 2026 in comparison 1; n = 2112 in comparison 2) Professionals: 345 providers (faculty, residents) (approx. 8 per session) Setting: outpatient (2 clinics with 20 half‐day sessions each in Atlanta, USA)
Interventions	Active control (comparison 1): educational meetings; distribution of educational material to physicians Intervention (comparison 1): physician reminder; educational meetings; distribution of educational material to physicians Active control (comparison 2): educational meetings; distribution of educational material to physicians; audit and feedback Intervention (comparison 2): physician reminder ;educational meetings; distribution of educational material to physicians; audit and feedback Duration of intervention: 3 years
Outcomes	Quality of care: providers who did enough (at 36 months) Patient outcomes: glycated hemoglobin level (primary); systolic blood pressure; LDL cholesterol; patients with glycated hemoglobin < 7%; patients with systolic blood pressure < 130 mmHg (no percentage data); patients with LDL cholesterol < 100 mg/dL (no percentage data)
Clinical area and targeted activity	Diabetes (general management)
Reminder	Description: The reminder includes a flow sheet showing clinically relevant parameters during the present visit and several previous visits, together with specific recommendations for management. The reminders document the course of critical values, provide notice when evaluations are due, include individualized recommendations for modifications in therapy. The patient‐specific reminder is printed out and attached to the front of the chart each time a patient with diabetes presents for a visit; a specific reminder sheet will be generated for each diabetic patient to be seen each week. Typology: Patient‐specific: YES; space for response: NO; explicit advice: YES; explanation: NO; reference: NO; at point‐of‐care: YES
Notes	Process data provided by authors.
*Risk of bias*
Bias	Authors' judgement	Support for judgement
Random sequence generation (selection bias)	Unclear risk	half‐day medical clinic sessions randomized; randomization not explicitly described
Allocation concealment (selection bias)	Unclear risk	not described
Blinding of outcome assessment (detection bias)   All outcomes	Unclear risk	not clear
Incomplete outcome data (attrition bias)   All outcomes	Unclear risk	not clear
Selective reporting (reporting bias)	Unclear risk	In published design paper, outcomes to be reported: level of glycated hemoglobin, blood pressure and lipid levels. Intensification of therapy, eye exam and foot exam: only results on intensification reported.
Other bias	Unclear risk	not clear
Outcomes at baseline similar?	Low risk	there were no differences in compliance rates among the different intervention arms at baseline (data sent by author)
Characteristics at baseline similar?	Low risk	there were no significant differences among the patients assigned to residents in the different intervention arms
Adequate protection against contamination?	Low risk	half‐day medical clinic sessions randomized

Open in a new tab

ACE: angiotensin converting enzyme, ARB: angiotensin receptor blocker, ED: emergency department, FOBT: fecal occult blood test, GEE: generalized estimating equation, HIV: human immunodeficiency virus, HMO: health maintenance organization, ICU: intensive care unit, LDL: low‐density lipoprotein, TB: tuberculosis, VA: Veterans Affairs

Characteristics of excluded studies [ordered by study ID]

Study	Reason for exclusion
Adelman 2013	Reminders delivered onscreen
Alfadda 2011	Ineligible comparison group or inappropriate control (assistant‐initiated reminder)
Anabtawi 2013	Reminders delivered onscreen
Anchala 2012	Ineligible comparison group or inappropriate control
Barkun 2013	Ineligible comparison or inappropriate control
Barnes 2014	Ineligible comparison or inappropriate control
Beeckman 2013	Ineligible comparison or inappropriate control
Beeler 2014	Reminders delivered onscreen
Belland 2014	Reminders delivered onscreen
Beste 2015	Reminders delivered onscreen
Braun 2013	Ineligible comparison or inappropriate control
Campbell 2014	Ineligible comparison or inappropriate control
Carroll 2013	Ineligible comparison or inappropriate control
Chen 2013	Not computer‐generated
Dexheimer 2013	Ineligible comparison or inappropriate control
Divinskiy 2015	Ineligible comparison or inappropriate control
dos Santos 2014	Reminders delivered onscreen
Erlingsdottir 2015	Not computer‐generated
Federman 2014	Reminders delivered onscreen
Gifford 2013	Not a reminder
Goldstein 2014	Reminders delivered onscreen
Green 2014	Ineligible comparison or inappropriate control
Gupta 2014	Reminders delivered onscreen
Hendrix 2015	Ineligible comparison or inappropriate control
Hye 2014	Reminders delivered onscreen
Jansink 2013	Ineligible comparison or inappropriate control
Kennedy 2012	Ineligible comparison or inappropriate control
Kousgaard 2013	Reminders delivered onscreen
Lai 2015	Reminders delivered onscreen
Levy 2013	Mixed reminder
Lusignan 2013	Ineligible comparison or inappropriate control
Lynn 2013	Ineligible comparison or inappropriate control
Majumdar 2012	Mixed reminder
Maximov 2013	Ineligible comparison or inappropriate control
McEvoy 2014	Not a reminder
McNulty 2014	Ineligible comparison or inappropriate control
Melo 2013	Ineligible comparison or inappropriate control
Neal 2012	Not original study
Nguyen 2014	Reminders delivered onscreen
O'Reilly 2012	Not original study
Persell 2016	Reminders delivered onscreen
Piazza 2013	Not computer‐generated
Roy 2016	Not computer‐generated
Schwalm 2015	Ineligible comparison or inappropriate control
Siersma 2015	Ineligible comparison or inappropriate control
Stockwell 2015	Reminders delivered onscreen
Szilagyi 2015	Reminders delivered onscreen
Tartaglia 2013	Ineligible comparison or inappropriate control
Tedja 2014	Mixed reminder
Teoh 2012	Not computer‐generated
Weiss 2013	Reminders delivered onscreen
Were 2011	Ineligible comparison or inappropriate control

Open in a new tab

Differences between protocol and review

The search strategies were revised and updated by Paul Miller, the Information Specialist at EPOC. Text sections of the review were updated to reflect the latest Cochrane and EPOC guidance for conducting and reporting reviews.(e.g. certainty of evidence).

We changed the wording of the outcomes: instead of 'process adherence outcomes', we now use 'quality of care outcomes', and instead of 'clinical outcomes', we now use 'patient outcomes', in line with EPOC guidelines (EPOC 2015d). We classified quality of care outcomes as primary outcomes and patient outcomes as secondary outcomes. We added adverse effects outcomes as secondary outcomes and added a subgroup analysis on disadvantaged populations.

Contributions of authors

CA led the update process. CA extracted and analyzed the updated data. CA amended the text of the review. BB reviewed the updated review. MR and PD read and commented on the final version.

Sources of support

Internal sources

Health Services Research Unit, University of Aberdeen, UK.
Centre Hospitalier Vaudois and University of Lausanne, Switzerland.

External sources

Loterie Romande, Lausanne, Switzerland.
Department of Community Medicine and Community Healthcare, Centre Hospitalier Universitaire Vaudois, Lausanne, Switzerland.

Research Grant

Declarations of interest

Chantal Arditi: none known.

Myriam Rège‐Walther: none known.

Pierre Durieux: none known.

Bernard Burnand: none known.

New search for studies and content updated (no change to conclusions)

References

References to studies included in this review

Barnett 1983 {published data only}

Barnett GO, Winickoff RN, Morgan MM, Zielstorff RD. A computer‐based monitoring system for follow‐up of elevated blood pressure. Medical Care 1983;21(4):400‐9. [DOI] [PubMed] [Google Scholar]

Becker 1989 {published data only}

Becker DM, Gomez EB, Kaiser DL, Yoshihasi A, Hodge RH Jr. Improving preventive care at a medical clinic: how can the patient help?. American Journal of Preventive Medicine 1989;5(6):353‐9. [PubMed] [Google Scholar]

Binstock 1997 {published data only}

Binstock MA, Geiger AM, Hackett JR, Yao JF. Pap smear outreach: a randomized controlled trial in an HMO. American Journal of Preventive Medicine 1997;13(6):425‐6. [PubMed] [Google Scholar]

Burack 1996 {published data only}

Burack RC, Gimotty PA, George J, Simon MS, Dews P, Moncrease A. The effect of patient and physician reminders on use of screening mammography in a health maintenance organization. Results of a randomized controlled trial. Cancer 1996;78(8):1708‐21. [PubMed] [Google Scholar]

Burack 1998 {published data only}

Burack RC, Gimotty PA, George J, McBride S, Moncrease A, Simon MS, et al. How reminders given to patients and physicians affected pap smear use in a health maintenance organization: results of a randomized controlled trial. Cancer 1998;82(12):2391‐400. [DOI] [PubMed] [Google Scholar]

Chambers 1989 {published data only}

Chambers CV, Balaban DJ, Carlson BL, Ungemack JA, Grasberger DM. Microcomputer‐generated reminders. Improving the compliance of primary care physicians with mammography screening guidelines. Journal of Family Practice 1989;29(3):273‐80. [PubMed] [Google Scholar]

Chambers 1991 {published data only}

Chambers CV, Balaban DJ, Carlson BL, Grasberger DM. The effect of microcomputer‐generated reminders on influenza vaccination rates in a university‐based family practice center. Journal of the American Board of Family Practice 1991;4(1):19‐26. [PubMed] [Google Scholar]

Dexter 1998 {published data only}

Dexter PR, Wolinsky FD, Gramelspacher GP, Zhou XH, Eckert GJ, Waisburd M, et al. Effectiveness of computer‐generated reminders for increasing discussions about advance directives and completion of advance directive forms. A randomized, controlled trial. Annals of Internal Medicine 1998;128(2):102‐10. [DOI] [PubMed] [Google Scholar]

Gilutz 2009 {published data only}

Gilutz H, Novack L, Shvartzman P, Zelingher J, Bonneh DY, Henkin Y, et al. Computerized community cholesterol control (4C): meeting the challenge of secondary prevention. Israel Medical Association Journal 2009;11(1):23‐9. [PUBMED: 19344008] [PubMed] [Google Scholar]

Heidenreich 2005 {published data only}

Heidenreich PA, Chacko M, Goldstein MK, Atwood JE. ACE inhibitor reminders attached to echocardiography reports of patients with reduced left ventricular ejection fraction. American Journal of Medicine 2005;118(9):1034‐7. [DOI] [PubMed] [Google Scholar]

Heidenreich 2007 {published data only}

Heidenreich PA, Gholami P, Sahay A, Massie B, Goldstein MK. Clinical reminders attached to echocardiography reports of patients with reduced left ventricular ejection fraction increase use of beta‐blockers: a randomized trial. Circulation 2007;115(22):2829‐34. [DOI] [PubMed] [Google Scholar]

Heiman 2004 {published data only}

Heiman H, Bates DW, Fairchild D, Shaykevich S, Lehmann LS. Improving completion of advance directives in the primary care setting: a randomized controlled trial. American Journal of Medicine 2004;117(5):318‐24. [DOI] [PubMed] [Google Scholar]

Javitt 2005 {published and unpublished data}

Javitt JC, Steinberg G, Locke T, Couch JB, Jacques J, Juster I, et al. Using a claims data‐based sentinel system to improve compliance with clinical guidelines: results of a randomized prospective study. American Journal of Managed Care 2005;11(2):93‐102. [PubMed] [Google Scholar]

Le Breton 2016 {published data only}

Breton J, Ferrat E, Attali C, Bercier S, Corvoisier P, Brixi Z, et al. Effect of reminders mailed to general practitioners on colorectal cancer screening adherence: a cluster‐randomized trial. European journal of cancer prevention : the official journal of the European Cancer Prevention Organisation (ECP) 2016;25(5):380‐7. [PUBMED: 26340058] [DOI] [PubMed] [Google Scholar]

Lobach 1997 {published data only}

Lobach DF, Hammond WE. Computerized decision support based on a clinical practice guideline improves compliance with care standards. American Journal of Medicine 1997;102(1):89‐98. [DOI] [PubMed] [Google Scholar]
Lobach DF, Hammond WE. Development and evaluation of a Computer‐Assisted Management Protocol (CAMP): improved compliance with care guidelines for diabetes mellitus. Proceedings / the Annual Symposium on Computer Applications in Medical Care 1994;‐:787‐91. [PMC free article] [PubMed] [Google Scholar]

Majumdar 2007 {published data only}

Majumdar SR, McAlister FA, Tsuyuki RT. A cluster randomized trial to assess the impact of opinion leader endorsed evidence summaries on improving quality of prescribing for patients with chronic cardiovascular disease: rationale and design [ISRCTN26365328]. BMC Cardiovascular Disorders 2005;5(1):17. [DOI] [PMC free article] [PubMed] [Google Scholar]
Majumdar SR, Tsuyuki RT, McAlister FA. Impact of opinion leader‐endorsed evidence summaries on the quality of prescribing for patients with cardiovascular disease: a randomized controlled trial. American Heart Journal 2007;153(1):22.e1‐8. [DOI] [PubMed] [Google Scholar]

Mazzuca 1990 {published data only}

Mazzuca SA, Vinicor F, Einterz RM, Tierney WM, Norton JA, Kalasinski LA. Effects of the clinical environment on physicians' response to postgraduate medical education. American Educational Research Journal 1990;27(3):473‐88. [Google Scholar]

McAlister 2009 {published data only}

McAlister FA, Fradette M, Graham M, Majumdar SR, Ghali WA, Williams R, et al. A randomized trial to assess the impact of opinion leader endorsed evidence summaries on the use of secondary prevention strategies in patients with coronary artery disease: the ESP‐CAD trial protocol [NCT00175240]. Implementation Science 2006;1:11. [DOI] [PMC free article] [PubMed] [Google Scholar]
McAlister FA, Fradette M, Majumdar SR, Williams R, Graham M, McMeekin J, et al. The enhancing secondary prevention in coronary artery disease trial. Canadian Medical Association Journal 2009;181(12):897‐904. [DOI] [PMC free article] [PubMed] [Google Scholar]

McDonald 1976a {published data only}

McDonald CJ. Protocol‐based computer reminders, the quality of care and the non‐perfectability of man. New England Journal of Medicine 1976;295(24):1351‐5. [DOI] [PubMed] [Google Scholar]

McDonald 1976b {published data only}

McDonald CJ. Use of a computer to detect and respond to clinical events: its effect on clinician behavior. Annals of Internal Medicine 1976;84(2):162‐7. [DOI] [PubMed] [Google Scholar]

McDonald 1980 {published data only}

McDonald CJ, Wilson GA, McCabe GP Jr. Physician response to computer reminders. JAMA 1980;244(14):1579‐81. [PubMed] [Google Scholar]

McDonald 1984 {published data only}

McDonald CJ, Hui SL, Smith DM, Tierney WM, Cohen SJ, Weinberger M, et al. Reminders to physicians from an introspective computer medical record. A two‐year randomized trial. Annals of Internal Medicine 1984;100(1):130‐8. [DOI] [PubMed] [Google Scholar]
McDonald CJ, Hui SL, Tierney WM. Effects of computer reminders for influenza vaccination on morbidity during influenza epidemics. MD Computing 1992;9(5):304‐12. [PubMed] [Google Scholar]

McPhee 1989 {published data only}

Bird JA, McPhee SJ, Jenkins C, Fordham D. Three strategies to promote cancer screening. How feasible is wide‐scale implementation?. Medical Care 1990;28(11):1005‐12. [DOI] [PubMed] [Google Scholar]
Fordham D, McPhee SJ, Bird JA, Rodnick JE, Detmer WM. The Cancer Prevention Reminder System. MD Computing 1990;7(5):289‐95. [PubMed] [Google Scholar]
McPhee SJ, Bird JA, Jenkins CN, Fordham D. Promoting cancer screening. A randomized, controlled trial of three interventions. Archives of Internal Medicine 1989;149(8):1866‐72. [DOI] [PubMed] [Google Scholar]

Morgan 1978 {published data only}

Morgan M, Studney DR, Barnett GO, Winickoff RN. Computerized concurrent review of prenatal care. QRB. Quality Review Bulletin 1978;4(9):33‐6. [PubMed] [Google Scholar]

Nilasena 1995 {published data only}

Nilasena DS, Lincoln MJ. A computer‐generated reminder system improves physician compliance with diabetes preventive care guidelines. Proceedings / the Annual Symposium on Computer Applications in Medical Care. 1995:640‐5. [PMC free article] [PubMed]
Nilasena DS, Lincoln MJ, Turner CW, Warner HR, Foerster VA, Williamson JW, et al. Development and implementation of a computer‐generated reminder system for diabetes preventive care. Proceedings / the Annual Symposium on Computer Application in Medical Care. 1994:831‐5. [PMC free article] [PubMed]

Oniki 2003 {published data only}

Oniki TA, Clemmer TP, Pryor TA. The effect of computer‐generated reminders on charting deficiencies in the ICU. Journal of the American Medical Informatics Association 2003;10(2):177‐87. [DOI] [PMC free article] [PubMed] [Google Scholar]

Ornstein 1991 {published data only}

Ornstein SM, Garr DR, Jenkins RG, Rust PF, Arnon A. Computer‐generated physician and patient reminders. Tools to improve population adherence to selected preventive services. Journal of Family Practice 1991;32(1):82‐90. [PubMed] [Google Scholar]

Rosser 1991 {published data only}

McDowell I, Newell C, Rosser W. A randomized trial of computerized reminders for blood pressure screening in primary care. Medical Care 1989;27(3):297‐305. [DOI] [PubMed] [Google Scholar]
McDowell I, Newell C, Rosser W. Comparison of three methods of recalling patients for influenza vaccination. Canadian Medical Association Journal 1986;135(9):991‐7. [PMC free article] [PubMed] [Google Scholar]
McDowell I, Newell C, Rosser W. Computerized reminders to encourage cervical screening in family practice. Journal of Family Practice 1989;28(4):420‐4. [PubMed] [Google Scholar]
Rosser WW, Hutchison BG, McDowell I, Newell C. Use of reminders to increase compliance with tetanus booster vaccination. Canadian Medical Association Journal 1992;146(6):911‐7. [PMC free article] [PubMed] [Google Scholar]
Rosser WW, McDowell I, Newell C. Use of reminders for preventive procedures in family medicine. Canadian Medical Association Journal 1991;145(7):807‐14. [PMC free article] [PubMed] [Google Scholar]

Rossi 1997 {published data only}

Rossi RA, Every NR. A computerized intervention to decrease the use of calcium channel blockers in hypertension. Journal of General Internal Medicine 1997;12(11):672‐8. [DOI] [PMC free article] [PubMed] [Google Scholar]

Thomas 1983 {published data only}

Thomas JC, Moore A, Qualls PE. The effect on cost of medical care for patients treated with an automated clinical audit system. Journal of Medical Systems 1983;7(3):307‐13. [DOI] [PubMed] [Google Scholar]

Tierney 1986 {published data only}

Tierney WM, Hui SL, McDonald CJ. Delayed feedback of physician performance versus immediate reminders to perform preventive care. Effects on physician compliance. Medical Care 1986;24(8):659‐66. [DOI] [PubMed] [Google Scholar]

Turner 1989 {published data only}

Turner BJ. A controlled trial of strategies to improve delivery of preventive care. Research in medical education:proceedings of the annual Conference. Conference on Research in Medical Education 1987;26:9‐13. [PubMed] [Google Scholar]
Turner BJ, Day SC, Borenstein B. A controlled trial to improve delivery of preventive care: physician or patient reminders?. Journal of General Internal Medicine 1989;4(5):403‐9. [DOI] [PubMed] [Google Scholar]

Were 2013 {published data only}

Were MC, Nyandiko WM, Huang KT, Slaven JE, Shen C, Tierney WM, et al. Computer‐generated reminders and quality of pediatric HIV care in a resource‐limited setting. Pediatrics 2013;131(3):e789‐96. [PUBMED: 23439898] [DOI] [PMC free article] [PubMed] [Google Scholar]

White 1984 {published data only}

White KS, Lindsay A, Pryor TA, Brown WF, Walsh K. Application of a computerized medical decision‐making process to the problem of digoxin intoxication. Journal of the American College of Cardiology 1984;4(3):571‐6. [DOI] [PubMed] [Google Scholar]

Ziemer 2006 {published and unpublished data}

Phillips LS, Hertzberg VS, Cook CB, El‐Kebbi IM, Gallina DL, Ziemer DC, et al. The Improving Primary Care of African Americans with Diabetes (IPCAAD) project: rationale and design. Controlled Clinical Trials 2002;23(5):554‐69. [DOI] [PubMed] [Google Scholar]
Phillips LS, Ziemer DC, Doyle JP, Barnes CS, Kolm P, Branch WT, et al. An endocrinologist‐supported intervention aimed at providers improves diabetes management in a primary care site: improving primary care of African Americans with diabetes (IPCAAD). Diabetes Care 2005;28(10):2352‐60. [DOI] [PubMed] [Google Scholar]
Ziemer DC, Doyle JP, Barnes CS, Branch WT Jr, Cook CB, El‐Kebbi IM, et al. An intervention to overcome clinical inertia and improve diabetes mellitus control in a primary care setting: Improving Primary Care of African Americans with Diabetes (IPCAAD). Archives of Internal Medicine 2006;166(5):507‐13. [DOI] [PubMed] [Google Scholar]

References to studies excluded from this review

Adelman 2013 {published data only}

Adelman JS, Kalkut GE, Schechter CB, Weiss JM, Berger MA, Reissman SH, et al. Understanding and preventing wrong‐patient electronic orders: a randomized controlled trial. Journal of the American Medical Informatics Association: JAMIA 2013;20(2):305‐10. [DOI] [PMC free article] [PubMed] [Google Scholar]

Alfadda 2011 {published data only}

Alfadda AA, Bin‐Abdulrahman KA, Saad HA, Mendoza CDO, Angkaya‐Bagayawa FF, Yale JF. Effect of an intervention to improve the management of patients with diabetes in primary care practice. Saudi Medical Journal 2011;32(1):36‐40. [PubMed] [Google Scholar]

Anabtawi 2013 {published data only}

Anabtawi A, Mathew LM. Improving compliance with screening of diabetic patients for microalbuminuria in primary care practice. ISRN Endocrinology 2013;2013:893913. [PUBMED: 24224095] [DOI] [PMC free article] [PubMed] [Google Scholar]

Anchala 2012 {published data only}

Anchala R, Pant H, Prabhakaran D, Franco OH. 'Decision support system (DSS) for prevention of cardiovascular disease (CVD) among hypertensive (HTN) patients in Andhra Pradesh, India'‐‐a cluster randomised community intervention trial. BMC Public Health 2012;12:393. [PUBMED: 22650767] [DOI] [PMC free article] [PubMed] [Google Scholar]

Barkun 2013 {published data only}

Barkun AN, Bhat M, Armstrong D, Dawes M, Donner A, Enns R, et al. Effectiveness of disseminating consensus management recommendations for ulcer bleeding: a cluster randomized trial. CMAJ: Canadian Medical Association Journal 2013;185(3):E156‐66. [DOI] [PMC free article] [PubMed] [Google Scholar]

Barnes 2014 {published data only}

Barnes ER, Theeke LA, Mallow J. Impact of the Provider and Healthcare team Adherence to Treatment Guidelines (PHAT‐G) intervention on adherence to national obesity clinical practice guidelines in a primary care centre. Journal of Evaluation in Clinical Practice 2014;21(2):300‐6. [DOI] [PMC free article] [PubMed] [Google Scholar]

Beeckman 2013 {published data only}

Beeckman D, Clays E, Hecke A, Vanderwee K, Schoonhoven L, Verhaeghe S. A multi‐faceted tailored strategy to implement an electronic clinical decision support system for pressure ulcer prevention in nursing homes: a two‐armed randomized controlled trial. Internation Journal of Nursing Studies 2013;50(4):475‐86. [DOI] [PubMed] [Google Scholar]

Beeler 2014 {published data only}

Beeler PE, Eschmann E, Schumacher A, Studt JD, Amann‐Vesti B, Blaser J. Impact of electronic reminders on venous thromboprophylaxis after admissions and transfers. Journal of the American Medical Informatics Association: JAMIA 2014;21(e2):e297‐303. [DOI] [PMC free article] [PubMed] [Google Scholar]

Belland 2014 {published data only}

Belland L, Rivera‐Reyes L, Genes N, Thum F, Winkel G, Soriano J, et al. The effect of a novel clinical decision support system on pain care for older adults presenting to the emergency department with acute abdominal pain. Annals of Emergency Medicine. 2014; Vol. 64, issue 4:S81.

Beste 2015 {published data only}

Beste LA, Ioannou GN, Yang Y, Chang MF, Ross D, Dominitz JA. Improved surveillance for hepatocellular carcinoma with a primary care‐oriented clinical reminder. Clinical Gastroenterology and Hepatology : the official clinical practice journal of the American Gastroenterological Association 2015;13(1):172‐9. [PUBMED: 24813175] [DOI] [PubMed] [Google Scholar]

Braun 2013 {published data only}

Braun R, Barandas A, Hancock‐Crear A. Improving access and addressing barriers to HPV vaccination for adolescents and young adults. Contraception. 2013; Vol. 88, issue 2:307.

Campbell 2014 {published data only}

Campbell F, Peleshok J, Yamada J, Krancevic A‐M, Pepper J, Dhaliwal M, et al. Impact of a quality improvement strategy for improving pain outcomes in a pediatric hospital setting. Pain Research and Management. 2014; Vol. 19, issue 3:e31.

Carroll 2013 {published data only}

Carroll AE, Biondich P, Anand V, Dugan TM, Downs SM. A randomized controlled trial of screening for maternal depression with a clinical decision support system. Journal of the American Medical Informatics Association: JAMIA 2013;20(2):311‐6. [DOI] [PMC free article] [PubMed] [Google Scholar]

Chen 2013 {published data only}

Chen YY, Chi MM, Chen YC, Chan YJ, Chou SS, Wang FD. Using a criteria‐based reminder to reduce use of indwelling urinary catheters and decrease urinary tract infections. American Journal of Critical Care 2013; Vol. 22, issue 2:105‐14. [DOI] [PubMed]

Dexheimer 2013 {published data only}

Dexheimer JW, Abramo TJ, Arnold DH, Johnson KB, Shyr Y, Ye F, et al. An asthma management system in a pediatric emergency department. International Journal of Medical Informatics 2013;82(4):230‐8. [DOI] [PMC free article] [PubMed] [Google Scholar]

Divinskiy 2015 {published data only}

Divinskiy T, Nemytova E, Liu X, Ajmal S. A quality improvement project increased documentation of advance directives among an older population in a primary care setting. Annual Scientific Meeting of the American Geriatrics Society. 2015:63.

dos Santos 2014 {published data only}

dos Santos MA, Tygesen H, Eriksson H, Herlitz J. Clinical decision support system (CDSS)‐‐effects on care quality. International Journal of Health Care Quality Assurance 2014;27(8):707‐18. [DOI] [PubMed] [Google Scholar]

Erlingsdottir 2015 {published data only}

Erlingsdottir H, Johannesson A, Asgeirsdottir TL. Can physician laboratory‐test requests be influenced by interventions?. Scandinavian Journal of Clinical and Laboratory Investigation 2015;75(1):18‐26. [DOI] [PubMed] [Google Scholar]

Federman 2014 {published data only}

Federman DG, Kravetz JD, Lerz KA, Akgun KM, Ruser C, Cain H, et al. Implementation of an electronic clinical reminder to improve rates of lung cancer screening. American Journal of Medicine 2014;127(9):813‐6. [DOI] [PubMed] [Google Scholar]

Gifford 2013 {published data only}

Gifford WA, Davies BL, Graham ID, Tourangeau A, Woodend AK, Lefebre N. Developing leadership capacity for guideline use: a pilot cluster randomized control trial. Worldviews on Evidence‐Based Nursing 2013;10(1):51‐65. [DOI] [PubMed] [Google Scholar]

Goldstein 2014 {published data only}

Goldstein NE, Kalman J, Kutner JS, Fromme EK, Hutchinson MD, Lipman HI, et al. A study to improve communication between clinicians and patients with advanced heart failure: Methods and challenges behind the working to improve discussions about defibrillator management trial. Journal of Pain and Symptom Management 2014;48(6):1236‐46. [DOI] [PMC free article] [PubMed] [Google Scholar]

Green 2014 {published data only}

Green BB, Anderson ML, Wang C‐Y, Vernon SW, Chubak J, Meenan RT, et al. Results of nurse navigator follow‐up after positive colorectal cancer screening test: A randomized trial. Journal of the American Board of Family Medicine 2014;27(6):789‐95. [DOI] [PMC free article] [PubMed] [Google Scholar]

Gupta 2014 {published data only}

Gupta A, Gholami P, Turakhia MP, Friday K, Heidenreich PA. Clinical reminders to providers of patients with reduced left ventricular ejection fraction increase defibrillator referral: a randomized trial. Circulation Heart Failure 2014;7(1):140‐5. [DOI] [PubMed] [Google Scholar]

Hendrix 2015 {published data only}

Hendrix KS, Downs SM, Carroll AE. Pediatricians' responses to printed clinical reminders: does highlighting prompts improve responsiveness?. Academic Pediatrics 2015;15(2):158‐64. [DOI] [PMC free article] [PubMed] [Google Scholar]

Hye 2014 {published data only}

Hye RJ, Smith AE, Wong GH, Vansomphone SS, Scott RD, Kanter MH. Leveraging the electronic medical record to implement an abdominal aortic aneurysm screening program. Journal of Vascular Surgery 2014;59(6):1535‐42. [DOI] [PubMed] [Google Scholar]

Jansink 2013 {published data only}

Jansink R, Braspenning J, Keizer E, Weijden T, Elwyn G, Grol R. No identifiable Hb1Ac or lifestyle change after a comprehensive diabetes programme including motivational interviewing: a cluster randomised trial. Scandinavian Journal of Primary Health Care 2013;31(2):119‐27. [DOI] [PMC free article] [PubMed] [Google Scholar]

Kennedy 2012 {published data only}

Kennedy CC, Ioannidis G, Giangregorio LM, Adachi JD, Thabane L, Morin SN, et al. An interdisciplinary knowledge translation intervention in long‐term care: study protocol for the vitamin D and osteoporosis study (Original Title; URL links to DARE Abstract) pilot cluster randomized controlled trial. Implementation Science 2012;7:48. [DOI] [PMC free article] [PubMed] [Google Scholar]

Kousgaard 2013 {published data only}

Kousgaard MB, Siersma V, Reventlow S, Ertmann R, Felding P, Waldorff FB. The effectiveness of computer reminders for improving quality assessment for point‐of‐care testing in general practice‐‐a randomized controlled trial. Implementation Science 2013;8:47. [DOI] [PMC free article] [PubMed] [Google Scholar]

Lai 2015 {published data only}

Lai CL, Chan HY, Pan YJ, Chen CH. The effectiveness of a computer reminder system for laboratory monitoring of metabolic syndrome in schizophrenic outpatients using second‐generation antipsychotics. Pharmacopsychiatry 2015;48(1):25‐9. [DOI] [PubMed] [Google Scholar]

Levy 2013 {published data only}

Levy BT, Xu Y, Daly JM, Ely JW. A randomized controlled trial to improve colon cancer screening in rural family medicine: an Iowa Research Network (IRENE) study. Journal of the American Board of Family Medicine 2013;26(5):486‐97. [DOI] [PubMed] [Google Scholar]

Lusignan 2013 {published data only}

Lusignan S, Gallagher H, Jones S, Chan T, Vlymen J, Tahir A, et al. Audit‐based education lowers systolic blood pressure in chronic kidney disease: the Quality Improvement in CKD (QICKD) trial results. Kidney International 2013;84(3):609‐20. [DOI] [PMC free article] [PubMed] [Google Scholar]

Lynn 2013 {published data only}

Lynn LA, Guerra CE, Ross KM, Holmboe E, Woo K, Heitjan DF, et al. A randomized controlled trial of an evidence‐based toolbox and guide to increase primary care clinicians' rates of colorectal cancer screening in diplomates of the ABIM. Journal of General Internal Medicine. 2013; Vol. 28, issue Suppl 1.

Majumdar 2012 {published data only}

Majumdar SR, McAlister FA, Johnson JA, Bellerose D, Siminoski K, Hanley DA, et al. Interventions to increase osteoporosis treatment in patients with 'incidentally' detected vertebral fractures. American Journal of Medicine 2012;125(9):929‐36. [DOI] [PubMed] [Google Scholar]

Maximov 2013 {published data only}

Maximov D, Lesnyak O. Implementation of the osteoarthritis clinical guideline: Results of a cluster randomized trial in primary care. Annals of the Rheumatic Diseases. 2013; Vol. 71:307‐8.

McEvoy 2014 {published data only}

McEvoy MD, Hand WR, Stoll WD, Furse CM, Nietert PJ. Adherence to guidelines for the management of local anesthetic systemic toxicity is improved by an electronic decision support tool and designated "Reader". Regional Anesthesia 2014;39(4):299‐305. [DOI] [PMC free article] [PubMed] [Google Scholar]

McNulty 2014 {published data only}

McNulty CA, Hogan AH, Ricketts EJ, Wallace L, Oliver I, Campbell R, et al. Increasing chlamydia screening tests in general practice: a modified Zelen prospective Cluster Randomised Controlled Trial evaluating a complex intervention based on the Theory of Planned Behaviour. Sexually Transmitted Infections 2014;90(3):188‐94. [PUBMED: 24005256] [DOI] [PMC free article] [PubMed] [Google Scholar]

Melo 2013 {published data only}

Melo Rodrigues R, Fontes AMS, Cesar Mantese O, Souza Martins R, Tanus Jorge M. Impact of an intervention in the use of sequential antibiotic therapy in a Brazilian university hospital. Revista da Sociedade Brasileira de Medicina Tropical 2013;46(1):50‐4. [DOI] [PubMed] [Google Scholar]

Neal 2012 {published data only}

Neal JM, Hsiung RL, Mulroy MF, Halpern BB, Dragnich AD, Slee AE. ASRA checklist improves trainee performance during a simulated episode of local anesthetic systemic toxicity. Regional Anesthesia and Pain Medicine 2012;37(1):8‐15. [DOI] [PubMed] [Google Scholar]

Nguyen 2014 {published data only}

Nguyen MC, Richardson DM, Hardy SG, Cookson RM, Mackenzie RS, Greenberg MR, et al. Computer‐based reminder system effectively impacts physician documentation. American Journal of Emergency Medicine 2014;32(1):104‐6. [DOI] [PubMed] [Google Scholar]

O'Reilly 2012 {published data only}

O'Reilly D, Holbrook A, Blackhouse G, Troyan S, Goeree R. Cost‐effectiveness of a shared computerized decision support system for diabetes linked to electronic medical records. Journal of the American Medical Informatics Association: JAMIA 2012;19(3):341‐5. [DOI] [PMC free article] [PubMed] [Google Scholar]

Persell 2016 {published data only}

Persell SD, Doctor JN, Friedberg MW, Meeker D, Friesema E, Cooper A, et al. Behavioral interventions to reduce inappropriate antibiotic prescribing: a randomized pilot trial. BMC Infectious Diseases 2016;16:373. [PUBMED: 27495917] [DOI] [PMC free article] [PubMed] [Google Scholar]

Piazza 2013 {published data only}

Piazza G, Anderson FA, Ortel TL, Cox MJ, Rosenberg DJ, Rahimian S, et al. Randomized trial of physician alerts for thromboprophylaxis after discharge. American Journal of Medicine 2013;126(5):435‐42. [DOI] [PubMed] [Google Scholar]

Roy 2016 {published data only}

Roy PM, Rachas A, Meyer G, Gal G, Durieux P, Kouri D, et al. Multifaceted Intervention to Prevent Venous Thromboembolism in Patients Hospitalized for Acute Medical Illness: A Multicenter Cluster‐Randomized Trial. PloS One 2016;11(5):e0154832. [PUBMED: 27227406] [DOI] [PMC free article] [PubMed] [Google Scholar]

Schwalm 2015 {published data only}

Schwalm JD, Ivers NM, Natarajan MK, Taljaard M, Rao‐Melacini P, Witteman HO, et al. Cluster randomized controlled trial of Delayed Educational Reminders for Long‐term Medication Adherence in ST‐Elevation Myocardial Infarction (DERLA‐STEMI). American Heart Journal 2015;170(5):903‐13. [PUBMED: 26542498] [DOI] [PubMed] [Google Scholar]

Siersma 2015 {published data only}

Siersma V, Kousgaard MB, Reventlow S, Ertmann R, Felding P, Waldorff FB. The effectiveness of computer reminders versus postal reminders for improving quality assessment for point‐of‐care testing in primary care: a randomized controlled trial. Journal of Evaluation in Clinical Practice 2015;21(1):13‐20. [DOI] [PubMed] [Google Scholar]

Stockwell 2015 {published data only}

Stockwell MS, Catallozzi M, Camargo S, Ramakrishnan R, Holleran S, Findley SE, et al. Registry‐linked electronic influenza vaccine provider reminders: a cluster‐crossover trial. Pediatrics 2015;135(1):e75‐82. [DOI] [PubMed] [Google Scholar]

Szilagyi 2015 {published data only}

Szilagyi PG, Serwint JR, Humiston SG, Rand CM, Schaffer S, Vincelli P, et al. Effect of provider prompts on adolescent immunization rates: a randomized trial. Academic Pediatrics 2015;15(2):149‐57. [DOI] [PMC free article] [PubMed] [Google Scholar]

Tartaglia 2013 {published data only}

Tartaglia KM, Walker C, Heacock A, Kuofie I, Liston B, West J, et al. Antibiotic timeout program: Implementing a hospital stewardship intervention. Journal of General Internal Medicine 2013;28:S429‐30. [Google Scholar]

Tedja 2014 {published data only}

Tedja R, Hassebroek E, Carrera P, Thakur L, Mielke C, Volkman T, et al. Reducing urinary catheter days and utilization in a chronic ventilator dependent unit: Mayo clinic quality improvement experience. Chest 2014;146:556A. [Google Scholar]

Teoh 2012 {published data only}

Teoh L, Latif M, Choi P, Wu S, Chan CY, Plowman G, et al. Use of a reminder sticker improves rates of documentation of resuscitation status and the appropriate prescription of venous thromboembolism prophylaxis. New Zealand Medical Journal 2012;125(1357):175‐7. [PubMed] [Google Scholar]

Weiss 2013 {published data only}

Weiss CH, Dibardino D, Rho J, Sung N, Collander B, Wunderink RG. A clinical trial comparing physician prompting with an unprompted automated electronic checklist to reduce empirical antibiotic utilization. Critical Care Medicine 2013;41(11):2563‐9. [DOI] [PMC free article] [PubMed] [Google Scholar]

Were 2011 {published data only}

Were MC, Shen C, Tierney WM, Mamlin JJ, Biondich PG, Li X, et al. Evaluation of computer‐generated reminders to improve CD4 laboratory monitoring in sub‐Saharan Africa: a prospective comparative study. Journal of the American Medical Informatics Association : JAMIA 2011;18(2):150‐5. [PUBMED: 21252053] [DOI] [PMC free article] [PubMed] [Google Scholar]

Additional references

Anderson 2007

Anderson JG. Social, ethical and legal barriers to e‐health. International Journal of Medical Informatics 2007;76(5‐6):480‐3. [PUBMED: 17064955] [DOI] [PubMed] [Google Scholar]

Axt‐Adam 1993

Axt‐Adam P, Wouden JC, Does E. Influencing behavior of physicians ordering laboratory tests: a literature study. Medical Care 1993;31:784‐94. [DOI] [PubMed] [Google Scholar]

Baker, 2015

Baker R, Camosso‐Stefinovic J, Gillies C, Shaw EJ, Cheater F, Flottorp S, et al. Tailored interventions to address determinants of practice. Cochrane Database of Systematic Reviews 2015, Issue 4. [DOI: 10.1002/14651858.CD005470.pub3] [DOI] [PMC free article] [PubMed] [Google Scholar]

Balas 2000

Balas EA, Weingarten S, Garb CT, Blumenthal D, Boren SA, Brown GD. Improving preventive care by prompting physicians. Archives of Internal Medicine 2000;160(3):301‐8. [PUBMED: 10668831] [DOI] [PubMed] [Google Scholar]

Balas 2004

Balas EA, Krishna S, Kretschmer RA, Cheek TR, Lobach DF, Boren SA. Computerized knowledge management in diabetes care. Medical Care 2004;42(6):610‐21. [PUBMED: 15167329] [DOI] [PubMed] [Google Scholar]

Baron 2010

Baron RC, Melillo S, Rimer BK, Coates RJ, Kerner J, Habarta N, et al. Intervention to increase recommendation and delivery of screening for breast, cervical, and colorectal cancers by healthcare providers a systematic review of provider reminders. American Journal of Preventive Medicine 2010;38(1):110‐7. [PUBMED: 20117566] [DOI] [PubMed] [Google Scholar]

Bennett 2003

Bennett JW, Glasziou PP. Computerised reminders and feedback in medication management: a systematic review of randomised controlled trials. Medical Journal of Australia 2003;178(5):217‐22. [PUBMED: 12603185] [DOI] [PubMed] [Google Scholar]

Biau 2008

Biau DJ, Porcher R, Boutron I. The account for provider and center effects in multicenter interventional and surgical randomized controlled trials is in need of improvement: a review. Journal of Clinical Epidemiology 2008;61(5):435‐9. [PUBMED: 18394535] [DOI] [PubMed] [Google Scholar]

Buntinx 1993

Buntinx F, Winkens R, Grol R, Knottnerus JA. Influencing diagnostic and preventive performance in ambulatory care by feedback and reminders. A review. Family Practice 1993;10(2):219‐28. [DOI] [PubMed] [Google Scholar]

Campbell 2007

Campbell MJ, Donner A, Klar N. Developments in cluster randomized trials and Statistics in Medicine. Statistics in Medicine 2007;26(1):2‐19. [PUBMED: 17136746] [DOI] [PubMed] [Google Scholar]

Craig 2008

Craig P, Dieppe P, Macintyre S, Michie S, Nazareth I, Petticrew M. Developing and evaluating complex interventions: the new Medical Research Council guidance. BMJ (Clinical Research ed.) 2008;337:a1655. [PUBMED: 18824488] [DOI] [PMC free article] [PubMed] [Google Scholar]

Davis 1992

Davis DA, Thomson MA, Oxman AD, Haynes RB. Evidence for the effectiveness of CME. A review of 50 randomized controlled trials. JAMA 1992;268(9):1111‐7. [PubMed] [Google Scholar]

Davis 1995

Davis DA, Thomson MA, Oxman AD, Haynes RB. Changing physician performance. A systematic review of the effect of continuing medical education strategies. JAMA 1995;274(9):700‐5. [DOI] [PubMed] [Google Scholar]

Dexheimer 2008

Dexheimer JW, Talbot TR, Sanders DL, Rosenbloom ST, Aronsky D. Prompting clinicians about preventive care measures: a systematic review of randomized controlled trials. Journal of the American Medical Informatics Association 2008;15(3):311‐20. [PUBMED: 18308989] [DOI] [PMC free article] [PubMed] [Google Scholar]

EPOC 2015a

Effective Practice, Organisation of Care (EPOC). Suggested risk of bias criteria for EPOC reviews. epoc.cochrane.org/epoc‐specific‐resources‐review‐authors. Oslo: Norwegian Knowledge Centre for the Health Services, (accessed 1 July 2016).

EPOC 2015b

Effective Practice, Organisation of Care (EPOC). What study designs should be included in an EPOC review and what should they be called?. epoc.cochrane.org/epoc‐specific‐resources‐review‐authors. Oslo: Norwegian Knowledge Centre for the Health Services, (accessed 1 July 2016).

EPOC 2015c

Effective Practice, Organisation of Care (EPOC). Worksheets for preparing Summary of Findings tables using GRADE. epoc.cochrane.org/epoc‐specific‐resources‐review‐authors. Oslo: Norwegian Knowledge Centre for the Health Services, (accessed 1 July 2016).

EPOC 2015d

Effective Practice, Organisation of Care (EPOC). What outcomes should be reported in EPOC reviews?. epoc.cochrane.org/epoc‐specific‐resources‐review‐authors. Oslo: Norwegian Knowledge Centre for the Health Services, (accessed 1 July 2016).

Fiander 2015

Fiander M, McGowan J, Grad R, Pluye P, Hannes K, Labrecque M, et al. Interventions to increase the use of electronic health information by healthcare practitioners to improve clinical practice and patient outcomes. Cochrane Database of Systematic Reviews 2015, Issue 3. [DOI: 10.1002/14651858.CD004749.pub3] [DOI] [PMC free article] [PubMed] [Google Scholar]

Flodgren 2011

Flodgren G, Parmelli E, Doumit G, Gattellari M, O'Brien MA, Grimshaw J, et al. Local opinion leaders: effects on professional practice and health care outcomes. Cochrane Database of Systematic Reviews 2011, Issue 8. [DOI: 10.1002/14651858.CD000125.pub4] [DOI] [PMC free article] [PubMed] [Google Scholar]

Forsetlund 2009

Forsetlund L, Bjorndal A, Rashidian A, Jamtvedt G, O'Brien MA, Wolf F, et al. Continuing education meetings and workshops: effects on professional practice and health care outcomes. Cochrane Database of Systematic Reviews 2009, Issue 2. [DOI: 10.1002/14651858.CD003030.pub2] [DOI] [PMC free article] [PubMed] [Google Scholar]

Gagnon 2009

Gagnon MP, Legare F, Labrecque M, Fremont P, Pluye P, Gagnon J, et al. Interventions for promoting information and communication technologies adoption in healthcare professionals. Cochrane Database of Systematic Reviews 2009, Issue 1. [DOI: 10.1002/14651858.CD006093.pub2] [DOI] [PMC free article] [PubMed] [Google Scholar]

Garg 2005

Garg AX, Adhikari NK, McDonald H, Rosas‐Arellano MP, Devereaux PJ, Beyene J, et al. Effects of computerized clinical decision support systems on practitioner performance and patient outcomes: a systematic review. JAMA 2005;293(10):1223‐38. [PUBMED: 15755945] [DOI] [PubMed] [Google Scholar]

Giguère 2012

Giguère A, Légaré F, Grimshaw J, Turcotte S, Fiander M, Grudniewicz A, et al. Printed educational materials: effects on professional practice and healthcare outcomes. Cochrane Database of Systematic Reviews 2012, Issue 10. [DOI: 10.1002/14651858.CD004398.pub3] [DOI] [PMC free article] [PubMed] [Google Scholar]

GRADEpro GDT 2015 [Computer program]

GRADE Working Group, McMaster University. GRADEpro GDT. Version accessed 10 August 2015. Hamilton (ON): GRADE Working Group, McMaster University, 2014.

Grimshaw 2004

Grimshaw JM, Thomas RE, MacLennan G, Fraser C, Ramsay CR, Vale L, et al. Effectiveness and efficiency of guideline dissemination and implementation strategies. Health Technology Assessment (Winchester, England) 2004;8(6):iii‐iv, 1‐72. [PUBMED: 14960256] [DOI] [PubMed] [Google Scholar]

Guyatt 2011

Guyatt GH, Oxman AD, Schunemann HJ, Tugwell P, Knottnerus A. GRADE guidelines: a new series of articles in the Journal of Clinical Epidemiology. Journal of Clinical Epidemiology 2011;64(4):380‐2. [PUBMED: 21185693] [DOI] [PubMed] [Google Scholar]

Haynes 1987

Haynes RB, Walker CJ. Computer‐aided quality assurance. A critical appraisal. Archives of Internal Medicine 1987;147(7):1297‐301. [PubMed] [Google Scholar]

Higgins 2011

Higgins, JPT, Green, S. Cochrane Handbook for Systematic Reviews of Interventions Version. Version 5.1.0. Cochrane Handbook for Systematic Reviews of Interventions. Chichester: John Wiley & Sons, 2011. [Google Scholar]

Hoffmann 2014

Hoffmann TC, Glasziou PP, Boutron I, Milne R, Perera R, Moher D, et al. Better reporting of interventions: template for intervention description and replication (TIDieR) checklist and guide. BMJ (Clinical Research ed.) 2014;348:g1687. [PUBMED: 24609605] [DOI] [PubMed] [Google Scholar]

Hunt 1998

Hunt DL, Haynes RB, Hanna SE, Smith K. Effects of computer‐based clinical decision support systems on physician performance and patient outcomes: a systematic review. JAMA 1998;280(15):1339‐46. [PUBMED: 9794315] [DOI] [PubMed] [Google Scholar]

Ivers 2012

Ivers N, Jamtvedt G, Flottorp S, Young JM, Odgaard‐Jensen J, French SD, et al. Audit and feedback: effects on professional practice and healthcare outcomes. Cochrane Database of Systematic Reviews 2012;6:CD000259. [DOI: 10.1002/14651858.CD000259.pub3] [DOI] [PMC free article] [PubMed] [Google Scholar]

Jha 2009

Jha AK, DesRoches CM, Campbell EG, Donelan K, Rao SR, Ferris TG, et al. Use of electronic health records in U.S. hospitals. The New England Journal of Medicine 2009;360(16):1628‐38. [PUBMED: 19321858] [DOI] [PubMed] [Google Scholar]

Johnston 1994

Johnston ME, Langton KB, Haynes RB, Mathieu A. Effects of computer‐based clinical decision support systems on clinician performance and patient outcome. Annals of Internal Medicine 1994;120(2):135‐42. [DOI] [PubMed] [Google Scholar]

Kahan 2013

Kahan BC, Morris TP. Assessing potential sources of clustering in individually randomised trials. BMC Medical Research Methodology 2013;13:58. [PUBMED: 23590245] [DOI] [PMC free article] [PubMed] [Google Scholar]

Kawamoto 2005

Kawamoto K, Houlihan CA, Balas EA, Lobach DF. Improving clinical practice using clinical decision support systems: a systematic review of trials to identify features critical to success. BMJ 2005;330(7494):765. [PUBMED: 15767266] [DOI] [PMC free article] [PubMed] [Google Scholar]

Keogh‐Brown 2007

Keogh‐Brown MR, Bachmann MO, Shepstone L, Hewitt C, Howe A, Ramsay CR, et al. Contamination in trials of educational interventions. Health Technology Assessment (Winchester, England) 2007;11(43):iii, ix‐107. [PUBMED: 17935683] [DOI] [PubMed] [Google Scholar]

Litzelman 1993

Litzelman DK, Dittus RS, Miller ME, Tierney WM. Requiring physicians to respond to computerized reminders improves their compliance with preventive care protocols. Journal of General Internal Medicine 1993;8(6):311‐7. [DOI] [PubMed] [Google Scholar]

Mahmud 2010

Mahmud N, Rodriguez J, Nesbit J. A text message‐based intervention to bridge the healthcare communication gap in the rural developing world. Technology and Health Care : official journal of the European Society for Engineering and Medicine 2010;18(2):137‐44. [PUBMED: 20495253] [DOI] [PubMed] [Google Scholar]

Mandelblatt 1995

Mandelblatt J, Kanetsky PA. Effectiveness of interventions to enhance physician screening for breast cancer. Journal of Family Practice 1995;40(2):162‐71. [PubMed] [Google Scholar]

McDonald 1976

McDonald CJ. Protocol‐based computer reminders, the quality of care and the non‐perfectability of man. New England Journal of Medicine 1976;295(24):1351‐5. [DOI] [PubMed] [Google Scholar]

Moher 2010

Moher D, Hopewell S, Schulz KF, Montori V, Gotzsche PC, Devereaux PJ, et al. CONSORT 2010 explanation and elaboration: updated guidelines for reporting parallel group randomised trials. BMJ 2010;340:c869. [PUBMED: 20332511] [DOI] [PMC free article] [PubMed] [Google Scholar]

Mohler 2015

Mohler R, Kopke S, Meyer G. Criteria for Reporting the Development and Evaluation of Complex Interventions in healthcare: revised guideline (CReDECI 2). Trials 2015;16:204. [PUBMED: 25935741] [DOI] [PMC free article] [PubMed] [Google Scholar]

Mollon 2009

Mollon B, Chong J Jr, Holbrook AM, Sung M, Thabane L, Foster G. Features predicting the success of computerized decision support for prescribing: a systematic review of randomized controlled trials. BMC Medical Informatics and Decision Making 2009;9:11. [PUBMED: 19210782] [DOI] [PMC free article] [PubMed] [Google Scholar]

Morris 2002

Morris AH. Decision support and safety of clinical environments. Quality & Safety in Health Care 2002;11(1):69‐75. [PUBMED: 12078374] [DOI] [PMC free article] [PubMed] [Google Scholar]

Ndiaye 2005

Ndiaye SM, Hopkins DP, Shefer AM, Hinman AR, Briss PA, Rodewald L, et al. Interventions to improve influenza, pneumococcal polysaccharide, and hepatitis B vaccination coverage among high‐risk adults: a systematic review. American Journal of Preventive Medicine 2005;28(5 Suppl):248‐79. [PUBMED: 15894160] [DOI] [PubMed] [Google Scholar]

O'Brien 2007

O'Brien MA, Rogers S, Jamtvedt G, Oxman AD, Odgaard‐Jensen J, Kristoffersen DT, et al. Educational outreach visits: effects on professional practice and health care outcomes. Cochrane Database of Systematic Reviews 2007, Issue 4. [DOI: 10.1002/14651858.CD000409.pub2; PUBMED: 17943742] [DOI] [PMC free article] [PubMed] [Google Scholar]

Ogrinc 2016

Ogrinc G, Davies L, Goodman D, Batalden P, Davidoff F, Stevens D. SQUIRE 2.0 (Standards for QUality Improvement Reporting Excellence): revised publication guidelines from a detailed consensus process. BMJ Quality & Safety 2016;25(12):986‐92. [PUBMED: 26369893] [DOI] [PMC free article] [PubMed] [Google Scholar]

Oxman 1995

Oxman AD, Thomson MA, Davis DA, Haynes RB. No magic bullets: a systematic review of 102 trials of interventions to improve professional practice. Canadian Medical Association Journal 1995;153(10):1423‐31. [PMC free article] [PubMed] [Google Scholar]

Pantoja 2014

Pantoja T, Green ME, Grimshaw J, Denig P, Durieux P, Gill P, et al. Manual paper reminders: effects on professional practice and health care outcomes. Cochrane Database of Systematic Reviews 2014, Issue 9. [DOI: 10.1002/14651858.CD001174.pub2; CD001174] [DOI] [Google Scholar]

Pearson 2009

Pearson SA, Moxey A, Robertson J, Hains I, Williamson M, Reeve J, et al. Do computerised clinical decision support systems for prescribing change practice? A systematic review of the literature (1990‐2007). BMC Health Services Research 2009;9:154. [PUBMED: 19715591] [DOI] [PMC free article] [PubMed] [Google Scholar]

Schedlbauer 2009

Schedlbauer A, Prasad V, Mulvaney C, Phansalkar S, Stanton W, Bates DW, et al. What evidence supports the use of computerized alerts and prompts to improve clinicians' prescribing behavior?. Journal of the American Medical Informatics Association 2009;16(4):531‐8. [PUBMED: 19390110] [DOI] [PMC free article] [PubMed] [Google Scholar]

Shea 1996

Shea S, DuMouchel W, Bahamonde L. A meta‐analysis of 16 randomized controlled trials to evaluate computer‐based clinical reminder systems for preventive care in the ambulatory setting. Journal of the American Medical Informatics Association 1996;3(6):399‐409. [PUBMED: 8930856] [DOI] [PMC free article] [PubMed] [Google Scholar]

Shiffman 1999

Shiffman RN, Liaw Y, Brandt CA, Corb GJ. Computer‐based guideline implementation systems: a systematic review of functionality and effectiveness. Journal of the American Medical Informatics Association : JAMIA 1999;6(2):104‐14. [PUBMED: 10094063] [DOI] [PMC free article] [PubMed] [Google Scholar]

Shojania 2009

Shojania KG, Jennings A, Mayhew A, Ramsay CR, Eccles MP, Grimshaw J. The effects of on‐screen, point of care computer reminders on processes and outcomes of care. Cochrane Database of Systematic Reviews 2009, Issue 3. [DOI: 10.1002/14651858.CD001096.pub2] [DOI] [PMC free article] [PubMed] [Google Scholar]

Siddiqi 2005

Siddiqi K, Newell J, Robinson M. Getting evidence into practice: what works in developing countries?. International Journal for Quality in Health Care : journal of the International Society for Quality in Health Care ISQua 2005;17(5):447‐54. [PUBMED: 15872024] [DOI] [PubMed] [Google Scholar]

Stata 2007 [Computer program]

StataCorp. Stata Statistical Software: Release 10. College Station, TX: StataCorp LP, 2007.

Thomas 1999

Thomas LH, Cullum NA, McColl E, Rousseau N, Soutter J, Steen N. Guidelines in professions allied to medicine. Cochrane Database of Systematic Reviews 1999, Issue 1. [DOI: 10.1002/14651858.CD000349] [DOI] [PubMed] [Google Scholar]

Tierney 2007

Tierney WM, Rotich JK, Hannan TJ, Siika AM, Biondich PG, Mamlin BW, et al. The AMPATH medical record system: creating, implementing, and sustaining an electronic medical record system to support HIV/AIDS care in western Kenya. Studies in Health Technology and Informatics 2007;129(Pt 1):372‐6. [PUBMED: 17911742] [PubMed] [Google Scholar]

Ukoumunne 1999

Ukoumunne OC, Gulliford MC, Chinn S, Sterne JA, Burney PG, Donner A. Methods in health service research. Evaluation of health interventions at area and organisation level. BMJ (Clinical Research ed.) 1999;319(7206):376‐9. [PUBMED: 10435968] [DOI] [PMC free article] [PubMed] [Google Scholar]

Wensing 1994

Wensing M, Grol R. Single and combined strategies for implementing changes in primary care: a literature review. International Journal for Quality in Health Care 1994;6(2):115‐32. [DOI] [PubMed] [Google Scholar]

References to other published versions of this review

Arditi 2010

Arditi C, Rège‐Walther M, Burnand B, Wyatt J. Computer‐generated paper reminders: effects on professional practice and health care outcomes. Cochrane Database of Systematic Reviews 2010, Issue 5. [DOI: 10.1002/14651858.CD001175.pub2] [DOI] [PubMed] [Google Scholar]

Arditi 2012

Arditi C, Rège‐Walther M, Wyatt JC, Durieux P, Burnand B. Computer‐generated reminders delivered on paper to healthcare professionals; effects on professional practice and health care outcomes. Cochrane Database of Systematic Reviews 2012, Issue 12. [DOI: 10.1002/14651858.CD001175.pub3] [DOI] [PubMed] [Google Scholar]

Gorman 1998

Gorman P, Redfern C, Liaw T, Carson S, Wyatt JC, Rowe RE, et al. Computer‐generated paper reminders: effects on professional practice and health care outcomes. Cochrane Database of Systematic Reviews 1998, Issue 3. [DOI: 10.1002/14651858.CD001175] [DOI] [Google Scholar]

[CD001175-bib-0001] Barnett GO, Winickoff RN, Morgan MM, Zielstorff RD. A computer‐based monitoring system for follow‐up of elevated blood pressure. Medical Care 1983;21(4):400‐9. [DOI] [PubMed] [Google Scholar]

[CD001175-bib-0002] Becker DM, Gomez EB, Kaiser DL, Yoshihasi A, Hodge RH Jr. Improving preventive care at a medical clinic: how can the patient help?. American Journal of Preventive Medicine 1989;5(6):353‐9. [PubMed] [Google Scholar]

[CD001175-bib-0003] Binstock MA, Geiger AM, Hackett JR, Yao JF. Pap smear outreach: a randomized controlled trial in an HMO. American Journal of Preventive Medicine 1997;13(6):425‐6. [PubMed] [Google Scholar]

[CD001175-bib-0004] Burack RC, Gimotty PA, George J, Simon MS, Dews P, Moncrease A. The effect of patient and physician reminders on use of screening mammography in a health maintenance organization. Results of a randomized controlled trial. Cancer 1996;78(8):1708‐21. [PubMed] [Google Scholar]

[CD001175-bib-0005] Burack RC, Gimotty PA, George J, McBride S, Moncrease A, Simon MS, et al. How reminders given to patients and physicians affected pap smear use in a health maintenance organization: results of a randomized controlled trial. Cancer 1998;82(12):2391‐400. [DOI] [PubMed] [Google Scholar]

[CD001175-bib-0006] Chambers CV, Balaban DJ, Carlson BL, Ungemack JA, Grasberger DM. Microcomputer‐generated reminders. Improving the compliance of primary care physicians with mammography screening guidelines. Journal of Family Practice 1989;29(3):273‐80. [PubMed] [Google Scholar]

[CD001175-bib-0007] Chambers CV, Balaban DJ, Carlson BL, Grasberger DM. The effect of microcomputer‐generated reminders on influenza vaccination rates in a university‐based family practice center. Journal of the American Board of Family Practice 1991;4(1):19‐26. [PubMed] [Google Scholar]

[CD001175-bib-0008] Dexter PR, Wolinsky FD, Gramelspacher GP, Zhou XH, Eckert GJ, Waisburd M, et al. Effectiveness of computer‐generated reminders for increasing discussions about advance directives and completion of advance directive forms. A randomized, controlled trial. Annals of Internal Medicine 1998;128(2):102‐10. [DOI] [PubMed] [Google Scholar]

[CD001175-bib-0009] Gilutz H, Novack L, Shvartzman P, Zelingher J, Bonneh DY, Henkin Y, et al. Computerized community cholesterol control (4C): meeting the challenge of secondary prevention. Israel Medical Association Journal 2009;11(1):23‐9. [PUBMED: 19344008] [PubMed] [Google Scholar]

[CD001175-bib-0010] Heidenreich PA, Chacko M, Goldstein MK, Atwood JE. ACE inhibitor reminders attached to echocardiography reports of patients with reduced left ventricular ejection fraction. American Journal of Medicine 2005;118(9):1034‐7. [DOI] [PubMed] [Google Scholar]

[CD001175-bib-0011] Heidenreich PA, Gholami P, Sahay A, Massie B, Goldstein MK. Clinical reminders attached to echocardiography reports of patients with reduced left ventricular ejection fraction increase use of beta‐blockers: a randomized trial. Circulation 2007;115(22):2829‐34. [DOI] [PubMed] [Google Scholar]

[CD001175-bib-0012] Heiman H, Bates DW, Fairchild D, Shaykevich S, Lehmann LS. Improving completion of advance directives in the primary care setting: a randomized controlled trial. American Journal of Medicine 2004;117(5):318‐24. [DOI] [PubMed] [Google Scholar]

[CD001175-bib-0013] Javitt JC, Steinberg G, Locke T, Couch JB, Jacques J, Juster I, et al. Using a claims data‐based sentinel system to improve compliance with clinical guidelines: results of a randomized prospective study. American Journal of Managed Care 2005;11(2):93‐102. [PubMed] [Google Scholar]

[CD001175-bib-0014] Breton J, Ferrat E, Attali C, Bercier S, Corvoisier P, Brixi Z, et al. Effect of reminders mailed to general practitioners on colorectal cancer screening adherence: a cluster‐randomized trial. European journal of cancer prevention : the official journal of the European Cancer Prevention Organisation (ECP) 2016;25(5):380‐7. [PUBMED: 26340058] [DOI] [PubMed] [Google Scholar]

[CD001175-bib-0015] Lobach DF, Hammond WE. Computerized decision support based on a clinical practice guideline improves compliance with care standards. American Journal of Medicine 1997;102(1):89‐98. [DOI] [PubMed] [Google Scholar]

[CD001175-bib-0016] Lobach DF, Hammond WE. Development and evaluation of a Computer‐Assisted Management Protocol (CAMP): improved compliance with care guidelines for diabetes mellitus. Proceedings / the Annual Symposium on Computer Applications in Medical Care 1994;‐:787‐91. [PMC free article] [PubMed] [Google Scholar]

[CD001175-bib-0017] Majumdar SR, McAlister FA, Tsuyuki RT. A cluster randomized trial to assess the impact of opinion leader endorsed evidence summaries on improving quality of prescribing for patients with chronic cardiovascular disease: rationale and design [ISRCTN26365328]. BMC Cardiovascular Disorders 2005;5(1):17. [DOI] [PMC free article] [PubMed] [Google Scholar]

[CD001175-bib-0018] Majumdar SR, Tsuyuki RT, McAlister FA. Impact of opinion leader‐endorsed evidence summaries on the quality of prescribing for patients with cardiovascular disease: a randomized controlled trial. American Heart Journal 2007;153(1):22.e1‐8. [DOI] [PubMed] [Google Scholar]

[CD001175-bib-0019] Mazzuca SA, Vinicor F, Einterz RM, Tierney WM, Norton JA, Kalasinski LA. Effects of the clinical environment on physicians' response to postgraduate medical education. American Educational Research Journal 1990;27(3):473‐88. [Google Scholar]

[CD001175-bib-0020] McAlister FA, Fradette M, Graham M, Majumdar SR, Ghali WA, Williams R, et al. A randomized trial to assess the impact of opinion leader endorsed evidence summaries on the use of secondary prevention strategies in patients with coronary artery disease: the ESP‐CAD trial protocol [NCT00175240]. Implementation Science 2006;1:11. [DOI] [PMC free article] [PubMed] [Google Scholar]

[CD001175-bib-0021] McAlister FA, Fradette M, Majumdar SR, Williams R, Graham M, McMeekin J, et al. The enhancing secondary prevention in coronary artery disease trial. Canadian Medical Association Journal 2009;181(12):897‐904. [DOI] [PMC free article] [PubMed] [Google Scholar]

[CD001175-bib-0022] McDonald CJ. Protocol‐based computer reminders, the quality of care and the non‐perfectability of man. New England Journal of Medicine 1976;295(24):1351‐5. [DOI] [PubMed] [Google Scholar]

[CD001175-bib-0023] McDonald CJ. Use of a computer to detect and respond to clinical events: its effect on clinician behavior. Annals of Internal Medicine 1976;84(2):162‐7. [DOI] [PubMed] [Google Scholar]

[CD001175-bib-0024] McDonald CJ, Wilson GA, McCabe GP Jr. Physician response to computer reminders. JAMA 1980;244(14):1579‐81. [PubMed] [Google Scholar]

[CD001175-bib-0025] McDonald CJ, Hui SL, Smith DM, Tierney WM, Cohen SJ, Weinberger M, et al. Reminders to physicians from an introspective computer medical record. A two‐year randomized trial. Annals of Internal Medicine 1984;100(1):130‐8. [DOI] [PubMed] [Google Scholar]

[CD001175-bib-0026] McDonald CJ, Hui SL, Tierney WM. Effects of computer reminders for influenza vaccination on morbidity during influenza epidemics. MD Computing 1992;9(5):304‐12. [PubMed] [Google Scholar]

[CD001175-bib-0027] Bird JA, McPhee SJ, Jenkins C, Fordham D. Three strategies to promote cancer screening. How feasible is wide‐scale implementation?. Medical Care 1990;28(11):1005‐12. [DOI] [PubMed] [Google Scholar]

[CD001175-bib-0028] Fordham D, McPhee SJ, Bird JA, Rodnick JE, Detmer WM. The Cancer Prevention Reminder System. MD Computing 1990;7(5):289‐95. [PubMed] [Google Scholar]

[CD001175-bib-0029] McPhee SJ, Bird JA, Jenkins CN, Fordham D. Promoting cancer screening. A randomized, controlled trial of three interventions. Archives of Internal Medicine 1989;149(8):1866‐72. [DOI] [PubMed] [Google Scholar]

[CD001175-bib-0030] Morgan M, Studney DR, Barnett GO, Winickoff RN. Computerized concurrent review of prenatal care. QRB. Quality Review Bulletin 1978;4(9):33‐6. [PubMed] [Google Scholar]

[CD001175-bib-0032] Nilasena DS, Lincoln MJ. A computer‐generated reminder system improves physician compliance with diabetes preventive care guidelines. Proceedings / the Annual Symposium on Computer Applications in Medical Care. 1995:640‐5. [PMC free article] [PubMed]

[CD001175-bib-0031] Nilasena DS, Lincoln MJ, Turner CW, Warner HR, Foerster VA, Williamson JW, et al. Development and implementation of a computer‐generated reminder system for diabetes preventive care. Proceedings / the Annual Symposium on Computer Application in Medical Care. 1994:831‐5. [PMC free article] [PubMed]

[CD001175-bib-0033] Oniki TA, Clemmer TP, Pryor TA. The effect of computer‐generated reminders on charting deficiencies in the ICU. Journal of the American Medical Informatics Association 2003;10(2):177‐87. [DOI] [PMC free article] [PubMed] [Google Scholar]

[CD001175-bib-0034] Ornstein SM, Garr DR, Jenkins RG, Rust PF, Arnon A. Computer‐generated physician and patient reminders. Tools to improve population adherence to selected preventive services. Journal of Family Practice 1991;32(1):82‐90. [PubMed] [Google Scholar]

[CD001175-bib-0035] McDowell I, Newell C, Rosser W. A randomized trial of computerized reminders for blood pressure screening in primary care. Medical Care 1989;27(3):297‐305. [DOI] [PubMed] [Google Scholar]

[CD001175-bib-0036] McDowell I, Newell C, Rosser W. Comparison of three methods of recalling patients for influenza vaccination. Canadian Medical Association Journal 1986;135(9):991‐7. [PMC free article] [PubMed] [Google Scholar]

[CD001175-bib-0037] McDowell I, Newell C, Rosser W. Computerized reminders to encourage cervical screening in family practice. Journal of Family Practice 1989;28(4):420‐4. [PubMed] [Google Scholar]

[CD001175-bib-0038] Rosser WW, Hutchison BG, McDowell I, Newell C. Use of reminders to increase compliance with tetanus booster vaccination. Canadian Medical Association Journal 1992;146(6):911‐7. [PMC free article] [PubMed] [Google Scholar]

[CD001175-bib-0039] Rosser WW, McDowell I, Newell C. Use of reminders for preventive procedures in family medicine. Canadian Medical Association Journal 1991;145(7):807‐14. [PMC free article] [PubMed] [Google Scholar]

[CD001175-bib-0040] Rossi RA, Every NR. A computerized intervention to decrease the use of calcium channel blockers in hypertension. Journal of General Internal Medicine 1997;12(11):672‐8. [DOI] [PMC free article] [PubMed] [Google Scholar]

[CD001175-bib-0041] Thomas JC, Moore A, Qualls PE. The effect on cost of medical care for patients treated with an automated clinical audit system. Journal of Medical Systems 1983;7(3):307‐13. [DOI] [PubMed] [Google Scholar]

[CD001175-bib-0042] Tierney WM, Hui SL, McDonald CJ. Delayed feedback of physician performance versus immediate reminders to perform preventive care. Effects on physician compliance. Medical Care 1986;24(8):659‐66. [DOI] [PubMed] [Google Scholar]

[CD001175-bib-0044] Turner BJ. A controlled trial of strategies to improve delivery of preventive care. Research in medical education:proceedings of the annual Conference. Conference on Research in Medical Education 1987;26:9‐13. [PubMed] [Google Scholar]

[CD001175-bib-0043] Turner BJ, Day SC, Borenstein B. A controlled trial to improve delivery of preventive care: physician or patient reminders?. Journal of General Internal Medicine 1989;4(5):403‐9. [DOI] [PubMed] [Google Scholar]

[CD001175-bib-0045] Were MC, Nyandiko WM, Huang KT, Slaven JE, Shen C, Tierney WM, et al. Computer‐generated reminders and quality of pediatric HIV care in a resource‐limited setting. Pediatrics 2013;131(3):e789‐96. [PUBMED: 23439898] [DOI] [PMC free article] [PubMed] [Google Scholar]

[CD001175-bib-0046] White KS, Lindsay A, Pryor TA, Brown WF, Walsh K. Application of a computerized medical decision‐making process to the problem of digoxin intoxication. Journal of the American College of Cardiology 1984;4(3):571‐6. [DOI] [PubMed] [Google Scholar]

[CD001175-bib-0047] Phillips LS, Hertzberg VS, Cook CB, El‐Kebbi IM, Gallina DL, Ziemer DC, et al. The Improving Primary Care of African Americans with Diabetes (IPCAAD) project: rationale and design. Controlled Clinical Trials 2002;23(5):554‐69. [DOI] [PubMed] [Google Scholar]

[CD001175-bib-0048] Phillips LS, Ziemer DC, Doyle JP, Barnes CS, Kolm P, Branch WT, et al. An endocrinologist‐supported intervention aimed at providers improves diabetes management in a primary care site: improving primary care of African Americans with diabetes (IPCAAD). Diabetes Care 2005;28(10):2352‐60. [DOI] [PubMed] [Google Scholar]

[CD001175-bib-0049] Ziemer DC, Doyle JP, Barnes CS, Branch WT Jr, Cook CB, El‐Kebbi IM, et al. An intervention to overcome clinical inertia and improve diabetes mellitus control in a primary care setting: Improving Primary Care of African Americans with Diabetes (IPCAAD). Archives of Internal Medicine 2006;166(5):507‐13. [DOI] [PubMed] [Google Scholar]

[CD001175-bib-0050] Adelman JS, Kalkut GE, Schechter CB, Weiss JM, Berger MA, Reissman SH, et al. Understanding and preventing wrong‐patient electronic orders: a randomized controlled trial. Journal of the American Medical Informatics Association: JAMIA 2013;20(2):305‐10. [DOI] [PMC free article] [PubMed] [Google Scholar]

[CD001175-bib-0051] Alfadda AA, Bin‐Abdulrahman KA, Saad HA, Mendoza CDO, Angkaya‐Bagayawa FF, Yale JF. Effect of an intervention to improve the management of patients with diabetes in primary care practice. Saudi Medical Journal 2011;32(1):36‐40. [PubMed] [Google Scholar]

[CD001175-bib-0052] Anabtawi A, Mathew LM. Improving compliance with screening of diabetic patients for microalbuminuria in primary care practice. ISRN Endocrinology 2013;2013:893913. [PUBMED: 24224095] [DOI] [PMC free article] [PubMed] [Google Scholar]

[CD001175-bib-0053] Anchala R, Pant H, Prabhakaran D, Franco OH. 'Decision support system (DSS) for prevention of cardiovascular disease (CVD) among hypertensive (HTN) patients in Andhra Pradesh, India'‐‐a cluster randomised community intervention trial. BMC Public Health 2012;12:393. [PUBMED: 22650767] [DOI] [PMC free article] [PubMed] [Google Scholar]

[CD001175-bib-0054] Barkun AN, Bhat M, Armstrong D, Dawes M, Donner A, Enns R, et al. Effectiveness of disseminating consensus management recommendations for ulcer bleeding: a cluster randomized trial. CMAJ: Canadian Medical Association Journal 2013;185(3):E156‐66. [DOI] [PMC free article] [PubMed] [Google Scholar]

[CD001175-bib-0055] Barnes ER, Theeke LA, Mallow J. Impact of the Provider and Healthcare team Adherence to Treatment Guidelines (PHAT‐G) intervention on adherence to national obesity clinical practice guidelines in a primary care centre. Journal of Evaluation in Clinical Practice 2014;21(2):300‐6. [DOI] [PMC free article] [PubMed] [Google Scholar]

[CD001175-bib-0056] Beeckman D, Clays E, Hecke A, Vanderwee K, Schoonhoven L, Verhaeghe S. A multi‐faceted tailored strategy to implement an electronic clinical decision support system for pressure ulcer prevention in nursing homes: a two‐armed randomized controlled trial. Internation Journal of Nursing Studies 2013;50(4):475‐86. [DOI] [PubMed] [Google Scholar]

[CD001175-bib-0057] Beeler PE, Eschmann E, Schumacher A, Studt JD, Amann‐Vesti B, Blaser J. Impact of electronic reminders on venous thromboprophylaxis after admissions and transfers. Journal of the American Medical Informatics Association: JAMIA 2014;21(e2):e297‐303. [DOI] [PMC free article] [PubMed] [Google Scholar]

[CD001175-bib-0058] Belland L, Rivera‐Reyes L, Genes N, Thum F, Winkel G, Soriano J, et al. The effect of a novel clinical decision support system on pain care for older adults presenting to the emergency department with acute abdominal pain. Annals of Emergency Medicine. 2014; Vol. 64, issue 4:S81.

[CD001175-bib-0059] Beste LA, Ioannou GN, Yang Y, Chang MF, Ross D, Dominitz JA. Improved surveillance for hepatocellular carcinoma with a primary care‐oriented clinical reminder. Clinical Gastroenterology and Hepatology : the official clinical practice journal of the American Gastroenterological Association 2015;13(1):172‐9. [PUBMED: 24813175] [DOI] [PubMed] [Google Scholar]

[CD001175-bib-0060] Braun R, Barandas A, Hancock‐Crear A. Improving access and addressing barriers to HPV vaccination for adolescents and young adults. Contraception. 2013; Vol. 88, issue 2:307.

[CD001175-bib-0061] Campbell F, Peleshok J, Yamada J, Krancevic A‐M, Pepper J, Dhaliwal M, et al. Impact of a quality improvement strategy for improving pain outcomes in a pediatric hospital setting. Pain Research and Management. 2014; Vol. 19, issue 3:e31.

[CD001175-bib-0062] Carroll AE, Biondich P, Anand V, Dugan TM, Downs SM. A randomized controlled trial of screening for maternal depression with a clinical decision support system. Journal of the American Medical Informatics Association: JAMIA 2013;20(2):311‐6. [DOI] [PMC free article] [PubMed] [Google Scholar]

[CD001175-bib-0063] Chen YY, Chi MM, Chen YC, Chan YJ, Chou SS, Wang FD. Using a criteria‐based reminder to reduce use of indwelling urinary catheters and decrease urinary tract infections. American Journal of Critical Care 2013; Vol. 22, issue 2:105‐14. [DOI] [PubMed]

[CD001175-bib-0064] Dexheimer JW, Abramo TJ, Arnold DH, Johnson KB, Shyr Y, Ye F, et al. An asthma management system in a pediatric emergency department. International Journal of Medical Informatics 2013;82(4):230‐8. [DOI] [PMC free article] [PubMed] [Google Scholar]

[CD001175-bib-0065] Divinskiy T, Nemytova E, Liu X, Ajmal S. A quality improvement project increased documentation of advance directives among an older population in a primary care setting. Annual Scientific Meeting of the American Geriatrics Society. 2015:63.

[CD001175-bib-0066] dos Santos MA, Tygesen H, Eriksson H, Herlitz J. Clinical decision support system (CDSS)‐‐effects on care quality. International Journal of Health Care Quality Assurance 2014;27(8):707‐18. [DOI] [PubMed] [Google Scholar]

[CD001175-bib-0067] Erlingsdottir H, Johannesson A, Asgeirsdottir TL. Can physician laboratory‐test requests be influenced by interventions?. Scandinavian Journal of Clinical and Laboratory Investigation 2015;75(1):18‐26. [DOI] [PubMed] [Google Scholar]

[CD001175-bib-0068] Federman DG, Kravetz JD, Lerz KA, Akgun KM, Ruser C, Cain H, et al. Implementation of an electronic clinical reminder to improve rates of lung cancer screening. American Journal of Medicine 2014;127(9):813‐6. [DOI] [PubMed] [Google Scholar]

[CD001175-bib-0069] Gifford WA, Davies BL, Graham ID, Tourangeau A, Woodend AK, Lefebre N. Developing leadership capacity for guideline use: a pilot cluster randomized control trial. Worldviews on Evidence‐Based Nursing 2013;10(1):51‐65. [DOI] [PubMed] [Google Scholar]

[CD001175-bib-0070] Goldstein NE, Kalman J, Kutner JS, Fromme EK, Hutchinson MD, Lipman HI, et al. A study to improve communication between clinicians and patients with advanced heart failure: Methods and challenges behind the working to improve discussions about defibrillator management trial. Journal of Pain and Symptom Management 2014;48(6):1236‐46. [DOI] [PMC free article] [PubMed] [Google Scholar]

[CD001175-bib-0071] Green BB, Anderson ML, Wang C‐Y, Vernon SW, Chubak J, Meenan RT, et al. Results of nurse navigator follow‐up after positive colorectal cancer screening test: A randomized trial. Journal of the American Board of Family Medicine 2014;27(6):789‐95. [DOI] [PMC free article] [PubMed] [Google Scholar]

[CD001175-bib-0072] Gupta A, Gholami P, Turakhia MP, Friday K, Heidenreich PA. Clinical reminders to providers of patients with reduced left ventricular ejection fraction increase defibrillator referral: a randomized trial. Circulation Heart Failure 2014;7(1):140‐5. [DOI] [PubMed] [Google Scholar]

[CD001175-bib-0073] Hendrix KS, Downs SM, Carroll AE. Pediatricians' responses to printed clinical reminders: does highlighting prompts improve responsiveness?. Academic Pediatrics 2015;15(2):158‐64. [DOI] [PMC free article] [PubMed] [Google Scholar]

[CD001175-bib-0074] Hye RJ, Smith AE, Wong GH, Vansomphone SS, Scott RD, Kanter MH. Leveraging the electronic medical record to implement an abdominal aortic aneurysm screening program. Journal of Vascular Surgery 2014;59(6):1535‐42. [DOI] [PubMed] [Google Scholar]

[CD001175-bib-0075] Jansink R, Braspenning J, Keizer E, Weijden T, Elwyn G, Grol R. No identifiable Hb1Ac or lifestyle change after a comprehensive diabetes programme including motivational interviewing: a cluster randomised trial. Scandinavian Journal of Primary Health Care 2013;31(2):119‐27. [DOI] [PMC free article] [PubMed] [Google Scholar]

[CD001175-bib-0076] Kennedy CC, Ioannidis G, Giangregorio LM, Adachi JD, Thabane L, Morin SN, et al. An interdisciplinary knowledge translation intervention in long‐term care: study protocol for the vitamin D and osteoporosis study (Original Title; URL links to DARE Abstract) pilot cluster randomized controlled trial. Implementation Science 2012;7:48. [DOI] [PMC free article] [PubMed] [Google Scholar]

[CD001175-bib-0077] Kousgaard MB, Siersma V, Reventlow S, Ertmann R, Felding P, Waldorff FB. The effectiveness of computer reminders for improving quality assessment for point‐of‐care testing in general practice‐‐a randomized controlled trial. Implementation Science 2013;8:47. [DOI] [PMC free article] [PubMed] [Google Scholar]

[CD001175-bib-0078] Lai CL, Chan HY, Pan YJ, Chen CH. The effectiveness of a computer reminder system for laboratory monitoring of metabolic syndrome in schizophrenic outpatients using second‐generation antipsychotics. Pharmacopsychiatry 2015;48(1):25‐9. [DOI] [PubMed] [Google Scholar]

[CD001175-bib-0079] Levy BT, Xu Y, Daly JM, Ely JW. A randomized controlled trial to improve colon cancer screening in rural family medicine: an Iowa Research Network (IRENE) study. Journal of the American Board of Family Medicine 2013;26(5):486‐97. [DOI] [PubMed] [Google Scholar]

[CD001175-bib-0080] Lusignan S, Gallagher H, Jones S, Chan T, Vlymen J, Tahir A, et al. Audit‐based education lowers systolic blood pressure in chronic kidney disease: the Quality Improvement in CKD (QICKD) trial results. Kidney International 2013;84(3):609‐20. [DOI] [PMC free article] [PubMed] [Google Scholar]

[CD001175-bib-0081] Lynn LA, Guerra CE, Ross KM, Holmboe E, Woo K, Heitjan DF, et al. A randomized controlled trial of an evidence‐based toolbox and guide to increase primary care clinicians' rates of colorectal cancer screening in diplomates of the ABIM. Journal of General Internal Medicine. 2013; Vol. 28, issue Suppl 1.

[CD001175-bib-0082] Majumdar SR, McAlister FA, Johnson JA, Bellerose D, Siminoski K, Hanley DA, et al. Interventions to increase osteoporosis treatment in patients with 'incidentally' detected vertebral fractures. American Journal of Medicine 2012;125(9):929‐36. [DOI] [PubMed] [Google Scholar]

[CD001175-bib-0083] Maximov D, Lesnyak O. Implementation of the osteoarthritis clinical guideline: Results of a cluster randomized trial in primary care. Annals of the Rheumatic Diseases. 2013; Vol. 71:307‐8.

[CD001175-bib-0084] McEvoy MD, Hand WR, Stoll WD, Furse CM, Nietert PJ. Adherence to guidelines for the management of local anesthetic systemic toxicity is improved by an electronic decision support tool and designated "Reader". Regional Anesthesia 2014;39(4):299‐305. [DOI] [PMC free article] [PubMed] [Google Scholar]

[CD001175-bib-0085] McNulty CA, Hogan AH, Ricketts EJ, Wallace L, Oliver I, Campbell R, et al. Increasing chlamydia screening tests in general practice: a modified Zelen prospective Cluster Randomised Controlled Trial evaluating a complex intervention based on the Theory of Planned Behaviour. Sexually Transmitted Infections 2014;90(3):188‐94. [PUBMED: 24005256] [DOI] [PMC free article] [PubMed] [Google Scholar]

[CD001175-bib-0086] Melo Rodrigues R, Fontes AMS, Cesar Mantese O, Souza Martins R, Tanus Jorge M. Impact of an intervention in the use of sequential antibiotic therapy in a Brazilian university hospital. Revista da Sociedade Brasileira de Medicina Tropical 2013;46(1):50‐4. [DOI] [PubMed] [Google Scholar]

[CD001175-bib-0087] Neal JM, Hsiung RL, Mulroy MF, Halpern BB, Dragnich AD, Slee AE. ASRA checklist improves trainee performance during a simulated episode of local anesthetic systemic toxicity. Regional Anesthesia and Pain Medicine 2012;37(1):8‐15. [DOI] [PubMed] [Google Scholar]

[CD001175-bib-0088] Nguyen MC, Richardson DM, Hardy SG, Cookson RM, Mackenzie RS, Greenberg MR, et al. Computer‐based reminder system effectively impacts physician documentation. American Journal of Emergency Medicine 2014;32(1):104‐6. [DOI] [PubMed] [Google Scholar]

[CD001175-bib-0089] O'Reilly D, Holbrook A, Blackhouse G, Troyan S, Goeree R. Cost‐effectiveness of a shared computerized decision support system for diabetes linked to electronic medical records. Journal of the American Medical Informatics Association: JAMIA 2012;19(3):341‐5. [DOI] [PMC free article] [PubMed] [Google Scholar]

[CD001175-bib-0090] Persell SD, Doctor JN, Friedberg MW, Meeker D, Friesema E, Cooper A, et al. Behavioral interventions to reduce inappropriate antibiotic prescribing: a randomized pilot trial. BMC Infectious Diseases 2016;16:373. [PUBMED: 27495917] [DOI] [PMC free article] [PubMed] [Google Scholar]

[CD001175-bib-0091] Piazza G, Anderson FA, Ortel TL, Cox MJ, Rosenberg DJ, Rahimian S, et al. Randomized trial of physician alerts for thromboprophylaxis after discharge. American Journal of Medicine 2013;126(5):435‐42. [DOI] [PubMed] [Google Scholar]

[CD001175-bib-0092] Roy PM, Rachas A, Meyer G, Gal G, Durieux P, Kouri D, et al. Multifaceted Intervention to Prevent Venous Thromboembolism in Patients Hospitalized for Acute Medical Illness: A Multicenter Cluster‐Randomized Trial. PloS One 2016;11(5):e0154832. [PUBMED: 27227406] [DOI] [PMC free article] [PubMed] [Google Scholar]

[CD001175-bib-0093] Schwalm JD, Ivers NM, Natarajan MK, Taljaard M, Rao‐Melacini P, Witteman HO, et al. Cluster randomized controlled trial of Delayed Educational Reminders for Long‐term Medication Adherence in ST‐Elevation Myocardial Infarction (DERLA‐STEMI). American Heart Journal 2015;170(5):903‐13. [PUBMED: 26542498] [DOI] [PubMed] [Google Scholar]

[CD001175-bib-0094] Siersma V, Kousgaard MB, Reventlow S, Ertmann R, Felding P, Waldorff FB. The effectiveness of computer reminders versus postal reminders for improving quality assessment for point‐of‐care testing in primary care: a randomized controlled trial. Journal of Evaluation in Clinical Practice 2015;21(1):13‐20. [DOI] [PubMed] [Google Scholar]

[CD001175-bib-0095] Stockwell MS, Catallozzi M, Camargo S, Ramakrishnan R, Holleran S, Findley SE, et al. Registry‐linked electronic influenza vaccine provider reminders: a cluster‐crossover trial. Pediatrics 2015;135(1):e75‐82. [DOI] [PubMed] [Google Scholar]

[CD001175-bib-0096] Szilagyi PG, Serwint JR, Humiston SG, Rand CM, Schaffer S, Vincelli P, et al. Effect of provider prompts on adolescent immunization rates: a randomized trial. Academic Pediatrics 2015;15(2):149‐57. [DOI] [PMC free article] [PubMed] [Google Scholar]

[CD001175-bib-0097] Tartaglia KM, Walker C, Heacock A, Kuofie I, Liston B, West J, et al. Antibiotic timeout program: Implementing a hospital stewardship intervention. Journal of General Internal Medicine 2013;28:S429‐30. [Google Scholar]

[CD001175-bib-0098] Tedja R, Hassebroek E, Carrera P, Thakur L, Mielke C, Volkman T, et al. Reducing urinary catheter days and utilization in a chronic ventilator dependent unit: Mayo clinic quality improvement experience. Chest 2014;146:556A. [Google Scholar]

[CD001175-bib-0099] Teoh L, Latif M, Choi P, Wu S, Chan CY, Plowman G, et al. Use of a reminder sticker improves rates of documentation of resuscitation status and the appropriate prescription of venous thromboembolism prophylaxis. New Zealand Medical Journal 2012;125(1357):175‐7. [PubMed] [Google Scholar]

[CD001175-bib-0100] Weiss CH, Dibardino D, Rho J, Sung N, Collander B, Wunderink RG. A clinical trial comparing physician prompting with an unprompted automated electronic checklist to reduce empirical antibiotic utilization. Critical Care Medicine 2013;41(11):2563‐9. [DOI] [PMC free article] [PubMed] [Google Scholar]

[CD001175-bib-0101] Were MC, Shen C, Tierney WM, Mamlin JJ, Biondich PG, Li X, et al. Evaluation of computer‐generated reminders to improve CD4 laboratory monitoring in sub‐Saharan Africa: a prospective comparative study. Journal of the American Medical Informatics Association : JAMIA 2011;18(2):150‐5. [PUBMED: 21252053] [DOI] [PMC free article] [PubMed] [Google Scholar]

[CD001175-bib-0102] Anderson JG. Social, ethical and legal barriers to e‐health. International Journal of Medical Informatics 2007;76(5‐6):480‐3. [PUBMED: 17064955] [DOI] [PubMed] [Google Scholar]

[CD001175-bib-0103] Axt‐Adam P, Wouden JC, Does E. Influencing behavior of physicians ordering laboratory tests: a literature study. Medical Care 1993;31:784‐94. [DOI] [PubMed] [Google Scholar]

[CD001175-bib-0104] Baker R, Camosso‐Stefinovic J, Gillies C, Shaw EJ, Cheater F, Flottorp S, et al. Tailored interventions to address determinants of practice. Cochrane Database of Systematic Reviews 2015, Issue 4. [DOI: 10.1002/14651858.CD005470.pub3] [DOI] [PMC free article] [PubMed] [Google Scholar]

[CD001175-bib-0105] Balas EA, Weingarten S, Garb CT, Blumenthal D, Boren SA, Brown GD. Improving preventive care by prompting physicians. Archives of Internal Medicine 2000;160(3):301‐8. [PUBMED: 10668831] [DOI] [PubMed] [Google Scholar]

[CD001175-bib-0106] Balas EA, Krishna S, Kretschmer RA, Cheek TR, Lobach DF, Boren SA. Computerized knowledge management in diabetes care. Medical Care 2004;42(6):610‐21. [PUBMED: 15167329] [DOI] [PubMed] [Google Scholar]

[CD001175-bib-0107] Baron RC, Melillo S, Rimer BK, Coates RJ, Kerner J, Habarta N, et al. Intervention to increase recommendation and delivery of screening for breast, cervical, and colorectal cancers by healthcare providers a systematic review of provider reminders. American Journal of Preventive Medicine 2010;38(1):110‐7. [PUBMED: 20117566] [DOI] [PubMed] [Google Scholar]

[CD001175-bib-0108] Bennett JW, Glasziou PP. Computerised reminders and feedback in medication management: a systematic review of randomised controlled trials. Medical Journal of Australia 2003;178(5):217‐22. [PUBMED: 12603185] [DOI] [PubMed] [Google Scholar]

[CD001175-bib-0109] Biau DJ, Porcher R, Boutron I. The account for provider and center effects in multicenter interventional and surgical randomized controlled trials is in need of improvement: a review. Journal of Clinical Epidemiology 2008;61(5):435‐9. [PUBMED: 18394535] [DOI] [PubMed] [Google Scholar]

[CD001175-bib-0110] Buntinx F, Winkens R, Grol R, Knottnerus JA. Influencing diagnostic and preventive performance in ambulatory care by feedback and reminders. A review. Family Practice 1993;10(2):219‐28. [DOI] [PubMed] [Google Scholar]

[CD001175-bib-0111] Campbell MJ, Donner A, Klar N. Developments in cluster randomized trials and Statistics in Medicine. Statistics in Medicine 2007;26(1):2‐19. [PUBMED: 17136746] [DOI] [PubMed] [Google Scholar]

[CD001175-bib-0112] Craig P, Dieppe P, Macintyre S, Michie S, Nazareth I, Petticrew M. Developing and evaluating complex interventions: the new Medical Research Council guidance. BMJ (Clinical Research ed.) 2008;337:a1655. [PUBMED: 18824488] [DOI] [PMC free article] [PubMed] [Google Scholar]

[CD001175-bib-0113] Davis DA, Thomson MA, Oxman AD, Haynes RB. Evidence for the effectiveness of CME. A review of 50 randomized controlled trials. JAMA 1992;268(9):1111‐7. [PubMed] [Google Scholar]

[CD001175-bib-0114] Davis DA, Thomson MA, Oxman AD, Haynes RB. Changing physician performance. A systematic review of the effect of continuing medical education strategies. JAMA 1995;274(9):700‐5. [DOI] [PubMed] [Google Scholar]

[CD001175-bib-0115] Dexheimer JW, Talbot TR, Sanders DL, Rosenbloom ST, Aronsky D. Prompting clinicians about preventive care measures: a systematic review of randomized controlled trials. Journal of the American Medical Informatics Association 2008;15(3):311‐20. [PUBMED: 18308989] [DOI] [PMC free article] [PubMed] [Google Scholar]

[CD001175-bib-0116] Effective Practice, Organisation of Care (EPOC). Suggested risk of bias criteria for EPOC reviews. epoc.cochrane.org/epoc‐specific‐resources‐review‐authors. Oslo: Norwegian Knowledge Centre for the Health Services, (accessed 1 July 2016).

[CD001175-bib-0117] Effective Practice, Organisation of Care (EPOC). What study designs should be included in an EPOC review and what should they be called?. epoc.cochrane.org/epoc‐specific‐resources‐review‐authors. Oslo: Norwegian Knowledge Centre for the Health Services, (accessed 1 July 2016).

[CD001175-bib-0118] Effective Practice, Organisation of Care (EPOC). Worksheets for preparing Summary of Findings tables using GRADE. epoc.cochrane.org/epoc‐specific‐resources‐review‐authors. Oslo: Norwegian Knowledge Centre for the Health Services, (accessed 1 July 2016).

[CD001175-bib-0119] Effective Practice, Organisation of Care (EPOC). What outcomes should be reported in EPOC reviews?. epoc.cochrane.org/epoc‐specific‐resources‐review‐authors. Oslo: Norwegian Knowledge Centre for the Health Services, (accessed 1 July 2016).

[CD001175-bib-0120] Fiander M, McGowan J, Grad R, Pluye P, Hannes K, Labrecque M, et al. Interventions to increase the use of electronic health information by healthcare practitioners to improve clinical practice and patient outcomes. Cochrane Database of Systematic Reviews 2015, Issue 3. [DOI: 10.1002/14651858.CD004749.pub3] [DOI] [PMC free article] [PubMed] [Google Scholar]

[CD001175-bib-0121] Flodgren G, Parmelli E, Doumit G, Gattellari M, O'Brien MA, Grimshaw J, et al. Local opinion leaders: effects on professional practice and health care outcomes. Cochrane Database of Systematic Reviews 2011, Issue 8. [DOI: 10.1002/14651858.CD000125.pub4] [DOI] [PMC free article] [PubMed] [Google Scholar]

[CD001175-bib-0122] Forsetlund L, Bjorndal A, Rashidian A, Jamtvedt G, O'Brien MA, Wolf F, et al. Continuing education meetings and workshops: effects on professional practice and health care outcomes. Cochrane Database of Systematic Reviews 2009, Issue 2. [DOI: 10.1002/14651858.CD003030.pub2] [DOI] [PMC free article] [PubMed] [Google Scholar]

[CD001175-bib-0123] Gagnon MP, Legare F, Labrecque M, Fremont P, Pluye P, Gagnon J, et al. Interventions for promoting information and communication technologies adoption in healthcare professionals. Cochrane Database of Systematic Reviews 2009, Issue 1. [DOI: 10.1002/14651858.CD006093.pub2] [DOI] [PMC free article] [PubMed] [Google Scholar]

[CD001175-bib-0124] Garg AX, Adhikari NK, McDonald H, Rosas‐Arellano MP, Devereaux PJ, Beyene J, et al. Effects of computerized clinical decision support systems on practitioner performance and patient outcomes: a systematic review. JAMA 2005;293(10):1223‐38. [PUBMED: 15755945] [DOI] [PubMed] [Google Scholar]

[CD001175-bib-0125] Giguère A, Légaré F, Grimshaw J, Turcotte S, Fiander M, Grudniewicz A, et al. Printed educational materials: effects on professional practice and healthcare outcomes. Cochrane Database of Systematic Reviews 2012, Issue 10. [DOI: 10.1002/14651858.CD004398.pub3] [DOI] [PMC free article] [PubMed] [Google Scholar]

[CD001175-bib-0126] GRADE Working Group, McMaster University. GRADEpro GDT. Version accessed 10 August 2015. Hamilton (ON): GRADE Working Group, McMaster University, 2014.

[CD001175-bib-0127] Grimshaw JM, Thomas RE, MacLennan G, Fraser C, Ramsay CR, Vale L, et al. Effectiveness and efficiency of guideline dissemination and implementation strategies. Health Technology Assessment (Winchester, England) 2004;8(6):iii‐iv, 1‐72. [PUBMED: 14960256] [DOI] [PubMed] [Google Scholar]

[CD001175-bib-0128] Guyatt GH, Oxman AD, Schunemann HJ, Tugwell P, Knottnerus A. GRADE guidelines: a new series of articles in the Journal of Clinical Epidemiology. Journal of Clinical Epidemiology 2011;64(4):380‐2. [PUBMED: 21185693] [DOI] [PubMed] [Google Scholar]

[CD001175-bib-0129] Haynes RB, Walker CJ. Computer‐aided quality assurance. A critical appraisal. Archives of Internal Medicine 1987;147(7):1297‐301. [PubMed] [Google Scholar]

[CD001175-bib-0130] Higgins, JPT, Green, S. Cochrane Handbook for Systematic Reviews of Interventions Version. Version 5.1.0. Cochrane Handbook for Systematic Reviews of Interventions. Chichester: John Wiley & Sons, 2011. [Google Scholar]

[CD001175-bib-0131] Hoffmann TC, Glasziou PP, Boutron I, Milne R, Perera R, Moher D, et al. Better reporting of interventions: template for intervention description and replication (TIDieR) checklist and guide. BMJ (Clinical Research ed.) 2014;348:g1687. [PUBMED: 24609605] [DOI] [PubMed] [Google Scholar]

[CD001175-bib-0132] Hunt DL, Haynes RB, Hanna SE, Smith K. Effects of computer‐based clinical decision support systems on physician performance and patient outcomes: a systematic review. JAMA 1998;280(15):1339‐46. [PUBMED: 9794315] [DOI] [PubMed] [Google Scholar]

[CD001175-bib-0133] Ivers N, Jamtvedt G, Flottorp S, Young JM, Odgaard‐Jensen J, French SD, et al. Audit and feedback: effects on professional practice and healthcare outcomes. Cochrane Database of Systematic Reviews 2012;6:CD000259. [DOI: 10.1002/14651858.CD000259.pub3] [DOI] [PMC free article] [PubMed] [Google Scholar]

[CD001175-bib-0134] Jha AK, DesRoches CM, Campbell EG, Donelan K, Rao SR, Ferris TG, et al. Use of electronic health records in U.S. hospitals. The New England Journal of Medicine 2009;360(16):1628‐38. [PUBMED: 19321858] [DOI] [PubMed] [Google Scholar]

[CD001175-bib-0135] Johnston ME, Langton KB, Haynes RB, Mathieu A. Effects of computer‐based clinical decision support systems on clinician performance and patient outcome. Annals of Internal Medicine 1994;120(2):135‐42. [DOI] [PubMed] [Google Scholar]

[CD001175-bib-0136] Kahan BC, Morris TP. Assessing potential sources of clustering in individually randomised trials. BMC Medical Research Methodology 2013;13:58. [PUBMED: 23590245] [DOI] [PMC free article] [PubMed] [Google Scholar]

[CD001175-bib-0137] Kawamoto K, Houlihan CA, Balas EA, Lobach DF. Improving clinical practice using clinical decision support systems: a systematic review of trials to identify features critical to success. BMJ 2005;330(7494):765. [PUBMED: 15767266] [DOI] [PMC free article] [PubMed] [Google Scholar]

[CD001175-bib-0138] Keogh‐Brown MR, Bachmann MO, Shepstone L, Hewitt C, Howe A, Ramsay CR, et al. Contamination in trials of educational interventions. Health Technology Assessment (Winchester, England) 2007;11(43):iii, ix‐107. [PUBMED: 17935683] [DOI] [PubMed] [Google Scholar]

[CD001175-bib-0139] Litzelman DK, Dittus RS, Miller ME, Tierney WM. Requiring physicians to respond to computerized reminders improves their compliance with preventive care protocols. Journal of General Internal Medicine 1993;8(6):311‐7. [DOI] [PubMed] [Google Scholar]

[CD001175-bib-0140] Mahmud N, Rodriguez J, Nesbit J. A text message‐based intervention to bridge the healthcare communication gap in the rural developing world. Technology and Health Care : official journal of the European Society for Engineering and Medicine 2010;18(2):137‐44. [PUBMED: 20495253] [DOI] [PubMed] [Google Scholar]

[CD001175-bib-0141] Mandelblatt J, Kanetsky PA. Effectiveness of interventions to enhance physician screening for breast cancer. Journal of Family Practice 1995;40(2):162‐71. [PubMed] [Google Scholar]

[CD001175-bib-0142] McDonald CJ. Protocol‐based computer reminders, the quality of care and the non‐perfectability of man. New England Journal of Medicine 1976;295(24):1351‐5. [DOI] [PubMed] [Google Scholar]

[CD001175-bib-0143] Moher D, Hopewell S, Schulz KF, Montori V, Gotzsche PC, Devereaux PJ, et al. CONSORT 2010 explanation and elaboration: updated guidelines for reporting parallel group randomised trials. BMJ 2010;340:c869. [PUBMED: 20332511] [DOI] [PMC free article] [PubMed] [Google Scholar]

[CD001175-bib-0144] Mohler R, Kopke S, Meyer G. Criteria for Reporting the Development and Evaluation of Complex Interventions in healthcare: revised guideline (CReDECI 2). Trials 2015;16:204. [PUBMED: 25935741] [DOI] [PMC free article] [PubMed] [Google Scholar]

[CD001175-bib-0145] Mollon B, Chong J Jr, Holbrook AM, Sung M, Thabane L, Foster G. Features predicting the success of computerized decision support for prescribing: a systematic review of randomized controlled trials. BMC Medical Informatics and Decision Making 2009;9:11. [PUBMED: 19210782] [DOI] [PMC free article] [PubMed] [Google Scholar]

[CD001175-bib-0146] Morris AH. Decision support and safety of clinical environments. Quality & Safety in Health Care 2002;11(1):69‐75. [PUBMED: 12078374] [DOI] [PMC free article] [PubMed] [Google Scholar]

[CD001175-bib-0147] Ndiaye SM, Hopkins DP, Shefer AM, Hinman AR, Briss PA, Rodewald L, et al. Interventions to improve influenza, pneumococcal polysaccharide, and hepatitis B vaccination coverage among high‐risk adults: a systematic review. American Journal of Preventive Medicine 2005;28(5 Suppl):248‐79. [PUBMED: 15894160] [DOI] [PubMed] [Google Scholar]

[CD001175-bib-0148] O'Brien MA, Rogers S, Jamtvedt G, Oxman AD, Odgaard‐Jensen J, Kristoffersen DT, et al. Educational outreach visits: effects on professional practice and health care outcomes. Cochrane Database of Systematic Reviews 2007, Issue 4. [DOI: 10.1002/14651858.CD000409.pub2; PUBMED: 17943742] [DOI] [PMC free article] [PubMed] [Google Scholar]

[CD001175-bib-0149] Ogrinc G, Davies L, Goodman D, Batalden P, Davidoff F, Stevens D. SQUIRE 2.0 (Standards for QUality Improvement Reporting Excellence): revised publication guidelines from a detailed consensus process. BMJ Quality & Safety 2016;25(12):986‐92. [PUBMED: 26369893] [DOI] [PMC free article] [PubMed] [Google Scholar]

[CD001175-bib-0150] Oxman AD, Thomson MA, Davis DA, Haynes RB. No magic bullets: a systematic review of 102 trials of interventions to improve professional practice. Canadian Medical Association Journal 1995;153(10):1423‐31. [PMC free article] [PubMed] [Google Scholar]

[CD001175-bib-0151] Pantoja T, Green ME, Grimshaw J, Denig P, Durieux P, Gill P, et al. Manual paper reminders: effects on professional practice and health care outcomes. Cochrane Database of Systematic Reviews 2014, Issue 9. [DOI: 10.1002/14651858.CD001174.pub2; CD001174] [DOI] [Google Scholar]

[CD001175-bib-0152] Pearson SA, Moxey A, Robertson J, Hains I, Williamson M, Reeve J, et al. Do computerised clinical decision support systems for prescribing change practice? A systematic review of the literature (1990‐2007). BMC Health Services Research 2009;9:154. [PUBMED: 19715591] [DOI] [PMC free article] [PubMed] [Google Scholar]

[CD001175-bib-0153] Schedlbauer A, Prasad V, Mulvaney C, Phansalkar S, Stanton W, Bates DW, et al. What evidence supports the use of computerized alerts and prompts to improve clinicians' prescribing behavior?. Journal of the American Medical Informatics Association 2009;16(4):531‐8. [PUBMED: 19390110] [DOI] [PMC free article] [PubMed] [Google Scholar]

[CD001175-bib-0154] Shea S, DuMouchel W, Bahamonde L. A meta‐analysis of 16 randomized controlled trials to evaluate computer‐based clinical reminder systems for preventive care in the ambulatory setting. Journal of the American Medical Informatics Association 1996;3(6):399‐409. [PUBMED: 8930856] [DOI] [PMC free article] [PubMed] [Google Scholar]

[CD001175-bib-0155] Shiffman RN, Liaw Y, Brandt CA, Corb GJ. Computer‐based guideline implementation systems: a systematic review of functionality and effectiveness. Journal of the American Medical Informatics Association : JAMIA 1999;6(2):104‐14. [PUBMED: 10094063] [DOI] [PMC free article] [PubMed] [Google Scholar]

[CD001175-bib-0156] Shojania KG, Jennings A, Mayhew A, Ramsay CR, Eccles MP, Grimshaw J. The effects of on‐screen, point of care computer reminders on processes and outcomes of care. Cochrane Database of Systematic Reviews 2009, Issue 3. [DOI: 10.1002/14651858.CD001096.pub2] [DOI] [PMC free article] [PubMed] [Google Scholar]

[CD001175-bib-0157] Siddiqi K, Newell J, Robinson M. Getting evidence into practice: what works in developing countries?. International Journal for Quality in Health Care : journal of the International Society for Quality in Health Care ISQua 2005;17(5):447‐54. [PUBMED: 15872024] [DOI] [PubMed] [Google Scholar]

[CD001175-bib-0158] StataCorp. Stata Statistical Software: Release 10. College Station, TX: StataCorp LP, 2007.

[CD001175-bib-0159] Thomas LH, Cullum NA, McColl E, Rousseau N, Soutter J, Steen N. Guidelines in professions allied to medicine. Cochrane Database of Systematic Reviews 1999, Issue 1. [DOI: 10.1002/14651858.CD000349] [DOI] [PubMed] [Google Scholar]

[CD001175-bib-0160] Tierney WM, Rotich JK, Hannan TJ, Siika AM, Biondich PG, Mamlin BW, et al. The AMPATH medical record system: creating, implementing, and sustaining an electronic medical record system to support HIV/AIDS care in western Kenya. Studies in Health Technology and Informatics 2007;129(Pt 1):372‐6. [PUBMED: 17911742] [PubMed] [Google Scholar]

[CD001175-bib-0161] Ukoumunne OC, Gulliford MC, Chinn S, Sterne JA, Burney PG, Donner A. Methods in health service research. Evaluation of health interventions at area and organisation level. BMJ (Clinical Research ed.) 1999;319(7206):376‐9. [PUBMED: 10435968] [DOI] [PMC free article] [PubMed] [Google Scholar]

[CD001175-bib-0162] Wensing M, Grol R. Single and combined strategies for implementing changes in primary care: a literature review. International Journal for Quality in Health Care 1994;6(2):115‐32. [DOI] [PubMed] [Google Scholar]

[CD001175-bib-0163] Arditi C, Rège‐Walther M, Burnand B, Wyatt J. Computer‐generated paper reminders: effects on professional practice and health care outcomes. Cochrane Database of Systematic Reviews 2010, Issue 5. [DOI: 10.1002/14651858.CD001175.pub2] [DOI] [PubMed] [Google Scholar]

[CD001175-bib-0164] Arditi C, Rège‐Walther M, Wyatt JC, Durieux P, Burnand B. Computer‐generated reminders delivered on paper to healthcare professionals; effects on professional practice and health care outcomes. Cochrane Database of Systematic Reviews 2012, Issue 12. [DOI: 10.1002/14651858.CD001175.pub3] [DOI] [PubMed] [Google Scholar]

[CD001175-bib-0165] Gorman P, Redfern C, Liaw T, Carson S, Wyatt JC, Rowe RE, et al. Computer‐generated paper reminders: effects on professional practice and health care outcomes. Cochrane Database of Systematic Reviews 1998, Issue 3. [DOI: 10.1002/14651858.CD001175] [DOI] [Google Scholar]

PERMALINK

Computer‐generated reminders delivered on paper to healthcare professionals: effects on professional practice and healthcare outcomes

Chantal Arditi

Myriam Rège‐Walther

Pierre Durieux

Bernard Burnand

Abstract

Background

Objectives

Search methods

Selection criteria

Data collection and analysis

Main results

Authors' conclusions

Plain language summary

Summary of findings

for the main comparison.

2.

3.

Background

Description of the condition

Description of the intervention

How the intervention might work

Why it is important to do this review

Objectives

Methods

Criteria for considering studies for this review

Types of studies

Types of participants

Types of interventions

Types of outcome measures

Primary outcomes

Secondary outcomes

Search methods for identification of studies

Electronic searches

Searching other resources

Data collection and analysis

Selection of studies

Data extraction and management

Assessment of risk of bias in included studies

Measures of treatment effect

Unit of analysis issues

Dealing with missing data

Assessment of heterogeneity

Assessment of reporting biases

Data synthesis

Summary of findings

Subgroup analysis and investigation of heterogeneity

Sensitivity analysis

Results

Description of studies

Results of the search

1.

Included studies

Design

Participants, setting and publication date

Interventions

Clinical domain and targeted behavior

Outcome measures

Excluded studies

Risk of bias in included studies

2.

3.

Allocation

Unit of allocation issues

Blinding

Incomplete outcome data

Other potential sources of bias

Effects of interventions

1. Improvement rates of quality of care, by study.

4.

Quality of care

Subgroup analyses: impact of reminder features on quality of care effect size

5.

Subgroup analyses: impact of study features on quality of care effect size

6.

Subgroup analyses: disadvantaged populations

Sensitivity analyses

2. Median improvement of quality of care across all comparisons and according to the presence of co‐interventions.

Patient outcomes