Patient reminder and recall interventions to improve immunization rates

Julie C Jacobson Vann; Robert M Jacobson; Tamera Coyne‐Beasley; Josephine K Asafu‐Adjei; Peter G Szilagyi

doi:10.1002/14651858.CD003941.pub3

. 2018 Jan 18;2018(1):CD003941. doi: 10.1002/14651858.CD003941.pub3

Patient reminder and recall interventions to improve immunization rates

Julie C Jacobson Vann ^1,^✉, Robert M Jacobson ², Tamera Coyne‐Beasley ³, Josephine K Asafu‐Adjei ⁴, Peter G Szilagyi ⁵

Editor: Cochrane Effective Practice and Organisation of Care Group

PMCID: PMC6491344 PMID: 29342498

Abstract

Background

Immunization rates for children and adults are rising, but coverage levels have not reached optimal goals. As a result, vaccine‐preventable diseases still occur. In an era of increasing complexity of immunization schedules, rising expectations about the performance of primary care, and large demands on primary care providers, it is important to understand and promote interventions that work in primary care settings to increase immunization coverage. One common theme across immunization programs in many nations involves the challenge of implementing a population‐based approach and identifying all eligible recipients, for example the children who should receive the measles vaccine. However, this issue is gradually being addressed through the availability of immunization registries and electronic health records. A second common theme is identifying the best strategies to promote high vaccination rates. Three types of strategies have been studied: (1) patient‐oriented interventions, such as patient reminder or recall, (2) provider interventions, and (3) system interventions, such as school laws. One of the most prominent intervention strategies, and perhaps best studied, involves patient reminder or recall systems. This is an update of a previously published review.

Objectives

To evaluate and compare the effectiveness of various types of patient reminder and recall interventions to improve receipt of immunizations.

Search methods

We searched CENTRAL, MEDLINE, Embase and CINAHL to January 2017. We also searched grey literature and trial registers to January 2017.

Selection criteria

We included randomized trials, controlled before and after studies, and interrupted time series evaluating immunization‐focused patient reminder or recall interventions in children, adolescents, and adults who receive immunizations in any setting. We included no‐intervention control groups, standard practice activities that did not include immunization patient reminder or recall, media‐based activities aimed at promoting immunizations, or simple practice‐based awareness campaigns. We included receipt of any immunizations as eligible outcome measures, excluding special travel immunizations. We excluded patients who were hospitalized for the duration of the study period.

Data collection and analysis

We used the standard methodological procedures expected by Cochrane and the Cochrane Effective Practice and Organisation of Care (EPOC) Group. We present results for individual studies as relative rates using risk ratios, and risk differences for randomized trials, and as absolute changes in percentage points for controlled before‐after studies. We present pooled results for randomized trials using the random‐effects model.

Main results

The 75 included studies involved child, adolescent, and adult participants in outpatient, community‐based, primary care, and other settings in 10 countries.

Patient reminder or recall interventions, including telephone and autodialer calls, letters, postcards, text messages, combination of mail or telephone, or a combination of patient reminder or recall with outreach, probably improve the proportion of participants who receive immunization (risk ratio (RR) of 1.28, 95% confidence interval (CI) 1.23 to 1.35; risk difference of 8%) based on moderate certainty evidence from 55 studies with 138,625 participants.

Three types of single‐method reminders improve receipt of immunizations based on high certainty evidence: the use of postcards (RR 1.18, 95% CI 1.08 to 1.30; eight studies; 27,734 participants), text messages (RR 1.29, 95% CI 1.15 to 1.44; six studies; 7772 participants), and autodialer (RR 1.17, 95% CI 1.03 to 1.32; five studies; 11,947 participants). Two types of single‐method reminders probably improve receipt of immunizations based on moderate certainty evidence: the use of telephone calls (RR 1.75, 95% CI 1.20 to 2.54; seven studies; 9120 participants) and letters to patients (RR 1.29, 95% CI 1.21 to 1.38; 27 studies; 81,100 participants).

Based on high certainty evidence, reminders improve receipt of immunizations for childhood (RR 1.22, 95% CI 1.15 to 1.29; risk difference of 8%; 23 studies; 31,099 participants) and adolescent vaccinations (RR 1.29, 95% CI 1.17 to 1.42; risk difference of 7%; 10 studies; 30,868 participants). Reminders probably improve receipt of vaccinations for childhood influenza (RR 1.51, 95% CI 1.14 to 1.99; risk difference of 22%; five studies; 9265 participants) and adult influenza (RR 1.29, 95% CI 1.17 to 1.43; risk difference of 9%; 15 studies; 59,328 participants) based on moderate certainty evidence. They may improve receipt of vaccinations for adult pneumococcus, tetanus, hepatitis B, and other non‐influenza vaccinations based on low certainty evidence although the confidence interval includes no effect of these interventions (RR 2.08, 95% CI 0.91 to 4.78; four studies; 8065 participants).

Authors' conclusions

Patient reminder and recall systems, in primary care settings, are likely to be effective at improving the proportion of the target population who receive immunizations.

Plain language summary

Do strategies to remind people to have vaccinations increase the number of people who receive vaccinations?

Aim of this review

The aim of this review is to determine whether strategies to remind people to receive vaccinations increase the number of people who receive vaccinations. This is an update of a previously published Cochrane Review.

Key messages

Reminding people to receive their vaccinations increases vaccination rates across different populations.

What was studied

Vaccinations are used to prevent a number of diseases but there is wide variation in vaccination coverage across different regions and countries. This can lead to diseases that are otherwise preventable by vaccines, having a large effect on individuals and communities. Informing people of an upcoming vaccination or telling them that they have missed a vaccination might help to increase coverage and reduce the effect and impact of disease preventable by vaccine. We reviewed 75 studies to evaluate whether reminding people to get vaccinated worked. The studies we looked at were from different settings, such as rural areas, schools, private practices, and state health departments. Most studies were done in the USA. The studies included a range of different groups: infants and children, adolescents and adults requiring routine vaccination, as well as adults who required the influenza vaccine. In most of the studies reminders took the form of person to person telephone calls, automated calls, letters or postcards. In few recent studies text messaging was used.

Main results of the review

Our review found that reminding people to have vaccinations likely increases the number of people who receive vaccinations by an average of 8 percentage points, although there was variation in the results of the studies. Reminding people by telephone and autodialer calls, sending a letter or postcard, or sending a text message increased vaccinations. Combinations of reminders were also effective. Reminding people over the telephone was more effective than the other types of reminders. The increases in vaccinations were observed among children, adolescents, and adults.

How up‐to‐date is this review?

We reviewed studies that were published to January 2017.

Summary of findings

Background

Description of the condition

Global coverage of routine immunizations varies widely (Hill 2016; European CDPC 2014; WHO 2016a). The global rate for three doses of diphtheria tetanus pertussis (DTP3) vaccine was estimated to be 86% in 2014, an increase of 66 percentage points since the 1980 level of 20% (WHO 2016a). However, approximately 18.7 million children did not receive DTP3 during 2014 (WHO 2016b). Rates vary by geographic area or country, with high levels of DTP3, ranging from 91% to 94% in the Americas, Europe, and the Western Pacific in 2014 (WHO 2016b). In 2014, the Eastern Mediterranean, South East Asia, and African regions had lower coverage rates of 73% to 84% (WHO 2016b). In the United States, immunization rates remain high and stable for infants and children, but coverage levels have not reached national goals for a number of vaccines, including the newer vaccinations introduced (Hill 2016; Seither 2016). Immunization rates remain nowhere near national coverage goals for influenza, human papillomavirus (HPV) vaccination, and adult vaccinations against herpes zoster or shingles and pneumococcal disease (Lu 2013; Reagan‐Steiner 2016; Stokley 2014). Vaccination rates also vary by state, race, and ethnicity (CDC 2016a; Hill 2015; Reagan‐Steiner 2015; Seither 2015; Williams 2015). For example, influenza immunization rates during the 2013 to 2014 influenza season varied by race or ethnicity, with reports of 46.7% among whites 19 years and older, compared with 36.5% among blacks, and 33.2% among Hispanics or Latinos (Williams 2016).

Burden of vaccine‐preventable diseases

As a result of unmet immunization goals, vaccine‐preventable diseases still have a significant effect in a number of countries (Clemmons 2015; European CDPC 2014; Williams 2016). This is evidenced by the continued occurrence of measles outbreaks, for example in Disneyland in California during 2015 (Clemmons 2015). In the European Union and three European Economic Area countries, 11,316 cases of measles were reported during 2012 (European CDPC 2014). The burden of vaccine‐preventable illnesses has also included approximately 226,000 influenza‐related hospitalizations, 3000 to 49,000 influenza‐related deaths, and 13,500 cases of invasive pneumococcal disease that occur each year in the US (Williams 2016). In Europe, an estimated four to 50 million symptomatic cases of influenza occur each year, and 15,000 to 70,000 deaths have been attributed to influenza annually (European CDPC 2016). Further, human papillomavirus (HPV) vaccination rates in the US and in some other nations are far lower than optimal, leading to many new cases of HPV infection and ultimately HPV‐related cancers (Viens 2016). Each year, 6.2 million persons are newly infected with HPV, and 26,000 new HPV‐related cancers are diagnosed in the US (Jemal 2013; Weinstock 2004). Cancers attributable to HPV infections lead to more than USD 4 billion in annual medical expenses in the US (Markowitz 2014).

Description of the intervention

Patient reminders notify populations, patients, or their parents or legal guardians of vaccines that are due because of age or other risk factors (AHRQ 2015; Jacobson 2016; Jacobson Vann 2005). The notification is delivered to populations or patients. Patient recalls refer to notifications of vaccines that are past due. For example, a letter sent to a patient at 60 years of age, informing her that she is now due for the shingles vaccine, is a patient reminder (Hales 2014; Jacobson Vann 2005). The patient who receives a notice at age 61 years about the shingles vaccine is getting a recall because the patient is past due, but would still benefit from receiving a shingles vaccination. Reminders and recalls require the source of the notification, whether a care provider, health services organization, public health authority, or community organization, to have access to the patient's contact information to facilitate the notification. The notification may be delivered by letter, postcard, telephone call, computerized telephone call, or text message (AHRQ 2015). The process would require the source of the notification to have determined the individual patient's status, in terms of being recommended for the vaccine by nature of the patient's age or risk condition or both, and not being vaccinated at the time of the notification.

How the intervention might work

Patient reminder or recall interventions work by addressing the common reasons that immunizations may be missed, such as forgetting or missing appointments, not knowing immunization schedules (Ahmed 2013; AHRQ 2015), and having concerns about vaccinations. The success of the patient or reminder recall intervention depends upon several factors, such as accuracy or currency of contact information, accuracy and completeness of vaccination records, viability of contact medium, readability or comprehensibility of the contact medium or message by the patient or patient's parent or legal guardian or caretaker, beliefs and attitudes about vaccinations, and access to health services or vaccinations (Esposito 2014; Pereira 2012; Thomas 2014).

Why it is important to do this review

In an era of increasing complexity of immunization schedules (CDC 1999a), rising expectations about the performance of primary care, and large demands on primary care physicians, it is important to understand and promote interventions that work in primary care settings. One strategy involves patient reminder or recall systems, which was recommended by the Task Force on Community Preventive Services (CPS Task Force 2016) and the Standards for Immunization Practices (National Vaccine Advisory Committee 2014). Despite the successes made in vaccinating populations under‐vaccination still occurs, resulting in vaccine‐preventable deaths and illnesses (Clemmons 2015; European CDPC 2014; Williams 2016).

Experts recommend that care providers utilize reminder or recall systems (CPS Task Force 2016; National Vaccine Advisory Committee 2014); however, there is evidence to suggest that few primary care providers actually use immunization reminder or recall systems, or both (Kempe 2012c; Pereira 2012; Tierney 2003). In a national survey of pediatric practices, only 16% of responding practices utilized them (Tierney 2003).

A range of different types of reminders and systems of recall are being implemented. Ten countries, including a number in Europe, have adopted computerized immunization information services or registries (Groom 2014). Immunization registries offer the potential to become the backbone of patient reminder or recall systems by identifying the populations at risk, providing algorithms to determine who is eligible based on vaccination recommendations, and providing systems to send postcards, letters, autodialer messages, and text message reminders (Kempe 2012c).

Centralized systems can include health systems, health maintenance organizations, practice 'networks' that share electronic health records, and state or national immunization registries. Several recent studies have demonstrated that centralized reminder and recall systems sent from health systems rather than from practices can raise immunization rates (Hofstetter 2015a; Hofstetter 2015b; Kempe 2013; Kempe 2015; Stockwell 2012a; Stockwell 2015; Szilagyi 2013). This strategy is intriguing because of the economies of scale that can be obtained from centralized reminder and recall.

This is an update of Jacobson Vann 2005 and Jacobson Vann 2008.

Objectives

To evaluate and compare the effectiveness of various types of patient reminder and recall interventions to improve receipt of immunizations.

Methods

Criteria for considering studies for this review

Types of studies

We included randomized trials, controlled before and after studies, and interrupted time series studies. We included good quality, controlled, non‐randomized studies to provide a sufficiently large number of studies to assess each intervention sub‐type. We excluded observational studies in which participants self‐selected to intervention groups. We reviewed the methods of each study to determine the study design, not relying on the authors' specified designs. We accepted a study that randomly selected intervention participants and selected matched controls in a non‐random fashion as a controlled before‐after design if pre‐intervention and post‐intervention immunization data were collected. Due to limited resources, we excluded non‐English language publications.

Types of participants

We included as participants children, from birth to 18 years, or adults who receive immunizations in any setting, including academic or non‐academic, and developed or developing countries. We excluded studies of patients who were hospitalized for the study duration.

Types of interventions

We included patient reminder or recall interventions, or both, that either reminded patients of upcoming immunizations or immunization visits that were due (reminders) or overdue (recall). Reminder and recall systems could be delivered by telephone, letter, postcard, text message, automatic electronic telephone calls (autodialer), within a secure online patient portal system, or in person, for example, a care provider giving a face‐to‐face reminder during a home visit, but not a clinic visit. Reminder and recall cues could also vary in specificity, number, and whether combined with other interventions, such as provider reminders or outreach. Specificity may vary from generic immunization reminders to personal reminders that address patient‐specific immunization needs. Frequency may be one‐time or multiple reminders. We included studies with multiple interventions if at least one study arm included immunization patient reminders or recall. We added text messages and messages occurring within patient portal systems in the current update.

Control activities

We included no‐intervention control groups, standard practice activities that did not include immunization‐focused patient reminder or recall interventions, media‐based activities aimed at promoting immunizations, and simple practice‐based immunization awareness campaigns.

Types of outcome measures

Our primary outcome measure was receipt of immunizations. We selected this outcome measure over other possible outcomes because of certain limitations, for example: the total number of vaccines would vary based on country, age group, and other factors; the proportion of the population that received all vaccines would depend on the specific population, such as age group; and on‐time vaccination is restrictive and is not expected to support clinicians' efforts to optimize receipt of vaccinations.

We accepted outcomes for individual vaccinations or standard combinations of recommended vaccinations, such as all recommended vaccinations by a specific date or age. If outcomes for a study were measured at multiple time points, we selected the outcomes designated as primary by the study authors. If unclear, we averaged outcomes over time periods.

We excluded immunizations that were sought for purposes of traveling to a destination where the disease may be widespread, and immunization orders or visits that did not also measure immunization status.

Search methods for identification of studies

We conducted searches of electronic databases, references lists of articles and reviews, grey literature, clinical trials websites, and identified articles from our team members, those already in use by our team for other clinical, teaching or project work, and experts in the field.

Electronic searches

For this update, we conducted a series of searches between February 2013 and 31 January 2017. We revised all our search strategies in January 2017 in order to reduce excess retrieval of papers and update the study design filter. We searched the Database of Abstracts of Reviews of Effects (DARE) for related systematic reviews and the following databases for primary studies on 31 January 2017:

Cochrane Central Register of Controlled Trials (CENTRAL; 2016, Issue 11) in the Cochrane Library;
Health Technology Assessment Database (HTA; 2016, Issue 4) in the Cochrane Library;
NHS Economic Evaluation Database (NHSEED; 2015, Issue 2) in the Cochrane Library;
MEDLINE Ovid, including Epub Ahead of Print, In‐Process & Other Non‐Indexed Citations: 1946 to 31 January 2017;
Embase Ovid: 1974 to 30 January 2017;
CINAHL EBSCO (Cumulative Index to Nursing and Allied Health Literature): 1981 to 31 January 2017.

The search strategies are presented in full in Appendix 1. Details of the previous search strategies are available in Jacobson Vann 2005 and Jacobson Vann 2008.

Searching other resources

We searched the reference lists of articles and reviews, and the files of study collaborators, for additional studies. Potentially relevant studies were also identified by experts in the field and by prior knowledge.

We conducted a grey literature search utilizing the sources below:

Agency for Healthcare Research and Quality website (www.ahrq.gov): searched March 2016;
ClinicalTrials.gov (www.clinicaltrials.gov): searched to February 2017;
World Health Organization International Clinical Trials Registry Platform (apps.who.int/trialsearch): searched 31 January 2017.

We also:

screened abstracts and conference proceedings (e.g. handsearching);
reviewed the reference lists of included studies, relevant systematic reviews, and primary studies;
contacted researchers with expertise relevant to the review topic or EPOC interventions.

Data collection and analysis

Selection of studies

One review author (JJV) screened all titles and abstracts, and two review authors (RMJ, TCB) independently reviewed half of all titles and abstracts.

Data extraction and management

Two review authors (JJV, RMJ, TCB) independently read and abstracted each potentially relevant study to assess for inclusion using a checklist developed by the Cochrane Effective Practice and Organisation of Care Group (EPOC) (EPOC 2017), and a supplemental form to collect new data elements to meet revised Cochrane guidelines. For each included study, we collected information on study design, study duration, intervention description, outcome measures, study findings, study setting, provider characteristics, participant characteristics, unit of allocation, unit of analysis, ethical approval, power or sample size calculations, and risk of bias assessments (EPOC 2017; Higgins 2011). We attempted to contact study authors when the data presented were not sufficiently detailed to include them in the analyses. One review author (JJV) compared both abstracts for each study and identified disagreements on inclusion and abstraction results. We resolved disagreements between review authors on study inclusion and abstraction results by a formal reconciliation process to achieve consensus.

We tracked the review process and status of each article and managed the study‐level data using Excel spreadsheets (Excel 2005). We assigned unique study identification numbers to support the tracking process. In the current update, we further tracked the major study activities and timelines in a project management system.

We entered dichotomous data from included randomized trials into RevMan 5 analysis data tables for intervention type, an overall summary measure, and participant‐immunization category: routine childhood, child influenza, other adult, adult influenza, and adolescent vaccinations.

Assessment of risk of bias in included studies

We assessed the risk of bias of the included studies using the Cochrane 'Risk of bias' tool (Higgins 2011). We scored each criterion, for each study, as low risk, unclear risk, or high risk of bias. We assessed the risk for the following.

Selection bias: random sequence generation, allocation concealment, and baseline measurement.
Performance bias: blinding of participants and personnel.
Detection bias: blinding of outcome assessment.
Attrition bias: incomplete outcome data.
Reporting bias: selective reporting.
Other bias: other sources of bias.

The results of our assessment are reported in the 'Risk of bias' tables (Characteristics of included studies), and are summarized by type of bias in our results (Risk of bias in included studies).

Measures of treatment effect

We used RevMan 5 to analyze data from randomized trials using risk ratios (RRs). We also computed absolute changes in immunizations received, as percentage point differences between pre‐intervention and post‐intervention measures because absolute changes are clinically meaningful to practitioners who make decisions about the types of interventions to adopt in their clinical practices. We computed absolute differences as post‐intervention immunization proportions minus pre‐intervention immunization proportions for each study group. We compared differences between study groups.

Unit of analysis issues

We did not combine data from randomized trials that allocated families, households, practices, or other clusters with trials that allocated individuals.

Dealing with missing data

When the reported data were insufficient to conduct our meta‐analyses, we attempted to contact study authors to obtain additional data, and analyzed all the available data (Pigott 2001). We excluded studies from the review if no relevant data were reported, based on the study protocol and minimum methodological inclusion criteria.

Assessment of heterogeneity

We used random‐effects models if Cochrane's Q test detected significant heterogeneity across studies. This test has low sensitivity, so we used a 0.1 significance level.

Assessment of reporting biases

We assessed for possible reporting biases, including publication bias, using a funnel plot (Egger 1997b). We created a funnel plot for the patient reminder or recall summary measure using the metafor package in R version 3.2.3 (Viechtbauer 2010), and also in RevMan 5 (Analysis 1.1), plotting the standard errors of log RRs against RRs (Sterne 2001). We also included a 95% confidence region based on a random‐effects model, where absence of bias is indicated by the inclusion of approximately 95% of the studies within this region (Sterne 2004).

1.1 — Comparison 1 Patient reminders (summary), Outcome 1 Immunized.

Data synthesis

We combined published data using random‐effects meta‐analysis for the number of people receiving immunizations. We grouped trials by population, including routine childhood, child influenza, other adult, adult influenza, and adolescent vaccinations, and the type of intervention. For studies with more than one patient reminder or with similar intervention types, such as two postcards groups with different messages, we combined intervention group data. For randomized trials with outcomes reported in multiple outcome categories, we reported each outcome separately by reminder type and combined data for the summary measure. For randomized trials with interventions delivered over time and multiple data collection points, we used average sample sizes.

We calculated risk ratios (RRs) and 95% confidence intervals (CIs) in RevMan for individual studies or study comparisons to assess the effect of patient reminder or recall interventions on receipt of immunizations. We computed pooled RRs and risk differences for each intervention type, and stratified the results within each intervention type by the five participant‐immunization categories. For example, we computed an overall pooled random‐effects RR for letter interventions, then stratified relevant study comparisons for participant‐immunization category.

Summary of findings

We summarized the findings of the main intervention comparison(s) for the most important outcome(s), specifically receipt of immunizations for each intervention type and participant‐immunization categories in two 'Summary of findings' tables. Two review authors (JJV, JKAA) independently assessed the certainty of the evidence or confidence in the estimate (high, moderate, low, and very low) for each outcome and intervention type using GRADE (BMJ 2016; Ryan 2016; Schünemann 2011). Our assessment included study design, risk of bias, inconsistency of effect size, indirectness, imprecision, and other considerations, including publication bias (EPOC 2017b). We present the certainty of evidence assessment results in GRADE evidence profiles (Appendix 2) and the 'Summary of findings' tables (Table 1; Table 2).

Summary of findings for the main comparison. Overview: Patient reminder or recall interventions for receipt of immunizations ‐ any kind.

Patient reminder or recall interventions compared with no patient reminder or recall for receipt of immunizations
Patient or population: children, adolescents, and adults with a need for routine immunizations, excluding travel immunizations Settings: patient telephone reminder or recall interventions are typically received in the home; the interventions originate from outpatient departments of hospitals, community‐based clinical settings, local and state public health departments, and other clinical settings Intervention: patient reminder or recall interventions Comparison: no‐intervention control groups, standard practice activities that did not include immunization‐focused patient reminder or recall interventions, media‐based activities aimed at promoting immunizations, and simple practice‐based immunization awareness campaigns
Intervention type	Outcome: received immunizations
	*Illustrative comparative risks (95% CI)**		Relative effect  (95% CI)	No of participants  (studies)	Certainty of the evidence  (GRADE)	Comments
	Assumed risk	Corresponding risk
	Without intervention	With intervention
Patient reminder or recall summary measure	290 per 1000	371 per 1000 (357 to 392)	RR 1.28^a (1.23 to 1.35)	138,625  (55)	Moderate^b	—
Patient telephone reminder or recall	164 per 1000	287 per 1000 (197 to 417)	RR 1.75 (1.20 to 2.54)	9120 (7)	Moderate^c	—
Patient letter reminder or recall	320 per 1000	412 per 1000 (387 to 442)	RR 1.29 (1.21 to 1.38)	81,100 (27)	Moderate^d	—
Patient postcard reminder or recall	327 per 1000	386 per 1000 (353 to 425)	RR 1.18 (1.08 to 1.30)	27,734 (8)	High^e	—
Patient text message reminder or recall	161 per 1000	208 per 1000 (185 to 232)	RR 1.29 (1.15 to 1.44)	7772 (6)	High	—
Patient autodialer message reminder or recall	365 per 1000	427 per 1000 (376 to 482)	RR 1.17 (1.03 to 1.32)	11,947 (5)	High	—
Combination of patient mail and telephone reminder or recall	277 per 1000	354 per 1000 (316 to 402)	RR 1.28 (1.14 to 1.45)	6506 (8)	Moderate^f	—
Combination of patient reminder or recall with outreach intervention	360 per 1000	439 per 1000 (396 to 486)	RR 1.22 (1.10 to 1.35)	2701 (3)	High	—
Combination of patient reminder or recall with provider reminder intervention	202 per 1000	588 per 1000 (540 to 644)	RR 2.91 (2.67 to 3.19)	4120 (2)	Moderate^g	—
The basis for the assumed risk,* e.g. the median control group risk across studies, is provided in footnotes. The corresponding risk, and its 95% confidence interval, is based on the assumed risk in the comparison group and the relative effect of the intervention, and its 95% CI.  CI: confidence interval; RR: risk ratio
GRADE Working Group grades of evidence  High quality: Further research is very unlikely to change our confidence in the estimate of effect.  Moderate quality: Further research is likely to have an important impact on our confidence in the estimate of effect and may change the estimate.  Low quality: Further research is very likely to have an important impact on our confidence in the estimate of effect and is likely to change the estimate.  Very low quality: We are very uncertain about the estimate.

Patient reminder or recall intervention for receipt of immunization, by type of immunization
Patient or population: children, adolescents, and adults with a need for routine immunizations, excluding travel immunizations Settings: patient reminder or recall interventions are typically received in the home; the interventions originate from outpatient departments of hospitals, community‐based clinical settings, local and state public health departments, and other clinical settings Interventions: patient reminder or recall interventions, including telephone calls, autodialer calls, letters, postcards, text messages, combination of mail or telephone, or combination of patient reminder or recall with outreach; this summary measure excludes patient reminder or recall interventions combined with provider reminders Comparison: no‐intervention control groups, standard practice activities that did not include immunization‐focused patient reminder or recall interventions, media‐based activities aimed at promoting immunizations, and simple practice‐based immunization awareness campaigns
Outcomes	*Illustrative comparative risks (95% CI)**		Relative effect  (95% CI)	No of participants  (studies)	Certainty of the evidence  (GRADE)	Comments
	Assumed risk	Corresponding risk
	Without intervention	With intervention
Childhood immunizations	333 per 1000	406 per 1000 (383 to 430)	RR 1.22 (1.15 to 1.29)	31,099 (23)	High^a	—
Childhood influenza immunizations	431 per 1000	651 per 1000 (491 to 857)	RR 1.51 (1.14 to 1.99)	9265 (5)	Moderate^b	—
Adult immunizations ‐ other than influenza or travel ('Other adult')	109 per 1000	227 per 1000 (99 to 521)	RR 2.08 (0.91 to 4.78)	8065 (4)	Low^c	—
Adult influenza immunizations	292 per 1000	376 per 1000 (342 to 418)	RR 1.29 (1.17 to 1.43)	59,328 (15)	Moderate^d	—
Adolescent immunizations	244 per 1000	314 per 1000 (285 to 346)	RR 1.29 (1.17 to 1.42)	30,868 (10)	High^e	—
The basis for the assumed risk,* e.g. the median control group risk across studies, is provided in footnotes. The corresponding risk, and its 95% confidence interval, is based on the assumed risk in the comparison group and the relative effect of the intervention, and its 95% CI.  CI: confidence interval; RR: risk ratio
GRADE Working Group grades of evidence  High quality: Further research is very unlikely to change our confidence in the estimate of effect.  Moderate quality: Further research is likely to have an important impact on our confidence in the estimate of effect and may change the estimate.  Low quality: Further research is very likely to have an important impact on our confidence in the estimate of effect and is likely to change the estimate.  Very low quality: We are very uncertain about the estimate.

Group or subgroup	RR (CI) for full set of included studies	RR (CI) after deleting studies with 'high' risk of bias for random sequence generation, allocation concealment, and/or incomplete outcomes	RR (CI) after deleting studies with primary outcome of received all needed vaccinations
Summary measure	1.28 (1.23 to 1.35)	1.29 (1.23 to 1.36)	1.32 (1.25 to 1.39)
Child	1.22 (1.15 to 1.29)	1.19 (1.12 to 1.27)	1.24 (1.15 to 1.34)
Influenza – child	1.51 (1.14 to 1.99)	1.51 (1.14 to 1.99)	1.37 (1.05 to 1.77)
Adult – other	2.08 (0.91 to 4.78)	2.35 (2.02 to 2.74)	2.08 (0.91 to 4.78)
Influenza – adult	1.29 (1.17 to 1.43)	1.33 (1.20 to 1.48)	1.29 (1.17 to 1.43)
Adolescent	1.29 (1.17 to 1.42)	1.26 (1.15 to 1.39)	1.33 (1.20 to 1.48)

Date	Event	Description
31 January 2017	New search has been performed	Updated searches conducted to 31 January 2017.
31 January 2017	New citation required but conclusions have not changed	Added text messages as a new reminder or recall intervention. Changed statistical method from odds ratio to risk ratio. Updated the review methods and reporting to align with current Cochrane and EPOC guidance. Added one additional author, a biostatistician.

Date	Event	Description
30 June 2016	New search has been performed	Updated searches conducted to 31 May 2013; 22 new studies identified. This review includes 69 studies.
12 November 2008	Amended	Minor changes.
14 August 2008	New citation required but conclusions have not changed	New search July 2007; four new studies.
12 June 2008	Amended	Converted to new review format.
15 February 2008	New search has been performed	New searches; no changes to findings.
25 May 2005	New citation required and conclusions have changed	Substantive amendment.

No.	Search terms	Results
1	reminder systems/	2870
2	preventive health services/	12000
3	primary prevention/	16132
4	immunization/	47274
5	vaccination/	70736
6	or/2‐5	141554
7	1 and 6	305
8	(postcard? or post‐card? or mail* or text messag* or sms or short messag* service or letter? or brochur* or pamphlet?).ti,ab.	126817
9	(recall or remind*).ti,ab.	58576
10	8 or 9	182258
11	(vaccine? or vaccinat* or immunis* or immuniz* or flu).ti,ab.	331177
12	10 and 11	3890
13	((vaccine? or vaccination? or immunization? or immunisation?) adj3 (registry or registries)).ti,ab.	461
14	or/7,12‐13	4431
15	randomized controlled trial.pt.	446587
16	controlled clinical trial.pt.	91788
17	multicenter study.pt.	217408
18	pragmatic clinical trial.pt.	521
19	(randomis* or randomiz* or randomly).ti,ab.	723931
20	groups.ab.	1670897
21	(trial or multicenter or multi center or multicentre or multi centre).ti.	205750
22	(intervention? or effect? or impact? or controlled or control group? or (before adj5 after) or (pre adj5 post) or ((pretest or pre test) and (posttest or post test)) or quasiexperiment* or quasi experiment* or pseudo experiment* or pseudoexperiment* or evaluat* or time series or time point? or repeated measur*).ti,ab.	7875589
23	non‐randomized controlled trials as topic/	119
24	interrupted time series analysis/	236
25	controlled before‐after studies/	213
26	or/15‐25	8803258
27	exp animals/	20689610
28	humans/	16378297
29	27 not (27 and 28)	4311313
30	review.pt.	2218518
31	meta analysis.pt.	74177
32	news.pt.	180911
33	comment.pt.	677926
34	editorial.pt.	424399
35	cochrane database of systematic reviews.jn.	12903
36	comment on.cm.	677925
37	(systematic review or literature review).ti.	89860
38	or/29‐37	7513987
39	26 not 38	6141300
40	14 and 39	1849

No of studies	Design	Risk of bias	Inconsistency	Indirectness^b	Imprecision	Other^c	Certainty (overall score)^d
Outcome: Receipt of immunizations ‐ for summary measure
55	Randomized trials	No serious risk of bias	Some inconsistency exists (‐0.5)	No serious indirectness	Some imprecision exists (‐0.5)	None	Moderate (3)

First author last name	Year of publication	Study designs (# of patient reminder‐recall studies)	Participants	Types of immunizations	Results
Groom	2015	Not clear (30)	Persons in high‐income countries	Vaccination rates	Median percentage point improvement: 6
Harvey	2015	Controlled studies (28 overall)	Children	Childhood	Risk difference (RD)²:  Postal: RD 0.11 (95% Confidence Interval (CI) 0.08 to 0.13)  Telephone: RD 0.04 (95% CI 0.01 to 0.07)
Niccolai	2015	Randomized (4 patient reminder recall (PRR)) and non‐randomized (3 PRR)	Adolescents	HPV	All 7 studies reported increases in at least 1 HPV vaccination outcome
Odone	2015	Observational or experimental (7 text messages; 1 smartphone application)	Parents of children and adolescents, pregnant women, providers	Variety of vaccination outcomes	Some evidence that text messaging, patient‐held web‐based portals and computerized reminders were effective
Oyo‐Ita	2011	Randomized trials, non‐randomized trials, interrupted time series (1)	Children in low‐ and middle‐income countries	Childhood	Not clear; study was patient card, not matching our eligibility criteria
Thomas	2014	Randomized trials (4)	60 years and older	Influenza	Pooled odds ratio 1.11 (95% CI 1.07 to 1.15) and 3.33 (95% CI 1.79 to 6.22)
Watterson	2015	Randomized trials and observational studies (3 vaccine)	Children in low‐ or middle‐income country	Childhood	9.7 percentage point increase in one study; positive qualitative findings in other studies
Williams	2011	Randomized trials (26), before and after studies (11), controlled intervention (9) (22 PRR)	Children in developed countries	Childhood	Median percentage point change of 11%; range of ‐11 to 24 percentage points

Outcome or subgroup title	No. of studies	No. of participants	Statistical method	Effect size
1 Immunized	55	138625	Risk Ratio (M‐H, Random, 95% CI)	1.28 [1.23, 1.35]
1.1 Childhood immunizations	23	31099	Risk Ratio (M‐H, Random, 95% CI)	1.22 [1.15, 1.29]
1.2 Childhood influenza immunizations	5	9265	Risk Ratio (M‐H, Random, 95% CI)	1.51 [1.14, 1.99]
1.3 Adult immunizations ‐ other	4	8065	Risk Ratio (M‐H, Random, 95% CI)	2.08 [0.91, 4.78]
1.4 Adult influenza immunizations	15	59328	Risk Ratio (M‐H, Random, 95% CI)	1.29 [1.17, 1.43]
1.5 Adolescent immunizations	10	30868	Risk Ratio (M‐H, Random, 95% CI)	1.29 [1.17, 1.42]

Outcome or subgroup title	No. of studies	No. of participants	Statistical method	Effect size
1 Immunized	7	9120	Risk Ratio (M‐H, Random, 95% CI)	1.75 [1.20, 2.54]
1.1 Childhood immunizations	2	234	Risk Ratio (M‐H, Random, 95% CI)	2.27 [1.12, 4.63]
1.2 Adult immunizations ‐ other	2	6630	Risk Ratio (M‐H, Random, 95% CI)	1.58 [0.55, 4.57]
1.3 Adult influenza immunizations	2	1838	Risk Ratio (M‐H, Random, 95% CI)	1.53 [0.73, 3.20]
1.4 Adolescent immunizations	1	418	Risk Ratio (M‐H, Random, 95% CI)	2.04 [1.05, 3.95]

Outcome or subgroup title	No. of studies	No. of participants	Statistical method	Effect size
1 Immunized	27	81100	Risk Ratio (M‐H, Random, 95% CI)	1.29 [1.21, 1.38]
1.1 Childhood immunizations	9	13009	Risk Ratio (M‐H, Random, 95% CI)	1.16 [1.06, 1.27]
1.2 Childhood influenza immunizations	5	9265	Risk Ratio (M‐H, Random, 95% CI)	1.51 [1.14, 1.99]
1.3 Adult immunizations ‐ other	2	1435	Risk Ratio (M‐H, Random, 95% CI)	3.13 [1.44, 6.84]
1.4 Adult influenza immunizations	11	44454	Risk Ratio (M‐H, Random, 95% CI)	1.35 [1.19, 1.52]
1.5 Adolescent immunizations	2	12937	Risk Ratio (M‐H, Random, 95% CI)	1.91 [0.71, 5.11]

No.	Search terms	Results
#1	[mh "reminder system"]	765
#2	[mh "preventive health service"]	28875
#3	[mh "primary prevention"]	4122
#4	[mh immunization]	4715
#5	[mh vaccination]	2485
#6	{or #2‐#5}	30429
#7	#1 and #6	328
#8	(postcard? or post‐card? or mail* or text messag* or sms or short messag* service or letter? or brochur* or pamphlet?):ti,ab	7408
#9	(recall or remind*):ti,ab	6902
#10	#8 or #9	13394
#11	(vaccine? or vaccinat* or immunis* or immuniz* or flu):ti,ab	12154
#12	#10 and #11	384
#13	((vaccine? or vaccination? or immunization? or immunisation?) near/3 (registry or registries)):ti,ab	2
#14	{or #7, #12‐#13}	621

No.	Search terms		Results
S1	MH Reminder Systems		1,517
S2	(MH "Preventive Health Care")		10,568
S3	(MH "Immunization+")		14,501
S4	S2 OR S3		24,671
S5	S1 AND S4		170
S6	postcard? or post‐card? or mail* or text messag* or sms or short messag* service or letter? or brochur* or pamphlet?		31,509
S7	recall or remind*		13,658
S8	S6 OR S7		44,296
S9	vaccine? or vaccinat* or immunis* or immuniz* or flu		35,591
S10	S8 AND S9		804
S11	((vaccine? or vaccination? or immunization? or immunisation?) N3 (registry or registries))		10
S12	S5 OR S10 OR S11		855
S13	PT randomized controlled trial		30,865
S14	PT clinical trial		52,906
S15	PT research		996,158
S16	(MH "Randomized Controlled Trials")	30,097
S17	(MH "Clinical Trials")	87,564
S18	(MH "Intervention Trials")	6,156
S19	(MH "Nonrandomized Trials")	182
S20	(MH "Experimental Studies")	15,224
S21	(MH "Pretest‐Posttest Design+")	27,974
S22	(MH "Quasi‐Experimental Studies+")	8,859
S23	(MH "Multicenter Studies")	21,526
S24	(MH "Health Services Research")	7,563
S25	TI (randomis* or randomiz* or randomly) OR AB (randomis* or randomiz* or randomly)	119,848
S26	TI (trial or effect* or impact* or intervention* or before N5 after or pre N5 post or ((pretest or "pre test") and (posttest or "post test")) or quasiexperiment* or quasi W0 experiment* or pseudo experiment* or pseudoexperiment* or evaluat* or "time series" or time W0 point* or repeated W0 measur) OR AB (trial or effect or impact* or intervention* or before N5 after or pre N5 post or ((pretest or "pre test") and (posttest or "post test")) or quasiexperiment* or quasi W0 experiment* or pseudo experiment* or pseudoexperiment* or evaluat* or "time series" or time W0 point* or repeated W0 measur*)	809,802
S27	S13 OR S14 OR S15 OR S16 OR S17 OR S18 OR S19 OR S20 OR S21 OR S22 OR S23 OR S24 OR S25 OR S26	1,347,398
S28	S12 AND S27	650

Methods	Study design: randomized trial Study duration: participants followed for 14 weeks; recruitment began 1 January 2013; study follow‐up ended 31 August 2013 Study aim: determine effectiveness of short message service reminders on immunization receipt
Participants	Inclusion: women or caregivers were recruited into study soon after delivery or during third and seventh day visits after delivery of baby; must have cell phone and resident of Kadoma city  Age: mothers of infants Exclusion: no cell phone  Setting: clinics in Zimbabwe n = 304
Interventions	Intervention group: short message service reminders indicating next appointment date and health education; 7, 3, and 1 day before immunization due date; repeated for 6‐, 10‐, and 14‐week appointments; message indicated immunization protects your child against deadly diseases, and reminder of vaccination appointment; n = 152 Control group: informed mothers or caregivers about next scheduled immunization visit and provided routine health education; n = 152
Outcomes	Receipt of scheduled vaccines at 6, 10, and 14 weeks 6 weeks OPV1, Penta1 and PCV: 97% versus 82%; 15 percentage point difference 10 weeks OPV2, Penta2 and PCV2: 96% versus 80%; 16 percentage point difference 14 weeks OPV3, Penta3 and PCV3: 95% versus 75%; 20 percentage point difference
Notes	Immunizations may have been measured at the date due or day after
*Risk of bias*
Bias	Authors' judgement	Support for judgement
Random sequence generation (selection bias)	Low risk	Participants were assigned by computer‐generated random numbers to intervention and control groups
Allocation concealment (selection bias)	Low risk	Participants were assigned by computer‐generated random numbers to intervention and control groups
Blinding of participants and personnel (performance bias)   All outcomes	Unclear risk	Blinding of participants and personnel not specified
Blinding of outcome assessment (detection bias)   All outcomes	Unclear risk	Blinding of outcome assessment not specified Entered and analyzed data in Epi Info 7
Incomplete outcome data (attrition bias)   All outcomes	Low risk	152 participants randomized and analyzed in each group; obtained outcomes by telephone follow‐up and clinic immunization registry; compared data sources
Selective reporting (reporting bias)	Low risk	Reported pre‐specified outcomes of interest
Other bias	Low risk	Did not identify other sources of bias
Baseline measurement	Low risk	Reported characteristics of mothers; similar for marital status, place of residence, educational levels, employment status, religion, and median age

Methods	Study design: group randomized trial Study duration: followed each infant for 9 to 12 months until 12 months of age; recruited from August to November 2012; cell phone reminder and recall occurred for 14 months, 2012 August through 2013 September Study aim: evaluated the effect of reminder‐recall intervention and primary care immunization provider training on routine immunization completion among infants
Participants	Inclusion: age 0 to 12 weeks at first immunization visits; parents living in study communities  Age: up to 12 months; 0 to 12 weeks of age at recruitment Exclusion: no cell phone; infant died; left service area  Setting: immunization clinics in Ibadan, Southwest Nigeria n = 605
Interventions	Intervention group 1: 2 cell reminder phone calls to child's parent or other contact person; made 2 and 1 day before immunization appointment; recall for missed appointments; n = 148 Intervention group 2: Primary Health Care Immunization Providers’ Training (PHCIPT); 2 days refresher training on theory and practice of immunization conducted for nurses, midwives, community health officers, and community health extension workers; 4 modules adapted from World Health Organization immunization training manual; not our intervention; n = 150 Intervention group 3: telephone reminder and recall with provider training; n = 147 Control group: usual care; no intervention; n = 150
Outcomes	Receipt of routine immunizations at 12 months of age Intervention group 1: 98.6% versus 57.3%; 41.3 percentage point difference
Notes	Data not included in meta‐analyses
*Risk of bias*
Bias	Authors' judgement	Support for judgement
Random sequence generation (selection bias)	Unclear risk	Randomly selected 2 local government areas from urban and 2 from suburban area; used ballot system to allocate areas into 3 intervention and 1 control group; randomly selected 1 ward from each area and purposively selected 1 primary health services facility from each ward
Allocation concealment (selection bias)	Unclear risk	As above
Blinding of participants and personnel (performance bias)   All outcomes	Unclear risk	Blinding of participants and personnel was not specified
Blinding of outcome assessment (detection bias)   All outcomes	Unclear risk	Outcome assessment blinding was not specified Researchers and research assistants used paper‐based immunization data system; integrated data collection into health facility activities; collected data using immunization records and cards, and qualitative feedback from mothers in reminder‐recall group
Incomplete outcome data (attrition bias)   All outcomes	Low risk	10 of 605 participants were lost to follow‐up
Selective reporting (reporting bias)	Low risk	Outcomes reported for study questions
Other bias	Low risk	Did not identify other sources of bias
Baseline measurement	High risk	Groups were similar with respect to mother’s age Groups differed for children’s mean age at first immunization visit, children’s sex, family type, birth order, family’s religion, maternal education, and place of delivery

Outcome or subgroup title	No. of studies	No. of participants	Statistical method	Effect size
1 Immunized	8	27734	Risk Ratio (M‐H, Random, 95% CI)	1.18 [1.08, 1.30]
1.1 Childhood immunizations	4	2806	Risk Ratio (M‐H, Random, 95% CI)	1.24 [1.05, 1.46]
1.2 Adult influenza immunizations	3	19265	Risk Ratio (M‐H, Random, 95% CI)	1.15 [0.95, 1.39]
1.3 Adolescent immunizations	1	5663	Risk Ratio (M‐H, Random, 95% CI)	1.23 [0.98, 1.54]

Outcome or subgroup title	No. of studies	No. of participants	Statistical method	Effect size
1 Immunized	6	7772	Risk Ratio (M‐H, Random, 95% CI)	1.29 [1.15, 1.44]
1.1 Childhood immunizations	1	304	Risk Ratio (M‐H, Random, 95% CI)	1.22 [1.11, 1.33]
1.2 Adult influenza immunizations	1	204	Risk Ratio (M‐H, Random, 95% CI)	1.05 [0.71, 1.58]
1.3 Adolescent immunizations	4	7264	Risk Ratio (M‐H, Random, 95% CI)	1.38 [1.16, 1.64]

Outcome or subgroup title	No. of studies	No. of participants	Statistical method	Effect size
1 Immunized	5	11947	Risk Ratio (M‐H, Random, 95% CI)	1.17 [1.03, 1.32]
1.1 Childhood immunizations	3	8583	Risk Ratio (M‐H, Random, 95% CI)	1.27 [1.19, 1.35]
1.2 Adolescent immunizations	2	3364	Risk Ratio (M‐H, Random, 95% CI)	1.08 [0.99, 1.17]

Outcome or subgroup title	No. of studies	No. of participants	Statistical method	Effect size
1 Immunized	8	6506	Risk Ratio (M‐H, Random, 95% CI)	1.28 [1.14, 1.45]
1.1 Childhood immunizations	7	4910	Risk Ratio (M‐H, Random, 95% CI)	1.27 [1.09, 1.48]
1.2 Adolescent immunizations	1	1596	Risk Ratio (M‐H, Random, 95% CI)	1.36 [1.20, 1.53]

Outcome or subgroup title	No. of studies	No. of participants	Statistical method	Effect size
1 Immunized	3	2701	Risk Ratio (M‐H, Random, 95% CI)	1.22 [1.10, 1.35]
1.1 Childhood immunizations	3	2701	Risk Ratio (M‐H, Random, 95% CI)	1.22 [1.10, 1.35]

Outcome or subgroup title	No. of studies	No. of participants	Statistical method	Effect size
1 Immunized	2	4120	Risk Ratio (M‐H, Random, 95% CI)	2.91 [2.67, 3.19]
1.1 Adult immunizations ‐ other	1	264	Risk Ratio (M‐H, Random, 95% CI)	4.07 [1.13, 14.70]
1.2 Adult influenza immunizations	2	3856	Risk Ratio (M‐H, Random, 95% CI)	2.91 [2.66, 3.18]

Methods	Study design: randomized trial; allocated children without grouping children in same family  Study duration: 6 months; 1 January 1991 to 30 June 1991 Study aim: evaluate effectiveness of mail and telephone interventions in increasing immunization rates among children less than 7 years of age in family practice residency clinic
Participants	Inclusion: children actively enrolled in family practice residency clinic; not up‐to‐date with immunizations  Age: older than 2 months of age as of 1 January 1991 and less than 7 years as of 30 June 1991  Setting: family practice residency clinic (USA)  n = 464 randomized, 446 analyzed
Interventions	Intervention: sent postcard reminder to parents, indicating types of immunizations needed by child, and urging parents to make appointment; made telephone calls to parents of unimmunized children, 6 weeks after postcard intervention; written in English; n = 213  Control: no intervention; no special contact; n = 233
Outcomes	Number and percent of children immunized: intervention 8.8 percentage point increase over controls  Number and percent of children receiving all needed immunizations: intervention 8.7 percentage point increase over controls
Notes	13% of postcards sent were returned as undeliverable Approximately 1% of families in practice Spanish‐speaking; postcards may not have been understood; 17.8% of telephones were disconnected 41 of 177 intervention families not reached by telephone Results seem to be inconsistently reported for children brought up‐to‐date with immunizations; reversed in study abstract compared with results in Table 3
*Risk of bias*
Bias	Authors' judgement	Support for judgement
Random sequence generation (selection bias)	Low risk	"immunization records of the 519 children were entered into a minicomputer” Probably randomized within computer even though method not explicitly specified
Allocation concealment (selection bias)	Unclear risk	Charts of infants were reviewed for immunizations received; supplemented these data with immunizations recorded in health department registry; entered immunization records into minicomputer prior to randomizing children; procedures not explicitly described
Blinding of participants and personnel (performance bias)   All outcomes	Unclear risk	No information provided
Blinding of outcome assessment (detection bias)   All outcomes	Unclear risk	Blinding not specified for review of practice billing codes and charts
Incomplete outcome data (attrition bias)   All outcomes	Unclear risk	Number of reviewed records not specified; asked parents by telephone about immunizations received Immunization status confirmed by reviewing practice billing codes and charts and county health department's records Did not confirm or record immunizations received at other sites
Selective reporting (reporting bias)	Low risk	Study purpose and methods are described; published data included all expected outcomes
Other bias	Low risk	Randomized patients within residency clinic; authors noted there may have been some contamination of control group because children with different surnames living in same household could have been assigned to different study groups
Baseline measurement	Low risk	"Immunization records of the county health department were reviewed." At baseline, reviewed charts of all infants actively enrolled in practice and older than 2 months and less than 7 years of age; supplemented information from health department immunization registry; selected participants behind on immunizations

Methods	Study design: randomized trial  Study duration: perhaps 1 influenza season; mailed reminders during third week of September 1995 Study aim: evaluate effect of 3 types of computer‐generated mailed reminders on influenza immunization rates
Participants	Inclusion: adult patients aligned with primary care physician within health system and at high risk for influenza complications based on age 65 years or older, or diagnosis of asthma, diabetes, end stage renal disease, sickle cell disease, ischemic cardiomyopathy, or nephrotic syndrome  Age: adults; mean age = 67.2 years  Setting: multispecialty group practice that serves patients in health system's affiliated nonprofit, mixed‐model health maintenance organization and patients in other fee‐for‐service health financing programs; southeastern Michigan (USA)  n = 24,743 randomized
Interventions	Intervention group 1: generic postcard to patient, standard message; n = 6169 Intervention group 2: personalized postcard from patient's primary care physician; n = 6252 Intervention group 3: personalized letter from the patient's primary care physician, addressed to specific patient; message tailored to specific health risk of patient; n = 6151  Control: no reminder, but comprehensive immunization program for all 4 groups; n = 6171 Comprehensive program included: walk‐in influenza vaccination clinics during October at all health system outpatient clinic locations, display of posters and take‐home postcards in clinic entrances and waiting areas, toll‐free information telephone line, developed program logo and theme used in all print media, and standard message in printed materials was based on Health Belief Model
Outcomes	Number and percent receiving influenza vaccination  Group 1: 2.9 percentage point increase over control group  Group 2: 4.1 percentage point increase over control group  Group 3: 4.6 percentage point increase over control group
Notes	Patient reminders were one component of comprehensive influenza immunization program. Used billing data to calculate rate of immunizations in study groups; some vaccinations may have been received at unaffiliated clinics, some of which provided vaccinations free of charge Authors identified possible threshold effect
*Risk of bias*
Bias	Authors' judgement	Support for judgement
Random sequence generation (selection bias)	Unclear risk	Patients were identified as being eligible for study using computerized appointment scheduling system and demographic data and computerized billing data; patients were randomized to 4 groups; method is not described
Allocation concealment (selection bias)	Unclear risk	Randomized patients into one of 4 groups; method not described
Blinding of participants and personnel (performance bias)   All outcomes	Unclear risk	No information provided
Blinding of outcome assessment (detection bias)   All outcomes	Low risk	Measured outcomes using billing data; blinding not specified
Incomplete outcome data (attrition bias)   All outcomes	Unclear risk	Used billing data to obtain immunization rates Authors note possibility that some participants may have received immunizations at non‐study clinics, some of which offered free vaccinations
Selective reporting (reporting bias)	Low risk	Study purpose and methods are described; published data included all expected outcomes to answer study questions
Other bias	Low risk	Study seems to be free of other sources of bias
Baseline measurement	Low risk	Identified eligible patients using computerized billing data Obtained data on date of birth, sex, race, and marital status Demographic characteristics similar between study groups

Methods	Study design: randomized trial, stratified by age and diagnosis  Study duration: 2‐week influenza shot clinic in October of study year Study aim: evaluate and compare effectiveness of letters and informational brochure on increasing influenza vaccination among persons who had not received vaccine in prior year and at high risk of getting influenza or complications
Participants	Inclusion: patients cared for in general medical clinic of 1 hospital; at high risk for influenza complications Exclusion: received influenza vaccination in previous year; living in nursing home; severe disabling mental, visual, or hearing impairment Defined high risk as: 65 years and older, or diagnosed with diabetes, chronic lung disorders, or chronic heart disorders  Age: adults  Setting: Veterans Administration Medical Center, general medical clinic, Seattle, Washington (USA)  n = 284 patients of 1093 eligible
Interventions	Intervention group 1: standard letter and informational brochure; developed using multi‐attribute utility‐based messages; sent to patients approximately 10 days before 2‐week special flu shot clinic in October; n = 66 Intervention group 2: augmented letter; added statement to standard letter that 70% of veterans from medical center were vaccinated last year; n = 57 Intervention group 3: augmented letter and informational brochure; n = 55  Control: standard letter; considered standard practice because it was in use prior to study; mentioned that flu season is approaching, potential risk for getting flu complication because at high risk, safety and effectiveness of flu vaccine, recommendation by doctor to receive flu shot each year, and time and location of flu shot clinic; signed by clinic chief; n = 57
Outcomes	Number and percent receiving influenza immunization  Group 1: 13 percentage point increase over standard letter control group  Group 2: 7 percentage point increase over standard letter control group  Group 3: 23 percentage point increase over standard letter control group
Notes	Control group includes patient reminder (standard letter), so no true control group Study participants had not received influenza vaccination in previous year, not general population Active influenza vaccination program had been operational in study setting since 1978; included sending mailed letters to patients at high risk of influenza, inviting them to receive vaccine
*Risk of bias*
Bias	Authors' judgement	Support for judgement
Random sequence generation (selection bias)	Unclear risk	Patients identified as high risk were stratified into risk groups and randomly assigned to one of 4 groups; allocation method not described
Allocation concealment (selection bias)	Unclear risk	Stratified high risk patients without history of influenza vaccination at the start of year 2 of larger study into those 65 years and older and less than 65 years, then randomly assigned to 1 of 4 groups; allocation process not described
Blinding of participants and personnel (performance bias)   All outcomes	Unclear risk	Blinding of participants and personnel not specified
Blinding of outcome assessment (detection bias)   All outcomes	Unclear risk	Blinding not specified Mailed vaccination status questionnaire to each participant to obtain outcome data; conducted second mailing and telephone follow‐up, if needed; also used clinic vaccination records
Incomplete outcome data (attrition bias)   All outcomes	Low risk	83% of participants remained in study at end of intervention Compared self‐report of immunization with clinic records; 94% agreement between data sources
Selective reporting (reporting bias)	Unclear risk	Study purpose and methods are described; published data included all expected outcomes
Other bias	High risk	Control group includes patient reminder (standard letter), so no true control group Study participants had not received influenza vaccination in previous year, so not general population
Baseline measurement	Low risk	All eligible patients had not received an influenza vaccination in the prior year

Methods	Study design: randomized intervention study Study duration: assessed HPV vaccination status 3 months after mailing; 12‐month evaluation period; 13 February 2013 to 12 February 2014 Study aim: evaluated effectiveness of reminder letter on HPV vaccine 3‐dose series completion
Participants	Inclusion: female members of health system for at least 1 year prior to study; received at least 1 dose of HPV4 during 3‐month period before 13 February 2013; valid address in membership file Age: 9 to 26 years when received first HPV4 dose Exclusion: more than 2 doses of HPV4; unresolved pregnancy; had not met the minimum HPV vaccine dosing intervals specified by ACIP; terminated health plan membership during evaluation period Setting: Kaiser Permanente Southern California Health Plan n = 12,205
Interventions	Customized reminder letter; 9th or 10th grade reading level; English and Spanish; indicating HPV4 immunization schedule, date of first dose and telephone numbers; encouraging follow‐up vaccination visits; sent to patients if 12 to 26 years and to parents if 9 to 11 years; 4 waves of mailings were scheduled quarterly; therefore, letters did not reach participants when a dose was due; n = 9760 Control group: usual care; author does not have information about reminder or recall systems used in individual clinical practices; n = 2445 System‐wide provider reminders in electronic medical records for intervention and control group
Outcomes	HPV vaccine 3‐dose series completion Intervention group: 56.4% versus 46.6%; 9.8 percentage point difference
Notes	Inconsistency in study method descriptions
*Risk of bias*
Bias	Authors' judgement	Support for judgement
Random sequence generation (selection bias)	Unclear risk	Randomization process not described; inconsistent description of allocation process; either 80% of eligible persons were randomly selected for intervention group and 20% for control group; or patients were randomized
Allocation concealment (selection bias)	Unclear risk	Randomization process not described
Blinding of participants and personnel (performance bias)   All outcomes	Unclear risk	Blinding of participants or personnel not described
Blinding of outcome assessment (detection bias)   All outcomes	Unclear risk	Blinding of outcome assessment and outcomes data source(s) not described
Incomplete outcome data (attrition bias)   All outcomes	Unclear risk	Data source(s) and follow‐up not described Not all reminder letters were successfully delivered; some were returned as undeliverable (n = 388; 4%); intention‐to‐treat analysis
Selective reporting (reporting bias)	Low risk	Reported results for all study questions
Other bias	Unclear risk	Methods are not fully described
Baseline measurement	Low risk	HPV4 vaccination history, including 1 or 2 doses, and age, race and ethnicity, Medicaid insurance, and length of managed care membership were reported and similar across study groups

PERMALINK

Patient reminder and recall interventions to improve immunization rates

Julie C Jacobson Vann

Robert M Jacobson

Tamera Coyne‐Beasley

Josephine K Asafu‐Adjei

Peter G Szilagyi

Abstract

Background

Objectives

Search methods

Selection criteria

Data collection and analysis

Main results

Authors' conclusions

Plain language summary

Summary of findings

Background

Description of the condition

Burden of vaccine‐preventable diseases

Description of the intervention

How the intervention might work

Why it is important to do this review

Objectives

Methods

Criteria for considering studies for this review

Types of studies

Types of participants

Types of interventions

Control activities

Types of outcome measures

Search methods for identification of studies

Electronic searches

Searching other resources

Data collection and analysis

Selection of studies

Data extraction and management

Assessment of risk of bias in included studies

Measures of treatment effect

Unit of analysis issues

Dealing with missing data

Assessment of heterogeneity

Assessment of reporting biases

1.1. Analysis.

Data synthesis

Summary of findings

Summary of findings for the main comparison. Overview: Patient reminder or recall interventions for receipt of immunizations ‐ any kind.

Summary of findings 2. Summary: Patient reminder or recall interventions by type of immunization.

Subgroup analysis and investigation of heterogeneity

Sensitivity analysis

Results

Description of studies

Results of the search

1.

Included studies

Study designs

Settings

Participants

Interventions

Excluded studies

Risk of bias in included studies

2.

Allocation

Random sequence generation

Allocation concealment

Baseline measurement

Blinding

Incomplete outcome data

Selective reporting

Other potential sources of bias

Effects of interventions

Patient reminder or recall

Different types of reminder or recall systems

Patient telephone reminder or recall interventions

2.1. Analysis.

Patient letter reminder or recall interventions

3.1. Analysis.

Patient postcard reminder or recall interventions

4.1. Analysis.

Patient text message reminder or recall interventions

1. Certainty assessment of evidence^a for each outcome ‐ Patient reminder or recall summary measure

2. Certainty assessment of evidence^a for each outcome ‐ patient telephone reminder or recall

3. Certainty assessment of evidence^a for each outcome ‐ patient letter reminder or recall

4. Certainty assessment of evidence^a for each outcome ‐ patient postcard reminder or recall

5. Certainty assessment of evidence^a for each outcome ‐ Patient text message reminder or recall

6. Certainty assessment of evidence^a for each outcome ‐ patient autodialer reminder or recall

7. Certainty assessment of evidence^a for each outcome ‐ combination patient mail and phone reminder and recall

8. Certainty assessment of evidence^a for each outcome ‐ combination patient reminder or recall and outreach

9. Certainty assessment of evidence^a for each outcome ‐ combination patient reminder or recall and provider reminder

10. Certainty assessment of evidence^a for each outcome ‐ childhood vaccinations

11. Certainty assessment of evidence^a for each outcome ‐ childhood influenza vaccinations

12. Certainty assessment of evidence^a for each outcome ‐ adult pneumococcal, tetanus, hepatitis B, and other non‐influenza vaccinations

13. Certainty assessment of evidence^a for each outcome ‐ adult influenza vaccinations

14. Certainty assessment of evidence^a for each outcome ‐ adolescent immunizations

Methods	Study design: randomized trial; stratified using 4 criteria, including clinical office, whether woman older than 50 years of age was in household, whether all family members were active in practice, and whether family had health insurance  Study duration: 2‐year study; 1‐year follow‐up per intervention Study aim: evaluate and compare manual health maintenance tracking system with computerized tracking system that generated patient reminders for all patients
Participants	Inclusion: families active in practice, defined as seen in clinic within past 2 years Exclusion: patients living in group homes and those living outside practice area; families that could not be reached by telephone or did not return mailed questionnaire to obtain demographic data for all adult family members; transferred care to another practice; or charts could not be located  Age: 21 years of age or older  Setting: rural, nonprofit, fee‐for‐service, family practice center that cares for patients in 5 offices; 4 of 5 offices participated; Dansville, New York (USA)  n = 1008 families; 1665 adult family members
Interventions	Intervention: telephone reminders to patients, computer‐generated health maintenance status report on chart and 2‐hour provider instruction session; n = 829  Control: manual flowchart‐based health maintenance tracking system; n = 836
Outcomes	Provider compliance with 11 health maintenance procedures within protocol, including per cent of participants immunized for tetanus diphtheria Considered providers compliant if: procedure was documented as done, not indicated, offered but patient refused, or it was provided somewhere else Intervention group: 20 percentage point increase over control group
Notes	Randomized families; data not entered in RevMan Study focused on 11 health maintenance procedures, with only one immunization measure: tetanus‐diphtheria immunization Other non‐immunization outcomes studied: tobacco use, blood pressure, weight, serum cholesterol, fecal occult blood test, physician breast exam, mammography, Papanicolaou test, teaching self‐examination, and teaching women to report post‐menopausal bleeding Control families received telephone reminders for health maintenance if requested by provider
*Risk of bias*
Bias	Authors' judgement	Support for judgement
Random sequence generation (selection bias)	Unclear risk	"Randomly assigned" families to intervention or control group within each of 32 strata based on 4 criteria; randomization procedure not described
Allocation concealment (selection bias)	Unclear risk	Randomized families to study groups; allocation procedure not described
Blinding of participants and personnel (performance bias)   All outcomes	Unclear risk	Blinding of participants and personnel not specified; however, the statement "the system must allow providers to specify or cancel sending patient reminders as well as specify the month in which reminders will be sent" implies that providers were not blinded; information is not sufficient to assess whether low risk or high risk
Blinding of outcome assessment (detection bias)   All outcomes	Unclear risk	Blinding not specified for conducting chart audits
Incomplete outcome data (attrition bias)   All outcomes	High risk	Final chart audit was conducted at end of intervention Outcomes were defined as provider compliance with Td vaccination, as defined above; immunization rates were not reported
Selective reporting (reporting bias)	Low risk	Outcome data were presented for all 11 health maintenance procedures
Other bias	Unclear risk	Control families received telephone reminders for health maintenance if requested by provider Generated a list of guarantor numbers for each participating practice, randomly; investigators attempted to contact families by telephone or mailed questionnaire to obtain demographic data; families were not included if demographic data not obtained
Baseline measurement	Low risk	Manually audited intervention and control charts at baseline Baseline characteristics of intervention and control groups were compared; small differences were observed in health insurance coverage for office visits; other characteristics were similar at baseline

Methods	Study design: randomized trial; randomized 9 practices in 3 districts Sequentially enrolled children in each practice until met sample sizes Study duration: not clear; measured vaccination of infants at 10 and 14 weeks; enrolled between February and October 2014 Study aim: evaluate the effect of text message and sticker reminders on vaccination of children
Participants	Inclusion: Kenyan districts with pentavalent 3 vaccine drop outs rates exceeding 10%; brought to selected health facilities in 3 districts for first dose of pentavalent vaccine; enrolled until sample sizes were reached Age: less than 12 months of age; media age 45 days; range of 31 to 99 days Exclusion: districts with high pentavalent vaccine coverage rates, geographically hard to reach, or security concerns; mothers did not have telephone number Setting: 9 practices providing vaccination in 3 districts; Kenya n: 1126 children assessed; 10 excluded; enrolled 1116
Interventions	Intervention group 1: short text messages reminding caretakers to return children for second and third doses of pentavalent vaccine 2 reminders from automated web‐based system 2 days before and on the day of second and third scheduled pentavalent vaccination due dates; Kiswahili and English; routine health education and advice on vaccinations; n = 372 Intervention group 2: stickers reminding caretakers to return children for second and third doses of pentavalent vaccine; not eligible intervention; n = 372 Control group: "no extra reminder messages"; next appointment date in a well‐child booklet; routine health education and advice on vaccinations; investigator contacted caretaker 2 weeks or more after immunization due date to determine reason for missed vaccinations; n = 372
Outcomes	Received scheduled pentavalent vaccines at 10 and 14 weeks Intervention group 1, 10 weeks: 98% versus 91% received pentavalent vaccine dose 2; 7 percentage point difference Intervention group 1, 14 weeks: 96% versus 83%; 13 percentage point difference
Notes	Not sure vaccines were only counted vaccines if given on the exact due date
*Risk of bias*
Bias	Authors' judgement	Support for judgement
Random sequence generation (selection bias)	Unclear risk	Random sequence generation not specified
Allocation concealment (selection bias)	Unclear risk	Randomization process not specified
Blinding of participants and personnel (performance bias)   All outcomes	Unclear risk	Blinding not specified; randomized practices
Blinding of outcome assessment (detection bias)   All outcomes	Unclear risk	Blinding not specified
Incomplete outcome data (attrition bias)   All outcomes	Unclear risk	Data collection process not clearly described; data possibly obtained by caretaker questionnaire or clinic records; data not obtained if immunizations obtained at other facilities
Selective reporting (reporting bias)	Low risk	Outcomes reported for study questions
Other bias	Unclear risk	Not able to determine if other sources of bias because methods not fully described
Baseline measurement	Low risk	Study groups were similar for demographic characteristics of caregivers and children

Methods	Study design: controlled before and after  Study duration: 1 year; 1 September 1992 to 31 August 1993 Study aim: assess the effect of interventions delivered by community‐based organization on immunization rates
Participants	Inclusion: children in Fulton County; patients of 4 public clinics or residents of one of 9 lower socioeconomic communities  Age: 3 to 59 months  Setting: community based organization in operation since 1984, serving disadvantaged populations in Fulton county, Georgia (USA)  n = 4 public clinics and 9 inner city communities; 2093 housing units within the 9 study communities; 755 parents of children were surveys in the housing units 2 clinics served predominantly African American populations, and 2 served predominantly Hispanic populations Allocated clinics to 1 intervention and 1 control clinic for each ethnic‐race category Communities consisted of 6 public housing communities with primarily African American populations, and 3 private housing communities Allocated public housing communities to 3 intervention and 3 control groups Allocated private housing communities to 2 intervention and 1 control group
Interventions	Intervention group 1: "clinic" group; telephone, mail or home visit with family Usually contacts with families were made by telephone or mail, but home visits were made when necessary; n = 2 clinics Intervention group 2: "community"; door‐to‐door campaign to identify under‐vaccinated children, provide immunization education, provide culturally sensitive promotional materials, and introduce them to services; these interventions were followed by a weekly mobile vaccination van or temporary on‐site vaccination stations, free child care, and incentives of food and baby products; ineligible intervention Interventions were applied for 1 year; n = 3 public housing communities and 2 private housing communities  Control: no intervention; n = 2 clinics, 3 public housing communities, and 1 private housing community
Outcomes	Age‐appropriate vaccination rates and series completion rates  Intervention groups: immunizations increased by 15 percentage points Controls: no change in immunization rates
Notes	Data not entered in RevMan data tables Selection of intervention sites was based on the community‐based organization's ties and perceptions of intervention feasibility Organization participated in selection of control sites
*Risk of bias*
Bias	Authors' judgement	Support for judgement
Random sequence generation (selection bias)	High risk	Allocation process was not random
Allocation concealment (selection bias)	High risk	Allocation occurred by community and practice; intervention clinics not located in control community; control clinics not located in intervention communities; allocation process was not randomized
Blinding of participants and personnel (performance bias)   All outcomes	High risk	Participants and personnel not blinded
Blinding of outcome assessment (detection bias)   All outcomes	Unclear risk	Outcome assessment blinding not specified
Incomplete outcome data (attrition bias)   All outcomes	Unclear risk	Georgia Department of Public Health district immunization officer visited each county clinic and reviewed health records of all children seen in clinic and who were 21 to 23 months of age at time of officer's visit In 2 intervention clinics, vaccination records were reviewed monthly
Selective reporting (reporting bias)	Low risk	Study purpose and methods are described; published data included all expected outcomes
Other bias	Unclear risk	Selection of intervention sites was based on the community‐based organization's ties and perceptions of intervention feasibility Organization participated in selection of control sites
Baseline measurement	Low risk	Reported baseline measures; in 1992, age‐appropriate immunization rate for children 3 to 59 months was 44% overall, and for intervention and control arm participants General characteristics of populations served by clinics and communities were similar between intervention and control groups

Methods	Study design: randomized trial; controlled before and after Intervention participants were randomized to 4 groups; controls were not randomized  Study duration: September 1996 to January 1997 Study aim: assess effectiveness of sending letters to families, delivering automated telephone messages to families, or both, in improving immunization adherence among under‐immunized 20‐month old children in health maintenance organization (HMO) setting
Participants	Inclusion and age: underimmunized 20‐month olds identified by HMO; lived in residence areas of 10 northern California medical centers of Kaiser Permanents Medical Care Program of Northern California  Setting: non‐profit group model HMO, Northern California (USA) n = 867 included in analysis, including 648 randomized to intervention groups and 219 non‐randomized controls; initially 752 were selected and randomized to 4 intervention groups; 67 were excluded because of gap in health insurance coverage
Interventions	Intervention group 1: automated telephone message followed by a letter 1 week later; n = 167 Intervention group 2: automated telephone message; 1‐minute prerecorded message stating that child was overdue for immunizations; it provided telephone numbers for advice or appointment lines at nearest Kaiser clinics; message was personalized with child's first name; system prompted listener to select language for message, either English, Spanish, or Cantonese; n = 165 Intervention group 3: letter; n = 162 Intervention group 4: letter followed by an automated telephone message 1 week later; n = 154  Control: no systematic intervention; n = 219
Outcomes	Primary outcome: receipt of any needed immunization by the 24‐month birthday Odds ratios for combined interventions = 2.1 and 2.5  Group 1: 17.7 percentage point increase over control group  Group 2: 8.2 percentage point increase over control group  Group 3: 8.6 percentage point increase over control group  Group 4: 22.2 percentage point increase over control group
Notes	Power: sample size was expected to have 80% power to detect a "16%" difference in immunization outcomes Reported intervention costs
*Risk of bias*
Bias	Authors' judgement	Support for judgement
Random sequence generation (selection bias)	High risk	Randomized patients to 4 intervention groups; selected comparison group of "similar patients"
Allocation concealment (selection bias)	Unclear risk	Eligible participants were identified by a computerized immunization tracking system; Randomized patients to 4 intervention groups but not a comparison group because their previous study found letters to be an effective intervention; investigators added "a comparison group of similar patients who turned 20 months old during January 1996"; selection of comparisons not described
Blinding of participants and personnel (performance bias)   All outcomes	Unclear risk	Interventions were sent from the health maintenance organization's regional office; blinding of participants and personnel not described
Blinding of outcome assessment (detection bias)   All outcomes	Unclear risk	Outcome assessment blinding not specified
Incomplete outcome data (attrition bias)   All outcomes	Unclear risk	Computerized immunization tracking system may not have complete vaccine information for children enrolled in the health maintenance organization after 42 days of age Tracking system does not consistently include data about immunizations that are given after child leaves health plan 67 of 752 patients were excluded because their follow‐up data may have been incomplete
Selective reporting (reporting bias)	Low risk	Study purpose and methods are described; published data included all expected outcomes
Other bias	Low risk	Study appears to be free of other sources of bias
Baseline measurement	Unclear risk	Children were identified using regional immunization tracking system Selected only underimmunized children Characteristics of participants were not described and contrasted between study groups

Methods	Study design: randomized trial; 2 by 2 factorial design  Study duration: interventions delivered over 18 months; March 1994 to August 1995 Study aim: measure effect of multi‐modal tracking and outreach intervention on improving vaccination coverage among children
Participants	Inclusion: all children in 9 practices, born between 1 March 1993 and 28 February 1994  Age: 0 to 12 months Exclusion: children who changed to nonparticipating provider or moved from Monroe County, New York were excluded from analyses  Setting: 9 primary care practices serving impoverished and middle class children; practices served more than half the city's preschool children, Rochester, New York (USA) Practices included: 2 pediatric urban group practices; 2 family medicine neighborhood health centers; 1 pediatric neighborhood health center; 1 hospital‐based clinic; 3 rural health centers  n = 3015 patients
Interventions	Intervention group 1: tracking with outreach; lay outreach workers, recruited from respective practice neighborhoods, were assigned to at least 1 practice; workers reviewed medical records to determine immunization status, and worked with parents of underimmunized children by sending postcards and making telephone calls; they made home visits to non‐responding parents; n = 630 Caseload: approximately 300 per outreach worker Intervention group 2: provider prompts; not an eligible intervention; n = 744 Intervention group 3: tracking, outreach, and provider prompts; received group 1 interventions, and distinct marker and "missed opportunity card" were placed on charts for children needing immunizations; n = 648  Control: no intervention; n = 719
Outcomes	Number and percent completing age‐appropriate vaccination series, including DTP, OPV, MMR, and Hib  Group 1: 21 percentage point increase over control group  Group 3: 21 percentage point increase over control group
Notes	Used 1‐month grace period to determine series completion outcomes Allocated patients; siblings not split between study groups; data not entered in RevMan Baseline immunization rates in area were relatively high, similar to national rates
*Risk of bias*
Bias	Authors' judgement	Support for judgement
Random sequence generation (selection bias)	Low risk	Allocated children to study groups using computer program
Allocation concealment (selection bias)	Low risk	Computer billing or encounter files were used to identify names and identifiers for participants Allocated children to study groups using computer program; outreach workers were provided with lists of intervention participants to conduct outreach and track
Blinding of participants and personnel (performance bias)   All outcomes	Unclear risk	Blinding of participants and personnel not specified; outreach workers did not document their interventions in the medical charts
Blinding of outcome assessment (detection bias)   All outcomes	Low risk	Collected data by chart abstraction; chart reviewers were blind to study group assignment
Incomplete outcome data (attrition bias)   All outcomes	Low risk	Independent research information group collected outcome data by conducting medical chart abstraction Study completion ranged from 88% to 94% within study groups Monroe County Health Department generally provided less than 1% of immunizations; health department provided written documentation to primary care providers or administered immunizations
Selective reporting (reporting bias)	Low risk	Study purpose and methods are described; published data included all expected outcomes
Other bias	Low risk	Study seems to be free of other sources of bias Performed quality control checks with dual independent review of 10% of charts; only provider‐validated immunization histories were accepted
Baseline measurement	Low risk	Independent research group at the University of Rochester collected baseline data by medical chart abstraction; this group was not involved with conducting interventions Study groups were similar for age, sex, insurance type, and baseline immunization status; study groups differed for racial composition; race was not recorded in charts for almost half the participants

Methods	Study design: randomized trial, allocated participants by week  Study duration: 11 months; enrollment from 19 January 1999 through 19 November 1999 Vaccine series completion was assessed through 16 June 2000 Study aim: evaluate effectiveness of telephone reminder‐recall intervention on increasing rates of hepatitis B vaccinations
Participants	Inclusion: male patients who reported susceptibility to hepatitis A or B; had accepted first dose of hepatitis A or B vaccine before enrollment; provided telephone number for nurse to call and leave a message with reminders about due or overdue doses; only men who have sex with men were included in analyses  Age: 18 years and older  Setting: Los Angeles Gay and Lesbian Center's Sexual Health Program, California (USA)  n = 524
Interventions	Intervention: telephone reminders; receive reminder 1 week before vaccination dose was due, and recalls at 2 and 6 weeks after dose was due; at least 1 other call attempt a different time of day, if patients not reached; n = 279  Control: no intervention; standard clinic follow‐up with appointment card listing date for next scheduled vaccine appointment and telephone number to reschedule appointment; n = 245
Outcomes	Number and percent receiving second hepatitis B vaccine dose  Intervention group: 6.3 percentage point increase over control group
Notes	Hepatitis A and B vaccines were provided free‐of‐charge to clinic 16.1% of intervention patients did not receive full intervention for second hepatitis B vaccine dose Vaccinations were free to patients
*Risk of bias*
Bias	Authors' judgement	Support for judgement
Random sequence generation (selection bias)	High risk	Allocated participants by week enrolled
Allocation concealment (selection bias)	High risk	Allocated participants by week enrolled
Blinding of participants and personnel (performance bias)   All outcomes	Unclear risk	Blinding of participants and personnel not specified; "Vaccine‐eligible clients were asked their willingness to be enrolled in an evaluation of a strategy to enhance completion of the vaccination series."
Blinding of outcome assessment (detection bias)   All outcomes	Unclear risk	Outcome assessment blinding not specified
Incomplete outcome data (attrition bias)   All outcomes	Unclear risk	Clinic employees recorded vaccination‐related information on vaccination record forms, including dates vaccinations were received at clinic, serious vaccine‐related adverse reactions, and reasons for dropping out of vaccine program 524 of 541 patients who accepted first vaccine dose included in evaluation
Selective reporting (reporting bias)	Low risk	Study purpose and methods are described; published data included all expected outcomes
Other bias	Unclear risk	To be eligible for the study, "clients had to accept a first dose of at least one of the vaccines…" Written information about hepatitis A and B infections, and availability, safety, and efficacy of the vaccines was available in clinic waiting room At the beginning of clinic visits, patients were informed by the interviewer about the availability of free hepatitis A and B vaccines Clinicians were asked to discuss availability of and recommend hepatitis vaccines to eligible patients Nurses explained to patients about vaccinations number and schedule
Baseline measurement	Unclear risk	Collected demographic data from each patient at each clinic visit: age, race, ethnicity, highest level of educational attainment; data were displayed for all clients, vaccine‐eligible clients, and clients who accepted the vaccine, but not stratified by study group Vaccine eligibility was based on self‐report of previous hepatitis B infection or vaccination

Methods	Study design: randomized trial nested within a larger study  Study duration: 5 months Study aim: evaluate effect of centralized computerized immunization register and reminder postcards on childhood immunization rates
Participants	Inclusion: infants entered in health department's computer system; intervention group participants were all infants born between 20 April 1985 and 31 December 1985; control group participants were all infants born between 1 January 1985 and 20 April 1985  Age: infants  Setting: Northland area (New Zealand)  n = 2088 patients
Interventions	Intervention: reminder card sent to parent early during any month in which vaccinations were due; monthly printout was sent to general practitioner with names of children due for immunizations; n = 709  Control: standard practice; infants' names were listed on printout if they needed vaccinations; n = 766 Non‐randomized controls: 613
Outcomes	Receipt of childhood immunizations: percent immunized at 6 weeks: 18.2% point increase, and at 3 and 5 months
Notes	Established centralized computerized immunization registry in New Zealand to address concerns about immunization data, such as using payment records that are grouped and not linked to patient‐level information, and overestimated vaccination levels using parent questionnaires, administered by public health nurses, which monitor children's immunization at the time of school entry
*Risk of bias*
Bias	Authors' judgement	Support for judgement
Random sequence generation (selection bias)	High risk	Allocated patients by even and odd dates of birth
Allocation concealment (selection bias)	High risk	Allocated patients to larger study based on date of birth, then further allocated into patient reminder study by even and odd dates of birth
Blinding of participants and personnel (performance bias)   All outcomes	Unclear risk	Practitioners received a monthly printout of names of all infants in the study with infants' dates of birth; blinding of participants and personnel not specified
Blinding of outcome assessment (detection bias)   All outcomes	Unclear risk	Outcome assessment blinding not specified
Incomplete outcome data (attrition bias)   All outcomes	Low risk	Completed printouts, listing patients needing vaccinations, by placing check mark in appropriate immunization column for each infant; wrote in unlisted infants that received vaccinations; lists were submitted monthly as claims; payments were made, and infants' computer files were updated with immunizations given Immunizations among infants who moved in and out of the area after birth were not recorded
Selective reporting (reporting bias)	Low risk	Study purpose and methods are described; published data included all expected outcomes
Other bias	Low risk	Study seems to be free of other sources of bias
Baseline measurement	Low risk	Infants were registered in a computer database as soon as possible after birth, with infant's date of birth, mother's name and address, and general practitioner's name

Methods	Study design: randomized trial; study subset randomized to intervention and control groups Study duration: 8‐month study; drew sample 1 August 2013 through 15 November 2013; study began 19 August 2013; claims reported by 1 April 2014 Study aim: evaluate feasibility of implementing multi‐level intervention to increase HPV vaccination among adolescents
Participants	Inclusion: enrolled in Medicaid or CHIP in June 2013; no HPV vaccine claims; residential zip code in North Central Florida; Gainesville, FL or surrounding primary care service area; at least 1 regular office visit between 1 July 2011 and 1 August 2013 Age: 11 to 17 years; mean = 13.7 years Exclusion: previously received HPV vaccine based on Medicaid or CHIP claims data Setting: clinics in Gainesville, Florida and surrounding service areas (USA) n: 5663 in non‐health information technology (HIT) groups
Interventions	Intervention group 1: 2 postcards sent to parents, one at study start and one 2 months later; sex‐specific; used learner verification framework; behavioral experts developed and refined postcards using iterative approach with focus groups of parents of Florida Medicaid and CHIP‐enrolled adolescents; 6‐ by 8‐inch full color postcards with images of adolescents and parents; English and Spanish; described vaccine benefits, costs, side effects, and safety; urged parents to discuss vaccination with the adolescent’s health services provider; n = 2839 Intervention group 2: HIT system; not eligible intervention; n = 1774 Control, non‐HIT: no patient reminder or recall; n = 2824
Outcomes	Initiation of human papillomavirus vaccine series Intervention group 1: 5.6% versus 4.6%; 1 percentage point higher
Notes	Randomly selected 200 parents of girls and 200 parents of boys to receive 3‐page survey to assess the acceptability of postcards; enclosed USD 5 cash and hand‐stamped return envelope; survey sent on 1 November 2013; follow‐up survey sent 28 January 2014 to non‐respondents
*Risk of bias*
Bias	Authors' judgement	Support for judgement
Random sequence generation (selection bias)	Unclear risk	Randomly assigned half of the boys and half of the girls in the HIT and comparison arms to receive postcard intervention; method of randomization not described
Allocation concealment (selection bias)	Unclear risk	As above
Blinding of participants and personnel (performance bias)   All outcomes	Unclear risk	Blinding of participants and personnel not described
Blinding of outcome assessment (detection bias)   All outcomes	Unclear risk	Used Medicaid and Children's Health Insurance Program (CHIP) claims and patient surveys to assess vaccinations; blinding not specified
Incomplete outcome data (attrition bias)   All outcomes	Unclear risk	Obtained immunization data from Medicaid and CHIP claims; however, 36% of adolescents without vaccination claims reported initiating HPV vaccination series
Selective reporting (reporting bias)	Low risk	Reported results for all groups and subgroups, answering study questions
Other bias	Unclear risk	Study methods not sufficiently detailed to assess other potential sources of bias
Baseline measurement	Unclear risk	Baseline vaccination rates initially differed between HIT and non‐HIT providers; investigators modified geographic areas to create better balance; did not present demographic characteristics, stratified by study group