Table 3.

Studies included in the review.

Author, and Publication Year	Study Design	Country	Care Context	Population	Intervention	Outcome	Result
Emily Gruber et al. 6 January 2023 [22]	Observational cohort study	USA Maryland	MDPCP practices (over 500 practices) and a matched cohort of other primary care practices not participating to the MDPCP	Total N: 245,349. N: 208,146 beneficiaries in the MDPCP group N: 37,203 beneficiaries in the non-MDPCP.	The MDPCP assisted in the COVID-19 response through four main initiatives: (1) offering data tools for targeted outreach, such as a real-time vaccine tracking system developed by CRISP; (2) facilitating early and coordinated distribution of resources, including vaccine allocation; (3) ensuring consistent communication between the MDH and practices; and (4) providing non-visit-based financial support to MDPCP practices.	Vaccination rates.	The MDPCP group was shown to have associations of higher uptake of COVID-19 vaccination, compared with the nonparticipating group. While 84.47% of the MDPCP group was fully vaccinated, 77.93% of the nonparticipating group was fully vaccinated (6.5—percentage point difference; p  <  0.001).
Yating You et al. 17 January 2023 [26]	Randomized Controlled Trial	China Shenzhen	In China, CHCs a total of 24 health centers in 4 districts were selected, among which half were assigned to the intervention group and the other half to the control group	Total N: 6886 Intervention group: N: 3814 Control group: N: 3072	PCPs working in the intervention health centers recommend FLU vaccination to their patients who were aged 60 and above. PCPs working in the control CHCs did not provide FLU vaccination recommendations for their patients.	Changes in the number of older patients vaccinated in the 24 studied health centers during the 2017–2018 flu season compared to the 2016–2017 vaccination campaign.	In the intervention group, 2457 patients received the FLU vaccine, marking an increase of 1100 patients compared to the 2016–2017 flu season. In contrast, the control group saw 1493 patients vaccinated, a decrease of 86 compared to the baseline 2016–2017 flu season.
Tracy A. Lieu et al. 17 June 2022 [27]	Randomized Clinical Trial	USA California	GPs from TPMG, working in medical centers	Total N: 8287 Latino and Black individuals aged 65 years and older from 4 KPNC services. This patients were randomly allocated in one of these groups: N: 2767 individuals in the culturally tailored PCP outreach group; N: 2747 individuals in the standard PCP outreach group; N: 2773 individuals in the usual care group;	Unvaccinated individuals were randomized into three groups: standard PCP outreach (secure message or letter), culturally tailored outreach (addressing additional concerns like cost, immigration status, and racial disparities), and usual care (no outreach). Messages, sent in the PCP’s name, emphasized vaccine trust, safety, side effects, and appointment booking. After four weeks, unvaccinated individuals in the outreach groups received a follow-up postcard, while the usual care group received no outreach.	Time to receipt of COVID-19 vaccination within 8 weeks after initial study outreach.	At 8 weeks post-intervention, vaccination rates were 24.0% (664 individuals) in the culturally tailored PCP outreach group, 23.1% (635 individuals) in the standard PCP outreach group, and 21.7% (603 individuals) in the usual care group. Culturally tailored PCP outreach significantly increased vaccination rates compared to usual care (adjusted hazard ratio [aHR], 1.22; 95% CI, 1.09–1.37; p  <  0.001), as did standard PCP outreach (aHR, 1.17; 95% CI, 1.04–1.31; p  =  0.007). There was no significant difference between culturally tailored and standard PCP outreach (aHR, 1.04; 95% CI, 0.94–1.17; p  =  0.42).
Peter G. Szilagyi et al. 18 May 2020 [28]	Randomized Clinical Trial	USA California	All 52 UCLA Health System primary care practices: 38 internal medicine, 5 medicine and pediatrics, 3 family medicine, and 6 pediatrics.	The sample included N: 164,205 active portal users, aged 6 months or older and eligible for FLU vaccination, who were primary care patients within the UCLA Health System and were randomly selected from the initial pool.	The study was a 4-arm, pragmatic, intention-to-treat randomized clinical trial involving 164,205 primary care patients assigned to one of four groups: no reminder (n = 41,070), 1 reminder (n = 41,055), 2 reminders (n = 41,046), or 3 reminders (n = 41,034). In the reminder groups, messages were sent via the patient portal, prompting patients with a secure email or text notification to log in and read a “message from your doctor,” without mentioning FLU vaccination in the title.	Receipt of one or more FLU vaccines, as documented in the electronic health record and supplemented with external data (e.g., pharmacies).	In the primary analysis FLU vaccination rates were 37.5% for those receiving no reminders, 38.0% for those receiving 1 reminder (p  =  0.008 vs. no reminder), 38.2% for those receiving 2 reminders (p  =  0.03 vs. no reminder), and 38.2% for those receiving 3 reminders (p  = 0.02 vs. no reminder)
Peter G Szilagyi et al. 1 September 2021 [29]	Randomized Clinical Trial 6-arm RCT	USA California	All 53 internal medicine, medicine-pediatric, and family medicine primary care practices at UCLA.	The study included adult patients who used the Epic™ EHR patient portal within 12 months, stratified as follows: young adults aged 18–64 years without diabetes (N = 145,166), older adults aged 65 years and older without diabetes (N = 29,795), and adults aged 18 years and older with diabetes (N = 21,525). Non-active portal users were excluded.	A total of 196,486 patients, including young adults (N = 145,166), older adults (N = 29,795), and patients with diabetes (N = 21,525), were randomized into six groups (6-arm RCT): (1) control (no messages), (2) pre-commitment letter only, (3) pre-commitment letter plus loss-framed reminders, (4) pre-commitment letter plus gain-framed reminders, (5) loss-framed reminders only, or (6) gain-framed reminders only. Pre-commitment groups received a message in mid-October, while loss- and gain-framed groups received up to three portal reminders from late October to December if no FLU vaccination was recorded.	FLU vaccination rates between 10 January 2019 and 31 March 2020	FLU vaccination rates were low: 37% in young adults, 55% in older adults, and 60% in patients with diabetes. There were no significant differences in vaccination rates across pre-commitment or message framing (loss vs. gain) within any group. Both unadjusted and adjusted risk ratios showed no statistically or clinically significant impact of pre-commitment or message framing on vaccination rates.
Natalia Y. Loskutova 29 February 2020 [24]	Non-Randomized Clinical Trial (prospective intervention)	USA North Carolina	Ten sites 43 providers within the same organization (AFPNRN) were included; 23 primary care providers in the intervention arm of the study while 20 providers were in the comparator group	This study included patients aged 18 and older who received services from participating providers during 2013–2015 and were eligible for vaccinations. Eligibility included pneumococcal vaccination for those aged 65 and older or 19–64 with risk factors, FLU vaccination for all aged 18 and older, and zoster vaccination for those aged 60 and older.	The following components were provided exclusively to providers in the intervention group: standing orders, provider audit and feedback on vaccination rates, improved documentation of patients’ medical history, enhanced provider education on vaccines and patient communication, and increased patient awareness and acceptance of recommended vaccines through educational materials.	Effects of intervention on vaccination rates	Vaccination rates increased after 12 months in both intervention and comparator groups as follows: FLU: intervention by 6.9 percentage points (p  =  0.001) and control by 6.2 percentage points (p  =  0.01); pneumococcal vaccinations in older adults: intervention by 18.6 percentage points (p  <  0.0001) and control by 16.7 percentage points (p  <  0.0001); zoster: intervention by 4.8 percentage points (p  <  0.0001) and control by 7.3 percentage points (p  =  0.001); pneumococcal vaccinations in at-risk adults remained stable in the intervention group, while the comparator group increased by 3.8 percentage points (p  =  0.003).
Marit Tuv 5 April 2023 [30]	Randomized Clinical Trial	Norway	Twenty-five GPs at 11 medical centers in Norway	Unvaccinated individuals over 18 years of age and at increased risk of severe COVID-19 were eligible. A total of 654 unvaccinated at-risk patients were identified: 202 were assigned to receive a phone call from their GP, while 452 were assigned to not receive a call.	Participants in the intervention group were contacted by their GPs via phone, while the control group received only standard care. During the calls, GPs explained that the purpose was to provide an opportunity to discuss and ask questions about the vaccine. GPs were given a one-page guide for conducting the phone call, along with a two-page document containing suggestions on how to address potential concerns raised by patients.	The proportion of participants registered as ’vaccinated against COVID-19′ in the Norwegian immunization Registry during the follow-up period was compared between the intervention and control groups.	The average follow-up period was 7.5 weeks. It is estimated that GPs successfully reached 76% (n = 154) of the patients they were assigned to call. At follow-up, 8.9% (n = 18/202) of the intervention group and 5.3% (n = 24/452) of the control group had been vaccinated (OR 1.72; 95% CI = 0.90 to 3.28).
Laurent Rigal 2023 [25]	Controlled non-randomized study	France	14 GPs in three multi-professional health centers	A total of 810 adults on a participating GP’s patient list were eligible for the 2019–2020 FLU vaccination campaign and were unvaccinated as of 2 January 2020, which was mid-campaign. Of these, 317 were assigned to the intervention arm, while 493 were assigned to the control arm.	On 2 January 2020, GPs in the intervention arm sent a standardized letter individually inviting their eligible patients not already vaccinated at mid-campaign to be vaccinated against FLU	FLU vaccination coverage estimated by the difference between the groups in their vaccination coverage at the end of the campaign (calculated from the NHIF databases)	At the end of the campaign, vaccination coverage was 14.7% (95% confidence interval [CI]: 11.6–17.9%) in the intervention group and 1.7% (95% CI: 1.0–4.3%) in the control group, resulting in a difference of 13.1 percentage points between the two groups (p < 0.001).
Hanley J Ho 2019 [31]	Pragmatic, cluster-randomized crossover trial	Singapore	22 private GP clinics in Singapore	The study included all patients aged 65 years or older, with or without chronic disease, who visited and were registered as clinic patients during the study period. In total, 8837 patients were considered. Of these, 4378 were included in the intervention periods, while 4459 were included in the control periods.	Clinics were assigned to a 3-month intervention period, which included a 1-month washout period, followed by a 4-month control period with usual care. The intervention materials consisted of informational flyers and posters with straightforward messages encouraging patients to receive FLU and pneumococcal vaccinations.	Differences in uptake rates for FLU and pneumococcal vaccinations between the intervention period and the control period.	Overall uptake rates were significantly higher in clinics during the intervention period compared with the control period for both FLU (5.9% vs. 4.8%; p = 0.047) and pneumococcal (5.7% vs. 3.7%; p = 0.001) vaccines.
Christophe Berkhout 2018 [32]	Randomized controlled trial	France	75 GPs’ waiting room	The study population consisted of patients aged 16 and older. The target group included patients over 65 years of age or those with chronic diseases requiring seasonal FLU vaccination, such as COPD or diabetes. In total, 3781 patients were included in the intervention periods, while 6816 patients were included in the control periods.	The study compared patient awareness between two settings: standard waiting rooms at 50 GPs’ offices (control group) and waiting rooms at 25 GPs’ offices where pamphlets and a poster on the FLU vaccine were provided, in addition to the standard mandatory information (intervention group).	Number of seasonal FLU vaccination units released in community pharmacies	No difference was observed in the number of FLU vaccination units delivered (Relative Risk = 1.01; 95% CI [0.97 to 1.05]; p = 0.561). However, having been vaccinated the previous year significantly increased the likelihood of revaccination (Relative Risk = 5.63; 95% CI [5.21 to 6.10]; p < 0.001).
Emily Herrett 2015 [33]	Cluster randomized trial	England	156 English primary care practices	The study involved 156 general practices that used text messaging software but had not previously used text message reminders for FLU vaccination. Eligible patients were aged 18–64 and classified as ’at-risk’. In total, 51,121 patients were included in the intervention periods, while 51,136 patients were included in the control periods.	Practices in the intervention arm (N: 77) were instructed to send text message reminders about FLU vaccination to their at-risk patients under 65. Practices in the standard care arm (N: 79) were asked to continue their FLU vaccination campaign as originally planned.	FLU vaccination rates uptake among patients aged 18–64 years in the seven prespecified risk groups during the period between 1 September and 31 December 2013.	In the standard care arm of the trial, mean vaccine uptake across practices was 50.7% and in the intervention, arm was 52.4% OR (95% CI) 1.11 (1.00 to 1.25).
Steven Kawczak 2020 [23]	Observational cohort study	USA	Primary care physicians employed by CCCAA and non-employed primary care physicians who are members of a regional Quality Alliance program	Out of 273 physicians from the Cleveland Clinic Quality Alliance network, 91.6% chose to participate in at least the first stage. Of these, 135 physicians (BMG [n = 8], CCF [n = 113], independent [n = 14]) progressed to Stage B (test group), while 100 physicians (BMG [n = 4], CCF [n = 87], independent [n = 9]) advanced to Stage C and completed the entire learning intervention.	The intervention was a three-stage quality improvement initiative incorporating CME learning activities. Stage A involved assessing practice to establish baseline performance. Stage B included participation in learning interventions and individualized action planning for practice change, while Stage C entailed reassessing practice. Data were also collected from a control group of clinicians who did not participate during the same period.	The rate at which patients of participating physicians received FLU and pneumococcal vaccines in accordance with guideline-based recommendations.	The intervention group showed significant increases in FLU vaccination rates, from 56.2% to 58.7% for patients aged ≥ 65 (p < 0.001) and from 38.6% to 40.4% for high-risk patients aged 18–64 (p < 0.001). Pneumococcal vaccination rates also increased, from 80.6% to 82.7% (p < 0.001) in the intervention group and from 56.7% to 58.2% (p < 0.001) in the control group for patients aged ≥ 65, with similar gains for high-risk adults in both groups.
Richard K Zimmerman 2016 [34]	Randomized controlled cluster trial CROSS-OVER RCT	USA	The study involved 25 primary care practices, stratified by city (Houston, Pittsburgh), location (rural, urban, suburban), and type (family medicine, internal medicine). In Pittsburgh, 19 clinics were enrolled, while 6 clinics participated in Houston.	GP’s patients aged 65 and older at baseline (N = 18,107; mean age 74.2; 60.7% female, 16.5% non-white, 15.7% Hispanic).	The 4PP, was implemented. At the end of Year 1, practices were given the option to continue the active intervention into Year 2, with four practices choosing to do so. Simultaneously, the Year 1 control sites began the intervention. For the Year 2 pre-post analyses, the four Pittsburgh practices that continued the intervention were combined with the Year 1 control sites and referred to as the active intervention group.	Vaccination rates for the 23-valent pneumococcal polysaccharide vaccine (PPSV) and pneumococcal conjugate vaccine (PCV), as well as percentage point (PP) changes in these rates. The primary outcomes were the cumulative PPSV and PCV vaccination rates reported at baseline, Year 1, and Year 2. Chi-square tests were conducted to assess differences in cumulative vaccination rates at various time points.	Cumulative PPSV vaccination rates for patients aged ≥ 65 increased significantly from baseline to Year 1 in both intervention and control groups, with gains of 6.5 to 8.7 percentage points (p < 0.001). Significant increases were observed at Houston sites (p < 0.001), but not Pittsburgh sites (p = 0.84). In Year 2, PPSV rates continued to improve, with 79% of practices achieving rates of at least 70% and 58% reaching 80%. Additionally, PCV rates increased significantly more in active intervention sites than in maintenance sites (p < 0.001 for Pittsburgh, p < 0.01 for Houston).
Chyongchiou J Lin 2016 [35]	Randomized controlled cluster trial CROSS-OVER RCT	USA	The study involved 25 primary care practices, stratified by city (Houston, Pittsburgh), location (rural, urban, suburban), and type (family medicine, internal medicine). In Pittsburgh, 19 clinics were enrolled, while 6 clinics participated in Houston.	GPs’ cohort of 70,549 adults seen in their respective practices (n = 24 with 1 drop out) at least once each year was followed. Baseline mean age was 55.1 years, 35% were men, 21% were non-white and 35% were Hispanic.	The 4PP was implemented. At the end of Year 1, practices had the option to continue the active intervention into Year 2, with four practices choosing to do so. Concurrently, the Year 1 control sites began the intervention. For the Year 2 pre-post analyses, the four Pittsburgh practices that continued in Year 2 were combined with the Year 1 control sites, creating the active intervention group.	FLU vaccination rate, was reported at the end of the baseline period (1 August 2012–31 January 2013) and the end of the intervention period (1 August 2013–31 January 2014) by site and intervention group for the Year 1 RCCT analyses.	After one year, both the intervention and control groups experienced significant increases in FLU vaccination rates, with improvements ranging from 2.7 to 6.5 percentage points (p < 0.001). Regression analyses indicated a higher likelihood of vaccination at sites with fewer missed opportunities (p < 0.001). After adjusting for missed opportunities, the intervention further increased vaccination rates in Houston (lower baseline rates) but not in Pittsburgh (higher baseline rates). During follow-up, vaccination likelihood improved at intervention sites and those that reduced missed opportunities (p < 0.005).
Mary Patricia Nowalk 2016 [36]	Randomized controlled cluster trial CROSS-OVER RCT	USA	The study involved 25 primary care practices, stratified by city (Houston, Pittsburgh), location (rural, urban, suburban), and type (family medicine, internal medicine). In Pittsburgh, 19 clinics were enrolled, while 6 clinics participated in Houston.	70,549 GPS’ patients ≥18 years who were seen in the practices ≥1 time each year, with a baseline mean age = 55 years; 35% were men; 56% were non-white; 35% were Hispanic and 20% were on Medicare	The 4PP was implemented. At the end of Year 1, practices could continue the active intervention into Year 2; four practices chose to do so. Meanwhile, the Year 1 control sites started the intervention. For the Year 2 pre-post analyses, the four Pittsburgh practices that continued the intervention were combined with the Year 1 control sites to form the active intervention group.	Cumulative Tdap vaccination rate reported at the end of baseline, Year 1 and Year 2	The baseline vaccination rate was 35%. In Year 1, Tdap vaccination rates increased more in the intervention groups (7.7 PP in Pittsburgh, 9.9 PP in Houston) than in the control groups (6.4 PP in Pittsburgh, 7.6 PP in Houston) (p < 0.001). In Year 2, active intervention groups had greater increases (6.2 PP) compared to maintenance groups (2.2 PP in Pittsburgh, 4.1 PP in Houston) (p < 0.001).

MDPCP: Maryland Primary Care Program; CHCs: Primarily Community Health Centers; TPMG: The Permanente Medical Group; UCLA: University of California, Los Angeles; AFPNRN: Academy of Family Physicians National Research Network; NHIF: national health insurance fund; CCCAA: Cleveland Clinic in Cleveland, Ohio and affiliated; 4PP: 4 Pillars™ Practice Transformation Program, also known as the 4 Pillars™ Immunization Toolkit (4pillarstoolkit.pitt.edu).