This randomized clinical trial evaluates the effect of patient reminders sent via a health care system’s electronic health record patient portal on influenza vaccination rates.
Key Points
Question
Can reminders sent through a patient portal increase influenza vaccination rates across a health care system?
Findings
In this randomized clinical trial of 164 205 patients served by 52 primary care practices across a health care system, 1, 2, or 3 influenza vaccine reminders sent through the electronic health record patient portal had a small, statistically significant effect on increasing influenza vaccination rates compared with no reminder.
Meaning
Generic patient portal reminders may be slightly effective in increasing influenza vaccination rates, but more intensive or more targeted patient motivational strategies appear to be needed.
Abstract
Importance
Influenza vaccination rates across the US are low. Because few practices send patient reminders for influenza vaccination, a scalable patient reminder system is needed.
Objective
To evaluate the effect of patient reminders sent via a health care system’s electronic health record patient portal on influenza vaccination rates.
Design, Setting, and Participants
This pragmatic, 4-arm randomized clinical trial was performed from October 1, 2018, to March 31, 2019, across the UCLA (University of California, Los Angeles) health care system. A total of 164 205 patients in 52 primary care practices who had used the patient portal within 12 months were included.
Interventions
Patients due for an influenza vaccine were sent a letter via the patient portal of the health care system reminding them about the importance of influenza vaccination, safety of the vaccine, and morbidity associated with influenza. Patients were randomized within primary care practices to 1 of 4 study groups (no reminder [n = 41 070] vs 1 reminder [n = 41 055], 2 reminders [n = 41 046], or 3 reminders [n = 41 034]).
Main Outcomes and Measures
The primary outcome was receipt of 1 or more influenza vaccines as documented in the electronic health record, which was supplemented with influenza vaccination data from external sources (eg, pharmacies). Secondary outcomes were influenza vaccination rates among subgroups and influenza vaccinations self-reported by patients in reply to the portal-based query as having been received elsewhere.
Results
A total of 164 205 patients (mean [SD] age, 46.2 [19.6] years; 95 779 [58.3%] female) were randomly allocated to 1 of the 4 study arms. In the primary analysis across all ages and not including patient self-reported vaccinations in reply to portal reminders, influenza vaccination rates were 37.5% for those receiving no reminders, 38.0% for those receiving 1 reminder (P = .008 vs no reminder), 38.2% for those receiving 2 reminders (P = .03 vs no reminder), and 38.2% for those receiving 3 reminders (P = .02 vs no reminder). In the secondary analysis not including patient self-reported vaccinations, among adults aged 18 to 64 years (vaccination rates: 32.0% in the control group, 32.8% in the 1-reminder group, 32.8% in the 2-reminder group, and 32.8% in the 3-reminder group; P = .001), male patients (vaccination rates: 37.3% vs 38.3%, 38.6%, and 38.8%; P = .001), non-Hispanic patients (vaccination rates: 37.6% vs 38.2%, 38.3%, and 38.2%; P = .004), and those who were not vaccinated in the prior 2 years (vaccination rates: 15.3% vs 15.9%, 16.3%, and 16.1%; P < .001), vaccination rates were higher in the portal reminder groups than in the control group; the findings in these 3 subgroups mirrored the findings in the entire population. When self-reported vaccinations received elsewhere were included, influenza vaccination rates were 1.4 to 2.9 percentage points higher in the portal reminder groups, with a dose-response effect (0 reminders: 15 537 [37.8%]; 1 reminder: 16 097 [39.2%]; 2 reminders: 16 426 [40.0%]; and 3 reminders: 16 714 [40.7%]; P < .001).
Conclusions and Relevance
Generic patient portal reminders were effective in minimally increasing influenza vaccination rates, but more intensive or more targeted patient motivational strategies appear to be needed.
Trial Registration
ClinicalTrials.gov Identifier: NCT03666026
Introduction
Influenza causes many respiratory illnesses, hospitalizations, ambulatory medical visits, and deaths in the US annually.1 Although the Advisory Committee on Immunization Practices recommends annual influenza vaccination for all US individuals 6 months or older,1 vaccination rates lag behind US Healthy People 2020 vaccination coverage goals of more than 80% for individuals younger than 65 years and more than 90% for those 65 years or older.2 In 2018 to 2019, vaccination coverage rates by age were 62.6% among those aged 6 months to 17 years, 34.9% among those aged 18 to 49 years, 47.3% among those aged 50 to 64 years, and 68.1% among those 65 years or older.3
Although patient reminders are associated with increased influenza vaccination coverage4,5 and the Task Force on Community Preventive Services recommends reminders,6 few primary care practices send reminders for any vaccine7,8,9,10 in part because of limited resources.10,11,12 Experts have recommended centralized reminders from state immunization information systems or health care systems as a scalable strategy.13 Most studies using centralized reminders for influenza vaccination have focused on children or adults with chronic diseases,7,14 subsets of the population,15,16 pregnant women,17 or small numbers of practices.18,19,20 A recent study21 that tested reminders for childhood influenza vaccination with state immunization information systems found little effect.
A scalable method to send centralized reminders involves patient portals, which are secure, web-based applications that provide 2-way communication between patients and health care systems or health care professionals. Most electronic health records (EHRs) have patient portals, and a recent survey22 found that 37% of adults with a health care visit in the past year had used portals in the past year. A randomized clinical trial in a health care system23 that evaluated the patient portal plus interactive voice recording reminders observed 1% improvement in influenza vaccination rates, but that trial23 involved a multispecialty group practice, excluded patients vaccinated before November 10, and had baseline vaccination rates below 15%.23 With Cochrane reviews4,24 reporting that reminder and recall work for influenza vaccination, it is important to study the effect of portal reminders on influenza vaccination across a health care system.
We performed a 4-arm randomized clinical trial that compared the effect of 1, 2, or 3 vs no portal reminders on influenza vaccination rates across a large health care system. We evaluated the effect of portal reminders on receipt of 1 or more influenza vaccinations in the entire population and in predesignated subgroups. We hypothesized that portal reminders would increase influenza vaccination rates at the population level and in subgroups (age, sex, race/ethnicity, insurance, and influenza vaccination in prior years).
Methods
Study Design and Setting
The intervention was conducted from October 1, 2018, to March 31, 2019. The design was a 4-arm, pragmatic, intention-to-treat randomized clinical trial of 164 205 patients randomized within primary care practices to 1 of 4 study arms (no reminder [n = 41 070] vs 1 reminder [n = 41 055], 2 reminders [n = 41 046], or 3 reminders [n = 41 034]). The setting was all 52 UCLA (University of California, Los Angeles) Health System primary care practices: 38 internal medicine, 5 medicine and pediatrics, 3 family medicine, and 6 pediatrics. The UCLA Institutional Review Board approved this study and provided a waiver of patient consent because (1) the trial was extremely low risk because it was designed to remind patients about a universally recommended vaccine and (2) it was impractical to obtain patient consent from tens of thousands of patients. All data were deidentified. The trial protocol is given in Supplement 1. This study followed the Consolidated Standards of Reporting Trials (CONSORT) reporting guideline.
Study Participants and Inclusion and Exclusion Criteria
Patients within UCLA Health System practices can register with the myUCLAHealth (Epic) portal. The Figure shows the flow of the patients through the study. First, we included any UCLA Health System primary care patient 6 months or older at intervention launch (ie, influenza vaccination eligible), defined by 2 or more primary care physician visits (by evaluation and management office codes) within 3 years or 1 or more primary care physician visits with a preventive service code within 1 year. We included managed care patients assigned to the UCLA Health System (even with no health care visits in the prior year) because we were certain about their primary care practice affiliation. Second, we identified the primary care practice most recently visited. Third, we grouped patients into family units using a series of algorithms that matched each patient’s telephone number, address, insurance member number, or patient guarantor identification. Fourth, we identified active portal users as patients or portal proxies (for minors, elderly persons, or disabled persons) who signed up for and logged into the portal 1 or more times during the prior 12 months not including the initial portal log-in; this sample comprised 223 191 (65.0%) of UCLA Health System’s primary care patients.25 Statisticians (S.V., C.-H.T.) randomly selected 1 active portal-using index patient per family for inclusion. This process generated the denominator of potential participants; all other study personnel and health care professionals were blinded to study allocation.
Figure. CONSORT Flow Diagram.
Patients were eligible for study participation if they were at least 6 months of age at the start of intervention (ie, eligible for influenza vaccination), a primary care patient within the UCLA Health System according to the health system’s criteria, and an active patient portal user (≥1 log-in during the past 12 months), other than initial account log-in. Patients were ineligible if they were not a primary care patient within the UCLA Health System or not an active patient portal user.
Exclusion criteria were individuals who were not active portal users and family members of index patients, who were excluded because of adequate sample size with index patients and for consistency with Cochrane criteria for meta-analyses using Cochrane RevMan software.4,24
Study statisticians randomized index patients to 1 of the 4 study arms, stratifying by their affiliation with a primary care practice. All index patients’ family members (whether proxies or not) received the same portal reminder to avoid confusion, but only index patients’ data were analyzed.
Intervention
For all patients randomized to 1, 2, or 3 portal messages, study statisticians (S.V., C.-H.T.) sent files to the health care system’s patient portal team, defining the portal messages to be sent. A secure notification was then sent by email or text (per patients’ standard portal preference) informing patients that “a message from your doctor” had been posted on the portal. Influenza vaccination was not mentioned in the title of the message. Patients had to log in to the portal to read the letter.
The portal reminder letter content was grounded in the Health Belief Model and principles of health literacy.26,27 We based the content on (1) prior patient interviews used to craft patient reminders for influenza vaccination,21 (2) positive framing principles of behavioral economics,28 and (3) input from external national vaccination experts. The letters (eFigure in Supplement 2) included the following: (1) information that influenza season was coming, the disease can cause substantial morbidity, and the vaccine is the best way to protect against influenza; (2) recommendation to receive an influenza vaccine by calling for an office appointment or going to a pharmacy or other setting; (3) a website link to input influenza vaccinations received elsewhere into the UCLA Health System record; and (4) another website link to a UCLA webpage containing information about influenza vaccine and video testimonials about influenza vaccination. Letters were in English, included the name of the patient’s primary care physician, and had a below seventh grade reading level per Flesch-Kincaid analysis.
Portal reminders were sent according to randomization arm at the beginning of October (all intervention groups), November (those receiving 2-3 reminders), and December (those receiving 3 reminders). The letter content was identical per round. Before each round, patients who had received an influenza vaccination according to the EHR were removed from the eligible list. For children younger than 8 years and eligible for 2 vaccinations, portal reminders were sent only for the initial vaccination.
Patient and Practice Characteristics
Patient and practice characteristics were obtained from the EHR. Patient factors (Table 1) included age, sex, insurance at the last primary care visit, race/ethnicity, and receipt of 1 or more influenza vaccinations during the prior 2 years.
Table 1. Demographic Characteristics of the Study Samplea.
| Characteristics | Patients, % | ||||
|---|---|---|---|---|---|
| Total (N = 164 205) | Control | Intervention | |||
| 0 Reminders (n = 41 070) | 1 Reminder (n = 41 055) | 2 Reminders (n = 41 046) | 3 Reminders (n = 41 034) | ||
| Age group, y | |||||
| 0.5-17 | 7.4 | 7.3 | 7.4 | 7.4 | 7.3 |
| 18-64 | 74.0 | 73.8 | 73.9 | 74.1 | 74.0 |
| ≥65 | 18.7 | 18.8 | 18.6 | 18.5 | 18.7 |
| Female | 58.3 | 58.4 | 58.3 | 58.2 | 58.4 |
| Insurance | |||||
| Private | 85.2 | 85.1 | 85.3 | 85.2 | 85.1 |
| Public | 13.5 | 13.5 | 13.4 | 13.6 | 13.6 |
| Other or unknown | 1.3 | 1.4 | 1.3 | 1.3 | 1.3 |
| Race | |||||
| White | 57.3 | 57.3 | 57.5 | 57.2 | 57.1 |
| Black | 4.6 | 4.4 | 4.6 | 4.7 | 4.8 |
| Asian | 10.3 | 10.1 | 10.4 | 10.3 | 10.3 |
| Other or unknown | 27.8 | 28.2 | 27.6 | 27.7 | 27.9 |
| Hispanic ethnicity | 9.6 | 9.5 | 9.6 | 9.8 | 9.6 |
| Received influenza vaccination in prior 2 y | 46.9 | 46.9 | 46.7 | 46.7 | 47.0 |
All information was obtained from the electronic health records.
Influenza Vaccination Data
Data from the EHR included influenza vaccinations (date and location) administered at any UCLA Health System site. In addition, UCLA health care practitioners could enter vaccination codes manually for vaccines received outside the UCLA Health System. The UCLA Health System merges data into the EHR from external sources, including SureScripts (pharmacy benefits manager), California Immunization Registry (mostly children’s immunizations), claims from select health care plans, and Care Everywhere (Epic’s information exchange application). Patients or proxies can also enter data when they log in to the portal. Data from these external sources (other than from claims) are normally incorporated into the medical record at office visits via a reconciliation process whereby health care professionals add or discard individual data elements. Our study did not modify these standards. However, we integrated data from all these external sources regardless of whether physicians reconciled the immunization record.
Finally, the portal reminder letters included a link for patients to update influenza vaccinations received elsewhere; these data were also integrated into the EHR. We tracked this method for updating influenza vaccinations received elsewhere and used this flag to distinguish primary vs secondary outcome measures.
Outcome and Process Measures
Primary Outcome
The primary study outcome was EHR-based documentation that included vaccinations obtained elsewhere if inputted into the EHR using the health care system’s standard electronic merging of 1 or more influenza vaccinations between October 1, 2018, and March 31, 2019. The primary analysis excluded vaccinations reported only by patients in response to the portal reminders because the control group did not have this opportunity for self-report, thus eliminating differential outcome ascertainment. This analysis created a conservative bias because portal reminders may have encouraged some patients to receive influenza vaccinations at sites (eg, employers, malls, and some pharmacies) that did not send data to the EHR.
Secondary Outcomes
We assessed 5 secondary outcomes related to receipt of 1 or more influenza vaccinations in the relevant season: (1) vaccinations in predetermined subgroups (age, sex, race/ethnicity, insurance, and influenza vaccination in prior 2 years); (2) vaccinations that were self-reported by patients or proxies via the portal in response to portal reminders; (3) influenza vaccinations before December 15, 2018, to determine whether portal reminders increased early vaccination; (4) vaccinations among those who opened 1 or more portal reminders vs those who did not open the reminders; and (5) vaccinations in patients in the upper half vs bottom half of overall portal use.
Process Metrics
We assessed whether patients (1) opened the portal reminder letter, (2) updated influenza vaccinations received outside the UCLA Health System in response to the portal reminders, and (3) clicked on the informational website link embedded in the portal letter.
Sample Size and Power Calculation
A sample size of approximately 41 000 patients per study arm would provide greater than 90% power to detect a 2–percentage point improvement in the rate of vaccination, which we deemed to be the smallest improvement with clinical effect. This number assumes a χ2 test, a control group rate of 50% (most conservative), and a significance level of P < .017 (3-fold Bonferroni correction for the pairwise comparisons of study arms with 1, 2, and 3 reminders vs control). Because patients were randomized within practices, clustering by practice was ignored in this power calculation for conservatism.
Statistical Analysis
We report descriptive statistics for all patient characteristics in terms of frequency distributions for the full patient sample and stratifying by study arm. The primary analysis compared vaccination rates between study arms using mixed-effects Poisson regression with robust SEs.29,30 Models included a fixed effect for study arm (0 vs 1 vs 2 vs 3 reminders), random practice effects, and adjustment for patient characteristics (age, sex, insurance, race/ethnicity, and vaccination history). Secondary subgroup analyses were performed by fitting separate models for each subgroup. In these analyses, the comparison was between the combination of the 3 intervention arms and the control arm. We further performed exploratory analyses to evaluate the effect of opening any reminder letters on vaccination status. In one approach, we replicated the primary analysis strategy but grouped patients by whether they opened any letters rather than by randomization assignment. In a second approach, we used 2-stage least-squares with cluster robust SEs at the practice level to estimate the local mean treatment effect of opening any letters using assignment to intervention as an instrumental variable.
For the primary analysis comparing each of the individual reminder arms with the control arm, a significance level of P < .017 was used. In all other analyses, we considered P < .05 as statistically significant. All statistical analyses were performed using SAS statistical software, version 9.4 (SAS Institute Inc).
Results
Practice and Patient Characteristics
A total of 164 205 patients (mean [SD] age, 46.2 [19.6] years; 95 779 [58.3%] female) were randomly allocated to 1 of the 4 study arms. Table 1 provides demographic data across study arms. A total of 161 268 patients (or proxies) (98.2%) spoke English as their primary language.
Primary Outcome: Influenza Vaccination
Across all patients, the vaccination rates were 37.5% among controls (no reminders), 38.0% in the 1-reminder group (P = .008 vs controls), 38.2% in the 2-reminder group (P = .03 vs controls), and 38.2% in the 3-reminder group (P = .02 vs controls). Table 2 gives the unadjusted and adjusted risk ratios (RRs) for influenza vaccination for each study arm in the entire population. Adjusted RRs were 1.02 (95% CI, 1.00-1.03) for the 1-reminder group, 1.02 (95% CI, 1.00-1.04) for the 2-reminder group, and 1.02 (95% CI, 1.00-1.04) for the 3-reminder group, showing a small effect and no dose response by number of reminders sent.
Table 2. Unadjusted and Adjusted Risk Ratios for Influenza Vaccination for Subgroups vs a Reference Subgroup and for Each Study Group vs the No Reminder Control Group.
| Subgroup | Risk ratio (95% CI)a | |
|---|---|---|
| Unadjusted | Adjusted | |
| Study group | ||
| 0 Reminders | 1 [Reference] | 1 [Reference] |
| 1 Reminder | 1.01 (1.00-1.03) | 1.02 (1.00-1.03) |
| 2 Reminders | 1.02 (1.00-1.04) | 1.02 (1.00-1.04) |
| 3 Reminders | 1.02 (1.00-1.04) | 1.02 (1.00-1.04) |
| Age group, y | ||
| 0.5-17 | 1 [Reference] | 1 [Reference] |
| 18-64 | 0.70 (0.66-0.75) | 0.78 (0.74-0.82) |
| ≥65 | 1.13 (1.04-1.23) | 0.97 (0.92-1.02) |
| Sex | ||
| Male | 1 [Reference] | 1 [Reference] |
| Female | 0.98 (0.96-1.00) | 1.00 (0.99-1.02)a |
| Insuranceb | ||
| Private | 1 [Reference] | 1 [Reference] |
| Public | 1.45 (1.36-1.54) | 1.04 (1.02-1.05) |
| Other or unknown | 0.95 (0.89-1.02) | 0.95 (0.90-1.00) |
| Race | ||
| White | 1 [Reference] | 1 [Reference] |
| Black | 0.78 (0.75-0.82) | 0.89 (0.86-0.92) |
| Asian | 1.07 (1.04-1.10) | 1.04 (1.02-1.06) |
| Other or unknown | 0.81 (0.79-0.84) | 0.91 (0.90-0.93) |
| Ethnicity | ||
| Non-Hispanic or unknown | 1 [Reference] | 1 [Reference] |
| Hispanic | 0.96 (0.93-0.99) | 1.01 (0.99-1.04) |
| Received influenza vaccination in prior 2 y | ||
| No | 1 [Reference] | 1 [Reference] |
| Yes | 3.85 (3.65-4.07) | 3.70 (3.51-3.91) |
1.00 is a rounded figure, but all the adjusted lower bounds of the 95% CIs were strictly greater than 1.
Most publicly insured patients in this health care system are older adults on Medicare or children on Medicaid, accounting for higher influenza vaccination rates among these patients.
Secondary Analyses and Process Metrics
Subgroups
For subgroups (Table 3), including age, sex, insurance, race/ethnicity, and prior influenza vaccination, the intervention had only a small (or no) effect on influenza vaccination rates. Some intervention subgroups vs the corresponding control subgroup had vaccination rates that were several percentage points higher, but there was no clear pattern or dose response by the number of reminders sent. As indicated in Table 3, portal reminders had a statistically significant effect among patients who were male (vaccination rates: 37.3% in the control group, 38.3% in the 1-reminder group, 38.6% in the 2-reminder group, and 38.8% in the 3-reminder group; P = .001), covered by private insurance (vaccination rates: 35.3% in the control group, 35.5% in the 1-reminder group, 36.1% in the 2-reminder group, and 36.0% in the 3-reminder group; P = .02), white (vaccination rates: 39.0% in the control group, 39.7% in the 1-reminder group, 39.9% in the 2-reminder group, and 39.6% in the 3-reminder group; P = .004), non-Hispanic (vaccination rates: 37.6% in the control group, 38.2% in the 1-reminder group, 38.3% in the 2-reminder group, and 38.2% in the 3-reminder group; P = .004), and not vaccinated in the prior 2 years (vaccination rates: 15.3% in the control group, 15.9% in the 1-reminder group, 16.3% in the 2-reminder group, and 16.1% in the 3-reminder group; P < .001).
Table 3. Influenza Vaccination Rates by Study Group Overall (Primary Analysis) and by Patient Subgroup at the End of the Study Period Excluding Self-reported Vaccinations.
| Subgroup | Patients who received an influenza vaccination, % | P valuea | |||
|---|---|---|---|---|---|
| 0 Reminders | 1 Reminder | 2 Reminders | 3 Reminders | ||
| All patients | 37.5 | 38.0 | 38.2 | 38.2 | .006 |
| Age group, y | |||||
| 0.5-17 | 52.2 | 51.6 | 54.0 | 52.7 | .88 |
| 18-64 | 32.0 | 32.8 | 32.8 | 32.8 | .001 |
| ≥65 | 53.2 | 53.1 | 53.0 | 53.8 | .31 |
| Sex | |||||
| Female | 37.6 | 37.8 | 37.8 | 37.8 | .39 |
| Male | 37.3 | 38.3 | 38.6 | 38.8 | .001 |
| Insurance | |||||
| Private | 35.3 | 35.5 | 36.1 | 36.0 | .02 |
| Public | 51.6 | 53.7 | 51.7 | 52.4 | .07 |
| Other or unknown | 32.5 | 38.6 | 33.7 | 34.1 | .15 |
| Race | |||||
| White | 39.0 | 39.7 | 39.9 | 39.6 | .004 |
| Black | 30.6 | 31.8 | 31.6 | 34.5 | .28 |
| Asian | 45.8 | 45.1 | 45.3 | 45.4 | .72 |
| Other or unknown | 32.5 | 32.8 | 33.0 | 33.2 | .04 |
| Ethnicity | |||||
| Hispanic | 36.4 | 36.2 | 37.0 | 38.3 | .41 |
| Non-Hispanic or unknown | 37.6 | 38.2 | 38.3 | 38.2 | .004 |
| Received influenza vaccination in prior 2 y | |||||
| Yes | 62.6 | 63.2 | 62.8 | 63.2 | .23 |
| No | 15.3 | 15.9 | 16.3 | 16.1 | <.001 |
P values are for differences between the combination of intervention groups and the control group adjusting for all other variables in the table.
Vaccination Rates Including Patient Self-reported Vaccinations in Response to Portal Reminders
Overall, 43 387 vaccinations (69.6%) were administered within UCLA clinics, whereas 18 935 (30.4%) were from external data sources, such as SureScripts. When we included self-reported data about influenza vaccinations received in the community in response to the patient portal reminders beyond external vaccinations that were already merged into the EHR (Table 4), 37.8% of controls (no reminders) received an influenza vaccination vs 39.2% in the 1-reminder group, 40.0% in the 2-reminder group), and 40.7% in the 3-reminder arms (comparison of combined intervention vs control; P < .001). There was a dose-response increase in vaccination rates depending on the number of reminders sent. In analyses of subgroups, statistically significant improvements in vaccination rates were noted among all subgroups except for children, with a small dose response.
Table 4. Influenza Vaccination Rates by Study Group Overall and by Patient Subgroup at the End of the Study Including Self-reported Vaccinations.
| Subgroup | Patients who received an influenza vaccination, % | P valuea | |||
|---|---|---|---|---|---|
| 0 Reminders | 1 Reminder | 2 Reminders | 3 Reminders | ||
| All patients | 37.8 | 39.2 | 40.0 | 40.7 | <.001 |
| Age group, y | |||||
| 0.5-17 | 52.3 | 51.9 | 54.4 | 53.1 | .70 |
| 18- 64 | 32.4 | 34.1 | 34.8 | 35.5 | <.001 |
| ≥65 | 53.6 | 54.6 | 55.1 | 56.7 | <.001 |
| Sex | |||||
| Female | 37.9 | 39.1 | 39.8 | 40.5 | <.001 |
| Male | 37.7 | 39.3 | 40.3 | 41.0 | <.001 |
| Insurance | |||||
| Private | 35.7 | 36.7 | 37.9 | 38.6 | <.001 |
| Public | 52.0 | 54.9 | 53.5 | 54.8 | <.001 |
| Other or unknown | 32.9 | 39.3 | 36.2 | 36.1 | .04 |
| Race | |||||
| White | 39.3 | 41.0 | 41.8 | 42.2 | <.001 |
| Black | 31.1 | 32.9 | 33.1 | 36.0 | .07 |
| Asian | 46.3 | 46.7 | 47.6 | 49.3 | .007 |
| Other or unknown | 32.7 | 33.6 | 34.6 | 35.4 | <.001 |
| Ethnicity | |||||
| Hispanic | 36.7 | 37.1 | 38.6 | 40.4 | .02 |
| Non-Hispanic or unknown | 37.9 | 39.4 | 40.2 | 40.8 | <.001 |
| Received influenza vaccination in the prior 2 y | |||||
| Yes | 63.1 | 65.0 | 65.7 | 67.0 | <.001 |
| No | 15.5 | 16.6 | 17.3 | 17.4 | <.001 |
P values are for differences between the combination of intervention groups and the control group adjusting for all other variables in the table.
Influenza Vaccines Received Through Mid-December
For the secondary analysis evaluating early vaccinations, 34.2% of controls (no reminders) received an influenza vaccination by mid-December vs 34.8% in the 1-reminder group (P = .001 vs controls), 35.1% in the 2-reminder group (P = .002 vs controls), and 35.2% on the 3-reminder group (P = .002 vs controls). Adjusted RRs were 1.02 (95% CI, 1.01-1.04) for the 1-reminder group, 1.03 (95% CI, 1.01-1.05) for the 2-reminder group, and 1.03 (95% CI, 1.01-1.05) for the 3-reminder group, showing a small effect and no dose response by number of reminders sent, consistent with the full-season results.
Patients Who Opened 1 Letter or More
After adjustment for all variables in Table 1 (including prior influenza vaccination), across all intervention groups, patients who opened 1 letter or more were more likely to be vaccinated than were those who did not open any letters (39.1% vs 25.4%; adjusted RR, 1.41; 95% CI, 1.38-1.44). We also used an instrumental variables approach to adjust the intention-to-treat effect of assignment to intervention for the proportion of intervention patients who opened any letters. Although the intention-to-treat effect of 1, 2, and 3 reminders was estimated to be a 0.66–percentage point increase in rates, the local mean treatment effect of opening a letter, using assignment to intervention as an instrument, was estimated to be 1.18 percentage points and was statistically significant (risk difference, 1.18; 95% CI, 0.29-2.07).
Intervention Effect by Level of Portal Use
We compared the intervention effect among those who interacted frequently with the portal (≥15 median log-ins across the population during 12 months) vs less frequently (<15 log-ins). The difference was not statistically significant.
Process Measures
A total of 52.9% of patients in the 1-reminder group, 55.9% in the 2-reminder group, and 58.8% in the 3-reminder group read the reminder letters at least once; for the 2-reminder group, 33.2% read both letters, and for the 3-reminder group, 12.8% read 2 letters and 24.5% read 3 letters. Of all influenza vaccinations recorded in the EHR, data for 0.8% of controls came exclusively from self-reporting vaccinations to the portal vs 3.6% of the 1-reminder group, 5.7% of the 2-reminder group, and 7.5% of the 3-reminder group, suggesting that the portal reminders prompted many patients to report vaccinations received elsewhere. Only 416 patients (0.3%) visited the embedded website providing information about influenza vaccination; even fewer viewed the video testimonials. There were no major patient or practitioner concerns from the study.
Discussion
This study has several important findings. First, if we exclude patient self-report of vaccinations received elsewhere (triggered by the portal reminders themselves), the portal reminders improved influenza vaccination rates by only 1 percentage point (with an adjusted RR of 1.02), and 2 or 3 reminders did not produce a dose-response effect. Second, portal reminders appeared to have a small, statistically significant effect on increasing influenza vaccination rates among adults aged 18 to 64 years, male patients, non-Hispanic patients, and those not vaccinated in the prior 2 years. Third, when we included self-reported vaccinations indicating a vaccine received elsewhere, the intervention had a larger effect: 1.4% for the 1-reminder group, 2.2% for the 2-reminder group, and 2.9% for the 3-reminder group; however, these findings may be biased by differential outcome ascertainment. Fourth, more than half of patients opened the reminder letters and many after each mailing. Patients who opened at least 1 letter were substantially more likely to be vaccinated than were those who did not. However, because opening a portal letter is a health-seeking behavior, we would expect higher vaccination rates among those who opened a letter even if our intervention did not have any effect. Overall, the primary intention-to-treat analysis revealed a small effect on increasing influenza vaccination rates across this health care system.
Our findings mirror results from a previous portal reminder study23 that also found only a 1-percentage point increase over a baseline coverage level of less than 15%. We speculate on possible reasons for our small study effect. First, perhaps after receiving a portal reminder, many adult influenza vaccinations were given outside the health care system but not recorded in the EHR despite multiple mechanisms capturing external vaccinations. This possibility would support our secondary analysis that included patient self-reported vaccinations and that found a larger effect. This possibility is supported by our finding that vaccination rates in this population were below national or state coverage estimates, suggesting that data may still be missing about many influenza vaccinations received by this population that has established primary care. Incomplete data on vaccinations received elsewhere may have obscured additional benefit of the intervention.
A second possibility is that the effectiveness of reminders has started to wane amid many patient-targeted health communications. The 2018 Cochrane Review of reminder and recall4 did not analyze whether the effect of reminders has decreased over time. A recent study21 noted that influenza vaccination reminders sent on behalf of state immunization registries had no effect on child vaccination rates. We had anticipated that portal reminders would have a stronger motivating influence because they come from the physician and from within the patients’ health care system. Although more than 50% of patients opened the reminder letters, the intervention’s effect was small but higher among those who opened the letter. Because only half of patients opened a letter, strategies that enhance reading of portal reminders might be helpful.
Studies20,31,32,33 of reminders have suggested that the message content might influence their effect. A postintervention focus group with the health care system’s portal patient representatives noted that patients preferred straightforward, relatively brief, user-friendly messages and positive messages and message titles. Because few study participants (<1%) viewed the linked website with videos, reminder letters themselves should contain critical educational content. Studies are needed to assess whether varying portal message content might improve its effect.
Another possible reason for the limited effectiveness of the portal reminders in the context of low baseline vaccination rates is vaccine hesitancy.34,35 The intervention was not designed to address hesitancy but rather to nudge patients who desired vaccination. Many studies35,36,37 have documented frequent influenza vaccination refusals. Vaccination hesitancy is not an all-or-nothing characteristic,35,38 and studies are needed to assess the degree to which reminders and education can affect vaccination uptake among hesitant individuals.
An important benefit to this intervention was that it captured more complete influenza vaccination data for the health care system because of the additional vaccinations self-reported by patients in response to portal reminders. This approach particularly benefits health care systems facing value-based incentives that incorporate vaccination rates as a quality metric.13,39
Strengths and Limitations
This study has strengths and limitations. This was a large, pragmatic trial across an entire health care system that tested a new communication mechanism (patient portal) with randomization within primary care practices to adjust for many unmeasured confounders and ascertainment of vaccinations received elsewhere via linkages with the EHR and patient self-reporting. A limitation is potential lack of generalizability to other health care systems (particularly safety-net populations or those with limited English proficiency) and to portal nonusers. In addition, we could not track all vaccinations received outside the health care system.
Conclusions
In this trial, patient portal reminders about influenza vaccination had a small effect on vaccination rates across a health care system. Further research is needed to enhance the effect of portal reminders for influenza vaccination.
Trial Protocol
eFigure. Example of Emailed Patient Portal Reminder Letter
Data Sharing Statement
References
- 1.Grohskopf LA, Sokolow LZ, Broder KR, Walter EB, Fry AM, Jernigan DB. Prevention and control of seasonal influenza with vaccines: recommendations of the Advisory Committee on Immunization Practices–United States, 2018-19 influenza season. MMWR Recomm Rep. 2018;67(3):1-20. doi: 10.15585/mmwr.rr6703a1 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.US Department of Health and Human Services Healthy People 2020. Accessed January 21, 2020. https://www.healthypeople.gov/2020/topicsobjectives2020/objectiveslist.aspx?topicId=23
- 3.Centers for Disease Control and Prevention Flu vaccination coverage, United States, 2018–19 influenza season. Accessed January 21, 2020. https://www.cdc.gov/flu/fluvaxview/coverage-1819estimates.htm
- 4.Jacobson Vann JC, Jacobson RM, Coyne-Beasley T, Asafu-Adjei JK, Szilagyi PG. Patient reminder and recall interventions to improve immunization rates. Cochrane Database Syst Rev. 2018;1:CD003941. doi: 10.1002/14651858.CD003941.pub3 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Szilagyi PG, Bordley C, Vann JC, et al. . Effect of patient reminder/recall interventions on immunization rates: a review. JAMA. 2000;284(14):1820-1827. doi: 10.1001/jama.284.14.1820 [DOI] [PubMed] [Google Scholar]
- 6.Briss PA, Rodewald LE, Hinman AR, et al. ; Task Force on Community Preventive Services . Reviews of evidence regarding interventions to improve vaccination coverage in children, adolescents, and adults. Am J Prev Med. 2000;18(1)(suppl):97-140. doi: 10.1016/S0749-3797(99)00118-X [DOI] [PubMed] [Google Scholar]
- 7.Dombkowski KJ, Harrington LB, Dong S, Clark SJ. Seasonal influenza vaccination reminders for children with high-risk conditions: a registry-based randomized trial. Am J Prev Med. 2012;42(1):71-75. doi: 10.1016/j.amepre.2011.09.028 [DOI] [PubMed] [Google Scholar]
- 8.Saville AW, Szilagyi P, Helmkamp L, et al. . Potential strategies to achieve universal influenza vaccination for children: provider attitudes in two states. Acad Pediatr. 2018;18(8):873-881. doi: 10.1016/j.acap.2018.07.003 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Tierney CD, Yusuf H, McMahon SR, et al. . Adoption of reminder and recall messages for immunizations by pediatricians and public health clinics. Pediatrics. 2003;112(5):1076-1082. doi: 10.1542/peds.112.5.1076 [DOI] [PubMed] [Google Scholar]
- 10.Cataldi JR, O’Leary ST, Lindley MC, et al. . Survey of adult influenza vaccination practices and perspectives among US primary care providers (2016-2017 influenza season). J Gen Intern Med. 2019;34(10):2167-2175. doi: 10.1007/s11606-019-05164-7 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Saville AW, Albright K, Nowels C, et al. . Getting under the hood: exploring issues that affect provider-based recall using an immunization information system. Acad Pediatr. 2011;11(1):44-49. doi: 10.1016/j.acap.2010.12.009 [DOI] [PubMed] [Google Scholar]
- 12.Kempe A, Wortley P, O’Leary S, et al. . Pediatricians’ attitudes about collaborations with other community vaccinators in the delivery of seasonal influenza vaccine. Acad Pediatr. 2012;12(1):26-35. doi: 10.1016/j.acap.2011.07.004 [DOI] [PubMed] [Google Scholar]
- 13.Fisher MP, Gurfinkel D, Szilagyi PG, et al. . Supporting and sustaining centralized reminder/recall for immunizations: qualitative insights from stakeholders. Vaccine. 2019;37(44):6601-6608. doi: 10.1016/j.vaccine.2019.09.055 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Mullooly JP. Increasing influenza vaccination among high-risk elderly: a randomized controlled trial of a mail cue in an HMO setting. Am J Public Health. 1987;77(5):626-627. doi: 10.2105/AJPH.77.5.626 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Baker AM, McCarthy B, Gurley VF, Yood MU. Influenza immunization in a managed care organization. J Gen Intern Med. 1998;13(7):469-475. doi: 10.1046/j.1525-1497.1998.00136.x [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Kempe A, Daley MF, Barrow J, et al. . Implementation of universal influenza immunization recommendations for healthy young children: results of a randomized, controlled trial with registry-based recall. Pediatrics. 2005;115(1):146-154. doi: 10.1542/peds.2004-1804 [DOI] [PubMed] [Google Scholar]
- 17.Stockwell MS, Westhoff C, Kharbanda EO, et al. . Influenza vaccine text message reminders for urban, low-income pregnant women: a randomized controlled trial. Am J Public Health. 2014;104(suppl 1):e7-e12. doi: 10.2105/AJPH.2013.301620 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.Nexøe J, Kragstrup J, Rønne T. Impact of postal invitations and user fee on influenza vaccination rates among the elderly: a randomized controlled trial in general practice. Scand J Prim Health Care. 1997;15(2):109-112. doi: 10.3109/02813439709018497 [DOI] [PubMed] [Google Scholar]
- 19.Stockwell MS, Kharbanda EO, Martinez RA, Vargas CY, Vawdrey DK, Camargo S. Effect of a text messaging intervention on influenza vaccination in an urban, low-income pediatric and adolescent population: a randomized controlled trial. JAMA. 2012;307(16):1702-1708. doi: 10.1001/jama.2012.502 [DOI] [PubMed] [Google Scholar]
- 20.Stockwell MS, Hofstetter AM, DuRivage N, et al. . Text message reminders for second dose of influenza vaccine: a randomized controlled trial. Pediatrics. 2015;135(1):e83-e91. doi: 10.1542/peds.2014-2475 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21.Kempe A, Saville AW, Albertin C, et al. . Centralized reminder/recall to increase influenza vaccination rates: a two-state pragmatic randomized trial. Acad Pediatr. 2020;20(3):374-383. doi: 10.1016/j.acap.2019.10.015 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.Anthony DL, Campos-Castillo C, Lim PS. Who isn’t using patient portals and why? evidence and implications from a national sample of US adults. Health Aff (Millwood). 2018;37(12):1948-1954. doi: 10.1377/hlthaff.2018.05117 [DOI] [PubMed] [Google Scholar]
- 23.Cutrona SL, Golden JG, Goff SL, et al. . Improving rates of outpatient influenza vaccination through EHR portal messages and interactive automated calls: a randomized controlled trial. J Gen Intern Med. 2018;33(5):659-667. doi: 10.1007/s11606-017-4266-9 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 24.Jacobson Vann JC, Szilagyi P. Patient reminder and patient recall systems for improving immunization rates. Cochrane Database Syst Rev. 2018;1(1):CD003941. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 25.Szilagyi PG, Valderrama R, Vangala S, et al. . Pediatric patient portal use in one health system. J Am Med Inform Assoc. 2020;27(3):444-448. doi: 10.1093/jamia/ocz203 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 26.Becker MH, Maiman LA. Sociobehavioral determinants of compliance with health and medical care recommendations. Med Care. 1975;13(1):10-24. doi: 10.1097/00005650-197501000-00002 [DOI] [PubMed] [Google Scholar]
- 27.Kim H, Xie B. Health literacy in the eHealth era: a systematic review of the literature. Patient Educ Couns. 2017;100(6):1073-1082. doi: 10.1016/j.pec.2017.01.015 [DOI] [PubMed] [Google Scholar]
- 28.Jenssen BP, Buttenheim AM, Fiks AG. Using behavioral economics to encourage parent behavior change: opportunities to improve clinical effectiveness. Acad Pediatr. 2019;19(1):4-10. doi: 10.1016/j.acap.2018.08.010 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 29.Zou G. A modified Poisson regression approach to prospective studies with binary data. Am J Epidemiol. 2004;159(7):702-706. doi: 10.1093/aje/kwh090 [DOI] [PubMed] [Google Scholar]
- 30.Zhu M. Analyzing Multilevel Models with the GLIMMIX Procedure: Proceedings of the SAS Global Forum 2014 Conference. SAS Institute Inc; 2014. Accessed January 17, 2020. https://support.sas.com/resources/papers/proceedings14/SAS026-2014.pdf [Google Scholar]
- 31.Dempsey AF, Zimet GD. Interventions to improve adolescent vaccination: what may work and what still needs to be tested. Vaccine. 2015;33(suppl 4):D106-D113. doi: 10.1016/j.vaccine.2015.09.032 [DOI] [PubMed] [Google Scholar]
- 32.Hofstetter AM, Vargas CY, Camargo S, et al. . Impacting delayed pediatric influenza vaccination: a randomized controlled trial of text message reminders. Am J Prev Med. 2015;48(4):392-401. doi: 10.1016/j.amepre.2014.10.023 [DOI] [PubMed] [Google Scholar]
- 33.Saville AW, Beaty B, Dickinson LM, Lockhart S, Kempe A. Novel immunization reminder/recall approaches: rural and urban differences in parent perceptions. Acad Pediatr. 2014;14(3):249-255. doi: 10.1016/j.acap.2014.02.003 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 34.Edwards KM, Hackell JM; Committee on Infectious Diseases, Committee on Practice and Ambulatory Medicine . Countering vaccine hesitancy. Pediatrics. 2016;138(3):e20162146. doi: 10.1542/peds.2016-2146 [DOI] [PubMed] [Google Scholar]
- 35.Jarrett C, Wilson R, O’Leary M, Eckersberger E, Larson HJ; SAGE Working Group on Vaccine Hesitancy . Strategies for addressing vaccine hesitancy: a systematic review. Vaccine. 2015;33(34):4180-4190. doi: 10.1016/j.vaccine.2015.04.040 [DOI] [PubMed] [Google Scholar]
- 36.Cunningham RM, Minard CG, Guffey D, Swaim LS, Opel DJ, Boom JA. Prevalence of vaccine hesitancy among expectant mothers in Houston, Texas. Acad Pediatr. 2018;18(2):154-160. doi: 10.1016/j.acap.2017.08.003 [DOI] [PubMed] [Google Scholar]
- 37.Shen AK, Warnock R, Selna W, Chu S, Kelman JA. Patient characteristics of Medicare beneficiaries who report not getting influenza and pneumococcal vaccinations, 2001-2013. Hum Vaccin Immunother. 2019:1-7. doi: 10.1080/21645515.2019.1688033 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 38.Schmid P, Rauber D, Betsch C, Lidolt G, Denker ML. Barriers of influenza vaccination intention and behavior: a systematic review of influenza vaccine hesitancy, 2005-2016. PLoS One. 2017;12(1):e0170550. doi: 10.1371/journal.pone.0170550 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 39.Scott A, Liu M, Yong J. Financial incentives to encourage value-based health care. Med Care Res Rev. 2018;75(1):3-32. doi: 10.1177/1077558716676594 [DOI] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
Trial Protocol
eFigure. Example of Emailed Patient Portal Reminder Letter
Data Sharing Statement