A Randomized Trial of Behavioral Nudges Delivered Through Text Messages to Increase Influenza Vaccination Among Patients With an Upcoming Primary Care Visit

Abstract

Purpose:

To evaluate if nudges delivered by text message prior to an upcoming primary care visit can increase influenza vaccination rates.

Design:

Randomized, controlled trial.

Setting:

Two health systems in the Northeastern US between September 2020 and March 2021.

Subjects:

74,811 adults.

Interventions:

Patients in the 19 intervention arms received 1–2 text messages in the 3 days preceding their appointment that varied in their format, interactivity, and content.

Measures:

Influenza vaccination.

Analysis:

Intention-to-treat.

Results:

Participants had a mean (SD) age of 50.7 (16.2) years; 55.8% (41,771) were female, 70.6% (52,826) were White, and 19.0% (14,222) were Black. Among the interventions, 5 of 19 (26.3%) had a significantly greater vaccination rate than control. On average, the 19 interventions increased vaccination relative to control by 1.8 percentage points or 6.1% (P = .005). The top performing text message described the vaccine to the patient as “reserved for you” and led to a 3.1 percentage point increase (95% CI, 1.3 to 4.9; P < .001) in vaccination relative to control. Three of the top five performing messages described the vaccine as “reserved for you.” None of the interventions performed worse than control.

Conclusions:

Text messages encouraging vaccination and delivered prior to an upcoming appointment significantly increased influenza vaccination rates and could be a scalable approach to increase vaccination more broadly.

Purpose

Influenza is a significant cause of illness, hospitalization and death worldwide.^1,2 While the influenza vaccine been demonstrated to reduce the burden of disease,³ less than half of adults obtain it each year.⁴ Encouraging greater vaccination uptake remains a significant challenge.^5–7

In early 2020, COVID-19, another respiratory virus, spread rapidly and caused significant morbidity and mortality around the world.^8,9 In the United States (US), the COVID-19 pandemic accelerated the use of new forms of digital technology and communication channels for patient care.¹⁰ Many health systems now have the capability to send automated text messages to patients, providing a scalable approach to interact with patients without needing more personnel intensive resources.^11,12 However, the best way to design a text message to nudge patients to get vaccinated had not been well examined.

In the Fall of 2020, prior to the availability of the COVID-19 vaccine, we conducted a randomized clinical trial to test 19 different text messages to promote influenza vaccination from September 2020 to March 2021. Messages were sent prior to a patient’s upcoming appointment with their primary care clinician. The goal was to understand which approaches could increase influenza vaccination and to develop a template for messages that could later be used to encourage COVID-19 vaccination when the vaccine became available. Influenza and COVID-19 both cause respiratory illness that can be prevented with a vaccine, but perceptions about their safety and effectiveness may vary.¹³ While, it was uncertain, ex ante, if the same nudges that worked to encourage influenza vaccination could nudge COVID-19 vaccination, this trial was conducted in the hopes testing promising ideas to nudge influenza (all designed with portability to COVID-19 in mind) could provide a helpful starting point for nudging COVID-19 vaccine adoption.

In December 2020, the COVID-19 vaccine became available in the United States. To inform the COVID-19 vaccination effort, we conducted an interim analysis of this trial with patients enrolled through the end of December 2020, which was published in short form.¹⁴ In this study, we report the full results from the entire influenza season from September 2020 to March 2021.

Methods

Design

This was a randomized clinical trial conducted at 90 primary care practices at two large health systems in the Northeastern United States. The trial evaluated differences in influenza vaccination from behavioral nudges delivered through text messages sent to patients in the three days leading up to a new or return visit with their primary care clinician. It was conducted as a megastudy – a field experiment in which many interventions developed by different teams of scientists were tested in the same population and on the same outcome.¹⁵ The trial occurred from September 20, 2020 (first appointments on September 24) to March 31, 2021 and analyses were conducted between April 16, 2021 and November 24, 2021. The trial protocol was approved by an Institutional Review Board, which granted a waiver of informed consent because this study was an evaluation of a health system intervention that posed minimal risk to clinicians and patients. This trial followed the Consolidated Standards of Reporting Trials (CONSORT) reporting guideline (Supplement 1). Neither clinicians nor patients were compensated for participation.

Sample

Eligible patients were adults with a new or return in-person clinic appointment with their primary care clinician during the study period. The first eligible patient visit within the study period was used. Patients were excluded if, based on electronic health record (EHR) documentation, they had received the influenza vaccine prior to the visit, had a documented allergy or adverse event related to the vaccine, did not have a cell phone number to send text messages to, or had previously opted out of receiving text message or participating in research. Patients who canceled their appointment less than four days in advance (N = 12,611), did not show up to their appointment (N = 5072), or converted their appointments to telemedicine visits (N = 1,153) were included in our analyses given that they could have already received intervention messages by the time they changed their plans. If a patient rescheduled their original appointment or scheduled a new appointment during the study period (N = 8492), their intervention messages remained the same and were re-started prior to their new appointment (i.e., up to three days prior to the new appointment). Clinicians whose patients were eligible to be enrolled in the trial included physicians, resident physicians, nurse practitioners, and physician assistants.

Patients were electronically randomized to one of the study arms stratified by site (Penn Medicine vs Geisinger), age (18 to 64 years vs ≥ 65 years), and influenza vaccination in the prior year (yes vs no/unknown).

Intervention

The trial included 19 intervention arms with text messages designed by 26 behavioral scientists tasked with crafting nudges to boost influenza vaccination that could later be redeployed to encourage COVID-19 vaccination. The 19 intervention arms contained 8 self-contained experiments, each with its own comparison condition(s) that could be analyzed separately. A control group did not receive any interventions from this trial.

Patients in the control group received only the usual care text message appointment reminders from their health system, indicating the date, time, and location of their appointments. These reminders were sent two business days prior to appointments at one health system, and one week, three days, and one day prior to appointments at the other health system.

In the 19 intervention arms, patients received the usual care text message appointment reminders and were sent additional intervention text messages. Interventions varied the content of the text messages patients received and could include interactive components (e.g., yes/no vs questions with different responses based on patient answers), links to external videos and surveys, variable numbers of messages (up to two, unless patients opted in to receive additional messages by responding to intervention messages), images, and messages sent at different times (messages could be sent as early as three days prior to a scheduled appointment and as late as 15 minutes prior to the appointment). Detailed descriptions and the full text of each intervention have been published previously¹⁴ and are available (Tables 1-19 in Supplement 2).

Measures

The primary outcome measure was influenza vaccination either on the day of a patient’s primary care visit or in the three days leading up to the visit. If patients rescheduled their appointment after the start of their intervention window, the intervention window extended from three days prior to the original appointment through the date of the new appointment. Patients who canceled their appointments during their intervention window and did not reschedule were included in our intention-to-treat analyses (the conclusion of their intervention window was defined based on the date of the canceled appointment).

Analysis

Data on patients, clinicians, clinic visits, and influenza vaccination were obtained from electronic health record databases at the two health systems participating in the trial. Patient data included information about patient demographics, insurance, comorbidities, the Charlson Comorbidity Index (CCI), medical appointment history and prior history of influenza vaccination. Data on clinicians included clinician type. Data on patient encounters included date, appointment time, practice site, visit type, and documentation of influenza vaccination.

A priori power calculations estimated that we had at least 90% power to detect a difference of 3.45 percentage points in vaccination rates between a given intervention group and the control group. This assumed a baseline vaccination rate of 33.0% and a 2-sided alpha of .05 as our threshold for statistical significance and our target sample of 4000 patients per group. To account for multiple comparisons, we report adjusted P-values computed using the Benjamini-Hochberg (BH) procedure, which controls for the false discovery rate when conducting multiple comparisons.¹⁶

All randomly assigned sites were included in our intention-to-treat analysis. We used the patient as the unit of analysis. In our adjusted model, we fit an ordinary least squares regression to predict whether a given patient received a flu shot on the day of their target appointment or in the three days prior (a binary indicator variable). The primary predictor variables in the regression were 19 indicators for assignment to each of the study’s 19 interventions (an indicator for the control condition was omitted, making this the reference group). The model was adjusted for study site, patient characteristics (age, gender, race/ethnicity), influenza vaccine receipt in the prior 2019–2020 flu season, clinician type, and the linear and squared number of days elapsed since the start of the study. We conducted pre-registered exploratory subgroup analyses by time period (before or after January 1), patient characteristics (age, gender, race, vaccination in the prior year), health system site, and clinician type.

In prior work,¹⁴ we defined 17 attributes representing characteristics of each text messaging intervention’s content and form. To account for the non-independence of attribute ratings, we identified a smaller number of attribute dimensions by performing principal component analysis and then assessed the relationship between these dimensions and efficacy in a simultaneous OLS regression predicting efficacy (e.g., using the 19 beta coefficients from the main analysis). This previous analysis found significant positive associations between intervention effectiveness and the use of the phrase “reserved for you” as well as congruence with the types of messaging that patients would expect to receive from a health care provider. This attribute analysis was repeated on the sample of patients in this trial for the full influenza season. For robustness, we also ran an OLS regression using these attributes to predict vaccination outcomes at the individual level, including the same control variables as our main analysis. All analyses were conducted in R (version 4.0.2).

Results

Participant Sample

In this trial, 74,811 patients were randomized, with 53.1% (39,741/74811) from one health system and 46.9% (35,070/74,811) from the other health system (Figure 1). Participants in the sample had a mean (SD) age of 50.7 (16.2) years; 55.8% (41,771/74,811) were female, 70.6% (52,826/74,811) were White, 19.0% (14,222/74,811) were Black, and 40.6% (30,409/74,811) had documentation of vaccination during the prior influenza season (Table 1). Participants’ characteristics in each intervention arm are available in Table 1 in Supplement 3.

Figure 1.

CONSORT Diagram. Participants in all arms were identified prior to an upcoming appointment with their primary care clinician and randomly assigned to a usual care control group or one of the 19 intervention arms.

Table 1.

Characteristics of the Patient Sample.

Characteristic	Control (n = 3742)	Pooled Intervention Arms (n = 71069)
Sociodemographics
Age, Mean (SD)	50.3 (16.2)	50.7 (16.2)
Female sex, n (%)	2056 (54.9)	39715 (55.9)
Race/ethnicity, n (%)
White non-Hispanic	2578 (68.9)	50248 (70.7)
Black non-Hispanic	751 (20.1)	13471 (19.0)
Asian non-Hispanic	86 (2.3)	1648 (2.3)
Hispanic	187 (5.0)	3204 (4.5)
Other	140 (3.7)	2498 (3.5)
Location, n (%)
Penn Medicine	2010 (53.7)	37731 (53.1)
Received flu shot in prior year, n (%)	1506 (40.2)	28903 (40.7)
Clinician type during visit, n (%)
Attending/Faculty physician	2852 (76.2)	54261 (76.3)
Resident	192 (5.1)	3567 (5.0)
Physician assistant	359 (9.6)	6732 (9.5)
Nurse practitioner	339 (9.1)	6501 (9.1)

^*Abbreviations: SD=Standard Deviation.

^**Patient characteristics for each of the 19 intervention arms are available in Supplement 3.

Vaccination

In the control group, 29.7% (1113/3742) of participants received an influenza vaccination. Table 2 reports the change in influenza vaccination rates for each intervention arm relative to control. Among the interventions, 5 of 19 (26.3%) had a significantly greater vaccination rate than the control group using a B-H adjusted threshold of P < .05. On average, the 19 interventions increased vaccination relative to control by 1.8 percentage points or 6.1% (P = .005), but we cannot reject the null hypothesis that all 19 effects have the same true value (Chi-sq = 23.4, P = .18). The top-performing intervention sent two text messages, described the vaccine to the patient as “reserved for you” and led to a 3.1 percentage point increase (95% CI, 1.3 to 4.9; P < .001) in vaccination relative to control. Three of the top five performing messages described the vaccine as “reserved for you.” The other two asked participants to watch a video about the flu and informed participants to protect themselves by getting the flu shot. None of the interventions performed worse than control.

Table 2.

Adjusted Difference in Influenza Vaccination Rate Relative to Control.

Intervention	Adjusted Difference in Percentage Points (95% CI)	Adjusted P Value
Flu shot reserved for you (2 texts: 72 hr + 24 hr pre-appt)	3.1 (1.3–4.9)	.009
Flu shot reserved for you (2 texts: 72 hr + 15 m pre-appt)	3.0 (1.2–4.8)	.009
Video about getting the flu (2 texts: 3 d + 1 d pre-appt)	2.8 (1.0–4.6)	.01
Protect yourself by getting a flu shot (2 texts: 24 hr + 15 m pre-appt)	2.7 (.9–4.5)	.01
Reply to receive the flu shot reserved for you (1 text: 1 d pre-appt)	2.7 (.9–4.5)	.01
Dedicate your flu shot to a loved one (2 texts: 3 d + 1 d pre-appt)	2.0 (.2–3.8)	.08
Remember to ask for your flu shot (1 text: 1 d pre-appt)	1.8 (.1–3.6)	.12
Protect a vulnerable loved one by getting a flu shot (2 texts: 3 d + 1 d pre-appt)	1.8 (.0–3.6)	.12
Don’t forget to get a flu shot (2 texts: 3 d + 1 d pre-appt)	1.7 (−.1–3.5)	.12
Easy health behavior quiz (1 text: 1 d pre-appt)	1.7 (−.1–3.4)	.13
Hard health behavior quiz (1 text: 1 d pre-appt)	1.6 (−.2–3.4)	.13
Reply to receive the flu shot (1 text: 1 d pre-appt)	1.6 (−.2–3.4)	.13
Vivid video about getting the flu (2 texts: 3 d + 1 d pre-appt)	1.5 (−.3–3.3)	.14
Improve the flu shot rate in your region (2 texts: 3 d + 1 hr pre-appt)	1.5 (−.3–3.2)	.15
Video about importance of exercise (2 texts: 3 d + 1 d pre-appt)	1.3 (−.5–3.1)	.19
Beat the flu shot rate in another region (2 texts: 3 d + 1 hr pre-appt)	1.3 (−.5–3.0)	.19
Protect others by getting a flu shot (2 texts: 24 hr + 15 m pre-appt)	1.1(−.6–2.9)	.24
Share a joke about the flu (1 text: 1 d pre-appt)	1.0 (−.8–2.7)	.30
Getting a flu shot is an easy way to be healthy (1 text: 1 d pre-appt)	.4 (−1.3–2.2)	.62
Usual care vaccination rate	29.7%
Patient sample	74,811

^*Abbreviations: CI=Confidence Interval; hr=hours; d=days; m=minutes; pre-appt = before appointment time.

^**Data presented are difference in predicted vaccination rates relative to the usual care control group. Point estimates and 95% Cis are unadjusted. P Values use BH adjustment.

Table 3 displays changes in vaccination rates for each intervention arm relative to control in the periods before and after January 1, 2020. The baseline vaccination rate for patients with appointments before the end of the year was 44.6%, compared to 7.6% after January 1^st. For the top three performing arms, effect sizes were larger in the period before January 1^st. Subgroup analyses by patient age, gender, race, vaccination in the prior year, clinician type and health system site are available in Tables 2-7 in Supplement 3. There were no significant differences between any of these subgroups. The trial interventions did not have a significant impact on appointment show rates (Table 8 in Supplement 3). In attribute analysis, there were no significant associations with intervention effectiveness and the use of “reserved for you” (P = .12) or “incongruence with usual health care messaging” (P = .28) (Tables 9-11 in Supplement 3). The results were similar when we examined the relationship between vaccination at the individual level and the use of “incongruence with usual health care messaging” (P = .11), however, there was a significant association with “reserved for you” (P = .04) (Tables 12-14 in Supplement 3). There were no reported adverse events.

Table 3.

Adjusted Difference in Influenza Vaccination Rate Relative to Control by Time Period.

	Enrollment Period: 9/24/20 to 12/31/20		Enrollment Period: 1/1/21 to 3/31/21
Intervention	Adjusted Difference in Percentage Points (95% CI)	Adjusted P Value	Adjusted Difference in Percentage Points (95% CI)	Adjusted P Value
Flu shot reserved for you (2 texts: 72 hr + 24 hr pre-appt)	4.5 (1.9–7.1)	.02	1.1 (−.8–3.1)	.30
Flu shot reserved for you (2 texts: 72 hr + 15 m pre-appt)	3.7 (1.1–6.4)	.06	2.3 (.4–4.3)	.07
Video about getting the flu (2 texts: 3 d + 1 d pre-appt)	3.3 (.7–5.9)	.09	2.4 (.4–4.4)	.07
Protect yourself by getting a flu shot (2 texts: 24 hr + 15 m pre-appt)	2.0 (−.6–4.7)	.29	3.6 (1.6–5.7)	.01
Reply to receive the flu shot reserved for you (1 text: 1 d pre-appt)	2.9 (.3–5.6)	.15	2.4 (.4–4.4)	.07
Dedicate your flu shot to a loved one (2 texts: 3 d + 1 d pre-appt)	2.0 (−.6–4.7)	.29	2.0 (.1–4.0)	.10
Remember to ask for your flu shot (1 text: 1 d pre-appt)	2.1 (−.5–4.8)	.29	1.5 (−.4–3.5)	.19
Protect a vulnerable loved one by getting a flu shot (2 texts: 3 d + 1 d pre-appt)	1.2 (−1.5–3.8)	.49	2.7 (.7–4.7)	.07
Don’t forget to get a flu shot (2 texts: 3 d + 1 d pre-appt)	2.7 (.0–5.4)	.19	.7 (−1.2–2.6)	.50
Easy health behavior quiz (1 text: 1 d pre-appt)	1.8 (−.8–4.5)	.34	1.6 (−.3–3.6)	.19
Hard health behavior quiz (1 text: 1 d pre-appt)	2.4 (−.2–5.1)	.22	.3 (−1.6–2.2)	.75
Reply to receive the flu shot (1 text: 1 d pre-appt)	1.0 (−1.7–3.6)	.57	2.6 (.6–4.6)	.07
Vivid video about getting the flu (2 texts: 3 d + 1 d pre-appt)	1.7 (−1.0–4.4)	.36	1.6 (−.4–3.5)	.19
Improve the flu shot rate in your region (2 texts: 3 d + 1 hr pre-appt)	1.5 (−1.1–4.2)	.41	1.4 (−.6–3.3)	.24
Video about importance of exercise (2 texts: 3 d + 1 d pre-appt)	1.2 (−1.5–3.8)	.49	1.3 (−.7–3.2)	.25
Beat the flu shot rate in another region (2 texts: 3 d + 1 hr pre-appt)	1.2 (−1.4–3.8)	.49	1.3 (−.6–3.3)	.24
Protect others by getting a flu shot (2 texts: 24 hr + 15 m pre-appt)	.9 (−1.8–3.5)	.59	1.8 (−.2–3.7)	.17
Share a joke about the flu (1 text: 1 d pre-appt)	.5 (−2.2–3.1)	.72	2.1 (.1–4.1)	.10
Getting a flu shot is an easy way to be healthy (1 text: 1 d pre-appt)	.7 (−1.9–3.3)	.62	.3 (−1.6–2.2)	.75
Baseline vaccination rate	44.6%		7.6%
Observations	45,396		29,415

^*Abbreviations: CI=Confidence Interval; hr=hours; d=days; m=minutes; pre-appt=before appointment time.

^**Data presented are difference in predicted vaccination rates relative to usual care Note: There were no significant differences between groups (F = 1.2113, P = .2367).

Discussion

In this randomized clinical trial, nudges delivered via text message to patients prior to an upcoming visit with their primary care clinician significantly increased influenza vaccination. On average, text messages increased vaccination by 1.8 percentage points relative to the control group, which did not receive any intervention text messages. This magnitude is similar to but slightly lower than that described in our previously reported findings through the end of December, which found a 2.1 percentage point change on average.¹⁴ While the magnitude of the effect may seem small, the near-zero cost of text messages makes possible scaling to a broader population, leading to a significant number of vaccinations that would not otherwise occur. In earlier findings, there were significant associations between intervention effectiveness and attributes of text message content and form. Specifically, text messages performed better when they were described as “reserved for you” and were congruent with the types of communications patients expect to receive from their health care provider. In this full evaluation, these associations were not significant with P values of .12 and .28, respectively. These differences between the evaluations of the two time points may be related to the much lower baseline vaccination rate of 7.6% after January 1^st, compared to 44.6% prior to this time point.

Our findings reveal important insights that can be used to promote vaccination and health behavior change more broadly. First, text messages are increasingly being used by health systems to communicate with patients about their care.^10–12 Our study indicates that these messages can have a meaningful impact on clinical outcomes. Communication sent by text message encouraging influenza vaccination is lower risk than communication sent by text message about other health behaviors because flu shots are recommended for all adults, and therefore communications do not reveal that a patient has any specific medical conditions. A text message is a non-secure channel and may be better received by patients for communications about recommended actions that do not reveal sensitive information. More sensitive information that could reveal patient conditions may be better sent through other channels that require a patient to login to see the message. Text message interventions are lower cost than more personnel intensive approaches to encouraging health behavior change and therefore could be a scalable approach to improving patient health behaviors.

Second, while our attribute analyses found no significant association between specific intervention attributes and intervention effectiveness, we did find that the phrase “reserved for you,” was used in three of the five best-performing messages and that the “reserved for you” messaging was significantly associated with increased vaccination at the individual level. Communicating that a vaccine is “reserved for you” may have increased a patient’s perceived value of the vaccine by creating a sense of ownership of their dose of the vaccine. This is consistent with research on the endowment effect, which shows that individuals are more motivated to take action to avoid losing something they feel is already theirs.¹⁷

Third, while these text messages were tested as a means of increasing influenza vaccination, they may provide more general insight into ways to encourage other types of vaccination where there is often a gap in one’s intention to get vaccinated and the action to do so. A randomized trial by Dai and colleagues implemented shortly after vaccines became available for COVID-19 and after we reported preliminary results from this study found that using similar ownership language in text messages (i.e., informing patients that the vaccine was available to them and that they should claim it) increased COVID-19 vaccine adoption.¹² The Dai et al trial leveraged the principle of psychological ownership that we proved could be effective as a means of nudging influenza vaccination to encourage COVID-19 vaccination, validating the portability of this messaging tactic.

This study has limitations. First, the trial was conducted among adults at only two health systems; however, it included 90 practices in urban, suburban, and rural areas of the Northeastern United States. Second, the interventions used text messaging, which required that patients had access to a cellphone and did not opt-out of these types of communications. Third, we focused on new or return in-person, non-sick visits with a patient’s primary care clinician and did not evaluate other types of interactions. Fourth, there may have been other initiatives at the health system or practice level to encourage vaccination; however, our study was a randomized trial with well-balanced patient characteristics across arms, suggesting that these other factors should also have been well-balanced across study arms in expectation. Fifth, a limitation of the megastudy design is that the effects of the top-performing interventions may be overestimated and the bottom-performing interventions may be underestimated (also known as the “winner’s curse”),¹⁸ and that with replication there may be some regression to the mean. Sixth, randomization was conducted at the level of the patient within a health system, not the practice location. While individual level randomization can raise the risk of spillover effects between arms, that risk appears low here because the intervention was delivered in close proximity to the patient visit and patients are unlikely to discuss their care with one other while in the waiting room at a doctor’s office. Seventh, this trial was conducted during the COVID-19 pandemic. The lower influenza vaccination rates after January 1^st coincided with the initial release of the COVID-19 vaccine, which may have influenced vaccination rates in our study.

Increasing vaccination against preventable illnesses is a significant challenge in the US and around the world. This challenge has become even more important to address given the global COVID-19 pandemic. Our findings suggest that behaviorally-informed text messages delivered to patients prior to an upcoming appointment with their clinician can increase influenza vaccination. These insights could be used more broadly to improve vaccination against other diseases.

SO WHAT?

What is already know on this topic?

Nudges are subtle changes to the way information is framed or choices are offered that can have an outsized impact on behavior. Health systems are increasingly using text messaging to communicate with patients. However, the best way to design nudges within text messages to encourage vaccination has not been well-examined.

What does this article add?

In this randomized trial, 19 different text messaging approaches were compared to a control group to evaluate their impact on influenza vaccination. On average, sending any of the text messaging increased vaccination relative to control. Three of the top five performing messages described the vaccine as “reserved for you.”

What are the implications for health promotion practice or research

Text messaging is an important communication channel for health systems and other stakeholders to encourage healthy behaviors. Nudges delivered through text messages in the days prior to a primary care visit can be designed to increase vaccination during the visit and could be used more broadly to improve healthy behaviors.

Data Availability

De-identified data may be made available upon request to the authors.