Abstract
Background
Vaccine hesitancy, especially related to COVID-19 vaccinations among Veterans, may limit uptake. Behaviorally informed text-based messages have the potential to improve uptake of COVID-19 vaccinations.
Objective
To evaluate the impact of two different behaviorally informed text message nudges on COVID-19 vaccine scheduling and completion, compared to standard control message.
Design
Prospective, three-arm patient-level randomized quality improvement trial.
Participants
Between March and May 2021, 20,523 Veterans were eligible for the initial series of COVID-19 vaccination, enrolled at two large Veterans Health Administration sites.
Intervention
Arm 1 (Control): standard scheduling message; Arm 2 (Social Good): standard message plus behaviorally informed text message “When you get a vaccine now, you help protect yourself, your family, and your community”; and Arm 3 (Scarcity): standard plus behaviorally informed text message “Only a limited number of vaccine appointments are available.”
Main Measures
Outcomes were vaccine scheduling and/or completion rate within 7 days of receipt of text message (primary), and within 14 days and 30 days after receipt of text message (secondary).
Key Results
Veterans had an overall rate of 19% of scheduling or receiving a vaccination in 7 days. In our adjusted intention-to-treat analysis, we found no difference between intervention social good or scarcity (aOR 0.98, 95% CI, 0.88–1.09, for both arms) compared to standard scheduling message. We found no statistical differences in our secondary outcomes.
Conclusions
During the initial phases of vaccine roll-out, two behaviorally informed text messages did not increase COVID-19 vaccination rates among Veterans compared to a standard scheduling message.
Supplementary Information
The online version contains supplementary material available at 10.1007/s11606-024-09170-2.
KEY WORDS: population health, COVID-19 vaccine, text messaging
INTRODUCTION
The creation and development of the COVID-19 vaccine is a remarkable success story.1 Yet, this success can only be realized if health systems can quickly and effectively deliver this vaccination to most of its population. The challenge of delivering these vaccinations is complicated by many factors including vaccine hesitancy, misinformation, and skepticism.2 Only 80% of US adults are fully vaccinated against COVID, with rates as low as 30% for COVID booster in the fall of 2023.3 COVID vaccination is recommended among high-risk populations, including those cared for by the Veterans Health Administration (VHA). Veterans are sicker, poorer, more likely to be from a minority population, and have fewer support services than age-matched non-Veterans—reasons that can lead to more severe outcomes related to COVID-19 infection.4 Given the ongoing burden of COVID-19, it will be imperative to find scalable solutions to help health systems encourage patients to receive COVID-19 vaccinations.
Nudges are behavioral approaches built to combat cognitive biases (e.g., systematic errors in thinking that affect how humans make decisions) and support individuals in making optimal decisions.5 Nudges can influence a behavior, without restricting or mandating, that could potentially support health systems improve vaccination rates. Recently, a large study conducted among commercially insured patients found that text messages sent prior to a primary care visit can boost flu vaccination rates by an average of 5% compared to usual care (no reminder message).6 Other studies suggest successful nudges including default scheduling of vaccination appointments, simple text reminders, and prompting people to jot down the exact date and time when they will get a flu shot can all increase vaccination rates.7–9 However, the effectiveness of these nudges may differ by populations and for the COVID-19 vaccination.
As programs are enacted to improve COVID-19 vaccination uptake, nudges offer a simple and low-cost way to encourage behavior change. However, to be effective, they must do three major things—the nudge must prompt, enable, and motivate a behavior change.10 While standard reminder messages may prompt and enable COVID-19 vaccine access, they may not provide sufficient motivation. Behavioral-informed messaging may provide additional motivation to change. Literature from behavioral science suggests that messages around scarcity can focus attention on the scarce resources, potentially leading to more efficient use of that resource.11 Messaging about the scarcity of vaccination resource may promote a more timely response to scheduling requests. Messages around social benefits and protecting others can also help motivate people to change behavior.12,13 Messaging around social benefit of vaccination can potentially lead to social normalization and promote increased participation in vaccination efforts. Our study aimed to test the impact of two different behaviorally informed text-based message nudges on COVID-19 vaccination scheduling among Veterans. Our interventions tested whether messages around (a) scarcity or (b) social good could improve vaccination scheduling compared to a standard scheduling text.
METHODS
We conducted a pragmatic randomized clinical trial to compare the effectiveness of a behavioral-informed text message to improve scheduling and uptake of COVID-19 vaccinations (ClinicalTrials.gov NCT04805931). This trial was completed as non-research quality improvement for pragmatic evaluation of primary care operations under the designation of the VHA Office of Primary Care and was therefore not subject to institutional review board review or exemption. The VHA Office of Primary Care is committed to the development and principles of the learning health systems model, embedding research teams within operations to improve care delivery for Veterans.14–17 There were no active recruitment or informed consent procedures given the minimal risks and purpose of quality improvement. This study followed the Consolidated Standards of Reporting Trials (CONSORT) reporting guideline. The trial protocol and statistical analysis plan are provided (Appendix 1).
Study Design
We performed a randomized, three-arm superiority trial with a 1:1:1 intervention allocation conducted at two sites of the VA Puget Sound Health Care System (VAPSHCS) (Seattle and American Lake). Participants were randomized to one of three groups: a control group who received a short message service (SMS) scheduling text and two intervention groups who received different behaviorally informed SMS text encouraging scheduling of a COVID-19 vaccine. Randomization was stratified by VHA site and enrollment took place between March 15, 2021, and May 1, 2021. The prespecified primary outcome of interest was vaccine scheduling or receipt within 7 days of text message.
Sample
Participants were identified from the Veteran Service Support Center Outreach database, a nationally maintained registry of Veterans eligible for primary COVID-19 series vaccination (Fig. 1). Selection criteria included enrollment in the VAPSHCS. Veterans were excluded if previously sent a direct scheduling vaccination text message from the VHA. Participants were eligible for participation regardless of ability to receive text messages or preference to not receive text messages from VAPSHCS.
Figure 1.
Consort diagram.
Randomization
Participants were assigned to one of three groups with a 1:1:1 allocation via a computer-generated randomization schedule (R version 3.6.1), stratified by site using permuted block randomization with random block sizes of 3 and 6. A master participant list with the group assignment was sent to the administrator responsible for programming in VHA’s national VEText tool. VEText is a direct scheduling system which facilitates two-way asynchronous communication between the Veteran and the VAPSHCS using electronic health medical record (EMHR) scheduling software.18 The study statistician was blinded to participant allocation for the primary outcome analyses. Participants were unaware of the messaging received by other groups and were not directly informed of group assignment.
Interventions
The design of our intervention was focused on key principles from the Fogg Behavior Model.10,19 First, nudges or reminders can increase vaccination rates by making the process more convenient (control arm). Second, that behavioral-informed text messages can increase motivation to get or schedule a vaccination (intervention arm). Finally, rather than using stated preferences on communication, we used revealed preferences by measuring a consumer’s response to different messages to define success (outcomes).
The intervention was created in partnership between academic and operational teams. The operational team (JG, AC) had experience working in both national and local primary care leadership. They were tasked to develop outreach to engage Veterans to receive COVID-19 vaccinations. In the design, we engaged academic partners with experience in behavioral economics and marketing to create messages (NA, JL, AR). Finally, we have established a multidisciplinary evaluation team (CW, SD, LS, KN, AR) to support study design and analysis.
Given constraints on the number of messages that could be sent and designed in a rapid evaluation, we selected two common behavioral messaging campaigns: (1) social good and (2) scarcity. A social good messaging as a nudge can appeal to an individuals’ desire to contribute positively to their community.20 When patients are reminded that getting vaccinated protects vulnerable populations and aligns with actions of their peers, they are more motivated to follow suit. This principle captures the psychological behaviors that conform to social expectations. Using scarcity messaging as a nudge can effectively increase vaccination rates by creating a sense of urgency and importance. When patients perceive that vaccine appointments are in high demand, they are more likely to prioritize getting vaccinated to avoid missing out.21 This approach leverages the psychological principle that people value opportunities more when they believe they might lose them. All text messages were reviewed for face validity by members of the study team and additional primary care staff. Moreover, all messages were reviewed by the national VHA Office of Public Communication.
All messages were sent via the VHA’s VEText. VEText reaches nearly all enrolled Veterans; in 2020, the program sent 134 million appointment reminders to nearly 7 million Veterans.22 Participant unique identifiers were uploaded to a portal that validated phone numbers from the VHA EMHR and facilitated outbound text message transmission. Veterans who did not have a text messaging capable phone number or who previously opted out of receiving text messages were automatically added to a report on the portal indicating failure to transmit. All groups received a baseline message with first vaccine dose appointment availability along with questions regarding prior vaccination status and interest in vaccination. Participants had the option to either schedule directly through the VEText system or call to schedule an appointment. The intervention groups received an additional behavioral informed message embedded within the baseline message that was based on either the idea of social good or idea of scarcity (Fig. 2).
Figure 2.
Behaviorally informed text message to schedule COVID-19 vaccination.
Outcomes
Our prespecified primary outcome was the proportion of first-dose vaccine appointments scheduled or vaccine received within 7 days after the text message was delivered. The secondary prespecified outcomes included the proportion of first-dose vaccine appointments scheduled or vaccine received within 14 and 30 days after text message. Outcomes on scheduling and/or receipt were obtained from VHA Corporate Data Warehouse. After the trial was completed, additional data was obtained from the State Department of Health on receipt of COVID-19 vaccination. We were able to link Veterans with the Washington State Immunization Information System (WAIIS) to see whether patients received vaccinations outside of VAPSHCS. WAIIS is a lifetime registry that keeps track of immunization records and provided a more comprehensive assessment of receipt of vaccination outside our facility.
Statistical Analysis
Sample size estimation assumed a baseline vaccination percentage of approximately 33%, based on historical COVID-19 vaccination data obtained via an electronic health record repository at the site. A clinically meaningful difference between groups was defined as 5%, and a sample size of 1437 participants per group (total N = 4311) was estimated to achieve at least 80% power.
The intention-to-treat (ITT) analysis examined the association of text message with primary and secondary outcomes. The per-protocol (PP) analysis excluded Veterans who were not sent a text message either due to previously opting out of text message reminders or due to not having a text messaging–enabled phone. In addition, we excluded patients who may have received the COVID-19 vaccination outside of the VAPSHCS prior to receipt of the text message.
Descriptive patient-level statistics were evaluated using the Pearson chi-square test for dichotomous variables and the Student’s t test for continuous variables. Logistic regression was used to test the association between randomization group and vaccine scheduling, adjusted for VHA site as a stratification variable, as well as age (years), sex (male/female), race/ethnicity (Alaska Native/Native American, Asian/Pacific Islander/Native Hawaiian, Hispanic, non-Hispanic Black, non-Hispanic White, Multiple/Other), neighborhood socioeconomic status (SES) index which is a surrogate marker for income, rurality (urban, rural, highly rural/insular islands), and drive distance (miles) to their main assigned primary care clinic site. Odds ratios and predicted probabilities from this model are reported. All descriptive and main analyses were performed using R version 3.6.1 and p-values <0.05 were considered statistically significant.
RESULTS
Sample Characteristics
A total of 20,523 Veterans were randomized with over 6800 Veterans in each arm (Fig. 1). A very small proportion (<1%) were excluded in the per-protocol analysis due to prior vaccination or invalid/non-working phone numbers (n=517). A majority of Veterans were non-Hispanic White (61%), male (82%), and with an average age of 41.0 (SD 8.6) years (Table 1). Over 85% of Veterans lived in urban areas with an average of 16 miles from a clinical site. All Veterans patient characteristics were balanced between the control and intervention arms.
Table 1.
Baseline Demographic Characteristics by Randomization Group
| Randomized group | ||||
|---|---|---|---|---|
| Control, N = 6838a | Scarcity, N = 6842a | Social good, N = 6843a | p-valueb | |
| Age (years) | 41.05 (8.57) | 40.99 (8.51) | 41.08 (8.57) | 0.9 |
| Female sex | 1182/6408 (18%) | 1174/6444 (18%) | 1213/6413 (19%) | 0.6 |
| Race/ethnicity | >0.9 | |||
| American Indian/Alaska Native | 67 / 6512 (1.0%) | 64 / 6523 (1.0%) | 66 / 6531 (1.0%) | |
| Asian/Pac Islander/Native Hawaiian | 591 / 6512 (9.1%) | 590 / 6523 (9.0%) | 616 / 6531 (9.4%) | |
| Hispanic | 623 / 6512 (9.6%) | 661 / 6523 (10%) | 638 / 6531 (9.8%) | |
| Multi-Race/Other | 223 / 6512 (3.4%) | 221 / 6523 (3.4%) | 228 / 6531 (3.5%) | |
| Non-Hispanic Black | 1036 / 6512 (16%) | 1016 / 6523 (16%) | 1011 / 6531 (15%) | |
| Non-Hispanic White | 3972 / 6512 (61%) | 3971 / 6523 (61%) | 3972 / 6531 (61%) | |
| Marital status | 0.3 | |||
| Married | 3319 / 6838 (49%) | 3357 / 6842 (49%) | 3266 / 6843 (48%) | |
| Other | 3519 / 6838 (51%) | 3485 / 6842 (51%) | 3577 / 6843 (52%) | |
| Socioeconomic status index (decile) | 0.7 | |||
| 0 | 97 / 5028 (1.9%) | 108 / 5073 (2.1%) | 122 / 5036 (2.4%) | |
| 1 | 244 / 5028 (4.9%) | 239 / 5073 (4.7%) | 232 / 5036 (4.6%) | |
| 2 | 500 / 5028 (9.9%) | 504 / 5073 (9.9%) | 482 / 5036 (9.6%) | |
| 3 | 464 / 5028 (9.2%) | 475 / 5073 (9.4%) | 474 / 5036 (9.4%) | |
| 4 | 707 / 5028 (14%) | 724 / 5073 (14%) | 757 / 5036 (15%) | |
| 5 | 596 / 5028 (12%) | 635 / 5073 (13%) | 600 / 5036 (12%) | |
| 6 | 787 / 5028 (16%) | 753 / 5073 (15%) | 751 / 5036 (15%) | |
| 7 | 624 / 5028 (12%) | 571 / 5073 (11%) | 565 / 5036 (11%) | |
| 8 | 577 / 5028 (11%) | 617 / 5073 (12%) | 631 / 5036 (13%) | |
| 9 | 432 / 5028 (8.6%) | 447 / 5073 (8.8%) | 422 / 5036 (8.4%) | |
| Rurality | 0.6 | |||
| Urban | 5182 / 6123 (85%) | 5166 / 6081 (85%) | 5174 / 6132 (84%) | |
| Rural | 904 / 6123 (15%) | 879 / 6081 (14%) | 910 / 6132 (15%) | |
| Highly rural/insular islands | 37 / 6123 (0.6%) | 36 / 6081 (0.6%) | 48 / 6132 (0.8%) | |
| Drive distance to primary care (miles) | 16.49 (11.21) | 16.72 (11.01) | 16.56 (11.27) | 0.10 |
| VA facility | >0.9 | |||
| American Lake | 4567 / 6838 (67%) | 4570 / 6842 (67%) | 4571 / 6843 (67%) | |
| Seattle | 2271 / 6838 (33%) | 2272 / 6842 (33%) | 2272 / 6843 (33%) | |
aMean (SD); n / N (%)
bKruskal-Wallis rank sum test; Pearson’s chi-squared test
Vaccination Scheduled or Received
Table 2 displays the unadjusted rates of vaccines scheduled or received for both the ITT and PP cohort at 7, 14, and 30 days. For our primary outcome, Veterans in the ITT cohort had an overall rate of 19% of scheduling or receiving a vaccination 7 days after receipt of the text message. There was incremental increase in vaccine scheduling/receipt to 22% and 29%, at 14 and 30 days, respectively. However, we found no significant difference between the control and intervention groups (social good or scarcity). Table 3 shows the unadjusted and adjusted model results for primary outcome of vaccine scheduling/receipt within 7 days of the text message. We found no statistical difference between groups in our unadjusted and adjusted analysis. The adjusted odds ratio for the ITT analysis was 0.98 (95% CI, 0.88–1.09) for both scarcity and social good groups versus control at 7 days. Results were similar in the unadjusted and adjusted PP analysis.
Table 2.
COVID-19 vaccination after text messaging by group, ITT, and PP
| Randomized group | ||||
|---|---|---|---|---|
| Outcomes | Control, N = 6838a |
Scarcity, N = 6842a |
Social good, N = 6843a |
p-valueb |
| Intention-to-treat analysis | ||||
| By 7 days (%) | 1306 (19%) | 1314 (19%) | 1298 (19%) | >0.9 |
| By 14 days (%) | 1505 (22%) | 1514 (22%) | 1550 (23%) | 0.6 |
| By 30 days (%) | 1988 (29%) | 2004 (29%) | 2017 (29%) | 0.9 |
| Per-protocol analysis | ||||
| By 7 days (%) | 1101 (18%) | 1135 (19%) | 1096 (19%) | 0.6 |
| By 14 days (%) | 1298 (22%) | 1330 (22%) | 1344 (23%) | 0.4 |
| By 30 days (%) | 1764 (29%) | 1810 (30%) | 1799 (30%) | 0.4 |
an / N (%)
bPearson’s chi-squared test
Table 3.
Odds of COVID-19 Vaccination After Text Messaging by 7 Days, by Group, ITT, and PP
| OR | 95% CI | p-value | |
|---|---|---|---|
| Intention-to-treat analysis | |||
| Unadjusted | >0.9 | ||
| Control | — | — | |
| Scarcity | 1.01 | 0.92, 1.10 | |
| Social good | 0.99 | 0.91, 1.08 | |
| Adjusteda | 0.9 | ||
| Control | — | — | |
| Scarcity | 0.98 | 0.88, 1.08 | |
| Social good | 0.98 | 0.89, 1.08 | |
| Per-protocol analysis | |||
| Unadjusted | 0.6 | ||
| Control | — | — | |
| Scarcity | 1.05 | 0.95, 1.15 | |
| Social good | 1.01 | 0.92, 1.11 | |
| Adjusteda | >0.9 | ||
| Control | — | — | |
| Scarcity | 1.02 | 0.91, 1.14 | |
| Social good | 1.00 | 0.90, 1.12 | |
OR, odds ratio; CI, confidence interval
aAdjustment covariates: age, sex, race/ethnicity, SES index, rurality, drive distance to PC
DISCUSSION
In this pragmatic randomized clinical trial, behaviorally informed text message nudges that focused on social good or scarcity messages did not improve Veteran scheduling or receiving COVID-19 vaccination compared to a standard reminder text. Moreover, the overall low uptake of COVID-19 suggests that Veterans may need additional approaches to increase COVID-19 vaccine uptake.
Overall, our findings reflect several similarities of Veteran behaviors and the general public in regard to COVID-19 vaccine uptake.23 First, the low vaccination rates we report are similar to national vaccination rates in younger Veterans during this time period. A retrospective study of COVID-19 vaccination rates in VHA as of June 2021 for Veterans aged 18–49 was less than 40%.24 Second, vaccine hesitancy and resistance rates are similar between Veterans and the general US public (36 vs. 33%, respectively).23 Vaccine hesitancy is multifaceted with no single reason that can explain why a Veteran may or may not get vaccinated.25 The most common cited reasons for Veterans not getting the vaccine were adverse side effects, lack of trust in the healthcare system, perceived novelty of the vaccine, and preferences for natural immunity.23 A number of studies from the general population have also documented similar reasons.26,27 Therefore, a simple behavioral informed messages and/or ease of vaccine scheduling may not be enough to get Veterans to schedule or receive COVID-19 vaccinations. Many public health efforts suggest that more tailored and individual conversations, especially with trusted peers, will have more influence to encourage vaccination.28
Our work also adds to the mixed evidence within the field of behavioral science that simple nudges can impact COVID vaccination rates. Our findings suggest that behavioral messaging may not provide sufficient motivation over standard messaging. All the messages provided prompts and reminders improving the ease to schedule a vaccination. However, the behavioral-informed messages did not significantly increase uptake of scheduling or receipt of the vaccine over the standard message. Two studies published after we completed our trial showed that reminders can increase COVID-19 vaccinations rates; however, both demonstrated that the influence of text-based nudges was between 2 and 5%.29,30 Other studies demonstrated that behavioral-informed messages did not improve COVID-19 uptake and may have limited effects in more vaccine-hesitant populations.21,28 An increasingly robust literature suggests that certain messaging maybe more effective than others in COVID-19 vaccine campaigns.31–33 Newer study designs such as the “megastudy” of text-based nudges to encourage flu vaccination tested over 19 different messages may provide the necessary information to find the ideal national message campaign and may lead to more targeted message for specific populations.6 But nudges and messaging alone may fail because the population impacted is heterogenous in their attitudes toward COVID-19 vaccinations. For example, if a Veteran is just apathetic to getting a vaccine, a simple prompt can be effective lowering the threshold to change behavior by making it easier to gain access and schedule the vaccination. However, if a Veteran is more vaccine hesitant, a nudge may not provide enough motivation to get the vaccine. While we only tested a limited number of behavioral-informed text messages to improve uptake, more high-touch interventions maybe needed to motivate vaccine-hesitant Veterans.
Limitations
This study had several limitations. First, the study took place among Veterans in a specific region of the country, and our results may not generalize to populations outside our region or non-Veterans. Second, we only tested two behaviorally informed messages. While these messages were guided by academic and operational partners and input from behavioral science experts, these messages may not have met the needs of our unique population. Third, we also did not test the timing or frequency of text messages—both of which have previously be shown to influence response.34 These decisions and constraints were made in conjunction with our operation partners and reflect the pragmatic nature of our trial.
CONCLUSION
This randomized clinical trial of behavioral-informed text message nudges to improve COVID-19 vaccination rates in Veterans did not improve vaccine uptake compared to a standard messaging reminder. Other more personalized interventions may be needed to address vaccine hesitancy and skepticism among Veterans.23,25
Supplementary Information
Below is the link to the electronic supplementary material.
Acknowledgements
This work was made possible through operational support and funding from VA Puget Sound Health Care System Leadership. Additionally, this work was funded by the Primary Care Analytics Team through the Veterans Health Administration Office of Primary Care. The funder had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication. The views expressed in this article are those of the authors and do not necessarily reflect the position or policy of the US government, the Department of Veterans Affairs, and the University of Washington.
Declarations:
Conflict of Interest:
The authors declare that they do not have a conflict of interest.
Footnotes
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
References
- 1.Fauci AS. The story behind COVID-19 vaccines. Science. 2021;372(6538):109-109. 10.1126/science.abi8397 [DOI] [PubMed] [Google Scholar]
- 2.Coustasse A, Kimble C, Maxik K. COVID-19 and Vaccine Hesitancy: A Challenge the United States Must Overcome. J Ambulatory Care Manage. 2021;44(1):71-75. 10.1097/JAC.0000000000000360 [DOI] [PubMed] [Google Scholar]
- 3.U.S. Department of Health And Human Services, CDC. Centers for Disease Control and Prevention COVID Data Tracker.; 2024. https://covid.cdc.gov/covid-data-tracker
- 4.Shulkin DJ. Why VA health care is different. fed pract health care prof VA DoD PHS. 2016;33(5):9-11. [PMC free article] [PubMed] [Google Scholar]
- 5.Thaler RH, Sunstein CR. Nudge: The Final Edition. Updated edition. Penguin Books, an imprint of Penguin Random House LLC; 2021.
- 6.Milkman KL, Patel MS, Gandhi L, et al. A megastudy of text-based nudges encouraging patients to get vaccinated at an upcoming doctor’s appointment. Proc Natl Acad Sci. 2021;118(20):e2101165118. 10.1073/pnas.2101165118 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Milkman KL, Beshears J, Choi JJ, Laibson D, Madrian BC. Using implementation intentions prompts to enhance influenza vaccination rates. Proc Natl Acad Sci U S A. 2011;108(26):10415-10420. 10.1073/pnas.1103170108 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Chapman GB, Li M, Leventhal H, Leventhal EA. Default clinic appointments promote influenza vaccination uptake without a displacement effect. Behav Sci Policy. 2016;2(2):40-50. 10.1353/bsp.2016.0014 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Regan AK, Bloomfield L, Peters I, Effler PV. Randomized Controlled Trial of Text Message Reminders for Increasing Influenza Vaccination. Ann Fam Med. 2017;15(6):507-514. 10.1370/afm.2120 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Patel MS. Text-message nudges encourage COVID vaccination. Nature. 2021;597(7876):336-337. 10.1038/d41586-021-02043-2 [DOI] [PubMed] [Google Scholar]
- 11.Aggarwal P, Jun SY, Huh JH. Scarcity Messages. J Advert. 2011;40(3):19-30. 10.2753/JOA0091-3367400302 [Google Scholar]
- 12.Opel DJ, Reich JA, Peek ME. Amplifying Appeals to the Common Good in COVID-19 Vaccine Messaging. JAMA Health Forum. 2022;3(5):e220991. 10.1001/jamahealthforum.2022.0991 [DOI] [PubMed] [Google Scholar]
- 13.Korn L, Böhm R, Meier NW, Betsch C. Vaccination as a social contract. Proc Natl Acad Sci. 2020;117(26):14890-14899. 10.1073/pnas.1919666117 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Nelson K, Reddy A, Stockdale SE, et al. The Primary Care Analytics Team: Integrating research and clinical care within the Veterans Health Administration Office of Primary Care. Healthc Amst Neth. 2021;8 Suppl 1:100491. 10.1016/j.hjdsi.2020.100491 [DOI] [PubMed] [Google Scholar]
- 15.Deeds S, Schuttner L, Wheat C, et al. Automated Reminders Enhance Mailed Fecal Immunochemical Test Completion Among Veterans: a Randomized Controlled Trial. J Gen Intern Med. 2024;39(1):113-119. 10.1007/s11606-023-08409-8 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Deeds S, Liu T, Schuttner L, et al. A Postcard Primer Prior to Mailed Fecal Immunochemical Test Among Veterans: a Randomized Controlled Trial. J Gen Intern Med. 2023;38(14):3235-3241. 10.1007/s11606-023-08248-7 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Marcotte LM, Deeds S, Wheat C, et al. Automated Opt-Out vs Opt-In Patient Outreach Strategies for Breast Cancer Screening: A Randomized Clinical Trial. JAMA Intern Med. 2023;183(11):1187-1194. 10.1001/jamainternmed.2023.4321 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.Deeds SA, Moore CB, Gunnink EJ, et al. Implementation of a mailed faecal immunochemical test programme for colorectal cancer screening among Veterans. BMJ Open Qual. 2022;11(4):e001927. 10.1136/bmjoq-2022-001927 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Fogg B. A behavior model for persuasive design. In: Proceedings of the 4th International Conference on Persuasive Technology. ACM; 2009:1-7. 10.1145/1541948.1541999
- 20.Bicchieri C, Dimant E. Nudging with care: the risks and benefits of social information. Public Choice. 2022;191(3-4):443-464. 10.1007/s11127-019-00684-6 [Google Scholar]
- 21.Mehta SJ, Mallozzi C, Shaw PA, et al. Effect of Text Messaging and Behavioral Interventions on COVID-19 Vaccination Uptake: A Randomized Clinical Trial. JAMA Netw Open. 2022;5(6):e2216649. 10.1001/jamanetworkopen.2022.16649 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.United States Department of Veterans Affairs. VA Employee Technical Innovation Reduces Nationwide Appointment No-Shows. DigitalVA. Published January 7, 2020. https://digital.va.gov/general/va-employee-technical-innovation-reduces-nationwide-appointment-no-shows/
- 23.Jasuja GK, Meterko M, Bradshaw LD, et al. Attitudes and Intentions of US Veterans Regarding COVID-19 Vaccination. JAMA Netw Open. 2021;4(11):e2132548. 10.1001/jamanetworkopen.2021.32548 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 24.Bajema KL, Rowneki M, Berry K, et al. Rates of and Factors Associated With Primary and Booster COVID-19 Vaccine Receipt by US Veterans, December 2020 to June 2022. JAMA Netw Open. 2023;6(2):e2254387. 10.1001/jamanetworkopen.2022.54387 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 25.Elwy AR, Clayman ML, LoBrutto L, et al. Vaccine hesitancy as an opportunity for engagement: A rapid qualitative study of patients and employees in the U.S. Veterans Affairs healthcare system. Vaccine X. 2021;9:100116. 10.1016/j.jvacx.2021.100116 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 26.Lin C, Tu P, Beitsch LM. Confidence and Receptivity for COVID-19 Vaccines: A Rapid Systematic Review. Vaccines. 2020;9(1):16. 10.3390/vaccines9010016 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 27.Fisher KA, Bloomstone SJ, Walder J, Crawford S, Fouayzi H, Mazor KM. Attitudes Toward a Potential SARS-CoV-2 Vaccine : A Survey of U.S. Adults. Ann Intern Med. 2020;173(12):964-973. 10.7326/M20-3569 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 28.Rabb N, Swindal M, Glick D, et al. Evidence from a statewide vaccination RCT shows the limits of nudges. Nature. 2022;604(7904):E1-E7. 10.1038/s41586-022-04526-2 [DOI] [PubMed] [Google Scholar]
- 29.Patel MS, Fogel R, Winegar AL, et al. Effect of Text Message Reminders and Vaccine Reservations on Adherence to a Health System COVID-19 Vaccination Policy: A Randomized Clinical Trial. JAMA Netw Open. 2022;5(7):e2222116. 10.1001/jamanetworkopen.2022.22116 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 30.Dai H, Saccardo S, Han MA, et al. Behavioural nudges increase COVID-19 vaccinations. Nature. 2021;597(7876):404-409. 10.1038/s41586-021-03843-2 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 31.Robertson DA, Timmons S, Lunn PD. Behavioural evidence on COVID-19 vaccine uptake. Public Health. 2024;227:49-53. 10.1016/j.puhe.2023.10.046 [DOI] [PubMed] [Google Scholar]
- 32.Jensen UT, Ayers S, Koskan AM. Video-based messages to reduce COVID-19 vaccine hesitancy and nudge vaccination intentions. PloS One. 2022;17(4):e0265736. 10.1371/journal.pone.0265736 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 33.Batteux E, Mills F, Jones LF, Symons C, Weston D. The Effectiveness of Interventions for Increasing COVID-19 Vaccine Uptake: A Systematic Review. Vaccines. 2022;10(3):386. 10.3390/vaccines10030386 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 34.Johansen ND, Vaduganathan M, Bhatt AS, et al. Electronic nudges to increase influenza vaccination uptake in Denmark: a nationwide, pragmatic, registry-based, randomised implementation trial. Lancet Lond Engl. 2023;401(10382):1103-1114. 10.1016/S0140-6736(23)00349-5 [DOI] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.