Key Points
Question
Can electronically delivered letter-based nudges informed by behavioral science increase influenza vaccination uptake among young and middle-aged patients with chronic diseases?
Findings
In this randomized clinical trial including 299 881 Danish patients aged 18 to 64 years with chronic diseases, participants were randomized to receive 6 different behaviorally informed electronic letters or usual care of no letter. All 6 intervention strategies markedly increased influenza vaccination rates with absolute effect sizes ranging between 11-14 percentage points.
Meaning
The results of this study suggest that simple, scalable, and cost-efficient electronic letter strategies may have substantial public health implications.
Abstract
Importance
Despite strong worldwide guideline recommendations, influenza vaccination rates remain suboptimal among young and middle-aged patients with chronic diseases. Effective scalable strategies to increase vaccination are needed.
Objective
To investigate whether electronically delivered letter-based nudges informed by behavioral science could increase influenza vaccination uptake among patients aged 18 to 64 years with chronic diseases.
Design, Setting, and Participants
Nationwide pragmatic registry-based randomized clinical implementation trial conducted between September 24, 2023, and May 31, 2024, enrolling all Danish citizens aged 18 to 64 years who met criteria for free-of-charge influenza vaccination in light of preexisting chronic disease. All trial data were sourced from nationwide administrative health registries.
Intervention
Randomized in 2.45:1:1:1:1:1:1 ratio to no letter (usual care) or 6 different behaviorally informed electronic letters.
Main Outcomes and Measures
The primary end point was receipt of influenza vaccination on or before January 1, 2024, assessed in 7 prespecified coprimary comparisons (all intervention groups pooled vs usual care and each individual intervention group vs usual care). Absolute risk difference in proportions and a crude relative risk were calculated for each comparison.
Results
A total of 299 881 participants (53.2% [159 454] female, median age, 52.0 [IQR, 39.8-59.0] years) were randomized. Compared with usual care, influenza vaccination rates were higher among those receiving any intervention letter (any intervention letter, 39.6% vs usual care, 27.9%; difference, 11.7 percentage points; 99.29% CI, 11.2-12.2 percentage points; P < .001). Each individual letter type significantly increased influenza vaccination with the largest effect sizes observed with a repeated letter sent 10 days after the initial letter (repeated letter, 41.8% vs usual care, 27.9%; difference, 13.9 percentage points; 99.29% CI, 13.1-14.7 percentage points; P < .001) and a letter emphasizing potential cardiovascular benefits of vaccination (cardiovascular gain, 39.8% vs usual care, 27.9%; difference, 11.9 percentage points; 99.29% CI, 11.1-12.7 percentage points; P < .001). Vaccination rates were improved across major subgroups.
Conclusions and Relevance
In a nationwide randomized clinical implementation trial, electronically delivered letter-based nudges markedly increased influenza vaccination compared with usual care among young and middle-aged patients with chronic diseases. The results of this study suggest that simple, scalable, and cost-efficient electronic letter strategies may have substantial public health implications.
Trial Registration
ClinicalTrials.gov Identifier: NCT06030739
This randomized clinical trial evaluates whether behaviorally informed electronically delivered letter-based nudges could increase influenza vaccination among adults with chronic diseases.
Introduction
Seasonal influenza constitutes a major public health burden and is estimated to cause 290 000 to 650 000 respiratory deaths worldwide per year.1 Annual influenza vaccination is effective in reducing the risk of influenza infection and its associated complications,2,3,4,5,6,7,8 but despite strong guideline recommendations from worldwide public health authorities, vaccination rates remain suboptimal, particularly among young and middle-aged adults.9,10,11 During the 2022 to 2023 influenza season in Denmark among patients aged 18 to 64 years, only 40.7% of patients with diabetes and 44.6% of patients with heart failure obtained vaccination.10,11 Therefore, effective, scalable, and cost-efficient strategies to increase influenza vaccination in this population are needed.
Letters and short messages incorporating behavioral science principles have previously been shown to be effective in increasing vaccination rates at a population level.12,13,14 During the 2022 to 2023 influenza season, the randomized Nationwide Utilization of Danish Government Electronic Letter System for Increasing Influenza Vaccine Uptake (NUDGE-FLU) trial used the unique Danish digital health infrastructure15 to demonstrate effectiveness of behaviorally informed letter-based nudges sent via the Danish governmental electronic letter system (Digital Post),16 and incorporating repeated messaging and information on potential cardiovascular (CV) benefits of vaccination in increasing influenza vaccination rates among adults aged 65 years or older.17 The results had a direct impact on health policy: during the ongoing 2023 to 2024 season, the Danish Health Authority implemented a repeated letter strategy for their informational letter delivered to adults aged 65 years or older.
In NUDGE-FLU, absolute improvements in vaccination rates were modest; however, background vaccination rates were very high at more than 80%, potentially limiting the number of participants who could be nudged. Chronic diseases are known to increase risk of influenza-related complications, even among younger populations. However, to date, no prior large-scale trial has specifically evaluated public health messaging among young and middle-aged patients with chronic diseases. Therefore, we designed the Nationwide Utilization of Danish Government Electronic Letter System for Increasing Influenza Vaccine Uptake Among Adults With Chronic Disease (NUDGE-FLU-CHRONIC) trial to evaluate whether behaviorally informed electronically delivered letter-based nudges could increase influenza vaccination among patients aged 18 to 64 years with chronic diseases and to exploratively assess whether such an increase would lead to improved clinical outcomes.
Methods
Study Design and Oversight
NUDGE-FLU-CHRONIC was a pragmatic nationwide randomized implementation trial conducted in Denmark between September 24, 2023, and May 31, 2024 (during the 2023 to 2024 northern hemisphere influenza season). The study design has previously been described in detail.18 Pragmatic trial elements included using broad eligibility criteria, enrolling all Danish citizens fulfilling these criteria, not requiring informed consent or in-person follow-up, and collecting routine health care data from administrative registries. An implementation trial is a trial that does not test new therapies but instead investigates strategies to implement and improve uptake of existing evidence-based interventions. The trial was granted a waiver from needing informed consent from participants by the Committees on Health Research Ethics in the Capital Region of Denmark, as the trial tested communication strategies without limiting participant autonomy. The study was approved by the data authority in the Capital Region of Denmark. The Danish Health Data Authority granted nationwide registry data access and approved the content of the intervention letters.
The study is reported according to the Consolidated Standards of Reporting Trials (CONSORT) 2010 reporting guideline. The trial protocol and the statistical analysis plan are available in Supplement 1.
Participants, Group Randomization, and Blinding
The trial included all Danish citizens aged 18 years or older who would not turn 65 until January 16, 2024, at the earliest and who had a chronic disease known to be associated with an increased risk of adverse influenza-related outcomes, a population that is eligible for free-of-charge influenza vaccination through the Danish governmental vaccination program.19 Participants were identified through nationwide administrative health registries using prespecified definitions based on International Statistical Classification of Diseases and Related Health Problems, Tenth Revision (ICD-10) codes (eTable 1 in Supplement 2). The disease-specific criteria were chosen based on recommendations from Statens Serum Institut, the institution responsible for infectious disease preparedness and control in Denmark, and included chronic CV and respiratory diseases, cancer, and diabetes, among others. Only patients without an exemption from the Danish governmental electronic letter system used for intervention delivery were included.
Participants were individually randomized in a 2.45:1:1:1:1:1:1 ratio20 to either usual care or 6 different active groups, each receiving different behaviorally informed electronic letters nudging participants to make the decision to obtain influenza vaccination. Randomization was not stratified. The intervention was not formally blinded, but given that consent was not required from patients, the usual care group was unaware that they acted as the control group in a randomized clinical trial.
Interventions
All intervention letters were written in Danish and based on the same standard letter template (eFigure 1 in Supplement 2), which was a 1-page electronic letter including information on why the recipient was eligible for free-of-charge vaccination, a short description of the risks associated with influenza among patients with chronic diseases, instructions on how to schedule vaccination, and details regarding potential coadministration with COVID-19 vaccines. The standard letter template was then varied across intervention groups to test different behavioral science concepts. A highlighted behavioral nudge was included in several of the letters. The trial had the following 6 intervention groups: (1) standard letter; (2) standard letter sent at baseline and repeated after 10 days (repeated letter); (3) CV gain-framing letter; (4) respiratory disease gain–framing letter; (5) active choice/implementation intention prompt letter; and (6) loss-framing letter (eTable 2 in Supplement 2). The specific interventions were selected based on prior evidence. The standard letter was included to estimate the effect of the standard letter template without a highlighted nudge. The repeated letter and the CV gain-framing letter had previously been effective in the NUDGE-FLU trial.17 The respiratory disease gain–framing letter was chosen to assess whether the previous success of the CV gain-framing letter relied on specifically mentioning CV benefits or whether informing on other disease-specific benefits could be similarly effective. Several prior trials12,21,22 have found an implementation intention prompt letter to be effective. Finally, the loss-framing letter was included as a way to assess the effects of loss-framing vs gain-framing messaging in a younger population where perceived risk associated with influenza may be lower compared with the older population included in NUDGE-FLU.
All letters were delivered on September 24, 2023, a week prior to the start of the governmental influenza vaccination period. The repeated letter was delivered on October 4, 2023. The study timeline is illustrated in eFigure 2 in Supplement 2. All letters included a link to the official booking website. Booking via phone call was also available.
The intervention letters were delivered through the Danish governmental electronic letter system (Digital Post) primarily used for official communication from public authorities, banks, and hospitals. All Danish citizens automatically receive official correspondence through this system from the age of 15 but can apply for an exemption to receive physical letters instead. Researchers are allowed to use the system for trial recruitment and communication during trial conduct. The system is similar to an email system and accessible through several web portals and smartphone apps. Notifications are sent via regular email and text message when a new electronic letter is received (eFigure 3 in Supplement 2).
The letters included a customized sender based on the recipients’ region of residence. The named senders were infectious disease attending physicians from university hospitals. The Capital Region of Denmark, the most populous of the 5 Danish administrative entities responsible for delivering public health care, figured as the sender in the mailbox. Participants randomized to the usual care group did not receive any letter on influenza vaccination but were exposed to any other standard public health vaccination campaigns.
Data Sources and End Points
The trial relied solely on nationwide Danish administrative health registries for data collection. These registries contain data from all health care encounters throughout the universal Danish health care system. All baseline and end point data were retrieved using prespecified definitions based on ICD-10, procedural, and Anatomical Therapeutic Chemical classification codes (eTables 3-6 in Supplement 2). The study database is situated at a secure server administered by the Danish Health Data Authority.
The primary end point of the trial was the receipt of influenza vaccination (Anatomical Therapeutic Chemical classification code: J07BB) on or before January 1, 2024. The secondary end point was time from intervention delivery to vaccination. Primary and secondary end point data were retrieved from the Danish Vaccination Registry.23 In Denmark, it is mandated by law to register all administered vaccines in the Danish Vaccination Registry, ensuring high data validity and completeness. Prespecified exploratory clinical outcomes included respiratory and CV hospitalizations, mortality, and primary care utilization, and participants were followed up for these outcomes from the date of initial letter delivery (September 24, 2023) through May 31, 2024. The full list of prespecified clinical outcomes can be found in eTable 6 in Supplement 2.
Statistical Analysis
The primary end point was assessed in 7 prespecified coprimary comparisons: all intervention groups pooled vs usual care and each individual intervention group vs usual care. A total α level of .05 was split evenly across all coprimary comparisons, testing each at an α level of .0071. We present an absolute difference in proportions and a crude relative risk for each comparison. The α-adjusted CIs are given along with unadjusted P values derived from χ2 tests.
We conducted prespecified sensitivity analyses to account for potential household-level clustering using binomial regression models with household-level clustered SEs. We did post hoc analyses comparing the repeated letter with the standard letter and the CV gain-framing letter with the standard letter. These analyses are reported at the same α level as the primary analyses to maintain consistency in reporting.
We assessed the secondary end point via the same prespecified comparisons as for the primary end point using the Kaplan-Meier procedure and Cox proportional hazards regression models with participants followed up from initial intervention delivery until death, emigration, or January 1, 2024, whichever occurred first. We tested the Cox proportional hazards regression model assumption using Schoenfeld residuals. The secondary end point was to be assessed at a total α level of .05 regardless of the primary results, also testing each comparison at an α level of .0071.
We tested for homogenous treatment effects across prespecified subgroups for the primary end point for the comparison of all intervention groups pooled vs usual care using binomial regression with identity link and interaction terms. Two-sided interaction P values <.05 were considered statistically significant.
For the exploratory assessment of clinical outcomes, we focused on the comparison of all intervention groups pooled vs usual care. Time-to-first-event end points were analyzed using cause-specific Cox proportional hazards regression models. Participants were followed up from initial intervention delivery until end of follow-up (May 31, 2024), (non-CV) death, or emigration, whichever occurred first. The number of general practitioner contacts end point was modeled using negative binomial regression. Effect estimates for these exploratory outcomes are shown with nominal 95% CIs without formal hypothesis testing.
The trial concept allowed for inclusion of all eligible participants in Denmark to maximize statistical power, and the detectable effect sizes were therefore determined by the available sample size. Based on an expected sample size of 325 000, each of the 6 comparisons of intervention group vs usual care had 80% power to detect a 1.05% absolute increase in the primary end point at a total α level of 0.05, assuming an expected influenza vaccination rate in the usual care group of 40%. All statistical analyses were performed using SAS Software, version 9.4 (SAS Institute) and Stata MP, version 18.0 (StataCorp).
Results
A total of 313 485 persons aged 18 to 64 years in Denmark met the disease-specific eligibility criteria, of whom 299 881 (95.7%) did not have an exemption from the governmental electronic letter system and were therefore included and randomized in the trial (Figure 1). Baseline characteristics were balanced across study groups (Table 1). Median age was 52.0 years (IQR, 39.8 to 59.0 years), and 159 454 participants (53.2%) were female. During the preceding 2022 to 2023 influenza season, 95 479 participants (31.8%) had received influenza vaccination. Most participants (85.2%) met only 1 disease-specific inclusion criterion; 44 286 participants (14.8%) met at least 2 criteria. A total of 61 707 participants (20.6%) met the CV disease eligibility criterion, 36 541 (12.2%) met the lung disease criterion, 43 427 (14.5%) met the diabetes criterion, and 158 776 (53.0%) had other chronic conditions (inclusive of cancer as well as hematologic, neurologic, and autoimmune diseases). Persons who were excluded (n = 13 604 [4.3%]) due to exemption from the electronic letter system were older, more likely to be male and to have received influenza vaccination during the preceding season, and had a higher prevalence of most comorbidities compared with those who were included in the trial (eTable 7 in Supplement 2).
Figure 1. Identification, Exclusion, and Randomization of Participants in the Trial.
Eligible participants were randomized in a 2.45:1:1:1:1:1:1 ratio to either usual care or 6 different active groups, each receiving different behaviorally informed electronic letters nudging participants to make the decision to obtain influenza vaccination (letters are described in the Methods section). No exclusions were made after randomization and group assignment. Citizens may apply for an exemption from the electronic letter system to receive physical letters instead. Exemptions are granted for several reasons including physical or mental impairment and lack of computer/internet access.
Table 1. Baseline Characteristics Across Randomization Groups.
| Characteristica | Randomization group, No. (%) | ||||||
|---|---|---|---|---|---|---|---|
| Letter type | Usual care (n = 87 059) | ||||||
| Loss-framing (n = 35 385)b | Implementation prompt (n = 35 701) | Respiratory gain (n = 35 320) | CV gain (n = 35 271) | Repeated letter (n = 35 649) | Standard letter (n = 35 496) | ||
| Demographic | |||||||
| Age, median (IQR), y | 52.1 (40.1-59.1) | 51.9 (39.9-58.9) | 51.9 (39.8-59.0) | 51.9 (39.6-58.9) | 52.0 (39.7-59.0) | 52.0 (39.7-59.0) | 52.0 (39.9-59.0) |
| Sex | |||||||
| Female | 18 745 (53.0) | 19 082 (53.4) | 18 678 (52.9) | 18 728 (53.1) | 19 038 (53.4) | 18 746 (52.8) | 46 437 (53.3) |
| Male | 16 640 (47.0) | 16 619 (46.6) | 16 642 (47.1) | 16 543 (46.9) | 16 611 (46.6) | 16 750 (47.2) | 40 622 (46.7) |
| Lived in household with ≥1 other trial participant | 3539 (10.0) | 3623 (10.1) | 3540 (10.0) | 3490 (9.9) | 3561 (10.0) | 3582 (10.1) | 8552 (9.8) |
| Comorbidities | |||||||
| Hypertension | 12 475 (35.3) | 12 695 (35.6) | 12 455 (35.3) | 12 279 (34.8) | 12 539 (35.2) | 12 818 (36.1) | 31 010 (35.6) |
| Cancer | 6223 (17.6) | 6175 (17.3) | 6124 (17.3) | 6204 (17.6) | 6251 (17.5) | 6306 (17.8) | 15 243 (17.5) |
| Any psychiatric diagnosis | 6030 (17.0) | 5988 (16.8) | 5978 (16.9) | 5951 (16.9) | 6129 (17.2) | 6000 (16.9) | 14 878 (17.1) |
| Ischemic heart disease | 3255 (9.2) | 3209 (9.0) | 3177 (9.0) | 3161 (9.0) | 3168 (8.9) | 3304 (9.3) | 7922 (9.1) |
| Atrial fibrillation | 2293 (6.5) | 2304 (6.5) | 2338 (6.6) | 2230 (6.3) | 2262 (6.3) | 2277 (6.4) | 5777 (6.6) |
| Substance misuse | 1752 (5.0) | 1677 (4.7) | 1725 (4.9) | 1703 (4.8) | 1712 (4.8) | 1738 (4.9) | 4239 (4.9) |
| Heart failure | 1386 (3.9) | 1484 (4.2) | 1360 (3.9) | 1291 (3.7) | 1323 (3.7) | 1468 (4.1) | 3415 (3.9) |
| Cerebrovascular disease | 1355 (3.8) | 1318 (3.7) | 1325 (3.8) | 1279 (3.6) | 1331 (3.7) | 1260 (3.5) | 3199 (3.7) |
| Chronic obstructive pulmonary disease | 1254 (3.5) | 1210 (3.4) | 1262 (3.6) | 1247 (3.5) | 1243 (3.5) | 1231 (3.5) | 2939 (3.4) |
| Disease-specific eligibility criteriac | |||||||
| Other chronic conditions with increased risk of severe influenzad | 18 687 (52.8) | 18 808 (52.7) | 18 627 (52.7) | 18 841 (53.4) | 18 995 (53.3) | 18 727 (52.8) | 46 091 (52.9) |
| Chronic CV disease | 7278 (20.6) | 7379 (20.7) | 7338 (20.8) | 7220 (20.5) | 7224 (20.3) | 7364 (20.7) | 17 904 (20.6) |
| Type 1 or 2 diabetes | 5239 (14.8) | 5109 (14.3) | 5124 (14.5) | 4952 (14.0) | 5199 (14.6) | 5257 (14.8) | 12 547 (14.4) |
| Chronic lung disease | 4314 (12.2) | 4343 (12.2) | 4379 (12.4) | 4247 (12.0) | 4406 (12.4) | 4311 (12.1) | 10 541 (12.1) |
| Immunodeficiency | 3955 (11.2) | 4045 (11.3) | 3891 (11.0) | 3916 (11.1) | 3923 (11.0) | 3870 (10.9) | 9764 (11.2) |
| Kidney or liver disease | 1806 (5.1) | 1828 (5.1) | 1856 (5.3) | 1780 (5.0) | 1731 (4.9) | 1844 (5.2) | 4370 (5.0) |
| Impaired breathing due to muscular weakness | 424 (1.2) | 450 (1.3) | 399 (1.1) | 460 (1.3) | 403 (1.1) | 426 (1.2) | 1042 (1.2) |
| Influenza vaccination in preceding season | 11 233 (31.7) | 11 333 (31.7) | 11 187 (31.7) | 11 242 (31.9) | 11 469 (32.2) | 11 310 (31.9) | 27 705 (31.8) |
Abbreviation: CV, cardiovascular.
The prespecified definitions used for disease-specific inclusion criteria and baseline characteristics are shown in eTables 3-6 in Supplement 2.
Participants were randomized in a 2.45:1:1:1:1:1:1 ratio to the 7 study groups.
The disease-specific inclusion criteria used stricter definitions than other baseline characteristics to increase specificity.
The inclusion criterion included cancer as well as hematologic, neurologic, and autoimmune diseases.
During the 2023 to 2024 season, a total of 108 471 randomized participants (36.2%) obtained influenza vaccination. Participants receiving any intervention letter had higher vaccination rates compared with usual care (39.6% [84 182 of 212 822 participants] in any intervention group vs 27.9% [24 289 of 87 059 participants] in the usual care group; difference, 11.7 percentage points; 99.29% CI, 11.2-12.2 percentage points; P < .001) (Figure 2). Each individual letter significantly increased influenza vaccination rates. The largest effect sizes were observed with the repeated letter strategy (41.8% [14 906 of 35 649 participants] in the repeated letter group vs 27.9% [24 289 of 87 059 participants] in the usual care group; difference, 13.9 percentage points; 99.29% CI, 13.1-14.7 percentage points; P < .001) and the CV gain-framing letter (39.8% [14 028 of 35 271 participants] in the CV gain group vs 27.9% [24 289 of 87 059 participants] in the usual care group; difference, 11.9 percentage points; 99.29% CI, 11.1-12.7 percentage points; P < .001). A sensitivity analysis accounting for household-level clustering yielded similar results (eTable 8 in Supplement 2). In a post hoc analysis, both the repeated letter and the CV gain-framing letter resulted in greater increases in influenza vaccination than the standard intervention letter (repeated letter vs standard letter: difference, 3.2 percentage points; 99.29% CI, 2.2-4.2 percentage points; P < .001; CV gain-framing letter vs standard letter: difference, 1.2 percentage points; 99.29% CI, 0.2-2.2 percentage points; P = .002). The results correspond to the number of individuals needed to be nudged to result in 1 additional vaccinated individual of 9 (99.29% CI, 8-9) for the comparison of any intervention letter vs usual care and 7 (99.29% CI, 7-8) for the repeated letter.
Figure 2. Results for Primary End Point of Influenza Vaccination Receipt.
The primary end point was receipt of influenza vaccination on or before January 1, 2024. A total of 7 prespecified comparisons were made and adjusted for multiplicity with each comparison performed at an α level of .0071. Nominal 99.29% CIs are reported to maintain an overall α level of .05 for the primary end point. Unadjusted P values are shown. The primary analyses shown in this Figure do not account for censoring. In total, during follow-up for receipt of influenza vaccination, 1008 participants (0.3%) died and 382 participants (0.1%) emigrated. CV indicates cardiovascular.
The secondary end point results were similar to the primary results (Figure 3 and Figure 4). When modeling influenza vaccination as a time-to-event outcome, participants receiving any intervention letter remained more likely to obtain vaccination compared with usual care (hazard ratio [HR], 1.56; 99.29% CI, 1.53-1.59; P < .001).
Figure 3. Results for 3 of 7 Comparisons for Secondary End Point of Time to Influenza Vaccination Receipts.
The secondary end point was influenza vaccination modeled as a time-to-event outcome. Cumulative incidence curves were estimated using the Kaplan-Meier method. Time 0 was the date of initial intervention delivery for all participants (September 24, 2023). Participants were censored at end of follow-up (January 1, 2024), death, or emigration, whichever occurred first. The numbers at risk at baseline differ from the total number of randomized participants in each group due to censoring occurring on the first day of follow-up. Cox proportional hazards regression models were used to estimate hazard ratios (HRs). No violations of the proportional hazards assumption were found. A total of 7 prespecified comparisons were made and adjusted for multiplicity with each comparison performed at an α level of .0071. Nominal 99.29% CIs are reported to maintain an overall α level of .05 for the secondary end point. Unadjusted P values are shown.
Figure 4. Results for 4 of 7 Comparisons for Secondary End Point of Time to Influenza Vaccination Receipts.
The secondary end point details are reported in the caption to Figure 3. CV indicates cardiovascular; HR, hazard ratio.
While the directionality of effect on the primary end point clearly favored any intervention letter in all subgroups, incremental effectiveness was suggested among older participants, those who were unvaccinated during the preceding season, those who had qualified due to chronic CV disease, and those who had not qualified due to type 1 or 2 diabetes (Figure 5).
Figure 5. Primary End Point Across Prespecified Subgroups for Comparison of Any Intervention Letter vs Usual Care.
We tested for homogeneous treatment effects across prespecified subgroups for the primary end point for the comparison of all intervention groups pooled vs usual care using binomial regression with identity link and interaction terms. Median age was 52.0 (IQR, 39.8-59.0) years. The P values shown are interaction P values.
The results for the prespecified exploratory assessment of clinical outcomes are shown in Table 2. In general, the observed increase in influenza vaccination did not result in substantial differences in clinical outcomes. Hospitalization for pneumonia or influenza occurred in 536 of 87 059 participants (0.6%) in the usual care group and 1297 of 212 822 of those receiving any intervention letter (0.6%) (HR, 0.99; 95% CI, 0.89-1.09). The incidence of cardiorespiratory hospitalization was 1918 of 87 059 in the usual care group (2.2%) compared with 4682 of 212 822 among those receiving any intervention letter (2.2%) (HR, 1.00; 95% CI, 0.95-1.05). Death from any cause occurred in 631 of 87 059 participants (0.7%) in the usual care group and 1557 of 212 822 participants (0.7%) in the pooled intervention letter group (HR, 1.01; 95% CI, 0.92-1.11).
Table 2. Prespecified Exploratory Clinical Outcomes for Comparison of Any Intervention Letter vs Usual Care.
| Clinical outcomea | Events, No. (%) | Absolute difference, % (95% CI) | Hazard ratio or rate ratio (95% CI)b | |
|---|---|---|---|---|
| Any intervention letter (n = 212 822) | Usual care (n = 87 059) | |||
| Lab-confirmed influenza | 1252 (0.6) | 496 (0.6) | 0.02 (−0.04 to 0.08) | 1.03 (0.93 to 1.15) |
| Hospitalization for pneumonia or influenza | 1297 (0.6) | 536 (0.6) | −0.01 (−0.07 to 0.06) | 0.99 (0.89 to 1.09) |
| Respiratory hospitalization | 2514 (1.2) | 1028 (1.2) | 0.00 (−0.08 to 0.09) | 1.00 (0.93 to 1.07) |
| CV hospitalization | 2270 (1.1) | 930 (1.1) | 0.00 (−0.08 to 0.08) | 1.00 (0.93 to 1.08) |
| Cardiorespiratory hospitalization | 4682 (2.2) | 1918 (2.2) | 0.00 (−0.12 to 0.12) | 1.00 (0.95 to 1.05) |
| All-cause hospitalization | 26 692 (12.5) | 10 732 (12.3) | 0.21 (−0.05 to 0.47) | 1.02 (0.99 to 1.04) |
| All-cause mortality | 1557 (0.7) | 631 (0.7) | 0.01 (−0.06 to 0.07) | 1.01 (0.92 to 1.11) |
| Incident HF, HF hospitalization, or CV death | 713 (0.3) | 352 (0.4) | −0.07 (−0.12 to −0.02) | 0.83 (0.73 to 0.94) |
| Myocardial infarction, stroke, or CV death | 787 (0.4) | 353 (0.4) | −0.04 (−0.09 to 0.01) | 0.91 (0.81 to 1.04) |
| Myocardial infarction, stroke, coronary revascularization, or CV death | 1040 (0.5) | 459 (0.5) | −0.04 (−0.10 to 0.02) | 0.93 (0.83 to 1.04) |
| Incident HF or HF hospitalization | 459 (0.2) | 229 (0.3) | −0.05 (−0.09 to −0.01) | 0.82 (0.70 to 0.95) |
| CV death | 278 (0.1) | 130 (0.2) | −0.02 (−0.05 to 0.01) | 0.87 (0.71 to 1.08) |
| Myocardial infarction | 273 (0.1) | 118 (0.1) | −0.01 (−0.04 to 0.02) | 0.95 (0.77 to 1.19) |
| Coronary revascularization | 453 (0.2) | 200 (0.2) | −0.02 (−0.01 to 0.02) | 0.93 (0.78 to 1.09) |
| Stroke | 247 (0.1) | 117 (0.1) | −0.02 (−0.05 to 0.01) | 0.86 (0.69 to 1.07) |
| Incident AF or AF hospitalization | 1058 (0.5) | 400 (0.5) | 0.04 (−0.02 to 0.09) | 1.08 (0.96 to 1.21) |
| No. of general practitioner contacts, median (IQR)c | 3 (1-6) | 3 (1-6) | NA | 1.00 (0.99 to 1.01) |
Abbreviations: AF, atrial fibrillation; CV, cardiovascular; HF, heart failure; NA, not applicable.
The prespecified registry-based end point definitions are listed in eTable 6 in Supplement 2. No procedure has been applied for controlling the type-1 error rate for these exploratory outcomes, and therefore, no inferences should be drawn from the 95% CIs.
Effect estimates are reported as hazard ratios for all outcomes besides the number of general practitioner contacts, for which the effect is reported as a rate ratio. Hazard ratios were generated from Cox proportional hazards models, whereas the rate ratio was calculated using negative binomial regression.
Vaccination visits were excluded from the number of general practitioner contacts.
Discussion
In this randomized clinical implementation trial, we found that behaviorally informed electronic letters markedly increased influenza vaccination rates in a broad at-risk population of young and middle-aged Danish patients eligible for free-of-charge influenza vaccination due to chronic diseases. All letter types were successful with absolute effect sizes ranging from 11 to 14 percentage points. The largest effect sizes were observed with a repeated letter strategy, where a standard informational letter was delivered at baseline and again 10 days later, and a letter incorporating a behavioral nudge highlighting potential beneficial effects of vaccination on CV health. The letters were found to be effective in increasing influenza vaccination across all major subgroups compared with usual care. In general, these large increases in influenza vaccination uptake did not result in any differences in clinical outcomes, although most effect estimates favored the interventions, especially for CV events.
The effect sizes observed in this trial exceed those observed in prior comparable trials. In a trial reported by Yokum et al,12 physically delivered intervention letters using similar behavioral science concepts in US Medicare beneficiaries increased influenza vaccination by just under 1 percentage point with a similar control group vaccination rate of 25.9%. Milkman et al13,24 found average effect sizes of 2 percentage points with short text message nudges in 2 separate trials, one with a relatively low usual care vaccination rate of 29.4%13 and another with a relatively low rate of 42.0%.24 Several factors may have led to the increased effect sizes in our trial. First, our letter-based format is substantially longer than the text messages tested by Milkman et al13,24; this may have allowed our interventions to address the recipients’ needs more accurately in terms of information on benefits and risks of vaccination as well as on the practical and logistical aspects of obtaining vaccination. Second, the governmental electronic letter system used for intervention delivery (Digital Post) carries significant authority in Denmark; as citizens are legally obligated to read and react to important governmental messaging delivered through this system, our interventions may have been regarded with increased salience in ways that prior research constructs did not. Our findings may provide a strong incentive for other countries, regions, or local entities to explore implementation of similar systems. Third, patients in this population may not have been aware of their eligibility for free-of-charge vaccination; monetary costs of vaccination were not clarified or specially called out in prior studies.
The magnitude of effect observed in this trial with simple, scalable, and cost-efficient electronic letter strategies may have substantial public health implications. The trial resulted in an additional 25 000 vaccinated Danish citizens across the 6 intervention groups compared with the usual care vaccination rate. Widespread implementation of the most successful strategy, the repeated letter, might be hypothesized to result in an even larger number of additional vaccinees. The overall number of individuals needed to be nudged to achieve 1 additional vaccination of only 9 underlines the cost-efficiency of the interventions, since digital letters can be delivered at almost no cost. These findings potentially have important implications that extend well beyond the unique Danish health system environment. Large-scale integrated health care delivery systems and insurers, some collectively caring for millions of lives, have also developed high-priority health-related communications patient portals. Data from the US National Health Interview Survey indicate that approximately 42% of US adults aged 18 to 64 years have at least 1 chronic condition,25 translating to a total of approximately 84 million people.26 Implementing an influenza vaccination implementation strategy such as we report could result in up to 9.3 million additional vaccinees at a number needed to nudge of 9, although this is a crude extrapolation and should be treated as such. In particular, the current fragmented nature of the US health care system with numerous separate health care delivery entities would potentially complicate centralized delivery of such interventions.
In the similarly designed NUDGE-FLU trial,17 a CV gain-framing letter and a repeated letter strategy increased vaccination rates by approximately 1 percentage point among Danish adults aged 65 years or older. These letters were also the most effective in the present trial. Several reasons may exist as to why we observed larger effect sizes in this present trial. First, while the usual care vaccination rate among older adults in NUDGE-FLU was approximately 80%, the usual care group in this trial had a vaccination rate of only 27.9%, and thus, the present results confirm prior beliefs that nudging interventions may be more effective in populations with low uptake of the desired behavior at baseline. Second, usual care in the older population enrolled in NUDGE-FLU included a separate informational letter sent by the Danish Health Authority and delivered through the same letter system, while the usual care group in this trial did not receive any other letters; this may have limited the incremental effectiveness of the nudges in NUDGE-FLU. While participants in the present trial may still have been exposed to other public vaccination campaigns, direct messaging has previously been shown to be one of the single most effective strategies to influence health-related behavior.27 Third, the trials enrolled distinct populations: NUDGE-FLU enrolled older adults aged 65 years or older regardless of concomitant disease, whereas this trial enrolled patients aged 18 to 64 years who qualified for free-of-charge vaccination due to chronic diseases. Younger patients may have been less aware about both their eligibility for free-of-charge vaccination and the potential health benefits of vaccination, thereby increasing “nudgeability.” These items, along with the separate usual care informational letter, are also the most likely explanations for the difference in usual care vaccination uptake between the 2 trials. Finally, the younger population enrolled in the present trial might be more responsive to electronic nudges than the older population evaluated in NUDGE-FLU, partly since older individuals may have stronger more established views on vaccination given greater durations of exposure to usual care vaccination campaigns.
The observed incremental effectiveness of repeated messaging is in line with prior implementation trials including attempts to increase influenza17,24 and COVID-19 vaccination rates.28 While a number of previous trials12,13 have tested different strategies to increase influenza vaccination in large populations, to date, no prior large-scale trial has specifically targeted young and middle-aged patients with chronic diseases. In addition, trials have often been characterized by limited data richness, ie, lack of baseline data beyond sex, age, and region, once again highlighting the suitability of the unique Danish health infrastructure for conducting randomized clinical trials using implementation strategies.15
Despite the large observed increase in influenza vaccination uptake, the interventions generally did not result in any differences in our prespecified exploratory assessment of clinical outcomes. The effect estimates did favor the interventions for most outcomes but did not reach statistical significance except for the 2 heart failure–related outcomes. The lack of observable effects on clinical outcomes may have several potential explanations. First, event rates were relatively low in this young and middle-aged population, limiting statistical power. Second, influenza may only cause a relatively small number of excess events in this younger population, limiting the number of preventable events. Third, the 2023 to 2024 influenza season in Denmark was characterized by predominance of influenza A and in particular the H1N1pdm09 strain, which in previous seasons has primarily affected older individuals.29
Limitations
This study has limitations. First, we were unable to register whether the electronic letters were actually opened due to privacy and autonomy concerns after delivery to the participants’ mailboxes; however, as citizens themselves are legally responsible for responding to high-priority communications from governmental authorities regularly received through the same system, very high open rates were expected. Monthly open rates of more than 90% have been reported previously.30 Second, the trial was conducted in Denmark and the results may not be generalizable to other countries or regions with different cultural, societal, and/or political norms. Third, the present results only directly apply to young and middle-aged patients with chronic diseases, although NUDGE-FLU did find consistent effectiveness of similar letters across patient subpopulations in the age group older than 65 years.10,11,17,31 Fourth, the trial relied solely on routinely collected health data captured in administrative registries as data source, which may introduce misclassification; however, this would not be expected to be differentially distributed across randomized groups.
Conclusions
In this study, behaviorally informed electronically delivered letter-based nudges markedly increased influenza vaccination compared with usual care in an at-risk population of patients aged 18 to 64 years with chronic diseases. The results suggest that simple, scalable, and cost-efficient electronic letter strategies may be utilized to encourage positive health behavior at a population level.
Protocol and Statistical Analysis Plan
eAppendix. NUDGE-FLU-CHRONIC Study Group
eFigure 1. Standard Letter Template
eFigure 2. Study Timeline
eFigure 3. Danish Governmental Electronic Letter System
eTable 1. Prespecified Definitions for Disease-Specific Eligibility Criteria
eTable 2. Description of Study Groups
eTable 3. Prespecified Definitions for Baseline Conditions
eTable 4. Prespecified Definitions for Medication Use at Baseline
eTable 5. Prespecified Definitions for Prior Vaccinations at Baseline
eTable 6. Prespecified Endpoint Definitions
eTable 7. Baseline Characteristics of Trial Population Compared With Those Excluded due to Exemption From Governmental Electronic Letter System
eTable 8. Sensitivity Analysis of Primary Endpoint Accounting for Household-Level Clustering
Data Sharing Statement
References
- 1.World Health Organization. Influenza (seasonal). October 3, 2023. Accessed January 19, 2024. https://www.who.int/news-room/fact-sheets/detail/influenza-(seasonal)
- 2.Gross PA, Hermogenes AW, Sacks HS, Lau J, Levandowski RA. The efficacy of influenza vaccine in elderly persons. a meta-analysis and review of the literature. Ann Intern Med. 1995;123(7):518-527. doi: 10.7326/0003-4819-123-7-199510010-00008 [DOI] [PubMed] [Google Scholar]
- 3.Rondy M, El Omeiri N, Thompson MG, Levêque A, Moren A, Sullivan SG. Effectiveness of influenza vaccines in preventing severe influenza illness among adults: a systematic review and meta-analysis of test-negative design case-control studies. J Infect. 2017;75(5):381-394. doi: 10.1016/j.jinf.2017.09.010 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.McLean HQ, Petrie JG, Hanson KE, et al. Interim estimates of 2022-23 seasonal influenza vaccine effectiveness - Wisconsin, October 2022-February 2023. MMWR Morb Mortal Wkly Rep. 2023;72(8):201-205.. doi: 10.15585/mmwr.mm7208a1 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Fröbert O, Götberg M, Erlinge D, et al. Influenza vaccination after myocardial infarction: a randomized, double-blind, placebo-controlled, multicenter trial. Circulation. 2021;144(18):1476-1484. doi: 10.1161/CIRCULATIONAHA.121.057042 [DOI] [PubMed] [Google Scholar]
- 6.Behrouzi B, Bhatt DL, Cannon CP, et al. Association of influenza vaccination with cardiovascular risk: a meta-analysis. JAMA Netw Open. 2022;5(4):e228873. doi: 10.1001/jamanetworkopen.2022.8873 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Modin D, Jørgensen ME, Gislason G, et al. Influenza vaccine in heart failure. Circulation. 2019;139(5):575-586. doi: 10.1161/CIRCULATIONAHA.118.036788 [DOI] [PubMed] [Google Scholar]
- 8.Modin D, Lassen MCH, Claggett B, et al. Influenza vaccination and cardiovascular events in patients with ischaemic heart disease and heart failure: a meta-analysis. Eur J Heart Fail. 2023;25(9):1685-1692. doi: 10.1002/ejhf.2945 [DOI] [PubMed] [Google Scholar]
- 9.Centers for Disease Control and Prevention . FluVaxView: Flu Vaccination Coverage, United States, 2022–23 Influenza Season. September 6, 2024. Accessed September 25, 2024. https://www.cdc.gov/fluvaxview/coverage-by-season/2022-2023.html?CDC_AAref_Val=https://www.cdc.gov/flu/fluvaxview/coverage-2223estimates.htm
- 10.Johansen ND, Vaduganathan M, Bhatt AS, et al. Electronic nudges to increase influenza vaccination uptake among patients with heart failure: a pre-specified analysis of the NUDGE-FLU trial. Eur J Heart Fail. 2023;25(8):1450-1458. doi: 10.1002/ejhf.2913 [DOI] [PubMed] [Google Scholar]
- 11.Lassen MCH, Johansen ND, Vaduganathan M, et al. Electronically delivered nudges to increase influenza vaccination uptake in older adults with diabetes: a secondary analysis of the NUDGE-FLU Trial. JAMA Netw Open. 2023;6(12):e2347630. doi: 10.1001/jamanetworkopen.2023.47630 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Yokum D, Lauffenburger JC, Ghazinouri R, Choudhry NK. Letters designed with behavioural science increase influenza vaccination in Medicare beneficiaries. Nat Hum Behav. 2018;2(10):743-749. doi: 10.1038/s41562-018-0432-2 [DOI] [PubMed] [Google Scholar]
- 13.Milkman KL, Gandhi L, Patel MS, et al. A 680,000-person megastudy of nudges to encourage vaccination in pharmacies. Proc Natl Acad Sci U S A. 2022;119(6):e2115126119. doi: 10.1073/pnas.2115126119 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Szilagyi PG, Albertin C, Casillas A, et al. Effect of patient portal reminders sent by a health care system on influenza vaccination rates: a randomized clinical trial. JAMA Intern Med. 2020;180(7):962-970. doi: 10.1001/jamainternmed.2020.1602 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Johansen ND, Vaduganathan M, Bhatt AS, Biering-Sørensen T. Nudging a Nation - The Danish NUDGE Trial Concept. NEJM Evid. 2024;3(1):w2300024. Published online December 26, 2023. doi: 10.1056/EVIDctw2300024 [DOI] [PubMed] [Google Scholar]
- 16.Agency for Digital Government. Digital Post. Accessed January 18, 2023. https://en.digst.dk/systems/digital-post/
- 17.Johansen ND, Vaduganathan M, Bhatt AS, et al. Electronic nudges to increase influenza vaccination uptake in Denmark: a nationwide, pragmatic, registry-based, randomised implementation trial. Lancet. 2023;401(10382):1103-1114. doi: 10.1016/S0140-6736(23)00349-5 [DOI] [PubMed] [Google Scholar]
- 18.Johansen ND, Vaduganathan M, Bhatt AS, et al. Rationale and design of NUDGE-FLU-CHRONIC and NUDGE-FLU-2: two nationwide randomized trials of electronic nudges to increase influenza vaccination among patients with chronic diseases and older adults during the 2023/2024 influenza season. Am Heart J. 2024;272:23-36. doi: 10.1016/j.ahj.2024.03.003 [DOI] [PubMed] [Google Scholar]
- 19.Danish Health Authority. Autumn vaccinations against covid-19 and influenza are to help us through the winter. September 13, 2022. Accessed November 27, 2023. https://www.sst.dk/en/english/news/2022/autumn-vaccinations-against-covid-19-and-influenza-are-to-help-us-through-the-winter
- 20.Dunnett CW. New tables for multiple comparisons with a control. Biometrics. 1964;20(3):482-491. doi: 10.2307/2528490 [DOI] [Google Scholar]
- 21.Milkman KL, Beshears J, Choi JJ, Laibson D, Madrian BC. Using implementation intentions prompts to enhance influenza vaccination rates. Proc Natl Acad Sci U S A. 2011;108(26):10415-10420. doi: 10.1073/pnas.1103170108 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.Keller PA, Harlam B, Loewenstein G, Volpp KG. Enhanced active choice: a new method to motivate behavior change. J Consum Psychol. 2011;21(4):376-383. doi: 10.1016/j.jcps.2011.06.003 [DOI] [Google Scholar]
- 23.Grove Krause T, Jakobsen S, Haarh M, Mølbak K. The Danish vaccination register. Euro Surveill. 2012;17(17):20155. doi: 10.2807/ese.17.17.20155-en [DOI] [PubMed] [Google Scholar]
- 24.Milkman KL, Patel MS, Gandhi L, et al. A megastudy of text-based nudges encouraging patients to get vaccinated at an upcoming doctor’s appointment. Proc Natl Acad Sci U S A. 2021;118(20):e2101165118. doi: 10.1073/pnas.2101165118 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 25.Boersma P, Black LI, Ward BW. Prevalence of multiple chronic conditions among US Adults, 2018. Prev Chronic Dis. 2020;17:E106. Published online September 17, 2020. doi: 10.5888/pcd17.200130 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 26.America’s Health Rankings, United Health Foundation. Population - Adults Ages 18-64 in the United States. Accessed March 17, 2024. https://www.americashealthrankings.org/explore/measures/pct_18-64
- 27.Head KJ, Noar SM, Iannarino NT, Grant Harrington N. Efficacy of text messaging-based interventions for health promotion: a meta-analysis. Soc Sci Med. 2013;97:41-48. doi: 10.1016/j.socscimed.2013.08.003 [DOI] [PubMed] [Google Scholar]
- 28.Dai H, Saccardo S, Han MA, et al. Behavioural nudges increase COVID-19 vaccinations. Nature. 2021;597(7876):404-409. doi: 10.1038/s41586-021-03843-2 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 29.Statens Serum Institut . Influenza 2022/2023. September 19, 2023. Accessed June 26, 2024. https://en.ssi.dk/surveillance-and-preparedness/surveillance-in-denmark/annual-reports-on-disease-incidence/influenza-2022-2023
- 30.Tamer M. The vast majority of people open their e-Boks every month. So why will we soon have a new digital mailbox? Politiken. December 10, 2021. Accessed January 15, 2023. https://politiken.dk/forbrugogliv/art8516920/Langt-de-fleste-%C3%A5bner-deres-e-Boks-hver-m%C3%A5ned.-S%C3%A5-hvorfor-f%C3%A5r-vi-nu-snart-en-ny-digital-postkasse
- 31.Modin D, Johansen ND, Vaduganathan M, et al. Effect of electronic nudges on influenza vaccination rate in older adults with cardiovascular disease: prespecified analysis of the NUDGE-FLU Trial. Circulation. 2023;147(18):1345-1354. doi: 10.1161/CIRCULATIONAHA.123.064270 [DOI] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
Protocol and Statistical Analysis Plan
eAppendix. NUDGE-FLU-CHRONIC Study Group
eFigure 1. Standard Letter Template
eFigure 2. Study Timeline
eFigure 3. Danish Governmental Electronic Letter System
eTable 1. Prespecified Definitions for Disease-Specific Eligibility Criteria
eTable 2. Description of Study Groups
eTable 3. Prespecified Definitions for Baseline Conditions
eTable 4. Prespecified Definitions for Medication Use at Baseline
eTable 5. Prespecified Definitions for Prior Vaccinations at Baseline
eTable 6. Prespecified Endpoint Definitions
eTable 7. Baseline Characteristics of Trial Population Compared With Those Excluded due to Exemption From Governmental Electronic Letter System
eTable 8. Sensitivity Analysis of Primary Endpoint Accounting for Household-Level Clustering
Data Sharing Statement