Behavioral economics implementation: Regret lottery improves mHealth patient study adherence

Abstract

Background

Nonadherence to study protocols reduces the generalizability, validity, and statistical power of longitudinal studies.

Purpose

To determine whether an automated electronically-delivered regret lottery would improve adherence to an intensive mHealth self-monitoring protocol as part of a longitudinal observational study.

Methods

We enrolled 77 adults into a 52-week study requiring five daily ecologic momentary assessments (EMA) of stress and daily accelerometer use. We performed a pre/post single-arm study to evaluate the efficacy of a lottery intervention in improving adherence to this protocol. Midway through the study, participants were invited to enter a weekly regret lottery ($50 prize, expected value <$1) in which prize collection was contingent upon meeting adherence thresholds for the prior week. Study protocol adherence before and after lottery initiation were compared using mixed models repeated measures analysis of variance.

Results

62 participants consented to lottery participation. In the 12 weeks prior to lottery initiation, weekly adherence was declining (slope −1.4%/week). The weekly per-participant probability of adherence was higher after lottery initiation when comparing the 4-week (32% pre-lottery vs 50% post-lottery, p < 0.001), 8-week (37% vs 49%, p < 0.001), and 12-week periods (39% vs 45%, p = 0.001) before and after lottery initiation. However, the rate of decline in adherence over time was unchanged.

Conclusion

The implementation of an automated, electronically-delivered weekly regret lottery improved adherence with an intensive self-monitoring study protocol. Regret lotteries may represent a cost-effective tool to improve adherence and reduce bias caused by dropout or nonadherence.

1. Introduction

Human subjects research often relies on sustained participant engagement for success. Attrition of study participants or lack of adherence with study protocols can reduce statistical power, introduce bias, and reduce generalizability [1]. Usual strategies to encourage participation for the duration of a study through predetermined fixed payments or regular communication from the research team carry financial and labor costs and may promote participant fatigue. Investigators have therefore been interested in using insight from behavioral economics (the study and manipulation of cognitive processes that influence valuation and subsequent decision making) to design cost-effective methods for improving study adherence. One such method is the use of money- or prize-based lotteries for which entry and/or prize receipt is contingent upon participants’ achievement of protocol adherence goals. In addition to providing an incentive to study adherence, “regret lotteries” – in which nonadherent participants learn that they are unable to collect a prize for which they have otherwise been selected – take advantage of innate aversion to loss and regret in order to motivate behavior change [2].

Prior studies that have assessed the effect of lotteries on adherence to recommended health behaviors have shown mixed success. While some investigations have shown lottery-based incentives to be effective for promoting weight loss [3] and medication adherence [4,5], others have shown no significant effect on a range of brief (venereal disease testing, breast cancer screening) and sustained (medication adherence, physical activity) health behaviors [[6], [7], [8], [9], [10], [11], [12]]. Few studies have assessed the impact of lotteries on adherence with a study protocol, for which the desired behavior carries no meaningful intrinsic benefit for the participant. Previous studies using cash/prize drawings to improve participation with one-time survey completion have yielded mixed results [[13], [14], [15], [16]], however past studies have not tested the effectiveness of lotteries to sustain long-term engagement among individuals already enrolled in a study.

We aimed to examine the impact of a lottery-based financial incentive on adherence with smartphone-based ecological momentary assessment (EMA) and wearable fitness tracker-based accelerometer protocols in a 52-week observational study assessing the bidirectional relationship between stress and physical activity. We hypothesized that the implementation of a weekly regret lottery midway through the study period would improve adherence to the self-monitoring measures (EMA completion and accelerometer use).

2. Methods

2.1. Participants

We enrolled intermittently exercising adults into a 52-week study of the association between stress and physical activity [17]. Participants were recruited at Columbia University Medical Center via flyers and word of mouth. Participants were included if they were 18 years or older, had access to a personal computer with an Internet connection, owned an iPhone or Android smartphone, and self-reported intermittently exercising (6–11 times per month). Participants were excluded if they had a medical condition that precluded regular physical activity, self-reported having occupational demands that required rigorous activity (e.g. construction workers) or would make responding to the EMA dangerous (e.g. bus/taxi drivers), were unable to read and speak English, were unable to adhere to the study protocol due to cognitive or psychiatric impairment, or were unavailable for follow-up over 52 weeks. Informed consent was obtained from all participants, and the study was approved by the Institutional Review Board of Columbia University Medical Center.

Access to the study dataset and information about the study's execution and materials is available publicly at https://osf.io/kmszn.

2.2. Measures

The background study required self-monitoring of stress through five daily EMAs, as well as automated monitoring of physical activity via accelerometer.

Physical activity was measured using Fitbit Flex activity trackers. Each device automatically uploaded data records to the Fitbit website whenever it was within 15 feet of the base unit. Participants were instructed to charge and sync their device at least every 5–7 days. Participants were considered adherent to accelerometer use for any given week if they displayed 5 or more days with ≥6 h of wear time.

EMA ratings of stress were assessed in the morning, at the end of the day, and at three random times during each day using the web browser on a smartphone or a personal computer. At each assessment, participants were prompted to respond to 2–3 questions regarding stress, environmental factors, and anticipated (morning assessment) or actual (end-of-day assessment) physical activity. Adherence with EMA prompts was based on the number of prompted assessments for which all questions were answered divided by the total number of prompts for that week (typically 35), and participants were considered to be adherent for any given week if they responded to ≥75% of EMA prompts.

Overall adherence with the background study protocol (i.e. adherence to self-monitoring and accelerometer use) was determined on a weekly basis. Participants were classified as adherent for each week if they were adherent with both EMA (responded to ≥75% EMA prompts that week) and physical activity monitoring (5 or more days with ≥6 h of accelerometer wear time in that week).

2.3. Lottery intervention

In response to decreasing study protocol adherence approximately midway through the overall study period, all participants were invited to partake in a weekly lottery for which prize collection was contingent upon having met pre-specified adherence thresholds during the prior week (Fig. 1). The goal of the lottery was to improve adherence accelerometer use and EMA-based stress self-monitoring. For two weeks prior to the start of the lottery, each participant who consented to lottery participation received an automatically-generated weekly email message listing his or her adherence for the prior week. On every subsequent Tuesday for the remainder of the study, participants continued to receive summaries of their adherence during the prior week. In addition, one participant was randomly chosen as the prize winner. The weekly winner was notified of his or her selection with an automatically-generated email message that also included an adherence summary for the prior week. Winners who met the threshold for prize eligibility were notified that they would receive the prize (weekly adherence summary followed by “Congratulations! Your name was drawn as the winner of the extra $50 compensation this week!”), whereas winners with insufficient adherence received a loss-framed notification that their prize would not be awarded for this reason (weekly adherence summary followed by, “Oh no! Your name was drawn as the winner of the extra $50 compensation this week; however, your adherence score(s) were too low to receive the additional compensation”). Participants who were not selected to be winners of the weekly lottery were notified of this, along with their adherence data for the prior week. Of note, participants who did not consent to lottery participation did not receive weekly compliance scores.

Fig. 1.

Schematic of lottery design.

The weekly prize was set at $50 for all drawings. The expected value (i.e. the predicted value based on the prize size multiplied by the probability of being selected) for the first drawing was $0.74, although the expected value for each drawing was occasionally nominally higher during subsequent weeks due to participant holidays or participant study completion, leaving fewer remaining participants in the lottery pool. However, participants were not aware of the odds of winning or expected value for any of the drawings.

2.4. Statistical analysis

All participants who consented to lottery participation were included in the primary analysis. Descriptive statistics were used to describe baseline participant characteristics. The primary outcome was the per-participant probability of weekly adherence (i.e. likelihood of each participant achieving the EMA completion and accelerometer wear time thresholds) in the 4 weeks prior to lottery initiation (excluding the 2-week training period after which the lottery was announced and weekly feedback was provided, but no winners were selected) compared to the 4 weeks after lottery initiation, and we hypothesized that the latter would be higher than the former. A mixed-models repeated measures analysis of variance was performed with adherence (defined at meeting thresholds of EMA completion and accelerometer wear time) as the dependent variable to compare average within-subject weekly study protocol adherence in these time periods.

As sensitivity analyses, we also compared adherence during the 8- and 12-week periods before and after lottery initiation (Fig. 1). We also compared the change in per-participant probability of weekly adherence over time for the 12 weeks before and after lottery initiation. We hypothesized that the slope of the change in participation over time before lottery initiation would be lower (i.e. more negative, assuming declining probability of weekly adherence over time) than the slope after lottery initiation.

Given that not all participants consented to lottery participation, we then performed a sensitivity analysis by repeating the above analyses including all participants (i.e. both lottery participants and non-participants) in order to determine whether there was a significant group-level improvement in adherence.

A p < 0.05 was used as the threshold for statistical significance. Model estimation and statistical testing were performed using SAS 9.4 (SAS Institute Inc., Cary, NC).

3. Results

A total of 77 participants were enrolled in the initial study. Participants had mean age of 32.1 years, 40.3% were White, and 58.4% were female (Table 1). 92.2% had full- or part-time employment. 62 participants (80.5%) agreed to participate in the regret lottery intervention, and there were significant differences in race, partner status, and education level, but not employment status, age, or sex, between participants who did and did not agree to lottery participation.

Table 1.

Baseline characteristics of study participants.

	All Participants (n = 77)	Lottery Participants (n = 62)	Lottery Non-participants (n = 15)
Age, years, mean (s.d.)	32.1 (9.5)	32.0 (9.6)	32.7 (9.5)
Female	45 (58.4)	38 (61.3)	7 (46.7)
Hispanic Ethnicity	21 (27.3)	18 (29.0)	3 (20.0)
Race*
White	31 (40.3)	26 (41.9)	5 (33.3)
Black/African American	11 (14.3)	8 (12.9)	3 (20.0)
Asian	15 (19.5)	13 (21.0)	2 (13.3)
Other/Multiple/Not Reported	20 (26.0)	15 (24.2)	5 (33.3)
Partner Status*
Single	43 (55.8)	37 (59.7)	6 (40.0)
Partner/Spouse	32 (41.6)	23 (37.1)	9 (60.0)
Divorced	2 (2.6)	2 (3.23)	0 (0.0)
Education Level*
High School Diploma/GED	1 (1.3)	1 (1.6)	0 (0.0)
Some College	12 (15.6)	11 (17.7)	1 (6.7)
College Graduate	32 (41.6)	23 (37.1)	9 (60.0)
Graduate/Professional School	32 (41.6)	27 (43.6)	5 (33.3)
Employment
Not working, personal choice	3 (3.9)	3 (4.8)	0 (0.0)
Not working, seeking employment	3 (3.9)	3 (4.8)	0 (0.0)
Working full time	63 (81.8)	49 (79.0)	14 (93.3)
Working part-time	8 (10.4)	7 (11.3)	1 (6.7)

All data are N (%) unless otherwise specified.

* = Fisher's exact test p-value <0.05.

Throughout the entire duration of the lottery intervention, a total of 38 weekly winners were selected (representing 25 unique study participants). Of these winners, 24 (63.2%) were ineligible to collect the prize due to not meeting the adherence threshold.

Among lottery participants, the proportion participants reaching the weekly adherence thresholds declined from 48% to 31% over the 12 weeks before the lottery. In the week after lottery initiation, adherence increased to 50%, but then declined to 43% over the subsequent 12 weeks (Fig. 2). The weekly probability of participant adherence to the study protocol was significantly higher in the 4-week, 8-week, and 12-week periods after lottery initiation compared to the comparable periods before lottery initiation (Table 2 and Fig. 3). The absolute increase in probability of weekly adherence was 18% when comparing the 4 weeks before and after the lottery started (p < 0.001), 12% when comparing the 8 weeks before and after the lottery (p < 0.001), and 6% when comparing the 12 weeks before and after the lottery (p = 0.001).

Fig. 2.

Protocol adherence by week among lottery participants (n = 62).

Table 2.

Effect of lottery initiation on likelihood of weekly adherence among lottery participants (n = 62).

Time Interval Before/After Lottery	Likelihood of Weekly Adherence Before Lotterya	Likelihood of Weekly Adherence with Lotterya	Absolute Difference in Likelihood of Weekly Adherence	p
4 weeks	32%	50%	18%	<0.001
8 weeks	37%	49%	12%	<0.001
12 weeks	39%	45%	6%	0.001

^aPercent represents the likelihood of protocol adherence per participant per week throughout the entire period specified, as estimated from the repeated measures model.

Fig. 3.

Weekly probability of adherence before and after lottery initiation.

The per-participant probability of adherence significantly decreased over time in the 12 weeks prior to the start of the lottery (slope = −1.4%/week, p = 0.001). The corresponding predicted probability of adherence in the first week after lottery initiation was 31% based on pre-lottery data, but the actual adherence rate was 50%. Per-participant probability of adherence subsequently decreased over time after initiation of the lottery. There was no significant difference in the rate of decline in weekly probability of adherence over time in the 12-week periods before versus after lottery initiation (p = 0.94).

We next performed a sensitivity analysis by analyzing changes in protocol adherence among all study participants together (i.e. lottery participants and non-participants). Over the 12 weeks prior to lottery initiation, the proportion of all study participants reaching the weekly adherence thresholds declined from 47% to 25%. Similar to the trend seen among lottery participants, adherence among all participants increased to 52% in the week after lottery initiation, but then declined to 29% over the following 12 weeks (Supplemental Fig. 1). The weekly probability of participant adherence to the study protocol was again significantly higher in the 4-week (16% higher in the 4 weeks after compared to before the lottery started), 8-week (11% higher), and 12-week (5% higher) periods after lottery initiation (Supplementary Table 1). Again, there was no significant difference in the rate of decline in weekly probability of adherence over time in the 12-week periods before versus after lottery initiation (−1.6%/week versus −1.8%/week, p = 0.68).

4. Discussion

We found that a weekly regret lottery improved study protocol adherence when implemented approximately midway through a 52-week intensive self-monitoring study about stress and physical activity. Remarkably, this effect was seen despite a modest prize value ($50) and low expected value (<$1). These findings suggest that regret lotteries may be useful as a low-cost strategy to improve participant adherence to study protocols that involve frequent or daily tasks.

Our findings add important new context to the literature on the use of lotteries to incentivize study participants. Previous studies on the effectiveness of lotteries to improve study adherence have largely focused on increasing response rates to one-time surveys [[13], [14], [15],18,19]. Our use case differed in that the goal was to increase adherence to a long-term, intensive study protocol requiring daily participation over 52-weeks, and thus suggests an expanded role of lotteries in human subjects research. These findings are consistent with other studies showing that lotteries can increase adherence with specific health behaviors [[3], [4], [5], [6], [7], [8], [9], [10], [11], [12],14].

Notably, participants in our study did not stand to receive any personal benefit from protocol adherence, unlike participants in studies aimed at improving weight loss, physical activity, medication adherence, or screening, in which the incentivized behavior is itself beneficial. While previous meta-analyses did not find target behavior to be an effect modifier on the efficacy of financial incentives to change patient behavior, all of the included behaviors (smoking cessation, diet/activity, vaccination, screening) were health-related [20,21]. We hypothesize that the effect of the lottery in our study is greater than it may have been if we were incentivizing a healthy behavior, since the financial incentive here was potentially the strongest factor motivating adherence (as opposed to augmenting an intrinsic motivation towards improved health outcomes). However, given the paucity of evidence regarding incentivization of behaviors such as those in our study, dedicated investigations are needed to compare lottery efficacy in behaviors that do or do not directly benefit participants.

Given the challenge of nonadherence that can bias results in EMA studies [22], prior investigators have examined fixed incentives to improve adherence with mixed results. One study found that a fixed incentive of $20 per month for the three-month study (not contingent on degree of protocol adherence) resulted in 76% completion of self-reports among all participants [23]. Among smoking cessation studies, investigators using smartphone-delivered EMAs to tailor smoking cessation messages achieved completion of about 70% of EMAs using fixed payments ($40-$120 based on adherence, further details not provided) for participants who completed ≥50% of EMAs [24], whereas incentivizing 2 weeks of EMA completion with gift cards ($25 for 50–74%, $50 for 75–89%, $80 for ≥90% completion) resulted in 83% EMA adherence [25]. However, lower-value fixed incentives for EMA completion to study smoking cessation (fixed payment of $30 for completion of study visits, $0.50 per completed EMA, and additional weekly bonuses for completion of both ≥90% of daytime EMAs as well as 100% of first-morning and bedtime EMAs, for a maximum total of $150 per participant with complete adherence) yielded scheduled EMA adherence of 76% for those quitting smoking, but lower adherence to event-contingent reporting (62%) [26]. Our lottery scheme required lower per-participant costs than these incentive designs, with the advantage of maintaining a predictable overall cost to investigators (whereas the fixed incentives cost more as adherence rises because a more participants will be paid).

The success of the lottery suggests that we were able to leverage the tendency to overestimate the probability of a rare event. While the prize value was $50, the expected value for each week was <$1 and the majority of participants never won the lottery. The weekly adherence report email to participants who were not selected for the lottery prize was intended to take advantage of loss aversion by leading nonadherent participants to consider that they would have been ineligible for prize collection had they won the lottery.

While the positive effect of the lottery on study protocol adherence was clear, adherence still decreased over time after lottery initiation. In fact, the slope of the decline in adherence after lottery initiation was similar to the decline preceding the lottery. This is an important addition to the knowledge regarding the limitations of lotteries. There have been prior suggestions that the effects of lotteries are not sustained after the intervention ends. For example, one study of lottery incentives to improve warfarin adherence demonstrated that, although the incentive was effective, anticoagulation control returned to baseline after the intervention was withdrawn [5]. Similarly, a weight loss intervention using lotteries and deposit contracts found that while participants in the intervention groups displayed greater weight loss at the end of the 4-month intervention, those in the intervention groups gained weight after the cessation of the incentives and the differences in weight compared to baseline were not significantly different from the control group 3 months after the intervention ended [3]. It therefore appears that not only does sustained behavior change require continuous incentivization, but individual incentives lose efficacy over time.

It is possible that the decline in protocol adherence in our study indicates that the lottery was not sufficiently frequent, and the effect of the lottery faded as it failed to induce enough regret or positive reinforcement. Lack of sufficient frequency of winning prizes may have changed participants interpretation of their probability of future wins [27]. Indeed, over the course of the lottery, only 25 unique winners were chosen, representing 32% of the total participants. The weekly adherence report emails were intended to combat this limitation. However, if instead the sustained success of the regret lottery requires increased frequency of selection for the prize (the extreme of which is a fixed interval schedule) or gradually increasing prize value, the increase in cost to study investigators may make lotteries less appealing. Alternatively, it is possible that the participants became tolerant to the effects of the lottery over time. If true, this would suggest that the ideal method of promoting adherence may involve rotating different types of incentives over time or increasing prize values. One strategy to increase each participant's frequency of receiving a lottery prize without significantly increasing the cost to investigators may be the combination of a more frequent low-value prize and a less frequent high-value prize. Such a strategy has been used in prior studies3,5,10 and its superiority is supported by previous research: Findings from a comparison of different lottery designs to incentivize physical activity in 209 adults with body mass index ≥27 kg/m² demonstrated that only a combined lottery that included both high-frequency/low-value and low-frequency/high-value rewards was effective at increasing adherence to the workplace wellness intervention [28]. Interestingly, the investigators found a decrease in adherence over time in the “jackpot” arm (low-frequency/high-reward) such that these participants were initially the most adherence, but by the end of study had lower adherence than even the control group. Further, the effects of even their combined lottery arm—which also had the highest expected value—gradually disappeared after intervention withdrawal. The lottery we present here had a probability of winning and a prize value that were in between each of the groups in that study. Together, these studies suggest that a variable incentive scheme (perhaps fixed plus variable incentives) should be tested in order to lead to a more sustained improvement in adherence. If ineffective, it may suggest that lotteries should primarily be employed in short-term interventions.

Notably, the background study in which our lottery was used is dissimilar to many other longitudinal studies because participants received multiple daily EMA prompts that also functioned as reminders. It is possible that the success of the regret lottery was positively influenced by these frequent reminders. Yet, in another study comparing the effectiveness of daily reminders, a daily lottery, or reminders plus lottery to improve warfarin adherence, both the lottery and lottery-reminder combination groups demonstrated better medication adherence compared to a control group (usual care); however, the combination of the lottery and reminders did not lead to an improvement in the outcome of anticoagulation control compared to reminders alone [4]. Further study is warranted to determine how these factors interact to change the behavior of study participants. Additional studies are also needed to determine whether similar improvements in protocol adherence could be achieved at a lower cost, or whether a larger incentive could yield superior results. For example, one study comparing financial incentives to improve mailed survey participation in Hong Kong found that while lottery entry for responders was effective at improving response rates, cash payments of a much smaller size (HKD $10-$40 versus $1000-$4000) were more cost-effective [13]. Moreover, further studies should assess the ideal verbiage of lottery results messaging in order to maximize adherence-promoting cognitive processes.

Our lottery was also unique in regards to the timing of the lottery compared to that of the parent study. In contrast to the studies above, our lottery intervention was implemented in response to declining study protocol adherence rather than at study initiation. It is possible that the influence of our lottery would have been attenuated if it was implemented at the start of the parent study, when adherence was already relatively high and therefore potentially at a maximum achievable level. Optimal intervention timing should be explored in future studies in order to avoid devoting resources to financial incentives at points where they may be ineffective.

Strengths of our study include its simple lottery design, objectively assessed adherence, and innovative delivery of the lottery using an automated electronic method. Limitations include the use of a pre-post design without a control group, making it possible that providing participants with feedback on their adherence contributed to the increased adherence. A dedicated lottery study with randomized assignment to a control group (with adherence feedback alone) or intervention group (with adherence feedback plus lottery) would help to clarify this issue. Additionally, not all of the eligible participants consented to lottery participation and no data was collected regarding the reasons that these participants declined to consent to lottery participation, raising the possibility of a selection bias contributing to the positive primary outcome. However, we noted the group-level improvement in study protocol adherence even when including those who did not participate in the lottery. Further, our small study size limited our ability to perform subgroup analyses to determine whether certain participant groups displayed different responses to the incentive. Similarly, the small number of lottery drawings during the period analyzed left us unable to study the behavior of participants immediately after winning the lottery.

5. Conclusion

In conclusion, implementation of a lottery-based financial incentive was effective at improving accelerometer and EMA completion adherence among participants in a study about the relationship between stress and physical activity. Regret lotteries should be considered as low-cost interventions to improve study protocol adherence.

Disclosures

The authors declare that they have no financial conflicts of interest to report.

Appendix A. Supplementary data

The following are the Supplementary data to this article: