Abstract
Background: To evaluate the effectiveness of a contingent financial incentive (£10 note in addition to a routinely provided £10 voucher) versus no contingent financial incentive, on improving the retention rate in a randomised controlled trial (RCT).
Methods: A two arm ‘Study within a Trial’ (SWAT) embedded within a host RCT (SCIMITAR+). Participants were randomised to the SWAT using a 2:1 (intervention:control) allocation ratio. The primary outcome measure was the proportion of participants completing a CO breath measurement at the first SCIMITAR+ follow up time point (6 months). Secondary outcomes were withdrawing from follow-up after contact and time from assessment due date to completion. Analyses were conducted using logistic or Cox Proportional Hazards regression as appropriate.
Results: A total of 434 participants were randomised into this SWAT. Completion of the CO breath measurement at 6 months was 88.5% (n=247) in the intervention arm of the SWAT and 85.4% (n=123) in the control arm. The difference (3.1%) was not statistically significant (p=0.36; OR 1.29, 95% CI 0.71-2.33, p=0.41). There was also no evidence of a difference in the proportion of participants withdrawing from follow-up after contact (intervention n=7 (2.5%), control n=5 (3.5%); OR 0.76, 95% CI 0.23-2.44, p=0.64), nor in terms of proximity of 6-month visit completion to due date (HR 1.07, 95% CI 0.86-1.33, p=0.55).
Conclusion: Contingent financial incentives did not statistically significantly increase rates of face-to-face follow-up completion within the SCIMITAR+ trial population. However, the sample size of this SWAT was constrained by the size of the host trial and power was limited. This SWAT adds to the body of evidence for initiatives to increase response rates in trials.
Keywords: SWAT, retention, randomized controlled trial
Introduction
Attrition is a major problem for randomised controlled trials (RCTs) with 25% experiencing more than 10% attrition 1.
Bower et al. (2014) 2 identified financial incentives as an effective retention strategy (RR 1.18; 95% CI 1.09 to 1.28), and effectiveness was increased if this incentive was provided on receipt of a completed questionnaire (RR 1.25; 95% CI 1.14 to 1.38). Bailey et al. (2013) 3 identified that varying the incentive level (£20 compared to £10) increased response to postal questionnaires by up to 10%.
This SWAT evaluated the effectiveness of a contingent financial reward - £10 cash in addition to a routinely provided £10 voucher - versus no contingent financial reward, on improving the retention rate in the SCIMITAR+ trial.
Methods
Design
This SWAT was embedded within the SCIMITAR+ RCT which evaluated the effectiveness of a bespoke, individually-tailored, smoking cessation programme, compared to usual care, for adult smokers with severe mental ill health conditions 4. The SCIMITAR+ Trial was registered prospectively: ISRCTN72955454
This paper refers to the methods and results of the SWAT only.
Participants
The SWAT 5 was conducted in 21 NHS Trusts and 16 primary care settings and was implemented after the start of SCIMITAR+ follow-up. Participants were eligible for this SWAT if they reached the SCIMITAR+ 6-month follow-up on or after 31 st September 2016.
Intervention
When participants in the SWAT intervention group were contacted by the research team to arrange their follow-up appointment, they were advised of the potential of receiving £10 cash contingent on providing a carbon monoxide (CO) breath measure as part of their 6-month face-to-face study appointment, in addition to the £10 gift voucher routinely provided to all participants. Participants in both groups received all other pre-planned retention strategies within SCIMITAR+.
Outcomes
The primary outcome for the SWAT was the proportion of participants completing a CO breath measurement at the SCIMITAR+ 6 month follow-up time-point. Secondary outcome measures were: i) the proximity of visit completion to visit due date; ii) the proportion of participants withdrawing from follow-up in the two months after initial contact was made to arrange the 6-month visit.
Sample size
The sample size was determined by the number of participants followed-up at 6 months in SCIMITAR+ from the point at which this SWAT was embedded.
Randomisation
Simple randomisation using random numbers was carried out by an independent statistician at the York Trials Unit using Stata v13 6. Participants were allocated with a 2:1 allocation ratio (intervention:control) due to the anticipated effectiveness of financial incentives increasing questionnaire response rates.
Blinding
It was not possible to blind research staff to the participant’s allocation. Participants were not informed about the SWAT so were blind to the study hypothesis.
Approvals
The SWAT was approved by the Research Ethics Committee Yorkshire and Humber – Leeds East (15/YH/0051). As the SWAT was deemed to be low risk, and to avoid disappointment for participants who did not receive the additional incentive, informed consent was not obtained for participation in this SWAT.
Statistical analysis
Analyses were conducted using Stata v15 7 on an intention to treat basis using two-sided statistical tests at the 5% significance level, adjusting for host trial allocation.
The proportion of participants who provided a 6-month CO breath measure was analysed using logistic regression. The odds ratio (OR), 95% confidence interval (CI) and p-value are presented.
The 6-month appointment due date was 183 days after randomisation. Participants who withdrew a month either side of the 6-month appointment due date were classed as withdrawn. The proportion of participants withdrawing from SCIMITAR+ in the two months after contact were analysed in the same way as the primary outcome.
A Cox Proportional Hazard model compared the proximity of the visit completion to visit due date (time in days). Participants who completed their visit before or on the due date had their time-to-visit set to 0.1.
Results
In total, 434 participants were randomised into this SWAT (n=286, 65.9% intervention group; n=148, 34.1% control group). Eleven participants withdrew from SCIMITAR+ following randomisation but prior to being contacted for their 6-month visit and were excluded from analysis. There were 423 eligible participants (intervention group n=279, 66.0%; control group n=144, 34.0%) ( Figure 1).
Overall, 87.5% (n=370) of participants completed the CO breath measurement at 6 months; there was no statistically significant difference between intervention (88.5%, n=247) and control groups (85.4%, n=123) (3.1% difference, OR 1.29, 95% CI 0.71-2.33, p=0.41). There was no significant difference in withdrawals between trials arms (intervention n=7, 2.8%; control n=5, 3.5%; OR 0.76, 95% CI 0.23-2.44, p=0.64) or proximity of 6-month visit completion to due date (hazard ratio 1.07, 95% CI 0.86-1.33, p=0.55).
Discussion
An additional £10 in cash did not statistically significantly increase the likelihood of participants completing a face-to-face follow-up, the proportion of the participants withdrawing, or have an effect on the proximity of the visit to the due date.
Strengths and limitations
A small positive difference was observed; however, despite the large sample size, the study was underpowered to confidently rule out a small ‘true’ effect. Due to the small effect size (3.1% increase in response) the cost per additional person attending would be in excess of £300.
Due to the sample size of this SWAT, it is most likely generalisable to the larger host trial population of patients with severe mental ill health disorders.
Data was not collected on how study staff followed the guidance on discussing the contingent £10 note to intervention group participants when arranging follow up visits. This may have diluted the effect of the intervention.
Conclusion
Contingent financial incentives did not statistically significantly increase rates of face-to-face follow-up completion in this trial. However, there were sample size and power limitations. Future SWATs are needed to add to the evidence base.
Data availability
Underlying data
Figshare: SCIMITAR+ Trial: A randomised study within a trial (SWAT) of a contingent financial reward to improve trial follow-up - Data Set, https://doi.org/10.6084/m9.figshare.10060202.v2 8.
Reporting guidelines
Figshare: CONSORT checklist for SCIMITAR+ Trial: A randomised study within a trial (SWAT) of a contingent financial reward to improve trial follow-up, https://doi.org/10.6084/m9.figshare.10060202.v2 8.
Data are available under the terms of the Creative Commons Attribution 4.0 International license (CC-BY 4.0).
Acknowledgements
The SCIMITAR+ team acknowledge the support of the PROMETHEUS programme (MRC MR/R013748/1) in developing this manuscript.
Funding Statement
This work was conducted as part of the SCIMITAR+ Trial which was funded by the NIHR Health Technology Assessment Programme (11/136/52). The views expressed are those of the author(s) and not necessarily those of the NIHR of the Department of Health and Social Care.
The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
[version 1; peer review: 1 approved
References
- 1. Hewitt CE, Kumaravel B, Dumville JC, et al. : Assessing the impact of attrition in randomized controlled trials. J Clin Epidemiol. 2010;63(11):1264–70. 10.1016/j.jclinepi.2010.01.010 [DOI] [PubMed] [Google Scholar]
- 2. Bower P, Brueton V, Gamble C, et al. : Interventions to improve recruitment and retention in clinical trials: a survey and workshop to assess current practice and future priorities. Trials. 2014;15: 399. 10.1186/1745-6215-15-399 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3. Bailey JV, Pavlou M, Copas A, et al. : The Sexunzipped trial: optimizing the design of online randomized controlled trials. J Med Internet Res. 2013;15(12):e278. 10.2196/jmir.2668 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4. Peckham E, Arundel C, Bailey D, et al. : Smoking Cessation Intervention for Severe Mental Ill Health Trial (SCIMITAR+): study protocol for a randomised controlled trial. Trials. 2017;18(1): 44. 10.1186/s13063-017-1789-7 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5. SWAT 48 – Effects of a £ 10 note on retention. Reference Source [Google Scholar]
- 6. StataCorp: Stata Statistical Software: Release 13. College Station, TX: StataCorp LP.2013. [Google Scholar]
- 7. StataCorp: Stata Statistical Software: Release 15. College Station, TX: StataCorp LLC.2017. [Google Scholar]
- 8. Arundel C, Coleman E, Fairhurst C, et al. : SCIMITAR+ Trial: A randomised study within a trial (SWAT) of a contingent financial reward to improve trial follow-up - Data Set. figshare.Dataset.2019. 10.6084/m9.figshare.10060202.v2 [DOI] [PMC free article] [PubMed] [Google Scholar]