Skip to main content
PMC home page
F1000Research logoLink to F1000Research
. 2020 Jul 17;8:1937. Originally published 2019 Nov 20. [Version 2] doi: 10.12688/f1000research.21059.2

The effectiveness of a contingent financial incentive to improve trial follow up; a randomised study within a trial (SWAT)

Catherine Arundel 1,a, Elizabeth Coleman 1, Caroline Fairhurst 1, Emily Peckham 1, Della Bailey 1, Simon Gilbody 1
PMCID: PMC7372525  PMID: 32742637

Version Changes

Revised. Amendments from Version 1

  • Manuscript title amended to make clear this manuscript relates to the SWAT and not the SCIMITAR+ Trial in the main which may have been implied by the original wording.

  • Two P values were inadvertently included in the Abstract Results section. The incorrect P value has been removed.

  • Given the limited power of SWATs, reference to statistical significance may be misleading. All references to ‘statistical significance’ have been amended to reflect the lack of clarity around the estimate due to wide confidence intervals.

  • Addition of further justification of SWAT in Introduction to make clear the evidence base on which this SWAT builds.

  • Revision of ‘reward’ to ‘incentive’ throughout the manuscript for consistency.

  • Addition of clarification on randomisation process (Methods – Randomisation; Results), to justify exclusion of some participants post randomisation

  • Addition of detail in relation to proportional hazards assumptions assessment in relation to secondary outcome (proximity to due date).

Abstract

Objectives

To evaluate the effectiveness of a contingent financial incentive (£10 note in addition to a routinely provided £10 voucher) versus no contingent financial incentive, on improving the retention rate in a randomised controlled trial (RCT).

Methods

A two arm ‘Study within a Trial’ (SWAT) embedded within a host RCT (SCIMITAR+). Participants were randomised to the SWAT using a 2:1 (intervention:control) allocation ratio. The primary outcome measure was the proportion of participants completing a CO breath measurement at the first SCIMITAR+ follow up time point (6 months). Secondary outcomes were withdrawing from follow-up after contact and time from assessment due date to completion.  Analyses were conducted using logistic or Cox Proportional Hazards regression as appropriate.

Results

A total of 434 participants were randomised into this SWAT. Completion of the CO breath measurement at 6 months was 88.5% (n=247) in the intervention arm of the SWAT and 85.4% (n=123) in the control arm (Difference 3.1%, OR 1.29, 95% CI 0.71-2.33, p=0.41). There was also no evidence of a difference in the proportion of participants withdrawing from follow-up after contact (intervention n=7 (2.5%), control n=5 (3.5%); OR 0.76, 95% CI 0.23-2.44, p=0.64), nor in terms of proximity of 6-month visit completion to due date (HR 1.07, 95% CI 0.86-1.33, p=0.55).

Conclusion

It is unclear if contingent financial incentives increased rates of face-to-face follow-up completion within the SCIMITAR+ trial population. However, the sample size of this SWAT was constrained by the size of the host trial and power was limited. This SWAT adds to the body of evidence for initiatives to increase response rates in trials.

Keywords: SWAT, retention, randomized controlled trial

Introduction

Attrition is a major problem for randomised controlled trials (RCTs) with 25% experiencing more than 10% attrition 1.

Bower et al. (2014) 2 identified financial incentives as an effective retention strategy (RR 1.18; 95% CI 1.09 to 1.28), and effectiveness was increased if this incentive was provided on receipt of a completed questionnaire (RR 1.25; 95% CI 1.14 to 1.38). Bailey et al. (2013) 3 identified that varying the incentive level (£20 compared to £10) increased response to postal questionnaires by up to 10%.

Contingent incentives have shown to be effective in increasing questionnaire response rate, but have not been tested in relation to face-to face-visits. This SWAT evaluated the effectiveness of a contingent financial incentive - £10 cash in addition to a routinely provided £10 voucher - versus no contingent financial incentive, on improving the retention rate in the SCIMITAR+ trial.

Methods

Design

This SWAT was embedded within the SCIMITAR+ RCT which evaluated the effectiveness of a bespoke, individually-tailored, smoking cessation programme, compared to usual care, for adult smokers with severe mental ill health conditions 4. The SCIMITAR+ Trial was registered prospectively: ISRCTN72955454

This paper refers to the methods and results of the SWAT only.

Participants

The SWAT 5 was conducted in 21 NHS Trusts and 16 primary care settings and was implemented after the start of SCIMITAR+ follow-up. Participants were eligible for this SWAT if they reached the SCIMITAR+ 6-month follow-up on or after 31 st September 2016.

Intervention

When participants in the SWAT intervention group were contacted by the research team to arrange their follow-up appointment, they were advised of the potential of receiving £10 cash contingent on providing a carbon monoxide (CO) breath measure as part of their 6-month face-to-face study appointment, in addition to the £10 gift voucher routinely provided to all participants. Participants in both groups received all other pre-planned retention strategies within SCIMITAR+.

Outcomes

The primary outcome for the SWAT was the proportion of participants completing a CO breath measurement at the SCIMITAR+ 6 month follow-up time-point. Secondary outcome measures were: i) the proximity of visit completion to visit due date; ii) the proportion of participants withdrawing from follow-up in the two months after initial contact was made to arrange the 6-month visit.

Sample size

The sample size was determined by the number of participants followed-up at 6 months in SCIMITAR+ from the point at which this SWAT was embedded.

Randomisation

Simple randomisation using random numbers was carried out by an independent statistician at the York Trials Unit using Stata v13 6. All potentially participants eligible for inclusion, at the time of randomisation, were allocated with a 2:1 allocation ratio (intervention:control) due to the anticipated effectiveness of financial incentives increasing questionnaire response rates.

Blinding

It was not possible to blind research staff to the participant’s allocation. Participants were not informed about the SWAT so were blind to the study hypothesis.

Approvals

The SWAT was approved by the Research Ethics Committee Yorkshire and Humber – Leeds East (15/YH/0051). As the SWAT was deemed to be low risk, and to avoid disappointment for participants who did not receive the additional incentive, informed consent was not obtained for participation in this SWAT.

Statistical analysis

Analyses were conducted using Stata v15 7 on an intention to treat basis using two-sided statistical tests at the 5% significance level, adjusting for host trial allocation.

The proportion of participants who provided a 6-month CO breath measure was analysed using logistic regression. The odds ratio (OR), 95% confidence interval (CI) and p-value are presented.

The 6-month appointment due date was 183 days after randomisation. Participants who withdrew a month either side of the 6-month appointment due date were classed as withdrawn. The proportion of participants withdrawing from SCIMITAR+ in the two months after contact were analysed in the same way as the primary outcome.

A Cox Proportional Hazard model compared the proximity of the visit completion to visit due date (time in days). Participants who completed their visit before or on the due date had their time-to-visit set to 0.1.

Results

In total, 434 participants were randomised into this SWAT (n=286, 65.9% intervention group; n=148, 34.1% control group). Due to randomisation occurring at a single time point, eleven participants were excluded from analysis as they withdrew from SCIMITAR+ following randomisation but prior to being contacted for their 6-month visit. There were 423 eligible participants (intervention group n=279, 66.0%; control group n=144, 34.0%) ( Figure 1).

Figure 1. Study flow diagram.

Figure 1.

Open in a new tab

Overall, 87.5% (n=370) of participants completed the CO breath measurement at 6 months and it was unclear if there was a difference between intervention (88.5%, n=247) and control groups (85.4%, n=123) (3.1% difference, OR 1.29, 95% CI 0.71-2.33, p=0.41). It was unclear if there was any difference in withdrawals between trials arms (intervention n=7, 2.8%; control n=5, 3.5%; OR 0.76, 95% CI 0.23-2.44, p=0.64) or proximity of 6-month visit completion to due date (hazard ratio 1.07, 95% CI 0.86-1.33, p=0.55). Assessment of assumptions noted that the Kaplan Meier curves crossed, however Grambsh and Therneau testing did not indicate any statistically significant non proportionality (SWAT allocation p=.047; Main trial allocation p=0.52; global test p=0.63).

Discussion

It is unclear whether an additional £10 in cash increases the likelihood of participants completing a face-to-face follow-up, the proportion of the participants withdrawing, or have an effect on the proximity of the visit to the due date.

Strengths and limitations

A small positive difference was observed; however, despite the large sample size, the study was underpowered to confidently rule out a small ‘true’ effect. Due to the small effect size (3.1% increase in response) the cost per additional person attending would be in excess of £300.

Due to the sample size of this SWAT, it is most likely generalisable to the larger host trial population of patients with severe mental ill health disorders.

Data was not collected on how study staff followed the guidance on discussing the contingent £10 note to intervention group participants when arranging follow up visits. This may have diluted the effect of the intervention.

Conclusion

It is unclear if contingent financial incentives increase rates of face-to-face follow-up completion in this trial. However, there were sample size and power limitations. Future SWATs are needed to add to the evidence base.

Data availability

Underlying data

Figshare: SCIMITAR+ Trial: A randomised study within a trial (SWAT) of a contingent financial reward to improve trial follow-up - Data Set, https://doi.org/10.6084/m9.figshare.10060202.v2 8.

Reporting guidelines

Figshare: CONSORT checklist for SCIMITAR+ Trial: A randomised study within a trial (SWAT) of a contingent financial reward to improve trial follow-up, https://doi.org/10.6084/m9.figshare.10060202.v2 8.

Data are available under the terms of the Creative Commons Attribution 4.0 International license (CC-BY 4.0).

Acknowledgements

The SCIMITAR+ team acknowledge the support of the PROMETHEUS programme (MRC MR/R013748/1) in developing this manuscript.

Funding Statement

This work was conducted as part of the SCIMITAR+ Trial which was funded by the NIHR Health Technology Assessment Programme (11/136/52). The views expressed are those of the author(s) and not necessarily those of the NIHR of the Department of Health and Social Care.

The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

[version 2; peer review: 2 approved]

References

  • 1. Hewitt CE, Kumaravel B, Dumville JC, et al. : Assessing the impact of attrition in randomized controlled trials. J Clin Epidemiol. 2010;63(11):1264–70. 10.1016/j.jclinepi.2010.01.010 [DOI] [PubMed] [Google Scholar]
  • 2. Bower P, Brueton V, Gamble C, et al. : Interventions to improve recruitment and retention in clinical trials: a survey and workshop to assess current practice and future priorities. Trials. 2014;15: 399. 10.1186/1745-6215-15-399 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 3. Bailey JV, Pavlou M, Copas A, et al. : The Sexunzipped trial: optimizing the design of online randomized controlled trials. J Med Internet Res. 2013;15(12):e278. 10.2196/jmir.2668 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 4. Peckham E, Arundel C, Bailey D, et al. : Smoking Cessation Intervention for Severe Mental Ill Health Trial (SCIMITAR+): study protocol for a randomised controlled trial. Trials. 2017;18(1): 44. 10.1186/s13063-017-1789-7 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 5. SWAT 48 – Effects of a £ 10 note on retention. Reference Source [Google Scholar]
  • 6. StataCorp: Stata Statistical Software: Release 13. College Station, TX: StataCorp LP.2013. [Google Scholar]
  • 7. StataCorp: Stata Statistical Software: Release 15. College Station, TX: StataCorp LLC.2017. [Google Scholar]
  • 8. Arundel C, Coleman E, Fairhurst C, et al. : SCIMITAR+ Trial: A randomised study within a trial (SWAT) of a contingent financial reward to improve trial follow-up - Data Set. figshare.Dataset.2019. 10.6084/m9.figshare.10060202.v2 [DOI] [PMC free article] [PubMed]
F1000Res. 2020 Jul 20. doi: 10.5256/f1000research.27418.r67390

Reviewer response for version 2

Frances Shiely 1,2

Thank you to the authors for making these revisions and providing a detailed response to my original queries. I think the revisions have greatly improved the article and am happy to recommend it be accepted.

Is the work clearly and accurately presented and does it cite the current literature?

Partly

If applicable, is the statistical analysis and its interpretation appropriate?

Partly

Are all the source data underlying the results available to ensure full reproducibility?

No source data required

Is the study design appropriate and is the work technically sound?

No

Are the conclusions drawn adequately supported by the results?

Partly

Are sufficient details of methods and analysis provided to allow replication by others?

Yes

Reviewer Expertise:

Epidemiology; trial methodological reserach

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

F1000Res. 2020 Apr 24. doi: 10.5256/f1000research.23176.r62472

Reviewer response for version 1

Sarah A Rhodes 1

  1. There seems to be an error in this sentence in the abstract – ‘The difference (3.1%) was not statistically significant (p=0.36; OR 1.29, 95% CI 0.71-2.33, p=0.41).’ – two different p-values are reported and the incorrect one should be removed.

  2. This is a paper about the financial incentive SWAT and not the SCIMITAR+ trial – the title seems to suggest that this is the trial report for the SCIMITAR+ trial. Something like ‘The effect of contingent financial reward to improve retention; A randomised study embedded within the SCIMITAR+ trial’ might be better.

  3. This SWAT has minimal power to answer a research question by itself, but as the authors rightly say it is an important addition to the body of evidence. I find the term ‘not statistically significant’ potentially misleading because it seems to suggest that the intervention wasn’t effective whereas the confidence intervals are very wide, and appear consistent with previous results. I’d recommend removing reference to ‘statistical significance’ and instead using terms like ‘it is unclear whether….’, given the very wide confidence intervals.

  4. The description ‘This SWAT evaluated the effectiveness of a contingent financial reward - £10 cash in addition to a routinely provided £10 voucher - versus no contingent financial reward, on improving the retention rate in the SCIMITAR+ trial’ is difficult to understand. It seems like the intervention is the offer of a contingent financial reward rather than the reward itself (which happens after outcome measurement) – should this perhaps be ‘the effectiveness of the offer of a contingent financial reward’? The £10 voucher happens in both arms so perhaps this should say ‘versus the routinely rewarded £10 voucher alone’ – unless I’m misunderstanding what the control group is here.

  5. It says ‘contingent financial reward’ in some places and ‘financial incentive’ in others – are these the same thing? It will be helpful if one term were used consistently.

  6. It says that analyses were ‘on an intention to treat basis’ but 11 randomised participants that withdrew before the 6-month visit were excluded from the analysis; this is not really compatible with a true ‘intention to treat’ analysis which aims to preserve randomised groups. Perhaps clearly describe the analysis population rather than calling it ‘intention to treat’ – e.g. the analysis population was participants randomised to the SWAT who had not withdrawn by time X days before due visit. Consider a sensitivity analysis including all randomised participants.

  7. The observed effect size was 3.1%; there is lots of uncertainty in this estimate so ideally it shouldn’t really be used to infer the likely cost effectiveness of future studies – instead, I’d recommend considering each end of the confidence interval (or perhaps the upper end as a best-case scenario) when considering cost per additional person retained.

  8. I don’t understand the sentence ‘Due to the sample size of this SWAT, it is most likely generalisable to the larger host trial population of patients with severe mental ill health disorders.’ A large sample size does not guarantee generalisability, and as noted the sample size is not really large enough to draw conclusions about this population. I think this point needs to be constructed more clearly.

  9. For the secondary outcome of ‘proximity to due date’, I’m really interested to know whether the proportional hazards assumption holds. It would be helpful to include a Kaplain-Meier curve and/or some comment about assumption checking.

Is the work clearly and accurately presented and does it cite the current literature?

Yes

If applicable, is the statistical analysis and its interpretation appropriate?

Partly

Are all the source data underlying the results available to ensure full reproducibility?

Yes

Is the study design appropriate and is the work technically sound?

Yes

Are the conclusions drawn adequately supported by the results?

Partly

Are sufficient details of methods and analysis provided to allow replication by others?

Yes

Reviewer Expertise:

Statistics, randomised trials, embedded trials, meta-analysis.

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

F1000Res. 2020 Jun 15.
Catherine Arundel 1

There seems to be an error in this sentence in the abstract – ‘The difference (3.1%) was not statistically significant (p=0.36; OR 1.29, 95% CI 0.71-2.33, p=0.41).’ – two different p-values are reported and the incorrect one should be removed.

We thank the reviewer for this observation. The error has been corrected accordingly

This is a paper about the financial incentive SWAT and not the SCIMITAR+ trial – the title seems to suggest that this is the trial report for the SCIMITAR+ trial. Something like ‘The effect of contingent financial reward to improve retention; A randomised study embedded within the SCIMITAR+ trial’ might be better.

We thank the reviewer for this observation. The title has been amended for clarity.

This SWAT has minimal power to answer a research question by itself, but as the authors rightly say it is an important addition to the body of evidence. I find the term ‘not statistically significant’ potentially misleading because it seems to suggest that the intervention wasn’t effective whereas the confidence intervals are very wide, and appear consistent with previous results. I’d recommend removing reference to ‘statistical significance’ and instead using terms like ‘it is unclear whether….’, given the very wide confidence intervals.

We agree with the reviewers thoughts here. The manuscript has therefore been updated to remove ‘statistical significance’ and to make clear the lack of certainty. P values have however been retained for completeness.

The description ‘This SWAT evaluated the effectiveness of a contingent financial reward - £10 cash in addition to a routinely provided £10 voucher - versus no contingent financial reward, on improving the retention rate in the SCIMITAR+ trial’ is difficult to understand. It seems like the intervention is the offer of a contingent financial reward rather than the reward itself (which happens after outcome measurement) – should this perhaps be ‘the effectiveness of the offer of a contingent financial reward’? The £10 voucher happens in both arms so perhaps this should say ‘versus the routinely rewarded £10 voucher alone’ – unless I’m misunderstanding what the control group is here.

We have amended the manuscript in line with the reviewer’s suggestion to make clearer the control group.

It says ‘contingent financial reward’ in some places and ‘financial incentive’ in others – are these the same thing? It will be helpful if one term were used consistently.

We apologise for this inconsistency. The manuscript has been updated throughout to ensure consistency,

It says that analyses were ‘on an intention to treat basis’ but 11 randomised participants that withdrew before the 6-month visit were excluded from the analysis; this is not really compatible with a true ‘intention to treat’ analysis which aims to preserve randomised groups. Perhaps clearly describe the analysis population rather than calling it ‘intention to treat’ – e.g. the analysis population was participants randomised to the SWAT who had not withdrawn by time X days before due visit. Consider a sensitivity analysis including all randomised participants.

We can see why the reviewer feels that this is not strictly ITT. The study randomisation was done in a single batch, rather than when the participant reached the Month 6 time point, and so participants received an allocation when they were not technically eligible for the SWAT, hence the exclusion.

 

As a result, we suggest that the analysis in the paper is ITT in the sense that the analysis includes everyone who would have been eligible for the SWAT. We have sought to make this clearer in the paper for avoidance of confusion.

 

For added confidence, the analysis has been re-run including those who withdrew post randomisation, but pre visit. When these are included the findings do not change .

 

The observed effect size was 3.1%; there is lots of uncertainty in this estimate so ideally it shouldn’t really be used to infer the likely cost effectiveness of future studies – instead, I’d recommend considering each end of the confidence interval (or perhaps the upper end as a best-case scenario) when considering cost per additional person retained.

We thank the reviewer for making this important point.

As the manuscript states, the cost of retaining an additional participant would be over £300. This statement is still correct in the worst case scenario, and as such covers the uncertainty; whilst also covering the estimate that we have found in this study.

 

I don’t understand the sentence ‘Due to the sample size of this SWAT, it is most likely generalisable to the larger host trial population of patients with severe mental ill health disorders.’ A large sample size does not guarantee generalisability, and as noted the sample size is not really large enough to draw conclusions about this population. I think this point needs to be constructed more clearly.

For clarity, this section has been amended to reflect that results are only applicable to the study population.

For the secondary outcome of ‘proximity to due date’, I’m really interested to know whether the proportional hazards assumption holds. It would be helpful to include a Kaplain-Meier curve and/or some comment about assumption checking.

The results section has been updated to detail assessment of hazards assumptions.

F1000Res. 2019 Dec 3. doi: 10.5256/f1000research.23176.r56823

Reviewer response for version 1

Frances Shiely 1,2

Thank you for the opportunity to review this SWAT. I am in favour of publishing negative results as well as positive results, so my reason for rejecting this SWAT is not because the findings are negative. I wish to be clear on that. 

This was a SWAT within the SCIMITAR+ trial. The purpose of the SWAT was to establish if £10 cash + £10 voucher would increase retention rates when compared to £10 voucher only. SCIMITAR+ was a smoking cessation trial. The authors conclude that contingent financial incentives did not statistically significantly increase rates of face-to-face follow-up completion within the SCIMITAR+ trial.

In the background section of the abstract, you describe the SWAT as  “…versus no contingent financial incentive” which is misleading. There is a contingent financial incentive because you have a £10 voucher as standard. I presume it’s contingent on them turning up to the appointment? I had to go to the body of the paper to ensure I had the correct interpretation. You should describe the SWAT more accurately as “to establish if £10 cash + £10 voucher would increase retention rates when compared to £10 voucher only” and change this in the main body of the paper also.

I struggled with the design of this SWAT and what the investigators hoped to achieve by it. I don’t think the primary outcome was suitable to measure what they intended. The conclusion drawn in the abstract is unrelated to the primary outcome. The purpose of the SWAT was to look at retention. The participants were eligible for the SWAT if they reached the SCIMITAR+ 6-month follow-up. I’d like the authors to comment on this in relation to their SWAT. It could be argued that if they were already coming for the 6 month follow-up they were motivated to continue regardless of the extra financial incentive. It seems like 100% of those contacted for the 6-month follow-up turned up to their appointment. Therefore, 100% were retained. I don’t see the point then in making the primary outcome contingent on the CO breath measurement and was not surprised there was no statistically significant difference between the two groups. Your secondary outcome, the proportion withdrawing after the contact would be more relevant. This is supported by the fact you used the proportion turning up for the appointment as your concluding statement.

Comparing two incentivised arms at 6 month follow-up doesn’t appear to be the best use of the money. What would have been interesting to find out is if the additional money at 12 months would make a difference to retention? Perhaps let the intervention arm know at 6-months that at 12 months, in addition to the voucher they would get £10 cash. Then you could establish if the additional incentive was effective. 

I would have liked to know what you told the comparator group when you phoned them to come to their 6-month visit. Did you tell them they were getting a £10 voucher to give a CO. Did you make them aware of what the intervention group were getting or did you tell them about the SWAT at all?

The participants were already incentivised with the £10 voucher. What is the evidence base for thinking that an additional £10 cash would increase retention at 6 months? I note the Bailey et al. 2013 study but this was in relation to response rates for a postal questionnaire. 1

Is the work clearly and accurately presented and does it cite the current literature?

Partly

If applicable, is the statistical analysis and its interpretation appropriate?

Partly

Are all the source data underlying the results available to ensure full reproducibility?

No source data required

Is the study design appropriate and is the work technically sound?

No

Are the conclusions drawn adequately supported by the results?

Partly

Are sufficient details of methods and analysis provided to allow replication by others?

Yes

Reviewer Expertise:

Epidemiology; trial methodological reserach

I confirm that I have read this submission and believe that I have an appropriate level of expertise to state that I do not consider it to be of an acceptable scientific standard, for reasons outlined above.

References

  • 1. : The Sexunzipped trial: optimizing the design of online randomized controlled trials. J Med Internet Res.2013;15(12) : 10.2196/jmir.2668 e278 10.2196/jmir.2668 [DOI] [PMC free article] [PubMed] [Google Scholar]
F1000Res. 2020 Jun 15.
Catherine Arundel 1

In the background section of the abstract, you describe the SWAT as  “…versus no contingent financial incentive” which is misleading. There is a contingent financial incentive because you have a £10 voucher as standard. I presume it’s contingent on them turning up to the appointment? I had to go to the body of the paper to ensure I had the correct interpretation. You should describe the SWAT more accurately as “to establish if £10 cash + £10 voucher would increase retention rates when compared to £10 voucher only” and change this in the main body of the paper also.

We thank the reviewer for this helpful suggestion. We have amended the manuscript to make clearer the control group.

I struggled with the design of this SWAT and what the investigators hoped to achieve by it. I don’t think the primary outcome was suitable to measure what they intended. The conclusion drawn in the abstract is unrelated to the primary outcome.

The purpose of the SWAT was to look at retention. The participants were eligible for the SWAT if they reached the SCIMITAR+ 6-month follow-up. I’d like the authors to comment on this in relation to their SWAT. It could be argued that if they were already coming for the 6 month follow-up they were motivated to continue regardless of the extra financial incentive.

  

It seems like 100% of those contacted for the 6-month follow-up turned up to their appointment. Therefore, 100% were retained. I don’t see the point then in making the primary outcome contingent on the CO breath measurement and was not surprised there was no statistically significant difference between the two groups. Your secondary outcome, the proportion withdrawing after the contact would be more relevant. This is supported by the fact you used the proportion turning up for the appointment as your concluding statement.

We thank the reviewer for their comments here and apologise for any confusion.

Having reviewed, we believe that the conclusion does relate to the primary outcome given it was unclear, due to the wide confidence intervals if there was any difference in completion of a CO breath measure between the two groups.

 Following review of this, and comments from Reviewer 2, we can see there has been some misunderstanding. The study randomisation was done in a single batch, rather than when the participant reached the Month 6 time point, therefore at the time of randomisation it was no known if the participant would attend their 6 month visit. We have made this clearer in the randomisation and results sections.

We are clear how the reviewer has derived this information. Figure 1 demonstrates that 32 intervention and 21 control participants did not attend a face to face visit and so did not provide a CO breath measure.

Comparing two incentivised arms at 6 month follow-up doesn’t appear to be the best use of the money. What would have been interesting to find out is if the additional money at 12 months would make a difference to retention? Perhaps let the intervention arm know at 6-months that at 12 months, in addition to the voucher they would get £10 cash. Then you could establish if the additional incentive was effective. 

The impact of the contingent reward on subsequent follow up time points is an important consideration. The effects were not assessed in this study, but this would be important to include in any subsequent studies of this nature.

I would have liked to know what you told the comparator group when you phoned them to come to their 6-month visit. Did you tell them they were getting a £10 voucher to give a CO. Did you make them aware of what the intervention group were getting or did you tell them about the SWAT at all?

As noted in the intervention description, during the call to arrange a follow up appointment intervention participants were advised of the contingent £10 if they were willing to meet face to face and provide the primary outcome. The intervention section has been reviewed to make this clearer.

The participants were already incentivised with the £10 voucher. What is the evidence base for thinking that an additional £10 cash would increase retention at 6 months? I note the Bailey et al. 2013 study but this was in relation to response rates for a postal questionnaire. 1

There is much available evidence on strategies to improve postal questionnaire response rates, but limited evidence for strategies to improve face to face visit completion.

We suggest that strategies used for postal questionnaire response rates may have similar effectiveness for face to face visits, hence the completion of this SWAT. We have made clear in the background the justification for this.

Associated Data

    This section collects any data citations, data availability statements, or supplementary materials included in this article.

    Data Citations

    1. Arundel C, Coleman E, Fairhurst C, et al. : SCIMITAR+ Trial: A randomised study within a trial (SWAT) of a contingent financial reward to improve trial follow-up - Data Set. figshare.Dataset.2019. 10.6084/m9.figshare.10060202.v2 [DOI] [PMC free article] [PubMed]

    Data Availability Statement

    Underlying data

    Figshare: SCIMITAR+ Trial: A randomised study within a trial (SWAT) of a contingent financial reward to improve trial follow-up - Data Set, https://doi.org/10.6084/m9.figshare.10060202.v2 8.

    Reporting guidelines

    Figshare: CONSORT checklist for SCIMITAR+ Trial: A randomised study within a trial (SWAT) of a contingent financial reward to improve trial follow-up, https://doi.org/10.6084/m9.figshare.10060202.v2 8.

    Data are available under the terms of the Creative Commons Attribution 4.0 International license (CC-BY 4.0).

    Articles from F1000Research are provided here courtesy of F1000 Research Ltd

    RESOURCES